If you are willing to invest in third party plugins, it can be done, if not, there is no way of doing it with just plain AE. It’s still gonna be a tough one since the expression would need to calculate the luminance with respect to how the layers overlap.
Mylenium
[Pour Myl