IF you use solid/flat colors for at least one of the transition layer, you can follow that up, in a nested composition with Auto-trace to create your mask.
One major issue that may crop up is with the edges of the layers. I suggest that your layers and precomps be sized about 20% larger than the ‘final comp’ to preclude having to fix unwanted/enexpected edge issues after applying Auto-trace.
Lastly, the Wipe transition is a raster effect.
HTH
RoRK
Intensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE