This “weird masking” is due to Keylight taking advantage of Dual processor systems.
The top half of the image being processed by one and the bottom half being processed by the other.
When the ram usage is max’d out it drops a process, if you cut the length of the clip and purge caches before the job this usually free’s up some space.
This does not happen in AE 6.0.
If you still have problems contact the Foundry support.