Can’t you work with the original DNG’s in AE? I was under the impression that this was possible, but haven’t tried it.
If not, go for the highest bitrate possible, for instance prores444.
You might be able to improve the key by grading (adjusting the levels and pushing the saturation to ‘standard’ levels) prior to keying (no need for inbetween renders, just apply these effects on the same layer first).
Shooting flat scenes in Raw seems to have become the standard, but I’ve always wondered if it doesn’t ‘downgrade’ the images when it comes to visual fx work.
The way I see it: you can have only X-amount of data per frame. Every bit that you throw away when grading is a waste. If you could shoot so that every bit of data is necessary, than you’d have more data available to pull that perfect key…