I know this is an older post, but hopefully this will help you or someone else in a similar situation.
When you get into heavy film effects and wild edits you need to give the encoder a lot of leeway to analyze the footage and nail down the best encoding for each segment. You won’t be able to get very good results with a hardware encoder unless it can do a bunch of passes to figure out where the tough parts are. I would recommend Cinemacraft’s CCESP as it can do as many as 99 passes to analyze, re-encode, and tweak the tougher parts. Leave the GOP structure open and the encoder will be able to make a few short GOPs where necessary on the tougher quick edits and leave the rest long (15 frame) for efficiency.
The only real catches with CCESP is that it runs on Windows and costs $2000. If you are using an Intel Mac you can run it under Bootcamp just fine. As far as the $2000 part, well that will be worth it once you see the results.
I’ve been very underwhelmed with the results of Compressor…too much macroblocking and pixelization for my tastes especially on saturated stuff and bitrates below 7.