After Effects is not intended for compression – only recently in last 3 years or so has dynamic link allowed this to be a possibility. Not sure why you are getting an error. I would export out a master file in quicktime ProRes 422 HQ. Then bring that into Media Encoder and set to .h264. May seem redundant, but other than a workaround for your immediate problem, this also allows you export multiple tests to find the right bit rate or to export to multiple sizes or codecs if need be. I always make a “master” QT lossless file and then compress deliverables from that. That way its only 1 long render and then multiple short compressions. Anyway, see if going direct out of AE first works. If AE still errors when rendering to ProRes, then you have a problem with the comp itself. Good luck!