No it’s not the 4K that’s doing it.
The best pipeline for AE and encoding is to first render out of AE as either an uncompressed format or something that is virtually lossless like ProRes or BlackMagic. Then take that rendered movie and use another tool to encode.
This is especially true with H264 which really needs to wade through every frame first to see what every frame looks like in order to get the best compression/size ratio. AE by it’s nature doesn’t know what’s coming up or what the last frame looked like, it takes each frame as it comes. So it can’t really do the 2 pass thing efficiently.
Premiere is also designed to cache elements and mix elements on the fly so you don’t have to wait for dissolves which require two streams at the very least.
Because AE assumes that:
1) there will be more streams of content than any reasonable processor or card or codec could handle in real time and will have effects that require subpixel sampling and 3D matrix math and 3rd party effects, so it takes one frame at a time, and
2) there are better tools to be doing editing and compressing in. Making a product do all things for all people leads to an overly complex interface and a result that isn’t as good as one that specializes.
While you can do many of things that Premiere can do and Encoder can do inside of AE, it’s not the ideal program for it. The same reason the Premiere doesn’t have the flexibility to do the things AE can do is the same reason that AE is a more complex program (or perhaps differently complex) so that it can do highly specialized things, but can’t do something as simple as dissolve two layers in real time. And compressing isn’t one of those features either.
Apart from saving the drive space (and drive space is cheap now) I can’t see a good reason for a render and compress pipeline out of AE. Do you really want to waste extra time with compressing a shot that isn’t right yet?
You are right though, given what AE is being tasked with I don’t find it takes full advantage of the procs or threads or even ram. There was a time that no matter how much ram you had AE could only use a few gigs of it. I have a feeling this is an infrastructure issue. Remember Adobe bought this program (it used to be called COSA) and they are probably still dealing with legacy code that is cheaper to piggy back on than rip it out and start again. Is it perfect? Not by a very long shot. But given the price and what it can do…
If you must encode from AE, set a watch folder for your compression software of choice and have it start it’s thing once AE has finished the render. If you really need to increase your speed use FFmpg. After using it you will have the same reaction to its speed as you are having with AE vs premiere – why isn’t everything this fast? Plus you can transcode from one format to another without creating an intermediary format and without losing quality.