I am having a similar problem: I have a comp with solids, text and vectors inside and I am using a 3D camera do get closer. A null controls the camera and my zoom is linked to camera-to-null by a expression that inverts its value, so, my zoom is locked on the null object.
Any time I change the camera position to get closer to this comp, it gets pixelated.
Now I tried a bigger comp to be zoomed. It gets better. If I turn deep of filed of, it improves a lot, but it is not the same thing…
If I turn on the vector quality (the little engine like icon next to layers name), my comp disappears…
What I am doing wrong?