The size issue is not wnat the videos play on, but what plays them. DVD? Hard drive?
If they will play on DVD, you have to make the videos at DVD resolution — use the DV preset in AE. You have NO choice in the matter.
The ugliness may be due to the MPEG-2 compression. You were right to start with the Animation codec, but it’s best to compress that Animation movie to MPEG-2 (for DVD) using a compressor that allows VBR, or variable-bitrate encoding. Squeeze or Compressor would be my choice.