The uncompressed avi files might be choking your system, especially if you’re trying to watch them in quicktime. So it might be a playback problem.
Also, AE isn’t the best for compressing. Can you go out uncompressed in QT and then compress it down to your youtube settings using a compression application?
And on a totally different note, try adding a little bit of motion blur to your animation. That often gets rid of judder.
Good Luck,
Reiner
Mac Pro 2 x 3 GHz 6-Core Intel Xeon
32GB Memory
Dual-channel 4Gb Fibre Channel PCI Express card
Dell Display (23″ flat panel)
ATI Radeon HD 5770
AJA Kona LSi SD/HD capture card
Rourke 16 TB
Flanders 2460
SOFTWARE
Mac OS X 10.6.5
FCP 7
After Effects CS5
Boris Continuum
Sapphire Plug Ins
All Trapcode Plugs
Zaxwerks Invig