H.264 may be using 4:2:0 or 4:2:2 chroma subsampling (depending on profile). That will give you really hideous stair-stepping — especially if the original has undergone multiple generations of lossy encoding.
If it’s possible to get a higher-quality original, that’s the way to go. If not, you can exchange some of the stairstepping for softness by blurring the chroma (Channel Combiner for RGB -> YUV, Channel Blur for increasing Green and Blue (now U and V) blurriness, then another Channel Combiner for YUV -> RGB).
Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events