Not an expert on this, but I think @TV needs a new profile level for encoded video. When they added 640×480 support for iPod Video the encoding profile changed from “baseline 1” to “baseline 3”, so even if your video had all the correct bit rate, size, etc. if it didn’t have the correct profile it would get rejected by the iPod.
Again, similarily, the first software to encode it properly was Quicktime’s mysterious “Encode for iPod” setting. Once people started to figure out what changed, the new profile was updated in other software and then EVENTUALLY Compressor got updated. I think it took a while for that upgrade to come out. Check out ffMPEGX and VisualHub to see if they support it yet.
This is from the manual:
Video formats supported
H.264 and protected H.264 (from iTunes Store): Up to 5 Mbps, Progressive Main Profile (CAVLC) with AAC-LC audio up to 160 Kbps (maximum resolution: 1280 by 720 pixels at 24 fps, 960 by 540 pixels at 30 fps)
So it looks like the CAVLC is the new profile that needs to be supported.
Also note that 720p24 is supported, but NOT higher framerates. If you want to encode 29.97 video you are limited to SD….for now