It’s possible that the jittery playback is a result of the speed of the drive(s) it’s stored on. Lossless compression produces huge files compared to compressed formats, and you need very fast drive(s) to shove that amount of data through. If you’re sure that drive speed is not the issue, open the movie and step through it frame by frame. If you find any duplicate frames (eg. in QuickTime or FCP you’re pressing the right arrow on the keyboard, and you find you’ve pressed it and nothing changed) then yes, something odd has happened in After Effects. If on the other hand, each press is accompanied by the expected frame change, then it’s the speed of your drives that’s the issue.
You could, instead of using lossless, output from AE using one of the ProRes codecs. They generally play back OK, produce a smaller file and are so slightly compressed as to be unnoticeable.
Unless you’re going to be outputting to tape, you could just ‘live with’ this stuttery playback in the knowledge that once you output your final compressed version, it should be fine. You could do a test, by compressing your stuttery movie into something like H264 and playing that back. If that plays fine, then you can be sure that your uncompressed version is fine. Then delete that H264 test version – you definitely don’t want to be editing/compositing with that.
– Paul