Phil,
Funny, I was thinking of doing the exact same thing as you with my home videos. I did a sample video today and now I am having the same issue(garbled output) using H.264 when playing in VLC Player(my favorite player).
I am using Windows 7 and it plays fine in WMP and Quick time but not in VLC Player. I also uploaded a clip to Vimeo and viewing is still “garbled”.
Let me know if you find anything out.