i find that doubling the target bit rate and using that for max is helpful for maintaining quality h264 compressions. thats admiddatly going a bit overboard but its a good place to start. when you are going for small file size VBR is your friend, just pump up the max bitrate and it will sort it out for you based on your target.