Personally, I’m a fan of h.265/HEVC, as in case of a quality loss, the blurring effect is more pleasant than the blockiness that h.264/AVC gives you… I would do .mkv container with h.265 (at above 5mbps, above 10 preferred for higher quality), and AAC 6+ channel audio (at least 32kbps per channel). That is just me, and only if your media/devices support it. Most devices in the past 18months or so will support HEVC for 4K material. The codec was designed to yeild better results for a give bandwidth over avc/h.264. Main concerns are getting enough room on a Bluray disc as well as online streaming at a reasonable/practical rate.
Compared to h.264, you should be able to use half the bitrate for similar results, or equivalent bitrate for better results. On the higher end the trade off becomes less of an issue. As I said though, I prefer the blending to artifacting. If you go too low on the bitrate, one of the first things you will notice is that body hair tends to blend in, instead of being able to pick it out. At least that’s the thing that I tend to notice first in test cases. If you use a constant quality of <= 23, you should be okay. (larger number means more loss, smaller size, and it’s a progressive scale). I actually will go lower than that for a lot of my rips, as I don’t mind the lower quality, again relative to h.264, and it reduces the space on my nas… I use about 1200-2500kbps for 1080p content, it will vary based on CQP (constant quality) needs. I also use NVENC, which isn’t as good for size/quality trade off as say x265 is.
Just mt $.02, I tend to be more concerned with size… and as mentioned doing test encodes of more complex scenes will tell you what will work and what won’t, it’s very subjective.