What kind of settings in Reference you use?
I tried Blu-Ray H.264 High Resolution with some changes of default settings:
Program stream (.mpg) setting changed to Elementary streams (.264, no audio – I have a seperate audio file which is BD compatable in Encore)
Bitrate set from 1-pass VBR 20000-24000 to 2-pass VBR 30000-35000
Higher bitrate and 2-pass mode is chosen due to intensive film grain in original video file. Movie is shot on 35mm and it has some natural film grain, especially in darker shots and I want this grain to be reproduced as good as possible.
Input file is self-contained QuickTime with DNxHD 185 X 25p video stream in it. Output framerate in Reference is set to 50i.
Also the Encore project is set to Blu-Ray h.264 50i.