Well, I ended up solving the problem by myself. For anyone who has this problem in the future, the problem seems to be when you set the target bitrate and the maximum bitrate as high as they will go.
After lowering the target bitrate the problem went away.
Still not sure why the problem only existed on the Mac side…