Looking at the example you just sent
1) You should work with 16b motion vectors pass (eg tiff 16 bpc) and a 16 bpc or floating point AE project (then also make sure there is no color management on the MV sequence). (zero motion is not properly defined at 8 bpc etc)
2) When you have multiple layers like you do, you sometimes with post-motion blur need to split in a foreground and a background pass (2 color renders) so the motion blur works as expected.
I am not sure what your animation looks like but in this case you might be able to do something with just an alpha of the foreground layer. You would apply RSMB as you have it now then overlay a second pass where you would cut out of the MV alpha the part that is not in the additional matte you rendered and motion blur just the foreground with that.
Pierre