I just tried it and it works for me.
You need to make absolutely sure that everything is routed and panned the right way.
In your example, and assuming stereo out, make sure that the Left part of the Full Mix is on A1, and the Left part of the M&E is on A3, and make sure that both are panned hard left. Right part of Full Mix goes on A2, Right part of M&E goes on A4, both panned hard right.
Apply the invert effect to both tracks of the M&E. Render. Now when you hit play, the M&E should be subtracted from the Full Mix. If that does not happen, either the panning is wrong, or the track assignment, or the M&E and the Full Mix do not line up correctly. Or maybe there are more mix differences, which will show up immediately.
If you need to convince yourself that it works, take any piece of audio, just one track, splice it timeline. Add the exact same piece to another track in the timeline. Pan both hard left. Playback, and you will hear the clip at a pretty strong level (adding up the two events of the same clip). Now apply the AudioSuite Invert to one of them and render. Playback and you will hear silence, and the meters of your Audio Tool will be pitch black.