Ideally compressors may need to be tweaked for animation because there’s large expanses of similar colors.
Handbrake might offer you more control but someone with animation experience might know the specifics. A long while back I had a friend who was the compressionist at the Cartoon Network and he said it would take hours of testing to come up with a custom setting.
Of course Vimeo and YouTube bulk encode so no matter how good you do, they’ll butcher it.
You could try uploading a ProRes version but I’m not sure if that would really be an advantage.
Also it’s not clear what settings you actually used. I had to make assumptions and even when you say “Vimeo guidelines” I can’t be sure you’ve interpreted them correctly. Of course I think their guidelines are for “live action” more so than your live/animation mix.
I do think you should have a look at Handbrake but keep in mind there’s little you can do to prevent the Online Butcher, if that’s where the problem lies.