A simple way might be to use the luminetri colour effect, there is a menu within it, where you can select by colour and it will create matte, you can then select show matte (making a the image greyscale) and use that to matte out an adjustment layer with levels effect on to effect the original footage at the bottom of the stack.
Or you can add a ‘hand drawn’ mask to the layer add a levels effect then in the effect properties (on the layer) use the compositing options and add the mask to it.
Or, use key light to key out the colour you want to keep, this will get rid of it, but then use an invert.
Or, there’s an effect called leave colour or something, it’s in the colour corrections part of the effects menu.
Hopefully one of these will get you some of the way.