I can’t tell you what your doing wrong, but, may I make a suggestion.
Export your video as a sequence of frames, set up a batch process in Photoshop to apply the charcoal effect..
The frames in the filmstrip are going to be so small, by the time you apply the effect, you’ll probably end up with something undesireable.
Good Luck