Need help with a matte

[Mike Fitzsimmons]

You might look at TrackX from Coremelt. they have great stuff.

[Mark Bowen]

Hi Mike,

Many thanks for that. Will take a look now.

Best wishes,

Mark

[Mark Suszko]

Sounds like you want to make the keys look like they are playing by themselves, like in “The Ghost & Mister Chicken” starring Don Knotts (A halloween favorite)?

I have an idea or two to make the hands disappear. Not sure how well they will work.

The first one is to use a chromakey effect on the hands, selecting their flesh tone as the matte target. You may need to adjust the shading sliders for contrast, brightness, color intensity, etc., and the result is going to look ugly but that’s okay, we just want the hands to key out. Next step is to take that generated matte and use it as a “holdout matte” in a new stack of video tracks. In a sandwich of tracks, the hold-out matte hides the hands and makes a “hole” thru to the next layer. if that next layer is an otherwise identical layer of the piano keys, the hands go invisible… or, more accurately, they go translucent.

But i see a problem if you are trying for the ghostly player piano keys thing, in that the replacement fill-in layer generated by the hold-out matte doesn’t have *depressed* piano keys, so it won’t match perfectly. I think my method would kinda work for you if you’re going for the almost-invisible “Predator” movie invisibility look… but I don’t think it will work 100 percent.

The other idea would have been to use AfterEffects’ new Sensei AI-driven background replacement tool to draw a border around the hands and let the machine replace it with it’s best guess of the keys underneath the hands, based on the surrounding nearby pixels. And maybe this will work on something as uniform as the keyboard… While this works great for fractal type textures like bushes, grass, trees, clouds, etc., I have not had the best luck with man-made objects and backgrounds trying this new tool. It takes a bloody long time to grind thru the computations, so test it on a a very few seconds of footage first, but, has the advantage of not making you sit and keyframe anything, though.

But we may be thinking about this all the wrong way, anyhow. If you want the keys to depress themselves without any fingers, I think you need to keyframe them in stop-motion fashion. Now THAT’s a tedious job! How I might approach that from a practical photography perspective is to first shoot a locked-off “plate” shot of the entire keyboard. Next, I’d use a small, inconspicuous wedge to hold individual keys down, and take screen grabs of all 88 keys, one at a time, this way. The next part would involve layering-in the specific down-keys from moment to moment, keyframed to the waveforms of the music. That’s a kind of “brute force” method, and painstaking. A better way would be to use a 3-D CGI keyboard, inset into the live plate shot to replace the real keyboard of the piano. There are MIDI scripts and clever plug-ins, I’m sure, to drive the piano keys in perfect synch to the score you want to play. I know they must exist because I’ve seen cool CGI animation videos where things like rolling marbles hit piano keys and drum heads in perfect timing. Where you go to get that advanced animation programming, I do NOT know – it’s pretty sophisticated and bespoke stuff.

[Mark Bowen]

Actually I want to do the exact opposite 😉

I need to keep the hands and get rid of the keys. Was just hoping there might be some way to mask out the keys (keeping the hands) by using some colour controls or something but seems like you can only use those types of things on effects applied to video.

Will have to check out Track X and if that (or something like it) doesn’t work then I’ll have to rethink.

Thanks for the post though, very much appreciated.

Best wishes,

Mark

[Blaise Douros]

The method that Mark is outlining is actually still 100% the best way to do this–and it will be easier than the other way around.

Now, caveat: I can’t see your footage, so there may be something preventing this, like black sleeves–those you might have to mask/roto. But hands over a black and white background ought to be easy!

All you do is add a Chroma Key effect, use the color picker to select the orange-y skin tones of the hand, and then play with the settings and range. Then, crucially: invert the effect. Normally it REMOVES all the color you select, like in greenscreen, but if you invert the effect: presto! Just hands.

You may get some artifacts from reflections on the black piano surface, but otherwise, I bet this is very doable. You will then have the challenge of any sleeves the performer might be wearing, and there’s probably no way except the rotobrush or fairly tedious masking to deal with them.

[Blaise Douros]

If you WERE trying to get the piano to play itself, like Mark Suszko is suggesting, you could just go shoot a Yamaha Disklavier for half a day. They play back MIDI files in real time, on a real keyboard, just like a ghost playing it. https://usa.yamaha.com/products/musical_instruments/pianos/disklavier/index.html

[Mark Bowen]

Hiya,

Thanks. Yes it’s the hands I need not the keys moving on their own. I’ve actually been a pianist since I was 3 (about 43 years now) and was very lucky to get to play on one of the very first grand pianos that could play back by itself. They’re pretty fun to watch those are!

Thanks for all the help everyone. I will give all of that a go and see what I come up with.

Best wishes,

Mark

[Mark Suszko]

I had a further idea about this piano problem and it occurs to me that the Deep Learning, AKA “Deepfake” technology would be great to use on this. Many MIDI music making programs featuring an animated piano keyboard that plays back visually as the music plays back. I know Garage Band does. Take that animated piano keyboard and apply it to the Deep fake program, re-mapping it to the real piano, and now you have the ghostly piano keys effect you really want.