sticking textures and images to moving face… Configurable DIY Snapchat filters

[Dirk De jong]

[Greg Sage] “Resulting cubist comp may or may not be recognized as a face by facial tracking softwares,”

yeah that’d probably be a huge problem for the social media filter authoring software face tracking (the cubist face)

[Greg Sage]

I have a weird situation (part of why I was trying to do in AE to begin with.)

I have a 4 minute clip of a cubist face I’ve composited from side and front shots of the same talking head in front of a green screen. Similar to this:

But it’s video, and he’s talking. Both shots are already motion stabilized, and I’ve already comped it together in AE. The cubist comping of the 2 shots requires warping each shot to get them to fit together correctly.

Resulting cubist comp may or may not be recognized as a face by facial tracking softwares, but it would be perferable to track it as such since that means textures and paint I overlay would help to hide the warps where, for instance, nose doesn’t quite move correctly with cheek as they’re comped together.

If necessary, I could track the front and side shots separately, paint them separately, then do the comping again from the painted versions of each, but then the warps would be more noticeable as they’d be baked into the comping.

Bottom line: I can either track and paint the comped cubist version (preferred path), or (if necessary) track the front and side shots separately (using either original footage or my pre-stabilized versions), paint them separately, then comp them together later. Depends really on whether software will allow me to track comped version.

So… would I then take the rest frame into PS, and paint it, then use that as the texture?

EDIT: Just thought of something. Maybe I’m looking at this all wrong, and there may be a third approach. The comping was a major pain because of all the skewing where even a slight turn in one direction translates differently in the front vs side shot. Since the head is being painted anyway, it might just be better to create a single talking 3d head driven by the front shot’s tracking data (and possibly texture information from side shot if it’s helpful for the head wrap)… then just render out the head mesh with the texture from both front and side angles and do the comping from that. In theory, it should be much more stable.

Speaking of stable: Trying to think through… where exactly in this workflow (or the one you mention with screen grab) would I lock down a particular tracking point if I want, for instance, to keep the left eye socket perfectly motionless so all motion translates from the center of the eye socket (not the pupil since it may move)?

[Greg Sage]

I find the layout of this forum quite confusing. Dbl post here was entirely unnecessary. Why is there no edit button?

[Greg Sage]

So in screen capture method you’re talking about… am I understanding correctly that the screen capture is then the painted version of the face over a black background?

Of course, that’s assuming I’m using a face it can track, and that I can find some way to do the side bit. It’s really just the nose and the alpha edge being used from the side, though, so I could just paint the nose separately, and use the alpha edge for the comping.

Just being able to paint the front face would get me halfway there.