-
How do I motion capture and replace a partially obscured object in greenscreen video?
Hi guys,
I am an Adobe Creative Cloud member, so I have After Effects CC. No commercial plugins beyond what comes with the package – i.e. for instance I only have the “AE” version of Mocha, not Mocha Pro.
I have shot a video of myself showing a white piece of paper from above, with a “greenscreen” below it. The paper is shown for a little bit, then I throw it out of frame.
Now, the greenscreen part of it, I think I can handle. I’ve had a play with Keylight and it seems like I will end up with something that works.
However, what I really want to be able to do here, is to replace the white paper with the red X on it with a Photoshop image I have of approximately the same dimensions (I have several of the images and the videos, and the final composition will look like someone looking at a bunch of pictures, then throwing them off screen).
Here is the video. Never mind the vigilito:
https://www.youtube.com/watch?v=bAxD5AhLI1o
I have about 15 minutes of experience with Mocha AE, and I am able to motion track the paper as long as movement is slow. However, if you study the video you will see that as the card is thrown, the motion becomes so fast that motion blur keeps mocha from doing its job;
I guess I can move the track points manually at that point, since there are only a few more frames where the position of the paper is relevant/visible.
However, this is where my knowledge ends and the questions pile up.
1) Is there a way to get Mocha to actually be able to capture such “large” motion as my clip shows? I’m thinking it would do a better job than me trying to place the points manually, but I realize that quick motion and motion blur makes it pretty hard.
2) Should I capture shape data, or should corner pin data be sufficient? I don’t really care if the paper bends slightly etc. – I’m happy for the Photoshop file to keep it’s shape, as long as it looks like it is being held and thrown by the hand.
3) The piece of paper is white, and so is the border of the Photoshop file (the images look kinda like classic polaroids, but not entirely). Should I try to remove the motion blur from the paper and let after effects add its own? The results don’t have to be ultra-realistic, but I’m willing to put in the work to make it look good, preferably real.
4) When doing the replacement, is there a simple way to keep the Photoshop file “under the thumb” as with the piece of paper? Or will I need to animate a mask?
If anyone could help me, I would be really grateful! Concrete advice, tutorials, anything is welcome! I’m not even on a budget, so as you guys can probably see, low quality video camera, sub-par lighting, I can’t afford mocha pro, but I’m hoping this should be possible with “plain” After Effects CC.
Thanks guys!
Best regards,
Rune Jacobsen
—
Rune Jacobsen -
9 Replies
-
Mocha should be able to handle the replacement – corner pin is what I would try first.
As far as the last few frames go, when the paper goes off screen, you should be able to do it manually and add Pixel Motion Blur to the PSD file to match the real life motion blur. There are hundreds of tutorials on object replacement with Mocha – a quick google search should provide you with quite a few.Tudor “Ted” Jelescu
Senior VFX Artist -
Thanks man!
I’ll look into learning more about mocha and corner pin, and definitely follow your recommendation about object replacement tutorials. It helps to get nudged in the right direction by someone who knows the terminology.
Thanks again!
—
Rune Jacobsen -
Here are my answers to your numbered Qs –
1) Large movements are usually OK for mocha to handle at default settings. However, when tracking rotations larger than 15 degrees across two frames, you will need to invoke mocha’s Angle property. It doesn’t look like you will need it here but just in case – -> mocha Tracking Large Rotation tutorial – https://www.imagineersystems.com/videos?tagFilter=tracking+rotation&sortBy=Newest
2) Shape data (in conjunction with tracking data) is required if your target does not conform to a four-sided object/shape which can be described, perfectly, by a Corner Pin effect.
3) You can export tracking from mocha and allow AE’s built-in motion blur (MB) to kick in as and when appropriate. If you’re exporting Corner Pin data from mocha, then ensure that you pick the tracking data export option that supports MB for Corner Pin. Exporting transform tracking data automatically provides the properties for AE’s MB feature.
4) There is no easy way. Ideally, you would have used a color that is easily keyed. White isn’t ideal but certainly doable – just a couple more steps to the workflow. To create a matte for the paper, you’ll need to extract a luma matte that isolates the paper from the rest of the video. You can use the Extract keyer for this.
You should be aware that you will require THREE major components in the final composite – (i) thumb (ii) insert layer (polaroid) (iii) fingers other than the thumb. I have numbered them from top-down in terms of their layer hierarchy, in order to complete the composite.
HTH
RoRKIntensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACI -
Hi Roland,
Thanks, that was very helpful. I’ve made a bit of progress, but not entirely there yet.
I will go trough the tracking large rotation tutorial in a few moments, just wanted to get this out there since I’m a new user here and still under moderation. 🙂
So here are my current batch of questions:
1) I did my tracking, and was reasonably happy with the result, even if I had to do some stuff manually (didn’t do the large rotation tutorial yet), so I went back to After Effects and pasted the tracking data (with motion blur) to the layer containing a test image with the same proportions as the finished ones will have (once I get my hands on them). I had to remove the keyframes for the anchor point, and move it back to the center coordinates of the comp (960×540, as this is a full HD project), and then it looked like this:
https://dl.dropboxusercontent.com/u/7958042/motionTracked.png
(I deliberately chose a frame where the original paper was approximately straight. I did actually paste the data from the first frame).
If I move around in time, the polaroid follows along as expected. Motion blur looks beautiful. However, as you can see in the image, scale and rotation are a wee bit off. Of course, there are key frames all over position, scale and rotation, in addition to the corner pin effect, so I can’t just scale and rotate as far as I know. How should I go about handling this situation?
2) I understand that I require three layers in the composite. Will I need to separately track the thumb shape, then export that as shape data, and use it as a mask for a duplicate of the composite layer? Or is there a more technically correct name for this process? If you are aware of any tutorials on the subject, I would be happy to know about that too!
Only two points this time, must mean I’m getting closer! 🙂
Again, thank you very much for the awesome help!
—
Rune Jacobsen -
[Rune Jacobsen] ” How should I go about handling this situation?”
Make a backup of the insert layer and delete the KFs for Scale & Rotation. If the results are better then great. If not – then try this on the layer with all the KFs -> Move the CTI to frame 1. Create a Null Object and place it in the middle of the “X” visual. Then Parent the Insert Layer to the Null Object. Keyframe the Null Object’s Rotation and Scale as and when required, to perfect its placement.
[Rune Jacobsen] “I understand that I require three layers in the composite. Will I need to separately track the thumb shape, then export that as shape data, and use it as a mask for a duplicate of the composite layer? Or is there a more technically correct name for this process? If you are aware of any tutorials on the subject, I would be happy to know about that too!”
If you already have the shape data or Corner Pin data of the white paper, you should be able to use it as a matte to isolate the thumb.
I suggest that you try out the composite hierarchy on a single frame. If it looks good on a single frame then there is a strong possibility that the hierarchy is correct and the mattes you are using are also correct.
Here’s a mocha tutorial on a tracking task with object occlusion. The finger in this example is moving and the background doesn’t lend itself well for a simple luma matte extraction (although it can be done twice, once for each of the white and black areas). Hence, Martin had to track and then roto the finger. This tutorial is the closest in terms of subject matter to what you’re doing that I can think of.
HTH
RoRKIntensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACI -
Roland,
Thanks. After fiddling a bit with the null, I got my image more or less perfectly following the paper tracking – the only thing that is “off” is when the card is thrown. Even after putting the angle up to 50 (which was the highest I was allowed to enter), it loses the card once the hand motion gets too quick.
I thought I would then do it manually – as mentioned, there are not a lot of frames left where the paper is visible at that point. So I moved the points for each frame – this made new keyframes in Mocha.
However, if I now export the tracking data to AE, everything is fine until the time where I start adding keyframes. At that point, the tracking data seems to jump around randomly, and it certainly doesn’t match where I put my points inside mocha AE.
Is moving the points of my x-spline the right way to handle this? Or should I learn to use the AdjustTrack feature (another thing I am clueless about)?
Also, the tutorial you mentioned about tracking with object occlusion sounds really interesting – but it seems like you didn’t include the link. Could you please post that as well?
Thank you a huge bunch – you’re a true lifesaver!
—
Rune Jacobsen -
Sorry about the missing link, here it is -> https://vimeo.com/8057500.
With regards to the manual tracking, I’d do it in AE. Start off by removing the transform and Corner Pin KFs ONE KF from the point where you intend to start manual tracking. Then use Position and Rotation to move the layer off screen. When doing this, DO NOT do this one frame at a time – that’s unnecessary work. Start off by looking out for natural KFs in the motion of the white piece of paper. Create KFs on these natural KFs first. Then fill in KFs between these natural KFs until the motion looks good.
If Motion Blur looks incorrect or messes up the motion tracking, then set Shutter Phase to -1/2 of Shutter Angle – you do this in the Advanced Tab of the Composition Settings dialog window.
HTH
RoRKIntensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACISome contents or functionalities here are not available due to your cookie preferences!This happens because the functionality/content marked as “Vimeo framework” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.
-
Hi Roland,
I know this is ages ago, but just wanted to let you know that thanks to your advice, I managed to get where I needed to be. It looks way beyond anything I could have hoped for without your help. Huge thanks for your great help!
All the best,
Rune Jacobsen
—
Rune Jacobsen -
Dang! It’s been awhile indeed! 🙂
I’m glad you’ve worked it out – this was quite an epic thread.
Cheers
RoRKIntensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACI
Reply to this Discussion! Login or Sign Up