How to rotoscope, for a beginner

[Al]

for this job, definately stick to AE rather than photoshop. And stay away from Paint; you want to stick to moving masks.

Assuming from your message that you understand how to make a mask and keyframe you mask to move with your object… but it looks less than crash hot.

My ten cent pointer for those new to rotoscoping is that before you start roto-ing your object, decide what you’re roto-ing. For example, your roto-ing the hand. Zoom right into the image so you can see some pixels defined, and pick your line to roto (probably the tan of her skin against whatever the bg was). Try not to roto that exact line, but pick a few pixels inside the line. I’ve done my roto time, and i will always roto just inside the object i’m roto-ing.

Once you’ve found the line you’re going to roto — zoom out a bit and follow your decision to the end –> most new roto-ers will change their mind of what they’re roto-ing half way through the shot. And i don’t mean they decide to roto the leg instead of the arm, but they choose a different pixel line to follow.

So maybe they started roto-ing a few pixels *inside* her hand, then they start roto-ing the actual line of her hand against the bg, then they might start roto-ing the pixels just outside her hand.

The other to watch for is your mask feathering. Always have your mask at a feather of 1 or 2 pixels, it avoids a harsh line. However, when theres fast movement (like the hand in front of the face moving quickly) — you’re going to need to feather your mask at this point. You might find it easier to ignore this until the end, then go back and add another mask that has a big feathered edge during the fast movement.

There are easier ways to do the above, but AE is not your tool. Heavy roto work is done in other programs, like Combustion, which has splines –> so you can feather your mask at each individual control point; rather than the mask as a whole.

Roto-ing is mind numbing but once you’ve got good at it, it’s an invaluable skill. I’ve seen people loose their jobs because they couldn’t get good at it; it’s a fundamental skill.

Good luck.

[Aconover]

Thanks, that was enormously helpful and gives me a great starting point. One place I’m getting tripped up, though, is exactly how to keyframe / move the mask points. For instance, through most of the shot, the hand is positioned more or less the same way — all five fingers are more or less extended. Towards the end of the overlapping portion (see note below), however, the hand dips down, forming more of a fist shape (i.e., no fingers). Should I create a new mask / new layer at this point, or should I keyframe the hand-shaped mask so that it forms a fist shape? To make matters more complicated, the character then stands up, again requiring an entirely different mask shape. What’s the best way to divide up these masks?

Note: The general shape of the shot is of one version of the actor sitting in a chair, and another version of the same actor squatting on the floor to the side / slightly behind him. At one point, the foreground character moves his hand in front of the other’s face — later, he stands up and begins to move across the second character. So I started with a simple “curved line down the center of the frame” mask, then added another version of the topmost layer (with the seated character), in which I masked the hand separately, starting when the hand first moves over the BG character’s face. Should I then create a third mask for the segment where the first character moves bodily across the second, or am I approaching this in the wrong way? Let me know if this is difficult to visualize and I will attach some screenshots. And thanks again for the advice!

[Al]

There are various ways people approach rotoscoping this kind of thing; most people and perhaps what is school/uni taught is that you should rotoscope everything seperately. Example; one mask for the arm, one for the leg, one for the torso, one for the head etc.

I personally avoid this technique as i find it too time consuming and generally painful. If my shape doesn’t change too much, i keep it all within the one mask. However, using AE for this (i use combustion for roto work), I would probably seperate the masks on the basis of the feathering issue i was explaining before.

SO you are approaching this the right way. With your hand that stays the same shape all the way through until the end when it makes a fist –> definately keep this in the one mask. A hand changing from outstretched fingers to a fist is not too complicated a shape change to warrant seperate masks.

Having said that; try and avoid the classic New to Rotoscoping Issue; too many control points. You want to minimise your control points drastically. Really really cut down on them as much as you can. Most newbies have far too many, which gets really complicated and plain ugly. The other thing to bear in mind with your hand, is if the hand doesn’t change shape until the end, keep your keyframes down to a minimal as well. Laying keyframes every single frame for every single control point is best avoided. It looks ok frame by frame, but when you watch it real time, it’s an unsightly mess.

And as per your note, yes, it sounds like you’re on the right track. Only mask when it’s needed and so if the body needs rotoscoping half way through then add a new mask for this time.

Hope that helps

[Aconover]

Thanks, muiisal — this advice was excellent, and really helped me get through the project after the first few atrocious starts that prompted this posting. There are still some details of roto that I need to work out — specifically, how keyframes relate to control points (does each set of control points constitute one keyframe position, or is each point keyframed individually, etc?), but those are things I can pick up myself — your advice definitely got me started on the right path. Thanks!