[Steve Bentley]

Hasve you tried locking the mask before applying the puppet tool? With out a locked mask the puppet tool thinks you want everything and may be trying to make a massive mesh. With the lock on the mask it will make the mesh only within the mask and ignore the alpha.

[Steve Bentley]

Hey Kory, Not sure if you solved this – I just saw it.
You can make some expresso nodes so that at as certain keyframe or object position the position commands switch from one procedure to another.
You could also approach it from a character animation point of view – when a character passes an object from one hand (where that object has been following that hand for while) to the other hand where it continues to follow the new hand. Or where they set an object they have been carrying down and leave it.

[Steve Bentley]

The Unity engine is even faster with just 2D. Its a nice package because you can use art or timelines or programming or all three to achieve your task. And there is a free version! It plays well with others – even Flash! and Flash doesn’t like playing with anything!

You shouldn’t have to worry about lag at all. We were fretting over 14 milliseconds of projector lag so a second or so is an eternity.

You could also use C4d (if that’s what you are used to) to make elements that would get used on sprites in your real time particle package. Particular does this in AE – for instance it uses volumetric renders of clouds done in C4d – little snippet movies- and puts them on thousands of sprites to simulate volumetric smoke or explosions – its very convincing but doesn’t take the volumetric render hit.

There’s so much already built for the kinect (from a input/output/coding point of view), even a beginner can get the data they want from it and put it to use in a whole plethora of packages – and for what its doing its very very fast. You just have to explain to your significant other why you are making odd gestures in front of your “web cam” in the middle of the night.

If you are going to use guitars as a mapping surface you could ask the band to put little tracking dots on the guitars – these could be UV or IR sensitive, so you might not even see them (but the camera would) but at the very least they wouldn’t show up as traditional tracking dots. That would add another layer of object confirmation for the camera system and would help with orientation (when the camera sees the guitar on oblique angles – research “fiduciary markers”). You can also build a library of shapes – all the ways a guitar can be oriented. The camera whips through that library every frame and returns a “hey, I know what that is and which way its facing” set of data.

There’s a lot of library’s out there for face recognition and body recognition. And there was some new stuff released just the other day regarding real time face replacement for movies (think “Tarkin” in Star Wars). While you don’t need that level of sophistication, when you have that kind of algorithm, you can do what your are after faster than the blink of an eye. I’ll find the link and post it.

Its a good field to get into because AR and VR (in real time) are going to be all the rage when the computing power and algorithms get just a little bit better.

[Steve Bentley]

A couple of things to keep in mind. C4d is not a real time renderer. And while you may have a killer card with a bagillion cores that will play a sequence of things in real time, it probably won’t start in real time. Add to that delay, the lag of the motion tracking processing system and the camera sensor and its offloader, then add to that the lag of the projector and video display system. Any one of these is significant, put them together and it can really add up.
Even using a kinect there is a lag when the hardware or software aren’t dialed right in or dedicated to those tasks.

As much as Processing was a terrific tool and led to all sorts of great things like Arduino, I wouldn’t call it speedy. I don’t know if that’s the app itself or the compiler or the handoff that happens in between.

I’m not sure if Unity would be a good choice or not – its fast enough for gaming but there is a bunch of overhead there that you don’t need. But real time for sure and the renders get better every day.

If you are thinking about generating canned 3D movies in C4d that play over a live actor, that can work, but remember live people don’t have a smpte track so they don’t always stick to the plan – its why they have live orchestras for live performances because anything can and will happen to delay a cue. A live generated system triggered by both actor and tech are the best bet. But often you need to have something that just can’t be rendered in real time so you can supplement with canned things that the tracker places.

We’ve done tracking and real time reactive particle systems in both Java and C+, and compiled, they run at a decent clip (paired with a kinect, or with a camera(s) and some machine vision tracking). We’ve even done some in Flash (although it started to choke when the particles got too heavy or the stage got too big)

For those with way too much money there’s CoolLux (it may be under a different name now as they keep getting bought out) and Green Hippo. Both are happy to work with you to customize your needs.

But you can hack something together in just about anything that can take a camera or kinect input and output to a video card. Just check how fast that input and output happen because there is a lot going on there.

Again, I assume you are talking real-time – if not just say so and we can recommend some tracking solutions that don’t work in real time.

[Steve Bentley]

Hi Victor,
Is this posted in the right forum? You are talking about AE correct?
I haven’t see that problem myself but then I went in and tweaked the prefs the first time 2018 booted.
It could be in the general tab – the dynamic link item or it could be in the autosave tab: the oitem that tells ae to save a copy every time you hit render (in case it crashes while the render is going on).

[Steve Bentley]

Try this one on for size. This uses the clips you uploaded (the original footage I’ve squished the head out of to get an incorrect but clean background plate so the lighting is right) and the last raw footage clip.
I think it boils less than the one you showed us and certainly doesn’t suffer from the edge blockys. It does crush her hair a bit more but another edge hair layer compensates and brings some of the wisps back in. This layer actually looks better on top but it does change the color of her hair a bit. If you can live with that difference in color (if you didn’t see the original you wouldn’t realize) it brings more wisps back in when placed above the footage.
I’ve added some noise to the BG to match the noisy green screen footage.
Let me know of you need it saved back before AE 2018.

If its not good enough it may give you some ideas that you could tweak yourself.

12281_greenscreenheadcomp.aep.zip

[Steve Bentley]

I think my jegs got a little crushed in the upload (the irony is not lost on me) so the details i’m pointing out aren’t really that evident – let me know if you want the originals

[Steve Bentley]

So here’s a couple of things:
This is a green screen image with H264 compression (and pretty mightily compressed at that) and you can see everything is sharper but more than that, the visible blocks that make up the H264 compression are away from the hairline, out in the green area where subtle shades have to be decided on. All blocks change every keyframe but with the obvious ones out in the green you won’t see the key dance every 15 frames or so at the hairline.

In your footage there is no where near the clarity of detail and there are all these little smudges where the codec (or the camera) has tried to invent what it’s seen or average a bunch of data into a few blocks. The more noisy a shot is (by noise I mean lots of different textures and colors and details – in other words the wrinkles in the fabric) the higher the Mb/s it needs to make sure all of those details come out sharp and at 3.5 mb/s sec its just not enough, given the content of this shot. Any codec is going to start to sacrifice things (usually color first) in order to keep details it finds important. One of my clients loves h264 because at a low through put rate all her wrinkles disappear.

If you zoom in on your footage to this level and step through it frame by frame (no key or effects) you can see the edge of the hairline boiling and little areas of distortion going on where chunks of hair literally move over 16 pixels for a few frames and then move back or sections get averaged to keep the bandwidth. This may be function of the camera or the codec, I’m not sure, but this is also going on at the edge of the hair so you are keying a moving target. It’s almost has a shower glass effect.
Still playing but I thought I would share.

[Steve Bentley]

There is no set angle for the shutter to be set at in filming – its all a dance between how much light you have to work with, the F stop you need for depth of field, the “speed” of the film or CCD and the kind of effect you want (the battles in Gladiator were shot with a very very narrow angle of shutter and tons of light to give it a strobey look and reduce the motion blur to nill).
But Broadcast cameras used in a studio don’t really have a shutter per se. And even if they did I doubt it would be a rotary one. You match the blur so it matches by eye – the shutter angle is just a number so you know if you need more, you need to increase the angle. The great thing about this is if you really want a lot you can go to 1000 or more, where a real shutter can’t be open more than all the way (360).
We get a lot of camera data when we’re doing match moves but we rarely get shutter data even when its on film. So the eyeball and reference is the best judge.
If the shot is still bugging you it’s probably the mismatch between the speed of the animated element and the real wheel. The blur you have done is very passable.

[Steve Bentley]

Good point Mark. shadows really ground people in an environment and really make them appear to be “there”. The brain also uses shadows to determine volume so it makes them seem less like cardboard cutouts in a keyed shot.
You can use the Videocopilot VCreflect tool (I think its free) to do shadows for vertical people on an angled plane – we find its the most intuitive way to do fake shadows especially for when their feet move and lift. You could also put a 3D plane in there and light the keyed “cards” with a back light to cast a shadow but that has its own headaches.