Reassign different source media file to entire sequence while preserving all edits

[Joseph Owens]

[Elijah Lynn] “it appears that FCP removes the pulldown, the other section looks very stuttery like there is a framerate mismatch, it is the same source file for the length of the sequence, they both have the same clip attributes and all numbers in the motion tab are whole numbers. I am lost as to why one section looks great and one section looks horrible, this issue could end right here if I knew the answer to this question (& could solve it).”

“Pulldown” is introduced into field cadence when making the transition from 23.98P to 29.97 interlaced. FCP doesn’t actually do the pulldown correctly because the cadence has been destroyed by the editing process. Been through this a million times, as I used to run a telecine operation.

Four “film” frames become ten video fields. There are several ways to accomplish this: SMPTE standard is 3:2:3:2 (B-Frame dominant), while what is in general practice is A-Frame dominant (2:3:2:3). There is Advanced Pulldown, which is 2:3:3:2 (which is rather clever for a number of reasons), and then there is what Final Cut does, which is 2:2:2:4, which is unacceptable for anything but previews, but that’s all it can do, because nearly all NLE’s are not field-aware. However, your HDV source material is 2:3:2:3, complicated by the fact that it is IBP-Long GOP. That it has been edited that way has introduced some serious errors for ever going back to 23.98P.

If you don’t recognize the ABCD notation:
The first 23.98 frame is mapped to the first two video fields (one ‘whole video frame’) and that is an “A” Frame. If film is shot at 29.97 and transferred to video at 29.97, then every frame is an “A” frame.
The next 23.98 frame, the “B” Frame is actually mapped to three video fields, in fact one-and-a-half whole video frames, and straddles video frames 2 and 3. A “C” frame is mapped to the next 2 fields, and straddles two video frames — the second field of Frame “3” and the first frame of Frame “4”. The last 23.98 frame in the sequence, the “D” frame, straddles field 2 of Frame “4” (Video) and both fields of Frame 5.
If you edit 29.97 material that has 2:3 field cadence, then every time you make a cut, you have an 80% chance of disrupting the ABCD field order. That is why some of your clips look great, and some of them look stuttery. If you apply the same “reverse telecine” dominance to all the clips, then you will get converted footage that has the wrong cadence progression.

If you used SHAKE to remove the ABCD fields, then you would realize that you have to reset the removal dominance to whatever is appropriate for that specific clip. Editing, statistically, is a random reordering process, and that is what you’ve got — random field cadence. You are correct that a re-edit, at 23.98, with material from which the pulldown has already been removed, is probably the only thing that will fix this. The Final Cut manual specifically states that you should remove the 3:2 pulldown from this kind of material prior to doing any editing, so that you are dealing with the native framerate of the source material, and editing “whole frames” instead of disrupting the pulldown.

jPo

You mean “Old Ben”? Ben Kenobi?

[Elijah Lynn]

Wow, this is an awesome answer Joseph!

[Joseph Owens] “You are correct that a re-edit, at 23.98, with material from which the pulldown has already been removed, is probably the only thing that will fix this”

I still need clarification on one more thing though. If I export an EDL from my 29.97 sequence to a new 23.98 sequence would that get me “close” I am trying right now to do this but keep getting the error “error importing EDL”, I know there are probably a lot of things that could cause that, and EDLs are probably a whole new bag of tricks but that would be good to know if there is any way to transfer the “essence” of the edit to the new sequence even if I have to go and manually look at every single edit and adjust, it would be easier than manually recreating the sequence.

This footage was shot in non-drop frame if that affects anything.

Thanks for your explanation!

[Elijah Lynn]

Thanks for your response Dave!

We shot in non-drop so would there be any hope of transferring a rough EDL to the new frame rate?

I at least feel relieved knowing that this is the correct way to do it now.

[Elijah Lynn]

Looks like EDL is pretty old school and isn’t really going to work the way I envisioned. Seems a better way would be to export a quicktime and make it half translucent and use it as a reference clip and then do the manual edit.

Gosh what a way to learn a lesson, nothing like on the job training!

Thanks again Joseph for your very through explanation, I really do appreciate that knowledge. Especially the mystery of why certain parts look stuttery and some don’t, that would have stayed with me for years if I had never found the answer to it!

[Elijah Lynn]

Okay, this is a bit of a sub discussion but is relevant to this thread so I will post it here. I was wrong about this:

[Elijah Lynn] “4. Moving on though, a possible solution we have arrived at is to take the source HDV .mov files and remove the pulldown using compressor, which takes forever and makes enormous files, but it works. I take the source footage (HDV,Mpeg-2 .MOV) and remove the pulldown via compressor by using the “reverse telecine” setting in “frame controls” > “deinterlace”, the resulting file shows in QT as 23.98 and has no “interlaced look”.”

It is actually pretty fast, much faster than realtime, for some reason the 8 core cluster I originally tested this on took 50 minutes to remove the pulldown and convert to ProRes HQ for a 3 minute test clip. I suspect the cause to be I/O related since it is on network storage. However, I have discovered something that is fascinating and completely defies logic of the Qmaster render farms we use. Everyone who is doing pulldown removal like this should take note of this as this could save you 2-35x in time spent!!!

==========================

Here goes:

3 minute test clip, HDV 1080i60 24p source footage, run through compressor on 4 different setups. The Raid 5 mentioned below is +200MB read/write. When I say “local” it means on the computer.

Setup 1 – 8 core local cluster using network storage- render=approx. 50 minutes

Setup 2 – 8 core local cluster using raid 5 local storage = 4minutes 25 seconds

Setup 3 – No cluster, “This Computer” only, using Raid 5 local storage = 1 minute 33 seconds

Setup 4 – 16 core cluster on Gigabit ethernet, 8 core controller is on raid 5, slave is on standard HDD = 4 minutes 56 seconds

Setup 5 – Same as above but over VOIP 10/100 switches = 8 minutes 51 seconds

==========================

As you can see above, in Setup 3. when submitting the batch to “This Computer” it removed pulldown in a 3 minute clip in 1 minute 33 seconds!!! And get this the CPU usage of all 8 cores when rendering that batch was only at 60-70%, Setup 2 had all 8 cores cranked to 100% and still took approx. 3x as long!!! This just goes to show how huge of a difference networks and I/O speeds make a difference. I ran all these test multiple times to make sure I wasn’t crazy in the head.

One thing to note is that in Setup 5 the slave and cluster controller were hooked up directly to our VOIP phones which only have a 10/100 (100 Mbit) switch in them, when this happened the CPU usage of the slave was only around 10-20 percent. When I physically connected them to Gigabit (10/100/1000) via a crossover cable all 16 cores were at around 95%-100% during the whole render, yet with 16 cores it took 4m56s!!!

In the end just the computer with no clusters involved was 3X faster than a 16 core cluster at warp speed & approx. 35x faster than a 8 core cluster on networked storage. Mind boggling… but also makes a ton of sense!

It is my suspicion that this is only the case since it is doing pulldown and it may just be able to allocate I/O resources better when not dealing with distributing tasks. I don’t think I would ever see this happen if it were actually doing transcoding and since removing pulldown isn’t really transcoding it is just a matter of how fast the I/O is.

So, I hope this comes in handy for anyone who has to remove pulldown on a ton of clips. I went from thinking it was going to take about 25 hours per 90 minute clip (4 days!!!) to doing them all in under 4 hours just by doing a test, and it is funny because my only intention was to reduce the time from 25 hours to 13 hours by using a 16 core cluster vs an 8 core one!!!