[Joe Marler]

[Gillian Marsollier] “I have all of the same media files on my drive, so I am not sure why it won’t relink…I have noticed as well that her drive has an extra file with ‘final cut optimized media’. It is GIGANTIC and seems redundant as all of the original files are there as well and they are 1/4 of the size. Can we deleted the optimized media, and how to we avoid this in the future if we don’t need it?…Is there an incredible online course out there that covers all of this stuff in great detail? I feel like we are missing some very key things in this area!”

If you have the same media files, it normally will relink. During the relink dialog, if you point it to the root of the volume it will if necessary search every file on the disk to find them.

However the files must be exactly the same to relink. If she generated optimized media, by default that is inside the library. But whether the optimized media was inside the library, outside the library, or deleted from the library after the original files were moved somewhere else — the library should relink to the original files if pointed to that disk volume.

Re whether you need the optimized media – depending on the machine, original media resolution and codec, this may not be needed. The main reason for optimized media is editing performance but it is roughly 6x the size of H264 files. In those cases proxy media can be better since they give similar or better performance but are a small fraction of the size. However you must remember to switch the viewer in FCPX to proxy, then switch back to original media before exporting.

If your original media is 1080p, most computers running FCPX do not need original or proxy media to edit this. If the original media is 4k H264, then optimized or proxy media might be needed to obtain good editing performance.

You can usually safely delete proxy or optimized media since they can be regenerated from the original files. It can take a while if the files are large.

Ripple Training has an excellent class on media management:

https://www.rippletraining.com/products/final-cut-pro/media-management-in-final-cut-pro-10-4/

[Joe Marler]

[Chad Greene] “About four years ago I chose to switch our shop to FCPX instead of Premiere. We have been happy with that decision. We cut long form documentaries. Recently we have been handing pieces back and forth between multiple editors and have found it to be bumpy. Now I am being asked to reconsider Premiere.

Do any of you work closely with multiple editors on projects? Do you have success in X? What are your technics? How do you share proxies with offsite editors?

Do you prefer Pr? Why/how?”

I also used Premiere CS4-CS6 and now FCPX for large documentaries with multiple concurrent editors. I still have CC and use it occasionally for testing but not production.

FCPX is really good at unscripted projects with high shooting ratios like documentaries. There’s a good argument that the rapid skimming and tagging features are more important than the magnetic timeline.

The FCPX multi-editor workflow can be complex when external proxies are involved and when the collaborators are not constantly connected via high-speed LAN. The external proxy workflow was obviously not a design priority since it involves some limitations, convoluted steps and there’s limited UI support. E.g, relink is not reliable when using external proxies. However we make this work fairly well, including sending metadata and timeline updates via XML. In this scenario some editors have proxy only media and others have full resolution.

While “collaborative editing” often refers to timeline editing, with FCPX (esp. on docs) much of the time and labor on a large project is during the lengthy pre-edit organizational phase. There has been much discussion of continuously connected, LAN-type collaboration and less discussion of geographically distributed internet-type collaboration.

The collaborative editing workspace can be categorized in four quadrants: the organizational phase and the editing phase, and for each of those two, the workgroup can be geographically co-located on a LAN or geographically distributed on a slower link, possibly intermittently connected. The procedures for each of these are different.

Each of those four cases can be further classified by whether the collaboration is partitioned and non-conflicting or concurrent and potentially conflicting. E.g, separate assistant editors each working in their own event to rate/keyword metadata are non-conflicting even though their work may be concurrent. Different editors on the same timeline isn’t supported, although they can each be working on separate projects which might be a previously-agreed portion of the overall timeline.

Our latest documentary is quite large – 230 hr of 4k H264 material comprising about 7,500 clips and 20 terabytes inc’l proxies. This is mostly in a single library, segregated by about nine different events. The geographically distributed editors all have a copy of the media, some of them proxy only.

Due to the amount of material, the pre-edit organizational phase is significantly more labor-intensive than the edit itself. So in this case we needed multiple collaborating assistant editors to devise and apply a consistent keywording and rating system. Once devised, that work could happen in parallel across multiple events and the updates could be gathered via event XMLs.

A limitation is FCPX does not properly handle XML updates if the underlying clips have duplicate filenames. Even though it applies a “uniqueifier” suffix upon import, it still gets confused if sending metadata updates via XML and will create spurious duplicate clips. Thus our workflow must include a careful renaming of all files before import to ensure they are globally unique. Use use “A Better Finder Rename” for this: https://www.publicspace.net/ABetterFinderRename/

The data org includes syncing multicam clips and selecting the proper audio source. This mostly happens before editing begins but it can overlap. Once in the edit phase, we can still (carefully) send updates via XML such as a missing multicam or a new keyword collection. Updating metadata when two people are in the same event can be tricky, so we use MergeX which has some limitations but works pretty well: https://www.merge.software

There is no support in FCPX for “event locking” or anything like that. We must manually coordinate who is working in what event. However it can be done and works fairly well.

The latest Premiere CC has “collaborative editing” called Team Projects. However it seems focused on the sequence or timeline phase of post production. It includes conflict resolution if two editors edit the same timeline, but apparently has no fine-grained locking within the timeline. IOW it knows if two editors alter the same timeline and either keeps one, the other or both versions. You can do that with FCPX (IOW edit the different timelines in the same event) and send XML files, but it must be done manually. Here’s a user review “Premiere Proxy Workflow with Adobe Team Projects”:

https://www.youtube.com/watch?v=NZJAD3C4tqc

The advantage is it doesn’t require additional hardware. For FCPX in a LAN environment there is the Lumaforge solution: https://lumaforge.com/workflow/

Others have experimented with various FCPX methods: https://www.fcp.co/final-cut-pro/articles/1938-collaborative-workflow-with-final-cut-pro-x-first-working-steps

I think it’s fair to say that Premiere CC now has a built-in solution that doesn’t require extra hardware. I don’t know how good a fit that is for a large documentary that consumes more labor hours in up-front tagging and classifying than it does in timeline editing. The Premiere solution is cloud-based, so it could supposedly work across geographically distributed editors. This likely assumes they all start with physical copies of the same media. Premiere now has proxy support.

I’d like to see more collaborative workflow features in FCPX, and not just those designed for LAN-type collaboration with co-located editors. This would involve better support for external proxies, relinking, fixing the XML problem for duplicate filenames, some type of check in/ check out locking system for assets, something like MergeX built in, and similar to Adobe would probably be cloud-based. However up to now most recent FCPX new features have been client-facing and UI centric. Maybe that’s in line with what users want: Out of the 128 feature requests on https://fcpx.tv/top.html, the word “collaborate” is not mentioned.

However in the real world, collaborative work is important — even for smaller projects.

Some contents or functionalities here are not available due to your cookie preferences!

This happens because the functionality/content marked as “Google Youtube” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.

[Joe Marler]

[Michael Gissing] “In that time frame I’ve moved from SD to 4k and have real time playback whilst grading. Standing still in terms of processor and going backwards in terms of GPU is unacceptable to me over the past ten years.”

My experience is even the highest end desktops don’t always have adequate performance — because of the now-ubiquitous use of 4k acquisition, the higher shooting ratios now common, and incredible computational demands entailed by 4k H264.

When I used Premiere CS5 in 2010 on standard-def DV, it was fast on a desktop or a top Windows laptop of that era. It was great to just drop in camera files without transcoding and edit with high performance. Shooting ratios were lower then, so that helped.

Today both Premiere CC and FCPX can struggle on 4k H264 — on any platform. Adobe’s “Mercury” playback engine is no longer like quicksilver when editing that format, esp. for multicam.

The worst combination is 4k H264 on Premiere on a Mac because Adobe doesn’t even use Quick Sync on Mac. But with *either* FCPX or Premiere CC on the latest hardware, we are often knocked back a generation to the previous workflow of “transcode before edit” — not to a mezzanine codec but to proxy.

Traditional GPUs cannot help this because the core algorithm of long GOP formats is inherently sequential. GPUs can muster thousands of lightweight threads which can attack certain parallelizable tasks, but many tasks cannot be (or have not been) parallelized. Highly compute-intensive plugins such as Neat Video, Digital Anarchy Flicker Free, and Imagenomic Portraiture only partially leverage the GPU or not at all. E.g, Neat Video is slower if configured to use the iMac Pro Vega 64 GPU than if using all CPU cores and no GPU. The problem isn’t the Vega GPU and it can’t be fixed by a faster GPU or an eGPU.

My documentary team can produce 1 terabyte of 4k H264 per day. I’d like to screen dailies without building proxies, but it’s just too slow, especially on a laptop. I’ve only tested one machine that can scrub though single-cam 4k H264 with moderate smoothness using FCPX, and that’s the top-spec 2017 iMac. It is way faster than the 12-core D700 Mac Pro and faster decoding 4k H264 than the 10-core Vega 64 iMac Pro. So we’d have to take a 2017 iMac 27 on site to get adequate editing performance to screen dailies without proxies.

For those doing scripted narratives or other productions with lower shooting ratios which can use ProRes or similar acquisition, even a laptop is pretty fast — at least with FCPX. Lower-compression intra-frame codecs are more an I/O problem than CPU. A top-spec MacBook Pro using SSD or Thunderbolt RAID storage can handle those codecs pretty well.

What I’d like is a desktop machine that regains the same timeline performance on today’s 4k H264 that we had in 2010 on Premiere using standard def DV. That machine does not yet exist, at least from Apple — even using FCPX.

So far Intel has remained absolutely intransigent on adding Quick Sync to any Xeon except the 4-core version. On the iMac Pro this forced Apple to write to AMD’s UVD/VCE transcoding hardware, which is better than nothing but thus far slower than Quick Sync on handling 4k H264. There are lots of factors at play here, but if Apple controlled their own CPU design for desktops they wouldn’t restricted by Intel’s decisions.

CPU design has now reached a point where major performance gains are difficult — as measured by traditional metrics such as clock speed and Instructions Per Clock. It’s unclear whether an A-series architecture would greatly improve this, as the problems seem fundamental. However — there are still major gains possible using “heterogeneous” processing — IOW specialized subsystems like Quick Sync. There are probably other software functions amenable to silicon-based acceleration — provided the chip vendor was cooperative and software harnessed this. Using an A-series CPU in a Mac would allow Apple to control both hardware and software.

The initial rumors of A-series CPUs on Macs focus on lower-end laptops, and those are a natural fit for some future iOS/macOS integration which a common instruction set might facilitate. The improved power consumption would help battery life. However this might be a testing ground to evaluate future use of higher-end A-series CPUs in higher-end desktop machines.

[Joe Marler]

You are evidently using external media, which is common for larger projects. If all the media was internal to the library (IOW a “managed library”) it would never be lost, no matter what drive volume or computer it was placed on. However managed libraries can be unwieldy to handle due to the large monolithic size.

For libraries using external media, if the files are placed on a different volume name or folder path than the original, they must be relinked. This normally isn’t difficult, since FCPX can search the entire volume and relink all media files in a single step.

If your collaborator sends you an updated library you will need to repeat the relink process. If the media folder paths are the same and only the volume name differs between the machines, it’s sometimes easier to just rename the drive volume to the same name on the destination machine.

This is covered in MacBreak Studio #273, starting at about 04:10:

https://youtu.be/ZFbtN1aTZh8?t=249

Some contents or functionalities here are not available due to your cookie preferences!

This happens because the functionality/content marked as “Google Youtube” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.

[Joe Marler]

[Michael Gissing] “this was my response to the article claiming this was big for DSLRs. I don’t think this is the market at all…”

The article never stated this was big for DSLRs. Rather, he said it would be big for some filmmakers who currently use upper-echelon DSLR or mirrorless cameras like the GH5. That category of user might be interested in ProRes RAW on a future GH6 or Sony A7SIII. He didn’t mean people will shoot ProRes RAW on a Nikon D3400.

The class of DSLR/mirrorless camera that might use this will not be impeded at all by SD card data rates. Those cameras already use UHS-II or XQD cards.

[Michael Gissing] “….They are not cameras that allow RAW sensor data to be output to external Shogun recorders so it will only work with internal recording if they want it too or with hacks like magic lantern maybe…”

Yes, that is a good point. You can’t get RAW sensor data from HDMI.

[Michael Gissing] “…Will some DSLRs adopt this? Sure but for all existing DSLRs it is unlikely…”

I think that is what he meant by referencing the GH5 and “other small form factor cameras”. He meant the category of filmmakers who are using that type of camera and already dealing with 400 Mbps data rates, yet only getting intra-frame H264 in return. The Sony FS5 is a “small form factor camera” that requires 600 Mbps for 4k XAVC-Intra. For those users, incrementally stepping up to 700 Mbps and getting internal RAW could be useful.

[Joe Marler]

[Michael Gissing] “DSLRs with ProResRAW? I think not. Their onboard cards are not up to the rigors of this sort of data rate.”

[Joe Marler] “So it doesn’t seem the data rate of current UHS-II cards is an impediment to ProRes RAW.”

[Michael Gissing] “…Yes some newer DSLR cameras and cards can support the required data rate. But a single 128 card shooting these codecs in 4k will last 22 minutes max. If ProResRAW is viable then why aren’t they shooting ProResHQ 4k already? Because most DSLR cameras need more record time and correct me if I’m wrong but the A7 Sony cameras don’t support high data rate codecs, especially ProRes…Panasonic DSLRs? More likely perhaps but the DSLR market is not really the target for this codec in my opinion….”

I don’t see the point about *current* DSLR/mirrorless card data rates having anything to do with recording ProResRAW internally. Those cameras don’t record ProResRAW internally because it was just announced, and their imaging pipeline doesn’t support that — not because current DSLR cards (CF, SDXC UHS-II, XQD) cannot support the required data rate.

Those cameras will likely not be updated via firmware to support internal ProRes RAW, it will probably require a redesign. When that redesign is done — for any which might support ProRes RAW — they will obviously use the correct card type, such as SD UHS-II or XQD. Those card types — commonly used in DSLR and mirrorless cameras already — support the required data rates.

The SD-size 256GB XQD card supports nearly 400 MB/sec right now and is used in the Nikon D4, Nikon D4s, Nikon D5, Nikon D850 and Nikon D500. So cards with the required data rate for ProRes RAW are already in wide use — in DSLRs.

The Panasonic GH5 handles 400 Mbps right now, which is nearly the data rate for 4k ProRes RAW.

The ProRes RAW announcement initially assumes people will use *external* recorders on whatever cameras they have, in which case the internal recording rate or card capacity isn’t an issue.

Re recording time and card capacity, 256GB UHS-II and XQD cards already exist and are used in DSLR and mirrorless cameras today. According to a graph in the Apple ProRes RAW white paper, data rate for this may be roughly equal or a little more than ProRes 422. According to the previously-posted ARRI chart, recording time at 4K UHD to a 128GB card is about 27 minutes. From this we might roughly calculate existing 256GB UHS-II or XQD cards would support 54 min recording time per card using ProRes RAW. I don’t see 54 minutes per card as a big restriction.

There’s a good question about how many DSLR and mirrorless shooters would shoot ProRes RAW, whether internal or externally recorded. I agree it’s not for everybody, but they are already shooting 4k 10-bit 4:2:2 All-Intra 400 mbps on the GH5.

For years they’ve been shooting 3.5k 10-bit lossless RAW on the 5D Mark III via Magic Lantern. That data rate is about 720 Mbps to the internal CF card.

Here is a feature documentary shot using RAW video on the 5D Mark III:

https://www.youtube.com/watch?v=k3rJiDN7cao

Some contents or functionalities here are not available due to your cookie preferences!

This happens because the functionality/content marked as “Google Youtube” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.

[Joe Marler]

I have a 10-core Vega 64 iMac Pro. It’s very quiet — the fans rarely spin up even under sustained FCPX transcoding. It has a UHS-II SD card reader, 10 gig ethernet and lots of ports on the back.

I’ve seen tests showing it’s much faster on an all-ProRes or RED RAW workflow than a top iMac.

However most of my acquisition and editing is 4k H264, and the picture is more mixed. It is much faster on this media than the 2013 12-core D700 trash can. So it’s a good upgrade for anybody with aging Mac Pros.

While Xeon does not have Quick Sync, FCPX is apparently using AMD’s UVD/VCE hardware transcoding acceleration. However in some tests (at least on this version of FCPX and macOS) it’s not faster than an i7 2017 iMac. When editing a 4k H264 timeline it is subjectively no quicker or more responsive than the 2017 iMac. It cannot smoothly edit multicam 4k H264 without proxies, no different than the 2017 iMac.

Below are some tests on my 10-core Vega 64 iMac Pro vs. 2017 i7 iMac 27 (min:sec)

iMac Pro export 60 sec 4k/29.97 H264 to 4k H264 Fast Encode: 00:45
2017 iMac export 60 sec 4k/29.97 H264 to 4k H264 Fast Encode: 00:38

iMac Pro export 60 sec 4k/29.97 H264 to 4k H264 Best Quality: 01:25
2017 iMac export 60 sec 4k/29.97 H264 to 4k H264 Best Quality: 01:11

iMac Pro export 60 sec 4k/29.97 H264 to 4k ProRes 422: 00:23
2017 iMac export 60 sec 4k/29.97 H264 to 4k ProRes 422: 00:36

iMac Pro export 60 sec 4k/29.97 ProRes 422 to 4k ProRes 422: 00:10
2017 iMac export 60 sec 4k/29.97 ProRes 422 to 4k ProRes 422: 00:13

Imac Pro create optimized media from 06:09 4k/29.97 H264: 02:05
2017 iMac create optimized media from 06:09 4k/29.97 H264: 03:40

(I don’t have the numbers for creating proxies from 4k H264 but the iMac Pro is a little slower than the iMac)

iMac Pro export 60 sec 4k/29.97 ProRes to 4k H265 8-bit: 00:43
2017 iMac export 60 sec 4k/29.97 ProRes 422 to 4k H265 8-bit: 02:16

iMac Pro export 60 sec 4k/29.97 ProRes to 4k H265 10-bit: 22:02
2017 iMac export 60 sec 4k/29.97 ProRes 422 to 4k H265 10-bit: 32:47

Neat Video 4.7, optimal config for iMac and iMac Pro, 4k/29.97 H264:
iMac Pro: 5.29 frames/sec
2017 iMac: 2.76 frames/sec

Digital Anarch Flicker Free on 10 sec of 4k/29.97 H264:
iMac Pro: 05:09
2017 iMac: 06:31

Imagenomic Portraiture skin processing on 10 sec 4k/29.97 H264:
iMac Pro: 01:07
2017 iMac: 01:10

Comments:

– FCPX is obviously not using hardware acceleration for HEVC/H265 10-bit export on either iMac Pro or iMac
– The 2017 top-spec iMac is a little faster at 4k H264 export than the 10-core Vega64 iMac Pro
– On effects, the iMac Pro performance advantage varies widely depending on the specific effect
– The iMac Pro is faster than the iMac at export from 4k H264 to 4k ProRes
– The iMac Pro is faster than the iMac on an all-ProRes workflow

iMac Pro: 10-core Vega64 iMac Pro, 64GB RAM, 2TB SSD, macOS 10.13.3, FCPX 10.4
2017 iMac: 4.2Ghz i7-7700K, 32GB RAM, Radeon Pro 580, 2TB SSD, macOS 10.12.6, FCPX 10.3.4
Camera and codec: Sony A7R2, XAVC-S, 4k/29.97 H264 100 mbps 8-bit 4:2:0

[Joe Marler]

[Michael Gissing] “DSLRs with ProResRAW? I think not. Their onboard cards are not up to the rigors of this sort of data rate. There’s a reason why non DSLR cameras go for CFAST and SSDs. To get from H264 at data rates under 50Mb/s to ProResRAW around 200Mb/s is not insignificant”

The Sandisk Extreme Pro UHS-II 128GB SD card in my Sony A7RIII has a tested write speed of about 250 MB/sec, and my iMac Pro’s UHS-II reader can import that at nearly 300 MB/sec.

https://www.cameramemoryspeed.com/reviews/sd-cards/sandisk-extreme-pro-300-mbs-uhs-ii-128gb-sdxc-memory-card/

This user measured 284 MB/sec transferring a video file from a UHS-II card via Finder on his iMac Pro: https://9to5mac.com/2018/02/01/imac-pro-uhs-2-sd-card-workflow-video/

ProRes RAW data rates are less than ProRes 422 HQ, which is about 734 megabit/sec or 92 megabytes/sec for UHD 4k. So it doesn’t seem the data rate of current UHS-II cards is an impediment to ProRes RAW.

https://www.arri.com/camera/amira/workflow/amira_workflow/image_format/codecs/

[Joe Marler]

[Bob Zelin] “You DO know, that unless Apple is planning on buying Intel, there will be no more Thunderbolt on those machines. What about your Thunderbolt investment – just throw that out ?”

The below article implies that Apple is now free to develop their own Thunderbolt chipsets or just integrate that logic on future Apple CPUs. This would cost less than previous Apple computers where they had to use Intel chipsets plus pay a licensing royalty.

https://arstechnica.com/gadgets/2017/05/intel-to-make-thunderbolt-3-royalty-free-in-bid-to-spur-adoption/

“Intel says that it is going to make the Thunderbolt 3 specification available on a non-exclusive, royalty-free basis. This will enable third parties to integrate the interface into their own silicon, opening the door to, for example, AMD systems with Thunderbolt 3 support and cheaper chips for the device end of the cable.”

[Joe Marler]

[Oliver Peters] “‘ve had much better luck when I cut stuff on the attached Promise RAID (additional locally attached storage)…I’m at NAB this coming week, so I intend to have a conversation about these issues with the QNAP folks.”

This might indicate a system-layer problem, not one with FCPX. However the system layer (macOS and the driver stack below that) should handle gracefully any network or disk I/O issue. It should report an error, a timeout, etc — not crash or hang.

Unfortunately it is common for silent errors or unexpected behaviors in either network or disk I/O to destabilize an app or the system. There are various threading APIs in macOS, but essentially the app is submitting many overlapped, ie asynchronous I/O calls, so there are a bunch queued up waiting for a completion signal. Each of those must also handle exception cases such as completion too slow, timeout, abort from the app, etc. All data structures, memory, handles, etc. must be cleaned up in all exception cases for each pending I/O — even if a multiple exception happens. E.g, if the I/O is slow to complete then the app issues a cancel. All this must happen in a thread-safe manner.

The bane of software testing is these asynchronous exception cases where the backout code path must work perfectly, yet it is often difficult to trigger this using normal test suites.

Just because FCPX crashes in your environment and other NLEs don’t doesn’t necessarily mean FCPX is at fault. There are various software “load paths”, and FCPX might be stumbling into on. E.g, consider if there was a bug in the code path for accessing Quick Sync. FCPX might crash on this, yet Premiere would not since it doesn’t use Quick Sync on Macs. Something like that might be happening in the network I/O realm.

Your scenario happens when executing rapid JKL commands when the storage is on a specific type of NAS. That could imply it’s triggered by a certain I/O pattern in combination with some unexpected response or exception case from the NAS. If this was understood it might be possible to write a small multithreaded test program which rapidly reproduces the problem, hopefully leading to quicker resolution.

It would be interesting if you could borrow or evaluate a totally different type of NAS such as a Lumaforge JellyFish, put your data on that and see if the problem goes away, stays the same or whatever.

In my case I’m using several different Macs, different H264 codecs, different RAID arrays, and all local Thunderbolt storage. It happens over multiple macOS and FCPX versions, so there is no obvious commonality. It is loosely related to timeline complexity as number of edits accumulate.

The FCPX data integrity seems quite good, as I almost never lose any work, despite all the crashes. However I’m always concerned that a spate of crashes has injected some damaged data which then makes subsequent crashes more likely. Unfortunately there is no FCPX database verifier.