[Joe Marler]

[Oliver Peters] “Yep, that’s slow. All I know is that it’s considerably slower than Premiere Pro doing the same relinking task.
“

I just did a smaller test of about 85 GB and 199 4k files, and Premiere CC 2017 is super fast at relinking — however it is also somewhat “dumb”. If you have lots of duplicate file names in different folders, as is common with Sony cameras (C00001.mp4, etc), it just grabs the first set of matching filenames — whether they are correct or not. If you point it to the right subfolder, then it’s both fast and correct since (a) you helped it, and (b) it was a simple case of a single folder name being different.

By contrast FCPX is doing some kind of more extensive match algorithm. I don’t know if it is hash or CRC or what. This takes considerably longer, but it always finds the right files, even if there are many duplicate filenames scattered throughout the disk volume in a greatly differing folder structure than the original.

This enables you to point FCPX at the root and it will do whatever it takes to find the correct match. There are cases where the source/destination structure is greatly different and the recipient may not know exactly what all those changes were — only that the media files are there, somewhere. I don’t think Premiere can handle this situation, but I have not tested it extensively.

The problem with FCPX is you can’t turn off that stringent file matching algorithm. Even in the above simple 85GB case, relink takes several minutes. There is also some kind of I/O constraint because the relink happens much faster on my Thunderbolt 2 SSD array than my spinning array. It also starts doing some kind of searching the moment you point it at the drive volume, without waiting until you navigate to the subfolder and press go.

In your case you have the exact same file structure and only the disk volume name is changed. It would be nice if you could somehow override FCPX and tell it “trust me, everything is the same so use a faster method”.

[Joe Marler]

Sorry, meant I did NOT do timed relink testing on earlier FCPX versions, so I don’t know if the current performance is worse or not. However it subjectively does seem kind of slow. 10 min to relink 2100 files in their own 1.5 TB folder on a Thunderbolt 2 RAID array seems a bit long. That said, relinking at least seems very reliable.

The ideal workflow would avoid relinking by renaming the volume and/or folder but that’s not always possible.

[Joe Marler]

[Oliver Peters] “I seem to recall that in the past, I would be able to launch the library from the new volume and FCPX was smart enough to relink everything. I may be wrong, but in any case, it no longer does that. You have to go through the relinking routine. BTW – that’s take a lot longer than it should.”

I don’t recollect it ever doing that. However if you rename the volume it should not require relinking at all. Maybe you are thinking about a case where the volume name was the same? However I did some tests on 10.3.1 (see below) and relink of a large library does seem slow, although I did timed testing on earlier versions.

Relink is supposedly faster if constrained to a folder tree vs searching a large drive containing lots of non-video assets. However my tests didn’t indicate that much difference.

I just tested a relink on a large project we shot last week — 2.3 terabytes totaling 199,000 total files and about 2100 4k video files, totaling about 50 hr. On a top-spec 2015 iMac 27 with media on a 16TB Thunderbolt 2 RAID-5 array, a full relink took 13 min to locate the files and about 60 sec to relink the 2100. However I had a lot of time lapse stills on that same volume (thus the 199k files) which weren’t part of the video data and that probably slowed down the file location search.

I then repeated the relink test, first separating out the video only files to a separate folder tree. But this only improved the relink speed to about 10 min, which is still a long time for 2100 files in their own folder tree.

The FCPX code doing the file location does not manage the UI well, especially during the initial “Looking for matching filenames” phase. The UI is unresponsive for long periods of time and could give the impression the process is irrevocably hung. During this phase it doesn’t do any I/O and each CPU core is at about 50%. The lack of I/O coupled with fairly low CPU could indicate an internal bottleneck, possibly serialized access to certain data structures — IOW Amdahl’s Law.

The UI then enters the stage of “verifying files for compatibility”. This can take several minutes on a large library, even on a fast machine and disk subsystem. During this phase the UI updates some (although still poorly) and the progress bar can misleadingly stay on one file.

It then starts doing a lot of small I/Os — the trace utility bitesize.d indicates an I/O histogram with distribution centered around 8k bytes. This might be FCPX using the Core Data SQLite database to update or build some table of the updated file paths, but I’m only guessing. But the initial lengthy CPU-oriented phase is not I/O limited.

I think for your case the easiest and fastest solution is just rename the volume using Finder>Get Info. If that is not possible then you’ll have to relink it.

I don’t have time to test it on my SSD RAID array but I don’t think that would improve it much because the majority of the waiting time is not I/O constrained. It appears they need to improve the algorithm.

[Joe Marler]

[Oliver Peters] “in comparing the two across several different machines, I find that when I do trimming via dragging clip edges, Premiere is a lot more responsive. This is not a new issue in my opinion.

The only time I feel FCPX truly lives up to its potential is when everything is SSD – computer hard drive, storage, etc. For example, it’s very slick on my MBP, but less so on a trash can connected to an external RAID – even via Thunderbolt. I always feels like I am outrunning the speed of the UI when I makes changes in the timeline. I don’t feel that way with Premiere.”

Premiere generally has a “crisp”, direct-coupled UI feel, but in some cases this can degrade much worse than FCPX. OTOH the FCPX UI is often severely degraded by computational load from plugins. A plugin like Digital Anarchy’s Flicker Free will cause the UI to lag severely. This isn’t pure CPU contention, since macOS UI events outside FCPX are still processed. Rather the UI event queue in FCPX is somehow in the same loop as back end plugin computation. I’ve never seen Premiere do that.

FCPX 10.3 has significant UI performance issues with stacked timelines (ie connected clips), which can cause several seconds of lag when dragging clips. This will probably be fixed pretty soon.

That said, FCPX (at least when running on identical Mac hardware) is much faster than Premiere in terms of viewer update rate and general keyboard responsiveness on a H264 4k timeline. On my top-spec 2015 iMac 27 with media on a Thunderbolt 2 SSD RAID-0 array, Premiere CC 2017 can lag several seconds updating the program monitor when fast forwarding on a 4k timeline. That is using Mercury GPU rendering and 1/4 resolution display. Viewer updates on FCPX are much faster, and that’s not including skimmer performance which is probably 20 times faster. Premiere lags severely on JKL input when FF/REV on a 4k timeline — it’s so sluggish it feels like the keyboard is broken.

Premiere does keep the playhead moving more smoothly than FCPX in such cases — almost as if they dedicated a separate high-priority thread to just doing that. However you can’t really tell where you are because the program monitor is like a slide show.

Improving performance in cases like this is often hard, tedious work. Years ago the senior managers at FoxBase would walk around with stopwatches around their necks, and the programmers were subject to an impromptu timed performance test of their code. The message was that performance — not just features — was critical. That definitely doesn’t happen anymore.

[Joe Marler]

[Michael Paul] “…footage (5D, C100, FS7…Now my FCPX project ends up being 571,23 GB with all the footage imported. I’m quite shocked that 195,92 blows up to this size but I guess it’s simply math because I create 2 different files: Proxies and Optimized…How big can a .fcpbundle file be and how much can FCPX handle….For a future project I will probably end up with 40 TB of FOOTAGE alone. If I transcode it using Proxies AND Optimized media the FCPBUNDLE will end up being MASSIVE. Can FCPX handle something like this or will it be an overkill and what are the minimum specs to handle this?…I’m on a 2012 iMac with a 3,4 GH Intel Core i7, 32 GB RAM and an NVIDIA GeForce GTX 680 MX. I fear this machine will struggle a lot under such a heavy load.”

It didn’t blow up to 571GB because of both proxy and optimized but because your original content is H264 and full-res optimized is much lower compression. Optimized media will often be several times the original H264 size.

40 TB of footage is a huge amount and typically would require a workstation-class machine like a top-spec Mac Pro with a very large Thunderbolt array. Then multiply that by at least 2x since it must be backed up. Even then you can’t blindly transcode all the content. On a higher-end machine you typically don’t need to transcode H264 1080p for performance reasons, and certainly not when skimming the content in the Event Browser. Once you’ve marked the initial favorites and whittled down the data to a few % of the original size, you might want to transcode that.

I don’t know whether FCPX can handle 40 TB of footage. If split across multiple libraries it might, given enough horsepower. One problem is at very large event sizes, FCPX does a lot of 4k and 8k random I/Os to build and maintain the thumbnails in the Event Browser. Even though a large spinning RAID array has good sequential performance, it doesn’t help on small random I/Os. An SSD array would help but those are very expensive at the 40TB size. However OWC makes one: https://blog.macsales.com/38713-owc-announces-40tb-ssd-thunderbay-4-mini-raid-the-highest-capacity-owc-desktop-ssd-solution-yet

Re the largest single library, it would be interesting if someone did some scalability studies on how factors like multiple events, workflow and various hardware configurations shape the achievable upper limit. However I don’t recall seeing anything like that.

FCPX is an integrated media database and editor. Managing a 40TB image database is a difficult task, regardless of actually editing. It’s possible a “lesser” editor without the database could handle that but how would you organize and find the clip ranges to edit? That in turn pushes you toward a stand alone Digital Asset Manager, which simply kicks down the road the database task. Due diligence would require you do scalability and stress tests on that DAM before committing production data to it. That in turn would lead to the same questions you’re asking now — hardware, configurations, scalability limits — but about an external product.

[Joe Marler]

I haven’t seen that but we don’t use much AVCHD, and what little we use is externally transcoded with EditReady.

What is the workflow for your files? It sounds like you are processing them externally before importing. If so, what utility? Or do you import the entire AVCHD bundle or copy files out of the bundle before importing? Do you use proxy or optimized media for the imported files?

Does it happening only during 1x playback, not when going FF/REV or skimming in the browser or timeline?

I have definitely seen performance problems if copying files out of the AVCHD bundle, but that happened on 10.2.3, and it was excessive I/O consumption not display degradation.

There are so many C100s in use I can’t imagine there is some widespread issue.

[Joe Marler]

[Robin S. Kurz] “So then why not get rid of folders in the Finder and just dump any and everything on the top level of every disk and just use Finder tags to filter everything? By that same logic I guess folders were a mistake of the original OS “

In most file systems, folders are an intuitive orgizational and navigational UI construct which maps to the underlying directory structure. Whether the UI draws a folder object or not, a heirarchical file structure exists. Using folders and sub-folders was an easy UI step, plus approachable and familiar to users.

FCPX is more like a relational database. In an RDBMS there is no intrinsic heirarchy of data, it exists in unordered rows that you query by attributes. On ingest you can define (ie tag) additional attributes to facilitate later retrieval. If the UI chooses to display that in folders corresponding to those attributes, it can, but (unlike a file system) the data is not heirarchically organized.

Since the advent of the RDBMS there has always been a conflict between human perception and relational data management. People often think of data navigationally or heirarchically. In their mind, rows are in an intrinsic order, like papers in a file. People have an ingrained (or is it learned?) tendency to store things within things, items with folders, folders within folders, etc.

This is likely one reason why FCPX originally used events. Not doing so would be too foreign, and transitioning or new users had enough to struggle with anyway.

Re “So then why not get rid of folders in the Finder and just dump any and everything on the top level of every disk and just use Finder tags to filter everything? By that same logic I guess folders were a mistake of the original OS…”

As previously described, a file system is very different from an RDBMS or FCPX. Today you can use FCPX perfectly well without any events, although events may have some residual value besides a comfort blanket of familiarity.

Object or database file systems have been proposed and some built, but none widely adopted. The lack of these is one reason we have so many problems when manipulating data outside iTunes, Lightroom or FCPX, which the app is then unaware of. Current file systems are very dumb, have no two-way communication with the app, and force database functionality to be redundantly implemented within each app. Each app must implement its own database, verification and repair functions — often rudimentary, undocumented or even non-existent.

Some apps have moved toward diminishing a pure folder-like organization. E.g, Gmail folders are not real folders but attribute tags which it displays as folders.

I’m OK using FCPX without events but I understand the need for them, at least for the foreseeable future. Eliminating events might make the FCPX storage model more pure, but I don’t see it bringing any other advantage, and it would create chaos if done.

To answer Oliver’s original question, even if database, query and performance factors would have allowed the original FCPX to be without events, I don’t think it would have been sellable to customers. It would have been too foreign.

[Joe Marler]

[Oliver Peters] “the original design was flawed and they’ve been trying to work around that ever since”

First, I think you’re correct about why do events still exist. This has been discussed before in various forums. I am personally working on a project with 5 terabytes, 7,000 clips an 160 hr of material in a single event of one library. I do everything with tagging and rating — it works mostly OK. There can be some I/O issues because FCPX does lots of random 4k and 8k I/Os when building and maintaining thumbnails in the Event Browser. I have read speculation that distributing content over multiple events somehow helps but I’ve never seen proof of this or tested it. This is on a 2015 top-spec iMac 27 with an 8TB SSD array in RAID-0.

With the advent of library-wide smart collections, library-wide searching, and finder tags to classify imported content, you mostly don’t need events from an organizational standpoint. However this doesn’t mean the original design was flawed, as if an architectural error. There are several possible explanations for events (all speculative):

– The internal database mechanisms to support library-wide searches and smart collections could not deliver adequate performance on then-existing hardware
– Finder tags did not exist which now makes it easier to duplicate the functionality of importing into a “bin”.
– It was difficult enough for editors to conceptually transition to FCPX. It would have been too much to demand they accept there’s no bin-like feature whatsoever and totally commit to metadata-only organization right off the bat.

Re why do they still exist, obviously people are using events for better or worse, so it would be a big disruption to remove them. Aside from media organization, there may in fact be reasons for retaining them, such as performance (unproven) or certain workflow or utility tasks. I have not studied this closely and other people could probably better list these.

[Joe Marler]

I have rarely had a problem getting the default FCPX H264 1080p export to play on laptops or mobile devices unless there is something wrong with them.

However Doug is right — FCPX does export 1080p/30 at about 20 megabits/sec, which is probably higher than needed. E.g, Youtube recommends 10 megabits/sec for 1080p/30.

A good fall-back option for smaller public projection or training is also export a version at 720p which reduces the data load by about 1/2, and is easier for ancient equipment to handle. For most venues other than large theaters, it is unlikely the audience can see the difference. Everything broadcast by ABC, Fox and ESPN is exclusively 720p.

[Joe Marler]

[Jeremy Freedberg] “Whenever I import my footage, I select “leave files in place,” and yet when I go into the contents of the library file, it clearly copies all my footage into the “original media” folder. For a project that’s over 500GB, this is unacceptable….This is my main issue with this program, as much as I am enjoying over premiere, it does what it wants, rarely what I want it to. Then I have to figure out what it did and work around that….

As Oliver said, your filenames indicate content from a Sony camera which may require re-wrapping by FCPX. This is also the case for AVCHD content. However in that case the “leave files in place” dialog is greyed out. So I’m not sure why it re-wrapped if the “leave files in place” dialog was active.

XAVC-S files from Sony A7-series cameras do not require re-wrapping, but apparently .MXF content does.

This can be a confusing area because the behavior is not documented and there are various sub-cases. E.g, if the .MTS files are copied outside the AVCHD file bundle they will not require re-wrapping. That is considered a poor practice and in my testing on 10.2.3 this caused problems with excessive I/O after the import, almost as if FCPX was dynamically re-wrapping each file each time it was referenced.

Ideally this should be documented in a KB article with a table showing the behavior in each case.

Normally FCPX can handle the native camera files pretty well, but (as Oliver said) if it is being re-wrapped on import my preference would be use EditReady and do that externally.

If you are on 10.3 and have installed Pro Video Formats 2.05, FCPX should handle MXF files natively, but I don’t know if that means via re-wrapping or what:

https://support.apple.com/kb/DL1898?locale=en_US

You are right that media management in FCPX can be complex. Ripple Training has an entire class on this, now updated for 10.3. OTOH FCPX has implemented a full-featured asset management database. Like most databases it can be more complex to learn than simple flat files.