Importing Proxies instead of originals

[Joe Marler]

Further testing shows relink has two main phases: “verifying files for compatibility”, and “relinking files”.

The first is the most time-consuming phase. It is doing many approx. 8 KB to 10 KB random reads, likely the video header of each media file. It is probably building an in-memory array which it then compares to the entry for each file in the SQL tables “Collection” and “CollectionMD”, which are contained in the bundle EventName/CurrentVersion.fcpevent.

During this phase it is doing only reads, not writes, and that constitutes the bulk of execution time on a mechanical array. If on an SSD the relative duration of each phase is more equal, likely due to the faster random I/O rate.

During the briefer 2nd phase “relinking files”, it is (1) rebuilding the pathname data in the SQL tables for each file, (2) rebuilding the inode for each file which is a secondary lookup method (3) updating the symlink for each file (all of which are in the library).

With 5,000 files comprising 11 TB on a 4-drive RAID-0 OWC Thunderbay 4 and with library on a dedicated SSD Thunderbolt array, the “verifying files for compatibility” phase takes 26 min, and the “relinking files” phase takes 5 min. Test system: 10-core Vega 64 iMac Pro, MacOS 10.15.7, FCP 10.5.

It is possible to determine the relative time spent for the 3rd step in phase 2 (symlink rebuild), since FCP will automatically do that if the files are moved or renamed on the same disk volume. You can rename or relocate 10,000 files within the same disk volume, and even manually delete all symlinks within the library. Upon re-launch FCP will use inode lookup to rebuild all symlinks within seconds.

This implies the relink bottleneck is not rebuilding the symlinks but (1) Fetching the video metadata from the header of each file, and (2) The array building and comparison phase. During portions of those periods the IO rate is very low and all CPU cores are low. This implies an algorithmic inefficiency, such as multiple threads blocked on a synchronization object.

This implies it may be possible for FCP development to greatly improve relink performance.

[Chad Greene]

Relinking to a local drive that is spotlight enabled works fantastic. However, our relinking slowdown comes from our shared storage that is not spotlightable. I can point Final Cut at the root of one drive and wait for it to search, or I can try and help it along by navigating to specific areas. Either way, it is not my favorite task. 🙂