[Joe Marler]

[Jeremy Garchow] “You could try simply relinking to the R3D files after you import the Red created Proxy files using FCPX. Or you could try this” [proxy cheat using Finder aliases” method]

FCPX is not currently designed to use new proxies created outside the product. It’s not a matter of those being a certain format. The SQL tables inside CurrentVersion.fcpevent have pointers to the symlinks inside the library, which themselves point to the proxy files. Besides those internal pointers, those tables also contain data about the file to prevent a mis-match. This includes pixel aspect ratio, audio config and timecode. The SQL tables also seem to contain Primary/Foreign keys which tie together the rows corresponding to a given proxy and media file.

However even if your constructed proxies match the config data exactly, they will not match the internal pointers or the inter-table relational links, so those will never work. In fact the SQL rows corresponding to proxies will not exist, since FCPX never created any proxies. The proxy and media pointers apparently each consist of two independent locators, the file inode to the proxy file plus some method of locating the symlink — either another inode or a path pointer.

If the FCPX-created proxies (or media files) are relocated within the disk volume, renamed, or even if the symlinks inside the library are totally deleted, FCPX will successfully relink those and rebuild new symlinks based on an inode lookup. However that doesn’t help for new proxies created by external methods.

The reason the above “proxy cheat” works is there is evidently an undocumented backup method of relinking *existing* proxy references inside the library to FCPX-created external proxies. That method is triggered by copying Finder aliases (which are different from symlinks) inside the library. That in turn causes FCPX to scan those, read the path information from the alias, and build new symlinks. The library proxy folder will be left with two sets of pointers to each proxy file, the aliases you dragged in plus the FCPX-created symlinks. The new symlinks are disambiguated with an (fcp1) suffix. The “proxy cheat” method also can be unreliable — even for existing FCPX-created proxies, if those are relocated to a different-named hard drive.

Note: Finder calls both aliases and symlinks “aliases”, so this is confusing. In Finder easiest way to differentiate an alias from a symlink is do “Get Info”. If there is a “select new original” button it’s an alias. If not, it’s a symlink. In terminal ls-l will show the symlink path whereas an alias appears as a regular file.

There are ways to create proxies that satisfy the FCPX check for audio channels, aspect ratio, and timecode — articles have been written about that. However this requires that FCPX-created proxies once existed — either inside or outside the library. If those FCPX-created proxies never existed, there are no references inside SQL tables inside CurrentVersion.fcpevent, hence there is nothing to relink those proxies to.

Since the data store is based on SQLite and that on-disk file format is documented, you may wonder why can’t a SQL guru write an app to do this. It’s because the application-layer schema and data dictionary are totally undocumented. IOW what the column names and values mean. Inside FCPX, the programmatic interface to the underlying database is apparently an object-oriented “graph database” using Core Data, which in turn uses SQLite as the persistent data store. The SQL column names and values are generally not human-readable CHAR fields, but either complex descriptors or Binary Large Objects.

The bottom line is I don’t know of any way to use externally-created proxies when FCPX-created proxies never existed.

[Joe Marler]

If you are importing into the library instead of “leave files in place”, that can take a lot of additional room.

If your import preferences automatically create proxies or optimized media, that also will take lots of room. If the original media is H.264, proxies add roughly 60% to that size and optimized media is about 6x that size. FCPX tries to predict the space needed for those and if insufficient to complete the task, will refuse to start with an “insufficient space” message.

If that’s not it, run Disk Utility First Aid on all volumes, then try again.

Here is a free app called OmniDiskSweeper that can help track down space consumption:

https://www.omnigroup.com/more/

If you have several libraries, Final Cut Library Manager can help safely free up space:

https://www.arcticwhiteness.com/finalcutlibrarymanager/

[Joe Marler]

[Hans Lucas] “I’m running QT 10.5 full screen and it looks just fine, which is not the case when I run FCP X full screen (same screen size)”

This is because QT10 will deinterlace in some circumstances — it reads the file metadata and chooses whether to apply playback deinterlacing. Sometimes it works, and other times the format or header confuses it and deinterlacing is not applied.

QT7 was manual — you had to invoke that. VLC was formerly manual — defaulted to off, current versions are like QT10 and will automatically try to detect interlaced content and apply playback deinterlacing. VLC also has various manual overrides for specific algorithms.

The FCPX viewer apparently does not deinterlace so this can be confusing. You just have to remember there is no single appearance “truth” — with interlaced material, something must deinterlace that and there are various methods and metadata which impact that. This means if you export interlaced, you must consider the playback environment. Will it be uploaded and deinterlaced by Youtube? Will the file be directly distributed to the recipient? Will it be burned to a DVD? How will they play it back — on a computer or hardware DVD player?

Re-testing on the current FCPX 10.4.5 shows the deinterlace filter in FCPX will visually deinterlace DV material in an interlaced DV project, inc’l when “show both fields” is enabled. Unlike how I previously thought, this carries through to output, which you don’t really want. You never want media flagged as interlaced which has been hard-deinterlaced.

So in theory (if you edit DV in an interlaced project) you could apply the deinterlace filter, then remember to turn it off before exporting.

However in my testing, NTSD DV content from a Panasonic DVX100 looks best when used in a progressive 720p project, deinterlaced and output as progressive. The output preset I used was Master File>Computer>H264>720p.

I did many screen captures using various playback methods and this seemed to look the best. Your case may be different.

Non-broadcast editing and distribution of interlaced content is hassle. Back when my doc team shot interlaced and distributed DVDs, we’d have to check various hardware and software players to verify we had encoded it for broadest compatibility with then-current playback systems.

ATSC broadcast uses 1080i (for distribution, not necessarily for acquisition and editing) — except for ABC, Fox, and ESPN who acquire and broadcast in 720p/60. Also Comcast has converted most of their channels from 1080i to 720p. Outside of live production, I’m not sure how much broadcast content is acquired and edited in interlaced format. The below photos show various networks shooting with cameras that are likely progressive:

ABC News: https://joema.smugmug.com/Photography/ABC-News-Using-DSLRs/n-BsScJC/
ABC Nightline: https://joema.smugmug.com/Photography/ABC-Nightline-Using-DSLR/n-HwH8hG/
CNN: https://joema.smugmug.com/Photography/CNN-DSLR-Video/n-scsdxs/
CNN Moneyline: https://joema.smugmug.com/Photography/CNN-Moneyline-DSLR-Shoot/n-ffF2JW/
CBS News: https://photos.smugmug.com/photos/i-CRnjPwk/0/2d7985a6/X3/i-CRnjPwk-X3.jpg

However this 60 Minutes field crew is probably shooting interlaced: https://joema.smugmug.com/Photography/60-Minutes-using-Panasonic-HMC/n-MFg8L9/

[Joe Marler]

[Joe Marler] “In theory the FCPX viewer should do that [deinterlace]”

[Robin S. Kurz] “How is it NOT doing that if you don’t tell it to show both fields? ????

If you don’t select “show both fields”, it’s not deinterlacing, it’s just discarding one field (as you mentioned). I only meant that FCPX obviously knows if the clip is interlaced and could theoretically apply playback-only deinterlacing based on the metadata, as QT10 does.

[Joe Marler] “…interlaced legacy material mixed in a progressive project. In that case each interlaced clip can (and should) be deinterlaced with the filter, and the output will be flagged progressive in the video header…”

[Robin S. Kurz]“No they shouldn’t. Unless of course you want the exact quality that he’s trying to avoid…

I just tested this with some DV material from a DVX-100, and if you add legacy DV content to a progressive project, don’t deinterlace that and output as progressive, the DV clips look bad. If you deinterlace the interlaced DV clips, then output the progressive project, it looks good.

[Robin S. Kurz]“…And whether an exported clip is “flagged” as progressive or not is solely defined by HOW you output it…”

The project type can definitely affect this. I just put a DV clip in an NTSC interlaced project, output it using the ProRes 422 preset and it was flagged as interlaced. Then I changed the project characteristics from 29.97i to 29.97p, output it the same way, and it was flagged progressive. The only thing that changed was the project characteristics.

OTOH what you said is generally correct. A DV clip in an interlaced project and output as “Apple Devices” will be flagged as progressive. If that same clip from the same project is output as DV, it will be flagged interlaced (as seen in inspection tools like Invisor or MediaInfo).

As you said, you might normally output in the original format. E.g, for interlaced DV material, output in DV and leave deinterlacing up to the playback chain as originally intended.

The problem is that’s such an old format, the playback deinterlacing (esp. on computers) doesn’t always look best. I tested this using QT10’s auto-deinterlacing, also I manually stepped through all of VLC’s manual deinterlacing options for many different clips. Even on DV clips with the correct metadata, it looked better if that was hard deinterlaced then output as progressive.

Also it’s less common nowadays to have all-interlaced material. More often it’s interlaced legacy content in a progressive project, and if you don’t deinterlaced the old stuff it looks bad, plus the output file metadata says progressive. Most playback systems can’t deinterlace the old clips locked inside a progressive project, effectively locking in “hard interlacing”.

I just did a lot more testing with DV material in FCPX 10.4.4, it appears to look best when the DV clips are put in a progressive project and deinterlaced. Another complication is web sites like Youtube have their own deinterlacing. In my tests Youtube did a good job of deinterlacing DV but it still looked a little better if hard deinterlaced inside a 720p project and uploaded.

[Joe Marler]

The shifting fields and comb effect on horizontal movement are expected characteristics on non-deinterlaced playback of interlaced material. QT Player is automatically deinterlacing. In theory the FCPX viewer should do that. I’ve seen some cases where it doesn’t. However as an editor sometimes it’s best to have visual feedback the clip is interlaced, esp in a mixed project.

FCPX has a deinterlace filter in Inspector, but it won’t work for an interlaced project (on export). By definition output from an interlaced project is interlaced and it’s left to the playback chain to deinterlace. Premier lets you deinterlaced an interlaced clip and output that file with interlaced metadata, which doesn’t seem right.

Nowadays it’s more common to have small amounts of interlaced legacy material mixed in a progressive project. In that case each interlaced clip can (and should) be deinterlaced with the filter, and the output will be flagged progressive in the video header.

I vaguely recollect in a past version around 10.3 or so the behavior of the deinterlace filter changed. Maybe before it was reflected in the viewer, and afterward it wasn’t. I can’t remember.

[Joe Marler]

I’ve had several FCPX 10.4.4 crashes on Mojave from using Imagenomic’s Portraiture skin processing plugin. I forwarded them the crash logs showing their plugin was crashing the entire FCPX process. They think it’s something related to their licensing code. Most of the time it works OK, but anytime a plugin crashes the host process that’s very serious. Metadata updates could be “in flight” from a database transactional standpoint which requires rollback or commit to preserve logical integrity.

Portraiture is the best available skin retouching software, so hopefully they will get this fixed or maybe it’s isolated to just a few cases.

[Joe Marler]

[Mark Smith] “one audio wav file from my mix pre 6 and my camera clip…I have tried every possible way to sync the clips and the resulting synced clip has no picture…the result is a synced clip with the proper audio and no video even though there is common time code, decent audio on both clips etc…”

My documentary team uses the Sound Devices MixPre-6; it’s really good. Label all clips from each camera or recorder in the Inspector, then create a multi-cam clip and select sync on audio (not timecode). It is often best to use a multicam clip even for one A/V and one audio clip.

To label each batch of clips, select those in the Event Browser, then in Inspector press the “i” button at top, then give them a camera name. Without labeling each group of clips, the audio sync attempt often will not work.

After doing that I suggest marking the parent clips rejected using the filter “hide rejected”. Then keyword/rate only the MC clip. This is because ratings and keywords for the parent clips are not passed through to the MC clip and you don’t want to accidentally use a parent clip in the timeline.

Sam Mestman mentioned the advantages of using multicam over sync clips here:

https://www.fcpworks.com/sync-or-multicam-clips/

His procedure for prepping the clips for sync is here:

https://blog.wemakemovies.org/2012/10/fcpx-batch-renaming-advanced-multicam-sync/

[Joe Marler]

[Oliver Peters] “https://digitalfilms.wordpress.com/2019/01/01/the-state-of-the-nle-2019/

My 2 cents

Oliver”

Oliver, thanks for that nice, well-round article.

Re FCPX optimization for Macs more than cross-platform NLEs, recent versions of Resolve have challenged this long-held notion. We formerly viewed FCPX’s hyper-responsive skimmer as something only Apple could achieve by owning the app, hardware and OS, and targeting only Macs. This was seemingly supported by how clunky and slow Premiere is on 4k H264 playback — ironic for a playback engined called “Mercury”.

However Resolve 15.x has a “skimmer” and it’s almost as fast as FCPX — yet it’s a cross-platform product. This is an astounding achievement. Blackmagic only controls the application layer — no kernel extensions, device drivers or add-on hardware was used to accomplish this.

Re FCPX improvements, people tend to emphasize user-facing elements, but it badly needs some significant architectural and media management improvements. Unfortunately these are not “sexy” from a marketing standpoint. From a development (and test) standpoint these tend to be difficult and time-consuming.

The current plugin architecture allows a software error in a single plugin’s code to crash the entire host process (FCPX). The more plugins, the more likely this can happen. Ideally the plugins should be run in some sandboxed fashion, similar to how most web browsers run each tab in a separate process address space.

For an app which emphasizes ease of use through UI simplicity, FCPX media management can rapidly become very complex and difficult to understand. One of the worst areas is management of external proxies which cannot be relinked if the drive name or underlying folder structure changes.

Overall FCPX data integrity is amazingly good (in terms of not losing edits). However there are an uncomfortable number of reports of lost render files, red clips, black clips, red clips that play, red audio which won’t play, red audio which *will* play, etc. This can’t all be blamed on hardware — I’ve had all these on a variety of machines.

From a software development and management standpoint, these things are “hard problems”. They may need the OS guys to fix something in the Core Data APIs or SQLite database, which involves cross-team prioritization and scheduling. If a senior developer is given the choice of working on a new user-facing feature vs cleaning up somebody else’s mess, the former is more attractive.

I’d personally like to see some collaborative features — and not just LAN-style, but cloud-based. Even small distributed teams who don’t share a LAN/SAN could benefit. Unfortunately to do that well and reliably — and test it thoroughly — is another hard problem.

[Joe Marler]

[Jiri Fiala] “Thing is, it’s not really cross-library editing. I was merely copying a project to a new library. I didn’t edit media from library A to library B. It’s a nasty bug.”

I just reproduced this; not sure how related it is to your situation. I had a library with proxy-only external media and copied an event to another library, and did not select the optimized or proxy checkboxes. Examination of the destination library bundle showed no /Transcoded Media/Proxy Media folder and no symlinks to the proxies on disk.

I then deleted the copied events with the black clips and re-did the copy but selected the optimized and proxy checkboxes. Examination of the target library bundle then showed showed a Proxy Media folder which contained symlinks pointing to the original proxies on the same volume.

Using external proxies might be an edge case, but the UI should really not let you do something that results in black clips. It’s also ambiguous. The normal interpretation of the copy proxies/optimized media checkboxes refers to the media itself, which you don’t want duplicated if using external media. However in this case those checkboxes were necessary to copy the symlinks, and without those it somehow copied only metadata resulting in black clips but no links to the media.

If you have proxies on some media (say just the 4k) and did not select the proxy checkbox during the copy, maybe that explains why only some clips were black in the destination library.

[Joe Marler]

[Jiri Fiala] “Thing is, it’s not really cross-library editing. I was merely copying a project to a new library. I didn’t edit media from library A to library B. It’s a nasty bug.”

By “cross-library editing” we meant copying the data (or references) then editing. You can’t really edit between libraries.

You’re right it should work perfectly. I’ve seen several cases of red clips or red audio after a cross-library copy, or if using multiple drives and one is momentarily off line. Those are often related to a cache issue and can sometimes be fixed by manually deleting the cache from Finder. Sometimes the clips are OK but the audio in a project is red, even though it plays OK.

There is another range of cases where proxies become unlinked, usually when using externally-stored proxies. This is really bad since you can’t re-link proxies.

However I haven’t seen cases where the clips are just black following a cross-library copy.

I’m actively investigating some FCPX media management issues now. If you could give any more details that would be helpful. You can email me directly at joema4(at)gmail(dot)com.