[Joe Marler]

Leave files in place is generally a good idea and it works fine on the 8-bit 4k GH4 and GH5 codecs. Proxy is usually needed for best editing performance, so you’re doing the right thing.

I have edited large amounts of 8-bit 4k content from the GH4 and GH5 and transcoded to proxy w/o any problem, including macOS 10.12.6. I’ve used the 2013, 2015 and 2017 iMac 27 and the 2016 MacBook Pro, plus 8TB and 16TB versions of the Thunderbolt G-RAID.

By default the proxies are created inside the library which also by default may be in your Movies folder. Is it possible your’e just running out of disk space as those proxies are added to the library and it’s destabilizing the operating system? If your main library is on your system drive you can just copy it to the G-RAID. Ideally you might want the I/O on a separate drive but it’s an easy troubleshooting step.

Just remember to switch the viewer from proxy back to optimized/original before you export, else the file will be at proxy resolution.

[Joe Marler]

[Eric Santiago] “I inherited a problem project due to the AVCHD (a7sii) file structure.
For the brave ones that don’t convert, I tip my hat to you.
If I ran into this file format again I would convert to ProRes period.
Just too many issues with that format.”

I think the OP is using XAVC-S not AVCHD. Those are two different long GOP formats. AVCHD is limited to 28 megabits/sec at 1080p, whereas XAVC-S can handle 4k at much higher bit rates and chroma sampling.

There are definitely issues with AVCHD under FCPX such as a significant I/O performance problem if the bare files are imported with “leave files in place”. This can happen even if a small % of this AVCHD content is imported to a library containing mostly other formats. I always rewrap AVCHD externally before import using EditReady. If that is done it can be safely imported with “leave files in place” and performance/reliability is very good. Or the AVCHD content can be imported from the original folder structure and FCPX will rewrap that and copy it to the library. That also works but I prefer using EditReady. If you have any libraries where bare AVCHD files were imported with “leave files in place”, I suggest those be rebuilt and re-imported using one of the two above methods. Transcoding the already-imported AVCHD material to optimized also might work.

By contrast XAVC-S works very well without transcoding to optimized media or rewrapping in FCPX — even at very large library sizes. I’m editing a documentary that includes a large amount of XAVC-S content which was imported with “leave files in place” by copying the bare files to another folder. It is currently about 6,000 clips or 200 hr of 4K H264 material, equating to 8TB of original media and 5TB proxies. This will eventually be about 30% larger. It is running on a top-spec 2017 iMac 27 and a 32TB RAID-0 Thunderbolt array (with identical backup), and performance is generally very good.

The problem with transcoding to ProRes optimized media is it does nothing to help image quality — FCPX is always editing on a ProRes buffer, no matter what the original codec. Optimized media consumes a huge amount of space. It does help editing performance but so does proxy at a much smaller size and conversion time.

The space penalty for proxy over XAVC-S is an additional 50%. By contrast, the space penalty for optimized ProRes is about 6x. The transcode time for optimized ProRes is 2.7x longer than proxy. So if I transcoded all our H264 media to optimized, that would increase the size from 8TB to 48TB. It must be backed up to equivalent performance RAID storage so the actual hardware cost would be for 96TB.

The backup is required because optimized media takes over 2.7x the time to generate vs. proxy. The fastest Mac for transcoding from H264 to optimized media is the 2017 iMac 27 — it is 3.5x faster than a 12-core D700 Mac Pro on this task. Despite this it would take about a week to re-transcode my original media to optimized, IOW one week of downtime despite not losing any source media. The Mac Pro would take nearly a month to do this. Even recovering from LTO-6 tape would take about 3.5 days. To avoid this large downtime potential the ProRes version must be backed up, not just the original H264 media. So there is a cascade effect of time, cost and storage issues when transcoding to optimized media for large projects with high shooting ratios. Fortunately FCPX generally handles original H264 media very well, and is very fast at generating proxies.

[Joe Marler]

[Thanos Papadopoulos] “This thing happens often with different clips and on different macs. These clips cannot be rendered! I have to export it and reimport it to see these clips”

I have seen this behavior periodically, and I’ve seen others report it. I think it’s a bug. However you don’t need to export and re-import it. The optical flow is usually being calculated correctly upon export, but it doesn’t always render visually properly in the timeline.

I’ve seen reports that deleting generated library files might make the problem temporarily go away. Also if you are using optical flow on a speed ramped clip that can cause it. In that case try blading the clip and only apply optical flow on the bladed section.

[Joe Marler]

I’ve edited terabytes of 4k XAVC-S video from the A7RII and A6500; they use the same codec as the A7SII.

You do not need to create optimized media. You can just import with “create proxy” option. When using 4k you need proxy for good editing performance. Optimized media does not help quality, and it consumes a lot more space.

A significant factor is achieving import with “leave files in place”. FCPX will not allow that if the card folder tree is copied to disk. Without that import option the media is copied to the library which takes a lot more time and space.

The only option is to copy the .mp4 files outside the folder tree before import. That is considered poor practice but we are forced to do that because FCPX does not have an override to allow in place import, and it does not interrogate the storage device to determine it’s not an SD card.

Another issue is these cameras do not permit customized in-camera filenames or a non-repeating filename sequence. Each time you insert a new SD card the filenames start over with C0001.mp4. Even if these are kept in separate folders before import this primes FCPX for a duplicate clip bug which happens if importing event XMLs. The only way to avoid it is rename all your files before import to maintain complete uniqueness. I use the tool “A Better Finder Rename” to assign a unique numerical suffix to each file before import: https://www.publicspace.net/ABetterFinderRename/

I also recommend using Sony’s vertical battery grip on the A7SII and A7RII. It greatly extends operational time. If you have any Canon L lenses they work fairly well on the A7 cameras using the Metabones IV adapter. A lot of the video I shoot in the A7RII is using the Canon 70-200 2.8 IS II. However we also use the Sony 70-200 2.8 G-Master which is excellent and avoids the adapter.

[Joe Marler]

[Declan Smith] “I never use RAID5 for anything anymore. A single disk failure means that the array still works yes, but at reduced performance, and the time it takes to rebuild after a failure is astronomical and degrades performance during the rebuild (and carries it’s own risks)….Then when you have more than one disk fail, which is more common than you think because most RAID5 arrays are built from disks that were made at the same time…..if something goes wrong with an SSD you will lose everything, no exception, everything, no recovery….I use RAID0 on most things for performance reasons.”

You have a lot of good advice here. I also don’t use RAID5 anymore for the same reasons. I formerly said I’d never use four-drive RAID0, but for 4k or uncompressed media this makes sense, and I use it all the time now. Of course the FCPX libraries are backed up elsewhere.

The media storage is very large and we always have redundant copies anyway, either on the original portable offload drives or another RAID array. It doesn’t make sense to take the performance penalty and size penalty of RAID5 when the media is already backed up and it isn’t changing much (if any). The RAID5 array (despite the redundancy) must itself be backed up because it can have a chassis failure, application database failure, filesystem failure or human failure. If it’s backed up anyway and the content isn’t changing, why use RAID5? The only reason is ability to continue work if a single drive fails. If a 2nd drive fails before the rebuild finishes, the entire array is lost.

However, in that case the RAID5 array becomes very slow during the rebuild phase, so the ability to continue work can be marginal. Also other drives in the array are then at statistical risk due to manufacturing batches and shared environmental issues. IOW if one drive fails, then others are more likely to fail. Higher RAID forms mitigate this but at even greater expense, storage or performance cost, and they do nothing to help filesystem, chassis, database or human failures.

If a four-drive RAID0 array fails, I simply switch to the backup array which is loaded via Carbon Copy every night. The media itself does not change during the day. Manual switchover to the backup array takes two minutes. This does require a backup array but the main array (whether RAID0, RAID5 or anything else) must be backed up anyway. If it’s backed up to an array with roughly comparable performance, that an be switched to the main array.

If it is backed up to slower media like LTO tape or bus-powered portable drives, the backup is cheaper per MB but recovery time becomes a significant issue. A good backup that nonetheless takes many hours to restore entails significant downtime.

Your comments about SSD are also correct. SSD can fail like any other drive and must be backed up. While SSD reliability in general is better than spinning drives, they still have a significant failure probability. In some past studies, SSDs failed at a higher rate than some spinning drives:

https://media.bestofmicro.com/4/T/302141/original/ssdfailurerates_1024.png

[Joe Marler]

[Nicolas Bori] “Almost every time I make a change, or specially when I undo a change, I get the Mac color spinning wheel, and need to wait 10 to 20 seconds, until I can continue….Sometimes…the computer does not respond and I need to force shut it down…H264 files from a Panasonic GH4 and GH5 (Project has about 2000 clips)…Final cut Pro X (10.3)….MacBook Pro mid 2015 retina 15 inch. “

I edit lots of 4k material from a GH4 and GH5 on a similar MacBook and it usually works pretty well, although proxies are required for good performance. That said, the GH5 4k 10-bit 4:2:2 H264 codec is not well supported and H265 won’t work until the next FCPX update. But the regular 8-bit codecs work OK.

If the computer becomes non-responsive this often indicates a system problem not an app problem. IOW hardware, disk drive, out of free space on the boot drive, etc. How much free space is on your system drive?

Can you run BlackMagic performance test on your external hard drive and get the results?

https://itunes.apple.com/us/app/blackmagic-disk-speed-test/id425264550?mt=12

You could also try running Disk Utility First Aid on both system drive and external drive.

[Joe Marler]

[Declan Smith] “tried exporting project xml and importing, but…the project browser doesn’t keep up with changes. I have also had a few problems with audio edits reverting back… “

I used event XMLs a lot and haven’t seen cases where the project isn’t updated when loading an event XML. However XML export/import of a multicam clip does not preserve the audio or video monitoring angle as selected in the angle editor. The selected audio in the angle *viewer* is preserved. For me it’s a minor issue. It might be by design — maybe they consider the monitoring angle to be application state data not video metadata.

[Joe Marler]

I forgot to add there is a bug in FCPX event XML processing which mandates that all media filenames be globally unique across the entire library. Many cameras (esp. some Sony NEX series) do not allow in-camera filename customization so will use repeating filenames each time an SD card is changed. Upon import FCPX appends a “uniquifier” but something still malfunctions during the import phase of XML processing on the destination machine.

Even if the media files are kept in separate disk folders (say by shooting day and by camera), after import to FCPX this creates a situation where loading an event XML will create spurious duplicate clips.

The only solution is maintain absolute global filename uniqueness before importing media. That might be a good practice anyway but it should not be required. If the media is in folders by shooting day and by camera, FCPX can automatically keyword ingested files by those folder names, so it’s easy to find using the FCPX query tools — regardless of filename.

Since you must assume that your material might later be combined with content from other projects, the global uniqueness requirement becomes more difficult. They must not only be unique within the event but unique across all libraries for all time, at least until Apple fixes the bug. I use the tool “A Better Finder Rename” before import, append a sequence number to each file, keep track of the highest number so each ingest batch uses sequential sequence numbers:

https://www.publicspace.net/ABetterFinderRename/

If your media also includes metadata files with matching filenames, those also must be renamed. However I think some cameras embed the original media filename inside the metadata files, so this might require a special utility to rename the files properly. To reiterate, this is only required if you are loading event or library XML files and the on-disk media filenames are not totally unique.

[Joe Marler]

[Declan Smith] “On the laptop “version” I am using proxy footage and on the desktop, I have the space for the original media. My question in a nutshell is what is the best way to sync the project between the laptop and desktop? I work on the laptop doing most of the edits on the train etc, then when back in the studio want to transfer to the desktop for further work, then back to laptop when I need to be out and about.”

There are several ways to do this, depending on the specific requirements and your configuration.

One option is to export an event XML from the laptop machine and import that on the desktop. That will include not only the project but any metadata changes you made such as keywords, favorites, rejects, etc. Note you will get a pop up warning upon import saying there are already duplicate clips in the library. You answer “replace” to this. Before loading any XML it’s a good idea to make up a manual backup copy of the library by shutting down FCPX, clicking on the library and selecting “duplicate”. If something goes wrong you can just roll back to that.

Another option is keep the library on a portable drive. By default the proxies will be in the library. The procedure is easy: in the field edit on laptop and keep FCPX in proxy mode (In the viewer drop-down menu, View>Proxy). Then when you get home you connect the portable drive to your desktop and it will automatically connect to the original media.

However if your library is on the laptop itself then you’d have to copy the entire library from the laptop to the desktop to resume work. If your work is mostly sequential, say you defer color and audio work until the end then do that on the desktop, this might not be inconvenient to copy it once, which includes all the contained proxy files inside.

However if you are frequently changing between desktop and laptop and if you aren’t using a portable drive, another option is use external proxies which keeps the library size small (typically < 100MB, often less than 50MB). Then when you get home you copy the library to the desktop, start FCPX and it should theoretically connect to the original media. However if the drive name is different this may require a relink. In my experience relinking when using external proxies is unreliable so this should be examined carefully before using it. A workaround is have the desktop and laptop both use the same volume name for the internal drive, if that’s where your media is.

Also, the procedure for creating external proxies can be a bit complex. The entire procedure including various issues is described in Ripple Training’s FCPX 10.3 Media Management tutorial:

https://www.rippletraining.com/products/final-cut-pro/media-management-in-final-cut-pro-10-3/

[Joe Marler]

[Thanos Papadopoulos] “Can somebody pinpoint the benefits of creating events for organising his clips?…cases you prefer events instead of folders and folders instead of keywords?”

Events are somewhat like a folder, in fact they are currently implemented physically as a folder inside the library package. However you need not use events at all beyond the single required event. I’ve previously had over 8,000 clips (181 hr of 1080p and 4K material), comprising 4.75 TB total size — in a single event. The organization was strictly based on keywords and ratings, and it worked fine.

It’s tempting to say events are required to organize content in a given shooting job, but that can also be done by keywording those clips with the event name. It can even be done automatically provided the clips are in a named Finder folder and you import with the option to keyword them based on the disk folder name. Whether you click on a keyword collection to reveal those clips or open a folder to reveal them (as in other editors) the result is similar.

However a clip can only be in one folder (or event) unless it’s duplicated, and you can’t put part of a clip in a folder or event. By contrast keywords and ratings are range-based and they can overlap, so are much more flexible.

Despite this there can be advantages to using events, such as partitioning metadata work among multiple assistant editors. E.g, consider an overall “project” which contains material shot at different times and locations, each in its own event. Multiple assistants having a copy of this media could each be busy rating & keywording their assigned event, then email that work via a event XML back to the home office. This parallelizes one of the most time-consuming aspects of a large project which is the media tagging and organization phase.

In FCPX there is no choice between events and folders as folders are not for media — only for keyword collections. The main purpose of folders is to de-clutter a large list of keyword collections by grouping similar ones together. E.g, say you have many keyword collections of different employee interviews by their name, and more keyword collections of their managers, and more keyword collections based on department. Instead of having all those in a huge scrolling list, it’s neater to have folders for “employees”, “managers”, and “departments”.

That could all be keyworded and you could query it without folders using the method shown in this video “One Smart Collection To Rule Them All”. However there’s a balance between a purely query-oriented approach and a navigational approach to data, so FCPX provides a few things like folders to help:
https://www.youtube.com/watch?v=sjuCfJFhdo0

Some contents or functionalities here are not available due to your cookie preferences!

This happens because the functionality/content marked as “Google Youtube” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.