[Joe Marler]

[Tangier Clarke] “A lot of my video has digital artifacts in that somehow occurred somewhere in the offloading from the cards and rewrapping to quicktime process.”

If you’ve already imported all this stuff to FCPX and edited it, there may be nothing you can do except fix it and re-import, but it’s important to understand what happened.

Do these artifacts only show up *after* re-wrapping, or are they visible in the original camera files if played in Quicktime or VLC? If they are there before rewrapping, it’s obviously not an issue with either FCPX or EditReady.

If they only show up after rewrapping, I suggest you contact DivergentMedia (maker of EditReady) and discuss with them. They have excellent support. If you haven’t updated your version of EditReady, rewrap some problematic content with that and evaluate. If the machine you rewrapped on does not have the latest 2.0.5 version of Apple Pro Video Formats, install that and evaluate: https://support.apple.com/kb/DL1898?locale=en_US

I think you are using a C100 Mk I which produces AVCHD, so re-wrapping that is definitely a good idea. I’ve used EditReady a lot and I haven’t see a case like this.

It would be interesting to compare the video file characteristics of the original AVCHD clip, a “bad” re-wrapped clip and a “good” re-wrapped clip. The Invisor utility lets you compare multiple files side-by-side in a spreadsheet format: https://itunes.apple.com/us/app/invisor-media-file-inspector/id442947586?mt=12

[Joe Marler]

[Oliver Peters] “…complained a number of times about JKL play performance in FCPX…What I’ve generally found is that across the board, if I’m working with ProRes, performance is OK. If I work with external i/o turned off and/or not installed, performance is OK. When I work with external i/o engaged, especially with non-ProRes codecs (C300, DSLRs, XDCAM, RED, etc.) performance is awful….”

I’ve tested H264 4k editing using Premiere CC vs FCPX many times back-to-back on a top-spec 2015 iMac 27. I use JKL heavily. In general FCPX is much faster without proxy or optimized media — on this machine, and when not using external I/O. However it seems Premiere has gotten faster within the past year or so, but it’s still significantly less responsive, especially to JKL. Using proxy they are both very fast on H264 4k.

I rarely use external I/O, and when I do it’s only Thunderbolt to another iMac. It’s interesting your case seems mostly confined to external I/O. Are other FCPX users with your I/O hardware and device driver also reporting problems? Can you test it with different I/O hardware?

I’d be frustrated if FCPX was slow. In a sense Apple has less leeway on this than Adobe, since a core design principle of FCPX is slippery-fast speed. You are talking mostly about timeline performance but if the Event Browser got sluggish that would deactivate one of the main reasons for using FCPX.

[Joe Marler]

[Ray Sherman] “I came across a large amount of .avi media…camcorder used was a JVC GR DVL9500 tape based consumer handicam….The media is non-widescreen 4:3 720×480 .avi and it appears to be interlaced….”

That is DV — FCPX can import that “in place” and edit that natively with no problems. Using proxy or optimized media is not required. Compressor can transcode it to ProRes before import but I don’t see the benefit of that. By modern standards the DV codec is simple and the resolution is low, so there is no performance problem editing it. Transcoding to ProRes does not improve image quality — that is determined by the camera and codec capture parameters.

The biggest problem with DV content is not image quality per se but interlacing. Some DV cameras shot “matted” 16:9 DV which is another problem but fortunately yours did not. The interlacing issue can be confusing for several reasons:

(1) Some playback software like Quicktime will auto deinterlace under some conditions but not others.
(2) VLC defaults to “no deinterlace” but it can be enabled and has multiple selectable algorithms.
(3) Youtube will auto-deinterlace NTSC DV content, at least sometimes.
(4) FCPX will not deinterlace on export for an NTSC project even if you set the “deinterlace” checkbox in Inspector under Settings.
(5) FCPX *will* deinterlace on export NTSC content in a progressive project IF you set the “deinterlace” checkbox.
(6) If you put NTSC content in a progressive project, don’t set the deinterlace checkbox, then output, the file will be degraded, apparently from discarding alternate scan lines.

Since most of your content is apparently progressive and you want to include some 4:3 NTSC DV stuff, use a 1080p or 720p project and for the DV AVI clips make sure you have the deinterlace checkbox set for those.

4:3 content in a 16:9 project will appear “pillar boxed”. That is expected and you can evaluate how to present that.

When dealing with interlaced content always inspect the output for comb-type interlacing artifacts and for signs of degraded resolution from “discard-type” deinterlacing. Check (1) Using Quicktime Player, (2) VLC with deinterlacing off, (3) After upload to Youtube or Vimeo, and (4) Local playback on mobile devices if that is a distribution method. The final output should be no worse than the original interlaced files viewed in VLC with Yadif 2x deinterlacing enabled.

Test and validate your workflow early on small media samples. Do not do a huge project then find out 20% of the clips are messed up from improper hard deinterlacing that happened several stages up the chain.

[Joe Marler]

[Oliver Peters] “Adobe’s Premiere Pro panels architecture?…It helps push the growth of Premiere Pro penetration into production companies and facilities. Nothing like that exists in FCPX and the presumption is that Apple is not interested in others altering their UI design….”

This is an interesting point. You can obviously have fairly elaborate plugins for FCPX such as Color Finale Pro, but it can’t be fully integrated into the UI. In FCPX you also see widely-varying techniques to shoehorn plugin UIs into the canvass. Some like Color Finale have a large floating window. Others like CoreMelt take over part of the Viewer. Neat Video launches an entire sub-interface with tabs and menus. There are other inconsistencies such as multicam doesn’t work with Drop Zones. There is no standard way to check plugin version numbers.

Besides UI issues, the plugin software architecture on FCPX can cause odd behavior. A fundamental principle of UI design is background tasks or computation should not interfere with the main UI thread. A familiar past example was on 16-bit Windows, OS/2 and (I think) macOS before X, the UI input queue was synchronous — if an app hung, it owned the input queue and you lost control of the mouse and keyboard.

With Windows 95, NT, OS/2 Warp 2.x and OS X, they adopted an asynchronous input queue so a hung or CPU-intensive app would not lock out user input: https://blogs.msdn.microsoft.com/oldnewthing/20130604-00/?p=4173

However it appears the plugin threading model used by FCPX has elements similar to a synchronous input queue. A CPU-intensive plugin such as Neat Video can lock out the input for significant periods. Normally — on most plugins and situations — this isn’t a problem but with some it is. I don’t know why Apple designed it this way but there are definitely possible improvements. I don’t recollect seeing this behavior on Premiere but I no longer use it that much. Maybe someone could comment.

[Joe Marler]

[Scott Witthaus] “I looked at Lumberjack but that looks as if you need to start the process during the shoot? “

Lumberjack can also log the material after the shoot — it’s called “Backlogger”: https://lumberjacksystem.com/backlogger.html

However Noah’s suggestion of MergeX is also good.

[Joe Marler]

[Mathew Farrell] “It seems odd to me that something like FCPX gets it knickers in a knot over the file wrapper, but can handle the codec passably. I would have though two or three lines of code would bypass whatever the wrapper is doing

It is obviously technically possible to handle bare .MTS files without major problems because Premiere does it. This is some kind of internal inefficiency with FCPX. Provided you understand it exists and use the solution you found, it’s no problem.

[Mathew Farrell] “After rewrapping, the filenames remained the same, but the extension (.MTS to .MOV) was changed…does anyone know a batchable way to replace files with a different format or codec, like I was trying to do?…”

It wasn’t just the filenames changed, but the internal file structure changed. Examination of a .MTS file using Invisor before and after rewrapping shows the byte count is different, duration slightly different and multiple header fields are changed: https://itunes.apple.com/us/app/invisor-media-file-inspector/id442947586?mt=12

From a human standpoint you know it’s the same content and codec, but by the stringent criteria FCPX uses, it looks like a different file. In theory these could still be relinked and used, and more flexible relinking has been requested many times.

[Mathew Farrell] “…In researching this problem, I came across advice to keep libraries under ~1000-2000 clips. This makes sense. The workaround for a “project” like this (7700 clips) is to split the clips over several libraries (no need to move or dupe the source files), and build out a timeline from several libraries. I might investigate splitting this library up down the line, but so far things a slow, but not the point of hampering my keywording mission.”

I think that recommendation came from Larry Jordan. This has been discussed several times and we don’t know the basis. I have a 3.5 terabyte library with 7,000 clips, and it mostly works OK. It had major performance problems before I cleansed it of all .MTS files, but it’s mostly OK now, but the media is on an 8TB Thunderbolt 2 RAID-0 SSD array.

Extremely large libraries can still have I/O performance issues because FCPX does a lot of 8k random I/Os to build and maintain thumbnails in the Event Browser. Since even the fastest spinning RAID array is not good at small random I/Os, the only solution for this is an SSD array, and even those are not infinitely fast on that I/O pattern. Apple really needs to improve this. You won’t see this on any “top 50” FCPX feature requests because most people either don’t work at that level or are unaware of the cause. Hence if Apple goes by the number of requests, they may never improve this.

If you are at 7,700 clips it might be a good idea to consider splitting that, but if your current performance is usable you could wait and watch. If you can afford it an 8TB SSD array is only about $3k, provided you use the OWC Thunderbay 4 Mini and 4x Samsung EVO 850 2TB.

[Joe Marler]

[andy patterson] “Have you tried using Premiere Pro with a PC? I will put my PC against your iMac”

I have used Premiere since CS4 and have obviously tested this many times on a PC. There is no question that a top-spec iMac can export H264 a lot faster than Premiere on either the same Mac hardware or a comparable PC — and maybe any PC. In theory Premiere on a multi-socket PC with enough cores might eventually beat FCPX on a 4Ghz iMac at exporting to H264, but it would take a lot of cores. If Adobe ever truly supports Quick Sync or the other hardware-accelerated transcoding methods like NVENC or VCE, that would make a difference.

[andy patterson] “…I don’t think Adobe supports Quick Sync. There is a plugin I believe. I think Adobe was concerned about quality over speed and they wanted the same experience for the XEON and i7 computers.”

This is a quote from Adobe’s Premiere CC 2015.3 feature list: “…initial support for Apple Metal and accelerated H.264 decode on Windows using Intel Quick Sync Video.” There was a tiny footnote somewhere (which I can’t find right now) which said “Windows only”. However if Adobe is using Quick Sync, they aren’t doing a very good job of it. https://www.adobe.com/products/premiere/features.html

[andy patterson] “…Can you email me the 4K files? I am not saying I can play them with out trans coding but I would like to try it. Can your iMac play 5 layers of the native Red One R3D files with multiple effects applied?”

These aren’t special 4k H264 files — they are from various cameras including Panasonic AG-DVX200, Sony A7RII, GoPro Hero5, DJI Phantom 4 Pro, etc. They all behave similarly on FCPX vs Premiere. They are too big to send but you can download sample files from those cameras from Vimeo or many other places. Just create a 3-camera 4k multicam, enable multicam playback, set the monitor to 1/4 res, then use JKL to scrub back and forth on the timeline. Re playing 5 layers of Red One with multiple effects — I have no idea and have no current need to investigate that.

[Joe Marler]

[andy patterson] “I think they felt as though the quality would not be as good. I think Quick Sync will only work with single pass renders. My system renders pretty fast with a mediocre GTX 1060 and a mediocre 3.4 GHZ Haswell Quad Core CPU. I imagine what a GTX 1080 and 8 Core CPU will do. AMD will be launching 8 core CPU for under $500.00 in a week or two. “

Several years ago, some felt that hardware transcoding had significant image quality issues. This 2012 article was titled “The Wretched State of GPU Transcoding”: https://www.extremetech.com/computing/128681-the-wretched-state-of-gpu-transcoding

It also misleadingly lumped Quick Sync into that group, even though it’s not technically GPU transcoding. Rather Quick Sync is “fixed function” logic, like an ASIC. It requires resources from Intel’s on-chip GPU, which is why it’s not in most Xeons — Intel would have to put the entire GPU in Xeon just to get Quick Sync. Why they designed it that way, I don’t know. But from an API or algorithm standpoint, Quick Sync and the GPU are different.

Back in 2012, Quick Sync and related software were less refined. Today I cannot easily tell the difference between a single-pass H264 export from FCPX (which uses Quick Sync) vs a multi-pass export (which does not). In 10.3.1, Apple even changed the export default to “Faster Encode”, IOW single-pass, and it’s not sticky — if you want multi-pass encoding you have to select this each time. They seem confident about the quality of single-pass encoding.

So the state of Quick Sync transcoding has definitely changed and the current quality level of single-pass encoding seems very high — at least from FCPX.

True GPU transcoding is a misleading term since no long-GOP encoding method (H264, H265, VP9, etc) can be greatly accelerated using traditional GPU methods. The core algorithm is inherently sequential and not amenable to the fine-grain parallelization a GPU can provide. More recently GPU vendors have added separate proprietary fixed-function logic to their GPUs, conceptually similar to Quick Sync. nVidia’s is NVENC and AMD’s is VCE. However these each have multiple versions with varying capability and if software used those APIs it would be vendor-specific and version-specific. IOW if Adobe wrote to NVENC ver. 4, that would only work on those cards which are in some PCs and not on any current Apple computer.

Each GOP can be processed in parallel, but examination of typical GOP sizes shows they are pretty large, so there aren’t enough GOPs in most files to harness thousands of lightweight GPU threads. However if doing purely software transcoding, each CPU core can simultaneously handle a separate GOP, but there aren’t usually enough cores, although the more cores the better. This is why all CPU cores are pegged when encoding to H264, even though the intra-GOP encoding algorithm is sequential — each core is processing a separate GOP.

This might partially explain why Adobe has been so slow at adopting Quick Sync or other hardware-assisted transcoding but I’m guessing there is a deeper explanation which nobody yet knows.

Intel has a PCIE card called Visual Compute Accelerator (VCA) containing three quad-core Xeon E3 chips, which is intended for server-side media transcoding. So some Xeons do have Quick Sync, but only 4-core versions — not useful to a workstation. However in theory you could put a VCA in a high-end workstation and if the software supported it, you’d have tremendous transcoding performance: https://www.intelserveredge.com/intelvca/ It is $2,500 but that’s no more than some high-end nVidia Quadro cards.

[Joe Marler]

[andy patterson] ” I am giving you actual information on why FCPX is faster when using an Intel CPU with Quick Sync. H.264 will render faster on a $2800.00 iMac than on an $8,000.00 Mac Pro. “

It’s not just rendering — it is any action involving decoding (not just encoding) H264, such as playing content from the timeline or quickly going forward or reverse on the timeline using JKL commands. Testing both Premiere CC 2017 and FCPX 10.3.2 back-to-back on the same iMac shows Premiere is not only a lot slower at H264 export but it burns a lot more CPU cycles just playing content or moving around the timeline.

Unfortunately H264 encode/decode cannot be meaningfully accelerated using GPU methods, but Quick Sync (essentially an on-chip ASIC transcoder) does make a big difference. I think Adobe started minimally supporting Quick Sync on Windows but not on Macs. Why Adobe has been so slow to support Quick Sync and why still no Mac support, I have no idea. Until Adobe can figure this out, FCPX will retain a significant performance advantage over Premiere when editing, timeline rendering or exporting H264 media on a MacBook.

It would be interesting to test H264 export performance and timeline CPU utilization on both Premiere and FCPX on the same Mac Pro, which uses Xeon. That would eliminate Quick Sync and would be a more pure software efficiency comparison.

Despite the better efficiency and performance of FCPX, for some common workflows it doesn’t make a huge difference. E.g, 4k H264 is now very widespread and even FCPX isn’t fast enough (despite Quick Sync) to smoothly edit 4k H264 multicam on the fastest iMac made — you have to generate proxy files. Premiere also now has proxy support and like FCPX, once proxy files are generated, multicam 4k H264 is lightning fast. So what’s the difference from a performance standpoint if they both require proxy and they are both fast after generating those? FCPX will still be faster at H264 export (at least on a CPU with Quick Sync) but that is only one element of workflow.

[Joe Marler]

[Joe Marler] ” rewrap the MXF using EditReady which should combine the metadata from the folder tree into the resultant .mov file. You could then import from those .mov files using “leave files in place”. Any workflow decision should be thoroughly tested and verified before adopting it for production.”

In line with the above, note that while EditReady in general supports rewrapping of MXF media, it does NOT support this from a Sony PXW-FS5. The rewrap seems to run but the resulted rewrapped files cannot be edited with FCPX or played with Quicktime. This is technically not a bug with EditReady but Sony is using some kind of special codec variant which can only be interpreted by Apple Pro Video Formats 2.0.5 if run against the original camera file. Note the FS5 is not listed as a supported device by Divergent Media: https://www.divergentmedia.com/devices/supporteddevices

This again illustrates why you must thoroughly test any workflow using the exact cameras and codecs before making an investment and committing to production.

My documentary team recently shot 2.4 terabytes of 4k H264 material in one week. Fortunately it was using camera media which FCPX can import with “leave files in place”. I would never use a camera (like the FS5) which FCPX cannot import with “leave files in place” AND cannot be rewrapped with EditReady. This isn’t a knock against the camera itself, but when you scale up workflow to large volumes, it must be supportable.

Interestingly Premiere CC 2017 has no problem importing 4k MXF material from an FS5 and with Premiere it’s always “leave files in place”. So if shooting huge amounts of MXF material from an FS5, Premiere would have a significant workflow advantage, even if both FCPX and Premiere had to create proxy files. With FCPX the import would have to copy all that to the library or you’d be stuck with a science project trying to figure out is it safe to remove the bare MXF files and import in place.