Is my external storage my bottleneck?

[Sam Frazier, Jr.]

I produce and edit an unscripted web series that involves a live event where we shoot around 9 cameras in varying formats and frame rates. I’ll describe them more below. I have two external storage devices:
– OWC ThunderBay 4 w 4 x HDs in RAID 0 and connected via Thunderbolt 3. It runs around 600 MB/s read and write.
– Oyen RAID 0 using 2 x SATAIII SDDs connected via USB C and runs more like 850MB/s read & write.

I think I can tell a definite difference b/t these options as the Oyen seems snappier. But, I don’t understand how this can be the case given that I don’t think the OWC Thunderbay is being overloaded. Given that the Thunderbay is supposed to be running at around 600 MB/s and the total MB/s for all these video sources should be 364 MB/s, I wouldn’t think the Thunderbay would be the bottleneck. I’d think it comes more down to CPU or GPU power on my 2017 iMac. And yet, I’m pretty sure I can tell a difference when I edit off the Thunderbay vs the Oyen on the same iMac.

So, any thoughts? Is my math off? Am I misunderstanding something?

Here’s how I calculated that given the cameras we use:

Panasonic EVA1 running 108 frames a sec in v-log at 46.6 Mb/s = 5.825 MB/s
– I had two of these cameras running but sped them up to around 450% to play at real time. Accordingly, I raised the MB/s by 4.5, so that = 26.2 MB/s. Two of them = 52.4 MB/s
Panasonic EVA1 in 24p = 402 Mb/s = 50.25 MB/s
Sony FS700 1776 mb/s = 222 MB/s
Nikon d7000 = 19.2 Mb/s = 2.5 MB/s
Panasonic GH2 = 33/5 Mb/s = 4.2 MB/s
Drone = 60 Mb/s = 7.5 MB/s
2 x GoPro 6 Black = 66.3 Mb/s = 8.3 MB/s each. So, 16.6 MB/s total.
Canon 7D = 41.4 Mb/s = 5.2 MB/s
iPhone 8 = 17.3 Mb/s = 2.6 MB/s
total = 364-ish MB/s.

[Neil Sadwelkar]

You’re playing a complex collection of formats, frame rates, and processing of images. Both your drives have more than enough speed to mathematically accommodate the required data rate.

So, there are likely to be one (or both) of the following factors

CPU-GPU speed in processing all these in real time

Drive latency, which limits a drive’s ability to read many small files in succession, quickly enough.

An example of latency and the limits it imposes, is a drive that can easily play one uncompressed video of, say, 2GB needing about 200 MB/sec data rate, finds it hard to play a 2GB image sequence of thousands of 7 MB files.

[Eric Santiago]

We own two Panasonic EVA-1s for my day job.

Dealing with that data is nowhere as easy as my personal camera a RED DRAGON 6K.

If you really want to avoid doing math, conform all EVA-1 media to ProRes422 or ProRes444 (not sure if there is any benefit to this one).

Another issue is that the EVA-1 media don’t play nice with Resolve.

My general workflow at work is FCPX/PPro > Resolve and back.

If I know I am not round-tripping then I proxy the lot in FCPX/PPro.

It’s safe to say that the other formats you’re dealing with should be easier to work with.

But again, I would convert them all to an editable container (ProRes).

[Sam Frazier, Jr.]

Thanks for that! I didn’t know latency could work that way w/ finding it hard to play an image sequence of thousands of smaller files. But, I have that going on in my timeline as I tend to keep many layers of alternative clips as options. I was thinking the latency was just more of the issue of SSD starting up faster that spinning drives, kind of like electric vs gasoline cars.

But yes, I highly suspect I’m also dealing with a bottleneck in my computer CPU and GPU. It’s probably time to upgrade.

[Eric Santiago]

Keep in mind that the codec can also cause the bottle neck.

H264 and AVC is known to cause CPUs to crawl during editing.

[Sam Frazier, Jr.]

Thank you for that. Can you tell me a bit more about how the EVA1 doesn’t play well w Resolve? If you transcode EVA1 files to ProRes, does it work better? I’m tempted to switch to Resolve for any number of reasons, including speed and PPro is about as buggy as that scene from Indiana Jones and The Temple of Doom…

As for my edit in PPro, I made offline ProRes Proxy files of all the clips and their online was their native files. This was impossible to playback w/ rendering first.

[Eric Santiago]

The last time I ran into an issue with EVA-1 and Resolve was a few weeks ago.

The files would not show data.

This was on a 2019 Mac Pro and R17.

I would have to try again what the results were.

I took the files into AME and converted to ProRes 422 to fix it.