Consolidate Weirdness

[Eric Santiago]

Another oddity with Consolidation is that once done and you copy the Library to another volume, you still have to do the hellish dance of relinking all the files to that newly copied library.
So I’m forced to un-mount drives just so FCPX can play nice.
Is this normal?
I don’t know if this is since I never had to move so much data.
Of course, with all this going on I find out that one drive has died.

Good times 😛

[Bret Williams]

It’s creating folders because you don’t have it set to consolidate the media in the library. It duplicates terabytes instantly because it doesn’t duplicate the actual media, it just creates hard links. Actual media files that share the bits of the other files. And once you copy to another volume you have to relink becaause the file path is absolute, not relative. It’s looking for a particular volume name and a specific file path. Change the name of the new drive to the same as the old drive and mimic the path and you won’t have to relink. Or just relink.

_______________________________________________________________________
https://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag https://bretfx.com/product/hang-tag
Overshoot Text https://bretfx.com/product/overshoot-text/
Outliner https://bretfx.com/product/outliner/
Clock Maker https://bretfx.com/product/bretfx-clock-maker/

[Eric Santiago]

Thanks, Bret that makes sense.
Strange I know for sure I selected Library during this process and it still creates the external folders.

[Eric Santiago]

Just looked at them again and they are no shortcuts to the original.
They are actually big files.
Again this is weird stuff.

[Jeremy Garchow]

[Eric Santiago] “Just looked at them again and they are no shortcuts to the original.
They are actually big files.
Again this is weird stuff.”

Can you please explain your workflow in more detail, and the problems you are seeing?

Obviously, multiple TBs can’t be duplicated in seconds, so there must be something that is happening that isn’t normal.

[Bret Williams]

Not shortcuts. Hard links. They’re real files, not aliases. BUT they’re simply real files utilizing the same bits and bites as another file. High Sierra supports this function directly in the finder when duplicating files on the new drive format, but FCPX has been doing it for years.

_______________________________________________________________________
https://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag https://bretfx.com/product/hang-tag
Overshoot Text https://bretfx.com/product/overshoot-text/
Outliner https://bretfx.com/product/outliner/
Clock Maker https://bretfx.com/product/bretfx-clock-maker/

[Eric Santiago]

[Jeremy Garchow] “Can you please explain your workflow in more detail, and the problems you are seeing?”

I have two projects both having the same issues.
Basically, the client gave up on editing this beast due to the sheer amount of data gathered.
I mix of RED, Canon, stock etc… spread across a few slow horrible USB 3 drives.
From past experience I would do the following:

1. Select Library and apply Consolidate Library Media. If I want it on a new volume I would choose that option which I have at times. However, for this one I specifically selected Library.
2. Select both Optimized and Proxy. And yes at times I will use the “Delete Generated…” option but only if am confident that I have all Original files. This has bit me in the butt many times.

Pretty straightforward as far as my workflow which is why I find it odd when the folders come up like that.
These projects are also daunting due to the Optical Analysis issues I am having (on another topic here).
This one project has no Optical Flow options used but tends to show as “Transcoding and Analysis” with Optical Flow Analysis as the culprit.
I can ignore it but it is keeping me from quitting the app without having to do a hard crash.
On top of that, a few files are missing due to a dead USB 3 drive but the client has ensured me the files exists at her end.
The issues with the Consolidate isn’t killing me just puzzling.
I know now that I won’t be taking anything like this again in the future.
I don’t mind dealing with thousands of RED files but mix crap formats and sketchy files is too much for me 😛

[Jeremy Garchow]

Can you provide a screen grab of the “folders”?

Usually what happens during consolidate is FCPX creates a symbolic link at the new destination (has a little arrow on it like an alias file) and then FCPX copies the files one by one to the new destination. Once the file is copied, the symlink goes away and is replaced with a full copy.

When consolidating, FCPX does not retain the external folder structure, and this may cause problems with red media, especially if you have spanned clips as FCPX probably doesn’t know to grab all those spanned clips as well.

Are you trying to trim the amount of media down to only transfer “used clips” or what gain is there to using the consolidate function instead of just a Finder level file copy?

Also, have you used the “Copy” function instead of the “Consolidate” function?

[Eric Santiago]

[Jeremy Garchow] “Are you trying to trim the amount of media down to only transfer “used clips” or what gain is there to using the consolidate function instead of just a Finder level file copy?

A few years ago, the copy Library at Finder level was fine for me.
Lately, they don’t seem to link to its own new Library (via copy), so I was faced with having to relink a ton of files.
Also, copying a Library via Finder is futile when it’s that big.
I’ve had failed attempts and sometimes power outages causing more work for me.
I know, get a proper power. Working on that.

[Jeremy Garchow]Also, have you used the “Copy” function instead of the “Consolidate” function?”

Same issue as Finder copy (for me). I tried this the first time and for some reason, I spent more time relinking.
I tested this workflow on a few files first but with this behemoth of a project, it was killing FCPX.
The last attempt I had to kill FCPX.

I have left this issue alone now since I spent way too much time searching and finding other ways to get around the constant crashing.
And of course with the Optical Flow Analysis issue, I think I need a break from this.

[Bret Williams]

I hear ya. i recently edited a 3 min corporate piece that was shot in 8k Red Raw. At one spot in the video I did a 4×4 grid of 8k Red Raw video. Of course I worked completely in Proxy, bringing that grid down to 4×4 grid of 4K ProRes Proxy files. Needless to say there was no real time playback.

_______________________________________________________________________
https://BretFX.com FCPX Plugins & Templates for Editors & Motion Graphics Artists
Hang Tag https://bretfx.com/product/hang-tag
Overshoot Text https://bretfx.com/product/overshoot-text/
Outliner https://bretfx.com/product/outliner/
Clock Maker https://bretfx.com/product/bretfx-clock-maker/