[Michael Angelo]

REMOTE PROXY ONLY COLLABORATION UPDATE 2015-0126

Discovered some new techniques in the last month so we can once again use libraries with external media for remote collaboration with proxy only drives. This is big as the XML workflow has many limitations: translating multi cam sync changes, compound clip editorial, titles, fx, etc. Now, once again, we can simply share a Library with a very low file size back and forth via the web and editorial translation is perfect!

Below find our latest workflow and this is using the 3 pool method of external media. You don’t have to use this method for the main system, but we find that when creating Proxy only Remote Libraries often FCPX will still copy Optimized Media as well. Having the Optimized Media on a separate drive that we can turn off when creating the Proxy only Remote Library makes this little quark go away, among other benefits. Find more info on the 3 Pool Media Method in my article here or search the web:
https://library.creativecow.net/angelo_michael/apple_fcpx_documentree/1

Here’s details for our new Proxy only Remote Collaboration workflow:

1. Create a Proxy only Remote Library on a travel drive that uses external media
2. When remote editor is ready to share library have them do this:
2a. Create a new Project
2b. Add every clip in their Library to the Project
(no need to do anything else to the clips in the timeline)
2c. Now .zip or .dmg the Library (we call this Transfer Library) upload to web
3. Download the Transfer Library to your main system uncompress file
4. Open the Transfer Library on the main system
5. Change storage location for Transfer Library to the External Location
6. Open the Original Library that you created the Remote Library from
7. Copy the Project from the Transfer Library to the Original Library (the Project that the remote editor created that has every clip in it from the Remote Library)
8. Copy that same Project from Original Library *BACK INTO the Transfer Library
9. The Transfer Library will now link to all of the External Media on the main system. Now you can move all your Remote Collaboration work to your main system.

You may have an issue with the Proxies on the Transfer Library not waking up. If this happens then simply select them, right click “transcode media”. This will not actually re-transcode the clips but rather relink them. For larger libraries this can take a minute but usually we are sharing specific edits and not an entire show, so we can be strategic and only “wake up” the clips we need to transfer. Hopefully this Project swap trick and “waking up” of the proxies will not be necessary in future releases of the software, but for now it’s not too much extra work

Finding a way to work remotely with proxies and sharing work with only small pointer files via the web opens many doors: traveling light with less drives, protecting full resolution material if lost of stolen while traveling, editing remotely on site with only bus powered drives, no need for AC outlets and adapters to edit, tether your phone to a laptop and share edits from a remote location like a beach, share work with complete accuracy that XML cannot provide, the list goes on. And with portable drive size on the rise (we use the LaCie Rugged 4TB portable drive), the possibilities to go anywhere with a large movie edit under your arm becomes a reality.

FCPX is a fantastic and groundbreaking edit system that is really still in it’s infancy. As the product continues to grow stronger, it’s through our sharing knowledge and ideas that we can all be apart of that growth. A big thank you to Creative Cow and it’s community for all the sharing, and to Apple for listening to users and constantly improving the product with each release.

Cheers

MA

Harvest the compromises…

[Michael Angelo]

My pleasure I hope it helps. Good luck with your project!

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Per my post:

“I’ve sent feedback to Apple. Has anyone else seen this behavior? Putting it out there in case anyone has come across this bug, has a consistent work around that is still efficient. This bug makes editing simple string out incredibly painful.”

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hello Elizabeth,

Oliver’s post is a good examination of your end game and finishing outside of FCPX, thank you Oliver:

https://forums.creativecow.net/readpost/344/40379

I put this together so you can get a look at a similar project to yours, explore media management, proxy work flow amd remote collaboration (can be tricky with a big show) as FCP 2009 and FCPX are very different. We are really in the same boat making a documentary with over 10TB footage. We have mixed source DSLR, RED camera, GoPro, archival sources, public domain etc. I hope what we’ve learned so far can be useful and stir discussion and discovery among the hive mind that is Creative Cow.

Like many of the comments in this thread you can be flexible and transcode only what you want to proxies or optimized media, it really is a matter of performance. Given the longer timelines involved with cutting a feature we have chosen to transocde most of our footage to optimized and use the 3 pool method of media management per this thread, keeping each type of media on it’s own drive for maximum flexibility. Most of the time we only use the Proxy drive for editing and only turn on the optimized or original media drives as needed:

1 – Original Media
2 – Proxy Transcodes/Library Files
3 – Optimized Media

ORIGINAL MEDIA
Our RED footage plays natively and performs well without optimizing which is great, so we leave that as is on separate original media drives. We have mirrored back ups of these drives at a separate location in case of fire etc.

OPTIMIZED MEDIA
FCPX does a very good job of transcoding/optimizing media to ProRes but we had a problem of FCPX crashing while transcoding plus it doesn’t do true background processing so takes forever. We used a separate application to transcode optimized media to their own separate drives allowing us to simultaneously work within FCPX, might be useful for transcoding your D90 footage. When we started the project 3 years ago we ran many tests with our DSLR 5D footage plus GoPro and found the only application where optimized transcodes looked as good as FCPX was 5DtoRGB, everything else was of lower quality or had phasing issues. We tested Magic Bullit Grinder, Quicktime Pro 7, Cineform, Apple Compressor just to name a few. I’m sure much has changed since then, but 5DtoRGB did the trick and has decent batch features:

https://rarevision.com/5dtorgb/

If you choose to transcode optimized media to their own drives per 3 pool method, make sure to uncheck “copy files to Final Cut Events folder” upon import otherwise you will create duplicates. Any “non-video” files like audio and pictures we chose “copy files to Final Cut Events folder” otherwise the audio and pictures wouldn’t live with our proxy clips (see PROXY MEDIA below) and this can bite you further down the line if you want to take a proxy only version of the show on the road.

PROXY MEDIA
Once our original footage (less RED) was transcoded to ProRes, we import the ProRes, RED, audio files and pictures into FCPX and let FCPX create proxy media and move the neccesary files to a separate Proxy/Library Files drive. We tell FCPX to store all this media, it’s cache files and back ups into separate folders on the same drive with the library files but not inside the library file itself. This “external media method” setting can be made in “File/Library Properties” and keeps the library files quite small in size. This creates maximum flexibility with regard to collaboration if you have duplicate systems with the same directory structure. By having the library and proxies on the same drive we can easily move our offline proxy edit between multiple computers by simply mounting the drive to another system. Here’s some info on using the “external media method”:

https://larryjordan.com/articles/fcp-x-convert-managed-to-external-media/

MANAGED MEDIA
It is tricky keeping all your media locations straight when you start using FCPX, especially with the 3 pool method, so you do need to pay close attention to where you are pointing and copying media to. This article is a good read on the topic:

https://www.bu.edu/comlabs/media-management-in-fcpx-where-how-to-save-your-work/

The most important thing to understand is that FCPX thinks differently, it works with “managed media” i.e. there is a lot of sensitive metadata that tells FCPX where the media is stored etc and FCPX gets very unhappy when you simply copy files using the finder. Coming from a traditional NLE background this can be pretty frustrating but the “managed media” aspect is what gives the application it’s greatest power. FCPX uses keywords to categorize shots rather than placing them into a single folder, very much like “tagging” media on the internet. This is the future of media management in my opinion whether it’s the internet, computer directories or in our case NLEs. It is especially useful for documentaries with tons and tons of footage to sort thru and organize. Because FCPX used “managed media”, once your clips are imported, you should always copy libraries, projects (edits) and events from within the application when collaborating to truly make aFCPX happy.

FCPX COLLABORATION – WHAT WORKS FOR US
Collaboration is a big topic for us as we have many interns logging footage remotely, editing meltdowns and we take the whole movie edit and source files with us when traveling on shoots. FCPX’s “managed media” works more like a database so it’s very particular about relinking media on other drives compared to a traditional NLE. The safest way we have found to collaborate is to have a master edit system with the 3 pool drive system. Then from within the application (very important) create libraries for your collaborators onto an external drive. Ideally these collaborator libraries should have the same directory structure as the master system (see PROXY MEDIA “external media method” above). After copying libraries, projects (edits) and events to the collaborator’s drive, make sure to click “Consolidate” under “File/Library Properties” otherwise FCPX only moves pointer files and won’t actually copy the proxy media.

The collaborating editor can then take these drives, do their work, return and mount the drive to share libraries, projects (edits) and events from within the application. You can share work without duplicating media by simply choosing not to transfer proxy or optimized media when prompted. We’ve found it is best to have the optimized media drive connected to the master system when importing media as sometimes FCPX get confused about the relinks. Conversely we’ve had to turn the optimized media drive off when exporting libraries and events to external drives at proxy resolution as FCPX gets confused and will export optimized media as well. Here’s some articles on the topic:

https://www.10dot1.co.uk/content/FCPXInASharedEnvironment_FINAL.pdf
https://phantommoose.com/2013/02/24/sharing-final-cut-pro-x-projects/

FCPX COLLABORATION – WHAT SHOULD WORK
Ideally one should be able to simply post a “pointer file” on the web (no media), have someone pull it down, then have access to your work. If the remote collaborators have duplicate optimized media with the same directory structure (“external media method”) then library files can be exchanged and with a simple relink work can be shared, but we’ve had challenges with these re-links. Even so, who wants to travel with over 10TB of optimized media? Having to hook up your remote collaboration drive directly to your master system is a big pain and very impractical when collaborating over a great distance. Proxy collaboration drives are key if you want to collaborate remotely and nothing under the current version of FCPX is dependable. We have had much success collaborating remotely with proxy only drives in the past by “tricking” the software, so it can be done. The problem is that as new FCPX versions evolve these back doors have been shut and this kind of functionality is not being built into the application. I’ve had direct correspondence with the one of the FCPX product managers at Apple, we’ve tried many things but have had inconsistent results. Below are some of these methods that we have used to work remotely with proxy only drives sharing only pointer files in case they become relevant again. When they worked it was fantastic, currently all of these methods are flawed.

FCPX COLLABORATION – WHAT USED TO WORK – EXCHANGE FILES WITHIN PACKAGE CONTENTS
Early on we were able to hack FCPX’s lack of remote workflow by right clicking the libray file, going inside the package contents, grabbing the “CurrentVersion” event file, then posting it for an editor who could then replace that file inside their library package contents. Their library could even be created on a remote drive by copying in the finder and avoiding the “managed media” sensitivity of FCPX. This method worked as long as we had duplicate media. There was no relinking of files necessary and occasionally you would have to re-render some clips. It worked like gang busters until FCPX 10.2.0 where something in the code changed. The beuaty of this method is the “CurrentVersion” event file is tiny, fits in an email. I really wish Apple could build this kind of functionality into the product natively. BTW this is a big no no messing around inside package contents, I don’t recommend it despite our success. Here’s where we learned the trick:

https://forums.creativecow.net/thread/344/11304#11309

FCPX COLLABORATION – WHAT USED TO WORK – EXCHANGE LIBRARY FILES VIA FINDER
If your proxy media is stored externally (see PROXY MEDIA “external media method” above) and you have a mirror drive with the same directory structure and same drive name, you were able to simply exhange the library file and see eachothers work remotely. This would work even if you had a proxy only drive and no re-linking was necessary. Unfortunately with more recent version of FCPX the “managed media” can tell that it’s not the same drive despite having the same name, the proxies won’t link. There is no way to relink proxies only in FCPX, so if you are working remotely on a proxy only drive you are out of luck. Here’s what happens if you link to the proxies, all hell breaks loose:

https://forums.creativecow.net/thread/344/31389
https://discussions.apple.com/thread/7053063

FCPX COLLABORATION – WHAT KINDA WORKS NOW – XMLs
XML is the most successful method that we have found to share work remotely without having to hook up the remote drives and by only sending small pointer files. It seems to work much more like a traditional NLE with paths stemming from the library file not the drive so it is able to relink to mirrored directory. The rub is that it makes mistakes like not transferring all your edits, losing any multicam sync changes, axis change errors and more. It is however reliable enough to get an idea of the collaborators edit decisions or logging and keywording that is done. It’s usually best to pull the XML from the collaborators library or event as pulling directly from the clips in latests versions of FCPX can cange the case of keywords. Here’s a good article on this work flow:

https://larryjordan.com/articles/fcp-x-collaboration-between-two-editors/

OFFLINE EDITING TOOLS
FCPX will make you pull your hair out over the traditional tools that are missing, but jump for joy at the incredible metadata and keyword workflow features. Learn how to properly name source material, set roles, keyword on import via your external directory structure before you start using FCPX. It’s lightening fast compared to FCP 2009, no rendering needed to see most things given the 64 bit architecture, it’s really coming along considering Apple is building this from the ground up. There are some crazy things like no way to undo a mark (insane) and if you get lots of keywords (like you will with a long form footage heavy show) you can’t expand the keyword window to see them, stuck scrolling in a tiny window, but life goes on. The magnetic timeline is really gonna take you a minute to get used to, best to not fight it and see where it takes you, here’s a good article:

https://library.creativecow.net/austin_charlie/FCPX-Timeline/1

ONLINE EDITING TOOLS
People are constantly making new tools for the software and Apple is charging features pretty hard. The color corrector is the weak link so I’m guessing most people finish color outside of the app. You can do a lot inside the app, previewing is a snap but the tools are a bit too automatic for the high end finisher, really depends on how much control you desire and what kind of things you need to do. Since you are coming from FCP 2009 it will blow your mind with regard to tools. We will likely use a mix of FCPX, After Effects, Resolve and Flame.

CONLCUSION
Despite remote collaboration workflow issues and some old school editing functionality, FCPX is a very powerful offline/online tool. It really is the perfect edit tool for long form documentary with mountains of footage if you really dive into it’s metadata features and keywording. I hope you can find some useful info in here, we have such similar projects thought it wise to share in case it helps you and others. If you have any questions feel free to PM me and good luck on your project!

Cheers

MA

PS This 4TB Lacie drive is awesome if you want to take your show on the road to edit. It holds a proxy version of our entire show including source clips, edits, library files etc (less hi rez stills), no power needed if you use the thunderbolt:

https://www.bhphotovideo.com/bnh/controller/home?O=&sku=1135375&gclid=CODV37zxhskCFY9bfgodkKcLiA&is=REG&m=Y&Q=&A=details

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hey Jereme,

I like the creativity of your suggestion to work proxy only as original source footage, definitely thinking outside of the box, nicely done. It does however create a number of obstacles. Mainly the inability to do any concurrent online work, breaking apart an established pipeline after more than 2 years, creating proxies for 1200 hours of media into a new pipeline, not using the software to it’s potential, no full full rez exports, the list goes on.

The big motivator to find a proper remote proxy workflow are not only the many benefits of using the software to it’s full potential but also the fact we had a perfect working system exchanging the “CurrentVersion.fcpevent” file until 10.1 (see original post), there is a way!

Great ideas though, again thank you

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hey Jereme

Proxy size is not at all the issue. We have very little RED footage and all the proxy sizes are quite standard. The question is what is the best work flow to share work while editing and logging remotely with proxies only using FCPX (please see details in my original post).

Appreciate your time and energy

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hi Jeremy,

So far 16TB of optimized media and more to transcode, proxies only systems are essential to our workflow.

Thanks for thinkin’ ’bout things.

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hi Jeremy,

RED, Canon 5D and GoPro

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Little update regarding this shareware (it’s call Keyword Extractor) did have a problem with one of my events out of 7 today and wasn’t able to parse the XML data due to a negative duration it found. It’s not 100 reliable but hey it’s free and does work most of the time.

Hopefully editing and printing a variety of data will be standard for FCPX soon as it’s a no brainer the variety of uses!

MA

“A life without cause is a life without effect.” -Dildano

[Michael Angelo]

Hey Cow People,

This free software works great to export event keyword data: https://www.spherico.com/filmtools/

It does group the keywords comma delimited in one column so I did have to use excel to split into different columns and rework those columns a little but only a few minutes. In the end I have a sortable road map to all the clips and their corresponding keywords with timecode in and outs!!

This is heaven as we have up to 6 people logging and sorting footage for a documentary with over 1200 hours of footage, provides a fantastic checks and balances resource.

Hope this helps.

Mangelo

“A life without cause is a life without effect.” -Dildano