Pawel Kasprzak
Forum Replies Created
-
Another thing about shared environment.
Version 10.1 made it considerable easier, but for many reasons we have to use proxies here. Various scenarios described in a cited paper rely on XML exchange. This, btw, can be easily done without exporting but simply replacing appropriate “currentversion” files from the Finder level. The advantage is that proxies jump in, which doesn’t happen with the project or event imported from XML.
In such cases proxies can be manually added (Finder again). Worth noting however this doesn’t work if the events structure and content changes within the libraries – the proxies, even while copied, don’t show up.
The paper doesn’t care about proxies that much assuming that thanks to powerful hardware solutions it’s easy to create and to store them. Well, our smallest libraries are still over 1TB in size.
What could help a lot would be to make proxies external. I don’t see the reason why they can’t be shared if originals can. But first of all it would help a lot if I could relink not only originals but proxies as well.
-
The article is informative indeed, but – as it suggests – there’s still a lot of testing ahead of anyone who wants to use FCP X in a complex shared workflow and a great deal of uncertainty as well, I’m afraid.
Picture a situation. You have a load of ingested HD media. Say 500 hours and you have it stored on an xSan networked drive. Most of that media comes in multi clips, say, 8 angles each, so forget about doing your job pulling the media directly from the server. You need proxies that are local. Your 500 hours is stored in 5 different events (A, B, C, D, E) and you have 4 (a, b, c, d) editors working on them editing different parts of the same episode.
Even though you have different events, there’s no way to tell, who of the editors will use which of them. They all need all five. Obviously there must be a collection of those events (say, single library) that has all proxies generated and identical copies of such “base library” is sent down to editing workstations (single file libraries make this easier). The base library contains things like multi clips and compound clips, which is worth noting.
Now editor “a” makes three different projects using media from events A, B, C (he’s actually barely aware of the relation between his projects and events – soon after he finishes, he hasn’t got the faintest idea, which of the events he used).
Editor “b” makes another two projects, based this time on B, C, D, E events. Etc. Finally one of the editors will have to put all those projects together and do some final editing that requires full access and involves some reediting.
If A, B, C, D, E were just events storing media plus some metadata like keyword collections, then a sheer XML export of the project file(no events required) would do the job, because FCP would simply create an appropriate event that contains all of the media required. There will still be some mess about the proxies and how to link them, but at least an EDL-like information is saved and secure. The thing is projects use multiclips and compounds. Those can’t be relinked when the event is missing. So obviously anyone exporting and exporting any project XML must have appropriate events loaded. And must know which of the events to load. Obviously all five should be loaded. This was sometimes tricky in FCP 10.09 (big events would sometimes take a day to open), now it’s way faster in 10.1.
But editors do their own multiclips and compunds during the job. To make things worse they also modify what was done previously in their base libraries. They import new files like music and they of course do create their own events.
So having 5 events and 4 editors you instantly end up with at least 20 events that must be loaded to see all of the edited stuff. And it’s just to start with of course, because people continue to work so that you not only have an original “base” event A, and not only its’ mutations Aa, Ab, Ac, Ad made accordingly by editors but also versions Acb and so forth – exponentially.
Of course this all can be done, but…
I think shared workflow will become a workflow rather than a total mess only after Apple finds a way to make libraries really shared with read and write access in database terms. Not that while one guy edits things others can only read (impossible now) but that all of them have a read-write access and only the “record” being currently edited gets blocked.
-
[Jeremy Garchow] “Even with Avid, you can’t have two people writing to th”
What Avid offers is still not really a database. In a database system only a single record in the database gets blocked and only for editing when some (authorized) user keeps working on it. Others can still read it seeing this record with data prior to last changes. The rest of the bin, folder or table remains accessible (according to privileges) to others in the network. Adobe did it pretty much this way.
You edit files metadata with the soft called Prelude there. What it does is it adds a text file that shares the same file name with your media. This file stores subclip information for use in Premiere and is saved in the same location as media file. Each time you import media to Premiere, the data from this file gets loaded as well. Even more – Premiere uploads current versions of metadata files on configurable time intervals, so from within your Premiere project you see current updates showing up as your loggers add new data. Certainly this all works in the network and only the file corresponding to a single media file that’s being edited gets blocked for editing, being readable for others.
I think the car analogy doesn’t really apply here. It’s not just car – it’s a spaceship or something 🙂 It’s being steered by a crew of people.
-
Jeremy,
No – I’m pretty new to Premiere, but so far I love what I see. I will for sure test it more. And our tests are really going to be crash tests – considering this lots of media we use, concurrent video streams we play etc.
It’s interesting what you mention about manually moving the proxies outside the event folder. I tried this – the proxies were stored on our xSan volume and took some tricks to fool FCPX into reading it from there. It didn’t work though – I got dropped frames which made editing impossible.
Looking at the transfers I thought FCPX reads San data some other way FCP 7 did. Reading multiclips from San under FCP7 you see pretty much constant transfer rate, under FCPX it gets jumpy.
The only configuration I find working is to store everything locally. New iMacs plus one of those LaCie 20TB thunderbolt arrays seem really great solution, OpenCL acceleration works great etc. Still – as our events swell we are in trouble. Pretty annoying to wait half a day for your events to get updated when your deadline is coming up…
-
[Walter Soyka] “I don’t think lowering the resolution will save you any disk bandwidth. Frame data is usually compressed and always encoded. Any application has to read the entire compressed frame from disk before it can decompress it for processing.”
This is what makes me wonder. I haven’t got the slightest idea how this is done, but I saw disk transfers on playback and it decreased when playback got switched to lowered resolution. As if Premiere was reading every other pixel – amazing considering what you wrote, true for sure. As if they got their way into the codec’s compression algorithms being able to still read sort of half the data required. Just give it a test – you get dropped frames on full res playback (clearly due to transfer issues), you switch and dropped frames vanish plus you see decreased network / disk transfers.
-
Jeremy,
True – back in the old days of FCP 7 and previous versions there were tiny little files as well. At our site we badly needed a central storage of media files and these included render files (so that each sequence could be opened rendered on any of the edit stations). But also cache files (especially waveform cache) were needed – these had to be common as well so that every editor could see them correctly. These – as you sure remember – were kept in a flat structure (single folder), there were lots of them and they were small in size. Stored on xSan volume they tended to slow the entire system down dramatically, so we learned to use another drive (just HFS+ formatted) and everything worked fine.
Such choices are not possible in FCPX anymore. Of course you can use media files that are external to your event folder, but proxies must be kept along with the rest of event. And we badly need proxies, typically using 9 angles multiclips, sound recorded as multiple separate files that must play in sync etc. We typically have way more than a 100 hours of video within a single episode we make and what we actually need is a common pool of over a 1000 hours of video. It’s not only a matter of bandwidth but also the fact that some 40 different files are being read on playback at a time. So we have the same volume for everything, the same block size parameter, while in fact this is an either-or choice. Our events tend to swell and beyond some level the system slows (I don’t know what really matters here and can only guess – the overall size of the media, the complexity, the sheer number of clips). It takes forever to get an event open and you have to wait quite a while before your playback starts. Pretty annoying when you try to edit to the music so that people dance to it or something. I can’t be sure what really causes these performance issues but my bet is that file sizes and inappropriate block sizes are one of the reasons.
You’re right – San can be and in fact is a bottleneck for our production, but avoiding San storage solves only a half of the problems we face. We still need RAIDs (even though local) and the formatting optimization remains a problem.
Plus the database mantra as someone pointed this out here – it’s a sheer absurd. I myself love the way keywords collections are organized (still some bugs in it though). But what’s the use of the database only single user can read at a time?
I am now in charge of another major production here which we chose to do with FCPX and we are having hard time here. Adobe and Premiere Pro is still an option for us. I don’t have to generate proxy files – if my San network proves a bottleneck (well it should easily allow for the transfers required, even though it involves 9 streams of HD video plus numerous audio tracks) we can always switch to half or quarter resolution and everything works just fine. It’s way faster than switching to proxy playback in FCPX plus I don’t need to render and store those files. Adobe’s Prelude solution for metadata works fine as well – only the metadata coming with the file that’s currently being edited is not accessible (to write – not to read) by other users. The only reason we actually went for FCPX now is that it offers kewords for multiclips and compound clips. I’m now not sure if it was really worthy as several overlapping ranges in keywords (and that’s what we need) cause errors.
After years spent with FCP we find it easier to switch to Adobe as it is simply more similar to old FCP. The advantages of FCPX are clear, but I feel it’s still a promise rather than a fact.
-
It’s actually less than it was in FCP 7. Then you could create a project file, make it read only and this project would store the metadata descriptions you needed for a networked workflow. This can’t be done now which is a shame really, even though the potentials of database architecture are really great.
I think what’s still missing in asset management tools like CatDV is that you can only work directly on media files and not on multiclips or compound clips the way FCPX handles them.
What’s also important for networked strategies is how media and metadata are organized. In FCPX you have no other choice that to store everything on the same drive. The only flexibility you have (thanks God) is to leave imported media files where they are, not really copying them. Unfortunately you’re not free to do the same with proxies. Nd you badly need proxies when it comes to editing more that 4 streams of HD in multiclips. All proxies or optimized media must reside on the same drive where the rest of your event stuff is recorded. When it comes to San strategies (this is never mentioned in Apple’s whitepapers) you end up having media files (so proxies most of all) saved along with billions of tiny thumbnail, audio peaks and other data files, which make your San volume (optimized for large media files) go crazy. Block size parameter remains an issue as well if you choose to store everything locally.
-
I think you’re wrong. The database is useful if and only if there’s a bunch of people that can access and use it. This is most of all about read (and not write) access – which is something not at all allowed under FCPX. You can have a single user accessing a “shared” library at a time. Normally – speaking of databases – you should have a crowd of them. Plus this crowd should instantly see updates made by those who have sufficient privileges. This is not at all possible and even though of in FCPX, which is a shame. So the car analogy doesn’t really apply here. What we need is several drivers seeing the same street lights, right?
-
Apple thinks of networked storage as if of could. It’s just files repository. Of course you can link your original files to several libraries owned by different users to have a multiple access to the same media files – but the metadata database is designed for single user only. Making a library or event read only would do a part of a job, but a real solution is probably of a kind Adobe has developed along with its’ Prelude. Here you have a text file that’s stored along with the media file – this is where the metadata description is being stored. So what gets blocked from multiple access is the metadata of the clip that’s being edited by someone over a network. You can still have access to everything else in the folder (which translates to event pretty much in FCPX).
-
Adobe lets you access your media file just as you’d naturally do it – simply importing them from a network (San or other) location. It’s then up to the bandwidth available how many users can really use this for editing. Pretty much the same what was available in FCP 7.
Plus Adobe has this nice solution to reduce playback resolution. The result is pretty much the same as if you used some proxy files – except you don’t need to generate them. The soft just reads every other pixel of the video (half the resolution or every fourth if you choose quarter the resolution) the result is that dropped frames vanish and you see smooth playback with frames slightly blurred (hardly visible in most cases).
Plus – Adobe’s Prelude stores metadata descriptions within test files that are stored along with media files. Any time you import such media into a PremierePro project, the metadata is loaded as well. Even more – PrepierePro refreshes media at time intervals so from within it you can see new metadata coming up as someone else in the network adds new records to the database.
Adobe’s database design is poorer than the one FCPX provides, but it’s a real multiuser accessible database. In FCP you can only have a single user connected at a time. But there’s no way to have multiclips or preedited sequencies described that way in Premiere / Prelude. FCPX allows for that which is great.