-
Folder Structure and File Naming
The more I read up on CatDV the more important I realise the folder structure and file naming is in terms of organising the ‘backend’ of CatDV. Currently I am using CatDV Professional (non server, non worker node, but certainly coming to that later this year).
I am organising footage from a wide range of sources and a wide range of time dating back many years. It is non project specific but stored in an archive style ready to be used when needed. Lets think of it as a stock footage library as that what it and I imagine a lot of small / medium production facilities could be using CatDV as and what closely represents my own uses.
I am trying to come up with a folder and file naming convention that will best organise the footage in a raw file based non CatDV reliant structure (mainly as I see it to be important as a backup to be able to manually locate incase there is a software issue, loss of computer etc etc) as well as to allow CatDV to ‘mine’ data from the folders and filenames. From what I have read in a few posts this is something CatDV is capable of doing and becomes very powerful when used eventually with the Worker Node. In fact from what I understand the worker node can in essence be programmed to do most of this organisation for me.
Essentially though we still need to have a structure in place to begin with. Here’s what I am currently thinking in terms of folder structure:
RAID VOLUME / YEAR / MONTH / COUNTRY / LOCATION / FILE FOLDER / File (file name will include as much data to make it usefully identifiable)
So something like as an actual example:
RAID VOLUME / 2012 / April / France / Paris / 2012_04_23_france_paris_city_broll / 2012_04_23_france_paris_city_broll.mov
So I can see from this file the date shot and a simple description of what it was. Of course the CatDV metadata will hold much more information on it, such as cameraperson, producer, markers with shot details, weather etc etc
I am wondering what others use as a folder structure and naming convention? Of course it will be different for everyone but perhaps there is some general rule to doing this kind of structure. I would be interested to know how some of the bigger facilities organise their data especially when dealing with tapeless clip based media (hence the importance of the file folder name).
Any advice on this would be greatly appreciated especially in reference to a long term structure that can adapt to the worker node and server solutions.
-
20 Replies
-
Well from my perspective you’re over thinking it.
I’m in pretty much the same situation. about 16 years of footage, dating from the time we switched from film to video. For the last 13 or so years we’ve created a great deal of derivative product from all the different material we’ve shot and continue to shoot. It’s all documentary in style. Not a spoken word is scripted. Lots of interviews and lots of transcripts. We create video-based products that are sold all over the world.
A few years ago we even changed the way our accounting works to match the way we actually create product. Whenever we shoot it’s research towards making an at-the-time-undetermined project. Product creation doesn’t begin until we sit down to put together a project in edit. The best way for us to work this way, and keep footage somehow compartmentalized and still make some sense of it all, was to separate it out by shooting location. So we have this factory, or that office site, or that hospital or school. Reel names (we shoot HDCam on a 750) use as two or three letter designator for the shooting location. So I can see a note or a line out of an EDL, or a tape case and know immediately what “shoot” it came from.
Ok. So the majority of the time I’m using FCP to load the footage. The project goes under the location name, say “Unity Hospital.” What become important later is the path to the directory Unity Hospital. Doesn’t matter how or where that footage goes later. I just need to find it. I use FCP to mark sections to load and label them such as: UN07 cafeteria cover 1.mov or UN10 Lacy Darcy interview part 2.mov {Note that I’m using ProRes for my files). When the shots are captured I bring them into CatDV, which will remember just exactly where the footage is. I can then parse them out even further with all sorts of key identifiers and markers and transcripts. I also create proxy movies of everything that live on a separate drive attached to one of my systems. Because I’m using “paths,” the structure is the same as the original footage.
Lately I’ve been moving footage off onto external drives for storage, and to clean out my systems. I’m sure some lucky souls get to use CacheA but for now we’re using raw external drives. It’s quick and easy and I can mount a drive using an eSata dock directly to one of the systems and edit from the raw drive when I need to. But the real change has been moving the original footage off to one of my archive drives FIRST and then bringing the footage into CatDV and making proxies from there. I’m doing the vast majority of my editing using the proxy movies, NOT the originals. As a result I have 100 of my footage from ALL shoots available ALL the time. Originals don’t come into play until I’m working on finishing. Then I can mount archive drives and conform/consolidate/relink or do whatever. It’s been quite slick. And by moving the originals to an archive drive first I’m not reconfiguring CatDV all the time to find the footage for both the originals and the proxies. I also have a portable clone of the proxy drive for working offsite (I used to just call that home). Because we’re using “paths” for proxies CatDV (and Final Cut) always thinks it knows where to look for a shot.
Sure, you can do what you suggest, but I think you’re adding a layer of complexity that’s not really going to help you that much and it will eat up even more prep time. Takes me back to the days when I setup the table of accounts for my old company’s accounting system in Quickbooks. I had everything parsed out to the Nth degree. I could have told you how much black gaffer’s tape was used on training projects versus commercials on any named Thursday. Of course I never did, nor did I ever actually need that level of detail. In the end it was a big lesson learned. One of the big lessons with CatDV is that you can add fields and descriptions later without messing up everything you already have. Are you willing to go back and redo your directory structure if someone suddenly decides if they want to know if shots were done under cloudy or sunny skies? Ouch.
Non-linear: all the time and nothing but.
-
Looking at your proposed directory structure and file name you have everything in triplicate. Just as it is you could drop four levels of directories.
Non-linear: all the time and nothing but.
-
Well from my perspective you’re over thinking it.
I’m in pretty much the same situation. about 16 years of footage, dating from the time we switched from film to video. For the last 13 or so years we’ve created a great deal of derivative product from all the different material we’ve shot and continue to shoot. It’s all documentary in style. Not a spoken word is scripted. Lots of interviews and lots of transcripts. We create video-based products that are sold all over the world.
A few years ago we even changed the way our accounting works to match the way we actually create product. Whenever we shoot it’s research towards making an at-the-time-undetermined project. Product creation doesn’t begin until we sit down to put together a project in edit. The best way for us to work this way, and keep footage somehow compartmentalized and still make some sense of it all, was to separate it out by shooting location. So we have this factory, or that office site, or that hospital or school. Reel names (we shoot HDCam on a 750) use as two or three letter designator for the shooting location. So I can see a note or a line out of an EDL, or a tape case and know immediately what “shoot” it came from.
Ok. So the majority of the time I’m using FCP to load the footage. The project goes under the location name, say “Unity Hospital.” What become important later is the path to the directory Unity Hospital. Doesn’t matter how or where that footage goes later. I just need to find it. I use FCP to mark sections to load and label them such as: UN07 cafeteria cover 1.mov or UN10 Lacy Darcy interview part 2.mov {Note that I’m using ProRes for my files). When the shots are captured I bring them into CatDV, which will remember just exactly where the footage is. I can then parse them out even further with all sorts of key identifiers and markers and transcripts. I also create proxy movies of everything that live on a separate drive attached to one of my systems. Because I’m using “paths,” the structure is the same as the original footage.
Lately I’ve been moving footage off onto external drives for storage, and to clean out my systems. I’m sure some lucky souls get to use CacheA but for now we’re using raw external drives. It’s quick and easy and I can mount a drive using an eSata dock directly to one of the systems and edit from the raw drive when I need to. But the real change has been moving the original footage off to one of my archive drives FIRST and then bringing the footage into CatDV and making proxies from there. I’m doing the vast majority of my editing using the proxy movies, NOT the originals. As a result I have 100 of my footage from ALL shoots available ALL the time. Originals don’t come into play until I’m working on finishing. Then I can mount archive drives and conform/consolidate/relink or do whatever. It’s been quite slick. And by moving the originals to an archive drive first I’m not reconfiguring CatDV all the time to find the footage for both the originals and the proxies. I also have a portable clone of the proxy drive for working offsite (I used to just call that home). Because we’re using “paths” for proxies CatDV (and Final Cut) always thinks it knows where to look for a shot.
Sure, you can do what you suggest, but I think you’re adding a layer of complexity that’s not really going to help you that much and it will eat up even more prep time. Takes me back to the days when I setup the table of accounts for my old company’s accounting system in Quickbooks. I had everything parsed out to the Nth degree. I could have told you how much black gaffer’s tape was used on training projects versus commercials on any named Thursday. Of course I never did, nor did I ever actually need that level of detail. In the end it was a big lesson learned. One of the big lessons with CatDV is that you can add fields and descriptions later without messing up everything you already have. Are you willing to go back and redo your directory structure if someone suddenly decides if they want to know if shots were done under cloudy or sunny skies? Ouch.
Non-linear: all the time and nothing but.
-
From what I understand you need to have a path that is constant for the ‘Full Res’ footage. The folder and file structure is then identical for the proxies. Are you able to have multiple drives for the full res that you can slot in and out? How does this affect your proxy creation?
I will not be keeping much of the full res on file to start with, currently bluray data and then cacheA. The proxies are kept on a Raid 5.
I have read conflicting arguments for having a deep folder structure and for not (like yours). Metadata like weather (cloudy/sunny) or course will not enter the folder structure or file name. I don’t really see it as being too complex, almost essential.
Lets say CatDV (for what ever, hypothetically) stopped working, how well would your structure hold up?
In regards to having everything in triplicate, could you expand on this? The reason I may need them like structured like this is for using the worker node to organise my footage in the future where essentially it can organise footage into the right year and country and location based on given fields in CatDV.
-
Ok, there are some semantics that we have to be careful of here. It’s EASIER if your paths are consistent. That MEANS: if you put them in one place, it’s hard for CatDV to find them if you move them.
Translation: Don’t keep changing where you store them. It doesn’t mean you can’t store them in multiple places. CatDV keeps track of where everything is. If you move them it doesn’t know where to look. However, if you move them you CAN tell CatDV where to look and all will be fine.
So, best thing to do is put them in a place, read them into CatDV and then let CatDV make the proxies. But because you are using paths (and not tape names) you can use other software, like Compressor, MpegStream Clip or something else to make the proxies. You just store them on your proxy drive(s) using the same structure: Volumes/Archive 1/Paris becomes for the proxies: Proxy Drive A/Volumes/Archive 1/Paris
you don’t want to have too deep a directory structure because you’ll run out of space for the OS to deal with it. I’d have to look up what the limits are (someone chime in here if you know), plus it gets very complicated to manage. You keep it simple and let CatDV do the heavy lifting. When you go to the workgroup version you’ll see how easy life becomes because you don’t have to deal with all these separate catalogs. All becomes as one (grasshopper). You just to a query and CatDV finds all the shots/clips for you. I do a LOT of drag and drop from CatDV to FCP directly. Works wonders and cuts down hugely on edit prep. Everything is already loaded and ready to go, as proxies.
Holding up? Each file name says what it is. The shooting location is already defined. After than I need the specifics of the shot itself and the other information is pretty much in the way. Again, CatDV does the heavy lifting.
And in triplicate: well your file name already has the date/country/city/broll and more, and yet you have directories that repeat the information. Actually if they’re all in the same directory a simple directory sort will give you all you need. CatDV isn’t going to die or blow up or something. Or at least it’s not likely. I do keep copies of catalogs locally in a DropBox folder and those are backed up six ways to Sunday. My date fields INSIDE of CatDV are what are important. If I tried to put the same degree of information in the directory structure and then duplicate it in the file name, well, talk about a plate of spaghetti. Start simple. You can always get more complicated later (inside of CatDV).
Look at what Bryson put up about how NASCAR uses the system. You’re not going to see that kind of directory structure.
Non-linear: all the time and nothing but.
-
Here’s what the directory structure looks like on one of my proxy drives.
I do like having the reel name as the prefix to every file name. Oh, and I window burn all the proxies and put the reel name on all of them. That works great when they’re being referenced by someone who isn’t used to dealing with footage.
Non-linear: all the time and nothing but.
-
Thanks for that. Yes I am certainly all for CatDV doing the heavy lifting. Yes the server solution is where I am heading in terms of multiple catalogues. This I am fully behind. Have you had any luck with getting CatDV to take certain elements from the file name and use that to populate fields or vice versa? Perhaps its only possible with the worker node.
Bryson referenced Nascar here:
https://www.youtube.com/watch?v=cpABlGRZ5EA
But did not go into folder or file name specifics, did you see something else from him? I searched but could not find it in the forum.
Some contents or functionalities here are not available due to your cookie preferences!This happens because the functionality/content marked as “Google Youtube” uses cookies that you choosed to keep disabled. In order to view this content or use this functionality, please enable cookies: click here to open your cookie preferences.
-
I see, so your ‘Archive’ folder relates to a drive?
-
Bryson didn’t go into that level of detail. But I believe that his description of their workflow details how they put your level (and mine too) of detail into CatDV rather than trying to create it in either directory structure or file names. I mean after all, that’s what CatDV is there for. If you’re going to do all the work, why have it.
I don’t have the Worker node. I create my first level of organization when I capture the footage using FCP 7. So far we rarely see file-based footage, though I’d like to get there. And I like long pieces of footage filled will shots and interview. The worst thing in the world for productivity is thousands of individual shots. I can’t imagine what it would be like if an interview came in a word at a time.
Non-linear: all the time and nothing but.
-
Yup, each “Archive XX” is a raw drive. Mostly they sit in a fireproof safe. I still have the tapes but once they’re on the raw drives I don’t actually have to capture them again. That saves a bunch of time downstream. It costs no extra time in prep or outside of traditional workflows.
Non-linear: all the time and nothing but.
Reply to this Discussion! Login or Sign Up