-
50 TB of Data Management
Hi all,
Not sure if this the place to put this, but its the only non-commercial storage forum I could find on here.
Anyway, I’ve recently been put in charge of managing a companies entire data pipeline. We produce Music Videos, Band Interviews and two shows that air in Canada. So we’re pumping out about 500 GB of content each month.
Up until the point when I came on board, they had someone who didn’t know what he was doing and who told them to purchase external hard drives, so they now have 50+ 1TB external hard drives filled with their old content.
The problem is (aside from none of it being redundant, which is scary) is that the guy who did this before me would put a project on one drive, and then transfer it to another for edit, which would have the edit on it, but then transfer it to another drive that has the export.. so I literally have a single project spread out over four drives, some with exports, some with projects, some with nothing but basic data.
The projects are organized by band name, so I had the idea of, as a temporary solution until they can afford a good server to host everything on. That each external drive would be a letter, all bands starting with A would go on the A Drive, and so forth.
Is this the best way of doing this, or would there be a better way to organize this data? Is there any best practice for doing something like this?
Also, as a temporary solution, whats the best back up plan without being able to buy more drives? Is there a best practice for backing up data that is already on a hard drive and/or redudant array?
-
2 Replies
-
today, it’s easy to buy two 32 terabyte drive arrays, and daisy chain them together. You can put them into a shared storage solution, so that everyone can share the same media. And you can use your fifty 1 TB single drives to back up individual projects, incase something happens to the main RAID arrays.
Companies like Maxx Digital and Sonnet and JMR make wonderful drive arrays that can be daisy chained together because of their SAS expander ports.How to manage all 50 terabytes to find your media quickly – CAT DV.
Bob Zelin
-
It seems to me that if not all the footage for all artists is required all the time, you could reduce the total amount of required shared (read: more expensive) storage by implementing some workflow strategies. For example, backing off projects as soon as they’re complete, and removing them from the SAN onto cheaper individual drives (which should each have its own clone for backup). We just buy bare drives and mount them using an eSATA dock.
I second Bob’s recommendation of CatDV (https://www.squarebox.co.uk) – amazing for the “10,000′ view” of all your projects. We use it as a “pre-editor” as well.
Gridiron Flow is also really handy for tracking what files go where – even if drives are offline. More useful for mixed-media projects, but great at tracking down that missing video clip across on- and offline storage.
– Allan White, Video Producer, Luis Palau Assoc.
Quad 3Ghz Mac Pro, 10GB RAM, X1900 GPU, XSAN, CatDV Server
Reply to this Discussion! Login or Sign Up