Project (and drive) management suggestions.

[Mark Suszko]

You have two issues: fast data access for day to day, reachable from anywhere, and insurance against catastrophic loss like drive failure, fire/theft.

What about a “belt-and-suspenders” approach? Such as: daily backups go to Cloud-based storage, and a weekly backup “ghosting” of the whole RAID goes onto hard drives, one of which can stay on-site, in rotation with three offsite back-ups? Once you have the computers set up to automate the backup process on a regular schedule, you can mostly forget about it.

It may help to imagine a couple of “worst case” scenarios, then design your methods and tech to answer that problem.

Worst case 1: in-house RAID failure, on the clock, deadline looms. What’s a faster/practical restore situation, I’d say cloud, *if* you have high bandwidth so it doesn’t take all day.

Worst case 2: can’t work from the shop, can you access remotely from home or another studio space?

If you’re doing incremental back-ups, what’s the maximum loss you’re comfortable with, in terms of how often the backups are made? Can you stand to lose the last hour of edits? The last three? A day’s worth?

Then there’s the “U-Boat” clients: they surface sporadically, without warning, asking for cr@p you did for them two years ago, pronto. How fast should you need to be able to restore/dub files for these guys? Is a day all right?

[Bob Zelin]

boy, I am going to regret answering this – no matter what, I will be chastised or reprimanded, or this post will be deleted quickly –
but here we go –

“we have our projects on two RAID enclosures (each with 4 drives configured as 4x 4TB Raid1). To back up the RAID enclosure I have 4x 4TB ext. harddrives (1 drive for each of the RAID1 setups). But I never know when is best to back these up”

So you are able to make a living with a single drive that is RAID 1 for “backup” , and you have two of these in a single array (4 drives in one enclosure). When you say “but I never know when is best to back these up”. Really – what if the power supply fails ? You lose it ALL. How about ANOTHER RAID ARRAY to clone this. Or how about all four drives at RAID 5, with another identical 4 bay RAID array, that you are running Carbon Copy Cloner on, or Chrono Sync –
oh – I forgot – that costs MONEY, and you can’t afford another 4 bay just for you (and another 4 bay for your other editor). Isn’t there a cheaper way to do this ? When do you back up ? Every day, every night ? Is this time consuming ?
How about hiring someone to do this, or running Chrono Sync or any scheduled backup program to do this for you, to do it when you leave for the day ?

.. It’s already a head-ache managing data, and I’m not even shooting 4K or RAW yet.

REPLY – you mean, that it is TIME CONSUMING and you are busy editing, and you and your other editor don’t have the time (or the money) to put all this time into managing data, because you EDIT for a living and all this backup stuff, and archiving, and media management is not billable. And God forbid that you have a drive failure, the meager amount you are charging for your editing will become ZERO (and possibly a law suit) if you LOSE this media.

How should I be organizing these? Should I have all Premiere Pro projects on one drive or folder that is backed up to the cloud (either with CrashPlan or Adobe Cloud)? It’s not a big deal to plug in an external drive once a week to back up (for off-site storage), but how do I manage the drives in a way to know which drives need backing up and which still have data that is already backed up?

REPLY – RAID5, RAID6, LTO Tape, redundant drive arrays. Crash Plan ? Really ? Buy ChronoSync and schedule the backup, and take it home with you (unless you are working from home). (But all this costs money, and requires my time, and I have no time !).

Unfortunately we area always working on multiple projects at a time- not just a single project, so footage (and especially project files) are constantly being updated/added.

REPLY – so you are BUSY with multiple projects – but are you BARELY getting by, because your budgets are SO LOW that you cannot afford someone else to assist you with this, and your budgets are SO LOW that you can’t afford redundant equipment or LTO to BACKUP, and you are exhausted from editing all day (perhaps you are shooting too) – and your budgets are SO LOW so that you can’t afford to do any of the things I am suggesting. Well, WELCOME to the real world. If you have LOW BUDGET CLIENTS that simply won’t pay you (we only have $800 for this project) – then you suffer, and you get screwed, and if you lose the data, then you don’t get paid at all. That kid who got his mom to pay for his Blackmagic camera, and shoot his music video that will become “viral” is not the client that you want.

I guess I’m wondering if there is a software solution, a user-disciplined/organized solution, or, more likely, a combination of the two. Any help would be greatly appreciated!

REPLY – well, you can WORK all night, run asset management software (who is paying for this ? ) like AXLE Video, or even $40 Neofinder, and you can BACKUP your media with Chrono sync ($40 – isn’t there a free version) –
and God forbid, you buy another IDENTICAL RAID ARRAY (your 4 bays) and backup to this – no – we can’t do that – your rent is due, and you have an infection on your finger that you got from banging your finger on the last production shoot that you got $800 for , for both shooting and editing – and you are a nice person – why can’t medical insurance be FREE ?

Aah – our lovely industry. Very different from the $800 dollars an hour (before ADO and Chyron) in the linear days of the 1980’s.

Believe me, I appreciate your struggle, but you cannot give your services away. And if that’s the only business you can find, which prevents you from purchasing the right equipment to keep your media safe, then perhaps it’s time to apply for a job somewhere. Because its possible that they will offer MEDICAL INSURANCE BENEFITS, which might be more money than you make all year.

I am an amateur musician, and I go see bar bands play all the time. I am going tonight. And I love them. And I say to my wife “how the F are these guys playing for 4 hours from 10 – 2am, setting up and sound check at 4:00, and at 2am, they have to pack up all the gear and go home – and for all of this, they get $75 – $100 each”. This is rapidly becoming the video business.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Todd Terry]

I guess Bob has some points, but upon revisiting the original post…

I didn’t notice one single time that Stephen complained about having to do the work to do proper backups, whined about the cost of it, moaned about not being able to hire someone to do it right, or claimed to be giving his work away for 800 bucks a pop.

He asked legitimate questions, and deserves a legitimate answer.

I too, frankly, doubt I do backups truly “properly.” And by the way I have no reticence about paying someone for good advice, spending money on hardware needed, or give away my work for nothing. I’m not a rich guy nor live in a McMansion, but when you get right down to it I’m remarkably well paid for doing the work that I love, and I’m not sure when people in our industry developed the reputation of being whining poor-as-churchmice complaining freeloaders.

T2

__________________________________
Todd Terry
Creative Director
Fantastic Plastic Entertainment, Inc.
fantasticplastic.com

[Stephen Pickering]

Hey Mark, thanks for your input!

I like the idea of the “belt-and-suspenders” approach. The project files themselves can sometimes be pretty hefty, but most often are fairly small in size, so cloud is a very easy solution on that front. So in this event, maybe it would be adventurous to have the actual project files in one location or one drive that is automatically synced to the cloud. My fear was that I would get too messy with too many different locations for projects and assets- but if I were in a better routine and more disciplined I imagine it would be easy (and consistent).

Setting up a semi-automated non-cloud backup would be really helpful. Bob Zelin mentioned a few software solutions that sound perfect, but it appears they are Mac only- I’m on a PC. But surely a google search will give me some ideas for this. I’ll look into this.

Thanks for the thoughts about the worst case scenarios. I actually chose RAID1 instead of 5 because I wanted to ability to pull a drive from the enclosure in the event the actual RAID box died (which has happened). It put me a few hours behind, but I was able to transfer the footage pretty quickly to another box and resume the project (then set the RAID drives aside) while awaiting a RAID box replacement. If there was a catastrophic hardware RAID crash (and both mirrored drives were lost) I would then have to grab the backup from off-site. This sounds like the reason you mention having more than one backup (and to keep one on site)?

Unfortunately cloud is not an option for any kind of footage, simply because of limited internet speed.

Remote access has never been a consideration, except to access files or assets when on the road- but never to edit. I wonder if I should consider this…

“If you’re doing incremental back-ups, what’s the maximum loss you’re comfortable with, in terms of how often the backups are made? Can you stand to lose the last hour of edits? The last three? A day’s worth?”

To be honest, a full day would be bad, but would not be detrimental. So even a nightly backup of editing projects would be acceptable, I think.

“Then there’s the “U-Boat” clients: they surface sporadically, without warning, asking for cr@p you did for them two years ago, pronto. How fast should you need to be able to restore/dub files for these guys? Is a day all right?”

Yes, these clients… I don’t mind at all telling these clients that it will be at least a day or two, which is why I think archiving to a few redundant hard drives would be fine. I could then plug the drives in to search for the client (or I could create an excel sheet that tells me what’s on each drive that is archived. But this is also where I would love some advice from you all who have been down this road already. I don’t mind buying some additional RAID boxes and a handful of harddrives that get plugged in once every month to archive. I used to use a bunch of external drives that I backed up to, but then if I ever needed to update one drive for some reason I would have to make sure to update the mirror as well.

So, I have a lot to think over. As I said, I know for sure there is discipline and routine that I need to just get the hang of- and I’m fine with that- I’m just hoping for some guidance so I don’t reinvent the wheel (when I know I’ll get it wrong anyways!). That’s why I’m wondering about how others organize their projects/assets. I also know that there must be a better software/hardware infrastructure that I can employ, and I understand that there will be extra costs. I don’t have money burning holes in my pockets, that’s for sure, but if there’s software that will make my life easier (for syncing or cataloging) I would love to know more. Or if additional RAID boxes are the way to go, I might just need to budget for it to make it happen. But since I don’t REALLY know the right way to do this, I hate spending money on boxes and enterprise drives, if there is a more cost-effective or future-proof/expandable solution I should be investing in.

Thank you again for your input- it’s given me some clues on what to start looking into!

[Stephen Pickering]

Hi Bob,

I apologize if my questions rubbed you the wrong way or implied that I was lazy or cheap. Your assumptions were not only wrong, but also pretty belittling. But I still appreciate your input since you did offer some valid ideas that I’ll look into.

Thanks Bob,
-Stephen

[Rich Rubasch]

One little nugget Bob did offer was Neofinder. Been using it lately to catalog a pile of archive drives and it is a pretty darn solid and flexible program. Very impressed. And the writer of the code is an email away even if he’s in Germany or the Netherlands, but he’ll answer your request!

Check it out if you want to catalog drives…..and we love Carbon Copy Cloner too!

Bob’s not too far off on the direction this industry is taking and did make some sweeping assumptions, but I don’t think he was making those assumptions about the poser, per se but rather on things he’s heard or read over time.

Not far off indeed.

Rich Rubasch
Tilt Media Inc.
Video Production, Post, Studio Sound Stage
Founder/President/Editor/Designer/Animator
https://www.tiltmedia.com