[Dave Klee]

Hey Nick, normally I would try to talk you out of backing up your whole RAID unless it has nothing but archival material on it — for example, tapeless files, original images or other original content that cannot be replaced. I would argue that on any RAID that’s used in video production, there are pieces of it worth keeping and pieces not worth the time, effort and expense to backup. Is it worth backing up terabytes of render files that could be re-rendered if lost? Is it worth backing up capture scratch that could be re-captured off tape (if properly logged)? Unless you’re in a high-throughput environment where you need close to zero recovery time, probably not.

But, you know your life and your world a lot better than I do, so I’m not going to try and talk you out of anything.

As you and others have said, backup is both a software and hardware issue. Some vendors do both, but often it’s separate.

For Mac backup software, Retrospect is an industry standard. Personally, I don’t enjoy it. If you’re a Retrospect person and have been using older versions, I can’t think of a good reason to ditch it. But, if you’re looking for something new to do simple backups, I think it is way more in-depth than you need, and not as user-friendly as it should be. Plus it’s more than you really need to spend.

My current favorite backup program for Mac is ChronoSync: https://www.econtechnologies.com/pages/cs/chrono_overview.html I don’t work for them, don’t care if you use it, but it’s made my life easier and I think it’s worth the $40. There are a few other mid-tier backup programs out there in that price range that I would recommend over Retrospect for most people.

For hardware, I feel like you’re aiming low for a good 16TB RAID for $5-7K. Honestly, like you said, you just need the thing to be reliable — speed isn’t that big a deal as long as you can meet an overnight backup window for all the changes made that day, and maybe a weekend window for an occasional full backup (in most environments).

While nothing like the top-notch stuff advertised here, I think the DroboPro makes a great backup solution. If you’re really nervous, you can put it into RAID6 (or RAID5) with e-mail notifications. Speed is not fantastic, but so far for me it has proven extremely reliable for very minimal cost. You can get a 16TB kit for somewhere around $2,600. And, if that’s not enough space when RAIDed, you can buy two and still meet your budget.

Besides cost, there are two key objectives to keep in mind for backup solutions: Recovery Point and Recovery Time. Recovery Point is the point you want to get back to after a failure: i.e. what is acceptable to lose if something dies. Recovery Time is how long it takes you to get back up and running. Getting both of those close to zero (zero loss and zero downtime) can be expensive and complicated in a video environment where you can’t afford the cost or performance hit to mirror everything all the time in real time. But just know it’s a trade off — zero loss generally means more down time, and less down time generally means higher loss (assuming you are keeping costs low).

Making smart decisions about what is REALLY important to you now — about your Recovery Point Objective — will keep both your costs and recovery time lower in the long run.

Good luck with everything!

[Dave Klee]

Hey Eric, I haven’t given iPhoto sharing a try — I think I’ll go play around with it. Nice idea for something simple to get going with. But, you’re right — you’ll probably outgrow iPhoto’s tagging capabilities, and it has the same problem of not TRULY embedding all its metadata back into the original files through IPTC write-back, making long term upgrading a little scary.

As for Bridge CS3 to CS4, there’s a pretty decent overview of the updates here:

https://blogs.adobe.com/jnack/2008/09/whats_new_in_bridge_cs4.html

From a cataloging perspective, CS4 has a couple small tweaks that make handling hierarchies of keywords easier (parent & child relationships). And, I can verify CS4 is much faster and much better with searching than CS3.

But, is that worth upgrading your whole suite for that? I’m guessing not. We started our process with CS2, and the CS2 to CS3 upgrade was HUGE. CS2 Bridge was borderline unusable for large-scale cataloging. But, everything you need is in CS3, including the ability to do those bulk-applications (metadata templates) for files. I’d say you can always start with CS3, and if you find you’re doing a lot of work and want a speed bump, CS4 will be nice.

Let me know how things go and if you come up with anything else clever — always looking for new ideas!

[Dave Klee]

Hey Eric, sounds like you’ve put some good thought into this! Very cool.

Portfolio isn’t bad, but I’m not a huge fan of the interface (at least I wasn’t on Version 8). Canto Cumulus is another popular choice, and we have an implementation here. Both are in the “thousands to tens of thousands” range for a good client-server setup. We’ve done a lot of shopping for Digital Asset Management (DAM) / Enterprise Content Management (ECM) solutions, and here are the key “sales-pitchy” things that have annoyed me:

1. They show you demos of how easy it is to search for things — search is their key strong point. However, if you regularly need to put new things into the system, or regularly need to make updates to metadata, you’re in trouble. Ingest and initial tagging is often a challenge.

2. Implementation time is longer than they lead you to believe. The demo shows, “Look, you just drag your current library into the software and, BAM, you’re done.” Not true. Sorting out the wreckage and configuring the system to your unique needs will take months — if not years. Plus, don’t forget about teaching your video editors a new interface.

3. They’re heavily print (not video) oriented. Plugins from the big players are becoming fairly common to allow you to import your assets into InDesign and Quark. They’re targeted at graphic design shops that make regular catalogs or magazines — high volume work where big groups of people need regular access to a centralized image database. Once you find an asset in your centralized catalog and want to use it in a video, it’s not as easy as right-clicking to reveal it in the Finder and/or drag it into your FCP timeline. You’re usually looking at an “export” process to give you a copy of the picture that has to be stored somewhere outside your catalog. In general, once you load your assets into a program like this, you’re tied to the ways it allows you to work — which is rarely through the Finder.

4. Licensing can be a challenge. With Canto Cumulus, you can get unlimited “read-only” access through a web interface. I believe Portfolio makes you buy a “web-publish” add on or something like that to get similar functionality. Still, for people to update and have full access to the catalog, you’re usually looking at either per named user or per concurrent user access fees. They add up quickly.

5. Don’t forget about maintenance. Often, there are annual “maintenance fees” associated with these programs. Translation: “Pay us a percentage of your up-front cost every year or we don’t help you with anything and your catalog eventually dies.”

6. Metadata write-back is often claimed to be supported, but rarely adequate. What that means is that when you add keywords to a picture through the interface for these software programs, much like Aperture or Lightroom those keywords live in a propitiatory database that can only be accessed by that particular software (or however the software lets you access it). The keywords are rarely “written back” into the picture itself. So, what happens in five or ten years when you need to transfer your image library out of that propitiatory database to something better? None of your images actually have keywords directly attached to them — they all live inside this software that you want to get away from. Good luck.

7. Updates are frequent enough to be annoying. In general, these systems get updated on a fairly regular basis — small updates take time to install and implement, and big ones cost money. True for life in general, but particularly annoying for what will inevitably become one of your core software packages.

Final Cut Server is a huge exception to many of these problems — which is why I think it has a lot of potential. It’s video oriented, has drag-and-drop import into FCP, isn’t updated too often (both good and bad), and doesn’t charge you extra license or maintenance costs. However, its metadata write-back is fairly immature, meaning most additional information you write into FCS metadata needs to be exported through XML files with some clever coding. And implementation time, as I’m sure you know, isn’t short. Also, while it can certainly handle images, it’s much better at video (in my opinion).

If I sound a little bitter about this stuff, I am. We’ve been through a couple different major pushes to find a DAM or ECM solution that would fit our video production and photo needs, only to end up pretty disappointed at the current batch of offerings. The last major look was a year ago, so it’s possible something has come out since that is worth a look, or that we just totally missed a good option.

Not knowing you and your situation, it’s totally possible there’s something out there that fits your needs. Please let me know what you find — I like looking into this stuff, and would love to see what a fresh set of eyes digs up.

As for spotlight, it hasn’t proven very effective for me on video volumes, but it’s fairly effective for photos given a few pre-requisites:
1. You’re using an actual OS X Server, Leopard or better.
2. You’re primarily accessing your photos via AFP shares from Mac OS X clients, Leopard or better.
3. You’re aware of which IPTC and EXIF (metadata) fields are indexed by Spotlight — it’s not all of them. I’ll look around for a list.
4. A good (gigabit) network is always a plus.

Spotlight Server (built into Leopard+ server) is meant to work in a shared environment — previous versions are not. If you want to give it a try, I’d recommend setting up a separate AFP share on a separate volume dedicated to photos. Personally, I’ve found that systems for photo and video require different optimization techniques that are often incompatible. Photo volumes typically need safety and search-ability, and are willing to sacrifice some speed to get it. Video volumes need maximum throughput with minimum latency. Maybe it’s more of a problem or archive vs. active storage; we treat our photo volume as an archive where we store things permanently, and our video volume as an active scratch disk where things live only for the production cycle. (We’re using Xsan for our video volume.)

Anyway, looking forward to trading thoughts on this stuff and seeing how things go for you! Honestly, I’m surprised video-oriented DAM isn’t a better niche in the marketplace. Hopefully you find something out there that works for you and you can share with me!

Dave

[Dave Klee]

Hey Eric, you’ve found a great problem. There’s a huge disconnect out there between software made for a single photographer at a single computer vs. enterprise level digital asset management systems. This is something we’ve been struggling a lot with over the last few years at our corporate production shop.

The solution we’ve come to rely on uses some ingest scripts that have been custom coded to ensure unique file names and proper image placement on the server (which is a standard OS X server running Leopard with some RAIDs attached). After that, files on the server are annotated in place with Adobe Bridge — keywords, descriptions, headlines, etc. are added to files through AFP file shares as they live on the server. Searches are later performed with Spotlight Server (assuming you’re using Leopard or better everywhere). Permissions are heavily regulated to prevent accidental deletion or overwrites using an Open Directory schema with some reliance on ACLs.

We’re on the fourth or fifth major revision to this workflow, and it seems to be going pretty well for us. Happy to share more details if this sounds helpful — just let me know!

[Dave Klee]

Hey Eric, just wanted to chime in about trying an internal software RAID for Pro Tools. As others have said, it doesn’t work. I didn’t believe it last week, and actually tried to put two internal drives into a software RAID-1 (mirror) using Apple Disk Utility. I wanted the redundancy. But, it wasn’t recognized as a valid audio drive.

I’ve been told this is a hold over from the old days. Apparently, when Pro Tools began, standard file systems on both the Mac and PC were not capable of giving the reliable throughput that Pro Tools required for multi-track recording and playback. So, the people with Pro Tools essentially redesigned a file system that would meet their needs. Now that file systems have improved, there would be a lot of re-coding to get Pro Tools to support software RAIDs or file systems like Xsan. Sounded believable to me.

My half-ass solution is to put three drives inside the Mac Pro: one for the system (boot) drive, one for the “audio” recording drive, and one for a “backup” of the audio recording drive. All formatted as single, stand-alone drives. The audio drive clones on a schedule to the backup drive using Carbon Copy Cloner, a free backup utility for Mac that allows for scheduled drive backups regardless of whether a user is logged in (https://www.bombich.com/). Then, whenever a user does log in, the backup hard drive unmounts using a script — so people don’t accidentally record to the backup drive. This gives us some piece of mind with fairly minimal cost, and so far Carbon Copy Cloner has proven very reliable.

Good luck!

[Dave Klee]

Hey Bob (or anyone), have you ever deployed one of your systems where the server and the primary editor were the same machine? As in, take a good Fibre Channel array and plug it directly into a good Mac Pro, then share that Mac Pro Fibre Channel array over the network (using good network gear). Then do the majority of your editing directly on the Mac Pro with the occasional laptop or other machine doing simple tape logging and/or basic editing?

[Dave Klee]

Okay, two basic options for you that, like Bob said, won’t give you any problem on the system you formatted it on.

If you format the drive NTFS using a Windows computer, it will work great on a Windows machine. Nearly unlimited file size. However, if you plug that drive into a Mac, the Mac will see it as a read-only volume. Your Mac can view most files, but cannot modify anything or write to the drive.

That problem can be fixed by purchasing this software for a Mac that Mikkell referenced (less than $40): https://www.tuxera.com/products/tuxera-ntfs-for-mac/
That software allows you to read and write to an NTFS drive attached to a Mac. Perhaps there’s a free version somewhere that I don’t know about.

The other option is to format the drive HFS+ using a Macintosh computer. It will work great on your Macintosh machine with almost no file size limitation (8 Exabytes). However, Windows will not understand the Mac-formatted drive at all if you plug it into a Windows computer.

Here again, you can install software on a Windows computer ($50) that will allow you to read and write to a Mac formatted drive called MacDrive: https://www.mediafour.com/products/macdrive/

Now, there are some more creative options for you if you’re interested in sharing files between systems — like using a NAS box as a file server to transfer files over ethernet. Or share files between computers on a local network using built-in file sharing on either OS. Neither may fit your needs, but both work well on local networks and avoid that 4GB file size limitation.

Really, though, if you’re primarily using a Mac, you generally want to format your hard drive with a Mac (HFS+, or as Apple Disk Utility calls it, “Mac OS Extended”), and if you’re primarily using a Windows computer, format it with Windows (NTFS). That’s my rule of thumb unless I know I don’t care about big files and I need maximum portability between Mac and PC (usually smaller USB thumb drives), then I’ll go FAT32, which Apple Disk Utility calls “MS-DOS (FAT).”

Hope that helps, and good luck!

[Dave Klee]

You probably want RAID-5. That would allow a single disk to fail in your RAID without losing data, with good speed. Other popular options are RAID-0, which is the fastest but has no protection (any disk fails and you’re data is lost), and RAID-1 (mirrored) which is the safest but slowest.

Not all RAID-5 is created equal, though. Different drives and manufactures give you vastly different speeds. And, you probably want a HARDWARE (rather than software) based RAID. Lots of good options listed here from Cow advertisers!

[Dave Klee]

Hey Doug, I just wanted to weigh in late and agree with Mark and Jordan on this stuff. The devil is in the details, and tuning an Xsan (like any complicated system) can be a challenge. No need to throw the baby out with the bath water before having an expert take a look.

As for the bigger question about working with your corporate IT folks to build or share a unified SAN system, I’d shy away from it.

I’m working in a video production department embedded in a larger corporate environment. Our core business is NOT video production — we’re a utility that happens to have a video production department. Our corporate IT folks, rightfully so, prioritize the core needs of the business over the needs of our creative department. We manage our own Xsan and only rely on our corporate IT folks for infrastructure things (like public network support, DNS and Static IP reservations).

We have some really talented IT people here, but (with a couple exceptions) they don’t know a lot about a few key topics:
-Macintosh Computers
-HD video
-Creative workflow

Those would all be really key things to understand in order to build and support a SAN used for video editing — especially using Final Cut Pro. Typical corporate environments need to support things like ERP, SQL, e-mail, Word documents and spreadsheets. The management methodology to build and support systems tuned for those needs are often incompatible with the needs of a fast-paced, HD, creative video production shop.

I’ve heard about two places in town that have tried (after much insistence from their centralized IT body) to integrate creative and core business systems — neither with Compellent, but both involving some level of SAN. Neither have gone well. In one case, after years of head-butting, the centralized IT body recently said, “We don’t get you guys. Go buy what you need on your own.” They’re the lucky ones.

There really is something to be said for experts who specialize in the management of SAN systems in creative environments. Find one you trust, and your life will be better.

Good luck with everything!

Dave

[Dave Klee]

Hey Marc, smart thinking on connecting to the network and moving data over. I was going to suggest that or Target Disk mode. Honestly, I’m stumped as to why they wouldn’t work. We’ve thrown some pretty random SATA disks in our new machines (never that exact model), and so far so good. I wonder if they’d work in an external enclosure over firewire or USB?

Anyway, sorry I’m not any help, but glad you got the data moved over. And, yeah, the fact that they changed sled lengths is pretty annoying. Meh. Not looking forward to migrating 35 machines next year… it could have been so easy…

Take care, and let me know if you figure out what’s up with those drives!

Dave