How best to backup our new 5Big 20TB?

[Chris Murphy]

Why not tape?

[Nick Tandavanitj]

Thanks for the reply. Good question.

Tape is still potentially an option for archiving for us but we often find ourselves dipping back in to review/reuse old footage which we found was pretty painful when we had DAT (nearly 10 years ago) so we’re skeptical that it’s quite right for us.

Would you recommend tape for day to day backups aswell?

[Chris Murphy]

I’m not recommending anything, just asking a question. Tape is not suitable for random access, the latency is huge (find the tape, load the tape, wait, search for what you want, select what you want, wait for the restore). However, once what you’re looking for is found, tape is as fast as a single drive – 120+MB/sec transfer rate. Even with an autoloader, which are expensive, it’s still going to involve quite some delay (minutes, not seconds). So don’t get tape as a random access device. Use it for backup and archive: stuff you know you need to keep, or you don’t know for sure you can purge.

LTO isn’t really comparable to DAT. DLT supplanted DAT. And LTO has supplanted DLT.

Time Machine is designed to backup/restore an OS, your applications, and some user data. It’s not really designed for your use case. The backupd consumes a lot of CPU, it does it every hour. If you’re willing to hack the preferences file to change the backup timing to something rational like noon and 6pm – or just 6pm – that’s better that this crazy daemon going off every hour and sucking the life out of system resources while you’re working. I’m not sure what use an hourly backup in this workflow would be, but if it’s working for you, fine.

However, don’t bet the bank that these Time Machine backups are going to last very long. I somewhat regularly have them totally implode on me – and that’s just backing up my laptop with an SSD in it. Pretty much once or twice a year I expect to reformat the Time Machine destination volume and start it from scratch. So it’s most definitely not a reliable archive. It’s a fairly reliable short term backup for an OS, apps, app prefs, and some local user files. For massive video files I’d use something else for your “nearline” (faster than tape) backup to drives: Carbon Copy Cloner or Super Duper, maybe someone here has advice on that. CCC is rsync based (it has its own current build of upstream rsync in the app, it doesn’t use the ancient version OS X ships with although you could use that too, or build a newer one from Macports), and Super Super is ditto based. CCC can leverage an rsync checksumming feature to confirm the source and destination are the same, although this is slow. The checksums aren’t stored anywhere as far as I know, it’s just used to independently confirm/deny the copy’s success.

[Nick Tandavanitj]

Thanks Chris. Understood re – Time Machine drives as bad for archiving. We’re just using them as short term backups in case of disk failure but – you’re right – we could probably schedule these out of office hours

Re backing up big video files, if we exclude big media files from Time Machine, would one solution be to RAID mirror a pair of drives for video on our 5big? Something like

4TB projects
4 + 4 mirrored for video/media files
4 + 4 JBOD for Time machine backups (excluding video files)

[Chris Murphy]

Backup for the system+apps+user preferences = Time Machine, or Apple Software Restore (asr is the CLI command, accessible via Disk Utility for GUI). ASR creates a disk image that you can store on any file system.

Nearline (faster than tape) backup for media files = Carbon Copy Cloner, or equivalent. If you search this forum, within the recent past others have made suggestions for other solutions that also manage organization, including telling you what drive a certain media file is located on. CCC won’t do that, it’s just a folder/file copy program.

I don’t know what you mean by 4 + 4 JBOD, or how you’re creating it in OS X. If you mean two 4TB drives in a linear/concat arrangement, so that they’re one big 8TB volume, I probably wouldn’t do that because it means that any single disk failure effectively means you’ve lost all of the data on that volume, so it has the same risk as raid0 but without the speed. But if you have 2nd backup or an archiving strategy, then you can of course get away with some additional risk with the 1st backup layout if the linear/concat arrangement makes the workflow easier for you. But if you only have one backup, then no I wouldn’t do either linear/concat (JBOD?) nor would I do raid0.

[Nick Tandavanitj]

Hi Chris,

Thanks again for your advice on this. We’ve now settled on a set-up of the 5big with…

4TB for project files
4+4 Raid0 for video
4+4 concat for short term Time Machine backups of the system and projects drive (with modified start interval)

We’re also using
– our old 4tb G-raid drives & CCC to keep short term clones of the Raid0 video drive
– a bunch of USB drives (2 copies of each) for long term archiving, with disk catalog maker for indexing

Hope this sounds reasonable as a setup?!

So far, the only issues we potentially have are…

1. Running our g-raid drives over usb2 – they have esata and fw800 but no thunderbolt so seem quite pretty slow

2. working out how to span CCC backups when content on the video drive goes over 4tb.

Cheers
Nick