-
Promise Pegasus R4 8tb RAID config?
-
David highlights the distinct difference between backup and archive. It’s a big difference.
I use LTO as well, but it’s “lowly” lto4 to a Cache-A. It’s actually been a pretty wonderful solution.
I will highlight my workflow do you can pick and choose what’s relevant.
We shoot, we make backups of the cards in the field to at least two drives if not a third.
That media then comes to the edit suite and gets copied to
local storage and starts the project and stays in the media folder that I create for that project.At that point, we can plug a hard drive right on to the ache system and archive straight to tape. I tend to “stage” media in tape size pieces. Once I have around 750 GBs of data, I make a tape, then I immediately make another one which is taken home (offsite). Depending on the number of files, a full tape takes about 3-4 hours and runs at or usually over 80MB/sec.
The footage is now backed up and archived.
I edit the project. Once that’s done, I network transfer that to the cache-a drive and backup all media save render files. This includes the original camera media, any transcodes, and all other ancillary media. This creates yet another copy of the original camera media, but it is now stored with all the project media, so when it comes time to restore, I have everything in one convenient package.
It works well for us.
I can’t recommend cache-a systems enough. They are flexible and pretty easy to use. They keep an internal catalog of all media which serves as a very rudimentary catalog. There’s no software, you plug it in to the wall, plug it in to your network, and power it up. You can even restore back to an attached hard drive if you want. They aren’t cheap, though, so it’s a big consideration. For us, the reliable archive system is worth it.
Ltfs represents a step forward, but since we are lto4, we don’t have that capability so I can’t speak to that.
-
[Jeremy Garchow] “Ltfs represents a step forward, but since we are lto4, we don’t have that capability so I can’t speak to that.”
LTFS is still fairly young and there are a few things one should be aware of but it opens up new backup possibilities and an interesting roadmap with specs for LTO6 and 7 already in the can plus cartridges are quite cheap.
While it is of course not recomended nor something that should be done in production, it is possible to play HD Prores HQ streams in QT player from an LTO5 tape in realtime without dropped frames. So in terms of speed that’s pretty amazing compared to tape based backups a few years ago. -
Several questions:
1. There are heavy costs to archiving: a heavy initial investment, time, attention, and even the tapes themselves. If you archive a client’s work, do you bill the client for that service, or is this part of your overhead?
2. It is often stressed, when transferring XDCAM footage, for example, that verification of the copied media is critical. If you are just using drag-and-drop to send data to an LTO-5 tape, is there any verification?
3. You write:
[Jeremy Garchow] “At that point, we can plug a hard drive right on to the ache system and archive straight to tape. I tend to “stage” media in tape size pieces. Once I have around 750 GBs of data, I make a tape, then I immediately make another one which is taken home (offsite).”Are you saying that you make an LTO tape immediately, or that you don’t make any LTO backups until you have accumulated 750GB of data? I guess that would be okay, if you had a very steady and predictable flow of data. But it would mean being exposed to the vagaries of hard drive failure for some period of time, at a stage when the project was most important.
(The sad truth is that for the vast majority of commercial projects, the finished product is the only thing the client really has much interest in, once the project is done. I would think that the most critical period for LTO backups would be during the editing phase, not after.)
Thanks for sharing the information about this important subject.
Bob C
-
[Bob Cole] “1. There are heavy costs to archiving: a heavy initial investment, time, attention, and even the tapes themselves. If you archive a client’s work, do you bill the client for that service, or is this part of your overhead?
If a client wants a back-up to LTO5 I will bill him. Most of my clients are regulars and I know I need to repurpose and revisit stuff all the time so this is part of my overall agreement with them.
2. It is often stressed, when transferring XDCAM footage, for example, that verification of the copied media is critical. If you are just using drag-and-drop to send data to an LTO-5 tape, is there any verification?
If you use tools such as BRU then verification is an option. If you just use drag-and-drop LTFS then there isn’t. I never had an issue restoring a backup from tape.
3. You write:
[Jeremy Garchow] “At that point, we can plug a hard drive right on to the ache system and archive straight to tape. I tend to “stage” media in tape size pieces. Once I have around 750 GBs of data, I make a tape, then I immediately make another one which is taken home (offsite).”Are you saying that you make an LTO tape immediately, or that you don’t make any LTO backups until you have accumulated 750GB of data? I guess that would be okay, if you had a very steady and predictable flow of data. But it would mean being exposed to the vagaries of hard drive failure for some period of time, at a stage when the project was most important.
In my case, I wait till I can fill-up a tape. I am working off RAID6 so I’d have to experience three drive failures for things to go up in smoke.
(The sad truth is that for the vast majority of commercial projects, the finished product is the only thing the client really has much interest in, once the project is done. I would think that the most critical period for LTO backups would be during the editing phase, not after.)”
Depends on the nature of your projects and what you do. My stuff ends up on HDCAM-SR for the client plus Blu-ray for replication/stores. I know I will need to revisit the projects all the time, often just the final product for slight modifications. So I could either put another 250 dollar SR Tape for 120 minutes on the shelve and recapture when necessary, pile up external hard drives or use an 80 dollar LTO5 to store 10 hours without the need to recapture. Obviously the latter makes more sense and is more economical both for me and the client. They save money, I save time.
-
[Bob Cole] “If you archive a client’s work, do you bill the client for that service, or is this part of your overhead?”
Mostly overhead at first, but you can certainly add an “archiving” fee to help defray the cost. It all comes down to how much your archive is worth to you, and perhaps to your clients. You don’t necessarily have to ask your clients how much it’s worth to them as they probably couldn’t put a value on it, nor should they. When we get repeat business, our clients expect us to have the footage, and we also charge for the time that this archive takes to restore. So, you can make it a line item, or simply wrap it in to the cost of the overall project. Either way, you should certainly be able to add this to your invoices in some way as it is part of the service you are providing.
[Bob Cole] “If you are just using drag-and-drop to send data to an LTO-5 tape, is there any verification?”
We use LTO4. The way I do it is that I setup the “tape sized” bits on the SAN (or it could be local storage as well). I then use Shotput Pro to run a verified copy to the Cache-A hard drive (known as a “Share” in Cache-A terms). This way I know the media is good from the SAN to the Cache-A. The Cache-A has it’s own verification system. You have the option to turn it off to gain a little bit of speed, but I leave it on as this is an archive after all! This way I have a verified copy to the share, then the Cache-A runs the verification to the tape.
[Bob Cole] “Are you saying that you make an LTO tape immediately, or that you don’t make any LTO backups until you have accumulated 750GB of data? I guess that would be okay, if you had a very steady and predictable flow of data. But it would mean being exposed to the vagaries of hard drive failure for some period of time, at a stage when the project was most important. “
Well, we have the original shoot drives that are in essence a backup, and any and all project media is backed up. Once it comes time to archive, I then archive to LTO. I don’t have to have 750GB sized bits. I archive to tape when the project is ready to archive. Sometimes, I have to split a project up as all of the media is more than 750GBs so it will go across two tapes (four with the offsite backup copies).
[Bob Cole] “(The sad truth is that for the vast majority of commercial projects, the finished product is the only thing the client really has much interest in, once the project is done. I would think that the most critical period for LTO backups would be during the editing phase, not after.)”
Then it sounds like for you, you need a solid backup system rather than an archive system. There’s a big difference that DRW touched on.
We need an archive system and clients do return to modify projects, or use existing footage in new projects, as well as backup. It has worked out great for us, but perhaps it doesn’t make sense for everyone.
AN LTO system could sort of be used as a backup system, but it’s more of an archive system. Tape backups would be fairly inefficient due to their linear nature.
Jeremy
-
WOW! Such great information on LTO archiving.
I’m in the process of going to LTO for archiving tapeless footage and projects/elements. I know it is going to be a huge investment. But I’m sitting on a lot of footage and projects that need backing up. Thank you to everyone on this thread for such great info. I’m looking at getting at the Tolis Group’s The Edit Bay Cube.
https://ww2.productionbackup.com/hardware-bundles/desktop-bundle-packages
Due to that lack of PCIe slots I’m also going to need to get a Atto R680 for both my SAS connected RAID and LTO drive. But it will be worth it in the end and I can breathe easier once all is archived on mirrored tapes.
Todd Gill
Editor.Colorist
Digital Post Ink
http://www.digitalpostink.com -
Backups and archives have become pretty crucial with the tapeless workflow. I still don’t feel safe the way I used to be when I had all the dv or hdv tapes named and lined up and two or three copied of the project I could batch capture and recover from. The LTO looks like a great choice, but the price is still very high. You save about 20,00/30,00$ on a 1,5 Tb tape compared to a HD drive of the same capacity, so it’s about 60 1,5 Tb drives before you pay the LTO driver. Years of backups in my line of work, even if I double security. And archived HD drives let sitting on a shelf should be less likely to go off than the actually used ones. I shoot outdoors documentaries so I first have to store the footage where I find myself working. I build a .dmg of each card daily on a RAID1 Western digital portable array from my MBP, and also copy them on one more drive. I keep the logged and transferred clips (I check and line up the footage daily) on the same RAID1, without copying them on the second drive (I still have the .dmg’s in case it fails). When I’m back home, I copy the RAID1 drive content onto a raid 5 “cold” NAS storage array, then the transferred clips for actual editing to the MP internal RAID, that gets synch’d onto another external backup system, so that if the RAID0 fails I do not have to find, copy and reconnect all the clips, and moreover, the graphics, voiceovers and so on. Just to feel more at ease, I keep a drive loaded with OS and FCP only, so that if I have a problem with the current system disk I can just connect that one and go on with the projects on the fly without having to wait for Time Machine restores or reinstalls (it saved my ### yesterday when FCP7 went out and wouldn’t launch or reinstall with a delivery to be completed tomorrow). When I’m done with a project I consolidate it using media manager into a drive I keep away. One drive per three or four projects. And time Machine takes care of the OS and projects on one more external drive. So far, it worked fine and I could survive to more than one drive or RAID failure. My next choice would be keeping the active projects media on an external RAID5 array like the Pegasus instead than on the RAID0 4 TB internal one.
Fabrizio D’Agnano
Rome, Italy -
[Fabrizio D'Agnano] “You save about 20,00/30,00$ on a 1,5 Tb tape compared to a HD drive of the same capacity,”
Your getting great prices on your hard drives. LTO tapes go for about $45.
[Fabrizio D'Agnano] “And archived HD drives let sitting on a shelf should be less likely to go off than the actually used ones”
Hard drives that aren’t spun up at least once a year are much more likely to fail then drives that see regular use.
The deal with LTO5 is that it gives you the safety of shooting on tape. Instead of consolidating on an external Raid5, which can and does fail – the raid can get corrupted even if the drives don’t fail, it has happened to me – archiving to LTO is like having it stored on DV tapes – it can sit on your shelf for 5 years and still work perfectly. The tapes are small and easily stored or shipped.
I will say that dealing with the LTO drives and the software that supports them is not as quick and easy as copying files to drives. The A-Cache system is probably the easiest way to go, although it costs almost twice as much.
Herb Sevush
Zebra Productions
—————————
nothin’ attached to nothin’
“Deciding the spine is the process of editing” F. Bieberkopf -
I think it’s important not to underestimate the power of long-term cloud storage as a viable tier 3 option going forward. The fact of the matter is, the cost benefit towards moving completely digital is fairly substantial on both a small and a large scale. On top of that you’re looking at a virtual fail safe. I think Amazon S3 storage is something like 1 file lost in over 10,000, and correct me if I’m wrong but I believe they claim to have never lost a file in Glacier Storage. And we’re talking pennies on the dollar for storage space as compared to LTO.
Reply to this Discussion! Login or Sign Up