Archiving: Keep a great lossless copy or not?

[Tim Jones]

Ask yourself these questions –

• How much did it actually cost to create the original media?
• How much was my time worth to create the original media?
• How much would it cost me to recreate the project from scratch?

Based on your honest answers, consider these costs to archive the work:

• A 2TB consumer-grade disk costs around $99
• A 2TB enterprise-grade disk costs around $160
• If you have an LTO-5 or LTO-6 tape drive, a 1.5TB LTO-5 tape costs around $30, 2.5TB LTO-6 around $70

Do the numbers balance (i.e.: is the disk or tape more cost effective than recreating the work)?

I recommend that you save everything – from originals through to rendered finals – unless you KNOW that it’s a one shot job and you’ll never need to revisit the subject (not just the job). You never know when today’s cast off clips might become a money and time saving b-roll in tomorrow’s project.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.productionbackup.com
BRU … because it’s the RESTORE that matters!

[Jon Doughtie]

Tim – very well said.

[Eric Hansen]

Tim made a great point that storage is cheap compared to labor and cost of creation/acquisition.

But I want to address your other point about project files and re-exporting.

It’s a bad idea.

You alluded to it right at the beginning: “I am using Premiere CS4(I know old but that’s what I got)”

Imagine what you’ll be using 10 years from now. Will it be able to open your project without screwing it up? Will the project file corrupt? Will the files all reconnect (that’s the biggest problem I’ve run into even a few months later). If you later take up Creative Cloud and then decide to end your subscription, you will be locked out of your project files. You will want masters that don’t require an app to play back.

I’ve always operated under the idea of the ‘tape on a shelf’ or ‘negative in a vault’, which is why I eventually adopted LTO tape. No matter what system you used to edit, if you printed an HDCAM tape, it can be played back in any HDCAM deck. This is a godsend when you realize that your intern accidentally deleted all your files, but you still have a master tape.

The other question is codec, and that’s a complicated one. Never save a master as a web-quality codec. You want to save the highest quality that you can. But, I’ve had feature length ProResHQ files develop errors in playback (glitches or ‘hits’ in the video), and some even become completely corrupted and won’t open. I’m not sure exactly why this happens. I switched to DPX for masters because it is frame based – if one frame corrupts, it doesn’t screw up the entire movie. yes, DPX exports take up a LOT of space, and slower systems can’t play them back in realtime. But as Tim said, storage is cheap and we’re only talking about the final export, not all the source media. For sources, i would just keep the original formats they were shot in, along with any files that were used in the timeline (basically, a media manage).

since film is going the way of the dodo, many production houses and networks require source footage and masters to be archived on LTO and not hard drives.

i hope that helps

e

Eric Hansen
Production Workflow Designer / Consultant / Colorist / DIT
https://www.erichansen.tv

[Jeff Pulera]

Hi Travis

Many reasons to keep a high-quality master, versus just the “YouTube” quality. What if you need to re-edit later, fix a mistake, use parts of the clip in another project? Now all you have is the nasty, heavily-compressed YouTube clip to work with, then when you export AGAIN and compress AGAIN it looks that much worse.

As someone said, if it’s a one-shot deal, been delivered and the customer signed off on it, then don’t keep any copy, that is up to you. No one says you MUST keep a “lossless archival copy”, that is up to you. For most of the weddings and events I deliver, once I know the customer is happy with the edit, all I keep is a copy of the DVD they got. If I have some really good clips that I might want to use for a future demo, then I will keep the original cameras clips or a high-quality master of them. Just depends.

You didn’t say if you were on the PC or Mac. The huge files you saw were from choosing the wrong codec, basically “uncompressed” as a series of still frame images.

Rather, you will want to use something “visually lossless” for mastering. On Mac, that would be ProRes for instance. On the PC, something like the free Lagarith or UT codecs will provide a good balance between great quality and reasonable file sizes. Or even the “Matrox” codec which is MPEG-2 I-Frame with 4:2:2 color and is smaller yet.

Thanks

Jeff Pulera
Safe Harbor Computers

[Bob Cole]

[Eric Hansen] “I’ve had feature length ProResHQ files develop errors in playback (glitches or ‘hits’ in the video), and some even become completely corrupted and won’t open. I’m not sure exactly why this happens. I switched to DPX for masters because it is frame based”

I’m so glad I saw this thread tonight.

Today a client with ancient (3/4″) videotapes told me she had been advised to transfer them to
“DV-50” for archiving. My immediate reaction was, why archive to a videotape format which will be more and more difficult to play back? (btw, I’m fairly sure she meant DVCPro 50 when she referred to DV-50.)

I was going to advise transfer to ProRes 422, onto hard drives and LTO tape. But your comment above, about corruption in ProRes files has me concerned. What do you use to create DPX files, and what assurance is there that DPX will be around, 10, 20, 30 years from now?

What digital format is “The Best” for archiving? Is there any consensus on this?

Thanks.

Bob C

[Eric Hansen]

I don’t think my problem was with ProRes. In reality, I think it could happen with any codec. Since I wasn’t able to figure out the source of the error, just that I experienced it with multiple random files spread out over years. It reminded me of the glitches I would see on old BetaSP tapes.

DV50 is just another codec. It just so happens that it’s also a tape-based codec, but I don’t think that really matters. I think any playback device that can do ProRes will also be able to do DV/DVCPRO in the future. It was one of the top codecs.

With DPX, the only reason that I prefer it is because any possible data corruption (not tape or disc or drive), will be limited to a single frame. It’s pretty easy to rebuild a single missing frame in After Effects or similar.

There’s never a guarantee in digital. I wouldn’t have guessed 10 years ago that DV would still be playable today, and it still is. I have no reason to believe that it won’t be playable 10 years from now, but I wouldn’t be surprised if other codecs go away.

The easiest way to create DPX now is with Resolve Lite – import your file and export it as DPX. You can also export as TIFF, which I’ve seen occasionally for DCP delivery. You will need to keep separate audio files that are synced.

e

Eric Hansen
Production Workflow Designer / Consultant / Colorist / DIT
https://www.erichansen.tv

[Chris Murphy]

A 2TB consumer-grade disk costs around $99
A 2TB enterprise-grade disk costs around $160
If you have an LTO-5 or LTO-6 tape drive, a 1.5TB LTO-5 tape costs around $30, 2.5TB LTO-6 around $70

Although this is somewhat esoteric information, the rate of unrecoverable reads for the above options decreases as you descend the list. Consumer SATA shouls have less than 1 unrecoverable read error (bad sector) in 12TB read. Enterprise SATA should be less than 1 in 120TB read. Enterprise SAS/FC, should be less than 1 in 1.2PB read. And LTO tape should be less than 1 in 12PB read. So that’s in the realm of 1200x the likelihood of losing a sector of data on a consumer SATA hard drive, than on LTO tape, or three orders magnitude greater. And their shelf life is longer also.

[Chris Murphy]

This is what we get for tolerating proprietary formats.

Some project choose to record the final to film for archiving because they’re solidly convinced in 30 or whatever years there will be no way to deal with the original digital data at all, or it will be prohibitively expensive, and therefore why bother paying to store all of that digital originally captured material in the meantime? The original material is a lot bigger than the final digital deliverable. And film, well that’s comparatively cheap to archive with a freezer. And it’s somewhat self-describing in that its encoding isn’t totally obscuring the content.

Back in the old plate making days, quite a few artists would break the plates after all members of an edition were made. The only thing that mattered were the edition prints. So another valid tactic would be to render out the “best practical” deliverable file you’d ever need and obliterate the rest. And even that’s not as aggressive as destroying the plates, because the digital final such as it is can be identically duplicated, unlike edition prints.