Moving Files to Harddisk for Archival: File Format

[David Fox]

Maybe also consider an LTO drive and a few 12TB LTO-8 tapes which are a better storage medium for long-term archive and won’t require spinning up periodically and replacing so quickly. Cost per TB will be lower (based on cost of tapes) so you’ll be able to have two copies at different locations.

[Neil Sadwelkar]

I wasn’t aware of RapidCopy. I use Hedge for all my copying tasks. Costs more than RapidCopy but lets you make two or more copies at the same speed as a single copy.

I like the idea of backup to spinning disks. I use enterprise drives in USB 3 docks, so the drives become almost like ‘cassettes’. For spinning drives Mac HFS extended (GUID) is a preferred format. Spinning disks are more easy to access at a later date, than LTO tapes.

For some tasks I have also used LTO plus drive and stored the contents of the LTO and the drive in a DiskCatalogMaker catalog. I prefer making plain catalogs (.dcmf) rather than the default catalog with thumbnails (.dcmd) because dcmf is a single file while dcmd is a package which can get corrupt across file systems.

One issue you could come across as you add backups is, while DiskCatalogMaker lets you ‘browse’ backups without mounting the drive, you don’t get any idea of what the asset looks like. I’m now considering creating 1/100th size H.264 or H.265 proxies of everything I backup so that this can be used to do quick ‘browse and select’ shortlisting without touching the backup.

Resolve lets you create proxies very fast while preserving the original folder structure. So, you could have 20 TB of video files in a complex folders structure, compressed to 200 GB with the exact same folder structure saved away in a local drive. If you have 50 backup drives of 20TB each, (1 Petabyte total) then proxies will take up just 10 TB.

Neil

[Doug Metz]

I’ve been cataloging everything with NeoFinder (was CDFinder) for a very long time (decades). Great product, great support, reasonable price. https://www.cdfinder.de/

For archive, I’m using LTO for nightly backups of servers and RAIDs, also using it for bulk archiving alongside HDD copies of those archives as near-line storage which doubles as a safety copy. Faster restore is a bonus.

[David Himmelberger]

Thank you both, very insightful!
I’ll look into adding LTO to the solution, have been pondering with it for a while but as hdd costs are coming down I’m not sure wether it’s not overkill.

I’ve looked into Neofinder and DiskCatalogMaker, so far I can’t decide which is better, most important for me is that the catalogs are stable and stay readable and don’t get corrupted.

[Neil Sadwelkar]

I have both of them. I use Diskcatalogmaker for its ability to make self-contained catalogs, to have multiple drives in one catalog, to copy drive catalogs from one catalog to another, and the ability to export a catalog as a .csv so I can do stuff with that data in Numbers/Excel.

I use NeoFinder to be able to import old Bru catalogs and create a browsable, searchable catalog of Bru tapes.

For the past couple of years, for some projects, I’ve also maintained a database of assets by importing raw camera files into a Resolve project. Compared to DCM or NF, Resolve stores more detailed metadata like timecode, resolution, codec camera details. And a Resolve media pool can be exported as .csv and imported into Numbers/Excel for further analysis.

Neil