Creative Communities of the World Forums

The peer to peer support community for media production professionals.

Activity Forums Storage & Archiving Moving Files to Harddisk for Archival: File Format

  • Moving Files to Harddisk for Archival: File Format

    Posted by David Himmelberger on June 6, 2023 at 3:00 pm

    Hello there,

    Is there a general consensus on how exactly (File Format, GUID/MBR, Sector Size) to Format big Harddisks (12TB+) when moving old Project Folder off the NAS and backing them up mirrored? We’re doing this mostly on mac and I wonder what’d be best.

    Thank you!

    FYI the workflow so far hasn’t been very consistent, but for the future I envision:

    1. Scanning Server + Disks with DiskCatalogMaker
    2. Moving Files/Folders off Server with Rapidcopy + Verify (Twice)
    3. Deleting Files on server
    4. Labeling Disks and storing them away
    5. Spinning disks up every 6-12 months, moving old disks to new ones every 3-5 years
    Neil Sadwelkar replied 11 months, 1 week ago 4 Members · 5 Replies
  • 5 Replies
  • David Fox

    June 8, 2023 at 10:35 am

    Maybe also consider an LTO drive and a few 12TB LTO-8 tapes which are a better storage medium for long-term archive and won’t require spinning up periodically and replacing so quickly. Cost per TB will be lower (based on cost of tapes) so you’ll be able to have two copies at different locations.

  • Neil Sadwelkar

    June 9, 2023 at 3:45 am

    I wasn’t aware of RapidCopy. I use Hedge for all my copying tasks. Costs more than RapidCopy but lets you make two or more copies at the same speed as a single copy.

    I like the idea of backup to spinning disks. I use enterprise drives in USB 3 docks, so the drives become almost like ‘cassettes’. For spinning drives Mac HFS extended (GUID) is a preferred format. Spinning disks are more easy to access at a later date, than LTO tapes.

    For some tasks I have also used LTO plus drive and stored the contents of the LTO and the drive in a DiskCatalogMaker catalog. I prefer making plain catalogs (.dcmf) rather than the default catalog with thumbnails (.dcmd) because dcmf is a single file while dcmd is a package which can get corrupt across file systems.

    One issue you could come across as you add backups is, while DiskCatalogMaker lets you ‘browse’ backups without mounting the drive, you don’t get any idea of what the asset looks like. I’m now considering creating 1/100th size H.264 or H.265 proxies of everything I backup so that this can be used to do quick ‘browse and select’ shortlisting without touching the backup.

    Resolve lets you create proxies very fast while preserving the original folder structure. So, you could have 20 TB of video files in a complex folders structure, compressed to 200 GB with the exact same folder structure saved away in a local drive. If you have 50 backup drives of 20TB each, (1 Petabyte total) then proxies will take up just 10 TB.

    Neil

  • Doug Metz

    June 12, 2023 at 9:46 pm

    I’ve been cataloging everything with NeoFinder (was CDFinder) for a very long time (decades). Great product, great support, reasonable price. https://www.cdfinder.de/

    For archive, I’m using LTO for nightly backups of servers and RAIDs, also using it for bulk archiving alongside HDD copies of those archives as near-line storage which doubles as a safety copy. Faster restore is a bonus.

  • David Himmelberger

    June 13, 2023 at 8:26 am

    Thank you both, very insightful!
    I’ll look into adding LTO to the solution, have been pondering with it for a while but as hdd costs are coming down I’m not sure wether it’s not overkill.

    I’ve looked into Neofinder and DiskCatalogMaker, so far I can’t decide which is better, most important for me is that the catalogs are stable and stay readable and don’t get corrupted.

  • Neil Sadwelkar

    June 28, 2023 at 7:23 am

    I have both of them. I use Diskcatalogmaker for its ability to make self-contained catalogs, to have multiple drives in one catalog, to copy drive catalogs from one catalog to another, and the ability to export a catalog as a .csv so I can do stuff with that data in Numbers/Excel.

    I use NeoFinder to be able to import old Bru catalogs and create a browsable, searchable catalog of Bru tapes.

    For the past couple of years, for some projects, I’ve also maintained a database of assets by importing raw camera files into a Resolve project. Compared to DCM or NF, Resolve stores more detailed metadata like timecode, resolution, codec camera details. And a Resolve media pool can be exported as .csv and imported into Numbers/Excel for further analysis.

    Neil

We use anonymous cookies to give you the best experience we can.
Our Privacy policy | GDPR Policy