Creative Communities of the World Forums

The peer to peer support community for media production professionals.

Forums Storage & Archiving Need some wisdom from the wise on storage solution

  • Need some wisdom from the wise on storage solution

  • Santanu Bhattacharjee

    January 23, 2021 at 6:16 pm

    During the pandemic, I shifted my editing workstation from studio to home. About 1 km apart. During the shifting, I lost a couple of hard drives too. So I also carried my backup server home. Some 40 TB of data. That’s another story.

    Here in India, the pandemic is almost in the decline. Time to work back from the studio. However I may shuttle between my studio and home. Home being more productive.

    I had originally thought of BackBlaze for unlimited backup of all my 40 TB at $60/y and their B2 cloud storage at $100/y for 1TB for my working data. I heard some negative reviews of indefinite time for restoring data during a crash etc etc…

    Now I am thinking of buying a NAS 40 TB+ that will work as backup as well as cloud storage, so I can work anywhere. I could sync the home PC. Grant access to other editors to work. Share my stock footage with others.

    If I keep the new NAS in the studio unattended, I worry about any fire hazards, cooling at nights. keeping the air conditioning or even the ceiling fan all night may get expensive. India gets hot up to 40 deg Celsius during the day. Power outages are common too. However, I have UPS and inverter backups to take of that.

    Any insights from the experienced? Many thanks in advance.

  • Tim Jones

    January 25, 2021 at 12:34 am

    Of course, my initial recommendation would be to look into an LTO tape solution. The initial cost of entry may be a bit frightening, but the long term costs and peace of mind will pay off quickly.

    For 40TB of data, you could use 7, 6TB LTO-7 tapes that are –

    • Easily transportable
    • Robust even in India’s summers
    • Require no power to maintain
    • Can be stored safely offsite in a vault or even duplicated to provide a geographically distant deep archive.

    And with appropriate connectivity (TB 3 or SAS) at home and your studio, you can carry the tape device between the systems so you only need a single tape device.

    Aside from that, you don’t mention the network capabilities between your home and studio. If you have a high speed service, you could take the disk route and place a low-cost disk array in both locations and then use something like rsync to keep both side sync’d so that you always have the same data available in both locations.

    In either case, you’ve provide at least a minimum of geographical (and hopefully power grid) separation between your data while making it easily available at both ends.

    Good luck!
    Tim J

  • Tim Gerhard

    January 25, 2021 at 5:10 pm

    For 40TB a NAS with a cloud backup would be perfectly fine. QNAP is a great choice.

    With BackBlaze be warned that it is not for backups only and not archive. You must keep the original data on the source NAS or they delete it from the cloud backup eventually.

    I’m an LTO tape guy, but wouldn’t recommend it in your situation unless it’s what you really want. LTO doesn’t really shine for cost until you hit around 60TB+ for archiving. You mentioned your environment is typically 40 degrees Celsius typically. LTO tapes should be stored up to 25 degrees Celsius long-term and when in use 32 degrees Celsius, so you’d have to manage the temperatures wherever you’re operating and storing tapes if you used them.

  • Paul Carlin

    January 27, 2021 at 8:12 pm

    A hard drive docking station and some “archive” rated SMR hard drives are an economical solution to backing up your data. Make two copies if you’re paranoid. Instant access to the data when you need to restore it. I take a snapshot of the directory structure and save them as text files so I can easily search for what I’m looking for and know which drive to plug in. I also recommend a hard drive storage case like the Orico 20-bay suitcase. Grab and go!

  • Bob Zelin

    January 29, 2021 at 3:30 pm

    I strongly suggest contacting Neil Sadwelkar in Mumbai. Neil is a great engineer, and can assist you with all of this. I work with him all the time in India.

    http://www.sadwelkar.com/neil.htm
    Bob Zelin

  • Neil Sadwelkar

    January 30, 2021 at 4:51 am

    Thanks Bob.

    Santanu, I’m in India too. And I too had to move my setup home in March as well. And am still operating from home. From your profile and web site it looks like you’re in Mumbai (Thane). I’m in Mumbai too (Andheri W), so we have similar climates. I’ve had similar challenges in temp and dust.

    About your server, you haven’t mentioned what kind of a server and how you connect to it. Or how many users currently connect to your present server.

    For 40TB of data, I don’t think Backblaze is a viable option. or practically any other cloud service. In fact, for these very use cases, in the last NAB I went to (2019, which was the last NAB anyone went to) I had meetings with Backblaze as well as Wasabi to figure out if we could work out a cloud backup structure based in India for Indian users. With the backup ‘seeded’ with shuttle drives, and then only updated online. That didn’t work out as neither was interested.

    Anyway, for 40TB of data which is shared between more than 3 users, who have to access the same media at the same time, a NAS is needed. If they only need to be able to occasionally share files, and never work on the same files, then its more cost effective to provide each user with sufficient local storage, and have them all connected over a fast network (not WiFi).

    Another interesting new cloud shared storage service is Postlab. For under Rs 10,000 per month, 3 users can share proxies up to 3 TB, and keep their FCP X libraries in sync over the cloud. Basically edit from anywhere with up to 28-50 hours of proxy footage in the cloud.

    That being said, the case for LTO drives is that they extend the capacity of your local storage – direct storage or NAS. Projects that have been delivered and are not likely to return immediately, but might one day, can be backed up to LTO tapes and erased off your primary storage. To be restored when the project returns for versions/changes.

    Cost-wise, LTO tape costs Re. 1 per GB now. Of course, you need the drive and software which will run about Rs 3.5 lakhs. Assuming a life of 5 years for the LTO drive (with moderate use of about 20-25 tapes per annum) and assuming you ‘sweep out’ data at the rate of about 10 TB per month, your effective tape ownership/backup cost goes to Rs 1.50 per GB. With heavy use, it will be about Rs 2.5 per GB. This is way cheaper than any kind of drive storage. Which is about Rs 2 per GB for the cheapest consumer drives up to about Rs 7-10 per GB for good RAID drives or NAS.

    There are also LTO service houses who do LTO Backups only. That might be more cost effective if your backup requirements are under 5 TB/month.

    So, drives+LTO effectively is sort of like unlimited storage.

    Hope this helps. Contact me offline and we can talk.

    My email is my name surname (one word all small) at me dot com

    Neil

  • Neil Sadwelkar

    January 30, 2021 at 4:56 am

    That’s a great bit of advice Paul. And more convenient than LTOs.

    What I use to track my ‘archive’ drives is Diskcatalogmaker. It saves’catalogs’ of your drives down to the last file, and you can ‘browse’ the drive without it being connected.

    For an entire feature film original camera files, we often have them backed up to, say, 8 drives of 12 TB. I ‘scan’ all 8 into one Diskcatalogmaker catalog, so I can search for a particular file across 8 drives and 96 TB without connecting a single drive.

    Diskcatalogmaker also exports all or selected files/folders as csv so you can make a Excel database of file names, sizes and paths and use an AppleScript or Excel formulae/macros to parse EDLs/XMLs and create a ‘copy script’ to extract selected files.

    Neil

  • Santanu Bhattacharjee

    May 17, 2021 at 3:50 pm

    Sorry guys for not being around for long. Thanks for the overwhelming response from all. There are some great suggestions while I guess I may not have explained my scenario in the best way.

    LTO is out, because although 95% of my data is not re-used often, I have old clients who come back to me after very long. So I some times need to revisit / re-edit their data. So I cannot catalog or update a serial device as often. Too much time consuming.

    Backblaze offers UNLIMITED Personal backup just for $60 / year / per PC. Not B2. I need just archival for my workstation. At 128MBPs broadband the estimate says ~40 days to backup the entire 40 TB. I can wait…Not sure of restore though. Need to ensure periodically that backups are in good health.

    NAS – Initial cost is higher with 10 x 4 TB HDD. May need 2 boxes for that many HDD. Will need additional UPS. May need to keep at home during lock downs to ensure all is working fine. Data recovery may be costlier for LINUX file system as against a PC NAS. Works as a cloud server too for teams to collaborate.

    PC as NAS – Has all benefits of a boxed NAS, in fact much cheaper. I have the required expertise to set it all up. Just that the foot print is bigger with mice, keyboard, monitor and an UPS for keeping at home. With 24 x 7 powered on will bother others at home.

  • Neil Sadwelkar

    May 23, 2021 at 6:32 am

    Santanu,

    Have you checked with Backblaze whether unlimited is really unlimited, as in, 40 TB is fine? Or maybe there’s some limit on a per day or per week/month basis. Or maybe they permit only ‘local’ files on your PC/Mac to be backed up, and do not NAS or external drives.

    Apart from this, you may run into one or both of the following.

    Backblaze deletes anything that’s not on your system for 30 days. And, they delete anything backed up off an external drive if that drive is not present when backing up.

    Two, there is probably a throttle on upload speeds further down the line. Or maybe for <font face=”inherit”>download speeds. So, getting back your data may be very slow. I </font>was<font face=”inherit”> reading the experience of a poster who wanted to </font>backup<font face=”inherit”> 64 TB to BB. After a year, he managed 29 TB. So he gave up. And he claims to have a 100 Mbps up and down line.</font>

    But if neither of these are true, then $60 per year is a good deal.

    Or, as Paul suggested, getting bare drives, and a dock, and treating drives like ‘cassettes’ that you pop into the dock and copy data to and from the drives is a viable option too. Its way faster than LTO, and will cost less than LTO. Consider the data safe for about 3-5 years, and be prepared to migrate some or all of it to new drives after that time passes. In any case, by 2025-26 or so, a 50 TB drive will cost as much as a 16 TB drive costs these days. So the number of drives you need will also shrink.

    Take a look at Seagate Backup Expansion drives of 16TB available on Amazon India for about Rs 25,000. Inside this there’s a Seagate Exos enterprise drive which you can consider safe for at least 5 years. With these 16TB drives, even over USB3, you’ll able to backup 40TB in 60-90 hours. If you get Hedge, you could also make two backups at the same time at the same speed if your source is fast enough.

  • Santanu Bhattacharjee

    July 27, 2021 at 4:44 am

    Thanks for your suggestions. I recently interacted with a Backblaze personnel to realize that backblaze is essentially designed for home users and not for video professional who have unusually large files to back up. The process of uploading large files with too many checksum in Backblaze is unfit for video professionals.

    I am going the bare drives way. Also tried creating a PC NAS with those, so that the data is always available. (2 birds with 1 arrow – Backup+Real-time editing). Discovered that they are good as a backup solution only. Not so good for real-time editing across network. Though a 1Gb Ethernet suffices, even a 7200 rpm struggles to transfer data at barely 50MB/s. For networked real-time access only an SSD works fine with ~100MB/s

    <font face=”inherit”>Therefore my current solution is all live project data / footage are stored on SSDs on a 1Gb networked server. On completion of a project they move from SSDs to the archive drives manually. The backup app, slowly makes a copy on old drives at its own pace. A cloud app on the same server also helps make the data available to team members for remote.

    No RAID. I found many people get in trouble when they do not find identical drives after 5-7 years. Using unmatched drives affects performance. So separate backup copy is better. More manageable.

Viewing 1 - 10 of 10 posts

Log in to reply.

We use anonymous cookies to give you the best experience we can.
Our Privacy policy | GDPR Policy