Creative Communities of the World Forums

The peer to peer support community for media production professionals.

Forums Storage & Archiving Need some wisdom from the wise on storage solution

  • Need some wisdom from the wise on storage solution

  • Santanu Bhattacharjee

    January 23, 2021 at 6:16 pm

    During the pandemic, I shifted my editing workstation from studio to home. About 1 km apart. During the shifting, I lost a couple of hard drives too. So I also carried my backup server home. Some 40 TB of data. That’s another story.

    Here in India, the pandemic is almost in the decline. Time to work back from the studio. However I may shuttle between my studio and home. Home being more productive.

    I had originally thought of BackBlaze for unlimited backup of all my 40 TB at $60/y and their B2 cloud storage at $100/y for 1TB for my working data. I heard some negative reviews of indefinite time for restoring data during a crash etc etc…

    Now I am thinking of buying a NAS 40 TB+ that will work as backup as well as cloud storage, so I can work anywhere. I could sync the home PC. Grant access to other editors to work. Share my stock footage with others.

    If I keep the new NAS in the studio unattended, I worry about any fire hazards, cooling at nights. keeping the air conditioning or even the ceiling fan all night may get expensive. India gets hot up to 40 deg Celsius during the day. Power outages are common too. However, I have UPS and inverter backups to take of that.

    Any insights from the experienced? Many thanks in advance.

  • Tim Jones

    January 25, 2021 at 12:34 am

    Of course, my initial recommendation would be to look into an LTO tape solution. The initial cost of entry may be a bit frightening, but the long term costs and peace of mind will pay off quickly.

    For 40TB of data, you could use 7, 6TB LTO-7 tapes that are –

    • Easily transportable
    • Robust even in India’s summers
    • Require no power to maintain
    • Can be stored safely offsite in a vault or even duplicated to provide a geographically distant deep archive.

    And with appropriate connectivity (TB 3 or SAS) at home and your studio, you can carry the tape device between the systems so you only need a single tape device.

    Aside from that, you don’t mention the network capabilities between your home and studio. If you have a high speed service, you could take the disk route and place a low-cost disk array in both locations and then use something like rsync to keep both side sync’d so that you always have the same data available in both locations.

    In either case, you’ve provide at least a minimum of geographical (and hopefully power grid) separation between your data while making it easily available at both ends.

    Good luck!
    Tim J

  • Tim Gerhard

    January 25, 2021 at 5:10 pm

    For 40TB a NAS with a cloud backup would be perfectly fine. QNAP is a great choice.

    With BackBlaze be warned that it is not for backups only and not archive. You must keep the original data on the source NAS or they delete it from the cloud backup eventually.

    I’m an LTO tape guy, but wouldn’t recommend it in your situation unless it’s what you really want. LTO doesn’t really shine for cost until you hit around 60TB+ for archiving. You mentioned your environment is typically 40 degrees Celsius typically. LTO tapes should be stored up to 25 degrees Celsius long-term and when in use 32 degrees Celsius, so you’d have to manage the temperatures wherever you’re operating and storing tapes if you used them.

  • Paul Carlin

    January 27, 2021 at 8:12 pm

    A hard drive docking station and some “archive” rated SMR hard drives are an economical solution to backing up your data. Make two copies if you’re paranoid. Instant access to the data when you need to restore it. I take a snapshot of the directory structure and save them as text files so I can easily search for what I’m looking for and know which drive to plug in. I also recommend a hard drive storage case like the Orico 20-bay suitcase. Grab and go!

  • Bob Zelin

    January 29, 2021 at 3:30 pm

    I strongly suggest contacting Neil Sadwelkar in Mumbai. Neil is a great engineer, and can assist you with all of this. I work with him all the time in India.

    http://www.sadwelkar.com/neil.htm
    Bob Zelin

  • Neil Sadwelkar

    January 30, 2021 at 4:51 am

    Thanks Bob.

    Santanu, I’m in India too. And I too had to move my setup home in March as well. And am still operating from home. From your profile and web site it looks like you’re in Mumbai (Thane). I’m in Mumbai too (Andheri W), so we have similar climates. I’ve had similar challenges in temp and dust.

    About your server, you haven’t mentioned what kind of a server and how you connect to it. Or how many users currently connect to your present server.

    For 40TB of data, I don’t think Backblaze is a viable option. or practically any other cloud service. In fact, for these very use cases, in the last NAB I went to (2019, which was the last NAB anyone went to) I had meetings with Backblaze as well as Wasabi to figure out if we could work out a cloud backup structure based in India for Indian users. With the backup ‘seeded’ with shuttle drives, and then only updated online. That didn’t work out as neither was interested.

    Anyway, for 40TB of data which is shared between more than 3 users, who have to access the same media at the same time, a NAS is needed. If they only need to be able to occasionally share files, and never work on the same files, then its more cost effective to provide each user with sufficient local storage, and have them all connected over a fast network (not WiFi).

    Another interesting new cloud shared storage service is Postlab. For under Rs 10,000 per month, 3 users can share proxies up to 3 TB, and keep their FCP X libraries in sync over the cloud. Basically edit from anywhere with up to 28-50 hours of proxy footage in the cloud.

    That being said, the case for LTO drives is that they extend the capacity of your local storage – direct storage or NAS. Projects that have been delivered and are not likely to return immediately, but might one day, can be backed up to LTO tapes and erased off your primary storage. To be restored when the project returns for versions/changes.

    Cost-wise, LTO tape costs Re. 1 per GB now. Of course, you need the drive and software which will run about Rs 3.5 lakhs. Assuming a life of 5 years for the LTO drive (with moderate use of about 20-25 tapes per annum) and assuming you ‘sweep out’ data at the rate of about 10 TB per month, your effective tape ownership/backup cost goes to Rs 1.50 per GB. With heavy use, it will be about Rs 2.5 per GB. This is way cheaper than any kind of drive storage. Which is about Rs 2 per GB for the cheapest consumer drives up to about Rs 7-10 per GB for good RAID drives or NAS.

    There are also LTO service houses who do LTO Backups only. That might be more cost effective if your backup requirements are under 5 TB/month.

    So, drives+LTO effectively is sort of like unlimited storage.

    Hope this helps. Contact me offline and we can talk.

    My email is my name surname (one word all small) at me dot com

    Neil

  • Neil Sadwelkar

    January 30, 2021 at 4:56 am

    That’s a great bit of advice Paul. And more convenient than LTOs.

    What I use to track my ‘archive’ drives is Diskcatalogmaker. It saves’catalogs’ of your drives down to the last file, and you can ‘browse’ the drive without it being connected.

    For an entire feature film original camera files, we often have them backed up to, say, 8 drives of 12 TB. I ‘scan’ all 8 into one Diskcatalogmaker catalog, so I can search for a particular file across 8 drives and 96 TB without connecting a single drive.

    Diskcatalogmaker also exports all or selected files/folders as csv so you can make a Excel database of file names, sizes and paths and use an AppleScript or Excel formulae/macros to parse EDLs/XMLs and create a ‘copy script’ to extract selected files.

    Neil

Viewing 1 - 7 of 7 posts

Log in to reply.

We use anonymous cookies to give you the best experience we can.
Our Privacy policy | GDPR Policy