free or low cost tape backup software? (and is there a point)

[Martin Greenwood]

The mechanisms and performance of all LTO-7 and 8 drives are the same, as they are all made by IBM, so your choice is on price or brand.

We would recommend using LTFS, it’s widely used and cross platform. Also we see customers using LTFS tapes as an interchange, rather than handing over a $100 hard drive why not send a $25 tape that can be accessed quickly.
An LTO-7 drive can read three generations of LTO tapes.

My company YoYotta develops workflow software for production and post and have used LTFS for over 7 years. Our Conform tool takes an editors timelines and partially restores the clips needed for the edit or grade. It can do this directly from an LTFS tape. Projects can have routinely have hundreds of TB of raw footage so it makes sense to store them on a shelf and restore files on demand.

Martin

CTO

YoYotta.com

[Tim Jones]

Hi Rachael and others,

Think about the value of your data and think about what it would cost you to recreate that data if it was lost. There is a reason why every major studio and post house has chosen BRU to protect their work. In fact, we can happily claim responsibility for more than 31 major motion pictures being recovered from massive system and storage failures since 2004.

If you are looking only for free backup software, you’ll need to look to a platform other than Mac OS and use tools like tar or cpio. There is no tape layer in Mac OS like there is on other Unix-based platforms for accessing tape, so the free options don’t exist on the platform. While tar, cpio, pax, and dd exist on Mac OS, they are not tape-aware. In fact, Apple actually removed the FreeBSD “sequential access” layer from the Darwin kernel back in the 0.99 beta days making native tape access on OS X unavailable. This means that real tape software vendors must create the tape IO layer themselves and that means engineering development expense that doesn’t exist on other platforms.

While you can get LTFS for free on Mac OS, it’s not a true backup/archival application any more than copying files from one hard drive to another is (which is all that the free LTFS tools basically are). Of course, others disagree with my feelings towards LTFS, but I’ve been at this for more than 32 years on over 50 different Unix platforms and I’ve witnessed a lot of “tape pretending to be disk” options come and go with the user generally caught with non-recoverable data.

When you mention your concern of lock in, I can speak for our BRU engine in that a) it has 33 years of cross-platform proven functionality, and b) you get real support from that single vendor that will cover you from the UI down to the manner in which the data is written to and recovered from the tape. BRU is available on more than 21 different platforms and and archive written on one platform can be restored on any of the others. The current version (BRU 19.0) can restore data from an archive that was written as far back as 1988 with our 6.1 version. As for support, our team knows the entire stack from the UI application down to the HBA and system device drivers to how the tape drives and libraries actually work. When dealing with free solutions like LTFS, we are already seeing platform and version incompatibilities depending on who’s LTFS suite and version you use. Also, you really have no one to go to for help if the LTFS core in use doesn’t work properly for you because of it’s “Wild West” open source nature. To quote the copyright and license header for LTFS:

The IBM Linear Tape File system Single Drive Edition is distributed in the
hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Having someone that you can communicate with when you have problems or questions is a very important part of a solution like backup and archival.

To read more about how BRU specifically differentiates itself from ALL other solutions, I ask you to take a look at this white paper for a bit more info into why a properly engineered and designed backup and archival solution is important for protecting your data:

The BRU Advantage – From 1985 to 2018, and Beyond

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Bob Zelin]

Hi Rachael –
everyone is being so nice and polite to you. Unfortunately, I will not be nice and polite.
So let me get this straight – you can afford editing systems, RAID arrays, an LTO drive (which as you know costs thousands of dollars) a disk drive host card to run the LTO (perhaps you have an MLogic MTape with is a thunderbolt LTO that also costs thousands of dollars), and you will pay about $60 per tape, but you want FREE LTO software ?

Well Rachael – you will get what you pay for. Here it is –
https://www.hpe.com/us/en/product-catalog/storage/storage-software/pip.hpe-storeopen-and-linear-tape-file-system-ltfs-software.4249221.html
This is HP StoreOpen
and let me tell you from first hand experience, it’s a piece of crap, and you will suffer when you run it
and from this very page on HP Store Open –
What’s new

Technical support is no longer available for StoreOpen Automation. Software is run at your own risk.

So when you have trouble with this FREE software, and you have problems – who will you call ? NO ONE.
No one will help you.

I don’t understand that you have thousands of dollars for the hardware to operate an LTO tape backup system, and it’s killing you (or your employer) to spend $500 for a wonderful piece of software from Tolis Group, YoYotta or Imagine Products that will work perfectly, and give you someone to ask questions, if it doesn’t work.

There ain’t no free lunch Rachael ! By the way, if I help you with this for free, would you edit a show for me for free ?

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Sam Lee]

Good thing I didn’t even bother to install HPStoreOpen many years ago. Right off the bat I saw that I have to manually fit the 2.5 Tb to each tape. That itself over time is a huge waste of time. $500 is worth it for me for a decent software that can easily span over multiple volumes and has cataloging feature such as BRU PE.

Red flag should always be raised whenever anything is free. Conventional wisdom has always been pay the price. That simple rule works for me. You’ll be disciplined and have no other options but to make whatever you purchase productive.

[Jeff Schaap]

Hi Rachael,

Just throwing in my two cents to say that we use YoYotta for LTO backups to a single drive and it’s been a good experience, worth every penny we paid for it (and it wasn’t very many pennies when compared to the cost of the rest of our system!).

Thanks,
Jeff Schaap

[Tim Gerhard]

LTFS is becoming the gold standard because it’s non-proprietary. The only real incompatibility issue I’ve seen is illegal characters between MAC and Windows, which the free software won’t handle for you, but the premium software packages do.

YoYotta and PreRollPost are both great options for LTFS, and if you end up switching to one or the other for some reason, you can still use your tapes.

If all else fails down the road switching your mind about software packages, you can always fall back on the free software to read your LTFS tapes. I’ve seen cases where people load content onto LTFS tapes and ship them somewhere else, and this frees up the need for the person n the other end to buy into the software you’re using.

If you’re concerned with the $500 price point on any of the software, the guys that make PreRollPost have a lite version called “myLTO” for $299.

Tim Gerhard
Magnext
614-433-0011 x114
tgerhard@magnext.com

[Rachael Busher]

Responding to things out of order.

I’m still in college, much of what I do is on a shoestring. There is not enough side paying work to justify the level of investment most other people are making in their projects – a few hundred dollars is still alot of money. My main observation is that LTFS seems to almost do a better job for free (by maintaining cross platform compatibility and near line access to files) that i’m trying to understand why I would want to lock away files in a proprietary format? I mean sure i’m willing to, but sell me on it, help me understand why to spend I dont know $500 maybe for software that is probably overkill for film student and talented amateur budgets, i’m trying to understand better what the differences are. If I were a production house clearing a third of a petabyte per year in offline media archives that seems like a completely different use case.

As to technical support on HP’s software I don’t know anything about it, I was just planning on using straight LTFS which is already more reliable than the straight mirroring to USB backup drives i’ve been doing for years because i’ve had bit rot and file corruption sometimes occur. The main advantage I would see in backup systems designed for petabyte level storage is probably keeping track of easy to lose files but I already have a system for that which I use with external USB drives. Most answers I can see coming from a backup company I would imagine to involve “well if it didn’t work then your tape was corrupt” and other than straight three way mirroring keeping one offsite, like I assume they aren’t going to be able to do much for me anyways. The tape is going to read in the hardware – or it’s not. If that data is further locked away in some proprietary format – that helps me even less than if I can boot it onto any system.

If you help me with this for free, I will pay time for time for something else I know about. Asking for best practices advice is not the same as volunteering hundreds of hours for an editing project. :-/

[Rachael Busher]

I’m just perusing some of the options now… having heard multiple perspectives.

BRU looks interesting – reading the white paper am I correct to assume that file integrity is it’s main advantage over LTFS, ie avoiding bitrot and that it is an off the shelf solution for this? (which i’ve already experienced on my mass of external hard drives) Just to be clear I am already aware of this problem (the hard way) and am already trying to implement a solution that makes sense at my lower budget to insure file integrity and recoverability. (the whole point of the ZFS filesystem for NAS’s was dealing with bitrot too)

I am curious what price ranges BRU costs down at the single user level – this seems more high end than I will be for awhile but I could see the time savings/convenience mattering as projects scaled way up. I don’t think i’ll be using it tomorrow but i’ll be bookmarking it for future consideration.

YoYotta looks like it’s mostly about being the ‘card catalog’ to a library and is already content aware/intelligent about the nature of needing old AV files. This too looks useful, but does it do anything like file integrity handling like BRU does or is that considered not it’s purpose? (if I already have ways of insuring file integrity this may be irrelevant i’m just trying to understand what each thing focuses on here)

“Think about the value of your data and think about what it would cost you to recreate that data if it was lost.” Already learned this the hard way, one difficulty is that we all have budget limitations. Simply putting files on LTO6 tape, with parity verification/recovery information over the top of LTFS, is my first step in this process. Meant to last long enough until I can upgrade to better solutions for older projects like those i’m trying to insure don’t experience any more bit rot. If a solution costs so much I can’t afford it until after “another hard drive crashes” it does me no good – so it’s more about understanding what I can do now, and what I should ideally do later, and not painting myself into the corner with the first solution somehow because the original solution at least works properly – even if it is not the most convenient, ideally time saving, or similar. Ie the minor hassle of “breaking projects into 2.5 TB segments” isn’t a big deal when my NAS is full of 3TB drives not overfull to avoid fragmentation and i’m not yet upgraded to a more convenient drive pool.

Another issue is interacting with outside parties. Mailing people an LTO tape using LTFS is something I can often expect people on the other side to handle, they may or may not have the same backup software I do and i’m not necessarily going to buy it for them. A system that lets me interchange large amounts of data and verify it didn’t arrive corrupted and the needed data is organized/findable will be needed whether I buy backup software or dont buy backup software is my point.

Basically it sounds like the two main differences between ‘free’ backup software and ‘real’ backup software is either data integrity OR easier librarian duties managing old versions and offline media archives, and the software either does one or the other. Both of these make sense and are good ideas, I was just wondering at what point they make sense on the smaller end of the scale or at what point in the growth curve/when to start using them. (probably when the time saved by convenience is more than the cost of the software)

[Bob Zelin]

Hi Rachael –
OMG ! I know that you feel that I am attacking you with my response, and I fully appreciate that you are a very VERY ambitious student (more than most professionals I know) – but I have to respond.

1) you are a student. You do not need LTO for your project. It is fantastic that you want to learn about LTO, and that you are becoming aware of the different types of LTO archive solutions on the market. But this is not your financial responsibility, and no offense, but your school project is not that critical that it requires LTO archive.

2) your school department (film department, video department, communications department) should purchase this software for both you AND the students (and the LTO drive, and tapes – well maybe not the tapes) if they feel that LTO is critical to the workflow of your classes or projects. The school can certainly afford it. It’s not your responsibility. I think it’s great that you want to do this, and are interested in learning this, as when you get out, your real job as an assistant editor will involve boring things that include transcoding, exporting, and archiving projects. Your LTO knowledge will be very valuable for you.

3) I am very anti school. While you have no money, your PARENTS that are funding you for an education whose degree will mean nothing in the real world, would be better suited to purchase real LTO software for you, instead of paying for school. What you will find, is that in the real world, your extensive knowledge of LTO, transcoding, storage, software from Adobe, Davinci Resolve, AVID, FCP X and assorted graphics programs (Cinema 4D, etc.)
will find you a job in this industry in a heartbeat, compared to a “student” with a 4.0 GPA and a nice independent film. Everyone is a film maker. Not everyone knows the knowledge that you appear anxious to learn. That knowledge will give you the opportunity to “get in” and move forward with your career – not your creative film making class.

Trust me Rachael – don’t waste 5 minutes with HP StorOpen. It will leave you with heartache and frustration.
And I just looked – YoYotta ID is 160 british pounds, and MyLTO for Mac from Imagine Products is $299.
Besides – who is going to get you the LTO drive, which costs thousands of dollars, even if you had the software for free ?

Just my stupid 2 cents.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Tim Jones]

I have to totally agree with Bob. In interfacing with the big studios and post houses, what you know about the technology and how to best utilize it weigh far more into landing a job in a real situation (you know, one that can pay the bills) and is far more important than an indie short and 2-4 years of film school – which is only going to get you an AD position or possibly an ADP position – mostly at the intern level.

However, KNOWING about mezzanine formats, media management, encoding and transcoding, and storage and archival will land you a serious position with almost any of them. Not everyone can be a director or DP. Most productions only need 1 of each. However, every production needs people that understand storage technologies and the manner in which the various platforms, filesystems, and devices work together.

Spending your money on an LTO solution and some storage so that you can learn about storage formats, filesystem pros and cons, media management, and archival needs is a much more productive – and far less expensive – school of hard knocks than spending 5x that cost on schooling that is more often out of date other than the basics by the time that you graduate.

To further restate Bob’s comments on free software –

LTFS is free, but it comes with the support and assistance that you get with anything free – if you break it, you get to keep the parts. Things like StoreOpen and OpenLTFS are simply GUIs that sit on top of the arcane command line incantations required to make LTFS work – assuming that the LTFS suite you have located actually works properly on your platform (Windows, Mac OS, Linux). And, if it doesn’t work, you’re at the mercy of your peers since the LTFS suite is released as Open Source and contains the caveat in the license that:

3. WARRANTY DISCLAIMER

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, QUANTUM AND ITS SUPPLIERS PROVIDE THE SOFTWARE “AS IS” AND WITH ALL FAULTS, AND HEREBY DISCLAIM ALL INDEMNITIES, WARRANTIES AND CONDITIONS, EITHER EXPRESS, IMPLIED, WHETHER BY STATUE, COMMON LAW, CUSTOM OR OTHERWISE, INCLUDING, BUT NOT LIMITED TO, WARRANTIES OF TITLE AND NON-INFRINGEMENT, ANY IMPLIED WARRANTIES, DUTIES OR CONDITIONS OF MERCHANTABILITY, OF FITNESS FOR A PARTICULAR PURPOSE, AND OF LACK OF VIRUSES. Quantum does not warrant that the operation of Software will be uninterrupted or error free or that the Software will meet Licensee’s requirements. Some states/jurisdictions do not allow exclusion of implied warranties or limitations on the duration of implied warranties, so the above disclaimer may not apply to Licensee in its entirety.

HPE, IBM, and others have very similar wording in their licenses and IBM even goes so far as to tell you that they offer no support for the LTFS tool chain.

Investing $300 to $500 in a software solution that DOES provide support is a smart investment unless:

• A) You really don’t care about the reliability of the solution
• B) You are a C/C++ coding wizard
• C) You are very literate in the way tape devices work at the low-level SCSI layer
• D) You know how to create and debug kernel-level code for your platform

Otherwise, as I said above, if you break it, you get to keep the pieces.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!