[Rachael Busher]

I’m just perusing some of the options now… having heard multiple perspectives.

BRU looks interesting – reading the white paper am I correct to assume that file integrity is it’s main advantage over LTFS, ie avoiding bitrot and that it is an off the shelf solution for this? (which i’ve already experienced on my mass of external hard drives) Just to be clear I am already aware of this problem (the hard way) and am already trying to implement a solution that makes sense at my lower budget to insure file integrity and recoverability. (the whole point of the ZFS filesystem for NAS’s was dealing with bitrot too)

I am curious what price ranges BRU costs down at the single user level – this seems more high end than I will be for awhile but I could see the time savings/convenience mattering as projects scaled way up. I don’t think i’ll be using it tomorrow but i’ll be bookmarking it for future consideration.

YoYotta looks like it’s mostly about being the ‘card catalog’ to a library and is already content aware/intelligent about the nature of needing old AV files. This too looks useful, but does it do anything like file integrity handling like BRU does or is that considered not it’s purpose? (if I already have ways of insuring file integrity this may be irrelevant i’m just trying to understand what each thing focuses on here)

“Think about the value of your data and think about what it would cost you to recreate that data if it was lost.” Already learned this the hard way, one difficulty is that we all have budget limitations. Simply putting files on LTO6 tape, with parity verification/recovery information over the top of LTFS, is my first step in this process. Meant to last long enough until I can upgrade to better solutions for older projects like those i’m trying to insure don’t experience any more bit rot. If a solution costs so much I can’t afford it until after “another hard drive crashes” it does me no good – so it’s more about understanding what I can do now, and what I should ideally do later, and not painting myself into the corner with the first solution somehow because the original solution at least works properly – even if it is not the most convenient, ideally time saving, or similar. Ie the minor hassle of “breaking projects into 2.5 TB segments” isn’t a big deal when my NAS is full of 3TB drives not overfull to avoid fragmentation and i’m not yet upgraded to a more convenient drive pool.

Another issue is interacting with outside parties. Mailing people an LTO tape using LTFS is something I can often expect people on the other side to handle, they may or may not have the same backup software I do and i’m not necessarily going to buy it for them. A system that lets me interchange large amounts of data and verify it didn’t arrive corrupted and the needed data is organized/findable will be needed whether I buy backup software or dont buy backup software is my point.

Basically it sounds like the two main differences between ‘free’ backup software and ‘real’ backup software is either data integrity OR easier librarian duties managing old versions and offline media archives, and the software either does one or the other. Both of these make sense and are good ideas, I was just wondering at what point they make sense on the smaller end of the scale or at what point in the growth curve/when to start using them. (probably when the time saved by convenience is more than the cost of the software)

[Rachael Busher]

Responding to things out of order.

I’m still in college, much of what I do is on a shoestring. There is not enough side paying work to justify the level of investment most other people are making in their projects – a few hundred dollars is still alot of money. My main observation is that LTFS seems to almost do a better job for free (by maintaining cross platform compatibility and near line access to files) that i’m trying to understand why I would want to lock away files in a proprietary format? I mean sure i’m willing to, but sell me on it, help me understand why to spend I dont know $500 maybe for software that is probably overkill for film student and talented amateur budgets, i’m trying to understand better what the differences are. If I were a production house clearing a third of a petabyte per year in offline media archives that seems like a completely different use case.

As to technical support on HP’s software I don’t know anything about it, I was just planning on using straight LTFS which is already more reliable than the straight mirroring to USB backup drives i’ve been doing for years because i’ve had bit rot and file corruption sometimes occur. The main advantage I would see in backup systems designed for petabyte level storage is probably keeping track of easy to lose files but I already have a system for that which I use with external USB drives. Most answers I can see coming from a backup company I would imagine to involve “well if it didn’t work then your tape was corrupt” and other than straight three way mirroring keeping one offsite, like I assume they aren’t going to be able to do much for me anyways. The tape is going to read in the hardware – or it’s not. If that data is further locked away in some proprietary format – that helps me even less than if I can boot it onto any system.

If you help me with this for free, I will pay time for time for something else I know about. Asking for best practices advice is not the same as volunteering hundreds of hours for an editing project. :-/

[Rachael Busher]

Well i’m all ears, you can recommend specific other sellers especially if prices for packages are competitive, curious what the package difference is though – I prefer to pick my own SAS card (because i’m using it for other things), I assumed at least LTFS was free for DL because that’s implied everywhere, cables are just… cables… so I order one with the drive, and tapes i’ll be ordering by the boxload soon enough.

Publically working through this – since i’m a newbie hopefully someone can give me a headcheck if I have a misunderstanding anywhere. Keep in mind i’m both still in college and not working on enough volume that hundreds of dollars is irrelevant to me for a minor enough inconvenience, I just have to migrate a few dozen TB of data to start, not petabytes.

If I can mention specific places I just have somewhere called BackupWorks bookmarked for now and I see half height internal dedicated LTO6 drives for like $1650 by Quantum for the TC-L62AN-BR-C and $1740 by IBM TS2260 which says it has dual SAS ports plus Ethernet. (does this mean it can be read/written through Ethernet instead of SAS else what would it be for, or connected to two separate computers with their own SAS cards? Or does the SAS just let you chain to something else after? I mean I sort of understand how SAS works, I just dont know how this specific DRIVE works/what is implied by them mentioning dual “SAS ports” because I don’t see why it should need it.) HP is like $500 more and I see no critical difference in features. Other models for LTO6 are notably more expensive but I see no real feature differences.

I’m pretty set on LTO6 for now – throwing the numbers into a spreadsheet it’s not worth another $500-700 for an LTO7 drive vs the ROI because by the time media might be cheaper for it, i’d have to upgrade my NAS drives to feed it. I’d rather jump two gens to LTO8 (and use LTO7 media formatted m8) when it’s time upgrading NAS and tape drive all at once, probably once LTO9 is out and prices drop in a few years. $500-700 can get me a GPU or SSD upgrade or second dSLR for two camera shoots, it’s not throwaway money.

Anyway i’m shopping for price alone and these are the two least expensive right now and it beats amazon and I dont trust saving $100 on ebay from an unknown supplier. I’m not willing to trust barely less expensive on something refurbished or an unknown supplier but willing to look at other suppliers – just didn’t see many others in my searching. Ebay prices on the IBM are notably more for some reason, is it in any way a better drive and worth the $90 extra from BUW? I assume they will all write from single drives “as low as 55MB/sec” without shoeshining, which is lower than any drive I want to back up from. Two of the drives list a 512MB cache, so that’s about 9 seconds of buffering of small files.

Externals of either are about $300 more and i’m not willing to pay that right much more, I originally assume this just fit in a half height 5.25 bay and used some standardized power connection but looking at dimensions on BUW i’m not sure – a different site lists the same Quantum drive with what I assume are normal 5.25 ATX case dimensions though.

One quirk I see is Macintosh not listed as a supported platform on either drive from BUW which I assumed should just be about a compatible SAS controller and some Mac version of LTFS. I would prefer having that ideally, but the only listing for sure is in an HP drive some $500 more and that’s more than i’m willing to pay for Mac compatibility right now because I don’t see any other reasons to buy the more expensive drive. (because if it’s plugged into a Mac or multiOS I can just directly write a tape from the system drive instead of imaging over the network) At a different website the same Quantum drive at least implies Mac compatibility.

I don’t have to buy overnight but my NAS is filling up at a rate that by the end of March I either add another drive or migrate old-old data off to some tapes so that’s my expected buying window.

I primarily use Windows and my NAS is simple Windows so i’m just planning on using LTFS for now. (in the future I plan to get more into Linux and Mac OSX and hope to use the same drive there, probably all from the same hardware computer – the handful of times I may move the drive physically to another machine or server will be counted on one hand so I don’t think i’ll spring for an external)

[Rachael Busher]

So in theory EVERY LTO6 drive should write fine as slowly as 55MB/second? Are any better than others? Is there anything specification or featurewise to suggest buying one drive over another then or not really/I can just buy on price?

I’m pretty sure I want LTO6 – by the time LTO7 media is cheap enough to matter i’m willing to upgrade to an LTO8 drive (and use M8 format) which can’t read my older tapes anyways so i’d still need the older drive. I see no point in paying more for an LTO7 drive now just to use LTO6 media – I see no point in using LTO7 media without an LTO8 drive to make better use of it. And economically the higher cost of the LTO8 drive doesn’t pay for itself over the LTO6 in my expected time window of usage and minimum vs potential writing needs.

Also if i’m going to newer drives i’m going to need higher performing writing systems and such – i’ve run a spreadsheet of how many hundreds of TB I realistically expect to have to use or need to scale up to over a period of time and nothing newer ever makes sense economically when i’d have to upgrade HD’s and change intended workflows even of migrating data around to faster drives. I just want to write straight to tape from each drive in an array of 3tb internals which I already have (along with overbought spares) which on the slow side max around 80MB/sec I think.

I’m sure the writing of small files to the drive will keep up with the cache, i’m just concerned about the reading side and how many seconds deep the cache is if I hit a slow point of too many files. I don’t plan to be storing lots of small files, it’s more like say a large video file then hundreds of small res snapshots at different timestamps in a preview directory, then another large video file. I’d still like to know if there’s any way to make a “tape image” of contiguous files that can read at full speed without thrashing around the FAT and whatever usual overhead is done… that’s less expensive than setting up an SSD or something.

[Rachael Busher]

I’m no expert on LTO (actually came here to ask questions more than give answers) but I thought the news release talking about M8 formatted tapes was very interesting. If not familiar, that’s the use of an LTO7 tape, formatted specially in an LTO8 drive, giving 50% more capacity for the money making them slightly cheaper than LTO6 per TB. (not enough to make up the difference in cost for my use, but likely to improve more and more over time) What I do know about LTO8 is it’s a bigger jump from the past – no longer compatible for even reading LTO6, and if you do that M8 format for an LTO7 tape it doesn’t read on an LTO7 drive either.

If you were desperate for both speed and capacity LTO8 using LTO7 tapes with M8 format sounds like a good deal though.