-
Data ‘migration’ as a fact of (data) life
I was re-reading a thread on LTO archives from 2018.
So much changed in these past 5 years.
LTO-6 has evolved up to LTO-9. LTFS has endured. While some non-LTFS softwares survive, Bru went away, but got revived as Argest. Even if one had Bru licences, one needed a MacOS 10.14 or earlier system to run it. Argest import of Bru catalogs off tape is a bit iffy, so, large quantities of Bru archives are best restored using an older MacOS 10.14 system.
Some of those that adopted cloud storage, are re-looking at on-prem. Because they’ve realised that over a long period, the monthly ‘holding’ charges quickly add up. Then there are ‘egress’ fees to get back your own data.
LTO backup’s one big issue is availability of compatible tape drives, and the software they were written in, after extended periods. Like, those who archived to LTO-1 to LTO-3 in the early 2000s, may or may not find the software and the system, to restore those tapes. Like NTBackup on Windows was quite popular (in my region) and I know people with stacks of those tapes. They have no easy way to read those tapes now, as the current Windows doesn’t support the software. Same with many other pre-LTFS softwares.
‘LTO migration’, and in general ‘data migration’ is a fact of life, and even on-prem storage is not ‘forever’. I tell clients that any on-prem or off-site archive storage is good for about 5 years. After that it needs to be migrated to something else.
Meanwhile those with a stack of LTO tapes, and TBs in the cloud, and an on-prem 5 year old NAS, will find the really old tapes can’t be (easily) read any more, and are taking up too much space. The TBs in the cloud have already costed way more than they had imagined and the next 5 years will cost the same or more. And the NAS is showing is age.
I’m about to undertake a large ‘migration’ project taking a client’s 9-11 year old LTO-4, 5, 6 tapes all written using Bru (in 2011-15), restoring them, and then writing them to LTO-9 using LTFS. About 23 LTO-4 tapes will be replaced with 1 LTO-9 tape. Or 12 LTO-5 tapes will be replaced with 1 LTO-9 tape.
Some people have stacks of drives. 2-6TB drives with data on them and still readable. These get ‘migrated’ to the largest available enterprise drives and get another 5 years of readable life. The newer 20TB drives can read at 250+ MB/sec so they make a good ‘near-line’ archive for recently cold data.
Neil
-
6 Replies
-
Neil, thanks for your insights. As I’ve gone back down the rabbit hole of archive and storage I’ve been reminded that storage and archiving cannot be all three of cheap, easy, quick.
We have 200 LTO-5 tapes backed up through Retrospect and probably 200-300 LaCie Rugged desktop drives. Fortunately we’re still able to recover using Retrospect and the LaCie drives have been holding up. Our company needs to make decisions regarding footage for defunct clients. I don’t think they’re going to want to spend the money to transfer it to new storage; there’s no contracts I know of that we would save their footage indefinitely.
I like the idea of an SSD hot storage, 180TB Nearline, then LTO 9 autoloader/library now that LTO capacity has reached 18TB per tape. Realizing we’re either paying to migrate in the future, or paying yearly to use the cloud providers.
If money was no object I’d go with AWS glacier deep in US West and Google archive in US Central since Google can rehydrate in seconds.
What are you using for software to do storage tiering?
-
Your assets are on LTO-5 tapes and LaCie rugged drives. The LaCie drives will run about 80 MB/sec and the LTO tapes about 100-120 MB/sec for retrieval.
Today, bare SATA drives of 20 TB each can run at two times this speed even over USB3.0. Bare drives run about $ 16 per TB. Many drives in a RAID enclosure, run about $ 25 – $ 40 per TB. This is your cost per TB for 5 years of storage.
Your 200 LTO-5 tapes will become 16 LTO-9 Tapes if you migrate then now. So, restoring the Retrospect LTO-5 tapes to hard drives, and then writing them to LTFS LTO-9 tapes will extend their life.
After 5 years, this cost of hard drives will halve, and most likely, take up half the physical space as well. So, your 300 TB of data, which takes up 15 drives of 20 TB now, will fit in 6 drives of 50 TB each of which will most likely cost exactly what you paid for the 20 TB drives now. If you paid $ 5,000 for 15 drives of 20 TB today, 6 drives of 50 TB will cost about $ 2000, 5 years from now. And they will hold the same amount of data. (I’m basing this on the fact that in 2017-18, 8 TB drives cost nearly what 20 TB drives cost today)
As for your clients, your best bet is to catalog their data using something like DiskCatalogMaker and send them a mail, showing their data as a pdf (made from DiskCatalogMaker), and offering them a certain amount per TB for 5 years. I suspect this will be less than what they would have paid for a cloud service. You could also let them pay annually.
Even if you bill your client $ 10 per TB per year, some clients with about 10 TB, may even be willing to pay for 5 years up front. And even if you manage to convince 100 TB worth of clients, you’re looking at recovering part or all of your costs for this storage venture.
There is, of course, the cost of the labour of doing this retrieval and migration.
-
Thanks Neil, those numbers are very helpful when trying to do calculations, especially looking 5 years into the future.
-
Thank you Neil for stating/re-stating what I feel many try and put their heads into the proverbial sand over–data migration is a fact of life–period.
Joby.
-
Hi Neil
Quality post, thanks for your time!
Argest is still full of bugs, nearly five years after it was first announced. They have strung me along saying “new version next week” for years. It doesn’t work. Bru is truly dead.
What are you using for LTFS software on Mac vs Windows vs Linux?
-
Jon,
All the LTO backup I do is only on Mac systems. For the past few years, it been an Intel i7 Mac Mini.
I use Yoyotta mostly, as it offers built-in cataloguing and project management. I also have Canister which I use for quick backups. And restore from tapes that were not made by me.
I have a different older MacBook Pro with macOS Sierra still on it, and a working Bru install. That I use when I need to retrieve an old Bru tape. Many of my clients have had their backups done by me which are on Bru, so I need to keep this system operational just for Bru.
Neil
Reply to this Discussion! Login or Sign Up