New LTO User looking for some sage advice

[David Fox]

Hi Shawn,

Just wanted to respond regards this statement from your post..

I was told that the drive you are writing from should be at least as quick as the write speed of the mTape LTO 8 TB3 drive (which I believe is 360Mb), however finding large capacity drives here that can also write that quick is a challenge and would be very costly.

My understanding here (I work for another software vendor Archiware) is that LTO drives are able to speed match the speed at which the tape runs through the drive, with the incoming data rate. So your LTO 8 drive will be spinning at its ‘top speed’ when date is arriving at 360Mb/sec, but as the incoming data-rate slows, the drive will be able to slow down to match. There is however, a lower lower limit, at which point the tape speed can no longer match the data speed. At this point, the drive will stop the tape, fill up its internal buffer with data, and then run the tape again when there’s enough data to write some blocks to the tape. If data continues to arrive at a slow pace, the drive will have to repeatedly stop, fill buffer, start. This is mechanically bad for both drive and tape. Sometimes referred to the ‘shoe shine’ effect.

All this speed matching and shoe-shining can be heard when listening to the drives noises.

The lowest speed at which your LTO drive will be able to accept data will be documented somewhere for the specific drive being used in the mTape enclosure. From my experience, it can vary a little between manufacturers.

Also note that the quoted 360MB/sec speed is for full height LTO drives, I think the drive used in an mTape TB enclosure will be half height, and max speed will be 300MB/sec.

So… your disks don’t need to be super-fast to write to LTO, just no too slow.

This is all based on knowledge of drives. If Canister has it’s own restrictions on data-rates. Then i’m not aware of them.

Best,

David Fox

[Shawn Convey]

So your LTO 8 drive will be spinning at its ‘top speed’ when date is arriving at 360Mb/sec, but as the incoming data-rate slows, the drive will be able to slow down to match. There is however, a lower lower limit, at which point the tape speed can no longer match the data speed. At this point, the drive will stop the tape, fill up its internal buffer with data, and then run the tape again when there’s enough data to write some blocks to the tape.

That is great to know David and much appreciated information which makes sense now that you spell it out for me 🙂 — You were also correct in the fact that mTape LTO 8 does have a max at 300Mb/sec and as such I suppose that the 250Mb/sec my Oyen Digital Novus 12TB External USB-C (3.1,Gen2) 7200RPM Hard Drive has wouldn’t slow down the process all that much.

NEW QUESTION
Bad Session Write — I have made several successful large transfers to my LTO tape however the folder that is currently being backed up is acting problematic (VERY SLOW transfer and verify speeds taking 100x longer than it should with the drive spinning up and down often) and I suspect that there is a good chance that this write session will not verify successfully in the end AS SUCH — What happens to the remainder of the data that is on the tape? Do I need to reformat the tape and start from scratch OR can I somehow delete that last block of data and continue?

[David Fox]

Glad that was useful Shawn. Regards you later point, one thing to bear in mind when writing files to tape, is the file sizes that you’re reading/writing and how they influence speed.

There’s a big different between asking your disk to read thousands of small files, totalling (let’s say) 1TB, versus a small number of large files that have the same total size. Even if your source disk is relatively fast, the reading of thousands of small files will be much slower than reading a few large files. Each file being read has an overhead – it has to be located on the disk, it’s meta-data read, verification checksum calculated etc etc.

Therefore, during the course of backup/archive of your folder, you’ll find the data-rate to tape will slow while reading many small files, and speed up when reading large files. There’s not a lot that you can do to overcome this, just the way of things.

[Shawn Convey]

Therefore, during the course of backup/archive of your folder, you’ll find the data-rate to tape will slow while reading many small files, and speed up when reading large files. There’s not a lot that you can do to overcome this, just the way of things.

OK that is great to know… I still think something was acting funny as 5.5hrs for 9.5GB (not TB) of data transfer / verification seems extremely excessive. But I have the developer looking into this as well…

Can you point me in a direction to find practical info such as this (small files of same data total will take longer to write / verify, how much space should be left on a LTO tape as buffer, Issues to watch out for, LTO workflows etc….) online? I am happy to do my own research on all of this and don’t mean to single out a forum or person on a forum individually for all these questions but I have been coming up short finding a location (or locations) that have non IT, end-user practical info on the basic in’s and outs of LTO archiving…

Also as you work for a software developer does Archiware have any plans on releasing a lower cost single user version of their software? The investment in a a LTO system (hardware and software) is very large for a single user / small shop and while I understand that hardware is regulated or controlled by IBM and they set the price I probably still would have justified the cost long ago IF the software wasn’t similarly expensive… It seems a shame that there are only one or two software options for the “little guy/gal” and the film / video field is becoming far more populated with single owner shops who have huge amounts of data that needs archiving… Anyway, I guess I am saying there seems there may be a growing demographic that could be served by more lightweight / low-cost options for software. 🙂

[David Fox]

Hello again!

Yes the 5.5hrs to write 9.5GB sounds like a problem somewhere. That’s far too slow.

I honestly don’t know of a good end-user targeted resource for info on LTO tape. Maybe a gap in the market. Archiware have a YouTube channel (https://www.youtube.com/c/archiwarep5) where I post explainer/training/demo videos, so maybe an opportunity for one explaining what we’ve been discussing here!

Our lowest cost for Archive to LTO product is USD 1,680 which is intended for single users such as yourself. This is already a big reduction in price from our workgroup targeted products. We’re not just throwing a folder onto LTFS but maintain a media index of everything written to all tapes, a database accessible via a browser.

Anyway, not wanting to get too sales-ey on these forums. Hope you’re able to resolve the speed issues. Seems to me that it should be fixable, you just need to figure out where the slowness is coming from.

[Neil Sadwelkar]

I routinely write tapes, like several times a week, using an Intel Mac mini or Intel MacBook Pro as host computer. I have not yet written a tape using any Apple M1 system as host computer.

I use a standalone tape drives (I have 5) connected to a SAS card inside a Thunderbolt – PCIe box (I have 3 of these). And for software, it used to be Bru PE for a decade but now it’s Yoyotta and Canister.

LTO write speed matches the read speed of the hard drive you’re reading from. I haven’t come across a lower limit, and I often have to backup to LTO-8, from 2.5″ portable drives that read at 60 MB/sec when full. I’ve even seen speeds as low as 40 MB/sec.

Writing to LTO takes longer with file sequences like .dpx, .ari, .arx, .tga etc. meaning 100 GB consisting of 14,000 files, will take longer than 100 GB consisting of 30 files. Longer means about 1.5-2 times as long sometimes. Nowhere near 5 hrs for 9 GB. So that’s definitely an anomaly.

Do you have access to an Intel Mac where to try out your entire setup? There is a likelihood of a driver issue with Apple M1. Your mTape TB3 LTO drive is actually an LTO drive connected to a SAS card attached to a TB3-PCIe converter all inside the same case. So its not vastly different from my setup where these two things are in separate cases.

I checked on the mLogic site and the drivers link links to the Atto drivers page so it looks like your mTape has an Atto 6GB SAS card inside it, same as mine.

The Help Centre at mLogic site also has links to Yoyotta 10 day demo. You could download and try that.

I’m sure the guys at Hedge with do all they can to try and fix the Canister issue. But I don’t think the problem is with Canister, in your setup. It either to do with SAS and LTO drivers for your M1 host computer, and/or errors on the drive with data. Have you tried doing a verify and repair on your source drives?

And one more thing. Are your source drives all HFS+? I’ve found some funkiness with ExFAT or NTFS drives sometimes.

[Neil Sadwelkar]

Shawn,

About the other things you’ve mentioned in your post.

You definitely need some cataloguing software to be able to browse, at one place, the contents of your drives, and to be able to store an offline record of the contents of the LTOs you are writing.

Take a look at DiskCatalogMaker or NeoFinder.

Writing many small sessions or 1 TB or fewer long sessions of 11 TB to fill out an entire LTO-8 tape. No difference, really. No gain or loss in doing either. But if you can choose between the two, I’d go with few large sessions than many small sessions.

Doing double backups. Yes there are dual drives, and software that can write to two tapes simultaneously. But if you don’t have access to either, then writing it all over again is the only way. Hence, fewer long sessions make this less painful.

Resources online. I’ve tried to find them too, but there don’t seem to be any except this forum. I’ve had to discover many things by trial and error. But I’ve been blessed with a continuous amount of LTO backup work to do, with no real time restraint, so I’ve had the time to try things out. I’ve bought and tested many software along the way.

In the pandemic, I’ve had to to a fair bit of this ‘remote ops’, not for LTO, but for editing, motion graphics, VFX systems and such like. It’s not the same as actually working on a machine, but at least it’s safer.

If it helps, I can have one of my team members (or me) operate the LTO backup for you remotely. All you have to do is to connect and disconnect drives and remove and insert tapes when needed, we can take care of the actual operations. Any troubleshooting we will do remotely. And it will all happen in front of you on your machine.

[Shawn Convey]

LTO write speed matches the read speed of the hard drive you’re reading from. I haven’t come across a lower limit, and I often have to backup to LTO-8, from 2.5″ portable drives that read at 60 MB/sec when full. I’ve even seen speeds as low as 40 MB/sec.

Writing to LTO takes longer with file sequences like .dpx, .ari, .arx, .tga etc. meaning 100 GB consisting of 14,000 files, will take longer than 100 GB consisting of 30 files. Longer means about 1.5-2 times as long sometimes. Nowhere near 5 hrs for 9 GB. So that’s definitely an anomaly.

Thanks for the insight @Neil!

– I don’t have an intel mac at the moment unfortunately.
– I will have to double check how all of my source drives are formatted but normally it is HSF+, I did have some ExFat / NTFS drives for shuttling between PC and Mac but I don’t believe I tried to use any of those as source drives in this scenario.

But I don’t think the problem is with Canister, in your setup. It either to do with SAS and LTO drivers for your M1 host computer, and/or errors on the drive with data. Have you tried doing a verify and repair on your source drives?

Right now the team at Canister is communicating with IBM trying to figure it out as well..
In the end it looks like I have a complete LTO 8 tape which which had about 8-9 sessions written to it using different source drives, and connection configurations write and verify. With the exception of that ULTRA-SLOW 9.5GB back up session everything else seemed to go smoothly with varying write / verify speeds depending on drive speed and connectivity. I did notice a much slower transfer when I had the mTape TB3 plugged into my OWC m.2 raid unit which was then plugged into the MBP, which makes sense I suppose, but even that session wasn’t as slow as the 9.5GB write.

[Shawn Convey]

@neilsadwelkar

Thanks again for your additional thoughts ( I think I posted my response to your first post at the same time your last post came through 🙂 )

All of this info is very helpful but I do have a few follow up questions:

1. Cataloging software — I checked out the 2 you mentioned and I didn’t see any specific LTFS / LTO support. Would I simply drag the mounted LTFS volume on my HD to either of these software choices for them to make the catalog or can I import a manifest or is there a better way to connect through Canister?

2. Remote Ops — that is very interesting proposition and not one I considered as possible… I would like to figure out a good workflow for my needs moving forward as once I am done archiving the past 10 years (which as a one man band isn’t nearly as bad as even a small production house) I will simply be archiving shoot data as it comes so it should all be very manageable moving forward… that is once I work out the workflow and the kinks. But with all that said knowing that your remote services even exist gives me some more piece of mind so thank you for mentioning that!

[Neil Sadwelkar]

1. Cataloging software — I checked out the 2 you mentioned and I didn’t see any specific LTFS / LTO support. Would I simply drag the mounted LTFS volume on my HD to either of these software choices for them to make the catalog or can I import a manifest or is there a better way to connect through Canister?

Both DiskCatalogMaker and NeoFinder treat a mounted LTFS volume as a drive. So you simply drag and drop the LTFS volume to the software and it scans and presents it as a drive. In DiskCatalogMaker, you can even have multiple LTFS volumes in one catalog, or you can drag catalogs from one to another. So, you could have multiple tapes inside one DiskCatalog and search across tapes without mounting them.

I have Disk Catalogs of even 40 drives inside one catalog and can search for one file or folder within those. And then mount that tape and extract that file/folder.

The file format of the catalog is .dcmf or .dcmd and you can park it in your Google Drive or Dropbox so that it accessible to you anywhere. There is a provision to save thumbnails of the media but that’s time-consuming so I don’t use it.

And, you can export this info as a csv and build a spreadsheet as well.

Neil