Creative Communities of the World Forums

The peer to peer support community for media production professionals.

Forums Storage & Archiving New LTO User looking for some sage advice

  • New LTO User looking for some sage advice

     Neil Sadwelkar updated 5 days ago 3 Members · 11 Posts
  • shawn convey

    January 10, 2022 at 6:34 am

    Hey all! I need some general pointers and a better understanding about this process than I can glean from YouTube and the white sheets. Any advice would be greatly appreciated.

    Workstation LTO specific gear — 2021 MBP 16″ M1 MAX 64GB, mTape LTO 8 TB3, OWC M.2 RAID (6/8TB), Novus 12TB USB 3.1 drive.

    Me – I am a one man band indie doc filmmaker who has made one feature (AMONG WOLVES) and is gearing up for some new long & short format projects. I am currently in Sri Lanka where tech and media are scarce and come at a premium. This last fall when I was back home in the US I picked up a mTape LTO 8 TB3 drive which shipped with Canister. The reasoning was I need to archive my feature film (and my other small gig’s) in a more permanent way and also I want a cost effective and safe way to transport my new footage back and forth intercontinentally for backup / archiving purposes.

    mTape / Canister – I ran into some serious hic-cups getting my setup up and running* and was communicating with Hedge / Canister** founder on a weekly basis in an effort to problem solve what was going on (I am only now attempting to “successfully” write my first LTO). Which brings me to some questions / concerns:

    1. Workflow / Write Time – I have many TB’s of data on various HDD’s and NAS’s that need to be organized and backed up. Currently I am working on my feature films archive and I can likely fit it all on one 12TB LTO tape. However this process is understandably taking a long time. I was told that the drive you are writing from should be at least as quick as the write speed of the mTape LTO 8 TB3 drive (which I believe is 360Mb), however finding large capacity drives here that can also write that quick is a challenge and would be very costly. I do have a m.2 softRaid (currently 6TB but soon to be 8TB) which read/writes ULTRA fast (2500Mbs +) however I purchased that as my edit drive. I also have a 12TB Novus Oyen Digital USB 3.1 drive for this express purpose BUT the r/w max’s out around 250Mbs.

      As the mTape only theoretically writes at 360Mbs and the m.2 RAID is far far faster and I want to have that space available for my editing jobs what drive / system makes the most sense in your opinion(s) to create the archives from.

      1a – LTO write speed Calculation? – Is there an easy way to figure out a basic understanding of how long each 1TB chunk of data will take to write / verify with each of my drive options?

    2. Sessions or Full Write backups – The above dilemma brings me to my concern about many small sessions over one large write. Currently I am most concerned with archiving my past work, as I have amassed tons of data that can easily fill up several LTO tapes but as the write / verify / cleanup times seem to be lengthy (days?) would you suggest I always brake the backup sessions into segments (chunks) of data say 1TB at a time. It seems like that may be more “safe” but am curious as to what you suggest.

      2a – Dupes – Without a LTO dual deck or replicator can anyone suggest any workflows on making multiple copies of the same LTO other than starting the lengthly process all over again (I think in my case that may be my only option but thought I would ask 🙂 ).

    3. Drive / Tape Storage – Again, I am going to be in Sri Lanka for the foreseeable future where both humidity and corrosion (by the sea) are huge obstacles. There is little I can do to protect the drive from these things when it is in use but when I am not using it I will be putting it into an electronic dry cabinet. Does anyone have any other advice / suggestions?
    4. Canister Settings / Workflows – Are there any Hedge / Canister users out there that can suggest their settings, folder structures and workflows to me?
    5. Resources / Online – Other than this forum, are there any other places where a LTO / IT n00b can get a better foundation on a small one man shop archival process (where LTO’s are concerned) as hinted in the forward, I am either getting white sheet data which is above my head or YouTube info which is often more of an “explanation” of what LTO is and less of a “how to make successful backups” and “future-proofing” your data… I see much of this forum of late is centered around “how to migrate or restore BRU archives” this is exactly the type of thing I would like to avoid if possible and this also is what informed me to stick with a simple interface using LTFS software like Canister.

      6. n00bie pitfalls – Anything for me to watch out for which as a new user I may overlook which can come back to haunt me on my archival journey?

    Thanks in advance for what I am sure will be some very informative insight!

    Cheers!


    * I was completely unable to write multiple sessions using my MBP 16″ M1 MAX (64GB) workstation which shipped with Monterey. It would write one session (but took for ever to clean-up) but would not write a new session nor was I able to retrieve the backup. Yesterday a new version of Canister was released and so far things seem to be working. (fingers crossed)

    ** Even though I had (maybe still have) issues getting up and running I have to say that the customer support at Hedge was fantastic and from what I was made to understand my issue has been a unique one for them (other users not having this issue at all).. I had personal ZOOMs with the developers who would “drive” my computer remotely in an effort to problem solve the issue and while I know there are more robust options out there, as mentioned before it is just me with one drive and once my backlog of past projects are backed up I will simply be backing up new footage incrementally as I shoot it. The other software is a bit prohibitively expensive for me to justify the costs at this point.

  • David Fox

    January 10, 2022 at 9:57 am

    Hi Shawn,

    Just wanted to respond regards this statement from your post..

    I was told that the drive you are writing from should be at least as quick as the write speed of the mTape LTO 8 TB3 drive (which I believe is 360Mb), however finding large capacity drives here that can also write that quick is a challenge and would be very costly.

    My understanding here (I work for another software vendor Archiware) is that LTO drives are able to speed match the speed at which the tape runs through the drive, with the incoming data rate. So your LTO 8 drive will be spinning at its ‘top speed’ when date is arriving at 360Mb/sec, but as the incoming data-rate slows, the drive will be able to slow down to match. There is however, a lower lower limit, at which point the tape speed can no longer match the data speed. At this point, the drive will stop the tape, fill up its internal buffer with data, and then run the tape again when there’s enough data to write some blocks to the tape. If data continues to arrive at a slow pace, the drive will have to repeatedly stop, fill buffer, start. This is mechanically bad for both drive and tape. Sometimes referred to the ‘shoe shine’ effect.

    All this speed matching and shoe-shining can be heard when listening to the drives noises.

    The lowest speed at which your LTO drive will be able to accept data will be documented somewhere for the specific drive being used in the mTape enclosure. From my experience, it can vary a little between manufacturers.

    Also note that the quoted 360MB/sec speed is for full height LTO drives, I think the drive used in an mTape TB enclosure will be half height, and max speed will be 300MB/sec.

    So… your disks don’t need to be super-fast to write to LTO, just no too slow.

    This is all based on knowledge of drives. If Canister has it’s own restrictions on data-rates. Then i’m not aware of them.

    Best,

    David Fox

  • shawn convey

    January 11, 2022 at 9:27 am

    So your LTO 8 drive will be spinning at its ‘top speed’ when date is arriving at 360Mb/sec, but as the incoming data-rate slows, the drive will be able to slow down to match. There is however, a lower lower limit, at which point the tape speed can no longer match the data speed. At this point, the drive will stop the tape, fill up its internal buffer with data, and then run the tape again when there’s enough data to write some blocks to the tape.

    That is great to know David and much appreciated information which makes sense now that you spell it out for me 🙂 — You were also correct in the fact that mTape LTO 8 does have a max at 300Mb/sec and as such I suppose that the 250Mb/sec my Oyen Digital Novus 12TB External USB-C (3.1,Gen2) 7200RPM Hard Drive has wouldn’t slow down the process all that much.

    NEW QUESTION
    Bad Session Write — I have made several successful large transfers to my LTO tape however the folder that is currently being backed up is acting problematic (VERY SLOW transfer and verify speeds taking 100x longer than it should with the drive spinning up and down often) and I suspect that there is a good chance that this write session will not verify successfully in the end AS SUCH — What happens to the remainder of the data that is on the tape? Do I need to reformat the tape and start from scratch OR can I somehow delete that last block of data and continue?

  • David Fox

    January 11, 2022 at 9:35 am

    Glad that was useful Shawn. Regards you later point, one thing to bear in mind when writing files to tape, is the file sizes that you’re reading/writing and how they influence speed.

    There’s a big different between asking your disk to read thousands of small files, totalling (let’s say) 1TB, versus a small number of large files that have the same total size. Even if your source disk is relatively fast, the reading of thousands of small files will be much slower than reading a few large files. Each file being read has an overhead – it has to be located on the disk, it’s meta-data read, verification checksum calculated etc etc.

    Therefore, during the course of backup/archive of your folder, you’ll find the data-rate to tape will slow while reading many small files, and speed up when reading large files. There’s not a lot that you can do to overcome this, just the way of things.

  • shawn convey

    January 12, 2022 at 3:34 am

    Therefore, during the course of backup/archive of your folder, you’ll find the data-rate to tape will slow while reading many small files, and speed up when reading large files. There’s not a lot that you can do to overcome this, just the way of things.

    OK that is great to know… I still think something was acting funny as 5.5hrs for 9.5GB (not TB) of data transfer / verification seems extremely excessive. But I have the developer looking into this as well…

    Can you point me in a direction to find practical info such as this (small files of same data total will take longer to write / verify, how much space should be left on a LTO tape as buffer, Issues to watch out for, LTO workflows etc….) online? I am happy to do my own research on all of this and don’t mean to single out a forum or person on a forum individually for all these questions but I have been coming up short finding a location (or locations) that have non IT, end-user practical info on the basic in’s and outs of LTO archiving…

    Also as you work for a software developer does Archiware have any plans on releasing a lower cost single user version of their software? The investment in a a LTO system (hardware and software) is very large for a single user / small shop and while I understand that hardware is regulated or controlled by IBM and they set the price I probably still would have justified the cost long ago IF the software wasn’t similarly expensive… It seems a shame that there are only one or two software options for the “little guy/gal” and the film / video field is becoming far more populated with single owner shops who have huge amounts of data that needs archiving… Anyway, I guess I am saying there seems there may be a growing demographic that could be served by more lightweight / low-cost options for software. 🙂

  • David Fox

    January 12, 2022 at 9:59 am

    Hello again!

    Yes the 5.5hrs to write 9.5GB sounds like a problem somewhere. That’s far too slow.

    I honestly don’t know of a good end-user targeted resource for info on LTO tape. Maybe a gap in the market. Archiware have a YouTube channel (https://www.youtube.com/c/archiwarep5) where I post explainer/training/demo videos, so maybe an opportunity for one explaining what we’ve been discussing here!

    Our lowest cost for Archive to LTO product is USD 1,680 which is intended for single users such as yourself. This is already a big reduction in price from our workgroup targeted products. We’re not just throwing a folder onto LTFS but maintain a media index of everything written to all tapes, a database accessible via a browser.

    Anyway, not wanting to get too sales-ey on these forums. Hope you’re able to resolve the speed issues. Seems to me that it should be fixable, you just need to figure out where the slowness is coming from.

  • Neil Sadwelkar

    January 13, 2022 at 5:14 am

    I routinely write tapes, like several times a week, using an Intel Mac mini or Intel MacBook Pro as host computer. I have not yet written a tape using any Apple M1 system as host computer.

    I use a standalone tape drives (I have 5) connected to a SAS card inside a Thunderbolt – PCIe box (I have 3 of these). And for software, it used to be Bru PE for a decade but now it’s Yoyotta and Canister.

    LTO write speed matches the read speed of the hard drive you’re reading from. I haven’t come across a lower limit, and I often have to backup to LTO-8, from 2.5″ portable drives that read at 60 MB/sec when full. I’ve even seen speeds as low as 40 MB/sec.

    Writing to LTO takes longer with file sequences like .dpx, .ari, .arx, .tga etc. meaning 100 GB consisting of 14,000 files, will take longer than 100 GB consisting of 30 files. Longer means about 1.5-2 times as long sometimes. Nowhere near 5 hrs for 9 GB. So that’s definitely an anomaly.

    Do you have access to an Intel Mac where to try out your entire setup? There is a likelihood of a driver issue with Apple M1. Your mTape TB3 LTO drive is actually an LTO drive connected to a SAS card attached to a TB3-PCIe converter all inside the same case. So its not vastly different from my setup where these two things are in separate cases.

    I checked on the mLogic site and the drivers link links to the Atto drivers page so it looks like your mTape has an Atto 6GB SAS card inside it, same as mine.

    The Help Centre at mLogic site also has links to Yoyotta 10 day demo. You could download and try that.

    I’m sure the guys at Hedge with do all they can to try and fix the Canister issue. But I don’t think the problem is with Canister, in your setup. It either to do with SAS and LTO drivers for your M1 host computer, and/or errors on the drive with data. Have you tried doing a verify and repair on your source drives?

    And one more thing. Are your source drives all HFS+? I’ve found some funkiness with ExFAT or NTFS drives sometimes.

  • Neil Sadwelkar

    January 13, 2022 at 5:30 am

    Shawn,

    About the other things you’ve mentioned in your post.

    You definitely need some cataloguing software to be able to browse, at one place, the contents of your drives, and to be able to store an offline record of the contents of the LTOs you are writing.

    Take a look at DiskCatalogMaker or NeoFinder.

    Writing many small sessions or 1 TB or fewer long sessions of 11 TB to fill out an entire LTO-8 tape. No difference, really. No gain or loss in doing either. But if you can choose between the two, I’d go with few large sessions than many small sessions.

    Doing double backups. Yes there are dual drives, and software that can write to two tapes simultaneously. But if you don’t have access to either, then writing it all over again is the only way. Hence, fewer long sessions make this less painful.

    Resources online. I’ve tried to find them too, but there don’t seem to be any except this forum. I’ve had to discover many things by trial and error. But I’ve been blessed with a continuous amount of LTO backup work to do, with no real time restraint, so I’ve had the time to try things out. I’ve bought and tested many software along the way.

    In the pandemic, I’ve had to to a fair bit of this ‘remote ops’, not for LTO, but for editing, motion graphics, VFX systems and such like. It’s not the same as actually working on a machine, but at least it’s safer.

    If it helps, I can have one of my team members (or me) operate the LTO backup for you remotely. All you have to do is to connect and disconnect drives and remove and insert tapes when needed, we can take care of the actual operations. Any troubleshooting we will do remotely. And it will all happen in front of you on your machine.

  • shawn convey

    January 13, 2022 at 5:34 am

    LTO write speed matches the read speed of the hard drive you’re reading from. I haven’t come across a lower limit, and I often have to backup to LTO-8, from 2.5″ portable drives that read at 60 MB/sec when full. I’ve even seen speeds as low as 40 MB/sec.

    Writing to LTO takes longer with file sequences like .dpx, .ari, .arx, .tga etc. meaning 100 GB consisting of 14,000 files, will take longer than 100 GB consisting of 30 files. Longer means about 1.5-2 times as long sometimes. Nowhere near 5 hrs for 9 GB. So that’s definitely an anomaly.

    Thanks for the insight @Neil!

    – I don’t have an intel mac at the moment unfortunately.
    – I will have to double check how all of my source drives are formatted but normally it is HSF+, I did have some ExFat / NTFS drives for shuttling between PC and Mac but I don’t believe I tried to use any of those as source drives in this scenario.

    But I don’t think the problem is with Canister, in your setup. It either to do with SAS and LTO drivers for your M1 host computer, and/or errors on the drive with data. Have you tried doing a verify and repair on your source drives?

    Right now the team at Canister is communicating with IBM trying to figure it out as well..
    In the end it looks like I have a complete LTO 8 tape which which had about 8-9 sessions written to it using different source drives, and connection configurations write and verify. With the exception of that ULTRA-SLOW 9.5GB back up session everything else seemed to go smoothly with varying write / verify speeds depending on drive speed and connectivity. I did notice a much slower transfer when I had the mTape TB3 plugged into my OWC m.2 raid unit which was then plugged into the MBP, which makes sense I suppose, but even that session wasn’t as slow as the 9.5GB write.

  • shawn convey

    January 13, 2022 at 2:22 pm

    @neilsadwelkar

    Thanks again for your additional thoughts ( I think I posted my response to your first post at the same time your last post came through 🙂 )

    All of this info is very helpful but I do have a few follow up questions:

    1. Cataloging software — I checked out the 2 you mentioned and I didn’t see any specific LTFS / LTO support. Would I simply drag the mounted LTFS volume on my HD to either of these software choices for them to make the catalog or can I import a manifest or is there a better way to connect through Canister?

    2. Remote Ops — that is very interesting proposition and not one I considered as possible… I would like to figure out a good workflow for my needs moving forward as once I am done archiving the past 10 years (which as a one man band isn’t nearly as bad as even a small production house) I will simply be archiving shoot data as it comes so it should all be very manageable moving forward… that is once I work out the workflow and the kinks. But with all that said knowing that your remote services even exist gives me some more piece of mind so thank you for mentioning that!

Viewing 1 - 10 of 11 posts

Log in to reply.

We use anonymous cookies to give you the best experience we can.
Our Privacy policy | GDPR Policy