[Doug Weiner]

OK, the problem itself turns out to be the HH LTO-8 drive itself. (Although the problem is still an annoying Gremlin. Drive passes self diagnostics as does the library, but Bru is spitting an error upon the hardware scan. I still want to transition off of Bru and to ArGest and would love help doing that.

[Doug Weiner]

My crashes were with larger files, (Video and it is all I am backing up with) so I am not convinced that is where their bug is. Some memory issue in Bru is my guess going back to old code. I am using the previous iteration of Bru (Bru Server 2.0.5). Maybe its two different errors, maybe not.

Veering into an LTO and software rant…..
Regardless, I am not here to solve Bru’s Issues, as they have their own timeline of things and seem content with how it all works. That might sound like a knock against Bru, I guess it kind of is but its also kind of isn’t. It is more of an annoyance and combined with the fact that there simply is not a more reliable, affordable, trustworthy LTO program out there. As an engineer friend of mine says and Tim at Tolis shares the sentiment “Its not the backup that is important, but the restore.” And Bru has never let me down. What I like about Bru, from my understanding of all their white papers, is basically Bru is writing a checksum at every block – making it much more reliable than LTFS. Now is this critical? Bru over engineered? I really don’t know, but I also don’t want to find out in 15 years that I have a problem. This is an archive. With only two tape copies in existence. I need to be able to restore 15 years from now reliably.

Another thing about Bru that I like is that Bru is easily searchable to restore individual files. My archiving needs are simpler than most big companies, I imagine, but I think they are similar to creative project based needs like people at creative cow encounter. ( I don’t do incremental updates so tracking tapes are easier in my workflow.) I perform an “archive” when a folder on my NAS fills a certain size, 12TBs for LTO 8, then make a Bru Archive (names Pass1) to LTO 8, run it, verify it, then duplicate the “archive” in Bru (re-name that one Pass2) and run a second archive. Then delete the original media on the NAS. LTO tapes are already barcoded uniquely, but I also label each tape box as well with Tape’s barcode number, archive name, date, the other tape numbers in the archive, and the Bru Hex name. This way if a tape is misfiled, I can figure out what archive it belongs to. I then put multiple tapes of an Archive into one plastic box ( cheap plastic ammo cases that hold about 13 tapes – silly I know but figure they might be water tight enough in case of “Sharknado”). If the Ammo case has room, I put multiple archives in the case. (labeling the outside of the ammo case as well so I can quickly locate an archive.) Bottom line, my archive procedure seems to scale fine for now. I have about 440 Tapes done over 4 years (220 are the 2nd copies and stored elsewhere) and they sit on 3 shelves. I imaging when I get to over 1000 tapes per copy, this will become another kind of scaling problem.

Anyway, I have explored YoYatta and P5, and both are better and worse than Bru. Yoyotta seems to copy near 300MBs, however its doing LTFS (non starter) and its database interface (Tape tracking) is not at all clear to me. P5 is a real alternative, but basically is going to cost me 10K to get it to do what I want. Pros: P5 is very well supported, updated to work with latter macOS, widely used, and I think is doing block checksums like Bru. Also its interface lets you just grow an Archive so you don’t have to track “completed’ archives. Just search for a file, locate the tape, and restore from that one tape. No need to locate and load all the tapes in that archive. BEST PRO: You can make 2 tapes that are clones of each other at the same time meaning the tapes are completely interchangeable. No tracking of Pass1 and Pass2 – just put one of the two tapes in a restore.. and poof it works. (FYI, I haven’t experienced if this actually works, only their videos says it works. Also I don’t know if P5 will verify both tapes at the same time. ) Cons: That powerful interface is also not intuitive, but this can be learned. The software is more expensive $4600 (Library version) versus $600. And to get the clone feature, I need to buy an extra LTO 8 drive ~ $5500. Oh frustratingly, P5 only writes at 150MBs as well. *** Note this is not due to my hardware specifically because in my testing of YoYatta on this exact same hardware, it writes at ~300MBs. I think its an issue of writing LTFS that is not doing checksums and Bru/P5 doing them. Or it could be both P5 and Bru were written a long time ago and YoYatta much more recently. Hence why I am seeking a full LTO 8 forum.

Bottom line, there is no happy LTO answer. I just want Bru to be a little less orphaned (easy and clear support) and clear and reproducible way to get Bru to write quicker. But Bru is affordable and works.

Rant over.

Answer to your previous questions:
1) 10GB network. With Aja, I can get reliably 350MBs reads across the network.
2) I am still using El Cap. I will at some point try High Sierra and see. (Whole bunch of Install issues with BRU Server and High Sierra, but lets let that sleeping dog lie.
3) What file protocol were you using on the Mac to connect to shares, AFP? SMB? CIFS?

Thanks
Doug

[Doug Weiner]

That’s good. The old problem was Bru would complete the back up, then “crash” on verify. However the real impact was that the “archive” was never written to the Bru database. Maybe my issue was I never let the full Verify continue. (This problem was years old so I don’t recall all my tests.) For you, as long as it is on Tape, Verified, and you can search and restore… awesome. Thanks for letting us know.

Now I need to find an LTO specialist group. I have LTO8 and would love to get the write speeds up closer to 300MBs. Currently I am getting 156MBs. (13.289 TB are written in about 24.8 Hours.) Verify is at about 300MBs (the same 13.389TB is verified in 12.9 Hours). (However this is not an accurate test of speed because I am not writing to Disk or network share, but at least I know the LTO8 can go that fast). If I could just get 225 MBs write speed….. The write job could be done in a little over 17 hours, then combine that with a verify, I could get a backup done in two human days instead of 3. (Run backup 12 Hours + 5, quit, Manual Verify 12 Hours.)

[Doug Weiner]

Dean,

Please check in the restore pane. Are the archives that have crashed available to be searched? Can you select a few files and restore from them? Please let use know the results.

Doug

[Doug Weiner]

I have had this same problem with Bru Server 2.0.4. so I don’t know if this is your exact problem, but it sounds exactly the same. Bru Server would crash on verify almost every time if you did a verify immediately after the write. (The problem I gleaned from Tolis support was this was some bug in the ‘database’ portion of the program.) Good news is there is a work around that reliably works. Try it for Bru PE.

1) Uncheck “Veirfy” in the back up job.
2) Run the back up.
3) After completion, Quit Bru.
4) Reopen Bru.
5) Run a Verify – manually – it should be at the bottom of the Verify list. However double check the Archive ID#. I use the Data Tools & Library Manager & Tapes to cross reference it all. (Sorry I forget my sequence to locate that data.). It’s not intuitive but all the data is there, you just have to cross reference.

** The key here is to quit out of Bru and re-start the program. This does something to avoid the problem.

I think the crash also loses the “archive” in the database – like it’s never saved. So there is no simple way to verify. Maybe you can reload the tape, but I think its best simply to re-run the job as described above. I suggest you only back up a few files and test, so you don’t waste 20 hours…. like I was doing for a few months.

[Doug Weiner]

Did you ever figure this out?

[Doug Weiner]

1) Sounds like a dirty head. Get a new cleaning tape. They don’t last nearly as long as I have heard. I have experienced 10 cleanings or so and then was done. Try that first.

2) The tape heads can be checked out, cleaned and resurfaced if need be. There are two people I kind of remember who service them. One guy in the LA area (got his name off of COW – but he was on vacation at the time so I didn’t use him) and another in Chicago https://www.mwtss.com. I used the latter and they did good work. They just cleaned and tested my drive. Sent back a bunch of reports.

I bet your LTO 6 drive is not bad yet. Those things are meant to take a serious, serious beating.

Doug

[Doug Weiner]

I have 2 Storinators 30 sitting right next to me in my office. They work great but they are also not exactly plug and play. I picked the 30 Drive units as they were quiet.

PROS
1) Quiet. 60 drives on right now and by far quietest set up possible. Not silent, but very reasonable.
2) Most affordable 30 slot server out there.
3) Affordably Expandable. I added an additional two port external LSI mini sas card in the server and another Storinator 30 with an Intel Expander card and got another 30 drive unit. I also have another 4 drive slots off the other external mini-sas port.
4) The Storinators 30’s are a small footprint.
5) Customizable (this is a Pro and Con.) If you are DIY or hacker type, its great.
6) You can pick and choose with 10gig card you want in the server.
7) Has been very reliable.
8) RaidZ2 with 10 drives over direct attach 10 Gig gets me ~500 MBs
9) I have done some crazy custom Zpool stuff with this set-up and NAS4Free. (I have Zpool shuttles. Meaning I have one small server at one office location recording to a Raid-Z1 (4 drives – 6TB avail) and when it fills up in about 3 weeks, I pull the bare drives, transport them to another office and then pop them into the main NAS4free setup and copy the files to the 60 drive set-up.) (If you are doing the math, yes, it fills up, so then I archive all the 4K material to LTO every month and leave the HD material behind. But I digress.)

CONS
1) You got to do it yourself. Have to know FreeNAS or NAS4free.
2) Trouble shooting drives is a completely manual process. On the Storinator 30’s, there are no indicator lights for drives. You got to make ZFS pools, record serial numbers and and kind of know what slots are where.
3) Rocket 750 cards require a specialize firmware from Storinator. People say it works but it scares me, so I opted for two LSI 9201 cards instead.
4) No real tech support or even a forum to help. I had a great FreeNAS guy help trouble shoot and help set me up.
5) Direct cabling in the machines are excellent for speed but a weak spot for repairability. I personally built my first unit and now one mini-sas port on one fan out cable (there are 6 in there?) is not working. (I am sure I didn’t install the cables so sweetly way back.) However, now I have to disassemble most of the server to repair.

UNKNOWN by me
1) I have not really set this up with multiple editing clients over a network, so I have direct data how that is handled. However, my guru friends has run FreeNAS in an editing environment, so I know it has been done. However maybe there are tweaks I don’t know.

OTHER THOUGHTS
1) 3 years ago, a 60 slot server with 10 gig and without drives was at least $20,000. I put mine together for about…. $4000. And its relatively very quiet. However, prices have come down a lot so I would compare more closely now.

2) I ghettoed the network. The server is a Supermicro and has built in two 1g Intel ports and I added a 2 port 10 Gig SFP+ (Myricom bought off eBay). I direct connected two SFP+ cables to two client computers and a third was connected via 1g although I have a little router on the 1g connection.) I never really hit the server hard with simultaneous connections, however it did seem to work the few times I did. I mostly now use the 1g connection for editing and only use the 10 Gig connection for LTO or other copying.

3) I have digressed a lot to my network in this post, and I am not sure its entirely relevant in comparing Storinator to Qnap and the like. But it is relevant in real world use so I have added it here.

4) When things go a little sideways with a drive, I really do miss the indicator lights on the front of the machine because trying to identify a failing drive without that on a NAS4Free/FreeNAS system is tricky and time consuming. You have to run iops terminal commands and watch for the drive to pop off line to find which one is slowing down. Then you have to trace which slot that is which is not a trivial matter. I have had only 1 or two drives go down in 4 years, so its not a common occurrence. Now that the Rocket 750 cards have proven themselves, I would go with them and the indicator lights they provide.

[Doug Weiner]

They are decent but lack some information. Here is a recent email I got back regarding some questions.

Q: Does YoYotta back up alias’s? (not the linked the file but just the alias. I personally use those as place holders for files that have been completely archived.)
1a) I see the alias listed in the PDF, but not in YoYotta database so I don’t know how these can be restored.

A:) Only media and metadata files are stored in the database. So you would restore either the whole tape or a folder.
https://yoyotta.com/help/restoreLTFS.html
Links are backed up. Unless Copy linked files is turned on in Preferences.
https://yoyotta.com/help/preferences.html

Q: I did a restore and it copies one file fast. Is there a way to make sure YoYotta restores sequentially from LTO tape? So that when I restore lets say 5 TBs from tape at once, YoYotta is not FF and REW the tape constantly to do the restore?

A:) When restoring folders are sorted into order, so the restore will be quick. We expect files inside folders to have been written in order.

[Doug Weiner]

Thanks. I am keeping the 2k and 4k masters. We are archiving them to LTO until there is more demand for them as stock footage or as drives become large enough that we do not need 60 x 4 TB drives but can go with 30 x 8 TB drives.