-
Large number of copy and verification errors in LTO using PreRoll Post
Hi All,
Following your responses to my questions in a previous thread, I decided to go with a Quantum LTO-7 drive (using LTO-6 tapes until the v7 media cost comes down) and PreRoll Post to back up my footage for a shoot in which we’re generating ~8TB a day of 4K ProRes. The footage is getting offloaded to a large RAID6 volume and, in theory at least, backed up to LTO-6 tapes.
However, we’ve been having a number of problems — first, we’re getting a large number of copy and verification errors in the PreRoll Post backup sessions. I’m new to LTO backup, so am confused. Is it normal to have copy and verification errors when writing to LTO? We’re talking in the range of 1-3 copy errors and 2-6 checksum errors per tape, consistently across now a half dozen attempts.
Second, when attempting spanned backups (e.x. ~4.5TB, so 2 tapes), PreRoll Post gets to the end of the first tape, reports the copy and checksum errors, and asks for another tape to complete the backup. However, I can’t get it to eject the first tape. The communication log in PRP says “resource busy, try ‘diskutil unmount,'” but diskutil unmount returns a ‘disk in use’ message and won’t unmount the tape. The only way out is to cancel the backup, restart PRP, and then unmount the tape, but of course at that point there’s no way to resume the backup. In each backup attempt we’ve formatted a fresh HP LTO-6 Ultrium tape in the PreRoll Post settings menu before starting the backup.
Any thoughts on why these issues might be occurring and what I can do to avoid them? I urgently need to find a solution, as the footage needs to be backed up ASAP…
Your help, as always, will be hugely appreciated.
Many thanks in advance,
Joe
My system:
Brand new Quantum LTO-7 drive in a rack mount enclosure, operating over SAS through a HighPoint RocketStor 6328. This is running into a 6-core late 2013 Mac Pro with 64GB RAM running OSX 10.11.4. We’re pulling data from a 12-bay rackmount RAID over 10GbE with read speeds in the 900MB/s range. -
16 Replies
-
Just to ensure that the tape drive and everything else in the system is functioning normally could you try doing a backup with something else? Like Bru-PE for instance. Its free for 30 days, and if a copy goes through without any error you can rule out your drive or system being the problem.
———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India -
Hi Joe,
Were you able to resolve your issues? I have had numerous errors with both YoYottaLTFS and PreRollPost and an mTape LTO-7 drive. I’ve run three tests, reformatting the same LTO-7 tape for each test. Is this ‘verboten’? Curious…Let us know what transpires. We’re all in the same boat. Good luck.
—
Tom Ervin -
Hi Tom,
Wish I had better news — I’ve been working through this problem for almost a month now without any conclusive answers. I’ve tried swapping out the SAS and Thunderbolt cables, the computer, the data source, and the software. I’ve downgraded and upgraded the firmware on the RocketStor and the LTO drive, and I’ve spent hours on the phone, email, and chat with the various tech support departments, and I still haven’t been able to resolve the issue — my backups only complete successfully about 40% of the time (out of dozens of attempts). At this point the folks I’ve been talking to seem to think it’s likely a hardware issue, but I can’t afford to buy an alternate controller card or drive to test.
So that’s my current status. Frustration level: high. I was warned that setting up an LTO system isn’t the easiest thing to do, but frankly this is getting ridiculous. I’ve spent thousands of dollars on this hardware/software and no one can suggest anything other than “must be one of the other vendors’ fault.”
Happy to fill you in on more things we’ve tried, and I’d be grateful to hear if you have any better luck.
Best,
Joe
-
I’ve been having the same problems and have tried several tapes and different files but constantly have error files yet don’t know what exactly the errors are. I’ve emailed Imagine Products and hope they will be able to help resolve this because it was quite an investment. I’m using mlogic LTO 6 and PreRoll Post.
Please keep us posted on any positive developments
-
Hi all,
Just a quick update on my situation — I’ve now been working with tech support at Imagine Products, YoYotta, HighPoint, and Quantum for over two months, and no one seems to be able to shed any light on why these errors continue to happen. I’ve forged ahead with backing up my data as much as possible, but am getting an approximately 40% error rate. Meaning 40% of my backups fail with file verification errors. In addition, I don’t seem to be able to reformat and reuse the tapes, so this is driving my cost through the roof.
Does anyone have any idea what else I can do to resolve this situation? This is a crazy outcome for an investment of over $5K in hardware and software.
For folks who have implemented successful LTO systems, what’s your setup and workflow?
Any input will be a huge help.
Thanks again for your thoughts.
-
I am having the exact same issue!
mLogic Thunderbolt LTO-7 drive
PreRollPost for Mac
Source drive is a G-SPEED Studio XLThe backups fail all the time. And you cannot span tapes (like PreRollPost claims you can). I called Imagine Products about the tape spanning issue and they told me to “stick to backing up 1 tape at a time”.
Even then, I’m getting tons of errors and failures. Rough going when it takes 10-12 hours per tape just to find out that you have to do it again.
I’ve been able to reformat tapes, but I worry that reformatting and retrying the backup over and over again could be compromising the integrity of the tape.
Had anyone made any progress with this issue?
Has anyone made any progress on this? -
Two different setups…
Joe Bender
Quantum LTO-7 rack mount, SAS HighPoint RocketStor 6328, 6-core late 2013 Mac Pro, 64GB RAM, OSX 10.11.4.
Source: 12-bay rackmount RAID over 10GbE, read speeds 900MB/s range.Jeff Lehman
mLogic Thunderbolt LTO-7 drive, PreRollPost for Mac
Source: G-SPEED Studio XLHardware spec totally different. Common software PreRoll Post among others.
I own PreRoll Post, as well as Bru-PE. Use Bru-PE almost always. PreRoll Post, tried ran into similar issues, gave up.
My setup
Tandberg LTO-6 connected via SAS to a Atto H680 SAS card inside a Sonnet PCIe expander.
Use MacBook Pro/Mac mini/MacBook Air and Bru-PE MacOSX is 10.10 and 10.9.Not one single backup failure in 4 years. Barring mains outages where even the UPS ran out.
Here are my thoughts.
Joe, I think your GigE card and/or SAS card is suspect. Is the SAS card recommended by the LTO drive maker? Have you tried backing up to LTO via some other drive not the NAS RAID.
Jeff, you too, Have you tried backing up to LTO via some other drive not the G-SPEED Studio XL.
And for both, have you tried backing up with Bru-PE? Its free for 30 days so you can try. If it backs up successfully, then your entire hardwares setup is doing just fine.
———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India -
Looking into BRU-PE. It does not use LTFS, correct?
It does look like a simplified version of PreRollPost, but I like the functionality of LTFS.
-
Sorry to hear you’re struggling with this. We have had a handful of folks running into it but have never been able to replicate the issue in-house which tends to say it’s hardware or data related.
That said, one recent report included the error log which showed that LTFS lost connection to the drive, thus resulting in cascading copy and verification failures. In that case it clearly was not an application issue; it was something in between the app and the LTO device (LTFS, OSXFuse, SAS/TB drivers, etc).
BTW, IBM just released another LTFS update about a month ago to “fix certain problems” with LTO-7 devices. So it’s a good idea to keep up to date with their drivers, especially if you’re constantly updating OSX.
One thing we can do is make the application more fault tolerant and we’re working on that. We already added a “reverify” feature that attempts to read and gather the checksum a second time when a comparison error is reported. Of course that doesn’t help if the ‘connection’ to the drive is lost.
Spanning does work in PRP, else we wouldn’t have put it in. But if you’re having issues it’s safer from a time investment to try to do just one tape at a time until you figure out what’s failing. What we’ve noticed in the 5+ years we’ve been working with LTO and Sony ODA is that people tend to throw all manner of files at it, and some just aren’t LTFS happy. Our original assumptions that people would only archive field footage or finished products proved false.
Offload with Confidence…
-
Update:
I changed 2 settings and have since gone 4/4 successful archives.
1: Default Link Choice — I know this should resolve itself within the prep process, but I decided to change it from “Cancel” to “Remove links from backup”
2: Turned off ALL options under the “Video” tab. Since 99% of the files I’m offloading are video files, I thought maybe the creation of thousands of proxy files and saving those to my computer was causing slow-down/communication interruptions. And I don’t really want/need them anyway.
Since these 2 changes, I have had no issues, but I will report back if the problem pops up again.
While I have you here, Dan – in regards to spanning: do I need to format all tapes BEFORE starting the multi-tape offload? And are there certain settings that need to be enabled to ensure smooth spanning? e.g. Auto Eject
Thanks.
Jeff
Reply to this Discussion! Login or Sign Up