PreRoll Post – abort verify and release hard drive?

[Martin Greenwood]

Here at YoYotta we have also had customer requests to delay verification. It’s not ideal, but the situation does arise in the real world.
So now in our YoYottaID LTFS software once the copy completes you can unmount the source media and the verify will continue. Hopefully the source media won’t be erased until the verify is completed !
Also you can abort the verify and then start it again later without the source drive.
You can reverify against the original camera checksums at any time in the future using a free version of our software.

Martin Greenwood.

CTO

yoyotta.com

[Neil Sadwelkar]

I began an archive in PreRoll Post and after the copying was done, and verify was under way, I hit cancel. Got a warning that the archive would not be complete, but I continued. PreRoll Post went into beach ball mode for about 10 mins. I gave up and force quit it.

Now, the tape shows up on the desktop, and all contents seem to be fine. I checked only folder sizes, of course. But there seems to be no way for getting this LTO tape into the PreRoll Post database. So I can’t verify it, nor use PreRoll Post for retrieve. Maybe I need to dig deeper to find a way. The manual doesn’t seem to have any info on this unusual mode I forced the software into.

———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India

[Dan Montgomery]

PreRollPost is an LTFS application. In LTFS, after copies complete the tape is recued to read each file (retrieve them) to compare MD5s to those of the source files. If you have a lot of files this step obviously can take some time to complete.

Once that’s done, the index partition on the tape is updated to reflect the data that’s been written to the tape. During this timeframe PRP also updates it’s SQLite database file with the new information. Again, this step may take some time if there are tens of thousands of files involved. Once all these steps are complete, you’ll get a ‘Done’ notification (or error information if any checksum problems are encountered, etc.).

So, if you completed the copy but quit the app and ejected the tape during the next phase your data should be intact. It’s just the indices and second checksum that didn’t get a chance to finish.

Depending on the severity of tape damage you’ll be prompted to do different recovery steps when the tape is next mounted. PRP will first attempt to repair the tape–getting the index on the tape to match what’s found in the data partition. Sometimes you may have to use implicit LTFS commands which are provided to you in a simple ‘Tape Recovery Utility’ interface under PRP’s application menu (above Quit). These recovery tools are designed to be used in sequence–usually the ‘Full Recovery’ is all that’s needed to get the tape’s partitions back into sync.

The next option is named Deep Recovery and it’s commonly needed when the tape is improperly ejected, power lost during the write process, etc. It should only be used if the Full Recovery is unsuccessful because this process can take hours to complete.

The last option is Read Only Mount. It’s the last resort and generally only needed in cases of actual physical damage to the tape or it’s memory, etc.

Any of these recovery options should never be interrupted once started. Otherwise you run the risk of the tape becoming permanently unmountable.

So, again, in your case it’s probably a simple matter of running the Full Recovery to get the tape’s index back in good shape.

To put this tape, or any other LTFS tape, into PreRollPost’s database: Go to Settings>Local Indices. Click on the lower left button to ‘Index Generic LTFS Tape’. This will copy the tape’s index information into PRP’s database and it will then be Retrievable/Searchable in that window. (To remove any tape or session from the retrieve view, select it and press Delete on your keyboard. To add it back, use the steps just described.)

Hope this helps. It’s generally not a good idea to stop an archive in progress if possible.

Offload with Confidence…

[Dan Montgomery]

Oh. And to verify the contents of the tape against the source files, you can retrieve them from the tape to a disk. Then run ShotSum to compare the two.

With ShotSum ($99) you can check any files or folders at any time. You can also find all the instances of the same files across several disks or network, etc. (regardless how they’re named).

ShotSum, like ShotPut Pro, has an option to save the MD5 checksum values to a text file with the files. If you do this, at any time in the future you can check the files integrity by just dropping this .md5 file into ShotSum (or ShotPut’s checksum utility) and the software will follow the embedded links to check the associated files against the md5 checksums of the originals. This of course has the advantage that you may move or copy the files and always be able to verify they’re exactly matching the originals.

Offload with Confidence…

[Neil Sadwelkar]

Thanks Dan. I’ll try that.

About the number of files, we use LTO-5 and LTO-6 to backup camera originals. So, if its from an Arri Alexa shooting ArriRaw, there are 1,440 files per minute (at 24fps). Each is 7 MB in size. So, for every TB there are 142,000 files. Plus sound files, xml files.
On an LTO-5 you can easily have 180,000 or more files, and on an LTO-6 over 320,000 files.

Sony F55/F65 or Red Raw produces much less files.

If one is backing up a Mac home folder containing address books, certain documents, iPhoto Library then too the backup quickly swells up to hundreds of thousands of files as many Mac document and data formats are packages. So, in PRP, just the ‘Prepare backup’ takes ages.

Which could be healthy, because I did my routine evening walk of 3km in the time Prepare backup got done with my RAID containing old iPhoto Libraries. This needs to be speeded up if possible.

———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India

[Dan Montgomery]

Who needs that iPhone Health app? lol.

Lots of files are simply going to take time. But there are a few things you can do. One is in the Settings>Video turn OFF the “Determine video type” feature. This setting creates thumbs for each known video file type, and also extracts the metadata for known types (timecode, frame rate, codec, etc.) so it takes processing time.

Alternatively, you can leave the ‘Determine type’ ON and set the advanced OPTIONS for it to ignore certain file extensions or small files, etc.

Also, on the Prepare Backup window, check the box to “Automatically begin backup after indexing is finished”. This eliminates the need for your presence to hit the Begin button once indexing is ready (so you can keep on walking).

PS: We routinely test with hundreds of thousands of files. However, there is no need to write an entire tape at once. You may add to it in several smaller sessions. After all, it’s not a WORM drive.

Offload with Confidence…

[Tim Jones]

And that’s a perfect example of what make’s BRU’s Anytime Verify different from any other tools out there. Because BRU records the 32bit CRC values with every 2K of data in the archive stream (far more dependable that a file-based checksum), you are able to fully verify and audit a BRU tape at any time and even on a completely different operating system.

So, even if the source disk IS erased before you run BRU’s verify, you can still perform the verify and audit of the tape’s contents. It’s always been this way with BRU – since 1985,

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Tim Jones]

Ugh – that’s one of the biggest foibles of LTFS – if something happens during the actual tape write, you can end up with a tape that can’t be recovered – meaning that all of the data on the tape is now lost.

With BRU-based applications, you can power off the tape drive during the write operation and everything that was written to the tape(s) up too that point can be fully recovered. No further recovery steps required.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Dan Montgomery]

Who said anything about losing data or an unmountable tape? Any data written to an LTFS tape is intact (just like the magical BRU software), it’s just the index that may need syncing.

Why do I always envision windmills whenever I see a post by Tim Jones? ;-\

Offload with Confidence…

[Tim Jones]

Dan, that’s a cheap shot (but I expect no less) and I’m sorry that you have a Don Quixote complex, but the proof is in the tasting of the pudding.

Insert and format an LTFS tape
Mount it
Start copying 300GB to it
Half way through, turn off the tape drive
Wait while Finder goes a bit wonky trying to warn you that the disk was improperly ejected
Turn the tape drive back on
Try to mount the LTFS volume
Ta Da – no windmill, just a failed tape.
Run “ltfsck –deep” and wait a few hours and maybe, just maybe, you’ll get access to some of the files written to the tape.

Insert a blank tape
Start a BRU backup of the same 300GB
Half way through, turn off the tape drive.
Restart tape drive
Insert that tape
Restore every file up to the point of the power off

Otherwise, if LTFS was so good why does it need the many recovery mechanisms that you describe?

Dan wrote: “The next option is named Deep Recovery and it’s commonly needed when the tape is improperly ejected, power lost during the write process, etc. It should only be used if the Full Recovery is unsuccessful because this process can take hours to complete. ”

and then: The last option is Read Only Mount. It’s the last resort and generally only needed in cases of actual physical damage to the tape or it’s memory, etc.

Any of these recovery options should never be interrupted once started. Otherwise you run the risk of the tape becoming permanently unmountable.

You’ll never find any such warnings and caveats with a BRU-based archive.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!