LTO file size mismatch in Finder

[Tom Goldberg]

Every LTFS implementation (except the open-sourced free versions) support tape spanning.

Tom Goldberg
TGCS
30201 Rainbow Hill Rd.
Evergreen, CO 80439
mailto:tomgoldberg@gmail.com
https://tomgoldberg.net

[Tim Jones]

There’s a difference between spanning tapes and breaking the data stream up to fit on tapes. The key being that the LTFS platforms that support automatic multi-tape operations do the data splitting for you.

Since you are concerned about BRU’s checksumming wasting tape, this type of data splitting also wastes tape since with few exceptions, the data is split into segments based on the “approximate” expected storage size of the tape used. It doesn’t (and really can’t) take into account any compression that occurs while writing to the tape.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Tim Jones]

Apologies to readers as this gets a bit long winded.

[Tom Goldberg] “I stand by my statement that the only way to know for sure is a full restore and compare. I did not mean to imply that has to be done manually, automated means such as your Autoscan certainly meets that criteria. And I don’t dismiss checksum comparison as a powerful way to verify without a full restore – only that it should be done against the source data set. Copied data IS more likely to have bit errors going between hard drives than going onto tape.
“
Tom, that’s simply not valid and I suspect that you make that statement from never having tested BRU’s resiliency in this situation. Because BRU’s checksum is calculated as the data is read from the disk, it is as valid as the original data. If the original data is bad on the disk, it will be bad on the tape, but even a bit-by-bit comparison (which BRU also supports) or restoring that data to an alternate location and comparing the files will report the compared bad data as good – so your example is not a good one to promote or refute the reliability of any of these methods. And our AUTOSCAN pass is not comparing the data on the tape to the source data – it’s rereading each block, recalculating the CRC and then comparing the calculated CRC to the CRC in the header of the block.

On the other hand, this does mean that BRU’s checksum is a completely reliable mechanism for checking the validity of the data in the archive container. And, with BRU’s CRC mechanism and our Any Time Verify, you can verify AND audit a tape 6 months from now or even 10 years from now with no knowledge of the original data source, filesystem or platform, unlike what you state above about needing the original source data set. Everyone else uses MD5 checksums because it’s the only way for them to provide ANY sort of checksum; they can’t modify the tar, MTF, or LTFS formats to add stream-based checksums because it would break the interchange capability.

Comparing our in-stream CRC algorithm against an external MD5 (or an even stronger SHA1) file-based checksum is not even close. BRU calculates the CRC in the stream, not as a separate process or pass. Additionally, we have shown that the BRU I/O engine running on an OS X system with a high-performance RAID source drive is still not topping out BRU’s performance capability. We have sustained performance of over 3.2GB/sec using BRU to create archives of the data on the array (faster on other platforms where PCIe-3 is available). This is far faster than any modern tape drive can handle, so the overhead relating to speed is nonexistent using BRU’s methodology.

Also, BRU has no issues archiving data sets with extremely deep folder structures, folders containing 100,000’s of files, filenames with special characters or international languages, so the user does not need to modify their workflow to take those things into consideration. We have one customer that recently left another LTFS solution because rather than rework their international workflow to fit LTFS, they were using ZIP to create archive containers using generic names so that the data could be backed up to the LTFS format used by the other product. Then they realized that they couldn’t restore a single file without first restoring the ZIP container. They bought a lot of BRU licenses.

Finally, the ~18% overhead that BRU generates is not simply the CRC (that’s literally 4 bytes – 32bits per block), but also includes all of the other filesystem and low-level metadata that we save with relation to the file. Things that LTFS simply throws away.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Tom Goldberg]

Well Tim, we can agree at least on one thing, that you are long winded 😉

Actually, I don’t take issue with any of the points you make in your post above.

I was in particular making the same point as you:
[Tim Jones] “If the original data is bad on the disk, it will be bad on the tape”

And then I was taking that point further – We saw many users doing extensive verification of copies of source data or even copies of copies of source data, and that is IMO a waste of time as the disk-to-disk copies are more likely to have errors than disk-to-tape.

I don’t doubt that BRU checksums are better than SHA1 or MD-5, just that they are bigger and slower. I note that no one else seems to think it is worth the overhead. And by no one else, I include IBM, HP, Quantum, Oracle, Spectra, and so on.

Tom Goldberg
TGCS
30201 Rainbow Hill Rd.
Evergreen, CO 80439
mailto:tomgoldberg@gmail.com
https://tomgoldberg.net

[Tim Jones]

[Tom Goldberg] “I don’t doubt that BRU checksums are better than SHA1 or MD-5, just that they are bigger and slower. I note that no one else seems to think it is worth the overhead. And by no one else, I include IBM, HP, Quantum, Oracle, Spectra, and so on.
“

Then you either did not fully read what I posted or are not understanding it – BRU’s in-stream CRC method is neither slower nor larger. In fact, it’s faster than MD5 and is in-stream rather than a sidecar to the process. An MD5 sidecar “file” is usually around 57 bytes each and requires a separate file entry in the data written (if they are included in the archive at all) while BRU’s 32bit CRC is 4 bytes and in-stream. You can’t “lose” BRU’s checksums, but you could lose the MD5 sidecar files that so many are promoting as “good enough”. As for speed, I’d love to see any tar or LTFS-based solution try to match BRU’s native CRC-inclusive I/O speeds while including all filesystem metadata, extended attribute information, ACLs, long paths, and full international character set support – even without including the MD5 sidecar files.

So far as mentioning hardware vendors, these companies aren’t responsible for what the software does with their devices. You attempt to imply that the fact that they don’t include such a solution is indicative that it’s not needed – two important points with regards to that:

• They don’t think it’s worth it because to do so would imply that their solutions are potentially prone to errors.
• Those that include software with their hardware provide software tools that are based on archive container formats that can’t provide in-stream checksums.

I can’t comprehend why you are having a problem following this point. They don’t offer or recommend it because they can’t. TAR, MTF (Microsoft tape format), NDMP, nor LTFS have a way to include the checksum mechanism within their formats without breaking the existing formats.

BRU has proven for more than 29 years to be as fast or faster, more reliable, more compatible and with broader platform support than any other solution (except maybe raw tar in that last point).

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters!

[Goce Shamakoski]

The reason I don’t let BRU span tapes is because the catalog is not as robust as I would like it to be. At the moment I use a cataloging software that allows me to see detailed information about a file or folder ( Thumbnails of videos and pictures, metadata, comments, creation date and modified, original XSAN path, tape number, column view list view icons…) Having to use the little arrow to collapse the folders is real pain when I have 30 sub folders.

Another reason is I have head people saying not good idea to split video files.