-
Retrieving files from LTFS tape — how to sort by startblock or copy in correct order?
All right, here’s a new question for anybody familiar with LTFS:
What’s the best process for retrieving files from an LTFS tape? Say I want to dump an entire 2.5 TB LTO-6 tape to a hard drive, what would be the best procedure?
If just select all the files in Windows Explorer and copy them over to a local hard drive, files seem to copy out of order — meaning the tape will seek to a location, a file will copy, then the tape will seek to another location, etc.
I read this from the HP LTFS “best practices” PDF:
Optimizing file retrieval order
When browsing the contents of a directory, the files are typically presented sorted by some criteria—for example filenames in alphabetical order,
or date of creation. Whilst this is logical for human interaction, it may lead to delays in retrieving the contents from tape because the perceived
order may not match the order in which they are stored in the LTFS volume. And because tape is essentially a linear sequential access device,
retrieving files in the “wrong” order may take many times longer than if they were accessed in their stored order. To address this, it is necessary
to peek inside the LTFS index and pre-sort the files into the “right” order, rather than allowing the operating system to select its own order.
This can be accomplished by looking at the reserved extended attribute ltfs.startblock which records, in the XML index, the logical block
number on tape at which the specified file starts. If the list of files to be retrieved is then sorted by logical block number, the operation can take
place in a more efficient and timely manner. An alternative is to view/process the XML index file, if that has been captured as described above in
“Saving and viewing index files.”Implementing on Windows
Although HPE StoreOpen for Windows now includes the ltfsattr utility, which can read and report the ltfs.startblock value, the current
implementation forces an open and read from a file in order to read the file’s extended attributes. This means that the tape must be positioned
to each file, which will actually take nearly as long as actually copying the file. Therefore the recommended solution for Windows at the time of
writing is to examine the XML index rather than accessing the extended attribute.
So I understand that if I could sort by lfts.startblock within Windows Explorer, I could possibly then select all the files that way and have them copy in the correct (most efficient) order. But I can’t find a way to do that, and I’m not quite sure where to go from here.
Opening the index XML file in a text editor, I can see each file has its own tag. So the information appears to be there, but I’m just not sure what to do with it. Hoping somebody else has tackled this before. 🙂
-
4 Replies
-
That info is primarily aimed at developers, not end users. Unless you are writing a wrapper for LTFS, there’s no way to simply “do” that in the Windows terminal (cmd or PowerShell). It basically translates to –
Read the XML file
Sort by Block instead of name
Create your copy list from that sorted data
Manually copy each file as a separate task in a loopAdditionally, if there have been file deletions or overwrites, things will become even more obtuse in dealing with the fact that the old version is still on the tape, just masked in the later XML entries.
Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters! -
Thanks for the response, Tim.
I’m very happy to announce that I found a solution! ☺
Turns out HP provides a command-line tool, called ltfscopy, that does exactly what I’m looking for. According to HP’s own PDF:
LTFS smart copy for fast data transfer from tape
Tape being a sequential access medium, retrieving large number of files from the tape requires
that the tape head has to seek the location of the file sequentially on the tape by winding the
tape back and forth many times until it reads all the files that needs to be copied. This causes
drastic delays when using the normal copy commands provided by the operating system because
those commands do not care about the location of the files on tape. When these commands are
used to copy multiple files from tape to disk or another tape it will result in poor data transfer
rates. The LTFS smart copy tool overcomes this limitation by rearranging the files to copy in such
a way that all files can be copied in just one pass of the head. This significantly reduces the total
seek time thus improving the overall performance for multi-file copy operations.I tried it out, and it works very well and is so much faster than copying multiple files through Windows Explorer.
Whew. 🙂
-
Good find.
Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.tolisgroup.com
BRU … because it’s the RESTORE that matters! -
Dear,
I’m trying to find a way to read the index of the LTFS tape (xml file).
Is there a command to do it ?I’m using the Linux HPE StoreOpen.
Thanks in advance for your support.
Best regards,
Reply to this Discussion! Login or Sign Up