Martin Greenwood
10
COW Coins
6
Storage & Archiving Coins
Forum Replies Created
-
Hi Ed,
We’re looking at importing Cache-A metadata into our LTFS app. If you want to email a link to your backup tgz then we can take a look and see what is possible.
Thanks,
Martin Greenwood
CTO
YoYotta.com
-
The attribute messages can be ignored, they just indicate an older format.
Data has been written to the tape as the available data space shown is approx 180GB. However the index space shows no usage… this isn’t expected. Even with an newly formatted tape the available space on the index partition should be less than the total index capacity. So this might indicate corruption. Do you see this same result with other tapes?
It’s as if the data has been deleted from the tape using Finder or Explorer. When this happens the data isn’t actually removed. So the space is still used, but a new index is created making the old files disappear.
This is because LTFS always appends new files, never overwriting existing files.Each time a group of new files are written to, or deleted from an LTFS tape a new index is created and it’s possible to rollback to previous generations (rollback points).
A plug for our YoYotta LTFS software here… if using our app then each generation will be labeled telling you the stage of the archive, writing/copied/complete and also the source file path. This makes it easy to find the required generation if files have been accidentally removed or the tape index has been damaged due to an equipment or power failure.
So the next step would be to list the available generations by typing this into a terminal.
ltfsck -l 0
Hopefully you will see some Generations listed. You can then mount the tape (in a read-only mode) at a previous generation when the files still existed using this command. Replacing the 10 with the required generation.
ltfs /Volumes/LTFS -o devname=0 -o rollback_mount=10
If nothing is listed then the empty index partition may be preventing access to the files. You would then need to get a recovery specialist to look at the tapes.
Good luck !
Martin
CTO
YoYotta.com
-
Indeed it is normal to see that message when reading older tapes with a newer LTFS. Older tapes can be read without issue. However if you don’t see any files that that suggests an issue with the tapes. If you want to post or email me the full dialog from the mount command I can take a look.
Best,
Martin Greenwood
martin@yoyotta.comCTO
YoYotta
-
“Reduces the tape life by half” must be assuming a completely different workflow, where you rewrite and verify tapes many times with new data. Performing a verify means you can only do this 4000 times rather than 8000 times. Which still sounds like a lot.
However your tapes are probably only written once and verified occasionally. So using those numbers you can still read back the entire dataset 8000 times ! So the verify has not reduced the life at all.
It does take time, but this is your archive. Also the verify performed by the backup application will check that each file is on tape, whereas the drive just checks the bit level data sent to it. So the drive cannot detect if a file is completely missing.
Martin
CTO
YoYotta.com
-
If you are looking at getting new hardware, then the extra cost of a library over a standalone drive is not that much. Standalone drives are obviously more portable, but the library wins otherwise. With two drives you can automate archiving large amounts of data spanned across two sets of tapes. With a single drive you can do the same thing sequentially. Also performing other tasks like a partial restore of selected files from a number of tapes is much easier.
How much data do you need to archive and how much time do you have before more data arrives?
This will help gauge the best system for your needs.Software like our YoYottaID Automation will control the library, create and restore industry standard LTFS archives that can also be used for interchange. Our Conform option allows automated restore and trimming of clips from the tapes in the library using FCPXML, ALE, EDL or AAF editorial timelines.
Martin Greenwood
CTO
YoYotta
-
Kevin,
A more detailed update regarding LTFS + SAS + LTO-7, please contact me if you have any questions.
There is a SCSI protocol in the T10 SAS Protocol Layer (SPL-4) called TLR that is required for IBM tape drives.
Note that all LTO-7 drives are manufactured by IBM.
The TLR protocol was rarely used before LTO-7, because of the lower transfer speeds.
If the SAS interface does not support TLR, then you will get read errors that are not signalled.
These errors would remain undetected unless the app used compares the data read back against a checksum of the written data.HPE, IBM, Quantum, Tandberg and YoYotta only recommend SAS interfaces with TLR support.
For the Mac the ATTO H680 PCIe card supports TLR, so this is the only interface we recommend.
To use the H680 with a Thunderbolt Mac put it into a Sonnet or other TB PCIe chassis.If you already have an ATTO R680 and SH2068 there is a workaround to limit the link speed, which does seem to reduce or eliminate TLR requests.
https://yoyotta.com/help/installATTO.html
We do not know of any fix for the HighPoint 6328, so this is not suitable for use with LTO-7 and LTFS.
Best,
Martin Greenwood
CTO
YoYotta
-
We have developed a web interface to view and search the metadata from your archived assets. Also you will be able to easily share metadata between multiple YoYottaID systems. More information coming soon. You will be able to see this on the Global Distribution booth at the BVE show next month in London or on the YoYotta booth at NAB in Las Vegas.
Martin Greenwood
CTO
YoYotta
-
Hopefully updating the firmware will enable detection of the tape drive. You need to install the HighPoint web gui to perform the update.
I’ve changed our RS6328 support page to include the firmware update procedure and also to fix the broken Highpoint download link !
https://yoyotta.com/help/installRocketStor.html
Martin
CTO
YoYotta
-
Hi Allen,
Try checking for SAS devices as described in our install page for the RS6328
https://yoyotta.com/help/installRocketStor.html
Once you have got it going you may want to try out a demo of our YoYottaID LTFS software.
https://yoyotta.com/help/demo.html
Martin Greenwood
CTO
YoYotta
-
Indeed you should have an near line backup of your current projects along with new material (graded, rendered, processed footage).
Using a NAS for this makes sense. But this should be in addition, not as an alternative to LTO backup.
LTFS tape can be part of your workflow from the very start of the project.There is the raw footage, the majority of which is not being used by the project. Plus all completed projects.
That is where LTFS tape fits in perfectly.
With 4K+, high dynamic range, high frame rate, multi camera capture being commonplace, along with longer takes even a small project can have multi TB of material.
However after editorial the majority of this isn’t needed, but still needs to be kept.
A lot of our users have projects with hundred of TB’s of footage. Most of this never touches online storage.
With LTO-7 tapes the restore speed is comparable to NAS speeds.So I would suggest online storage (direct attached or SAN) plus near line NAS backup.
Then archive from the NAS to LTFS tape as archiving direct from “live” online storage is not a good idea.Here is a suggested workflow.
When footage is created as soon as possible make two copies on LTFS tape.
If there is a lot of material, then at the same time transcode it to a proxy format for editorial.
After editorial restore the original footage from tape (if you didn’t keep it online).
Link rather than copy material into your project.
Create new material outside the project. That way you keep the project small in size.
Backup the projects and new material to NAS.
The original footage doesn’t need to go to NAS, because you have two tape copies.
Backup projects and new material from the NAS to tape.
On completion remove projects from online. If there is space then keep then on the NAS for a bit longer.Martin Greenwood
CTO
YoYotta