CatDV Re-archive footage with Cache-A

[Jeff Schaap]

Wow… thank you Tom for the information, especially about the end user needing to restore the files to the Vtape first… our integrator never told us this and all our frustrations now make sense. This is the reason why we can’t get anything to restore properly. In all the demo’s we saw of the plug-in the tape was never ejected so the files never left the Vtape… they were just getting recalled from the Vtape when the plug-in called for a restore. That must be why it worked in those demo. settings but not in our “real-world” tests. No wonder the plug-in is saying things like “Can’t restore from VTape” while I bang my head on the wall and say “It’s on the LTO!”

The information about not using spanned volumes in CatDV is also helpful. Again, I wish we had understood that at the outset so we wouldn’t have wasted hours trying to figure out why it wasn’t working.

I guess this is why they say information is power… sigh.

Thank again Tom,
Jeff

[Nathan Mcalpine]

[Lindsay Simpson] ”
Nate, you had mentioned looking at file size by hovering over a file in the CacheA UI correct?”

You must hover over the file size to get the uncompressed file size (figure 01) I haven’t noticed the media directory being larger at the end of transfer. I’ll watch for that.

[Lindsay Simpson] “Nate, what type of media are you archiving through CatDV? Does your production house convert all media upon return from a shoot out of it’s native format?”

We do sometimes get footage from outside sources that has been transcoded but mostly it’s our native format and we shoot ProRes. We also have some DVCPROHD footage. Coming up we have an XDCam shoot and am contemplating purchasing the MXF option so we can ingest that directly.

~ Nate

[Nathan Mcalpine]

[Lindsay Simpson] “Forgive my ignorance, but is worker node a feature only accessible if you have a server-based CatDV system?”

The server plays a vital role in our workflow because we have CatDV on ten computers as well as two Web Client licenses. We save all of our catalogs to the server so any of the computers can access them. Worker node, I believe, is only available for server-based systems, as we have it on a Windows server (although all our licenses are on Macs). You can send commands to make proxies through Worker Node from any of our systems.

~ Nate

[Nathan Mcalpine]

Tom, thanks for your input. You brought up several key issues with how we’ve been archiving.

[Tom Goldberg] “First, the CatDV plugin does not deal with spanned tapes aka Multiple Volumes.”

We have been trying to add media from multiple volumes.

[Tom Goldberg] “In order to restore files with CatDV’s plugin, they must already have been restored to the VTAPE by the user first.”

I simply followed the instructions from Matt Stamos on this YouTube video where there is no manual intervention:
https://www.youtube.com/watch?v=Ivb251Qgk7U&context=C4e9f2e1ADvjVQa1PpcFNojjn7TSuKP7OCh5n26WcqzYJi9nglF7k=

We have been using LTFS but currently are working with a tape that somehow became a .tar format. We have over a TB of “space lost”. What do you make of that?

[Tom Goldberg] “have you contacted our tech support? We can help you through some of these issues. What version is your Cache-A (you should be at v2.1.15)?”

We have contacted tech support. Rolf actually gave us a helpful response today which we haven’t had a chance to test since we’re also having an unrelated archive issue with another company and we’ve been working with that all day. I will give you an update as to if this fixed some of our problems.

It also looks like we have an old version as we’re operating on v2.0.22. I’ll update that and see if that helps. Hopefully we will have time to run some test and devote all of our attention to this so I can give a detailed update by the end of week.

~ Nate

[Tom Goldberg]

Nate,

I’m sorry that your expectations were not properly set for the interactions between our products. Not much I can do about that except to disclose all the facts to the best of my understanding here. As I’ve mentioned, I know Rolf is actively working on the restore issue.

As noted in our 2.0.22 release notes, once you format a tape as LTFS, you can’t change it back – this explains your “Space Lost” issue. As noted in our 2.1 release notes, this has been fixed, so once you get the update, you will be able to fix that tape – format it back to LTFS and then back to tar again to get all the space back.

We’ll need your serial number to schedule that software update for your machine – send me an email with that info or simply fill out the Software Update request form on our web site (https://cache-a.com/uprequest.php)

Tom Goldberg
Cache-A Corporation
602 Park Point Drive
Golden, CO 80401
mailto:tom.goldberg@cache-a.com
https://cache-a.com

[Jeff Schaap]

Ok… we are SO confused.

We just did a test and CatDV Cache-A WAS able to recall files from the LTO tape… we deleted the VTape, called for a restore and watched the Cache-A seek up to the files, copy them to the Vtape and then CatDV Cache-A plug-in copied them to our SAN. So, Tom, it seems that this is working after all?

However, some of the footage we wrote to the tapes is missing… LARGE portions and we don’t understand why. We wrote in two sessions and only some of the footage from the FIRST session is on the tape.

Tom, can we not write multiple sessions to the same LTO? That poses a big problem if it’s true.

Jeff

[Jeff Schaap]

Hey All,

I just wanted to post an update after we did more tests today.

Here’s what we did:

1) With CatDV Cache-A plugin wrote a catalog of 91 files (30 GB) to an LTO tape. No logged errors or other problems.

2) We ejected this tape and cleared the vtape.

3) Re-inserted the tape and then restored the files with the CatDV Cache-A plugin. PROBLEM. Six files were 1/2 or less of their original size. CatDV could not play them. Couldn’t play them in the OS. They are incomplete and / or corrupt. BLARG.

4) We compared the problem files (now on our SAN) against the files on the vtape. THE FILES ON THE VTAPE WERE FINE… wow. So the problem came when CatDV Cache-A plugin copied them from the vtape back to the SAN… WHY?? That seems like a relatively simple operation.

5) We deleted the problematic files from the SAN and the vtape and then called for a restore of just these files through the CatDV Cache-A plugin. They came back from the LTO to the vtape to the SAN without any issue. We are perplexed.

6) We started over… we deleted all the files from this catalog on the SAN and the Vtape. With the CatDV Cache-A plugin we called for a restore of all the files. They were all pulled from the LTO tape to the vtape to the SAN without any issue… HEAD SCRATCHING ENSUES. Why did this not work the first time and now it was flawless?

We move on to seeing if we can write multiple sessions to the SAME tape (NOT spanning tapes). It works only if you NEVER EJECT THE TAPE. If you do eject the tape and then try to write more files to tape in a later session it will overwrite whatever is on the tape… so that means you have to fill the tape (if you want your money’s worth) without ever ejecting it.

My question is, is that normal operation or is this some sort of problem with CatDV Cache-A plugin? Any thoughts Tom?

[Lindsay Simpson]

Ah yes, I understand now. Thanks for clarifying Nate.

Lindsay

[Lindsay Simpson]

Hi Tom,

Thank you for clarifying the restore procedure for CatDV. I will be sure to follow that process, once I have an error-free, successfully archived a catalog.

As for the issues that seem to specific to our system, at this point I am at a bit of a loss. I have also been working with our IT specialists on everything from examining permissions on all related hard drives to our network connections, including testing different Ethernet cables.

I will continue archiving directly to the CacheA using the staged transfer method for now, and fingers crossed perhaps will have a new revelation or idea to test out. I am open to any suggestions you might have.

Lindsay

[Scott Tupper]

Jeff,
Thanks so much for the extensive testing!
I sure hope that either Rolf or Tom can answer your question, “can you perform multiple sessions to the same tape, eject, erase, reinsert tape and perform restore via CATDV?” This in my mind would be a deal breaker as we would be archiving projects on a daily basis with the chance of needing to restore from a different tape at any given moment.

Let’s see how the day pans out.

-scott