CatDV Re-archive footage with Cache-A

[Jeff Schaap]

Whoops… figured it out. The material was still on the Cache-A “VTape” and thus the reason CatDV / Cache-A was refusing to rewrite it. We deleted the material on the VTape through the Cache-A interface and then were able to start the backup again.

Jeff

[Scott Tupper]

Jeff,
I was evaluating the Cache-A integration last year but I don’t believe I had the latest Cache-A software. We haven’t purchased CATDV yet but we are using the Cache-A and tape capacity notification for me is an issue. Can you clarify for me that in order for CATDV to ask for a new tape when it reaches capacity that “multiple volumes” must be enabled on the Cache-A? I can’t test this and the last time I chatted with Cache-A tech support they suggested that I not enable that option.

Thanks
-scott

[Jeff Schaap]

Scott,

When we enabled multi-volume and Cache-A did ask for another tape (through the web interface) when the first was full. However, we have not successfully recovered files from a spanned set. In fact, the Cache-A plug-in for CatDV doesn’t seem to be performing reliably at all and we aren’t using it or the Cache-A at this point. I can’t detail everything we’ve tried but basically, it doesn’t work. We get conflicting stories from our integrator, CatDV and Cache-A about whether or not we can use spanning. It’s been very frustrating because we can’t get a straight answer out of anyone.

Right now I have a technician dedicated to getting it figured out. I personally can’t spend more time on it as we have actual projects to get done. I would be happy to update you on our progress and what we discover. If you don’t hear from me in a while give me a nudge and I will update you here.

Thanks,
Jeff

[Lindsay Simpson]

Hi Jeff,

Wow, you have no idea how strangely comforting it is to have evidence of another Cache-A/CatDV user experiencing difficulty with products and workflow. Our issues are different however.

I have yet to even play with the multiple volumes options,as at this point with the media types we utilize (3 varieties of MXF)I have yet to have a successful archive out of CatDV to Cache-A. That is an archive that transfers error free as logged by the Cache-A UI

May I inquire as to what media types you are importing into CatDV for cataloging?

Based on our staff, equipment, software etc., it has been convienent and effective to edit directly from the native MXF pulled from tapeless camera media, including P2 cards, SD cards and Ikegami GFPaks. So of course these formats would provide some great challenges in terms of archive! We do have the MXF option purchased for CatDV as well as all of the Calibrated MXF plug-ins too, but it seems that we will be forced to convert from MXF to something else to archive using CatDV. At this point we have no money to invest in a worker node, and I guess I can really only blame myself for not recognizing that issue when the CatDV purchase was made specifically for the archiving function.

We are a PC based system, just FYI.

At this point…I almost miss tape…

Lindsay

[Nathan Mcalpine]

I feel for both of you guys. We’ve had two techs, myself and another, working for the last couple weeks trying to get our archiving system on track. Archiving from CatDV to Cache-A has been a roller-coaster of an experience.

At first things were going very well. The footage would be backed up to LTO and a field in CatDV would auto-fill with the path to where the footage was archived. Our first problem was trying to easily verify that the footage had been successfully backed up. There is no way to compare file size, as Cache-A compresses the files. We did learn that by hovering over a file, it does show you the original size, but it took several days to discover that. You also cannot view the footage from Cache-A, to double check that it’s actually there.

We proceeded with archiving this way for a few hours, then started running into issues. Occasionally, a file would not back up and we would receive an error message that CatDV could not find the path to Cache-A. I would try to reconnect to the Cache-A and it would work for a little while before failing again. Sometimes files will not back up and we receive an error message “Unable to make directory.” We have contacted Cache-A and CatDV and have yet to get a useful response.

I attempted to bring footage back in from after purging some test footage. I got an error message saying Cache-A cannot locate the file. Using the Cache-A web gui, we can locate the files on the LTO, but for whatever reason it is not found when trying to bring back the footage.

Every time we think we’re making progress, something quirky happens. The Cache-A wont mount, error messages pop up. Our most recent issue, we have a 1.5TB LTO. Put 150GB onto it through CatDV and it said we had 1200GB of “space lost” so the tape was full.

So frustrating. I keep waiting for things to just fall into place and work but as of now, it hasn’t happened. I hope things change, because we have a lot of time and money thrown into this and we really want it to work as advertised.

~ Nate

[Jeff Schaap]

Hey Lindsay,

Yeah, I hate to be a pessimist but I have yet to bump into anyone who is using this successfully. I have had integrators demonstrate it to me by writing a couple files and bringing them back but that is a far cry from backing up and recalling thousands of files without issues.

We aren’t having any issues on the transfer or errors during Cache-A writing. It’s recall that’s the issue for us right now. We did discover today that you have to OPEN the catalog and then do the backup in order to retain the information in the “Archived Status” field. If you are in read-only server mode it will show the tape name and such that the clip was archived too but the info. will be lost if you restart CatDV and no other client will see those changes (because they are not actually saved). So, you have to open the catalog, send to to Cache through the plug-in and then save the catalog.

Strangely, we have discovered that when we don’t do that CatDV STILL knows where the clip was saved (but doesn’t display it in the “Archived Status field)… If you select a clip that has nothing in the “Archive Status” and then go Tools > Archive Status it reports the tape name where it’s located! Figure that one out.

I was also going to mention to you and Nathan to watch out for illegal characters in filenames… we are in a Mac environment and anything that had non-alphanumeric characters gave us errors. And unfortunately, we had a ton of those files. Those files would trip the creation of “Apple Double” files and they took up a LOT of space. We eventually got this mostly resolved with CatDV. I am wondering if you and Nathan are getting the same type of problem on the PC side… since you both get seemingly random errors. Could it be filename issues?

To answer your question about file formats we have quite a variety… We have some stuff prior to when I was here shot on Panasonic P2 format but that has mostly been dropped to ProRes. We now shoot AVCHD footage generated from our Sony NEX VG-10’s (LOVE this camera).

We did not pay for the MXF plug-in for CatDV because ultimately we decided we didn’t need it. We don’t really have a need but I could create a Worker Node routine to trip ClipWrap and generate ProRes preview files so the clips could be reviewed in CatDV just as we do with our AVCHD files. But we just haven’t had call for it yet since most our stuff was “log and transferred” by FCP 7 into ProRes and the MXF files are long gone. We have a ton of other stuff that is mostly .MP4 or some flavor of Quicktime. But, we haven’t had any particular issue with Cache-A or CatDV not wanting to back up a certain type of video… which I think was the point of your question.

We are doing more testing today. I will share anything I think that is useful to you, Nathan, or anyone else interested. Let’s keep talking.

Thanks,
Jeff

[Lindsay Simpson]

Hi guys,

Is it sad that at this point I am happy to accept a pessimistic outlook on this workflow? It’s just nice to not feel quite so crazy or that it is entirely based on operator error.

Yet its very frustrating indeed. Over the past few months, as I have been solely responsible for establishing this media archive system, I have felt like a hot potato at times, bouncing between CatDV and Cache-A support with little headway made.

I also have little trust when it comes to having absolute knowledge that files have been properly archived. My latest attempt to ‘Restore Media Files’ in a CatDV to CacheA test resulted in this error:

Perhaps I am missing a step in terms of restoring through CatDV? But it seems to me if I have to manually locate the file, or media directory containing a specific file, through the Cache UI and move it to the VTAPE within the CacheA UI, this defeats the purpose of restoring through CatDV.

Jeff, correct me if I’m wrong, but what you are doing is archiving by moving a CatDV catalog file to the VTAPE. Correct? I haven’t tried that method yet. I have only used the function built into CatDV itself, and am careful to select 20 to 30 GB chunks of clips inside a specific catalog to send to archive at one time. The chunk size limit was recommended by CatDV or CacheA (I can’t remember at this point). Also just FYI we do not operate with a CatDV server here. We are a two-person crew with only one edit system utilizing a raid array for main media storage and attempting to establish this CatDV/CacheA based archive system.

I just did a quick repeat test of a P2 media directory archive, performed three different ways on a freshly re-initialized tape.

Attempt 1 (first event in attached transfer log) was out of CatDV utilizing the ‘Archive Media Files’ tool.

Attempt 2 was a basic, staged transfer where the media directory was copied to a ‘staging folder’ on the CacheA share drive and then copy/pasted from the stage folder to the VTAPE folder. This is when the directory file size seems to change.

Attempt 3 was a basic, staged transfer where the media directory was copied to a ‘staging folder’ on the CacheA share drive and then dragged and dropped from the staging folder to the VTAPE. This is the only way so far that I have performed successful transfers. I have tested this recall successfully (going to test again now…just to verify).

4032_aedirectorycatalogcopypastdragdroptranferlog42412.htm.zip

ae_catalog_test.png

Nate, you had mentioned looking at file size by hovering over a file in the CacheA UI correct? I had not noticed that feature yet, but when I tried it, I get nothing. One thing I have noticed in regards to file size is that on occasion a media directory will be a larger size according to the Cache UI log at the end of the transfer.

Nate, what type of media are you archiving through CatDV? Does your production house convert all media upon return from a shoot out of it’s native format?

Jeff thanks for sharing media type…yes I was curious as to whether it was a media type specific issue. Forgive my ignorance, but is worker node a feature only accessible if you have a server-based CatDV system? Not sure I entirely understand the role of the server though I realize it is a key tool when working in a larger production house.

I had some concerns about portions of the MXF file structure not being archived with the associated media files through CatDV, such as MXF .txt files and empty folders built into the structure for proxies and additional metadata. I feared that when restored our NLE wouldn’t be able to read the MXF structure if it was missing components, empty or not. I have been able to ease my fears of this matter with a bit of testing at least with my P2 media.

In terms of basic archives to the CacheA, skipping CatDV, has anyone noticed any difference when performing a staged transfer as to whether the desired media in the staging folder on the CacheA Share is copy/pasted to the VTAPE folder or dragged/dropped to the VTAPE folder? Numerous tests on my end with these slightly different methods with an almost empty LTO tape and CacheA share drive, have proved that if media is copy/pasted to the VTAPE rather dragged/dropped, the transfer will produce various, seemingly random errors and not be completely successfully.

As far as illegal characters and file naming issues, I have tested that as well and haven’t come up with any road blocks on our end in regards to that. I performed various tests with the same directory with a name from the most basic to more complex using underscores and numbers, all with success if, and only if I transfer via the basic, staged transfer using the drag and drop method. I even tested file path length as the MXF structure can be deep and file path did not prove to be a limiting factor.

Having spent a significant amount of time importing, adding metadata, and sorting our raw media directories in category/subject based catalogs for efficient, long term accessibility post-archive and with our main raid storage almost at capacity, I am extremely eager to verify a successful CatDV to CacheA Archive and Restore with a successful test. I’ve taught myself not to get my hopes up too much. Every time I think I’ve sorted it out, something different seems to pop up.

Anyhow, now that I have mastered the staged, basic transfer, for now I am sadly skipping CatDV until I can figure out how to successfully restore from an archived catalog etc. Any suggestions on that issue?

Lets’ continue this conversation. Thanks and good luck to all on this archiving journey!

Lindsay

[Jeff Schaap]

Lindsay,

We are using the CatDV plug-in exclusively, what you are calling the “Archive Media Files” tool. This only appears in CatDV if you have the plug-in. It didn’t for us until our licenses came through.

We have to do it this way because we need CatDV to remember which LTO tape a given video clip was cataloged too.

I had a thought on your issue with it working when you drag and drop from Staging to VTape but not when you COPY from Staging to the VTape.

I know that as soon as you drop anything in the Vtape directory Cache-A takes off writing it immediately to tape. Just like a CD-writing session if the writing device (the LTO writer in this case) gets “starved” for data it will create an error (a buffer under-run). I wonder if that is what’s happening… Cache-A starts writing but “catches up” to the copying and gets starved for data.

Early on, when we were using Cache-A outside of CatDV to create backups the people at Cache-A were telling our tech. to copy everything to a “Staging” folder first (seems like they are telling you that too) and then once everything was there, drag and drop it into Vtape. Their reasoning was that if you copy across a network directly into the Vtape folder the Cache-A might get starved for data at some point (like when copying a large file) and cause an error.

However, this doesn’t work for our situation. First problem: we have to work with the plug-in as I mentioned. Second: we can’t tell the plug-in to write to the staging folder because that is non-unique name… if all our footage was written to “staging” (as far as CatDV knows) then it is going to try to restore from “staging” instead of the LTO tape’s serial number. So, we MUST write across the network to the Vtape and the plug-in intends for it to work this way. When you start the archiving process you can “see” Cache-A writing the clips from the catalog into the Vtape folder but the serial number of the tape gets recorded in CatDV.

At first we had our Cache-A passing through a 10/100 network and we were getting buffer under-run issues (I think… we were getting errors anyway). However, now we have a gigabit network and we can write with the Cache-A plug-in to the Vtape (through the plug-in) directly without any trouble at all- we never have errors. We’re just working on the recall issues.

So what is your network speed?

We are going to get this to work! 🙂

Jeff

[Lindsay Simpson]

Jeff,

Good to know ‘Archive Media Tools’ = plug-in for future correspondences.

As for the drag/drop vs. copy/paste your explanation definitely makes sense. I’ll continue using the drag method, and I was mostly just curious on that issue as I found it strange. Another thought I had on that was permissions differences between simply moving a folder (drag/drop) and re-writing a folder (copy/paste).

Yes when working with the plug-in which is ideally what I want to do, the staging folder does not play and a role and it shouldn’t have to. Not sure why our plug-in seems to be failing…which again seems to bring me back to media type, though Rolf and others have said it shouldn’t be an issue.

I just checked our network connection and we are running at 1 GB/s just as you are. I just performed another quick archive test and as soon as the data starts appearing in the Cache UI transfer log so do the red ‘x’s and associated errors which are frequently ‘Cannot open: Resource temporarily unavailable’ as noted in the transfer log I provided earlier.

I have another thought about the archive fields in CatDV, but it will have to wait until tomorrow. Thanks again for your feedback as I am strangely more optimistic now than I have been for some time regarding this situation. Have a great evening!

Lindsay

[Tom Goldberg]

As the Cache-A guy on the Cow, I would like to jump in here and try to address a number of the issues being mentioned in this thread.

First, the CatDV plugin does not deal with spanned tapes aka Multiple Volumes. As you add tapes to a spanned set on a Cache-A, we automatically rename each subsequent tape with the volume name of the first tape in the set (i.e. “myfiles) followed by an underscore tape number (i.e. myfiles_2, myfiles_3). The CatDV plugin does not recognize this policy at this time.

The CatDV plugin does display the % space left on tape upon invoking an archive. This allows you to manage your archive transfers to fit on tapes. In general, spanned sets are to be avoided if possible – they’re ok for archiving large data sets that need to remain intact, but not needed for CatDV managed files.

In order to restore files with CatDV’s plugin, they must already have been restored to the VTAPE by the user first. CatDV is actively working on being able to restore directly from tapes, but they will have to comment on the availability of that feature for what version of their plugin.

Lindsay, the kind of media you are archiving is irrelevant, we are a data device and when it comes to archiving, bits are bits. Because you are getting file transfer errors, I’d suspect your network, your physical connections, your archive source hard drives, your PC configuration… something in your system. Because we have a lot of units deployed, and problems like yours are most likely to be due to such end-user system issues.

Nathan, have you contacted our tech support? We can help you through some of these issues. Are you trying to use LTFS (not tested with the CatDV plugin)? What version is your Cache-A (you should be at v2.1.15)? Is it under warranty (Lindsay’s is not)?

Jeff, you seem to be on the right path. I hope some of these explanations help explain issues you’ve had.

I hope all of you can work through your issues and will be here to address this kind of topic… usually over on the Archiving and Backup forum, but happy to contribute here as well.

Tom Goldberg
Cache-A Corporation
602 Park Point Drive
Golden, CO 80401
mailto:tom.goldberg@cache-a.com
https://cache-a.com