-
CatDV and Cache-A Archiving Questions
I’m hoping someone can help with some archiving questions that have recently arisen.
First, let me say that we are using CatDV 8.1.1 Pro/Server with the accompanying Cache-A plugin. Our workflow had been working fine up until about a month ago:
We would add media files to our catalogs, use the Archive Media function under Tools. Choose the Cache-A as the path. CatDV would correctly pick up the LTO tape name. We’d tell it to archive and the files would be immediately copied to the VTape in a mirrored (Volumes/Hard Drive/Media folder/Media File) pathway. And then Cache-A would automatically copy the files to the LTO.
I discovered recently, that the Cache-A was no longer copying the files to the LTO after CatDV had copied them to the VTape. Why the Cache-A is not syncing the VTape is a matter I am discussing with Cache-A Support, so I don’t want to focus on that for this post.
What I am trying to determine is if CatDV is actually doing what it is designed to do and whether I correctly understand what it is doing.
The reason I ask this is because Cache-A Support is telling me that it is a bad idea to modify folders within the VTape. They suggest staging the files outside of the VTape and then copying over complete folders to the VTape root. When we archive files from the Finder, this is exactly what we do, however, this is not the way that the CatDV/Cache-A plugin works, at least on our system. On our system, it copies the files directly to a subfolder structure on the VTape and if you then archive other files onto the same tape, it just adds them to the existing folder structure.
My first question is: Is this actually what is happening, or is there something more complicated happening behind the scenes?
My second question is: Can the plugin be reconfigured to do the staging that Cache-A Support recommends?
And lastly: We’re debating about upgrading to the latest version of CatDV. Would upgrading to the new Cache-A plugin do what Cache-A Support is advising us to do?
I appreciate any help with this.
Gerald Goldman
Director of Post Production
Grace Creek Media -
7 Replies
-
I’m curious about this as well. (The new version that is.)
Cache-A, BRU, Atempo, anyone, LTO tape doesn’t like a lot of little files and that’s how DAM’s tend to deliver them.
From what I know of the Cache-A SDK, you cannot perform “staging” through the SDK. This isn’t a CatDV limitation, it’s not provided for.
Can some of the CatDV and Cache-A crew confirm this?
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Hi Gerald,
I will defer to the CatDV experts on the advantages of v9.x vs. v8.x and focus on the Cache-A side of your questions.
The reason we say it is a bad idea to modify anything inside the VTAPE is that the VTAPE is a watchfolder. Your Cache-A appliance is constantly monitoring that folder for changes and will archive anything it sees that is new. You move a file – it will get re-archived. You rename something – we see it as new and will archive it again.
Bryson’s comment that LTO doesn’t like a lot of little files is not quite correct – we handle small files fine, what causes problems are lots of little archive sessions. So the easiest way to make CatDV prevent small sessions is to get all your content organized and ready for archiving and select a large data set before you select Send to Archive. If you can run the job with 10 GB or more per session, you will have fewer problems and get better tape utilization.
There is a work-around if you want to manage your content within the VTAPE: First, insert the tape you will be archiving to, the Cache-A will create a VTAPE for that tape. At this time you can check available space, rename the volume, etc. Then eject the tape but select “Keep the VTAPE.” Now you freely move your stuff around manage to your heart’s content, and when you’re all done, re-insert the tape. The Cache-A will tell you that the tape’s contents doesn’t match the VTAPE and gives you the option to Sync the contents – select that button and everything that wasn’t in the VTAPE when you ejected will get added to that tape.
Finally, I’d like to take issue with Bryson’s assertion that “you cannot perform “staging” through the SDK” – you certainly can. Our SDK will allow you to archive content from any folder on any volume it can see, and will allow for lists of files or folders to be archived at once (which is effectively staging). CatDV chose to not use that class of command but rather initiate archiving by moving data to the VTAPE. That said, even though I disagreed with this one point, I want to thank Bryson for being a knowledgeable resource for this community (and an all-around great guy).
Tom Goldberg
Cache-A Corporation
602 Park Point Drive
Golden, CO 80401
mailto:tom.goldberg@cache-a.com
https://cache-a.com -
Hi Tom,
Thanks for your feedback. I had been wondering if a “Sync Whole Tape” approach would be the best solution to this issue. This is essentially what staging in the share can do when you wait until you have a tape’s worth of data and then move it all into the VTape at once.
Your suggestion of ejecting the tape would essentially accomplish this. I will try this out. It would also be a work-around for our other issue of the Cache-A sync not automatically being triggered.
Your suggestion of staging archives within CatDV is, however, not very practical. At least not for our workflow. I like to archive from within each catalog that is modified. Keeping current means lots of small (file number, not necessarily data size) archive sessions. I could do a Query to pull up files to archive across catalogs, but I wouldn’t trust that I would catch everything.
Thanks again.
Gerald Goldman
Director of Post Production
Grace Creek Media -
I recently sent an email to Cache-A support about this, but I have yet to receive a response. This seems to be along topic (and probably answers it), but I will post here to make sure:
We recently realized that our archive was only being copied to VTAPE and not actually writing to LTO. We then reinserted a tape and it started archiving, but we ended up tape spanning, which CatDV does not support. So then I deleted the VTAPE and started re-archiving to avoid tape spanning. However, I am getting “lost space” and would like to try and avoid/minimize this. On a daily basis, we would not archive that much. Therefore, what is the best workflow? If I archive through CatDV, then it will automatically write to tape; at least it does now the way it is set up.
I see in the manual it mentions creating a ‘Staged Data’ folder on the Cache-A Share. Should I do this and “archive” the files from CatDV to this folder and then when that has a decent amount of files on it, I then drag the contents to the VTAPE folder? If so, then what? Will it automatically write to LTO or is there another step?
Cheers!
Cache-A version 2.1.17
CatDV version 9.0.6 -
Hi Whitney,
It sounds like we are having similar problems. I’m pretty sure that the files archived from CatDV were ending up on the LTO until about a month ago.
The tape ejection method sounds like it will be an effective work around, but it will require greater attention from the staff. Not ideal, but I should be able to easily confirm that the files are actually on the LTO.
Gerald Goldman
Director of Post Production
Grace Creek Media -
The main difference between CatDV 8 and CatDV 9/10 is that we use the Cache-A API to restore files. When we first developed the archive integration the archive API wasn’t available which is why we use the vtape watch folder, but obviously that is supposed to work – the fact that you can just copy files to the Cache-A disk, secure in the knowledge that they will be archived, is Cache-A’s selling point.
If you’re worried about fragementation (in the sense of lots of separate sessions on the tape) you can create another folder on the Cache-A drive and choose that as the Archive Location (and select Other rather than Cache-A for the Archive System when you configure the archiving in CatDV). Then it will use the folder name as the “tape name” and you can manually drag the contents of that folder onto the vtape to archive it. Better yet, you can follow Tom’s suggestion of ejecting the tape without erasing the vtape, though I haven’t personally tried that myself.
-
Thanks for the info on version 9 and 10. We’ll be upgrading in the near future.
The tape ejection method actually works pretty well. CatDV gets to do what it wants to do. Separately, we can manually stage files in the manner that we like, and then we can insert the tape and basically hit the “burn” button.
It could be a real hassle if someone needs to restore something before a tape is filled, but I think we can deal with that.
It’s a work around, but I think it will keep us going for now.
Thanks again.
Gerald Goldman
Director of Post Production
Grace Creek Media
http://www.gracecreek.com
http://www.sportskool.com
Reply to this Discussion! Login or Sign Up