-
CatDV archives and Amazon AWS
I’ve been digging around and I haven’t seen a lot of information outside Proxy storage for client review, but has anybody been working to figure out how to handle offline/near-line storage with s3 and Amazon Glacier with CatDV integration?
Thanks.
-
10 Replies
-
We handle proxies on S3 and stream them both to the Client and Web Client. I will be looking into using Glacier to handle offsite backup of the full res. As with the proxy workflow this will require using the Worker and some other third party apps to upload to glacier.
What is your use case exactly?
Scott Goddard
Neo Verite Limited
https://neoverite.com -
(Edit since my ipad post was rife with errors!)
Good timing.
In a few weeks North Shore will be in beta of our CatDV to AWS app.
It will support:
Upload to S3 (you can then life cycle to Glacier) via API which is far more powerful for control and exchange of metadata than simple FTP.
Storage of URL and Object ID in CatDV, as well as the ability to poll for status and see where your asset is in the storage as far as S3 to Glacier.
Restores from S3 (easy.)
Restores from Glacier. (Not so easy but we’re doing it by queuing the jobs for the 6 hour wait time, polling AWS long the way, and then pulling down to original path.)
The big news is that we will have drive imports (AWS import/export). You can prep and our finalization scripts to prepare your drives for shipping to Amazon.
1. Prep the drive with our app.
2. Fill it via Worker actions, recording all the AWS job info and transfer drive info, as it does.
3. Then finalize with our app and ship the drive to AWS. (We create all your signatures, uuid’s and manifests with a double click. We also notify your AWS account to create a job and report that it’s on the way.)
4. Once there, we will link the assets to your CatDV db and then you can manage the assets in CaDV.
Uploading 4 tb via internet won’t be fun, but shipping a drive linked to your db will be fairly easy.
Also, by using only API calls we will retain the media link and you’ll be able to manage assets in AWS completely.
(More info on AWS import/export here: https://aws.amazon.com/importexport/)
Stay tuned. I’ll keep this thread posted as to progress.
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Hey Bryson, sounds great. This will be an excellent workflow and there is a big demand for this as you are no doubt aware. Can’t wait to see a prototype.
Anything else in the works? Server 7 sounds like it’s going to be a big improvement.
A new client interface design is desperately needed though…
Scott Goddard
Neo Verite Limited
https://neoverite.com -
Scott, thank you. And yes, our developer meeting was really solid. I’ll reach out to you as soon as we have something to show and there’s a slide deck in the works as well.
There are so many things in the works over there.
I predict more speed and capacity in the short term (we are also testing some more extreme db server configs as well.)
Interface will be fairly similar but the web will be moving quickly and the partner integrations are going to be more and more important so that Squarebox can focus on making CatDV stronger and stronger as a platform and all us partners will be handling archiving, extended automations and other outside workflows.
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Hey, a reader pointed out that my link to AWS is wrong above. (Good catch, Joe!)
https://aws.amazon.com/importexport/details/ should be the correct one.
Cow links also seem to not be working so you may have to cut and paste.
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Hey Bryson,
How’s it going? Any news on the Glacier project?
Cheers,
Scott
Scott Goddard
Neo Verite Limited
https://neoverite.com -
Funny you should mention that, yes.
We actually finished the code a while ago but in testing we discovered an AWS bug with unicode characters, which they acknowledged publicly and fixed internally. We are awaiting the release but in the meantime our client has sent several terrabytes of data to AWS and has linked metadata for all but one asset which had an unusual character. (No, 99.99999% of the data isn’t good enough for us.) 😉
We have no way to know how long until Amazon releases the fix, but we’re hoping for the next 30 days. The app should be ready to sell after that. We’re working on the docs as I type.
Squarebox will test and then add it to the price list once Amazon is done, but we can sell it as custom for now to solid integrators like yourself.
Anyone who has no unusual characters in their data set and wants to run the code can certainly contact us.
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Hi Bryson, this sounds great. Did the client use a HDD to send the assets to AWS or can this also work through upload?
Scott Goddard
Neo Verite Limited
https://neoverite.com -
Oh yes, I guess I should update this with the final features.
(Note, the Cow doesn’t do bullets well.)
Features:
Send assets to AWS from CatDV via Amazon uploader
No FTP needed
More reliableSend assets to AWS on a hard drive via import/export job
Metadata recorded in CatDV links to the asset in AWS
Details on I/O here: https://aws.amazon.com/importexport/View/track AWS job info & metadata in CatDV interface
Metadata:
DriveID
JobID
AWS Key
AWS md5 checksum
AWS Bucket Name
AWS Import Log filename
For import/export job
AWS Storage ClassView lifecycle status in CatDV i.e. S3 or Glacier
Call restores from S3 or Glacier in the CatDV interface.
Restores from S3 initiate immediately
Restores from Glacier require time for AWS to move asset from Glacier to S3 bucket and then Archive App downloads asset back to home system once asset appears in S3This process is queued and managed by the Archive application, no user input required.
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
Bryson, this sounds great.
I have a client who is ready for this with no bad characters.
I will contact you via email.
Cheers,
Scott Goddard
Neo Verite Limited
https://neoverite.com
Reply to this Discussion! Login or Sign Up