-
DATA And Storage Genius needed for this one,
I’ve been tasked with making a projection for CLOUD storage for the next 2-3 years.
Here is what I know today.
on LTO I have 1400TB
NAS 450TB
CLOUD S3 type 600TB
I can see the dilemma in my head and will try to communicate it clearly.
Part of our legacy video and project assets are on LTO which we started using heavily in 2019 and sending 5 year backlog to tape.
ALL of tape is being pulled back year by year to NAS and we’re moving to CLOUD.
Part of the LTO assets have been moved from tape and are still “online” NOT moved yet to cloud.
Part of what is on NAS is in CLOUD but not on tape because our storage sponsor software went end of life and we didn’t send them to tape while waiting for CLOUD to be stood up.
Part of what is on LTO is on NAS and CLOUD because we had it on tape, brought it back online
Anything pushed to TAPE or CLOUD have been on NAS and tracked through CATDV so we have record of where all copies of any single piece of project data/video lives and where it may have lived prior to where it is currenly accessible.
Our video library has grown extensively by a measurable % since 2019.
Which formula would assist in making the best predictions for future CLOUD needs which would be the balance of LTO coming back online and pushing to cloud..with what is currently online on NAS and excluding what came from LTO and what is on both NAS and CLOUD.
I have an idea of taking a two years near 2019 and finding the % of growth. Pick another year like 2022 and finding the average growth to then multiply out 2x years to get some kind of viable estimate?
Does this sound right?
-
4 Replies
-
This is something that many media houses that have a large quantity of video and audio assets, will have.
I’m intrigued by your multi-tiered approach. The assets some here some there, some in multiple places, is typical, and it will all get sorted over time.
But one question I have, since you’re making projections for the future. How are you going to afford to pay for the ‘holding costs’ that most cloud services charge per month which over many years and many TBs quickly run into six figures (in $$). Plus with some providers, there are ‘egress fees’, so you pay to get your own data back.
The question actually is, are the assets worth these costs? Do they return these investments?
Maybe they are.
About projecting future data requirements, the ‘formula’ you have, seems fine. Although data estimation is tricky. One cannot accurately predict how much (more or less) data you’ll create, going forward. And, with new formats coming out all the time, one cannot predict how small (or large) future assets will be. Back in the day, archives were 10-bit dpx. then came ProRes and files shrunk, but more files were created. Now we have J2k, JPEG-XS, HEVC, so you may have to factor that in.
The other thing some people have begun to consider is ‘data audit’, ‘data pruning’, ‘data mining’. To make the data stored and archived, richer and more usable. But that’s a whole new story.
Neil
-
I see that Neil has been posting quite a bit recently. I kind of gave up on this forum, as I just observed utter desperation of people with old equipment like Tolis BRU, and simply refused to accept that they had old equipment, and had to spend money. The very idea that someone won’t spend the money for a license of Hedge Canister, or YoYotta, and just wants to do it for free is a big turn off for me. What kind of professionals are these ?
As the years have gone by, and I have observed what happened with all of these companies (Tolis BRU, Cache-A, etc.) – it is very clear. Data migration is a fact of life, just like Neil said in a similar post. You can say that drives are not safe, but be it LTO, Cloud, or drives, you are not getting more than 5 – 6 years out of any of these. And if you say “cloud” – then you will be paying for what a large drive based system costs within 5 – 6 years. There ain’t no “free unlimited dropbox account” for this type of professional application.
If you simply get a large NAS, like a 24 bay QNAP TS-h3087XU-RP (it’s $8000 before the cost of the drives – you load it up with 20 or 22 TB SATA drives), then you now have 400 – 440 TB of usable storage after RAID 60. If you have a disaster with your main system, there is no slow download from a cloud site like S3, or trying to restore 400 TB from LTO tapes. It’s instant. Will this last 30 years – or even 10 years – absolutely not. But neither will your LTO 5 tape library that you spend a fortune for. Data migration is a fact of life, and it will cost you money every few years to migrate to whatever is new. I found it very frustrating to observe on this forum, how people were hysterical about Tolis BRU, and simply refused to spend the money to get the “next thing”. These are the same people that are still trying to use their 2009 Mac Pro’s and Apple FCP 7.
Bob Zelin
-
To all. I had to jump into this neck deep this past summer. We moved from 16 separate 53 TB units to 3..and managed to sort out a complimentary portion for each business unit. Here is where we are.
1. IT is using on prem cloud-like storage (s3 type) called CEPH. There were larger numbers finally calculated for the near PB of storage.
2. I had to disappoint everyone with the bad news that “we can’t keep everything” and yes thank you for sure..if there no possible return in the future. We’re keeping our business internal legacy items but I’ve been able to purge almost 400+ TB of just data alone.
3. Going forward – everyone has been teased with if you shoot bigger file size 4 or 6k plan on punching in and use fewer cameras.
4. I have a meeting with Telestream at NAB to rework our internal delivery system with transcodes, archives, and proxy delivery.
ps my sleeves are still rolled up..
-
One thing about your comment “if you shoot 4k and 6k…”
What has been a bigger impact is that nobody says “cut” any more. Shooting ratios have gone sky high.
Reply to this Discussion! Login or Sign Up