-
Cost per GB for large data
I am doing this study for production companies that generate large amounts of data like with Red Epic/ArriRAW/F65 feature shoots, need to back it up, and at some point of time need access to all of it at good speeds.
I’ve considered bare drives and single drives which are the lowest cost/GB, and LTO tape which too is low cost/GB. Both of these are reasonably good for backup, but don’t permit access to all data at once and speedily. Meaning if you’re looking at Firewire/eSATA/USB3, then even 50 TB it would become a forest of cables, pass-through or hub connections, multiple eSATA cards etc.
Then I considered RAIDs. Good RAIDs from Promise, Maxx, GTech etc, and again found that to be practical, they would need to have 8 or more drives. Larger arrays with 24 or more drives have higher, per GB costs compared to smaller 8-bay RAIDs. And they are larger and heavier and noisier so, a rack mount in a separate room becomes inevitable.
So I found the sweet spot is a 8-bay 24-32 TB Thunderbolt RAID. You can daisy chain 6 of them per Thunderbolt port and an iMac has two Thunderbolt ports so you can add remove, mix and match up to 12 drives totalling nearly 280 TB of RAID6 storage (in 24 TB increments) for about $57000. This is using good reliable drives from good vendors. Not some DIY stuff with no warranty.
Also, Thunderbolt means no in-between eSATA/SAS cards and expansion boxes involved. Direct connect with today’s Macs.
Anyone with better ideas? SAS/FC RAIDs? Larger, reliable arrays but economical?
———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India -
8 Replies
-
Are you looking for active near-line, or as working volumes?
As an active near-line setup, I believe you are correct, the 8-bay TB RAID from Areca with quality drives is the sweet spot. The price amazes me.
But as active work volumes, there’s a few things to keep in mind. The tests at Barefeats show the 8 bay Areca maxing out at 900MB/s, demonstrating that this is more or less the practical speed limit for a single TB1 connection. If you daisy chain a bunch of RAIDs over this connection, that 900MB/s has to be shared, which could get ugly depending on what you’re trying to do. If you want to share these volumes out to ethernet connected clients, you could run out of bandwidth if you have too many people hitting the storage. TB doesn’t scale nearly as well as SAS for this use. SAS has higher speed limits (depending on the cards) and there’s the ability to put multiple cards in a single server.
but if the 900MB/s speed limit isn’t an issue, i don’t think you can do much better. Thunderbolt is nice and clean. But dealing with daisy-chained Thunderbolt drives is bringing back awful memories of daisy-chained FW800 drives, 6-10 sitting on a desk, trying to figure out which are plugged in, and in which order. I haven’t worked with more than 4 TB drives connected, and not long enough to comment on connection reliability. But hopefully TB is much better than FW800.
e
Eric Hansen
Production Workflow Designer / Consultant / Colorist / DIT
https://www.erichansen.tv -
Are you specifically trying to avoid stuff that’s rack mountable? If so, could you explain why? (Is it more about space restrictions, or cost, or is it because you plan to physically move the arrays around, something else…?) Is the primary $/GB goal to get the best upfront purchase price, or is it TCO over a long term?
I have to agree with Eric’s post above in the sense that daisy-chaining a bunch of TB arrays conjures nightmarish images. That’s especially true if you’re planning to frequently move around the arrays.
Eric Newbauer
Studio Network Solutions
https://www.studionetworksolutions.com -
No I’m not trying to avoid rack-mount arrays though I haven’t come across any which cost less than non rack mount.
I agree about the cable spaghetti with Thunderbolt, but that would be true of miniSAS too, particularly with multiple 8-bay boxes.
I’m keeping the cost down because hard disk costs change so rapidly that it makes no sense on spending a fortune on something that will be hopelessly outdated in just 3-5 years. I know, I’ve helped spend $2 million on a 20TB SGI Stone Shared in 2004, about the same on a 48TB Bright in 2006, and somewhat less on a 120TB DVS in 2009.Even a 8-bay Thunderbolt or SAS RAID with SATA drives today would probably match all of these for speed and capacity. So I’m much rather spend as less as I need to now, because I’ll need to replace these by 2016.
The 900MB/sec speed limit is just fine since this array will store Red/Alexa/Sony camera originals with the need to initially transcode rapidly and then simply make them available for occasional browsing. Eventually this data is needed to be accessible all at once for conform and export selects. So it can be characterised as occasionally online, mostly archive.
If 12, 16 or 24-bay boxes would have cost less than the sum of 8-bay boxes, I’d rather go with more bays. But more bays for some reason runs up the per GB cost. Meaning one 24-bay seems to cost more than 3 times 3×8-bay boxes. And rack-mount boxes are noisier than non-rack-mount ones.
———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India -
[Neil Sadwelkar] “No I’m not trying to avoid rack-mount arrays though I haven’t come across any which cost less than non rack mount.”
Hi Neil,
A decent 16-bay rackmount MiniSAS expander with redundant PSUs (ProAvio IS316JS, for instance) is around $2700. An 8-bay MiniSAS expander tower – $1.5K. Purely among MiniSAS expanders, cost per GB is lower with rackount boxes – and you get extra benefits in terms of resiliency.
What makes the Thunderbolt 8-bay box possibly cheaper (and of course, more portable), is the inclusion of a RAID controller in that price – an extra $800 for MiniSAS expanders. Indeed, can’t beat that, and like Eric said, the price point is amazing. The only thing against it is non-expansion (can’t make a RAID set larger than 8 drives) and lack of battery or NVRAM backup, a potential “write hole” and data loss issue.
The former also may make the Thunderbolt box more expensive per GB: with dual parity RAID6 and a hot spare, your space utilization is 5/8th or 62.5%. The same on a 16-bay box – 13/16th or 81.25%. Hot spares can also be shared between boxes with MiniSAS expanders, not something you could do with Thunderbolt boxes.
HTH.
Alex Gerulaitis
Systems Engineer
DV411 – Los Angeles, CA -
[Neil Sadwelkar] “If 12, 16 or 24-bay boxes would have cost less than the sum of 8-bay boxes, I’d rather go with more bays. But more bays for some reason runs up the per GB cost.”
Expander boxes have been less expensive (per bay or per GB) for me with more bays. What boxes have you been looking at, that are more expensive with higher bay count?
[Neil Sadwelkar] “And rack-mount boxes are noisier than non-rack-mount ones.”
Not the case with that ProAvio box I mentioned above. Certainly the case with some older 1u and 2U boxes, and with most CineRAID EditPro rackmounts, unfortunately.
-
i definitely wouldn’t run RAID6 plus hot spare on an 8 drive RAID. I’m paranoid, but not that much. about half of my 8 bay RAIDs are RAID6 and half are RAID5. the more “active” ones are RAID6, but i don’t much care either way as long as there’s cold spares sitting around.
Neil, another thing you may consider is bare hard drives and a MAM system. For most of my clients, that’s a much better option than leaving everything on growing near-line RAIDs, running 24/7. But it depends on your workflow.
e
Eric Hansen
Production Workflow Designer / Consultant / Colorist / DIT
https://www.erichansen.tv -
It probably is too DIY, but this is very high density in terms of both space and $/GB:
Install FreeBSD on that and use ZFS across all those HDDs and you’d have a pretty stout NAS box.
Best,
Andy -
A Net store 16 bay array with Thundeboldt connection would be my choice.
Raid5 is suitable for 8 bay raids, With 16 bay I would choose Raid6.
I think thunderboldt is the connection of the future.Cheers
Rainer
factstory
Rainer Wirth
phone_0049-177-2156086
Mac pro 8core
Adobe,FCP,Avid
several raid systems
Reply to this Discussion! Login or Sign Up