Isilon… anybody?

[David Gagne]

I’ve got an Isilon 6000x that I’m about ready to retire. 4 nodes, around 20TB total.

I’ve talked about it before… but I’ll do it again…

First, the good stuff:

• Single filesystem across cluster.
• Never had a performance problem.
• Very resilient – even had a node die and lost no data.
• Management UI is decent, and has realtime monitoring.

Next, the bad stuff:

• Fairly expensive, especially when you factor the ridiculous support costs (20%+ per year?)
• Support is slow. Some of my issues took weeks to resolve, and a couple issues are still unresolved.
• You need support. I had 3 NVRAM batteries go bad, and one raid card. Software updates require support. Any troubleshooting requires use of proprietary command line commands.
• NFS is not always ideal for editing, permissions get weird sometimes.

Who should buy it?
All in all, my experience has been kind of negative, but honestly, my company should not have bought it in the first place. Isilon is great for people who have huge amounts of data and can benefit from having it consolidated in one place, with a purchasing strategy that will take advantage of the increasing rewards of adding nodes. If you’re not working with half a petabyte, I’d look elsewhere. Four nodes of Isilon does not make good financial sense.

Final thoughts
Isilon IS expensive, but it is also some cool tech, but few and far between are the people who actually can take advantage of it. Also it seems like it’s being retrofitted for more typical IT environments with additions of features like iSCSI, Hadoop, VM integration, etc. If I had tons money and I had to pick ONE storage platform for the rest of my life, I might pick Isilon. Then again, I’m not sure I could live with their support.

[John Heagy]

Thanks for the detailed reply David.

You mentioned you had no issue with bandwidth. Can you describe the workflows the Isilon volume supported? Any live ingest?

Thanks
John

[David Gagne]

Yeah, live ingest is no problem for us, because we’re using ProRes and DVCProHD.

But you should be able to upgrade to 10Gb if you needed uncompressed or something.

Our setup:

4x nodes, 2x 1Gb ports on each, for 8 total.
2x playout servers
2x capture servers
Each server is connected to a separate node (or sometimes I double up if I want to free up a node for edit).

For one event I captured 7x live feeds DVCProHD, while also allowing editors access. I posted it here somewhere in the forums, you can look up that report.

I use NFS as my protocol and use Softron Movie Recorder for capture. No problems with it. The only problems I had were with some wonky NFS permissions messing up my files, but once that was sorted it was smooth sailing.

Pretty cool, but in theory you should be able to do the same with a much cheaper system. Maybe if I had to capture like 16 channels while editing multichannel something like this might be worth it…

[John Heagy]

Interesting you mention 16 channels of ingest. That’s how many we send to each volume for a total of 32. We do require growing file playback also.

You mention connecting clients TO a node. Don’t all the node and client connections go to a switch?

I agree that the cost and the typical IT targeting are drawbacks. I put them in the same category as NetApp.

Thanks
John

[David Gagne]

Yes they connect through a switch, but you can access the cluster through a specific IP address allocated to a single gigabit port. Of course you can do bonding etc., but it’s pretty easy to just segregate your traffic, which also means that file transfers won’t slam into a recording process.

Are you doing 32 channels of uncompressed or compressed?

-David

[John Veron]

Hi,

I’ve read your post and was wondering why you are looking at an Isilon for the kind of work you want to do.
Are there any IT features that you require that makes you want this solution?

There are heaps of vendors offering scalable (not auto balancing though) storage that does it in one 3 HU box at a fraction of the price.

Editshare, Facilis, GBlabs, Smalltree, etc.
They all are well suited for many streams of compressed (and even uncompressed HD).
Looks like a waste of money to go for Isilon unless you need all the other features.

Regards,
John

[John Heagy]

We do ProRes LT for the mass ingest.

ProRes SQ for everything else.

[John Heagy]

The big selling point for Isilon is easy expandability and increased resiliency the bigger it gets.

Editshare can’t be expanded once a volume is created and Smalltree is very limited to how large it can grow. Can’t comment on Facillis. Isn’t that a block level SAN system?

We have high hopes for GBLabs and will be testing a system. While expandable and redundant, it is far more complicated to achieve.

The IT market Isilon targets is both good and bad. Bad that it expends resources to cater to that huge market, but good in that helps keep the company in the black. A small media only focused company, like GBLabs, does limit it’s market and therefore it’s sustainability.

John

[David Gagne]

If you’re interested in Isilon, I’d say that Coraid is also worth a look. Again it’s IT focused, but their ZX series gear is pretty slick for large media stuff.

You get dual heads for increased bandwidth and failover, ZFS filesystem, NFS mounts, and you can grow on the fly by simply adding shelves of storage or even individual drives. Also cool is that they don’t charge an arm and a leg for drives or support, so it’s really cost effective to grow.

There’s a lot of options here for growth, you can add SSDs for faster read/write cache, SAS if you need IO, or just cheap sata drives for more capacity.

Also it’s based on Solaris and so you don’t have to deal with all the proprietary commands and horrid support of EMC/Isilon.

It’s definitely not for the smaller shops, but for larger operations it might be fitting. The cost savings start to kick in at around 200TB.

[John Heagy]

Thanks for the tip David!