General questions about SAN from new user

[Caspian Brand]

Hi Mike,

I can’t address all of your questions in one sitting, but one point I wanted to make between some of the differences between SAN and NAS. They are both Shared Storage (which seems a more apt name for this forum these days) but they accomplish sharing storage in different approaches.

SAN = Storage Area Network
NAS = Network Attached Storage

They don’t make this easy…but they are both valid approaches to “Shared Storage”.

One way to consider the differences are that a NAS is basically a File Server, that can be fairly task or performance specific, say for video editing. Conventionally in this model, the storage connects directly to the NAS server, which can be based on Windows, Mac, or Linux. This server then presents network shares to it’s direct attached storage through the server’s own network ports. These network ports are often connected to a switch, and subsequently multiple client computers. The NAS is what arbitrates access to the storage. With a NAS the client sends the NAS a request for Media, the NAS then accesses this media on it’s direct attached storage, then re-shares this media out through its requested network protocol: AFP, CIFS/SMB, NFS, each of which have differing levels of performance and support depending on the client OS.

A SAN on the other hand, is historically a hardware RAID based storage array with multiple network ports on-board (Fibre Channel or iSCSI). These ports will usually connect to a network switch of some sort (usually Fibre or GbE), and require a multi-platform SAN management software (such as SANmp, MetaSAN, or XSAN) to arbitrate the access to the storage in place of a single server/NAS approach.

In the SAN approach clients can have this “block level” access to storage in that the client machine is accessing the media directly from disk vs. requesting the media from a NAS (which then requests the files from storage and re-shares them). With the SAN “block level” connection, as far as the client computer is concerned, that high speed storage is residing inside the client computer, even though it’s really in a centralized RAID system on the network. This “block level” difference is mainly what enables SANs to get higher throughput in the range of the 800MB/s + you’re looking for, by using Fibre Channel or iSCSI.

With some SAN management software, additional MetaData Controllers and MetaData networks may be required, such as with MetaSAN and XSAN. The MetaData Controllers do not present the clients with the media files themselves, just the metadata about the files and where they reside on the storage array. These MetaData Controllers arbitrate the locks to the storage file system, but do not stream the actual media; the media files travel down the high speed Fibre or iSCSI connections directly to the client machines. A NAS, on the other hand, arbitrates the locks to it’s local storage and file system, as well as presents the media files over it’s own network ports.

These days with such products as the EVO, which my company provides, the lines between conventional SAN and NAS appliances is blurring, as our EVO appliance is a storage server which provides a combination of NAS shares and SAN block level storage targets in the same box.
https://www.studionetworksolutions.com/products/product_detail.php?pi=12

The long and short of it, both approaches provide a great way to share media, and you may even find yourself using a multiple, “tiered” approach to solving your storage needs, depending on the number of clients you have and what type of media you need to share.

I hope I’ve helped answer some of your questions, I’m sure others will chime in with good input here as well. Thanks for participating on the COW!

Regards,
-Caspian
https://www.studionetworksolutions.com

Product Specialist
Studio Network Solutions

[Bob Zelin]

replies below, but before I read one word of your text, you are going to PAY SOMEONE MONEY to help get this working.
Replies below –

Keeping it short, I need a fast storage (about 800MB/s read write speeds) and probably available to 2 systems (to begin with). It is my understanding that I would need a Fibre SAN system to achieve this.

REPLY – no shared storage system is going to run at 800MB/sec, nor do you need these speeds. This is the speed of a modern drive array, not your network connection speed.

My questions are:
1) I keep reading that SAN reads the files block based while a NAS is file based, or something like that…what is up with this block, file thing ? Does it have something to do that with a SAN your files show up as a normal partition to your operating system while with a NAS they show up as…what exactly ?

REPLY – when you buy Super Share, you will have to spend $1000 per copy of SOMETHING on each client- like XSAN, MetaSAN, CommandSoft Fibre Jet, or SANmp. Some of these are file based, some of these are volume based. You do not get this software when you buy Super Share. It’s extra.

2) Using a NAS doesn’t let 2 people access and work on the same file at the same time, can this be possible in the first place as long as a NAS doesn’t present itself as a normal disk to the system, this is why one would use a SAN for this type of thing ?

REPLY – a simple NAS, like one based on Apple file sharing, does allow multiple people to use the same file at the same time.

Also in big networks do NAS devices function on their own, I mean users connecting to them through their browser and getting the files they need as oppossed to being controlled by a server ?

REPLY – you enter your account names and passwords into a server, and then you can connect to your shared storage. SAN’s allow for direct connection to the storage, without going thru a server (except for XSAN).

3) Do all SAN solutions need a separate computer which acts as the server (or metadata controller ? is it the same thing) that controls the SAN software which manages the whole system,

REPLY – some systems use a server, some systems don’t. Fibre Jet does not require a server for example, XSAN does. There are lots of examples. You want SuperShare – this does not require a server, but requires SAN management software at $995 per copy.

can’t I have the computer that i’m editing, compositin etc. on act also as a server and control the SAN (of course in this case it would always have to be turned on for the SAN to function) ?

REPLY – no, you cannot edit on your server computer.

If I have the software installed already on the server can’t I just connect to the server and see the shared storage there as a normal disk drive and start using it, do I also need to have a copy of the software installed in my client machine ?

REPLY – some yes, some no. Super Share requires a SEPERATE COPY of software on each computer. Using SANmp with the EVO product, you get 4 SANmp licenses, and 2 iSCSI licenses. XSAN requires a seperate licenese for every computer. Final Share does not require licenses, because it uses File Sharing, but uses a server computer.

4) In one of Bob Zelin’s tutorial articles that I found on the Cow websites he shows how to design your own SAN ( through Ethernet ). After i’ve read the article I didn’t quite understand the following.
You use the multiple ethernet port card to connect to the SAN storage box. So wouldn’t the storage box need 4 ethernet ports to be able to make this connection or did I understand things wrong and you actually connect to the SAN with the help of the managed switch in which you plug the 4 cables and you also plug into it the SAN storage box and the other editing computers ?

REPLY – you don’t understand many concepts of networking. The 4 ports are used for Link Aggregation. It’s ok if you don’t understand what that is, and even if you did, I wouldn’t show you how to do it.
The storage box (the drive array) is attached to a server computer, and in the same server is the 4 port or 6 port card, which goes to a switch. This is how the client computers connect. We use a managed switch, which costs $1200. No, the $300 Netgear switch does not work.

If we do this I still do not understand how will the 4GB link aggregated bandwidth be available to all of the comps because you only have this bandwidth between the server computer and the switch. There were some users that commented that actually the link aggregation arhitecture doesn’t in fact give you 4Gb/s bandwidth if just gives 4 1Gb/s separate sessions and that you would in fact obtain the same result by ditching the switch entirely and connecting directly from the server to the clients if you do not have more clients than ethernet ports on the card.

REPLY- You need to CALL ALL THESE COMPANIES to investigate. You will never build a system by yourself without help. I dont’ care how smart you are. Ethernet connection will do 50MB/sec. Ethernet with jumbo frames or with iSCSI will do about 90MB/sec. Fibre will do over 200 – 300MB/sec. And 10 Gig ethernet will do 350MB/sec with jumbo frames enabled. All you need to worry about is the resolution you are working at, and the number of streams you need.

Also what does MetaLAN actually add to the whole mix ? The management of data needed to keep the bandwidth up ?

REPLY – this is SAN management software.

Another user said it offers volume sharing vs file sharing , what’s that ?

REPLY – Volume sharing means one guy writes to one volume, file based sharing is that multiple people can write to a single volume.

5) In the Final Share solution and similar solutions, software such as MetaLAN/SAN is not needed ?

REPLY – true, Final share uses Apple File Sharing, it does not use MetaLAN.

Does the server take on the responsabilites and management of the bandwidth ?

REPLY – no, the bandwidth is based on your network connection. Final Share uses ethernet and jumbo frames. You get 90MB/sec per client.

And shouldn’t you use the link aggregation on all the computers that are attached to the storage to be able to benefit from the bandwidth ? The shared storage is accesible like a normal disk in these systems ?

REPLY – you can’t link aggregate a client, only the server to the switch. That’s the rules – believe me, if you could get more bandwidth with 2 ethernet cables to one client, we would be doing it.
Want more bandwidth that 90MB/sec per client – fibre or 10 gig ethernet.

6) Can I build such a system with a PC server ?

REPLY – no

YOU WILL FAIL IF YOU TRY TO DO THIS YOURSELF. Call Final Share, Call SNS, Call Cal Digit, Call everyone. GET HELP. You will look like a fool if you have your company spend money, and try to piece it together yourself. Everyone asks for help, and you need help.
All of the companeis you mentioned will help you set up these systems. Stop your “researching”, and CALL THESE COMPANIES, and ask direct questions about these systems. Then choose what works for your budget, and you will be a hero. If you try to piece this together yourself, you will fail, and be fired.

Bob Zelin

[Mike Nicolau]

Thank you both for the lengthy and patient answers.
Bob believe me i’m not trying to do anything on my own and know that trying to do everything by yourself leads to doing nothing right but one week ago when people started telling me about SANs NAS etc. I realised I need to have at least a basic grasp of what these terms mean before I start looking for the people who offer this solutions to the market. So now I at least can have a conversation on common grounds with any storage manufacturer. I also have the utmost respect for the people with the experience and know how of making these systems work together and integrating them into the right workflow.
So…that being clarified my need for such high speeds come from the fact that i’ll be working with up to 4K resolution material and speed hungry formats like DPXs and such. Maybe not 800 read/writes but somewhere close to that.
I’ve found some SAN systems that claim they can achieve this speeds and i’ll look into it is the CalDigit solution can also support this.

The reason I was asking so many questions about your “building your own SAN” tutorial is because I might even want to go with this solution for a personal project and because there was this user saying this:
“EXCELLENT article, thanks for sharing! My studio has a very similar setup to the one you describe in this article. The primary difference is:

– No MetaLAN software (which appears to be Windows only)

– No 4-port ethernet card on the server. All machines (xserve and 8 Mac Pro clients) use the included 2-port ethernet cards, aggregated into single 2Gbe connections

– PROAVIO EditBox EB8ML 8-drive array in RAID 5 mode

Using this setup, we get 70-90MB/s connections on all machines, depending on which speed tests you trust more (Aja Performance Test vs. Blackmagic Disc Test).

I was wondering, is there a particular benefit to using the 4-port card vs. this setup? We do get rare situations where a workstation will suddenly slow to a crawl, usually when 5 or more people are working on the same project/footage. Would you guess this has more to do with the lack of a 4-port server card, or more to do with sharing assets with no SAN software in the mix?”

and I would just simply like to understand how did he also got this speeds and what are the drawbacks of his systems.
As to my “link aggregation” concept misunderstandings there was this other user who commented on your article:

“Fantastic article, but there’s one very important thing you left out. If it’s a small shop with 4 edit stations or less, you do NOT need nor want Link Aggregation.

For a 4 station setup, what you do is assign one port to each client and use subnet masking so that each port is on it’s own virtual private network. You can use any cheap gigabit switch, or no switch at all (just plug it straight in). This will give the identical performance, without the headache or extra cost of doing Link Aggregation. Link aggregation balances all the links to all the clients, so it’s useful if you have more clients than you do ports on the server. But if you have 4 or less stations and 4 ports, it’s pointless.

Another thing. If you aggregate 4 ports, you don’t exactly get a single 4gbps port as you suggested. Instead you get up to 4 sessions at 1gbps max each. In other words, say you use Link Aggregation on a Mac client and link 4 ports together. Then connect to a server that also has 4 linked ports. Your max bandwidth for a single video stream is still only 1gbps.

Just wanted to clear that up. When I first heard of Link Aggregation, I too thought it was the same a having a 4gb ethernet connection. Unfortunately it’s not the case.”

Again i’m not trying to question your methods i’m just respectfully asking if it’s possible for some clarifications about this.

Also on my question about whether it’s possible to build such a system around a PC server you answered no, but reading the article again I saw that it is also possible to have a PC act as the server ?

best,
Mike

[Bob Zelin]

my need for such high speeds come from the fact that i’ll be working with up to 4K resolution material and speed hungry formats like DPXs and such. Maybe not 800 read/writes but somewhere close to that.

REPLY – this is done with local storage, not on a SAN. You CONFORM with 2K and 4K, you don’t edit in 4K. You don’t edit at 800MB/sec.
Do you know how much drive space this takes up? Are you rich ?

I’ve found some SAN systems that claim they can achieve this speeds and i’ll look into it is the CalDigit solution can also support this.

REPLY – good idea. Call Cal Digit directly, and ask them.

The reason I was asking so many questions about your “building your own SAN” tutorial is because I might even want to go with this solution for a personal project and because there was this user saying this:

REPLY – this is for a LOW BUDGET SAN. Not for doing 800MB/sec.

– No MetaLAN software (which appears to be Windows only)

REPLY – this is wrong information.

– No 4-port ethernet card on the server. All machines (xserve and 8 Mac Pro clients) use the included 2-port ethernet cards, aggregated into single 2Gbe connections

REPLY – this is wrong information.

– PROAVIO EditBox EB8ML 8-drive array in RAID 5 mode

This drive will do about 600 MB/sec on a SINGLE dedicated MAC or PC.
This bandwidth splits using shared storage.

I was wondering, is there a particular benefit to using the 4-port card vs. this setup? We do get rare situations where a workstation will suddenly slow to a crawl, usually when 5 or more people are working on the same project/footage. Would you guess this has more to do with the lack of a 4-port server card, or more to do with sharing assets with no SAN software in the mix?”

REPLY – I am tired of answering your questions on this forum. Why don’t you CALL ME, call the other companies, and ask questions. Why 200 questions on a user forum ?

and I would just simply like to understand how did he also got this speeds and what are the drawbacks of his systems.
As to my “link aggregation” concept misunderstandings there was this other user who commented on your article:

“Fantastic article, but there’s one very important thing you left out. If it’s a small shop with 4 edit stations or less, you do NOT need nor want Link Aggregation.

REPLY – this is wrong information.

For a 4 station setup, what you do is assign one port to each client and use subnet masking so that each port is on it’s own virtual private network. You can use any cheap gigabit switch, or no switch at all (just plug it straight in). This will give the identical performance, without the headache or extra cost of doing Link Aggregation. Link aggregation balances all the links to all the clients, so it’s useful if you have more clients than you do ports on the server. But if you have 4 or less stations and 4 ports, it’s pointless.

REPLY, then why don’t you get a 4 port card and try it. Go to Staples or Office Depot, get a cheap switch and TRY IT. Exactly what are you looking to accomplish here ?

Another thing. If you aggregate 4 ports, you don’t exactly get a single 4gbps port as you suggested. Instead you get up to 4 sessions at 1gbps max each. In other words, say you use Link Aggregation on a Mac client and link 4 ports together. Then connect to a server that also has 4 linked ports. Your max bandwidth for a single video stream is still only 1gbps.

When you link agg 4 or 6 ports together, you get a large data pipe between your switch and server. Your client only gets 90MB/sec PER CLIENT with a single ethernet cable.

Just wanted to clear that up. When I first heard of Link Aggregation, I too thought it was the same a having a 4gb ethernet connection. Unfortunately it’s not the case.”

REPLY – that is correct – you get 90MB/sec with jumbo frames.

Again i’m not trying to question your methods i’m just respectfully asking if it’s possible for some clarifications about this.

REPLY – if you call me, and if you call the other competing companies, they will explain to you on the phone .

Also on my question about whether it’s possible to build such a system around a PC server you answered no, but reading the article again I saw that it is also possible to have a PC act as the server ?

REPLY -anything is possible. I have used MAC’s to do this. If you want to research this yourself, then go for it. You originally stated that you wanted to go with Cal Digit Super Share, but I get the impression that you want to spend ZERO money, and build this yourself, and be the big hero. All I can say to you is good luck with your research. Do you know what happens when you do research – you spend a LOT OF TIME AND MONEY just like all of us did, figuring it out. And if it actually works when you get done, you have a product that you can sell. If you only have interest in building a single system for your company, it will be cheaper and easier just to buy something.

I am here if you want to talk. So are all the other companies.

bob Zelin

[Steve Modica]

Hi Mike,
How this all works is a lot more complicated than you might thing.

For example, FCP and Quicktime use AIO (asynchronous IO) and other things like Finder don’t. The AFP daemon uses multiple threads to handle reads, but it uses Pread and Pwrite rather than AIO. Of course, each application and codec might have its own IO size too.

The story of block access, File based (NAS) access and shared block access (clustered filesystems) is long. I’ve got a number of articles out there. Here’s one that’s recent.

https://www.postmagazine.com/Publications/Post-Magazine/2010/December-1-2010/The-NAS-vs-SAN-argument.aspx

What you’re going to be doing is asking a server to do something a billion times a second in coordination with a switch and a client computer (also required to do everything a billion times a second). Really small 1 in 10,000 glitches kill performance. Since you are editing video, this is unacceptable.

In a way, TCP messes me up. It enables “green light syndrome”. You can hook up any computers over just about any switch and TCP will deal with the errors silently. You won’t know anything is wrong. You’ll get green lights, you’ll see your files and all is right with the world til you drop frames.

In the tougher world of FCoE or AOE where TCP isn’t involved, you don’t get that easy pass. IO errors occur and problems get figured out quickly.

Poke around for a few more of the articles I have out there and you’ll get a better idea of how all this works. We worked at SGI/Cray during the invention of Clustered filesystems like cxfs and Xsan.

Steve

Steve Modica
CTO, Small Tree Communications