Matt Geier
Forum Replies Created
-
Christian,
This all sounds fun! 🙂 As usual, I remain neutral in my posting, giving you information that will help you start more researching and get you thinking about this.
From a bandwidth perspective, I know that Ethernet can support the data rates you need running for SD and also HD (Compressed, and Uncompressed) This can also be scaled to match the amount of users you have on the network (even if it means multiple servers, and storage)
—- I don’t think however, bandwidth will be the bottleneck for you as much as I suspect you’ll need faster storage then what you have now or might be considering purchasing. Perhaps, even more then one RAID to support doing HD Uncompressed editing with multiple users at one time.
Whats interesting about what you want to do is that there are very little people doing this this way. A lot of the problem is that it’s very expensive to get it done right, not that it can’t be done.
Doing this in a shared environment or any environment is possible, but a lot of raids out there, don’t go as fast as they would need to in order to support the requirements to pass the stream input/ouput’s back and forth) — Most people are using Pro Res, or sticking to Direct Attached Storage for their HD Editing.
Here’s some things to consider…
(8/10Bit HD Uncompressed x 5 Streams)
10Gb Ethernet on a single Mac will allow you to go 250-500MB/sec. However, in order to support this many streams of HD Uncompressed at once, you will need multiple RAIDS to support multiple people. As well as bandwidth, the RAID will also need to be fast enough to pass the input / output going back and forth off the disks. (This will likely mean a couple or a few more 8Gb Fiber Channel RAID’s will be needed)Now consider this –
(Pro Res HQ x 5 Streams)
Gigabit Ethernet on a single Mac will allow you to run up to speeds of 90MB/sec. Please note: This is the same bandwidth as running 5 x HD Uncompressed streams, however, the RAID requirements are not as stringent, and this road will certainly cost the lesser of the two options.The idea here that I’m pointing out:
The amount of time it takes to pass an HD Uncompressed is much more stringent then the requirement needed to pass a Pro Res HQ stream.
Both of these examples can run in an environment that is shared simultaneously with Ethernet and Fiber Channel, or Ethernet with SAS/SATA Storage. (If configured with proper network, and hardware of course)
Having said that, as long as your network, servers, and raids, are fast enough and ALSO provides the needed bandwidth requirements, and assuming all the network configuration is properly set up, you’d be able to edit an HD Uncompressed stream over the Ethernet in real time just fine with 10gb. In the end though, you’d get more for your money, if you decided to do this with Gigabit Ethernet and used Pro Res for your compressions.
I don’t know of any RAIDS, other then perhaps multiple 8GB FC RAIDs that can support the HD Uncompressed requirement you are seeking…..
I would be weary of any solution out there, that is not already designed to meet your specific needs of Uncompressed HD editing (Ethernet OR Fibre Channel), and be even more cautious if you are building this on your own (unless you are an expert on things like Flow Control, RAID Performance, Network Performance, etc)
When you enter a world of real time editing in an environment with multiple people all accessing data at once, hitting the network at once, hitting a server at once, and more, then you have one shot only to get it right, if it’s not right, as far as a lot of people are concerned, it’s failed. This is obviously the part you want to avoid.
-
Hi Christian,
You were not real specific on the kind of editing you want to do.
What kind of video formats do you intend to use primarily?
Do you want Shared Storage which is capable of Real Time video editing, or are you looking for centralized shared storage to push and pull files from?I see that you have 3 Mac’s from the above.
If you are not invested in Fiber Channel, enough that you are open to the idea of doing an Ethernet based network with some very fast Shared Storage performing for Real Time editing, there are some inexpensive solution options for you.
Even 10Gb Ethernet may may perk your interest for some additional reasons to go Ethernet. Especially with Fiber Channel over Ethernet technology coming around and what it’s promising!
Keep us posted!
Matt G
-
Hey Bob,
I’ll make a slight adjustment to my statement from before –
10GBASE-T will work up to 55 m (180 ft) with existing Cat 6 cabling. In order to allow deployment at the usual 100 m (328 ft), the standard uses a new partitioned Category 6a (a.k.a “augmented Cat6”) cable specification, designed to reduce crosstalk between UTP cables, known as alien crosstalk. Some manufacturers of cable have released “Cat7” cable which is specifically marketed for 10GBASE-T compatibility and claims greater shielding beyond cat6a specifications.
Hope this sheds some light on why I was referring to specifying CAT6a for future 10Gb use.
-
Rikard,
If you can, spec in CAT6a (Different from CAT6) –
Cat6a meets the 10Gb specification which will be being used for such things as Fibre Channel over Ethernet, and likely more things in the future.
-
Chris,
What causes a bog down of a SAN?
Well that’s like asking what’s making it snow outside….. okay ….
Its generally never just one thing, it’s usually several things at once. For example, if you design the network to support so much bandwidth, but your storage doesn’t go that fast – that could cause slowness. This could be the opposite as well, the storage could be super fast (MB/s), but the network bandwidth is not that fast, therefore causing a bottleneck.
It’s also common for some devices to be sitting behind a Shared server, but the devices were not designed to be hit with large IO/s, but a bunch of small ones – like in a Point of Sale system …. depending on what the storage was designed to do, this could be a bottleck.
The RAID cards, they can cause slowness. The processors can cause slowness, the memory can cause slowness, the fiber channel connections can cause slowness, the communication between client / sever / storage and back, can be long, causing slowness (also referred to as latency)
Answering the question for you, is just going to generalize this. Most SAN’s that are slow, are having problems because of something in their environment, be it to many users, to many servers hitting one storage, etc etc…. it would require looking (reviewing logs, kernal i/o’s, etc) over the configuration to determine what’s going on.
Some of the Storage SANs are performing well with SD environments from what I’ve seen around. The problem is starting to creep up when people try to do something larger then SD format. Formats which require larger size IO/s, and require “back and forth” movement in a certain time period, otherwise the frames drop….
It’s good for people like me when all other people think about are bandwidth requirements. Since there’s more to it then that when your working with larger IO/s, I could care less about bandwidth in most cases, since that’s not typically an issue. 90% of the time, it’s Raid Cards, and Storage Devices which are causing bottlenecks for editing in real time.
For example, I know it’s entirely possible, probable, and works very well to edit up to 2 x Pro Res HQ streams on a single Gigabit wire
(about 60MB-70MB/sec needed for this (using pro res @ 30-35MB/sec as the basis)) —- if you multiply this by 4 people, now you need to push 240MB/s just to support 4 users running 2 streams each … So I know if I have 400MB/s of network bandwidth, the network will not be a bottleneck. However, what I also need to know, is also how quickly each of those streams need to move their peice of data before the next piece of data. This is usually done in miliseconds, and moves faster or slower depending on ALL the pieces of the hardware / network solution.If you just live in the MB/sec world, and you need to be realizing a world where you have Real Time needs, well, that “real time” requirement changes how you need to design and implement a Real Time Shared Solution to work without the hiccups we all hear about and hear so many people failing at some point or another. (Fiber Channel SAN, or otherwise…)
Did that help at all ???
-
David,
You’re simply asking about something that can be done both ways. (Speaking in terms of capability of a “SAN” network)
About AE, I don’t know specific requirements, however, in a world where perhaps Final Cut exists and you have properly implemented your hardware and network configurations, yes, you can share media files off the same storage where the project files reside.
Again though, this may not be the case with your situation, based on limitations of your actual application in use, and software/hardware installed.
If you are truly in a SAN, where you have multiple reads, and multiple writes going on at one time with many users, then you likely have some traffic cop software overlying on the network that will prevent people from reading and writing over each others files, and using files which are locked for projects, or media files which someone else is accessing. Another thing that might affect what you want to do, is the File locking, or Volume locking capabilities of your installation.
Again, this depends on what your set up is capable of in it’s current form, what room it has to upgrade or expand, etc etc etc …
Can you expand your current network description so we might all understand just how you’re set up now, etc etc… ???
Does this help any?
-
Ian,
Me again –
Are you running 10.5 or 10.6?You may be experiencing some kind of bug in Final Cut, AFP, or perhaps something else.
I’m hearing about this kind of similar problem on some of the other forums here on The Cow, and externally, and also through some other “grape vines” —
Post us back with your o/s and let us know.
-
Ian,
You should know that there are solutions available today that rely solely on the use of Ethernet, and on Apple’s AFP running on an Apple server (Xserve as well as Mac Pro) .
Apple’s File Protocol (AFP), is not “file sharing” — Apple File Sharing is not AFP.
AFP will allow you to transfer up to 90MB /sec on a Gigabit wire using Jumbo Frames (with respect to bandwidth) — reading all this, I don’t think, based on what you described about the file being corrupt, you have an AFP issue.
Apple File Sharing – allows you to share stuff for others to access (hence, sharing your storage volume)
In my mind (and I’m looking for some others that use AFP and no overhead software to chime in…) — If you were capturing from your capture system, to the server/storage, the server is managing that connection, inode numbers, and all the kernal and i/o information being passed to the storage device…. the server is the access point…not each editors workstation.
Corrupted data could simply be caused by a mismatch in information, or perhaps a bump on the table (vibrating your disks in the storage) causing it to have a bad write, or numerous other things, like even a bad tape.
It’s very effective and possible to have multi amounts of users sitting on your server editing, all accessing different projects from the same storage volumes, and sharing media, without them reading and writing over each other, causing corrupted data, or data that is overwritten.
In a fibre channel environment, where each user hits the storage on their own connection, you need SAN software to manage each of those people from reading and writing over each other.
In a Ethernet environment with AFP running on your Apple server, and a switch or multi port gigabit connection serving each of the clients bandwidth, each user needs their own account on the server, and then each user can edit video with final cut (while at their own mac, in real time, over the wire.) – WITHOUT ANY SAN SOFTWARE ….
I’m not confident that running out and buying “traffic cop software” is going to solve your issue.
You really need to be on the phone talking to someone who can help support you, except you’re doing this all on your own, which means you will be at the mercy of this forum, or at the mercy of your wallet when you need to talk to someone like Bob Z, or Jordan Woods, etc !!
Regards,
Matt G
-
Ian,
When you ask what’s wrong, that’s a good question. I say this because there could be several things wrong at once, or just a couple of things going wrong. It’s hard to say.
If you’d like to shed some light on a list of hardware pieces you are using and EXACTLY how you’ve set things up, perhaps someone would be willing and able to provide some additional input.
I’d like to point out something you said—
Even when I attach a computer directly to my Xserve via an ethernet cable I still cannot capture a DV stream. The capture as always starts and stops when expected (I am capturing a full tape), however the file is always corrupted at the end.
My reply to that (there is some room for error here, I’m not a FCP expert…)
If the capture is starting and stopping as expected, it’s not dropping frames. (Usually dropped frames means your “captures or edits” are failing) — Simply having a corrupt file could mean a lot of things, even perhaps unrelated to the “networking” you’ve done up to this point.It’s hard to say what the problem is without taking the time to provide some additional input from your side about what further things you’ve tried and the different things you’ve done to reach the point where you are now.
Can you elaborate further about how you’ve set this up, and what steps you’ve gone through to configure everything. it also wouldn’t hurt to let us know what equipment you are using, just in case someone has experience with this already and can shed some additional light on the subject.
Regards,
Matt G
651-209-6509 x 1 -
Michael,
What Nate told you, really is true, if you can afford and really want the management overhead for traditional Fibre Channel SAN implementations for your video editing. (Or pay someone a lot of money to do this for you…)
Don’t be afraid to consider Ethernet however, which is also a viable solution you can propose to your management team, and depending on what kind of work flow you need, can save you a load of money in software, hardware, and management overhead, and all the other hardware and software bits that are associated with a traditional SAN implementation.
What you should know —
Final Cut Pro supports File Level locking on the project files, while the media files can be shared between project files. — this means that all of your users can do their work on a single server, and allow Final Cut to do the project locking, leaving each user to be able to work on their own project files at the same time as the rest. (off one server .. or multiple servers, depending on how many connections you need.)The one thing i cannot speak to though, is how your Facilis storage will hold up under the real time conditions of the environment, assuming you’re successful in hooking it all together. – Test it, and start pulling video streams off of it, if and when it’s start dropping frames, that’s how many streams of that video format you can pull from it at one time.
I would be proposing a new solution dedicated to the Final Cut Pro video editors. At the entire cost of an Ethernet based solution, your looking at a fraction of what a traditional SAN will run you.
If you’d like to talk some more about this in further detail, I can certainly introduce you to some alternative solutions to your FC based SAN that you had previously. Ethernet based environments can still provide the performance you need for Video Editing, and with things like Fibre Channel over Ethernet coming, anyone in an Ethernet based solution will be geared to getting into the next phase of where Ethernet is really going!
Let me know….
Matt Geier
651-209-6509 x 1