Chris Gordon
Forum Replies Created
-
I’m expect this will be out of your comfort level, but I’ll throw it out there anyway.
My personal favorite tool for this kind of thing is rsync. It’s an extremely powerful unix command line tool that can do exactly what you want and a whole lot more — I’ve built an entire backup system around it. If you move lots of files or keep lots of things in sync, it is probably worth the learning curve. Incidentally it was originally written by the author of Samba.
-
The only reasons I can see that going to 8 GB in a MBP (a model that supports 8 GB,that is) would cause crashes are:
– Some odd firmware problem that Apple has. I really doubt this, but there could be some bug.
– Poor quality memory. I’ll second Thomas’ recommendation for Kingston. I’ve had very good luck with it in a number of different machines at work and home (home built PC’s, Macs, exorbitantly expensive IBM UNIX servers). When I got my 2008 MacPro (back in 2008), I bought some lesser expensive memory (the price gap was huge between Kingston and this other memory and I saw lots of posts of good luck with it) and have had it throw lots of ECC errors over time and a number of the DIMMs just completely die. I’ve since put Kingston memory in my machine (much better prices). The other vendor RMA’d all of it for $0 and no hassle (actually very impressed with that customer service), but I’m reluctant to put it back in.
– You just aren’t giving enough electrical power to your machine. Remember that RAM takes electricity. With a system that now need more power than before, previously acceptable voltage fluctuations may not be acceptable any longer and cause various issues. The point where the Mac switches to battery on a low voltage may be lower than the power you need to run with 8 GB of RAM — I don’t know the details here, so I’m doing a bit of guesswork. Solve this with a good UPS (don’t be cheap here!).Hope that helps.
Chris
-
The ATI HD 5770 seems to be compatible with the 3,1. This is the lower of the two cards currently sold in Mac Pros and as an upgrade kit for others. Not sure why the 5870 isn’t.
https://store.apple.com/us/product/MC742ZM/A?mco=MTY3ODQ5OTY
-
Long Answer:
The OS has limits on the maximum number of file handles that can be on on a system wide, per user and per process basis. I’m guessing you’re hitting the max files per process limit (kern.maxfilesperproc) which defaults to 10240. You can see the limit by using “sysctl kern.maxfilesperproc” at a shell prompt. “sysctl -a” will show you all sorts of limits and settings of the system. Some/most of these you can change with sysctl -w parameter=value. Some must be set at boot time, so you put the entry in /etc/sysctl.conf (this will make a change permanent, too) and reboot. Some limits are hard coded into the kernel. You will also need to look at kern.maxfiles. I’m not sure if these values are 32bit or 64bit integers. Assuming 32bit, the biggest I’d expect this to be is a touch over 65,000. I’m also pretty sure you’ll chew up more kernel memory in doing this — especially if you can get to your 600,000 target. Remember that when you start messing with these settings, you’re playing with a loaded gun. Adjusting kernel settings is something you want an experienced Sys Admin to do based on well thought out understanding of the impacts of the change. Don’t complain if you shoot yourself with this loaded gun.
Shorter Answer:
Don’t try to open 600,000 files on your system. That’s an insanely huge number of open files and sure to cause you some other kind of heart ache. Since this is an FCP forum, I assume you have Final Cut. Use QT to create multiple time lapses with more reasonable number of files per video and then pull those videos into Final Cut and combine together.
Also be careful with the number of files you put into a single file system or directory. All filesystems hit a point where their performance starts to go down the tubes with huge numbers of files in a single directory, some with total number of files in the file system, regardless of directory location. If really want to store 600k+ files like this, you really need to read up on the internals of HFS+ (the Mac’s file system) and understand how best to accomplish this. My guess is that you’ll want multiple smaller file systems with few files in them. You may be able to also do this with a number of disk images — essentially nesting file systems on top of each other, but performance probably won’t be so great.
Good luck.
-
You need to consider a couple of factors in determining what is the “right” configuration:
– What performance do you need. You really should do some math and figure out what data you’re going to want to support. Look at the data rate of the video streams and the number of streams you’re going to work with simultaneously.
– How much data protection (if any) do you want? Depending on the number of disks and RAID level chosen, you can survive a different number of drive failures before you lose data
– How much space do you need? Different RAID levels result in different amounts of usable space.
– How much money do you want to spend?Now for a bit more theory and general comments:
So what does RAID do for us? You really get 3 things from RAID:
– Better performance. This comes from striping data across multiple drives. A given disk can handle only so much throughput — both in terms of data rate and in the number of IOPS. By spreading your data across multiple drives, you get increase performance since each drive only has to service a fraction of the total IO work.
– Increase reliability. This is done by either mirroring or parity. This essentially makes a second copy of your data across your array (or at least the ability to recreate your data). Of course you lose “usable” disk space when doing this.
– Larger volume size. You may need a volume larger than a disk, so you need to combine several disks into one to give you a big enough volume.What are the different RAID levels and what do they mean in practical terms? (See the wikipedia RAID article for more details)
– RIAD 0: This is a simple stripe across all of your disks. Like David said, for a given number of disks, this gives you the most performance since all disks are contributing to the IO work load. Of course there is redundancy here, so the loss of a single disk kills the entire RAID set.
– RAID 1: This is simple mirroring of disks. Two disks are (essentially) exact copies of each other. When you write to the volume, each disk writes the same thing. You only get half of the raw space as usable for storage with mirroring.
– RAID 5: Striped parity (see wikipedia for details). With RAID5, you lose essentially 1 disk worth of storage for parity. A big performance caveat of RAID5 is the parity calculation. Any write you make requires parity to be calculated in addition to performing the actual IO operation. Modifying data also incurs additional performance overhead in that you have to read all of the data in the stripe being modified to perform a new parity calculation. Then you have to write the new data and new parity. You need to make sure your RAID controller (for hardware RAID) can keep up with this.
– RAID 6: Similar to RAID 5, but with two different parity blocks. This is a common solution to recovery issues with RAID5 arrays (see below).
– RAID 10: This combines RAID 1 and 0 (also called RAID 1+0). In this scenario you take pairs of disks and mirror (RAID1) them and then create stripe (RAID0) across the mirrored pairs. This gets you the best performance and redundancy at the cost of space. Depending on how nice Mr Murphy is, you could lose half of the disks (just not two in a mirrored pair) and have no data loss.In practical terms, you really only see RAID5, 6 and RAID10 for significant data storage with RAID1 hanging around for boot/OS/System drives.
What happens when a disk dies?
– RAID 5: You keep running, but with degraded performance. Whenever you have to a block of data that was on the failed disk, you have to recreate that data from parity resulting in some performance degradation. When you replace the failed disk and rebuild the volume, all of the data on all of the disks must be read to either populate the new disk (recalc the parity or recreate the data). This can take a long time and has a performance hit across the entire array. The heavy work placed on the remaining drives in the array can help Mr Murphy appear and give you a second failure during the recovery resulting in loss of data.
– RAID 6: Similar to RAID5, but you have a second parity bit and can suffer 2 disk failures before data loss vs the 1 disk in RAID5.
– RAID 10: When you lose a disk, you keep running without any degradation in performance. The rebuild only requires reading from 1 other disk and no parity or other calculations, resulting in a faster recovery. The performance of the array during the rebuild is only for data on the mirror set being rebuilt — you may have IO contention on that one disk since it’s being read for the rebuild.My preferences (take as you like):
– I don’t like RAID5 with disks bigger than 1 TB. Too many vendors and storage experts I work with consider the double failure too likely. I also don’t like that the performance degradation during recovery and long recovery times. Of course I see a lot of people on these forums happily using RAID5 and I see a lot of video specific storage vendors proposing it, so I may just be overly conservative.
– My preference is RAID 10. I get the best performance and recovery and disks are relatively cheap, so total volume isn’t so much of an issue. -
The Quick Time Export will create a reference video which really just points back to all of your original sort material and whatever you’ve had to render out. This should not affect quality at all, but the video won’t play on anything other than your machine since it doesn’t actually contain the video clips, but just points to them. Quick Time Conversion will encode everything into a completely new file with whatever settings you choose. You can then hand this file to anyone to watch (or post to YouTube), but there is some amount of loss, however small, due to the encoding pass. The amount of loss depends on the settings you choose. Something like Compressor (as Walter suggested) should give you even more control over the output and probably better quality than the QT Conversion in the editor. If you’re going to go the Compressor route, then the reference movie is a better option since you only encode the video once. Going through QT conversion and then to Compressor (or MPEG Streamclip) results in multiple encodings and the potential for more data loss.
-
You were actively using the disk and it was unplugged? This mean you never properly unmounted the file system. HFS+ is supposed to be a journaling file system which should help maintain consistency in these cases, but from my experience it’s pretty poor. Any time you don’t cleanly unmount a file system (eject the disk, clean system shutdown, etc), you should run a file system check and repair anything amiss.
For your firewire drive, you can use the Disk Utility GUI and run “Repair Disk” on the drive. You need to run the repair repeatedly until it comes back with no-errors. One pass may not get everything. You can also do this from the command line with either diskutil or straight with fsck (which is what diskutil actually uses). The man pages on diskutil and fsck can give you more information.
If you had to shutdown your Mac uncleanly (is that a word?) — like pull the power — you need to check the file system, of your boot disk. The trick is that you can’t do this when it’s mounted read/write, i.e., when the OS is running. You have two options here. (1) Boot to single user mode by holding down “command-S” when booting. This dumps you at a shell prompt and even gives you the syntax for the the fsck command. (2) Boot from your OS X DVD and run the GUI Disk Utility from there.
Hope that helps, and good look. Remember, you should always back up data to other media!
-
Those are little SOHO NAS arrays. Perfectly fine for sharing some documents or music at home or in a small office, but not for any heavy IO work. They typically have rather weak CPUs in them (a single core Atom if you’re lucky) which has to service all of the IO, manage the RAID (mirroring, parity, whatever you set) and run the CIFS/SMB server (assume that’s what you’re using). For the amount of data you want to move, this thing just can’t keep up.
What I would recommend doing is to move from a NAS solution to something directly connected. This means an enclosure to hold the drives and a controller that sits in your computer to connect to the enclosure. What you get is going to depend on your budget. You can go to somewhere like Newegg.com and get the parts and build it yourself with the drives you have from your NAS or buy solutions. Bob mentioned a couple of companies in his initial reply and there are others such as G-Technology or even Dell and HP (look at the server storage solutions). I’d recommend getting a packaged solution (with support) instead of the build your own (to get good performance and reliability, you need some expertise).
Some other things to consider:
– Never fill your drives all the way full. First, any file system needs free space to maintain itself. Typically you never want to go over 80% full, but that can vary. Second, you end up taking a performance hit as the drive gets more and more full. Remember a disk is a spinning platter and you have to move the drive head over it to read or write. With more data on the disk, you’ll probably be reading/writing to it more. That means more time you wait for the head to get positioned properly.
– You have SATA drives now. These are good for holding large volumes of data cheaply and tend to perform relatively well for large sequential IOs. They can’t keep up with extremely high volumes of IOs or a lot of random IO. In those cases you need to move to faster disks — SAS or FC (there are disks that connect internally to the array with FC and this doesn’t mean specifically a piece of glass coming out of your PC to an array, though it typically ends up there) spinning at 10k or 15k RPM. These drives are smaller in capacity and more expensive than SATA disks.
– Which type of RAID are you using and why? Each of the RAID types has trade offs between performance, cost and redundancy. The Wikipedia article on RAID can give you some background on the different types. Also remember that RAID is NOT a backup solution. You are backing up your data somewhere, right?
– You may want to hook up with a storage expert to help set you up. If you’re in a university, there are probably people on staff there that could help (bribe them).Hope this helps a bit more. There isn’t a magic bullet here, each solution is a balance of what you need and can afford.
Chris
-
Bob,
Thank you for the compliment.
I work for VeriSign as a System Architect designing highly scalable and highly available (at least we hope) infrastructures. (Corp disclaimer: anything I say is my thought or opinion and in no way connected with my employer.) My LinkedIn profile is https://www.linkedin.com/pub/chris-gordon/1/541/788.
With respect to video, I’m a bit of a neophyte. It’s a hobby that I’ve dabbled with from time to time and increasingly enjoy. I’ve been following a number of forums on the COW and a few other places to educate myself on the art and science of video.
Thanks,
Chris -
A couple of questions first:
– Which NAS are you using now?
– What are the specs on it (number/type of drives, type/number of CPUs, amount of cache, etc)?
– Have you verified that the NAS device itself is not IO or CPU bound, not just the pipe between your machine and the storage array?
– Which NAS protocol are you using (SMB/CIFS, AFP, NFS, iSCSI)?
– What is the nature of the IO? Reads and writes? Sequential or random?
– You say “PC”. Does that mean your machine is running Windows or is it running some other OS?As you probably know, this is all a game of moving bottlenecks around. That means you need to find out what part of the chain is slow and fix that, then move to the next part until everything performs as you want. There are several places to look:
– Make sure your host is not memory/CPU/bus constrained (you’ve already done this).
– Look at the storage array itself. You need to first check to see if it is constrained somewhere. Remember, these aren’t magic devices, but just computers built for a very special purpose — servicing IO requests quickly and efficiently. Within the array, or any storage system for that matter, you get your speed by two methods. First is the number of disk spindles you throw at it. A given disk can service only so many IOPS before performance goes down. The more disks you spread the load across the more IOPS you can support. Second, you use cache (RAM). If you have a lot of bursty writes, you can absorb that in cache and get really good performance. In the low write periods, the data is then de-staged out of cache and written to disk. Cache can also help with reads if you read the same blocks repeatedly (say a reference table in a database). The array (assuming its OS is worth a darn) will hold that data in cache and service the read requests from there instead of constantly having to read from disk.
– Look at how you access the array. There are typically two ways to do this: file access protocols and block level protocols.
— File level protocols are things like SMB/CIFS, AFP or NFS which run over IP and present files to the client. File access locking and management of the file system itself are done on the array. These are typically simple and cheap to setup and run and can perform will in many/most cases.
— Block level protocols (Fibre Channel, SAS, iSCSI, eSATA) present a “raw” LUN to the host and the host manages the file system, etc. These are typically far more complex to setup and manage, but you can typically get much better performance out of these. The performance increase often comes from (a) more efficient protocols and (b) you are only sending the data/disk blocks back and forth between the host and array whereas with file access protocols, there is a lot more data going back and forth.So what can you do (assuming your only bottle neck is the network connection):
– Assuming the only bottle is your network connection, then increase that. Either bond several 1 GigE interfaces together (different OS’s call it something different: bonding, etherchannel, link aggregation, trunking, teaming etc) or move up to a 10 GigE network. If you do either of these, you need to make sure you can do something similar on the NAS side along with your network switches. If you go the bonding route, be sure that both your host and array will load balance across the different NICs and not just do simple fail over.
– Ditch the NAS and get a directly connected array with a decent RAID controller. This would put a lot of disks (spread out the IO) directly connected (local disks) to your host.
– Move to a Fibre Channel based solution. You’ll need your array to support FC, an FC HBA for your host and possibly a FC switch. Similar to ethernet, you can aggregate multiple FC connections to get increase performance. If you are going to go this route, you need to get someone that has experience with FC and SANs. It’s a bit of a different world than simple IP connections and you can easily shoot yourself in the foot — especially since this stuff isn’t cheap. This should be your last stop as its by far the most expensive.Hope that helps.
Chris