[Steve Modica]

one of the nice things about OS X is Dtrace. You can run various Dtrace scripts to see what’s going on.
iosnoop is one such dtrace script. With -D it will show you all of the IO going through the system and the latency associated with each. We use that to evaluate storage solutions and determine whether they can sustain video streams. Most can’t.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

Small Tree loaded up when we heard about the flood. We have drives in stock for our storage solutions.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

In a terminal, type:

cd /var/log

Then type:

open .

This will open a finder. Then open kernel.log with a text editor and see what’s at the bottom when this is going on. There might be driver errors. Cable length is one possibility.

Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

I’m guessing cable length. Commands hangings will cause lots of weird stuff. Look in /var/log/kernel.log

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

Atto had a bug with one of the latest driver revisions where the raid would come up with a drive degraded (even tho it wasn’t). We caught this in testing. They also had a bug with NCQ that would take a drive out.

When we see a drive fail in an unusual way, we get them back to Hitachi. We’re an official OEM and they send it through an analysis to tell us what happened. Our two most recent cases were faulty head actuator and a bad midplane in the chassis (power sagging).

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

I’d find the disk utility process using ps and get its PID. Then dtruss it:

sudo dtruss -p PID

While that’s running in a terminal, hit the erase button.
One possibility is that apple issues the scsi “erase” command and the drive simply doesn’t like it or is locked.
In either case, you should see the failing system call in the output.
Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

most raid technologies store their setup data on the drive volume headers. The idea is that they can move elsewhere or even move to a new controller (of the same brand) and function.

In my experience, some vendors intentionally don’t mount raids when they are moved to a new system because it’s assumed the volume may be shared. So you have to explicitly mount it the first time. (For example, you may have to tell Atto to automap the device).

I don’t know about apple’s mechanism.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

Avid uses their own inhouse protocol running over UDP, so they can stripe things out however they choose (including over two ports). They also use TCP at times when they have offload cards available (non-mac).

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

If you know exactly what you want to test, nothing beats FIO. We support the code on mac platforms in the open source community.

Aja and blackmagic do not use the right APIs for simulating most of the things we care about. iozone and iometer are not detailed enough for me either.

Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

You shouldn’t be using desktop drives in a stacked RAID like that. They don’t have rotational vibration sensors and you may be hitting heat issues as well. (although putting server drives in, you might overrun your power supply)

You have to look at the atto logs and see why the drives degraded. Only then will you know what’s really going on. I believe atto “fakes” media errors for timeouts and you’ll get some other error for heat related issues. I’m betting on heat.

Steve

Steve Modica
CTO, Small Tree Communications