ZFS anyone?

[Bob Zelin]

Small Tree systems use ZFS.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
maxavid@cfl.rr.com

[Frank Wit]

Hi Marcus,

I have been fiddling around with Openindana and napp-it
started as a hobby project to explore ZFS

an old gigabyte (2008) ma-785 mobo and a phenom quad-core and only 8 gis of ram
6 x 2 TB seagate as a raidz
onboard sata and a supermicro controller and lsi FC controller
worked like a charm on windows machines,
however could not get the fibre channel card to work on my hackintosh,
replaced the FC cards with 4 port quad gigabit, same on the hackintosh.
SNS iscsi initiator on the mac.

works like a charm! read speeds via BMD and Aja system test around 280 Megabytes/sec sustained.
write speeds around 100 MB/sec.

because the machine was rock-solid exchanged the 2 TB drives with 12 x 3TB seagates RAIDZ2
had to change the supermicro controller because of the 2 TB limit, to a M1015 IBM in IT mode

no luck with jumbo frames so far…..

tried editing from the storage of this machine with FCP-7 and X no problems!
the trick is using iscsi instead of AFP/SMB/NFS there is to much latency here.

not bad for a 27TB nearline backup, next to LTO tapes of course!

kind regards, Frank

[Marcus Lyall]

iScsi sounds interesting.
Wasn’t thinking of using this for editing just yet.
But it’s tempting…

So far we’ve got an old 8 drive Proavio linked up to a Supermicro box running FreeNAS on a little USB stick.
Going to get a second Proavio hooked up this week and then look into getting a proper enclosure if it’s all working OK.

Got good speeds over gigE, but haven’t tried FCP / Premiere yet.

Dunno if SanMP would work on FreeBSD…

[Chris Murphy]

I get read/write speeds of 100+MB/s over non-special GigE network, not even jumbo frames, using NFS with an async export. I see more downsides to iSCSI than upside, mainly because it’s easy to configure incorrectly while NFS is easy to configure correctly. If NFS is incorrectly configured, the client won’t connect, or it’ll be slow, or maybe it’ll have random disconnects. If iSCSI is incorrectly configured, data can get corrupted and you won’t necessarily be informed of this. Also iSCSI is a block device which implies either SAN (multiple layers of additional complication) to share it with other users; otherwise as NTFS or JHFS+ formatted it can’t be shared with other users.

I’d guess most use cases are: sharing files among users, which thus points to NFS not iSCSI; or pushing files to NAS for longer term storage in which case the performance differential is unlikely to be worth considering.

[Frank Wit]

Hi Chris!
you sure have a point here!
make sure you know what you are doing when it comes to iScsi!

my situation:

3 seats, always editing from LOCAL storage 4×1 TB raid-0, great read an write speeds, but VERY unreliable when it comes to a HD crash.
so the nearline backup is the ZFS iscsi which is ONLY connected to 1 seat.
this seat shares the storage via AFP to the other 2, still getting 100 MB/sec transfer rates..
with TCP offloading one doesn’t notice performance issues in a normal everyday situation on the 1st seat that is sharing.
on the other hand “very unreliable” these sets have been running for 3 years without any HD crashing…
but I am pretty paranoid about losing data, this is why the ZFS nearline and LTO are in place.

kind regards,

Frank

[Frank Wit]

still contemplating my first storage adventure on my first Powermac 9600…
8x 10k 9GB drives via an Atto UW card and getting 40 MB/S… with 25.000 dollars for 72 gigs of storage in 1997….
life is sweet!

Frank

[Chris Murphy]

Yeah in terms of price/performance, direct attach storage is unbeatable.

I think NTFS/JHFS+ over iSCSI on ZFS could be confusing what aspects benefit from ZFS. If the server drives’ ECC were to not detect read errors, ZFS would catch and correct for it even while hosting a guest filesystem within a ZFS sparse volume. But if there were iSCSI link level corruption, ZFS couldn’t protect from this. And as neither NTFS nor JHFS+ have either data journalling, nor use checksums in the metadata or metadata journal, they can’t even protect themselves from such events (not that this is limited to iSCSI of course, the same issue applies to direct attach storage with these file systems).

Another advantage of NFS over iSCSI in a ZFS context is the NFS export metadata for a zpool is included in ZFS metadata. So when exporting the file system (ZFS send/receive), the NFS configuration follows. Really though, they are quite different and it comes down to what you need it to do rather than it being only about performance considerations.

Interesting you get 100MB/s via AFP over GigE. I seem to top out at 55-65MB/s on the same physical setup as NFS async over GigE where I get 100+. But good to know that it’s possible. I think I’d rather have oral surgery than configure Samba.

[Bob Zelin]

Chris writes –
I think I’d rather have oral surgery than configure Samba.

REPLY – well guess what Chris, we will be doing exactly this (and not oral surgery) as of OS X 10.9 in a couple of months, with everything becoming smb2

And I get similar results with NFS that you got (glad to see that it’s not just me).

Bob Zelin

Bob Zelin
Rescue 1, Inc.
maxavid@cfl.rr.com

[Chris Murphy]

Oh yes, this. I’m not really sure what to think of it yet. On the one hand Apple should make the UI rather simple. And it also ought to mean they feel their home rolled version has matured enough they can ditch AFP. (Or it means they need a psychiatrist.) But SMB isn’t simple.

If the workflow is an all Mac situation, it Should Just Work, right from the GUI. The cases where it might have problems would be integrating with Windows or Linux or BSD on the other end of the connection. And if I’m out in the weeds without a GUI push button make it work solution, I think I’d rather get Samba built on the Macs, with its significant pile of documentation and support community, rather than learning Apple’s 3rd flavor of SMB under the hood.

Also, Microsoft has been contributing to the Samba project for a while now, so that ought to be working pretty reliably with either 3.6.16 or 4.0.x. I suspect Apple’s implementation is a distinct subset of SMB capability which is why there have been so many problems since 10.7.

[Marcus Lyall]

So, one question.
Any recommendations on a sata drive enclosure.
Looking for something with…

either 24 or 36 bays.
sata multilane connectors.
redundant power supplies.
economical.
rack mount

We have a supermicro rack server. Apparently you can pull the motherboard out and put it in a raid-style chassis. But would prefer not to. Unless it’s relatively painless.