[Francois Stark]

Hi Kevin

I kept thinking Mac – but thanks for this suggestion. I’ll look into it. Question: Can you go out of another decklink card at the receiving end? I need another composite signal.

Regards
Francois

[Francois Stark]

I think I found the last remaing problem with the SAN:

I tested how long it takes between the dropped frames and it was exactly 6 minutes – the time I set up for SanMP’s volume sync.

So I disabled SANMP’s volume automatic sync, and the problem went away. Since then, I’ve captured long pieces without any problems.

This morning’s first 55 minutes must have captured fine because the SAN was empty. When it’s empty, the volume sync must be fast enough not to cause dropped frames.

So now we will start using the SAN properly – I hope.

Regards
Francois

[Francois Stark]

I just realised the playback problems happen with even three streams. While I was typing, the fourth stream was paused. It was playing three streams reading 60MB/s for a random time, when it would drop frames, data rate drops and then shoots up to 120MB/s where it stays for about three minutes. Then they just continue like before.

It’s like something gets stuck on the array, and causes this dropped frame on playback. It could be the same thing that is causing the problem for capturing from next door. But then, why did the first 55 minutes capture OK this morning?

Regards
Francois

[Francois Stark]

Hi Bart and chrispy

I’m just gonna ramble on about my experience because I find that typing it up makes you think deeper about the problem…

Well, last night I thought I had messed around enough with the Raid 0 test LU’s. I deleted all the LU’s and restarted the ADTX array. Created two new 5 drive arrays, one on each controller. Created 6 LU’s on each array – all raid 5.

While the first LU’s were being initialised ( the others were”waiting to be initialised”) I could access all the LU’s from the G5’s! I tried creating stripe sets in the disk utils, but that did not work properly, so I went home to let the raid 5 LU’s finish initialising.

This morning I found the LU’s had finished initialising. I started up in 10.4.1 Tiger on the G5, but Tiger seems to exhibit similar problems on the G5 as os X.3.9 did on the G4: All the LU’s are visible in the system profiler, but only one port on the ADTX array’s LU’s are visible in the disk utility. So I went back to os X.3.8 on the G5 and could see all the LU’s in the disk util.

Bad news for Tiger migration… The Astera cards might work but the apple fibre card is really not working well yet in Tiger.

Then it took me about 30 minutes just to figure out which LU’s must be striped together, because most of them are the same size! I had to format them as seperate drives, copy stuff over and see where the data goes using the fibre switch software – one by one. Eventually I figured out how to identify which array and which LU is which disk is in the disk utility.

Then I striped the corresponding LU’s from each array together and started doing some tests (journalling off – yesterday I had it on). Yesterday I could copy stuff from one stripe array to another at 55MB/s (raid 0 stripes), today it ran at 50MB/s (raid 5 stripes). OK, I guess. Raid 5 makes it about 10% slower.

Setting up the LU mapping in the ADTX: I set it up so that
1) every port has one LU 0 – so the G4’s can see all the LU’s
2) every real LU only shows up on one ADTX port
3) every real LU only shows up on the port of it’s own controller
4) I have 12 LU’s it total, 6 on each array
The LU mapping looks like this:
Port 1: 0 – 1 – 2 – – – – – – –
Port 2: – 0 – 1 – 2 – – – – – –
Port 3: – – – – – – 0 – 1 – 2 –
Port 4: – – – – – – – 0 – 1 – 2
(PS Only people who have a ADTX array will recognize this LU mapping)

Then I set up FCP to capture to the one Sanmp disk (stripe raid 50), SD PAL uncompressed using the Decklink Xtreme on a G5. Checking on the fibre switch performance graphs, I could see the G5 is splitting the data over both fibre links, 10MByte/s each, and the ADTX is also getting the data on two ports at 10Mbyte/s each, one on each controller. I captured about 55 minutes in 10 minute clips without problems.

Then I started some playback tests. Yesterday I could play back 8 streams uncompressed on the raid 0 stripe, today I could to 6. I continued capturing, while playing back 4 streams on the next G5 machine.

Then something went wrong. I don’t know what. The capturing started dropping frames – abort capture. At random times, between 1 min and 5 minutes from start of capture. The SanMP capture disk, mounted as read-only on the second G5, started acting strangely. It would show the folders on the drive, without icons. As I click on the folder name, it would vanish. Freaky. I restarted the read-only machine and it restored disk access to the sanMP disk.

I restarted the capture machine three times. Continued dropping frames on capture to the SAN disk. Trashed preferences. I even changed capture drive to the boot drive and it continued dropping frames as long as I had the sanMP disk mounted. Only after another reboot and by not mounting the sanMP disk at all, would the system capture to the boot drive wihtout dropping frames.

I’m sitting here on the read-only machine and it can not even play back four streams SD. It starts off fine, but after about 2 minutes it drops frames. Badly. And the capturer machine does not even have the SanMP disks mounted. No other machine is mounted. Yesterda it also dropped frames on capture, but I though that could be related to the journalling that I left on when creating the stripe sets. It seems I was wrong.

On the playback machine I playing four of the captured SD files (apple unc422 8 bit) in quicktime at the same time – looped. Plays at 80MByte/s. Then after a random time it drops fames and the datarate momentary drops, whereafter it shoots up to 130MByte/s as quicktime tries to catch up. It stays in this state indefinitely, reading from the array at 130MByte/s while continuing to drop frames in all 4 windows. When I pause one window, the others start playing normally after about 10 seconds – datarate drope to 60MB/s. Bottom line, it can not handle 4 streams continously, even though it has much higher datarate capability.

I checked the drive lights on the front of the ADTX to look for “sticky” lights indicating a drive with many retries slowing the ADTX down. Nothing strange – all lights blinking similarly.

I wonder if this has to do with the SanMP automatic sync. I set all my drives to ON for read and write, with a 360 second volume sync interval.

So I’m stuck again…

Regards
Francois

[Francois Stark]

Hi Bart
thanks for this discussion.

[Bart Harrison] “You can share a Mac OS Extended, FAT 32, or NTFS volume via any networking topology you have available.”

When sharing over SMB to PC’s we can only access the user;desktop etc. We can not see any non-boot drives. So we’d have to get os X server software.

[Bart Harrison] “7 drives”

Working towards that.

[Bart Harrison] “Look Ahead Table to 128 Entries”

I’ll try that today.

Regards
Francois

[Francois Stark]

That’s what apple says. I don’t have the tools to test it, but have found two issues regarding rendering accuracy:

-when using Joe’s filters color glow, I get banding on 8 bit YUV setting, but 10 bit YUV then banding is gone, even when it is a DV timeline!!!

-when combining two filters – I think it was joe’ glow and 3 way CC, I switched to floating point rendering, and got some rendering errors on the visual highlights. Like the numbers “punched through” the brightest levels and replaced it with darker values.

I just changed the timeline to 10 bit rendering and rerendered. Problem gone.

Regards
Francois

[Francois Stark]

Hi Bart

I found where Jeff Bernstein talks about the change from os X.3.7:
——————————————————————

Name: Jeff Bernstein
Date: May 27, 2005 at 04:44 gmt
Subject: Re: Not solved yet – seems G5 is better.

Okay, this is starting to sound a little familiar. I have a feeling you are running OS X 10.3.7 or greater. This could mean you are not running the latest firmware on the ADTX in which half the LUNs would not mount due to the way OS X changed the behavior in LUN discovery.

Another thing to note is that there is a nasty bug on the ADTX when used with SANmp. If your ADTX needs to rebuild, you not be able to recover the data. We just discovered this the hard way. ADTX is working on it and I welcome you to apply some pressure for I would hate for this to happen to you.

Let us know how it goes.

Can you tell us which OS and which rev of the firmware on the ADTX?

Jeff

Jeff Bernstein

Digital Desktop Consulting
Apple Pro Video VAR
XSAN Certified

323-653-7611

[Bart Harrison] “set them up as an NFS volume”

Why NFS? Maybe NTFS? for PC graphics machines? (we all make typo’s sometimes…) or is it really NFS?

[Bart Harrison] “we usually see well over 300MB/sec read and write at every workstation” Today I got about 180MB/s on the decklink disk speed meter. Remember, only 10 drives (two 5 drive raid 0’s striped using apple disk util), not a fully populated ADTX yet.

On your ADTX settings: What is your setting for cache pre-read streams? Mine is set to 32 at the moment, default is 8. If I remember right it can go up to 128? I’m not interested in HD streams (yet), I need as many SD (20MB/s)uncompressed streams as possible.

Thanks
Francois

[Francois Stark]

Hi Bart

Thanks for the encouragement – I seem to be sorting the problems out one by one. Sad about having to go back to an earlier version of os X, but someone said apple changed fibre port discovery from os X.3.7. So it seems I can go back to X.3.6.

I don’t know what’s going to happen with tiger – I know SanMP is not ready for tiger yet, but that cannot be far off. And if this problem is not solved, I’ll be stuck in Panther on the G4’s. And FCP 5 does not run on X.3.6; it needs at least X.3.9…

Q: Did you notice any difference between G4 and G5 machines on the Astera cards? What version of os X are you running?

I’m at home now, so I’m not sure about the rev of apple fibre card. Will check tomorrow.

On the G5 that froze during capture: I’m running the Apple fibre card (LSI) in the top slot (PCI-X 133MHz by itself), and the decklink extreme with the Atto UL4D in the second and third slots (sharing PCI-X 100MHz). I copied some files off the Huge 320S array to the fibre array and got 135MByte/s read from Huge, and 135MB/s write to the Fibre array. No crashing.

The only time it froze was during capture. I’m going to try the decklink card in the top slot (to isolate it from other cards), and the LU4D and Fibre cards in the second and third slots.

Hopefully, if I can get this fibre array working very reliably, I can sell the UL4D and Huge array – which means I will only have the fibre card and Decklink installed.

struggling on…
Francois

[Francois Stark]

Here I re-did the post because the previous links worked for our internal LAN only:

Our G4 was running on os X.3.9:

All 8 LU’s in the system profile but only 2 in the disk utility.

And here I tried it in Tiger X.4.1:

In Tiger also all 8 LU’s in the system profile, but only 2 in disk util.

So here I did a raw installation of X.3:

And suddenly I could see all the LU’s in the disk util.

Finaly I went up to os X.3.4 and this is what I saw:

All the LU’s in disk util! I installed sanMP and could mount and read and write to the shared fibre arrays.

Conclusion: Os X.3.9 and X.4.1 are not able to see LU’s properly on a ADTX array on a G4 mirrordoor. Os X.3.8 works in G5 machines.

At the moment I’m getting 6 streams SD from the 10 drive raid 0 array (two 5 drive raid 0’s striped by apple disk util). I can play 6 streams on one G5 in FCP real time, or two streams each on three machines (two G5’s and one G4). This is quite reliable – I looped these six playing streams for about 1 hour.

However I have had three crashes on our G5 while capturing SD PAL to the SANMP fibre raid. FCP stops with disk IO error, and then finder also does not see the SanMP disk anymore. Everything freezes and I have to do a hard reboot. SanMP recovers after it’s 15 second delay because I did not unmount properly.

So there is still a reliability problem when capturing – more tests to follow.

Regards
Francois

[Francois Stark]

Our G4 was running on os X.3.9:

All 8 LU’s in the system profile but only 2 in the disk utility.

And here I tried it in Tiger X.4.1:

In Tiger also all 8 LU’s in the system profile, but only 2 in disk util.

So here I did a raw installation of X.3:

And suddenly I could see all the LU’s in the disk util.

Finaly I went up to os X.3.4 and this is what I saw:

All the LU’s in disk util! I installed sanMP and could mount and read and write to the shared fibre arrays.

Conclusion: Os X.3.9 and X.4.1 are not able to see LU’s properly on a ADTX array on a G4 mirrordoor. Os X.3.8 works in G5 machines.

At the moment I’m getting 6 streams SD from the 10 drive raid 0 array (two 5 drive raid 0’s striped by apple disk util). I can play 6 streams on one G5 in FCP real time, or two streams each on three machines (two G5’s and one G4). This is quite reliable.

However I have had three crashes on our G5 while capturing SD PAL to the SANMP fibre raid. FCP stops with disk IO error, and then finder also does not see the SanMP disk anymore. Everything freezes and I have to do a hard reboot. SanMP recovers after it’s 15 second delay because I did not unmount properly.

So there is still a reliability problem when capturing – more tests to follow.

Regards
Francois