ArriRaw to LTO

[Martin Greenwood]

You should be limited by the Firewire 800 speed, this is probably around 50MB/s. The LTO drive will try to speed match slow sources however this may just be too slow. So the tape may be overshooting and rewinding. However 20Mb/s or 2.5MB/s is very very slow !
Ideally for shuttle drives use USB3 or Thunderbolt. With either of these interfaces you will see the full drive speed which is normally around 100MB/s for a portable drive. The LTO will work OK at this speed.
Ideally LTO6 would want at least 160MB/s source speed.
You could try the demo of our OS X software YoYottaID LTFS to see what speed it copies to tape. You could also use YoYottaID to copy from the Firewire drive to another faster drive, this will give you a real world idea of the max transfer of the ARRIRAW files from that drive. Please let us know how you get on.

Martin Greenwood
CTO
martin@yoyotta.com

[Neil Sadwelkar]

We do a lot ArriRaw to LTO backups using Bru-PE and we too have observed the same slowness with ArriRaw compared to RedRaw. But the difference using Bru is not as different as you’ve mentioned. We too use a 2009 MacPro with a SAS LTO-6 drive.

We keep the report that Bru puts out at the end of the backup which mentions start and end times for backup and verify. And from that in an Excel sheet, we derive the time taken.

Checkin my logs for recent backups I find that a single eSATA drive can backup around 130-150 MB/sec and verify can happen at 150-200 MB/sec when dealing with large contiguous files – Red Raw, QT, Smoke Archive etc.

With ArriRaw or DPX files – large number of small files, also over eSATA, the speed drops to 100-120 MB/sec for backup and 60-100 MB/sec for verify.

These figures are substantial smaller with 2.5″ drives (over any interface) and for Firewire connected drives. Even Firewire RAIDs don’t deliver anything over about 80 MB/sec.

I haven’t tried with LTFS much, but I’m guessing there’s a performance penalty for a large number of small files over a small number of large files.

———————————–
Neil Sadwelkar
neilsadwelkar.blogspot.com
twitter: fcpguru
FCP Editor, Edit systems consultant
Mumbai India

[Andrew Proude]

“I haven’t tried with LTFS much, but I’m guessing there’s a performance penalty for a large number of small files over a small number of large files.”

That’s correct, LTFS and tape is really bad at handling small files.
Best solution is to TAR the folder first, then write the package to tape.

[Martin Greenwood]

Andrew is correct that small files will transfer slowly to LTFS. However note that small means less than a few MB. ARRIRAW is 8MB or greater and these images sequences will transfer to LTO-6 up to the full speed of 160MB/s. If compression is enabled when formatting an LTFS tape then you will see speeds up to 10% faster. This is because ARRIRAW unlike most camera formats is uncompressed.

I just ran a test using our YoYottaID LTFS app copying 300GB of ARRIRAW to LTO-6 and the average speed was 160MB/s. This includes the time when the drive has to update indexes etc.

As mentioned at the start of this thread this depends on the source drive being able to source this speed. A single spinning drive whatever the interface will probably not be fast enough.

Martin Greenwood
CTO
martin@yoyotta.com