Sparse Bundle Users: How do you backup?

[Jeff Kirkland]

Just to keep a sort-of apples and apples comparison, I downloaded AJA System Test and ran it again…

Volume: Thunderbolt (LaCie 2big 4tb RAID 0 about 50% full)
Video Frame Size: 1920×1080 (8bit)
File Size: 1.0 GB
Write: 258.4 MB/s
Read: 260.9 MB/s

Volume: Sparse Bundle (1tb / 24gb in use)
Video Frame Size: 1920×1080
File Size: 1.0 GB
Write: 252.1 MB/s
Read: 282.7 MB/s

Jeff Kirkland | Video Producer | Southern Creative Media | Melbourne Australia
http://www.southerncreative.com.au | G+: https://gplus.to/jeffkirkland | Twitter: @jeffkirkland

[Andreas Kiel]

@ Jason
That’s a pretty poor result.

I never got results that had been less than 15% off the volume speed.
Since sparsebundles are only directories (with a little overhead) there is no reason why the should be way slower than the volume itself.

@ Jeff
I personally never got a corrupted sparsebundle and I’m using them for many things since many years (even before X).
I don’t know about Chronosync as I a use CCC since the day it was released. But there can’t a big difference between those apps cause both (and all backup apps) just use the system Unix calls.
Normally during the backup the checksum of files are compared so nothing can happen (in most cases) if the source file is okay.

A good idea though is to unmount sparsebundles during backup times, they might be busy for whatever reason and this might influence the way they are “backuped”.

-Andreas

Spherico
https://www.spherico.com/filmtools

“He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee.” – Friedrich Nietzsche, Beyond Good and Evil

[Brett Sherman]

For me the sparse bundle is okay, media is okay, but the event and project files can get corrupted. This has happened on more than one occasion. This is with the bundles unmounted. I think there is an issue with incremental backups of sparse bundles and the way FCP X modifies the database files. I’m not 100% sure, but I think it may be more likely to happen when the backup sparse bundle is launched and opened in FCP X in between backups.

[Dave Gage]

Brett,

This may have nothing to do with your issue, but have you tried running DiskWarrior on the Sparse Bundle? I don’t use them, but I seem to remember hearing that DiskWarrior should work find on Sparse Bundles as well as standard drives.

Early on with FCPX, I major issues with external drives used solely for use as FCPX drives getting corrupted to the point where I had to wipe them clean and start over. I then began to use DW regularly and have not had a problem since. As I said about a year and a half ago, it “appeared” that FCPX was hosing my external RAIDs, which might have translated into the drive directory structures becoming a mess. All is fine now.

Dave

[Craig Alan]

[Dave Gage] “This may have nothing to do with your issue, but have you tried running DiskWarrior on the Sparse Bundle? I don’t use them, but I seem to remember hearing that DiskWarrior should work find on Sparse Bundles as well as standard drives.”

I’ve heard that too but is there any issues with DiskWarrior and FCP X file structure? Now that I think about it, I used it on my system drive when running FCP 4-7 where the project file was on the system drive and never had an issue. But FCP X has a completely different type of file. Hmmm. Gonna do a search.

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.

[Dave Gage]

[Craig Alan] “Gonna do a search.”

Cool, let us know what you find out. By the way for the record, I’m still on 10.6.8 and FCPX 10.0.4. I really doubt the file structure has changed since it’s debut, but I have no idea. You might consider contacting the DW Tech Support at Alsoft. The senior engineers there are awfully helpful and knowledgeable.

Thanks,
Dave

[Craig Alan]

Google was no help but this might be:

From Diskwarrior tech support:

“As for DiskWarrior, yes it can be used with drives that contain Final Cut Pro X
media. However, a project file being damaged while in a .sparsebundle may or
may not indicate a directory problem.”

“To Rebuild a .sparseimage or .sparsebundle, the image can be mounted or
unmounted. Simply, select “Rebuild Disk Image” from the File menu within
DiskWarrior.”

“Is it possible for you to send both a working as well as damaged project file? ”

I did not have a corrupted project to send them. But here’s the address if anyone cares to:

Alsoft Technical Support

Mac Pro, macbook pro, Imacs (i7); Camcorders: Panasonic AG-HPX170/AG-HPX250P, Canon HV30/40, Sony Z7U, VX2000, PD170; FCP 6 certified; write professionally for a variety of media; teach video production in L.A.

[Andreas Kiel]

The most important thing is that you use sparsebundles — not sparseimages.
Sparsebundles in most cases even can be repaired with Disk Utility.
But I do really recommend to have Disk Warrior in your toolset — it’s a must have!

A corrupted “CurrentVersion.xxxx” in most cases has nothing to do with the location where it is saved. But a disk might be bad and on a bad disk a sparsebundle (remember it’s a folder) will work as bad as any other folder.

So finally you should have a backup strategy like having your projects or events saved quite often to a different file — a different file on a different disk is even better. You can do that within FCPX by duplicating the project and collecting used things into a new event or you can do that by using ProVersioner or my X-Files Pro which both will only save events/project without anything around like render files etc.
Next thing is to keep media at a safe place and organize them upfront before using them. If media need to be transcoded put the transcoded media in a new folder at a safe place and start from there.
Having a strict organization will allow a good backup strategy using backup apps which allow additional scripts.
Another (not cheap) recommendation is to have a disk/raid for each of the disks/volumes. If you let your machine 24/7 running the OS will try to repair and defragment things (means also you should have at least 20% free space on the disk/volumes). Fragmentation will happen anyway as really big files won’t be touched. So a safe copy to a wiped disk is the only safe way and will make your system more responsive — take the weekend off, have some fun with your family or friends.

Just some notes.

-Andreas

Spherico
https://www.spherico.com/filmtools

“He who fights with monsters should be careful lest he thereby
become a monster. And if thou gaze long into an abyss, the abyss will
also gaze into thee.” – Friedrich Nietzsche, Beyond Good and Evil

[Brett Sherman]

[Andreas Kiel] “A corrupted “CurrentVersion.xxxx” in most cases has nothing to do with the location where it is saved. But a disk might be bad and on a bad disk a sparsebundle (remember it’s a folder) will work as bad as any other folder.”

I don’t think that’s the case here. I’m not sure what the problem is exactly. But somehow my backup with sparse bundles is causing corruption in the event and project files. Not all the time, but enough to be a concern and not rely on.

For me, ProVersioner is a buggy mess right now. To the point that it is impossible to work with. I’m using Backups for Final Cut, but don’t like the laborious manual setup process. Can’t wait to try out X-Files Pro when it’s out of beta.

[Matt Larson]

Looks like a lot of good info on this thread already, but I’ll add my solution (it’s worked well so far…)

I’ve been keeping my events on a separate image than my projects. This works well for me for two reasons: 1) I tend to have video that I use across multiple projects, so I can start a new project with an Event I’ve already keyworded, etc and not have the old project cluttering up FCPX; 2) The Images with the Events are large and don’t need to be backed up as often as the project files do. So I have just the project file on a small disk image that backs up quickly. I add a “_X” to the end of the file name to distinguish it from the event image, but this could be anything. (As long as it’s consistent it lets you do this next part.)

I’ve also set up an Automator action to make daily backups of any file that ends in “_X.sparsebundle” to another drive.

There are also some Terminal commands that let you either resize the Image to make it larger if you need to, or compact the image and reclaim free space when you have deleted Render Files or Transcoded media before you archive the project. Reminds me of my days dealing with the Avid Unity system!

2 x3Ghz Quad MacPro
9 GB RAM
Mac OS X 10.5.5
QT 7.5.5
FCP 6.0.5
AJA Kona 3 (6.0.1 drivers)
G-Speed XL 12 RAID