Chris Murphy
Forum Replies Created
-
As long as you understand how to revert the change to this command in case it makes things worse, I wouldn’t do it. It means editing the nsmb.conf file and removing the two lines that this command adds.
-
Backup for the system+apps+user preferences = Time Machine, or Apple Software Restore (asr is the CLI command, accessible via Disk Utility for GUI). ASR creates a disk image that you can store on any file system.
Nearline (faster than tape) backup for media files = Carbon Copy Cloner, or equivalent. If you search this forum, within the recent past others have made suggestions for other solutions that also manage organization, including telling you what drive a certain media file is located on. CCC won’t do that, it’s just a folder/file copy program.
I don’t know what you mean by 4 + 4 JBOD, or how you’re creating it in OS X. If you mean two 4TB drives in a linear/concat arrangement, so that they’re one big 8TB volume, I probably wouldn’t do that because it means that any single disk failure effectively means you’ve lost all of the data on that volume, so it has the same risk as raid0 but without the speed. But if you have 2nd backup or an archiving strategy, then you can of course get away with some additional risk with the 1st backup layout if the linear/concat arrangement makes the workflow easier for you. But if you only have one backup, then no I wouldn’t do either linear/concat (JBOD?) nor would I do raid0.
-
Yes, sparse bundles can be used for Time Machine backups – this is how Time Machine backs up over a network. There are guides on the internet on how to create them with hdiutil. The problem is that once configured, backupd automatically makes an authenticated connection to the remote host, and also mounts the disk image file, does the backup, then unmounts the disk image. All automatically. You won’t get this behavior locally, you’ll have to manually mount (double clicking) the disk image and probably always just leave it mounted. So now you have two mounted file systems, one on top of the other, and if you have a crash or panic during the backup there’s a higher chance of one of the file systems being corrupted. And Time Machine backups, I find, are already really sensitive to file system problems which is why I don’t consider them worthy of being an archive. It’s just a matter of time before a Time Machine backup starts to exhibit problems, and before it exhibits problems backing up, it’s not assured it won’t have a problem with a restore.
Time Machine is the backup/restore strategy I wish worked. I use it because it’s brain dead simple, and if the boot drive dies and the restore works, I’m up and running much faster than alternatives. But I simply don’t trust it to be the only backup, and I don’t trust it at all to be an archive.
What you’re best off using Time Machine for is making a “snapshot” of your boot drive, OS, applications, user settings, so that if the boot drive dies and is replaced, you can restore and get up and running again quickly. I’d suggest turning off Time Machine in between OS/Application updates because on a production machine, not much is changing so what are you really backing up? Not much, yet backupd consumes a lot of system resources when it runs each hour which I notice degrading system performance in the most basic use case, not even doing live video production work. So I’d think that Time Machine going off every hour would be a show stopper for video people.
For your important files, those don’t even belong on a boot drive. And I wouldn’t have Time Machine backing them up. I’d use something like a scheduled rsync script, which Carbon Copy Cloner can do for you. Set it to backup once a day outside production hours, maybe over lunch, or maybe shortly after the end of normal working hours. Or possibly with a logout script.
-
This is a confusing post.
1. 90 and 40 gigs ~20 mins via usb3.0
It makes no sense that a 90GB and 40GB transfer would take the same amount of time. But 40,000 MB in 1200 seconds is only 33MB/s which is slow. Even 90GB in 20 minutes is 75MB/s which is not exactly a speed demon either.2. 10800 seconds to transfer 90,000 MB is 8MB/s. Obviously something is wrong. This is the sort of transfer rate you get with a.) bad drive; b.) bad connection; c.) improper alignment of either the raid to the physical sector, and/or file system to the raid; d.) small file random I/O. e.) other.
So that leaves c and e. And I don’t know what e would be. And c is a pretty obscure problem to have these days, and HFS+ doesn’t even permit alignment options because it really isn’t designed/optimized for use with RAID.
What sort of results do you get from Blackmagic Disk Speed Test for each drive? You can get this from the App Store for free.
-
I’m not recommending anything, just asking a question. Tape is not suitable for random access, the latency is huge (find the tape, load the tape, wait, search for what you want, select what you want, wait for the restore). However, once what you’re looking for is found, tape is as fast as a single drive – 120+MB/sec transfer rate. Even with an autoloader, which are expensive, it’s still going to involve quite some delay (minutes, not seconds). So don’t get tape as a random access device. Use it for backup and archive: stuff you know you need to keep, or you don’t know for sure you can purge.
LTO isn’t really comparable to DAT. DLT supplanted DAT. And LTO has supplanted DLT.
Time Machine is designed to backup/restore an OS, your applications, and some user data. It’s not really designed for your use case. The backupd consumes a lot of CPU, it does it every hour. If you’re willing to hack the preferences file to change the backup timing to something rational like noon and 6pm – or just 6pm – that’s better that this crazy daemon going off every hour and sucking the life out of system resources while you’re working. I’m not sure what use an hourly backup in this workflow would be, but if it’s working for you, fine.
However, don’t bet the bank that these Time Machine backups are going to last very long. I somewhat regularly have them totally implode on me – and that’s just backing up my laptop with an SSD in it. Pretty much once or twice a year I expect to reformat the Time Machine destination volume and start it from scratch. So it’s most definitely not a reliable archive. It’s a fairly reliable short term backup for an OS, apps, app prefs, and some local user files. For massive video files I’d use something else for your “nearline” (faster than tape) backup to drives: Carbon Copy Cloner or Super Duper, maybe someone here has advice on that. CCC is rsync based (it has its own current build of upstream rsync in the app, it doesn’t use the ancient version OS X ships with although you could use that too, or build a newer one from Macports), and Super Super is ditto based. CCC can leverage an rsync checksumming feature to confirm the source and destination are the same, although this is slow. The checksums aren’t stored anywhere as far as I know, it’s just used to independently confirm/deny the copy’s success.
-
Btrfs is still under much development, so it’s not production comparable to ZFS yet, but is getting closer to stable. The Btrfs single, raid0, 1, 10 code has been in the linux mainline kernel since 2008. The raid5/6 code went in almost a year ago and still needs some work.
File system creation takes less than a second. There is no parity initialization. Additional disks of any size and number can be added, online. The device add occurs in less than a second. Online restripe is supported, but not required. Upon new device add, the allocator writes across all present drives. There’s preliminary work on supporting object level raid, I’m not sure if the idea is per file granularity or only at the subvolume level. Devices can also be replaced online which causes their data to be migrated to the replacement.
-
Why not tape?
-
If the idea is to avoid most storage stack related file corruption and make it easy to use, the solutions are: a.) use enterprise drives, since they have an order of magnitude fewer unrecoverable errors and corruptions than consumer drives; and/or b.) use a file system that checksums every block and uses data redundancy; any detected corruption causes the replicated data to be used instead. Examples are ZFS and Btrfs. Of course, this means non-local storage.
It also does nothing for ingestion. There are no file system mechanisms with exFAT, NTFS, or HFSJ/HFSX to detect or correct data corruption. So you’re at the mercy of the media being used by the camera, the card reader, its drivers, the local computer and its drives, and the network. Ideally, there’d be ingestion right at the storage stack itself to limit the opportunity for corruption to occur.
-
find -type f -name *.jpg -exec md5 {} +
Instead of jpg put in the file type you’re after.
-
Yes good point and actually it’s likely SMART testing won’t work through the USB bridge chipset anyway to see if there are reallocated or pending sectors.
This performance sounds like something is causing literally every file to be seeked at once, so the drive is just seeking to death. Read one or two big files at a time. Much bigger than 100MB. You need something about a gig in size for reads, and 2x RAM for writes to make sure OS caching isn’t affecting the performance stat. But these days writes should be the same as reads.
A linux live CD will have tools that are useful for getting a decent (basic) idea of unbuffered drive read performance with hdparm or dd. dd is also available on OS X, use a bs size of at least 128k.