Checksums for entire AVCHD or Canon XF folders?

[Tim Jones]

md5 theClipFolder

Retraction – I just discovered that the OS X md5 utility won’t operate on a folder…

[Chris Murphy]

find -type f -name *.jpg -exec md5 {} +

Instead of jpg put in the file type you’re after.

[Tim Jones]

It appears that the only way to get this to do what Seth is looking for is to use ‘shasum’ to create a checksum list for the original media folder, copy that file into the destination folder after the copy, and then run the ‘shasum -c’ option to compare those sums against the files in the copy folder.

mount the card (/Volumes/CanonCard for this example)
cd ino the mount point
find . -type f -exec shasum {} >> checksums.sha \;
copy the clips to the new location
copy the checksums.sha file to that same folder
cd into the new folder
shasum -c checksums.sha | grep -v “OK$”

That should only print out the files that don’t match.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.productionbackup.com
BRU … because it’s the RESTORE that matters!

[Tim Jones]

Another option would be to use the “diff” utility:

Mount the camera card as before
copy the clips to the new folder
diff -q /Volumes/CameraCard /Volumes/Array/NewFolder
that will only print the names of the files that differ.

Tim
—
Tim Jones
CTO – TOLIS Group, Inc.
https://www.productionbackup.com
BRU … because it’s the RESTORE that matters!

[Chris Murphy]

If the idea is to avoid most storage stack related file corruption and make it easy to use, the solutions are: a.) use enterprise drives, since they have an order of magnitude fewer unrecoverable errors and corruptions than consumer drives; and/or b.) use a file system that checksums every block and uses data redundancy; any detected corruption causes the replicated data to be used instead. Examples are ZFS and Btrfs. Of course, this means non-local storage.

It also does nothing for ingestion. There are no file system mechanisms with exFAT, NTFS, or HFSJ/HFSX to detect or correct data corruption. So you’re at the mercy of the media being used by the camera, the card reader, its drivers, the local computer and its drives, and the network. Ideally, there’d be ingestion right at the storage stack itself to limit the opportunity for corruption to occur.