-
Finding ALL Duplicates?
Hello Everyone,
I hope you guys can help!
I am currently working on a massive project using catDV Pro 8.1.5 and CatDV Server. Due to the amount of videos imported (about 100 000 of them, usually imported by batches of 500 to 10 000 in a catDV XML File), and due to the fact that the Worker Node has proven highly unreliable for automatic imports, we have ended up doing the import manually, and therefore created a big number of duplicates (3-4000) along the way.
I do not want to have to delete them manually (although I am at the moment!).
Can anyone explain to me how I can find ALL DUPLICATES of the clips within a single catalog? as in, which ones in that catalogs are there twice…By duplicates, I mean clips that either:
– have the exact same name (we have a naming convention at import making sure all names are unique)
– and/or point to the same file (same media path)
– and/or have the same x/y/z (a few other fields that are unique to each video as well).For some reason, when using the “Find Similar” option, I either get a bunch of files that are not only duplicates (meaning I still have to double-check all entries, which is counter productive) or I get no result.
The only way I managed to make it work is to do it with a single entry selected (therefore making me checking all entries individually, which again is counter productive).
The only explaination on how to use the “Find Similar” option is this:
Find Duplicate Clips
The Tools > Find Similar command will find duplicate clips which are similar to the selected clips based on a particular attribute. It will either search all the open catalogs in memory or compare the current catalog with the CatDV server. For example, you might compare on File Hash or Media Signature to see if anyone has already imported a particular media file.
Can anyone help? Please? You’d make my day a lot better! I’ve been on this for days. And I did try to speak to Rolf, but he’s now away I believe. So, I’m hoping I get more luck here!
Thanks!
-
9 Replies
-
Try doing this with “media signature” field as it is fairly accurate and compares multiple fields. Then sort in list view by this and you can pretty quickly see groups of multiple duplicate clips. Keep in mind in CatDV these are database records and not “media files” by default for example the delete key only deletes the record leaving the media file untouched. You may also choose too delete media files from the media menu but could be dangerous when the files reside in multiple project folders for example.
-
Matthew,
Thanks a lot for your answer!
I have tried this and it seems to partially work only. I am trying to find identical “database records” that also point to the same media file. In our case, there should be only a single record per clip. And actually, if there was copies of that same entry, they would still hold the same “Clip ID”, right? But I digress
The result should then be the clips that have duplicates and their duplicates… But I get also all the clips I selected in the first place on which I ran “Find Similar” option, which do not have duplicates.
And on top of that, the delete key (backspace on my mac) doesn’t allow me to remove the duplicated entry…
Does it work for you?
ps: all clips are spread across only 2 folders, and clips are distributed to a bunch of different catalogs right after import.
-
After you use the find similar command in the other tab in the tree there is a duplicates icon on the left you can open in a new window and delete from there. By default find similar is a read only result as a safety precaution. Duplicate database records are generally not a big deal as they take up very little space unless you have a inordinate amount or make a lot of thumbnails. I would rather get too many results of a query than none at all. Also with the copy and paste of metadata you can merge the metadata to ensure your records match! You may also want to try Media file as it looks for entries with the same path.
-
Thanks Matthew.
it has been a really painful process, but we managed to clear the system of duplicates somehow. Deleting clips from the query result (in editing mode) is not always working on here. Somehow, moving the clips from the query result window is also a no-no. I would have thought I could treat this search as a regular query and move/edit/delete any entry as I please.
Instead, we painfully went through all catalogs, searched for duplicates, marked them with a custom field specifically created for the purpose; we then moved them into a single catalog of duplicates where we could compare entries side-by-side and then delete the ones that were not needed or not up to date.
After that, we moved the lot back to the catalogs where they belonged to in the first place….
Thank god it’s over!
-
Were you getting an error when you tried to delete or move clips in editing mode?
-
yes, an exception error mentioning java: https://dl.dropbox.com/u/6320711/CatDV-Error-Deleting.png
Any idea why is that happening? -
Okay we will take a look and send me an error log CatDV that shows the error! Are you on a mac?
Home > Library > Logs > CatDV Error Log
Home > Library > Logs > Diagnostic Reports -
I am on a mac – I will search for those log files and send it to you if I have them.
-
Hi guys,
I wanted to add something: The catDV version we now have (8.1.6) has fixed all of this. We can now delete from queries really easily. The process is a lot smoother, and it is much better since we can delete duplicates as and when we find them, being in a query or while browsing a catalog.
The more I use catDV, the more I realize its potential. And although I still find it quite memory hungry (speaking of RAM here), it does the job pretty well.
Thanks for all your help anyway.
Best,Romain
Reply to this Discussion! Login or Sign Up