-
Is it normal for Worker to take much longer to open/import than client?
I’m in the process of importing a SFX library into CatDV. All the files are in subfolders (for instance Animals, Cars, etc.). There are about 150 folders.
I figured I would automate the task somewhat, and create a drag/drop Worker task. Essentially it just creates a catalog with the name of the folder, and imports/publishes the contents of said folder to that catalog. No conversions, moving, anything…just read the file and publish it to the catalog.
So far so good.
The issue is that the process of opening and publishing the files takes about 5x longer (if not 10x!) than working in the CatDV client and selecting Import Folder to import the files into a catalog.
Is that normal? I’m guessing it may be because Worker is opening/scanning/publishing/closing…perhaps importing a folder through the client is more efficient where it doesn’t close the connection after each file or whatever? But importing a large group of files in the client by dragging them from Explorer is also much, much faster than Worker too.
I’ll just leave it here working for a day I suppose.
Thanks!
-
2 Replies
-
You are correct, in most cases.
The worker is treating each asset as an import, whereas the client app can import 100 and then save them as one transaction.
The database adds some overhead and so, in our experience, one save is faster than 100.
Also, you may have analysis steps turned on in the Worker that may not be running in the client. (check this and thumbnails and other such ops.)
We don’t usually use a Worker for large imports unless it’s something that a human shouldn’t be waiting on like transfers from outside. If it’s assets off of a drive, etc, we offer the idea that a user putting the data where it goes and then importing it into CatDV Pro is probably far faster and can be better, as you’d often want a person to look over any incoming media anyway, before it was published into the db.
This makes new users mad to hear. (What??? I bought this to automate things!! You guys are idiots!!) You can of course automate all sorts of things but sometimes it’s best to manually publish to start things off and make sure everything is placed and happy.
Exceptions: Very regimented workflows where the media is very consistent and arrives in a very clean and organized fashion. Also, imports where no one is waiting on the media right away. Or shops where a lot of automation horsepower is utilized and the users simply monitor those actions. (One Worker Node is almost never enough to accomplish more than basic automation’s. We average 2-3 on a mid-sized system.
The Worker is amazing, but every automation has some overhead. You might want to save your cycles for more important things that can’t be fixed with a simple “drag and drop.”
Finally, the good news, as you said, you don’t have to watch that run. You can do other good stuff, like rest. 😉
bryson
bryson “at” northshoreautomation.com
northshoreautomation.com
-
You’re probably right, but since this is essentially stock footage stuff, it’s already sorted into appropriate folders, etc. Would be nice if Worker could import folders the same way as client, although it’s probably only useful for bulk imports like I’m doing now.
Been about 40 hours so far. Am on 33,000 of 50,000 haha
Reply to this Discussion! Login or Sign Up