[Andrew Richards]

[Oliver Peters] “That’s what I meant and not that we’d eventually see two products. I think the capabilities will get ramped up, but still within this magnetic/trackless/metadata paradigm.”

Agree. I hope the practical shortcomings of the magnetic timeline are being addressed in parallel to the spec sheet upgrades that are coming. They really need to let you explode audio tracks out from a clip for manipulation without detaching them and risking sync loss, for one thing. Role-based mixing would be handy too, though it is probably be easier said than done.

Best,
Andy

[Andrew Richards]

[Oliver Peters] “The comparison of DAMs was as compared with Avid Interplay or CatDV.”

CatDV does have much more in terms of asset manipulation for the end user. FCSvr was great for the admin (me). I’ve never used Interplay.

Best,
Andy

[Andrew Richards]

[Oliver Peters] “Rich is a stand-up guy, but this data point isn’t really verified anywhere else. Nevertheless it gives credence to the claims that FCP X was intended as an FCE successor.”

One of Philip Hodgetts’ recent podcasts (I forget which one) put forward the idea that FCP7 was going to be the 64-bit FCP, but Apple’s about-face on 64-bit Carbon scuttled that. The FCP team were supposedly forced to start over and took it as an opportunity to really start over. Richard’s aside about there being a 64-bit FCP they had to abandon (regardless of version number) aligns with that.

[Oliver Peters] “My guess is that FCP X was released in its current form with the idea of buying time to continue rewriting a more “pro” version that still fits this design. I have heard from other folks close to third party developers about two versions in development. Might be BS or might be part of the same rumors. In any case, no matter how skilled we think the Apple coders are or aren’t, it takes time to make the sort of about-face that they had to do.”

Maybe at some point prior to FCPX’s release there were two products in the pipeline, but I’d be pretty surprised to see something beyond FCPX coming down the pike. Maybe they merged the products and the heavy duty stuff is what is being added in incrementally. With all the NDA sidestepping any of this info would need to navigate to get to us, there is plenty of room for a lot of truth, even if the details are muddy.

[Oliver Peters] “Plus, the OS team is equally compartmentalized. Since FCP X is so tied to components in the OS (unlike Avid or Adobe), there are certain things FCP X simply can’t do until that portion of the OS is ready for the FCP X team. All of this leads us to a lot of anticipation for the Q1 2012 update. If that doesn’t show significant core improvements beyond just multicam and broadcast output, then we’ll know Apple isn’t all that interested in our world anymore.”

This is a very important point. I’ll bet you a fiver that broadcast I/O absolutely depends on a new API in Lion (CoreMediaIO). I really hope we get a successor to the cheese grater Mac Pro to go with it. The hardware is a bigger indicator for where Apple wants to be- that is where they make their money. Software isn’t a profit center.

Best,
Andy

[Andrew Richards]

[Oliver Peters] “It’s quite possible that too much of the database handling came over from FC Server. It’s not a particularly good asset management tool and when it works well, it’s running under Mac OS X Server usually on an Xserve. If that’s the case, then there are some built-in design problems.”

FCSvr ran on top of a PostgreSQL database that was being queried directly by the Java front end. FCPX calls CoreData to transact with SQLite databases (that’s all CoreData uses). Perhaps there are some schematic similarities between the two, but probably only at a high level (if at all). There are virtually zero architectural similarities given the technologies each uses.

I also disagree with your opinion of FCSvr as an asset management tool, but that is another discussion for another forum.

Best,
Andy

[Andrew Richards]

[Timothy Payton] “I have several large mySQL databases that run web sites on my web server. The hardware is 2.1 GHZ, Dual Core, 4 MB of Ram, and a regular old 200 GB sata internal drive. Typically the database files are small, about 2mb. But I have a very large site with a 67MB database, 168,683 records. Does the size of the database effect the speed? Not that I can tell at all. Typically my queries to return a hundred or so records are less than .5 seconds. Adding a record to the database just took 0.2085 seconds. Just as they are on a site with a small 2MB database.”

I don’t think the sheer size of the database is the issue, but rather the volume of queries (and possibly the nature of the queries). How would your mySQL database fare if it were getting hit with tens or hundreds of concurrent queries? Even if you could assume your performance wouldn’t suffer under many concurrent queries (it would), your .5 and .2 second benchmarks quickly become whole seconds, and that’s going to feel slow. If every little thing FCPX is doing is being logged in a database, we can expect a lot of transactions to be taking place virtually constantly. Spinning disk is going to struggle with that no matter how it is connected to the host.

[Timothy Payton] “Now I don’t know alot about SQLLite (which as far as I can tell is the structure of the FCP X project and event files) but I did a quick Google and found this: (it may or may not be relevant) Basically it says “As you can see, most operations are slower on SQLite3 and “write” operations (create/update/delete) are really bad.””

FCPX is calling CoreData, and that is what uses SQLite3. Maybe CoreData isn’t up to the task of the kind of strain an NLE like FCPX places on it, or maybe the schema implemented in FCPX is to blame.

[Timothy Payton] “So there are two issues here. 1) project databases has unbelievable bloat. 40 units of bloat for every 1 unit of real data. 2) database access is incredibly slow, which seems to get worse with size.”

The level of detail in FCPXML is clearly much lower than in an actual project. FCPXML doesn’t even retain audio levels. The telling thing to me is your original project was three times bigger than the one that resulted from importing the XML. We don’t know what is getting stripped out along the way from FCPX Project to FCPXML to AAF to FCP7 project, so judging them strictly by file size isn’t really telling a complete story. Maybe it is bloat, maybe CoreData is inefficient given that it is an abstraction layer between the app and the actual SQLite database, maybe it is bad design, or maybe it doesn’t matter much because the real bottleneck is the IOPS performance of the storage that hosts the database.

[Timothy Payton] “I would conclude from this that the database format that we currently see in FCP X is horribly flawed. If I were a project manager, I would get as many programmers as possible and set this as the top priority. Either scrap it completely or do a rewrite. Although a complete rewrite sounds like a big deal, and I can’t dream of working on a large project like FCP X, I have done smaller rewrites of database structures and it is often needed during the coarse of a project. It is going to be what determines if FCP X flies or not. No amount of hardware you throw at it can help.”

Leaving aside the myth of the man-month, I don’t agree that we have any kind of conclusive evidence this is strictly bad software and that hardware has nothing to do with it. There probably is lots of room for performance tuning with respect to how FCPX is calling CoreData, and there is probably also room for improvement of CoreData itself. No software is perfect and can always be improved upon. However, dismissing the role of hardware out of hand when database performance is in question is simply incorrect. IOPS matter a ton for busy databases. Contemporary consumer SSDs at their worst will deliver about 100 times the IOPS performance of a typical SATA HDD. That kind of gap absolutely matters in this analysis.

Storage requirements are a well-known factor when it comes to streaming video for an NLE. If you don’t have the bandwidth, you are going to drop frames. Period. The same thing holds for databases. If you want to go fast, you need high IOPS storage. The net benefit of using storage that is hundreds or even thousands of times quicker than spinning disks is going to deliver a much higher return than software tuning.

It’s fair to criticize Apple’s decision to go with an auto-saving-persistent-database model instead of a volatile-memory-to-flat-file model like they had in FCP7, especially in light of the problems it introduces. Both approaches have their plusses and minuses. I just don’t agree that there is nothing we can do about it, such as it is, out here in deployment-land.

Best,
Andy

[Andrew Richards]

Reading the experiences here and via your link my hunch is the weakness is in the storage hosting the project and event databases. IOPS are critical to snappy database activity, and I suspect most editors are storing projects like media, on storage that is very big and can stream big bitrate data. That same storage, likely HDDs (even if it is RAID) will not be particularly good at serving database queries against large databases.

If anyone has an SSD in their rig, testing a large project or event on the SSD vs an HDD might yield a considerable performance gap in favor of the SSD.

Best,
Andy

[Andrew Richards]

[Darren Kelly] “The PC, which I built from scratch cost me less than $2K, has 13.5TB of storage, 16GB of RAM and 6 cores (It’s an AMD – I’d like to move up to the Bulldozer 8 core this winter).”

You might be better off with today’s AMD chip, or so I read.

Best,
Andy

[Andrew Richards]

You won’t be able to use those HBAs in a Mac that is newer than Leopard. Qlogic ceased development for Mac OS for its HBAs at 10.5. If you want to use them, it will have to be in Linux or Windows or Leopard hosts for which Qlogic distributes drivers for that HBA.

What can you do with it? You can upgrade to Lion and get some Apple FC cards for you Mac Pros and you’ve got everything you need for Xsan save for some FC RAIDs.

If you don’t want to dedicate one of the Mac Pros to MDC duty, get a Mac mini Server with a SANlink and a USB Ethernet adapter as well. Get two if you want redundancy.

Also, unless you really know what you are doing, hire some help if you want to build out Xsan. You can also look into metaSAN or SANmp. metaSAN claims to not need an MDC, but they quietly recommend one (or used to). SANmp is volume locking.

Best,
Andy

[Andrew Richards]

[Craig Seeman] “Evan did mention the significance of Lion having server built in, when he made apparently tangential comments but later also mentioned that the separation between Events and Projects was important. It sounds like a clues about FCS like features returning maybe through Lion server as opposed to a separate product.”

Another tea leaf to add to this line: Lion dropped prior OS X Server versions’ included MySQL in favor of PostgreSQL, the database that underpinned FCSvr.

Best,
Andy

[Andrew Richards]

[Bob Zelin] “EVERYONE uses ATTO. “

…for Macs. Everyone else (Dell, HP, IBM, Oracle, etc) OEMs Qlogic.

Best,
Andy