Final Cut Pro unusable for big projects?

[Walter Biscardi]

[Christoph Strothjohann] “our project has about 8000 files (sound and picture), which take up 3,5 terabyte spread on 3 internal sata drives and 2 external esata drives.”

There’s a HUGE problem right there. You have a massive project spread out over multiple drives. FCP has to search through all of those to find the files and that takes time. This REALLY slows down your performance.

With a project this big I don’t see why you or whomever owns the system did not invest in a large, high speed SATA array. We own an 8TB MaxxDigital EVO HD unit which is 8 drives striped together for around 400 – 500MB/s. And more importantly, it’s all one media array.

[Christoph Strothjohann] “This setup runs extremly instable. with one or two crashes on a good day or up to 8 crashes (including complete system crashes) on a bad day. most of the crashes happen during playback or when we try to move files to a second opened project. all crashes seem to be random.”

Again, a symptom of your media being spread around instead of in one location. Very bad project management for a project this large.

[Christoph Strothjohann] “each time we screen rough cuts ( at the moment ca. 100 mins long) out of one sequence after a 15-20 minutes fcp gets a short “hiccup” with visibly stuttering frames for a moment then continuing to run on, but completly out of sync. when we then hit pause and play quickly it continues running in sync again. Until after another 15 to 20Minutes we have the same “hiccup” occurs again…”

My guess is the hiccups happen when FCP is switching from one media drive to another and there’s a lag in the information getting to FCP, so you get a hiccup.

[Christoph Strothjohann] “after we tried all tips that were given (like turning of “high quality scrubbing”, deleting preferences, using without the aja io hd, etc.) we moved our project two days ago to a brand new mac pro with the same specs and fresh install of leopard and final cut.
The same problems occur on this machine.”

If you’re using the same media arrays spread out, this will continue to be a problem. You need a high speed, single media array for a project this large, preferably running RAID 5 in the case of a drive failure. MaxxDigital, Dulce, Sonnet, Ciprico, Facilis all make excellent large media arrays for this type of work

Walter Biscardi, Jr.
Biscardi Creative Media
HD and SD Production for Broadcast and Independent Productions.

STOP STARING AND START GRADING WITH APPLE COLOR Apple Color Training DVD available now!
Read my Blog!

[Rafael Amador]

Run DiskWarrior or TechTools in all your HDs, starting by the System HD.
Rafael

http://www.nagavideo.com

[Mike Parfit]

Hi,

Yes, we have a big project, and we have almost exactly the same problems. Our crashes happen mostly during rendering, but also at apparent random. We also have had the playback problems you describe, and sometimes they happen even when the program is set to stop on dropped frames. In other words sometimes the program does not think it has dropped frames but apparently it has; it then goes out of sync and we have to stop and start to get it back.

We reduce the crashes by reducing the size of the project file, mostly by restricting the numbers of sequences in the project. We tend to duplicate sequences often for backup and to retain earlier versions, so we often save a new project and delete the old sequences. We find that it helps to have the project size well under 100 MB. When we need to do a lot of rendering, we also remove our Kona 3 card from the system, which helps a great deal.

In the case of the IoHD, we used one briefly and found something different. With our Kona 3 card all we had to do was remove the card itself, not the software, to dramatically improve our stability. With the IoHD, however, unplugging the unit and rebooting did not help at all, but when we uninstalled the software, that did help.

These removals do not solve the problem entirely, but greatly reduce the crashing issues. (The playback can’t be helped this way, of course, because we have to playback through the Kona.)

We think there is a memory leak in Final Cut that steadily locks up memory and eventually leads to a crash. We have come to that conclusion because several times we set the program to render a long clip on one monitor and used the other monitor to show the OSX Activity Monitor. Since it was a very long render, we set up a video camera to record the Activity Monitor. As the render continued, the memory use gradually increased, until it passed 2GB (we have 4GB in the machine), at which point it worked for about another five or ten minutes, then crashed. We repeated this test several times with different size projects, and it was always the same. With the Kona 3 card out of the system the machine took longer to get to the 2GB spot and longer to crash, but it usually did.

This guess at the source of the problem is also supported by another symptom. We sometimes crash just when scrolling through a non-rendered clip on the timeline, even when we have not been doing much rendering. These kinds of crashes never occur during the first few hours of working with Final Cut. They always happen after we’ve been at the desk for some time. This, too, would seem to indicate that there’s a memory leak somewhere, that builds up as the program is used over time.

We significantly reduce our problems by rendering small clips from the timeline of the bigger projects, then putting those smaller clips on a separate timeline. When the timeline is restricted to one video track and two or four audio tracks, we are able to render 93-minute clips with no problem. With a single big clip like that on a single drive, the playback is no problem.

Like you, we have tried MANY, MANY fixes suggested by people on this forum. Not one of those fixes have changed the outcome in any way at all. The only fix we haven’t tried is what Walter suggests, the addition of a very fast array that looks like a single drive to the system. I think that might help, but we simply don’t have the thousands of dollars necessary. And if there’s a memory leak that makes large project files crash more frequently than smaller ones, it won’t help much.

My additional suggestion is that you do what we’ve been planning but haven’t gotten around to — chop your film into reels and have a separate (much smaller) project file for each reel. It’s not difficult to work with a couple of files in FCP when you need to shift sequences from one to another, either nested or just cut and pasted, so it shouldn’t handicap you much. Also, it seems to help somewhat to have as few video tracks as possible, so if you have a few places with complex composites, I’d put them on a separate timeline, render them out and put the result on the main timeline on track 1.

I hope you find some solutions. Personally, I think this is a problem that has been with FCP from the start and will only be fixed by the eventual complete code rewrite. But I may be wrong and there may be some simple fix. So, good luck.

Best wishes,

Mike

[Walter Biscardi]

[Mike Parfit] “The only fix we haven’t tried is what Walter suggests, the addition of a very fast array that looks like a single drive to the system. I think that might help, but we simply don’t have the thousands of dollars necessary. And if there’s a memory leak that makes large project files crash more frequently than smaller ones, it won’t help much.”

All I can tell you is that if you’re doing very large projects, there’s no reason to bite the bullet and get a very fast array. The prices are so much cheaper these days.

The 8TB unit we’re running is $5995 US and that includes the ATTO R380 Host Adapter. The 6TB is just $4995 US. Considering I spent about $8995 4 years ago for a 2TB Fibre Channel array, that’s an amazing price. I purchased two of the 8TB units last year and they have been cutting projects that have as many as 200 raw tapes per project with zero stability issues. The only issues we ran in to were some Quicktime / Render issues with one version of Quicktime.

Fast and reliable storage is simply a must for any large project or really any shop that is doing a lot of work. You just can’t be stringing your media along multiple drives or even regular DV projects can have slow-down issues, instability and the like. One, large fast media array is always better than multiple media drives.

Yes, FCP does have some sort of a memory issue that has been around since probably day one. The longer you run your FCP system, the slower it will get, especially if you open any other apps while you’re working. We always quit FCP during lunch and re-launch the app after lunch. Sometimes we’ll do a complete reboot after lunch. This is just common practice to ensure a smooth edit session.

Walter Biscardi, Jr.
Biscardi Creative Media
HD and SD Production for Broadcast and Independent Productions.

STOP STARING AND START GRADING WITH APPLE COLOR Apple Color Training DVD available now!
Read my Blog!

[Mike Parfit]

Hi, Walter,

Thanks. I don’t think we disagree here, except in terms of budget. $5,000 is just out of our reach. I mean, it is more than we paid for the machine itself.

It has so much to do with what a filmmaker is doing. You are putting out a series and a lot of other stuff regularly and your cash flow is quite different from ours. We’ve been working on a single feature doc for four years. We’ve won awards out on the festival circuit and have significant distributor interest, but we’re taking the highest-risk route here and trying to have an effective theatrical release, so we are flat broke, and have been for the whole four years, as we’ve put everything into shooting and editing time, and then publicity. Five grand is just out of reach. It doesn’t mean that we’re less professional, but it does mean that we’re on a different budget track.

Sure, it probably makes sense for us to have borrowed enough at the start to buy one of those excellent external raid systems that you and many others have been reviewing for us all these years, but this comes down to my only real crankiness about all this. If working on large projects requires such a system in order for FCP to be stable, Apple should tell us in its system requirements, and then we could take that cost into consideration at purchase time. Instead, the stated requirements and ALL the advertising indicate otherwise. The only stuff published about high-performance raid systems in both the FCP and AJA documents is related to whether you have a system fast enough to avoid dropping frames on capture or playback. There is nothing about system stability, but that is the big problem.

That’s my gripe — that Apple does not accept this problem. And the other thing that is still preventing us from finding that five grand somewhere is the strong likelihood that it just won’t improve things much. For a long time we used DVCPRO HD and HDV all on a 3-disk internal array, with great read and write numbers, far beyond what those high-compression codecs require, and that didn’t make any difference in the marginal stability of the system.

Oddly, one of the other very effective things we’ve done is to take OUT 2 GB of ram, which takes our system down to just 2 GB total. The system doesn’t crash then, but it does give a lot of “Out of Memory” errors. (By the way, just in case you’re thinking, ‘Ah ha, he has bad memory,’ I have alternately swapped out different sticks of memory just to be sure it’s not something in one of the sticks.) It’s as if FCP has proper error-catching code at 2GB of memory, but at 4GB or larger it just crashes.

Thanks for the response,

Mike

[Michael Sacci]

My 2 cents, I think it is crazy to think you should be able to do large editing projects with JBOD, internal and external. A good array is part of the tools of the trade. Now you can get the speed you need for less but you give up redundancy, I put together 7.5TB Raid 0 systems in the 2K range. With my system I have edited 8 streams of ProRes HQ (HD) multicam so it can keep up, All things being equal I would like to get a higher end array but I also want to eat, but there is no way I would except to do my job with single Sata drives.

If you have to use multiple single drives you ready need to be editing DV25 footage and then do a final recapture of the HD footage.

[Steven Gonzales]

I’ve edited a lot of features which had very low budgets, through all permutations of Final Cut.

You might have to scale down your codec to match your hard drives. Also, keeping the project files size down makes everything work better.

FCP can definitely do large features at a low budget. However, using a very high throughput codec, without the proper drive subsystem, will be very painful and unworkable.

You might have to sacrifice editing image when using low cost hard drives and a smaller “offline” type of codec. If you’re finishing on film, the important thing is your film lists are correct, and you can deliver sound properly, not the quality of the codec you’re editing with.

[Walter Biscardi]

[Mike Parfit] “Thanks. I don’t think we disagree here, except in terms of budget. $5,000 is just out of our reach. I mean, it is more than we paid for the machine itself.”

Absolutely, high speed, high volume storage does cost more. When you get into shared FibreChannel it goes through the roof, maybe $20,000 for 6 to 8TB shared for two workstations that cost maybe $8,000 combined.

[Mike Parfit] “Sure, it probably makes sense for us to have borrowed enough at the start to buy one of those excellent external raid systems that you and many others have been reviewing for us all these years, but this comes down to my only real crankiness about all this. If working on large projects requires such a system in order for FCP to be stable, Apple should tell us in its system requirements, and then we could take that cost into consideration at purchase time.”

Apple has been recommending their own X-RAID for years for working with video production. That’s a high speed, very expensive array. So that should be a warning right there.

Also, Final Cut Pro is designed to work in many MANY different configurations depending on your needs. Small projects require small arrays. Larger projects require careful consideration, especially on media management. It really makes common sense that if you are going to need a lot of storage, it’s better to purchase a large, fast array in the beginning.

[Mike Parfit] “It has so much to do with what a filmmaker is doing. You are putting out a series and a lot of other stuff regularly and your cash flow is quite different from ours.”

Ok, let’s start from the beginning. In 2001 when I opened Biscardi Creative Media I was coming off a failed partnership that put me $50,000 in the hole. As in I still owed my ex-partner $50,000 to pay off the debts of the failed company. I had zero equipment.

So I borrowed $27,000 to purchase a brand new Final Cut Pro System with a Pinnacle Cin�Wave card along with a BetaSP Recorder and what was at the time a very large 240GB Med�a SCSI high speed array.

So opening my doors I’m $77,000 in the hole before I start my first project. I had exactly 1 full time client. A corporate client for whom I did a lot of short children’s stories. Sure I had a great reputation and resume with Emmy awards and the like, but just 1 full time client and whole heaping load of debt opening my doors.

You know how I got Good Eats? A very lucky circumstance with the D.P. asking for help on the Cow and I happened to answer his call. Then I went out on a limb and invested approx. $32,000 that I didn’t have into a Panasonic 1200A DVCPro HD VTR. When Good Eats switched to HD, they asked me to help them develop the HD Post workflow and we’ve now delivered over 60 episodes of the series in HD.

So I get very annoyed when I see posts that say “Well, you’re in a different league, we can’t spend that kind of money.” I’m just a small business guy who believes I the only way to make a successful business for yourself and to make you stand out from the others AND to have a STABLE system, you need to spend money in certain areas.

High speed, stable, reliable storage is one of those places.

[Mike Parfit] “Oddly, one of the other very effective things we’ve done is to take OUT 2 GB of ram, which takes our system down to just 2 GB total. The system doesn’t crash then, but it does give a lot of “Out of Memory” errors. (By the way, just in case you’re thinking, ‘Ah ha, he has bad memory,’ I have alternately swapped out different sticks of memory just to be sure it’s not something in one of the sticks.) It’s as if FCP has proper error-catching code at 2GB of memory, but at 4GB or larger it just crashes.”

4GB of RAM is a major problem as I have noted in my blog and in other posts here on the Cow. Our Mac Pro Quad had all sorts of issues with 4GB RAM that went away when we dropped down to 2GB. Now it’s running 8GB RAM with zero issues. The new Octo Core is running 10GB with no issues.

Walter Biscardi, Jr.
Biscardi Creative Media
HD and SD Production for Broadcast and Independent Productions.

STOP STARING AND START GRADING WITH APPLE COLOR Apple Color Training DVD available now!
Read my Blog!

[Steve Eisen]

Thank you Walter. You said it all. I wish more shooters/editors can understand the requirements needed in this business. If you are going to shoot and edit in HD, you need to do it the right way. If not, expect undesirable results.

I too get annoyed when I see posts complaining that something doesn’t work or blaming Apple. The majority of the time it is user error. Apple is giving you the basic tools needed to get you started in this business. You still have to add more RAM, hard drive storage, monitors, capture card, etc.

I am a small business owner that started with a $15k loan. I don’t need to go into detail on the equipment I own or how much money I make. All I can say is I am very educated when it comes to the necessary equipment needed to complete a project. My system has never hiccuped or experienced any crashing like Mike and Christoph has described above.

Can an HD project be done on a budget? Yes it can. It still costs money.

Steve Eisen
Eisen Video Productions
Board of Directors
Chicago Final Cut Pro Users Group

[Rafael Amador]

Instead of talking about how to get an stable system, we end up talking about to spend more money.
My bigger project (no that long) have been 47 minutes. I worked with some 120 tapes. A 4 years old G5 and few external FW400-800 HDs. Not a single crack.
I live 600 miles away from the closet Apple service. I can not allow my self any kind of crashes and I get it. I don’t understand why people with the same equipments and configurations, some have problems, some have no problems. For me the word is : Maintenance.

http://www.nagavideo.com