Quadro 4000 shuts down during long preview renders and export encodes

[Matthew De jongh]

You could try the premier pro benchmark project and see if it craps out rendering that…

https://www.ppbm6.com

I have nothing to do with those folks, I just found their site very helpful as I’ve been upgrading my older dual Xeon quad core system…

I also have the quadro 4000, just upgraded from a quadro 3700

Matthew

[Paul Thurston]

Hi Bruce,

When your computer monitor is “TCO’ 03 power management compatible,” it means your monitor will go into “sleep” mode whenever it detects there is no movement of the mouse or keyboard after a measured amount of time. The Nvidia Quadro 4000 is also “TCO’ 03 power management compatible” aware. So when the monitor reaches it’s time limit of “no action by the user” moment, it communicates that to the Quadro 4000 and the Quadro literally sends a type of “sleep mode” signal to the monitor. The monitor will appear to not have an image, like it’s turned off, but in fact it is still on. It’s like the backlight of the monitor was turned off. You can tell the monitor went into this “sleep mode” as the computer is still running and in your case still rendering.

If your monitor is made by Dell, you’ll notice when it goes into the “power management mode” as it states so on the screen ever so often.

The design process to get out of the sleep mode or “power management” was to just move the mouse. BUT, there is a software bug in the Nvidia Driver that does not allow the Nvidia Quadros to quickly “wake up the monitor.” In fact the Driver may fail to actually wake up the monitor at all (depends on the Nvidia driver being used.) In that case, you’ll just have to press the On/Off button of the computer so it gracefully exits windows.

So how do you overcome this limitation?
1) Connect another monitor to the motherboard VGA out, so you can continue to see what goes on after you discover the Nvidia Quadro has gone into the “power management” subroutine. If you have an old Tube ViewSonic monitor somewhere, those work great with motherboard VGA outs. (This of course implies you’ve not turned off in BIOS the VGA out of your motherboard.)
2) Stick around while the rendering takes place and move the mouse ever so often. (Not recommended, because it’s a pain to do, but it works.)
3) Work with Nvidia to figure out what to do to avert the “TCO’ 03 power management” from executing. I understand Nvidia knows about this issue, and has written about it in its REAMME PDF associated with the latest Driver update. They know about it, they just can’t fix it right now for some reason.

On a side note, the TCO’ 03 power management is controlled solely by the Nvidia Quadro Driver. It is not controlled by windows 7 power management at all.

-Paul

———————————————–
Paul Thurston
Producer
Chile

[Bruce N. goren]

So here is an update for anyone following this thread.

This is clearly a heat induced component failure.

Turning off the GPU based Mercury Engine in favor of software only in the project settings allows me to render, albeit a bit more slowly.

Encoding in AME directly also is successful since it does not task the GPU.

In leafing through the release notes for recent nVidia drivers I noted that in dual monitor usage Quadro cards are forced into enhanced performance mode full time. Sure enough, using CPUID HWMonitor I observed my fans were roaring at 100% and the card was cooking near its operational limits, 95 to 100 C. I’ve gone back to single monitor mode now but perhaps the damage is done, since even at “idle” the card fans blow at 35-50% and the heat runs 75 – 85 C. Using GPU intensive apps pushes things back up to the red line.

You’ve got to wonder if this design limitation is one of the reasons for Adobe/nVidia coming up with the Maximus configuration solution; moving GPU operations over to a headless, expensive, dedicated GPU cores card and relegating the Quadro to handling display only.

I’ve vacuumed and blown out the fans and heat sinks and ensured all connections are secure.

Perhaps I need to consider liquid cooling the Quadro 4000, have to cost that out . . .

Thanks to those of you who have contacted me directly via email with helpful info and suggestions!

After consulting with Xi, they confirmed my GPU temps were abnormal based on the rigs they were running, since I am beyond the one year build warranty I next called PNY.

Excellent phone support from PNY, but odd exchange policy. I’m used to advanced replacement at the Pro level, meaning that you issue a P.O. or allow a credit card hold to be placed and they send you a new unit to swap out — no down time, just send back the defective unit within a few days. PNY gave me two choices, send them my defective Quadro4000, they would then send me a replacement, that would mean several days down time, not acceptable. I chose plan “B” which was they charge my credit card for the replacement board, ship it to me by next day air, then I return the defective card and they issue a refund — but the caveat is that it takes 4 to 6 weeks for the refund to post. That sucks, but what the heck, I needed to get this done.

Installed the replacement card, did not seem 100% fresh and new, I could tell by the connectors and hold-downs it had been in and out of motherboards many times, but the firmware was more recent than mine, so it goes. Problem solved, this card idles between 45 and 60 C and the highest I was able to push it was 92 C under the GPU torture of Boinc/Seti . Premiere Pro also no longer trips the panic switch, with GPU rendering hovering between 75 and 85 C.

I’ll let you know how long it takes PNY to credit me for the return.