Dave Haynie
Forum Replies Created
-
I still have a Firewire to SCSI adapter around here somewhere. I had SCSI HDDs on my PCs in the very early days (well, kinda early, I has successfully avoided The PC until 1994), and of course, Amigas before that. The last SCSI device I actually used was my old Polaroid film scanner… but I have a far superior Epson. But I’m sure that adapter is around somewhere.
I still have Firewire, since that’s what I use to talk to my Drobo… haven’t used a DV/HDV camera in longer than I’d like to admit 🙂
-Dave
-
Given that it’s a new card, be sure you have the very latest drivers straight from either the card vendor or nVidia. The drivers on the disc that came with the card (assuming you installed those, rather than keeping the drivers for your GT610) are GUARANTEED to be too old to use. I don’t think nVidia does separate CUDA drives, but double-check — AMD did that for awhile, updating OpenCL indpendently of the video driver.
-Dave
-
[John Rofrano] “People are always worrying about quality when re-rendering and most of it is unfounded.”
That’s basically it.
Every time you render MPEG-2, MPEG-4, or any other of the family of Direct Cosine Transform CODECs, there’s a conversion process. The pixel data is bucketed into macroblocks (usually 8×8 or 16×16 pixels, but particularly in AVC, that’s not the whole story). Then the DCT is run,w which translates spatial information into frequency information. This step isn’t the lossy part.
Next comes the lossy part — a low-pass filter is run over the frequency data, tossing out some of it. Sometimes. The bitrate, some of the VBR analysis of the “big picture” if you’re encoding variable bitrate, etc. all contributes to shaping that low-pass filter. After the low-pass filter, there’s additional stuff: lossless Huffmann or similar compression of the frequency data, etc.
When you recompress with the same or greater compression settings on identical material, you’re pretty likely (but not guaranteed) that the AVC encoder is making the same blocking, bit-budget, and filtering decisions. Which means the video is going to change very little. And of course, you probably did something to the video in the process of editing: color correction, noise reduction, level or contrast adjustments, etc. That’s going to change the encode-time decisions, but if you’re doing your edits well, the bottom line is that the re-encoded video looks better than the original video.
And these days, we can often start with much higher bitrate video than we target, whether that’s DSLR video at 50-100Mb/s, AVC-Intra or some other higher-end formats, etc. At this point, you’re going to be essentially indistinguishable from RAW at the target video, aside from any processing errors (letting video saturate and clip, etc… most of which can be avoided by rendering with 32-bit precision). Given the higher quality video, you can also do things to ensure that the final encoding preserves more of the intended video, less of the bad stuff — like light noise filtering, if you have a little digital noise in the image.
Back in the early days of compressed music, one of the figures of merit on the CODEC was the number of re-compressions you could stack without some level of damage to the audio. I recall, when MD first came out, Sony’s ATRAC encoder could manage about 4-5 recompressions; by the time they were done editing with it, it could do 20-25.
-Dave
-
When Vegas 11 came out, first version of Vegas to use OpenCL rather than CUDA (well, CUDA-proprietary; as nVidia will tell you, CUDA is also OpenCL compliant these days, but when one speaks of CUDA, that’s usually to indicate the CUDA-only support, which was limited to the small bit of acceleration in Vegas 10 and maybe a few earlier version, just in the CODECs themselves), I did a comparison. I ran a GTX 570 against a Radeon HD6970. The HD6970 was as fast or faster at everything, so I kept it.
A couple of things. Back then anyway, I had to track down the better OpenCL drivers; the one that came bundled in the generic ATi/AMD install package didn’t work. Back then, the nVidia driver was buggy… didn’t see it fail on Vegas, but it did fail on a few sections of a few OpenCL benchmark programs.
I haven’t run benchmarks on my new system, but I’m pretty convinced that the level of acceleration in Vegas is limited to helping out slower PCs only. There’s always some time wasted in compiling OpenCL programs, transferring them to the GPU, transferring data to/from the GPU, etc, the CPU waiting on the GPU to finish, etc. So despite the fact that your GPU may be 25x faster than your CPU on paper, only some of that speed gets tapped, and a faster CPU eliminates the advantage. That’s of course implementation-dependent; if Vegas managed to run the GPU stuff asynchronously and pipeline processing, keeping the CPU at around 100% and maybe the GPU a bit more used, we might see a big win from the GPU. But that’s probably a huge change to the way Vegas works right now.
I’d be surprised if Vegas didn’t work right with the newer GPUs, unless there’s something rotten in their OpenGL implementation. One of the main premises of OpenGL is that it make the actual hardware a non-issue, as long as its capable of supporting the needed OpenGL ops. AMD actually has a driver that lets you run OpenGL on an AMD CPU… not necessarily what you want, but it does work.
-Dave
-
[Norman Black] “I’ll disagree that TV has more contrast and colors. Rec 709 (HDTV) and sRGB (typical computer display) are pretty similar color spaces. Nearly identical in fact.”
Similar enough color spaces, sure. But a typical LCD PC monitor these days has a low contrast ratio, usually around 1000:1. Ok, some of the newer models have a dynamic/LED mode, but that’s usually disabled by default. CRT for television is usually a minimum of 20,000:1, and you’ll get better range from pretty much every modern television… of course the LCD televisions are using dynamic backlighting to achieve their extended dynamic range.
It’s also a question of your target, modern or ancient TVs. If you’re thinking of an old CRT, you have NTSC issues to contend with. Digital televisions will certainly accept and upconvert NTSC, but they’ve never had to deal with NTSC color, interlace, or any of that nonsense in their native mode — in fact, most current televisions don’t have an interlaced mode; they’ll scan convert it to some form of progressive.
And of course, on a PC, you also have to contend with the fact that your display program isn’t necessarily synched to the actual refresh rate. Most folks pop up a player, and that player is just dumping data into the video card’s frame buffer. The buffer may be updating at 24p, but the screen is probably being refreshed at 60p. There’s usually enough sync, or double-buffering, to avoid updates of the live display, but still, it’s going to look much different on a television, where the video is displayed in sync with the display.
-Dave
-
You can certainly avoid overheating or underpowering an internal RAID. But what you can’t avoid is that it’s entirely dependent on the PC, on Windows, etc. to operate. That always spooked me — that PC goes down and my files are inaccessible, my really critical ones that I put on RAID to make them more likely to be available. And if the RAID controller is on a main board, do I even know for certain the replacement PCB will have compatible RAID firmware? Nope!
I hadn’t messed with in-box or motherboard RAID for some years, but I’m playing with it now, even though I have the main RAID on a Drobo. Part of this experiment was a need for speed — the Firewire 800 Drobo is too damn slow for editing… ok for long-term storage. And they’re pretty pricey, but have the really cool feature of supporting mixed drive types, and automatically rebuilding on larger drives as necessary.
So I set up a 3-drive RAID5 using Intel’s so-called Rapid Storage Technology, Enterprise edition (well, that’s marketing for you I guess). Worked ok, was slow on small writes as described above, but suitably fast on large writes and pretty much all reads… better than half the speed of the SSD, faster by far than a single 3TB drive.
I was fooling around with boosting the CPU clock (I know, not a good idea, but new system, I’m playing around) and got to a point were I had to reset the BIOS. The system came up in non-RAID mode. Switched to RAID, and it couldn’t boot — something ate the Intel RAID driver, not sure if was a software update or a glitch, but nothing else was missing…. I had the boot drive on that chipset-based SATA, that’s why it wouldn’t boot. Moved that, booted, installed the Intel RAID driver… rebooted. Now the RAID BIOS claimed that 2 out of 3 RAID drives were not RAID drives at all but standard separate drives. Looked up on Intel support sites, pretty much everyone says you’re screwed at that point. So I tried a frightening hack: I deleted the RAID, set up the same structure anew, rebooted. Then I used a partition restoration program, which found the original RAID (which it could, since the controller had the right parameters to run that particular configuration), wrote that, and was back in business. But no thanks to Intel’s software… if you can’t trust your RAID environment, dump it. I’ll be using a different solution on this system for a faster data drive… maybe the Drobo’s not such a bad deal after all 🙂
-Dave
-
As John says, could be a bitrate issue. There are Blu-Ray templates for both 8 and 16Mb/s in the standard Sony AVC templates, as I recall. You might also try Main Concept’s 25Mb/s Blu-ray templates, or boosting Sony’s somewhat, assuming you’re already using 16Mb/s and still seeing the artifacting.
Here’s the likely problem — warning, I’m about to go all video encoding geek on you: you’re shooting at 60p. That means you have one I-Frame (the “independent” frame, basically a glorified JPEG) every 1/4 second, and 1/60th of a second of motion between each B or P Frame. Less motion means less “error” video to encode in the AVC stream.
As the amount of “error” video increases, the more bits you need to encode that without showing blocking… really no different than back in the old days of low bitrate MPEG-1. When you take your 60p video to either 60i or 30p to make it Blu-ray compliant (Blu-ray doesn’t actually support 30p, but it’ll happily encode a 60i stream that’s actually 30p), you now have an I-Frame even 1/2 second, and 1/30th of a second between each B and P frame. So in essence, you have to deal with twice as much motion between frames. Thus, fast motion can degrade, where it didn’t before.
The other issue is variable vs. fixed bitrate. Most camcorders these days use variable bitrate encoding — nothing crazy, but they can boost up the bitrate for high motion and drop it for low motion, within limits, to match the bit budget to the needs of the video, while still meeting the AVCHD 2.0 (or whatever) requirements. Sony’s AVC CODEC only does fixed bitrate.. so you may be between a rock and a hard place, dropping your frame rate and your bitrate and your peak bitrate. I have not used the Main Concept CODEC for Blu-ray (Sony only blessed it for that in Vegas 12, before then, it was only recommended for Internet bitrates), but this one allows peaks up to 40Mb/s, from the base 25Mb/s rate.
Another option, if you’re getting no love at 1080/30p, is to try 720/60p — also a legal Blu-ray resolution. I don’t know how much fast motion you have in your video, but I shot quite a bit of high school soccer video, and I got much better quality at 720/60p, due to these issues.
-Dave
-
[John Rofrano] “[Dave Haynie] “What they really mean is that the RAID chip, card, or box with its own processor to run the RAID software. Otherwise, that software runs on you PC’s processor.”
Yes, and that’s the point I as trying to make. If your CPU has to run the RAID software it’s going to be painfully slow with RAID 5. “Yeah… I wanted to point out two things. One, that you really have to check your specs, if you’re buying a device. I suspect just about every JBOD out there has RAID software included, but that’s running on the host CPU, and it’s probably very slow. So when you’re looking at an external box, make sure it’s real RAID with its own CPU, as with yours.
It’s not just the checksum calculation with the external hardware, though. When you do a write to the external RAID, that’s write cached — goes into RAM on the RAID controller board. So the main PC is isolated from the time it takes to complete that write.
There are several things that cause software/firmware RAID to be slow. One is your RAID stripe size. When you write files smaller than the stipe size, the controller has to read-modify-write… it’s reading in the data, making the change, calculating the checksums, and writing it back out. I’ve been playing around with Intel’s latest software/firmware RAID on my new PC, and in one test, I saw writes of 512 byte blocks going at 1.4MB/s, pretty pathetic. And I saw CPU use jump briefly to 12%, which reflects this extra work. About what I saw, at least by degree, the last time I played with this stuff. This is no different from the hardware RAID, really, except that those slow writes are handled by the RAID controller’s CPU… Windows never gets bogged down by them.
This reinforces a hard rule of modern HDDs — large writes go fast, small writes go slow. This pretty much worst-cases it, and thus, if you’re putting in a RAID for something like an internet server — pretty much all small writes, this is lethal.
On the other hand, I looked at larger writes. At 2MB writes, I was seeing 179.8MB/s, which doesn’t suck, and you don’t get much smaller on a media drive. The CPU use didn’t peak above 4% on any of the larger reads or writes, and was usually a 1-2%. I looked that up and found that, while the Intel software is still using the host CPU for checksum calculations, it’s using a hardware coalescer to transform a large number of would-be small checksum writes into a very few single I/O operations. That actually makes sense; the checksum isn’t a great deal of overhead to calculate, but writing it (a small block of data for every large-ish RAID stripe) is a killer. The other thing they did to improve performance is allow write caching (haven’t played with that yet)… applications just write to a RAM cache, and the data gets flushed to HDD later. Very fast, but dangerous if you don’t have a UPS on the system.
Anyway, my main goal was to really find out why these things were so slow… the checksum answer just didn’t do it for me. This was with ATTO… I need a better, more modern disc benchmark, but still kind of a surprise. I still expect to port over the RAID controller from the old PC, once I’m done with it.
-Dave
-
A couple of things.
One observation: all RAIDs are software RAIDs. When you say “hardware RAID”, that can mean a few things. At the least, it means SATA hardware with some extra RAID a support. Some folks think that mandates an external box, but some external boxes are still “software” RAID. What they really mean is that the RAID chip, card, or box with its own processor to run the RAID software. Otherwise, that software runs on you PC’s processor.
My main RAID is an external Drobo box.. They have a proprietary tech, but it can run things similar to RAID 1, 5, or 6 on some models had some performance issues on older models looks mine, but it did let me add drives gradually. Many motherboards offer RAID support, as does Windows itself under NTFS. Filesystem RAID is alwaysv software only. BIOS RAID is either software only or software with some kind of acceleration in the SATA hardware. Either can give you a decent RAID 0 or 1, but usually nothing better without a huge slowdown.
To define these, for those who don’t know… RAID is a “stripe set”… You add together the storage of multiple drives into one volume. This adds speed, too, but there’s no redundancy. The life of this is the lesser of all drives: one fails, all data is lost. RAID 1 sets up two or more identical drives in a mirror config, writes go to two drives, reads come from either until an error is detected, then the other drive can supply the data. RAID 5 stores error check and correct data for fragments of data written, and allows one drive to fail completely and be replaced
No RAID let’s you add and subtract storage — use individual drives. I use separate SATA for projects, with a cheap enough drive bay that fits in a 5.25” bay.
-Dave
-
I would say memory problems are possible. Not super likely, but it’s possible that in typical use, the OP never hits the faulty cell(s) outside of Vegas just based on the amount of memory being used. Memtest is a good one, as these things go these days. Might also see if there’s a memory test in the BIOS, which can test all RAM without using it – that can let the test stress things in different ways. Key is running these long enough to find a flaky cell, maybe just a chance of failure in normal use.
-Dave