[Dave Haynie]

Depends on the format used for recording.

The Canon 5D Mk III, like my 6D, has the option of either IPB or “All-I” recording. All-I is the same thing as AVC-Intra, records at about 100Mb/s. This decodes much faster, since each frame is entirely independent of the next. However, if you use IPB mode, you’ll get the usual ~40Mb/s video you expect from a Canon, but it’ll take longer to decode than video from your T3i.

The T3i uses IP (sometimes written IPP) encoding, AVC, at about 40Mb/s. This will take longer to decode than AVC-Intra.

So here’s the story.. this is simplified, and covers pretty much any MPEG family video CODEC. Back in the dark old days of DV, you just had the DV CODEC. This was, itself, a formalization of “Motion JPEG”. Basically, each frame was encoded as a JPEG image, with some additional special sauce for dealing with interlacing in an effective way. You can imagine shooting 24 JPEGs per second with either Canon, and you’ll get the idea.

HDV moved to MPEG-2, which uses both interframe and intraframe compression. It specifies an I-Frame, which means “Independent” frame, which is pretty similar that original JPEG. That’s compression within the frame; it was a 5:1 compression for regular DV25; MPEG-2 in HDV goes a bit stronger. But consider: DV was 25Mb/s. DVD, which looks just as good, runs at around 6-8Mb/s typical. That’s where the intraframe compression goes.

So after than I-Frame, let’s do something different. Since it’s pretty common for most of one frame to be in the next frame, let’s figure out the difference between the next frame and the one you just encoded as an I-Frame. Using a motion search algorithm, I can produce a set of vectors, which tell me pretty accurately where each chunk of one frame went, in the next frame. So then, apply those vectors to the first image, subtract the second image, and you get a very strange “difference” image, which is basically just the error between the algorithms used to find and apply the vectors, and the actual image. So we store those vectors and that difference image, which is highly compressible, into a new frame time, dubbed a “P-Frame”, for prediction. The final kind of frame is a B-Frame… B standard for “bidirectional”, and it’s a frame that can essentially use details from either the previous or the next frame as the basis for compression. The earlier Canons didn’t use B-Frames; DVD and Blu-ray do, as do the newer Canon models.

So, the decoding job when it’s I-Frame only is easy: every frame to itself, just a JPEG decoder. For the P-Frame, you have to take that previous frame, apply the motion vectors, take the error frame, uncompress it, then apply it to the vectored frame. Much more work, thus, a much slower decompression.

And AVC itself has a bunch of extra small details, in both its “JPEG” equivalent for I-Frames, the analysis and application of vectors, etc. which lead to better images, but slower decoding and encoding.

-Dave

[Dave Haynie]

There is no 4K profile for Blu-ray yet. Period. Not supported.

There’s apparently a group at the Blu-ray Disc Association studying the problem. That would presumably result in the definition of a new Blu-ray disc profile, as they did with BD Live!, BD Audio, 3D, etc.

This would not likely be directly compatible with today’s players. They might go to the existing BD-XL format, which supports 3 and 4 layer Blu-ray discs (so far, this is just a recording format, no one’s making glass mastered BD-XL discs). They might rather employ the HEVC video CODEC, rather than/in addition to AVC, since HEVC offers about twice the coding efficiency of AVC. And just when you thought those AVC rendering times were getting controllable.

Sony and Red are each currently offering 4K media players, the FMP-X1 and the RedRay, respectively. They’re not cheap, and they’re each using proprietary video CODECs. Red claims to be using only about 20Mb/s and a proprietary CODEC; Sony’s using H.264 today, but claim that HEVC/H.265 support will be forthcoming. They’re both concentrating on online rentals.

-Dave

[Dave Haynie]

[John Rofrano] ” [Dave Haynie] “…so a misbehaving program could lock out every other program. Today, you’d have to lock up a hardware resource for that to happen… also possible in pretty much every other OS.”

This must be what I’m experiencing. OS X does not have this problem. In Unix, kill means kill! In Windows kill means shut down the computer and reboot.”

Kill is relative, even in UNIX. When you send a normal kill signal to a process (SIGTERM), it’s just a signal to that process… a bit is set. At some point, that process’s signal handler will respond to that, and take the appropriate course of action. Usually that’s “clean up and exit”, but the program can do whatever it wants to do. It could ignore that, or it could even hit a big in the function responding to SIGTERM, causing the program to hang.

In Windows, the equivalent is a WM_QUIT message, sent through that same single message queue. In theory, if the application doesn’t reply to a WM_QUIT message, the OS will intervene — that’s when you see that “program is not responding” pop-up after you hit the close box and nothing happens. Once the program responds to WM_QUIT, it’s just like SIGTERM.. it’s left up to the program to quit, or not.

The “not”, in either case, may be trying to shut down asynchronous I/O. You have a thread that’s dealing with I/O, and it’s stuck waiting for something to finish. So the main program gets the “quit” message, but other threads or processes may not be able to respond. This is the same in UNIX as in Windows, too.

There’s also a signal call SIGKILL in UNIX; that’s what you get when you type “kill -9 ” from a shell. When you send a SIGKILL, the signal is not handled by the process it’s sent to, but rather, by the Init process, which is basically the king of all processes. There are a few cases in which SIGKILL can be ignored.. but not many. Windows doesn’t use a message for this, because as discussed, messages can be ignored.

Some of problem in Windows are what’s actually being done versus what’s expected. When you fire up the Windows Task Manager, go to the Applications window and select that misbehaving application, and click on “End Task”, you’re really just sending WM_CLOSE to that program… the same thing that’s supposed to happen when you click on the close box. So it’s not terribly powerful if the application is hosed, but does have the advantage of letting the application clean up after itself if it is still listening.

If you go to the Processes Windows, select a process and click on End Process or End Process Tree, you’re doing something more powerful. This actually calls an OS function that starts killing the process you’re clobbering (and in the second case, any child process started by that original). The OS can apparently clobber any user-mode aspect of the process immediately. But anything that’s kernel level can’t just be killed. So if the offending process happens to be doing some I/O, for example, that’s waiting on a kernel process and can’t be killed. The OS has to signal that kernel resource (often some driver somewhere) to cancel whatever it was doing and clean up. That doesn’t always happen quickly.

Again, UNIX itself is no different, and I’ve had plenty of Linux programs that hang and stay hung (occasionally even those I was writing). I think Windows is still more prone to this for whatever reason. The flow of code in a Windows program is often convoluted; for a Windowed application, you’re actually building a bunch of subroutines that are going to be called by Windows, which may then go on to call Windows functions, which may in turn create other unexpected things, like interprocess communications turned into message queue events (eg, parallel events being converted to serial, a bad hangover from before the Windows GUI was multitasking).

MacOS may be better still. I’m not sure how much of the original Mach idea of Microkernel architecture they’ve kept, but that’s a big help. As I mentioned, Windows can clean up user-mode stuff pretty easily, but the kernel stuff can be problematic. Same with any OS — they have the same kind of problems, even if they differ by degree. But the more “stuff” you have running in user mode, the less likely you are to run into something being locked because of waiting on failed kernel resources.

This is very different than Linux. Linux is still very much a macro-kernel. No, you don’t have to compile everything in to change a driver. But the drivers basically work like plug-ins do in NLEs and DAWs… they make the kernel modular, but they’re still run entirely within the context of the kernel.

-Dave

[Dave Haynie]

That’s to be expected. H.264/AVC/MPEG-4 Part 10 delivers about twice the coding efficiency of MPEG-2. You would expect H.264 at 25Mb/s to be about the same quality as MPEG-2 at 50Mb/s.

More or less. This comes from many times the encoding and decoding complexity. I recall when PCs got fast enough to do DVD decoding plus upscaling in software… on a Pentium at 133MHz. Back in the early 1990s. A buddy of mine worked for one of the companies doing DVD hardware decoders for PCs, and they had just released a software product that, for the first time, looked better than the hardware decoders.

My laptop, a 2.4GHz Core 2 Duo, was just enough CPU to decode HD quality AVC in without jerking around. And it actually did depend on the right decoder… the freebie in VLC didn’t make it. This was around the time they started using GPUs for decoding acceleration in players. With the accelerated player, the same video went from about 90-95% CPU to about 12% CPU.

But keep that in mind: just decoding AVC on two CPUs used them up. Imagine encoding from AVC back to AVC… it’s going to take a really long time on this computer. I do recall renders that ran 24 hours, back when I actually did rendering on this PC. MPEG-2, on the other hand, was no big deal for this computer.

-Dave

[Dave Haynie]

I wasn’t shocked so much at the RAID5 slowness, once I figured out why it was so slow (small blocks = small writes) on small files. But I was actually surprised by how many things are doing small sized writes to disc, anyway… my system has so much more memory, but many apps seem to think they’re back in the 1990s, memory-wise, at least for file buffering. That’s really what killed it.

But that’s ok.. the RAID10 is rebuilding. I did a couple of benchmarks, and yeah, the RAID10 is faster than the single drive. Tiny writes are slow even on an SSD, but they scale up pretty fast on the RAID10. Here’s my SSD as a baseline:

And here’s the RAID10 (4 x 3TB HDD):

Not too shabby. I’m seeing 90-100MB/s on the restore, limited by the original 3TB WB drive.

-Dave

[Dave Haynie]

[John Rofrano] “Unless of course you wanted an operating system that actually performs preemptive multitasking and can’t get locked up by a user application or slow down every year until it’s practically unusable and you need to re-install Windows… then you might want to use a “real” OS based on Unix that runs as fast 3 years later as it did the first day you got it! (i.e., then you might want a Mac) ;-)”

Windows has done real preemptive multitasking for quite some time. The Windows NT kernel, at the time, was actually more advanced than any version of UNIX around. Some UNIX-like operating systems, like MacOS, have caught up, at least to some degree.

That given, the Windows UI does, in places, ruin that power. Much of the way Windows works was designed in the days of fake multitasking… then again, that’s also true of the Mac API. It’s just that, the way the Windows API is architected, this is kind of still exposed. You CAN write good multitasking/multithreaded software, but you have to watch out — there are things in the OS, still, that try to artificially serialize all that great parallel code through the Windows event queue.

At least event queues are per-application now. In the past, there was a single queue in the whole system… so a misbehaving program could lock out every other program. Today, you’d have to lock up a hardware resource for that to happen… also possible in pretty much every other OS.

Both systems had pretty primitive ideas about GUI events. In AmigaOS, your message port only ever saw message types you actually requested to see — the OS managed all of the others. One UI port per window, too, so nothing blocking anywhere.

Back to UNIX… MacOS is actually best described as “UNIX-like”, not UNIX. The kernel is CMU’s Mach 3.x kernel, with decades of Apple modifications (like getting decent multimedia performance… historically, UNIX didn’t even care about this sort of thing). But classic UNIX defines a “macro-kernel”… the kernel has pretty much everything including the kitchen sink in it, including all drivers of any kind. Mach defined a microkernel, which allowed both plug-in kernel-level drivers, kernel-level multitasking, user-mode drivers, etc… like the NT kernel. Neither is a pure microkernel, and to an extent less today than historically… video drivers used to be user-mode in early versions of NT, for example, but they were deemed too slow, and moved into the kernel.

UNIX also classically didn’t have multithreading. Multiprocessing, sure.. you could pretty easily create big fat processes, clone your current one, communicate via pipes or files between processes, but for much of what you want in a modern OS, that’s very clumsy and inefficient. Mach also had real threads from early on, as did NT. Linux got them more recently, and some versions still give you “user mode threads” rather than “kernel mode threads”… which is fancy geek talk for “cooperative multithreading in user programs”. Fake multithreading.

Pre-NT versions of Windows always slowed down for me over time, and since I was trying to do realtime stuff, music and all, this could need a re-install every six months. That never happens in modern Windows. Careless users can definitely get bogged down with hundreds of crap background programs sitting around polling for the one stupid thing they manage… one of the major innovations in today’s mobile OSs and appstores was central management of this kind of thing. But the OS itself doesn’t seem to rot, the way Win9x did.

Of course, Microsoft has recently ruined desktop Windows by grafting their tablet UI on it, as many folks have discovered in Windows 8. They really do seem intent on screwing up their desktop stuff. Apple has intentionally resisted most of that, instead trying to actually figure out what you can adapt specifically for the desktop from what they’ve learned about touch-screen systems. But that may have been driven by SJ. And given how small MacOS is becoming relative to iOS (about 12% of Apple’s business), I’d be nervous about where they take MacOS, too, in the next few years. The announcement of the new A7 SOC has all the chip geek blogs predicting, again, that Apple goes to ARM for the desktop Real Soon Now. But I digress… killing time while I restore a RAID from backup.

-Dave

[Dave Haynie]

[John Rofrano] “I would look at gaming laptop like those from Alienware”

Not sure Alienware has been the same since Dell took ’em over. But in general, yeah, most anything that makes for a good high-end gaming experience is good for a video machine too. You’re probably still going to be “stuck” with a lower performance mobile processor… an i7, definitely, but only two RAM lanes. But you’ll be able to get a decent GPU, enough memory, and at least a decent screen, though tiny (I have two 2560×1440 and one 1920×1200 monitors on my desktop).

And while I know John likes Macs, you’ll be able to far more performance in a gaming laptop than available in any Mac laptop; lots more performance for the same, a little more performance for less cash.

B>[John Rofrano] “[Angelo Mike] “You wouldn’t want a Mac to run Vegas, since it’s impossible unless you dual boot it with Windows,”
Not true. I do it all the time without dual boot. “

I run Linux in Windows much the same way. Virtual PC environments are pretty sophisticated these days. You’ll take a hit on graphic-related things, but not much on the actual CPU-related stuff, other than of course doubling the basic resting workload on your system by running two operating systems at once.

Since AMD’s 64-bit x86 architecture, there has been CPU-level hardware support for virtual environments. So you’re not losing much performance virtualizing the CPU itself. Virualizing other things, you can drop performance some.

This is also just good tech to know about. When I was setting up my new PC, I had things about golden on the new PC. I had started it out with some old unimportant GPU, but needed to swap the Radeon HD6970 to the new PC, drop the bogus GPU into the old one, put the new one in the office and the old in the lab — still had things to transfer.

Well, after surgery, the new one started up just dandy. The old one never woke up from surgery. Not sure what died, but it hadn’t been healthy for awhile. But I had this old PC that I still wanted around… what to do?

Of course, I could mount my old HDD on the new PC. But what if I needed to actually run something on the old PC? So I did add the old HDD to the new PC, but rather than using it directly, I set up a virtual disc link for the HDD, start up VirtualBox (free virtual software), and launched, in essence, my old PC in a window on my new PC.

-Dave

[Dave Haynie]

Yeah, I thought about that, too. Certainly in some areas, it works — there are plenty of competitive NLEs that sell fine without offering the whole suite that Adobe offers.

But here’s the thing… let’s say I successfully create a viable Photoshop alternative, something strong enough to compete against Photoshop (and not as mind-numbingly arcane as The GIMP). This could definitely be a big sell to anyone who’s on the Creative Cloud just for Photoshop.

First problem: my easiest target would be those most unhappy with Photoshop on CC… which as I pointed out up above, are also the most difficult customers to please in general. The guys with one foot out of the Adobe door are the most demanding, just based on the way the service model works. So I might inherit all the toughest customers first.

Second problem is that I only get the Photoshop-only customers. Everyone else is using some other CC tools, and the price for everything is about twice that for just one thing. If I can’t also supply that “other thing”, I may have no success at luring away all those malcontents.

I think where this may find traction is among existing companies. There are definitely some folks out there with good photo editing software, who’ve kept their packages strategically away from Adobe, maybe in the zone between Photoshop and PS-Elements or whatever. This may be incentive to up their game and take on Photoshop. Then again, this may have Adobe themselves upping their game on Elements. There have been plenty of users of Photoshop who don’t use half its power, but needed a bit more from Elements. Now with the distinctly different business models, maybe they find room for “Elements Pro”.

Either way, yeah, it’s a big climactic shift in software. And we haven’t begun to see what the new point of stability looks like.

-Dave

[Dave Haynie]

I’m ending this experiment… in regular use, the occurrence of small writes is just too common to make the “chipset” RAID5 worthwhile, even with the latest extra stuff in Intel’s C600 I/O chip.

My main purpose of RAIDing it was to have a larger, faster “work” drive; I have the slower external RAID for mass storage. My old system has the single 3TB HDD, which had ok performance for this. But with that much storage, I know it’s not getting as much flow-through to the external RAID or backup as it once did, so I still like the idea of RAID for reliability. I’m going to move this to a RAID1 or RAID10 and see if that changes the performance.

-Dave

[Dave Haynie]

[Steve Rhoden] ” Seems to be the software
companies who think they are too big to fail are embracing
this ransom type nonsense.”

I think so. The old purchase model works in the customer’s favor. Sony or Adobe or whomever has to convince me to upgrade, or I’m sitting pat for the next upgrade cycle. No big deal. This way, they effectively end that choice — I’m either upgrading or I’m giving up the tools. Pretty nasty, IMHO. In the specific case, I only use Photoshop, but now I have to figure that, at some point, I may have to buy CS6 just to lock in on for-purchase versions while they’re still available.

The worst problem here isn’t this, though. But rather, think for a moment about those things you buy (cameras, Vegas, computers, etc) versus those services to which you subscribe (cable/satellite, ISP, smartphone, etc). Which one delivers the most value?

It’s inevitably the purchase. Part of that’s the retail market: I can shop for a good price on New Software Package, New Lens, New Camera, etc. I do the research, try a demo, etc… I’m certain that New Thing is just what I want. I pay, I’m done, it’s mine!

Subscription services usually optimize at the point at which we find cable, satellite, cellular today. They fully expect that some small percentage of users will be so frustrated with their services, they leave. They could make those people happy, but those are the most expensive customers to retain, so they don’t bother. As long as they can bring in more new users than they lose old users, they’re good… most of those new users won’t be the high maintenance kind.

So what this ultimately means is that, while some users are perfectly happy, there many, perhaps a significant percentage, who are in varying degrees of unhappiness, just not having passed that threshold of canceling the service. And this BY DESIGN — this is the services company spending the least money to retain the most customers. And I believe it’s inevitable, this model.

I actually did pass that threshold of frustration last fall, on both my satellite TV and Internet service. In both cases, pretty much all of the company efforts were spent on attracting new customers — they did virtually nothing to keep current customers happy… either you were or you were not, it didn’t seem to matter much.

And that’s the big trap I see with plans like Adobe’s: you pay or you leave.. you have lost any other leverage on the company’s performance. They may very well optimize like most other service companies, to push formerly satisfied users toward the edge of wanting to leave, only the threshold is significantly higher, given the eventual loss of access to existing projects. They can spend more money on applications with competition, like Premiere, but ignore those with little competition, like Photoshop.

This will also have the effect of pushing users slightly dependent on Adobe tools into being all-Adobe. The same kind of dynamic that won Microsoft the Office crown — you already had these other tools, so why pay for alternatives? Once you need more than one Creative Cloud tool, you might as well pony up for them all, based on the current pricing model.

-Dave