unexpected zip compression of DNxHD file

[Keith Mcgregor]

Your compression scheme is not listed, which software are you using? In mac, the standard right click and choose compress just throws out redundant and repeated code, providing in a smaller file. In video that is high quality and not GOP there really is no redundancy or repeated code so hence no real reduction. When I use stuffit deluxe I might get a 25% reduction when I really try to squish it. Again, with video we need that information and there is no repeating code so hence no reduction in file size.
If you change the depth to 8, millions of colors, then you’ll see a reduction in both size and quality, but if you do a test can you visibly see the difference? Most peeps won’t notice, and if it still looks good to the eye then mebbie the producers won’t know either.

Here’s an interesting read I found:
https://www.maximumcompression.com/data/jpg.php

Hope this helps.
-Keith

Reality? What did you make it?

[Graham Macfarlane]

Hi Keith,

Thank you for your reply!

I’m using After Effects CS5 on Win 7 x64.
I’ve attached a screen grab of the settings explained in my first post:

I had a closer look at all my DNxHD files and noticed that compared to the ProRes originals there is a very slight noise softening that is just appreciable at around 800% zoom. This fact though, should affect both A2 and B2 equally.

I also realised that video A contains a slightly higher proportion of areas of near solid colour (eg a clean painted wall) compared to video B (eg containing a bush). Could this really attribute to near 50% compressibility?

Below is a list of all the other compression tests I’ve run to try to make sense of this puzzling situation.

ProRes originals (I suspect they were originally 8bit but supplied to me saved as 10bit, unsure how to check):
* A1 = 1800MB (zipped: 1791MB) (approx. 0.5% compression)
* B1 = 2010MB (zipped: 1985MB) (approx. 1% compression)

DNxHD: (only 100 frames compressed for speed)
* A2: 8bit prj – output depth-millions – 1080p/25 DNxHD 185 8bit = 90.5MB (zipped: 49MB)
* A3: 16bit prj – output depth-trillions – 1080p/25 DNxHD 185 10bit = 90.5MB (zipped: 60MB)
* A4: settings as with A3 but 100f are from different point in video = 90.5MB (zipped: 41MB)

Strangely A4 zips better than A3. Both have locked off cameras but A3 has two moving actors in shot compared to A4 only having one actor moving in shot.

* B2: 8bit prj – output depth-millions – 1080p/25 DNxHD 120 8bit = 59MB (zipped: 58.5MB)
* B3: 8bit prj – output depth-millions – 1080p/25 DNxHD 185 8bit = 89.5MB (zipped: 88.5MB)
* B4: 16bit prj – output depth-trillions – 1080p/25 DNxHD 185 10bit = 89.5MB (zipped: 88.5MB)

Graham Macfarlane
3D animator and VFX specialist
London UK

[Walter Soyka]

The efficiency of lossless compression like ZIP will be very dependent on the source material. This is a stark contrast to lossy visual compression, which can be constrained, sacrificing quality as necessary to maintain bitrate.

Roughly speaking, DCT-based image compression (like DNxHD or JPEG) works as follows:

• Convert the image to multiple grayscale channels (often spliting an image into luma and two channels of chroma, similar to YUV video)
• Break each of these monochromatic planes into blocks
• Transform the spatial pixel data into a frequency space for each block through a process called discrete cosine transform
• Quantize (round off) the coefficients that represent the frequency data (this is the first opportunity for data reduction through eliminating visual data)
• Re-order the DCT coefficients in a zig-zag pattern to increase the likelihood of getting a run of several zeroes in a row, and performs run-length encoding (basically saying 10×0, instead of 0000000000).
• Huffman encoding, which means that the data is optimized so that frequently-occurring symbols are abbreviated to take up less space, and rarely-occurring symbols are expanded as necessary.

DNxHD is a highly-constrained VBR intraframe scheme. It is not intended to yield maximum compression of the signal, but rather a predictable bit rate. It’s therefore possible that there would be further opportunities for compression after the fact, especially where there’s repetitive data (perhaps that cleanly painted wall that appears largely unchanged in frame after frame after frame).

So basically, yes, it’s certainly plausible that in a few cases, but certainly not all, you’ll get huge efficiency out of ZIP compression for video.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events

[Graham Macfarlane]

Thank you Walter for the confirmation and the interesting info on compression!

Graham Macfarlane
3D animator and VFX specialist
London UK