gh2 file sizes

Started Dec 20, 2011 | Discussions thread
Franksson Forum Member • Posts: 93
Re: Pansonic RAW/RW2 Encoding Info From DCraw Source Code ?

Detail Man wrote:

Franksson wrote:
Seasons Greetings!

To you, as well ! ...

... Knowing how the decoder works, gave me a good idea what the encoder must be doing to generate the compressed raw data in the file.

To a point (fortunately, for us). I (and others, quite likely) appreciate your efforts in this regard. Differentiating between speculation and reasonable senses of certainty is always the dicey part in these types of matters.

I agree. It is in my nature to want to know how things work. It is essential to discriminate between what is certain and what is speculation.

... the CR2 mechanism allows for difference values to be only as long as necessary, up to 12 bits if needed, which means that there is no loss of precision. Difference values are further compressed using a lossless Huffman encoding. The CR2 compression is completely lossless. But that doesn't help us here at all.

Did you base that determination on having a look at the "raw" data itself as well as the DCraw (decoding) program statements/commands, etc. ? Just curious.

There is a lot of good information about the CR2 file format available on-line. Much more than for RW2. To get the fine detail of the decoder worked out, I used DCRaw source, raw data from my own pictures and also ITU-T specification T.81.

There is no doubt in my mind that the way the RW2 decoder works means some image precision is lost whenever the size of the difference value requires it to be shifted. How much this affects image quality I cannot say.

Differencing large-valued numbers (in any base-system) does lead to numerical errors, indeed. By "shifting", I take that you mean the intentional truncation of the number of (binary) digits contained in the results of these subtractions ?

Yes. If the difference has been shifted (left shift) then the LSBs of the difference are filled with zeros. Those bits from the original 12 bit difference in the encoder are lost. It means there will be an equivalent error in the pixel value produced by the decoder. Once the difference has been applied to the previous pixel value in DCRaw, these bits in the new pixel will be the same as they were in the previous pixel. (But effectively random now).

I could try to go looking for extreme shift/difference cases in my RW2 files, and get the co-ordinates to enable a visual examination. I'm not sure if it will work, or how long it might take.

It sounds like that may be rather hard to do from just looking at the RW2-level data that has been encoded (as it would already have been rounded-off in encoding process ? What do you think ?

The situation that worries me most is this: in a group of 3 pixels one has a large difference requiring a 4 bit shift value. One or both of the remaining pixels (which may be a different colour to the one with the big difference) have a low previous value and require a low difference to be applied. An error of + or - 15 will have a negligible effect on a large pixel value, but it could be very significant for small values. It will be possible to look for a combination of a 4 bit shift AND a small previous value AND a difference value which is so small it would not have required any shifting. I am not convinced there is a real world problem here. I expect (hope) that the frequency of this situation will be small, and the errors are not noticeable. I will have a go at looking for this and report back. It's a busy time visiting family etc. So it might take a while.

I must correct an error in my earlier post. Blocks of 14 raw pixel values (21 bytes) get encoded into a block of 16 bytes in the RW2 file. So the image is reduced to 76% of its original size. Raw pixels are, of course, only 12 bits so two can be squeezed into 3 bytes.

Thanks again for your interest an diligence. (Obviously), when one posts indications of less than ideal (potential) realities on such a forum as this, it (usually) goes over like a "lead-balloon" ...

I (for one, anyway) would be very interested in your best guess as to the actual state of affairs - regardless of whether or not that comports with the optimistic hopes of Panasonic RW2 image-file recorders and users. While some "truths" may be painful, indeed, I (for one, anyway) would rather be aware of them. Please do look into this matter further as much as you are able. Thank you !!!

I will have a go at it.

PS - I am also curious if the (larger, 20 Mbytes or so for a 10 Mpixel FZ50 RAW image-file) older Panasonic ".RAW" image-files may not be similarly "compromised". I know DCraw decodes these.

I have not looked at these at all. Was that a RW1 file type? If they were 20 Mbytes for a 10 Mpixel image, it doesn't sound like there was any compression at all. 10 Mpixels * 1.5 bytes per pixel (12 bit) gives 15 MByte, allowing another 5 for metadata and an embedded jpeg thumbnail (probably).

 Franksson's gear list:Franksson's gear list
Panasonic Lumix DMC-GH2 Panasonic G85 Panasonic Lumix G Vario 7-14mm F4 ASPH Panasonic Lumix G Vario HD 14-140mm F4-5.8 OIS Panasonic Lumix G Vario 100-300mm F4-5.6 OIS +1 more
Post (hide subjects) Posted by
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow