gh2 file sizes

Started Dec 20, 2011 | Discussions thread
kenw Veteran Member • Posts: 6,402
Re: Raw: why?

Great catch on this. Late reply because I was out and about for the holidays and when I saw this couldn't really post.

I checked the DCRAW source code and indeed that is exactly how it behaves.

As far as the impact goes my thoughts are:

  • Within a given color channel I don't think it is possible for the imaging system (AA filter and the sharpest lens imaginable) to have enough contrast across just three pixels to actually have a count large enough to require a bit shift in one pixel and a count small enough to be impacted by rounding error in the next. The MTF would never be better than say 50% at that pitch so if you had a count of around 4000 that might cause differences to be bit-shifted by two the adjacent pixel in the same color channel could at the very least be 2000 and the rounding error not relevant. Image forming light really couldn't create a situation with a count in the thousands in one pixel and a count in the tens in the very next.

  • Across color channels (as the bit shift is for both color channels in a row) I also suspect it isn't likely since the CFA filters don't completely reject all out of band light nor do we encounter monochromatic light in most photos. I suppose it might be possible to pick just the right wave length of monochromatic light with the result that one color channel was in the thousands and the other channel much lower - but I still doubt it could be low enough to be affected by the stronger channel's difference rounding. Basically I'm saying I have trouble thinking of a scenario in which the color channels can be so dramatically different that you have counts in the thousands in one and counts in the tens in the other. Keep in mind R and B are never in the same row, the bit shift sharing is only ever between R and G or between G and B which in both cases are adjacent in the spectrum.

So those two together mean I think there is no way for image forming light to be impacted by the rounding bit-shifts in the RW2 format. This is on account of the MTF of the real optical system and the spectral characteristics of the CFA filters as well as the spectrum of most light.

There is another interesting case and that is hot-pixels. These of course can be a huge isolated count in a single pixel right in the middle of a shadow. If that is left in the RAW file it could affect rounding of immediately adjacent low value pixels (the bit shift only affects three pixels at a time). The RAW converter detects the hot pixel and interpolates over it but the rounding error is left in the adjacent normal pixels. This post about the RW2 format was really timely because it did explain a perplexing result from my dark frames:

That's ISO2000 where the camera is applying a digital multiply so there are output values that don't exist. See the wide regular gaps in the histogram. But in fact some of the "don't exist" values do in fact exist at low counts. At the left of the histogram there is a "shadow" of the main histogram in the "excluded" output values. It is this difference rounding that is causing them to appear - these odd balls were adjacent to a particularly hot pixel and after the rounding ended up at a value we wouldn't expect to be valid with digital multiplying occurring in the camera. Note also the histogram bifurcates just above 144 (which is 128+16 or the threshold at which the first bit shift would need to occur for hot pixels).

This sort of mystified me when I did the tests, but now that I understand how the RW2 difference bit shifts work it all makes sense.

This histogram is also very informative for another reason. Above I noted that hot pixels could cause rounding errors in adjacent pixels, the question is would it be significant. The histogram gives the answer - no it isn't significant. We have here that second "shadow" of the histogram that clearly shows us all the pixels that have had a rounding error and we can see they are very infrequent - many orders of magnitude below all the pixels with equivalent magnitudes of sensor noise. So the hot pixel rounding error in the final image is just like adding a vanishingly small amount of additional read noise which is actually way, way, way less than the native read noise of the sensor.

Anyway, those were my thoughts - some tested and some just supposition.

Again, big thanks for pointing this out - I hadn't code dived DCRAW closely to see what was going on with RW2.
Ken W
See plan in profile for equipment list

 kenw's gear list:kenw's gear list
Panasonic Lumix DMC-GM1 Olympus E-M5 II Nikon Z7 Nikon Z 14-30mm F4 Nikon Z 24-200mm F4-6.3 VR +37 more
Post (hide subjects) Posted by
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow