RAW vs White balance

I have an XT (plan to get K10D). Since I shoot RAW almost exclusively findings of this discussion really bother me. I always used RAW on assumption that it packs WB-neutral image data. Tried following:

1. Took with XT on a tripod a series of photos under a daylight balanced light of a color chart. The only variable is WB.

2. Converted RAW files into TIFF with dcraw in document mode (-d) with same RGB multipliers (-r).

3. Loaded TIFFs into PS CS and compared histograms and color channels on various 100% crops.

4. Result - There're essentially no color differences.

Maybe something is wrong with my procedure but it seems to prove that Canon just records WB information in a header without applying it to the image data itself.

--
http://www.pbase.com/klopus

that can try to do this (if the software works with the K10D PEF files)?

--
Jonas

Here a quick and dirty "real world" test of the influence of WB settings for color rendition in RAW.

The setup: Ds on a tripod. DA40, manual focus, ISO 200, manual exposure 1/3s, f4. Light: Three candles in an otherwise dark room. No other light sources. I took four pictures with WB set to 1) Auto, 2) Sunshine, 3) Tungsten and 4) Shadows and then opened the pics in ACR 3.6 (PS CS2). Everything "auto" disabled of course, default values were used. Here how it looks unprocessed (just resized) at Auto WB:



Camera WB reported in ACR was 5500 both for Auto WB (!) and Daylight settings, 3050 for Tungsten and 6550 for Shadow

I then developed all pics to the same WB of 5500 Kelvin and compared those four pics carefully. It was not easy to discern any differences at all to the naked eye - but switching between them one could detect minuscule ones; I detected some differences in the blue channels, especially while comparing the instances with Tungsten and Shadow WB with each other; but even here nothing really strong. Here the four pictures:



(For those who want to make their own comparisons here a larger version without text captions: http://www.pbase.com/phsan/image/70665021/original )

To follow up the difference in the blue channel I then decided to go to the extremes. I upped EV in ACR to +1,80 - still avoiding any clipping but making differences more prominent this way, and chose a corrected white balance for the musical clock on the right of the pic - that was 2000 Kelvin; remember the clock was lit by the candles only! I then took the two pics with the most extreme WB - the Tungsten one with originally 3050 and the Shadow one with 6550. Here a crop of those two:



You see now a visible difference between the two pics. In spite of being developed with identical values the one that had Tungsten WB has a brighter blue channel; the Shadow one has an overall warmer look because of its darker blue channel.

Frankly I still find the difference of those two shots under those rather extreme conditions (raised EV, opposing presets, color temperature lowered rather extremely) not that pronounced. Yes, there is an influence, but have a look here - I just upped color temperature a tiny little bit on the Tungsten WB pic from 2000 to 2100 to get almost identical renditions:



Interestingly enough the other way round isn't possible - in ACR you can't lower color temperature below 2000, so the Tungsten WB pic indeed has a little bit more headroom for adjustment here.

My personal conclusion: It may be a good idea to go the preset or manual balance way in rather extreme conditions. For normal conditions I think adjusting WB for RAW shooting is not called for. In camera that is. ;-)

And of course you're invited to draw your own conclusions here!

--
Phil

GMT +1

Jonas B said:

that can try to do this (if the software works with the K10D PEF
files)?

Click to expand...

Will be interesting to see how (if) this applies to DNG and if there's any difference from PEF.

--
http://www.pbase.com/klopus

Jonas B said:

that can try to do this (if the software works with the K10D PEF
files)?

Click to expand...

I don't think the software in its current form will work with K10D PEF files as it depends on being able to read byte packed raw data and won't be able to read compressed PEF K10D files.

Regards, GordonBGood

philzucker said:

My personal conclusion: It may be a good idea to go the preset or
manual balance way in rather extreme conditions. For normal
conditions I think adjusting WB for RAW shooting is not called for.
In camera that is. ;-)

And of course you're invited to draw your own conclusions here!

Click to expand...

I think you are right in your assessment, and your conclusions are the same as mine. I think one can use either AWB or a fixed medium WB such as Sunny for normal conditions without much effect as the gain tags in the file will take care of whatever is happening to the pre-ADC gain for the data in the raw file.

Regards, GordonBGood

I would second that.

And it means I'm back to the first conclusion again, it doesn't really matter. At least not in this case. Perhaps it is best not to use Tungsten setting for outdoor work.

Thank you Phil. Some day when I have some spare time I think I'll check the opposite situation.

But again, I'm still surprised that the whole topic exists and in a way happy that the differences aren't bigger than we have seen.

--
Jonas

GordonBGood said:

Jonas B said:

that can try to do this (if the software works with the K10D PEF
files)?

Click to expand...

I don't think the software in its current form will work with K10D
PEF files as it depends on being able to read byte packed raw data
and won't be able to read compressed PEF K10D files.

Click to expand...

I guess you are right. Then the IRIS software mentioned here:
http://forums.dpreview.com/forums/read.asp?forum=1036&message=19789342
probably can't do it either.

regards,

Jonas

Jonas B said:

GordonBGood said:

Jonas B said:

that can try to do this (if the software works with the K10D PEF
files)?

Click to expand...

I don't think the software in its current form will work with K10D
PEF files as it depends on being able to read byte packed raw data
and won't be able to read compressed PEF K10D files.

Click to expand...

I guess you are right. Then the IRIS software mentioned here:
http://forums.dpreview.com/forums/read.asp?forum=1036&message=19789342
probably can't do it either.

Click to expand...

Nope. Has anyone sent Dave Coffin of dcraw a sample PEF or two so he can show us how to decompress them?

Regards, GordonBGood

Hi Phil!

Wow, interesting thread. Thanks to vortout for starting it and Jonas for telling me about it.

The ISO 1600/3200 = shifted ISO 800 fact was known before, so no surprise. Gordon's K10D examination and the implication of using a fixed low-noise amp are especially interesting in this light..

I always suspected there might be some WB processing done. I could not see it, though. And Phil's post probably explains why.

But I'd assume that by correcting the RAW WB, even if a very similar colour can be achieved, there would be higher noise in the channel with the most correction. Whethe it's enough to worry about, is another question, though...

Cheers
Jens

--

'Well, 'Zooming with your feet' is usually a stupid thing as zoom rings are designed for hands.' (Me, 2006)
http://www.jensroesner.de/
--=! Condemning proprietary batteries since 1976 !=--

Thought I'd give the technique a spin in a bit more practical circumstance.

The first shot is just shot with preset daylight WB, second WB with my trusty pink paper (and then WB reset by grey point in PPL3.) Sunlight, manual exposure (actually the -1 shot of a bracket.)

Too lazy to crop 'em for y'all, cuz Bond is about to start again.

CurtisR said:

Thought I'd give the technique a spin in a bit more practical
circumstance.

The first shot is just shot with preset daylight WB, second WB with
my trusty pink paper (and then WB reset by grey point in PPL3.)
Sunlight, manual exposure (actually the -1 shot of a bracket.)

Click to expand...

Could you clarify if you reset WB for both pics by grey point? Only then they would be comparable, and even then only if you got exactly the same grey point as WB "anchor". Or asked the other way round: Was identical color temperature in Kelvin set to both pics in RAW development?

--
Phil

GMT +1

philzucker said:

CurtisR said:

Thought I'd give the technique a spin in a bit more practical
circumstance.

The first shot is just shot with preset daylight WB, second WB with
my trusty pink paper (and then WB reset by grey point in PPL3.)
Sunlight, manual exposure (actually the -1 shot of a bracket.)

Click to expand...

Could you clarify if you reset WB for both pics by grey point? Only
then they would be comparable, and even then only if you got
exactly the same grey point as WB "anchor". Or asked the other way
round: Was identical color temperature in Kelvin set to both pics
in RAW development?

Click to expand...

No and no. Though I don't think that makes them uninteresting to compare. The colors are a bit different, but the degree to which detail is lost to clipping in the red channel is appreciable (although silkypix does a pretty impressive job of faking the clipped red channel.)

CurtisR said:

philzucker said:

CurtisR said:

Thought I'd give the technique a spin in a bit more practical
circumstance.

The first shot is just shot with preset daylight WB, second WB with
my trusty pink paper (and then WB reset by grey point in PPL3.)
Sunlight, manual exposure (actually the -1 shot of a bracket.)

Click to expand...

Could you clarify if you reset WB for both pics by grey point? Only
then they would be comparable, and even then only if you got
exactly the same grey point as WB "anchor". Or asked the other way
round: Was identical color temperature in Kelvin set to both pics
in RAW development?

Click to expand...

No and no. Though I don't think that makes them uninteresting to
compare. The colors are a bit different, but the degree to which
detail is lost to clipping in the red channel is appreciable
(although silkypix does a pretty impressive job of faking the
clipped red channel.)

Click to expand...

If you wanted do see if channel clipping would be less in the red channel with a custom WB done on a pink paper compared to the daylight WB preset - did I get this right? - you would have to develop the RAWs to the same color temperature and then compare the results. Only then you can see how much an analog gain in the red channel prior to ADC influences red channel clipping.

With the pics you present here you just show that different WB in "developed" or processed images - but not in RAW files - influences color channel clipping. But the point of this thread was IMO to see how WB influences results in RAW files, and how this influence manifests itself in later processing of those files.

Maybe you could develop both PEFs again - the joys of RAW! ;-) - to exactly the same tint and color temperature in Silkypix (I don't know the interface of Silkypix, but I guess there will be the possibility to enter tint and color temperature numerically) and then present us crops showing the red flower of both versions? That would be interesting!

--
Phil

GMT +1

I'm resurrecting this old thread, as I believe one of the OP's observations that there is something strange about the DS camera's exposures between ISO 200 and ISO 400 apply to the K100D and explain why Phil Askey's review of this camera showed more DR "headroom" at ISO 400 than at ISO 200. As follows:

GordonBGood said:

GordonBGood said:

GordonBGood said:

The second part of the observations by the OP was that there is a
discrepency in gain for all channels between ISO 200 and ISO 400.
As per your earlier post with the multitude of charts, it looks
like this is true in every setting for WB (I would assume for
Custom WB as well). With my knowledge of the Pentax raw format for
the K100D, I assume that there is the same offset on all channels
in raw of about 128 counts, so what we seem to be seeing here is
that the ISO 400 charts are just reduced by a factor of 2 which is
just another form of bit clipping - not using the upper half of the
range. I'm sure that if one looks in the raw file, they will find
that the stated maximum range reflects that the gain has been
reduced. In other range, there is just one ISO sensitivity setting
that is the real gain of the pre-amplifier and ADC: ISO 200. It
looks like all of the others are just software manipulated,
first just amplified by the maximum range setting for ISO 400, then
just bit shifted for each subsequent increase in ISO by a factor of
two.

Click to expand...

Click to expand...

I'm not looking at the missing codes, which are clear indications
of bit shifting for ISO 800 and above, including ISO 800 as
Jonas B himself sees, too. I'm referring to the double width of
the ISO 200 histogram as compared to all of the other histograms
even though the exposure values were changed to compensate for the
higher differences in ISO by using half/double the shutter speed
with a fixed aperture. This was also the one of the observations
in the OP. It looks like ISO 400 may be accomplished as compared
to ISO 200 not by using a bit shift but merely be keeping the same
gain and changing the maximum white point value so that the raw
values will be scaled by a factor of two in the raw convertor
(which is basically the same effect as a bit shift).

Click to expand...

The following images are "lifted" from Jonas B's post to this thread, with all credit due to him.

As the DS raw readings at ISO 200:



and for the DS raw readings at ISO 400 with half the exposure time but with otherwise identical settings:



Note that the width of the histogram which actually has meaningful data is something like a third again wider at ISO 200 than at ISO 400, where for a half the exposure time the histogram widths of meaningful data should be exactly the same width.

What we seem to be seeing here is that, when one sets these "D" cameras or the K100/110D to ISO 400, they are not actually increasing the gain of the signals coming from the sensor before conversion by the ADC by a factor of two, but rather just increasing the gain by about a factor of 1.5 with a tag to indicate that the range is now two thirds (or whatever the factor that was chosen was). I believe this explains why there is more headroom on ISO 400 and higher ISO sensitivities for these cameras than for at ISO 200.

It may also explain why those who use raw with any application (including ACR) that doesn't know how to read the tag indicating true gain or to make allowances for this characteristic of these cameras will find ISO 400 and above raw images to look under exposed and require positive exposure compensation.

This discrepency does not seem to apply to the new 10 MP K10D in that it appears that all pre-scaling is done in that camera using the new PRIME engine before the raw data is stored.

This is not specifically a problem as long as one is aware of it in that, first, one may enjoy the extra headroom and, second, boosting the exposure in post processing does not add any noise amplification that would not have been added in the same way by amplifying the signal before ADC conversion.

Regards, GordonBGood

GordonBGood said:

I'm resurrecting this old thread,

Click to expand...

Yes. and why not. You are confirming something I have wondered about, but never really understood the reasons for. Or how I shall put it.

Nice pictures also! The PP is a little so so but one can't complain about the composition. Everything in the middle for better, closer look? Well, that's boring...

Nearly OT: Do you know of any free available software that can do the same with raw files from a Canon?

regards,

Jonas

Jonas B said:

Nearly OT: Do you know of any free available software that can do
the same with raw files from a Canon?

Click to expand...

Hmmm, what model Canon and what OS do you run? Among other things, I am working on Canon CR2 files now, and could likely quickly cobble together something rough and ready, but I work in C# under DotNet2 (Windows 2K+). I could likely get it to work on the Mac or Linux under Mono, but I haven't tested it. e-mail to the address in your profile?

Regards, GordonBGood

GordonBGood said:

Jonas B said:

Nearly OT: Do you know of any free available software that can do
the same with raw files from a Canon?

Click to expand...

Hmmm, what model Canon and what OS do you run? Among other things,
I am working on Canon CR2 files now, and could likely quickly
cobble together something rough and ready, but I work in C# under
DotNet2 (Windows 2K+). I could likely get it to work on the Mac or
Linux under Mono, but I haven't tested it. e-mail to the address
in your profile?

Click to expand...

In short: Canon 5D, Windows XP, the profile mail addy works

All I would like to do is to find out if the 5D do something to the raw files in the line with what we have seen in this thread. Maybe this is allready well known? As I'm new to do the 5D I have some catch up to do in some areas.

Thank you,

Jonas

Jonas B said:

GordonBGood said:

Hmmm, what model Canon and what OS do you run? Among other things,
I am working on Canon CR2 files now, and could likely quickly
cobble together something rough and ready, but I work in C# under
DotNet2 (Windows 2K+). I could likely get it to work on the Mac or
Linux under Mono, but I haven't tested it. e-mail to the address
in your profile?

Click to expand...

In short: Canon 5D, Windows XP, the profile mail addy works

All I would like to do is to find out if the 5D do something to the
raw files in the line with what we have seen in this thread. Maybe
this is allready well known? As I'm new to do the 5D I have some
catch up to do in some areas.

Click to expand...

Jonas, after thinking about this last night, I thought it might be easier to just patch the Java source files for the PEF program you already used so as to read the raw data from CR2 files, uncompress it, and pass it to the routines the guy already wrote. In that way you don't need to download the DotNet 2.0 runtime (if you don't have it), as if you could run this program, you should be able to run the patched version.

Haven't used Java for a while, and have misplaced my compiler, but am downloading a free free version. Shouldn't take too long after that.

Regards, GordonBGood

GordonBGood said:

Jonas B said:

GordonBGood said:

Hmmm, what model Canon and what OS do you run? Among other things,
I am working on Canon CR2 files now, and could likely quickly
cobble together something rough and ready, but I work in C# under
DotNet2 (Windows 2K+). I could likely get it to work on the Mac or
Linux under Mono, but I haven't tested it. e-mail to the address
in your profile?

Click to expand...

In short: Canon 5D, Windows XP, the profile mail addy works

All I would like to do is to find out if the 5D do something to the
raw files in the line with what we have seen in this thread. Maybe
this is allready well known? As I'm new to do the 5D I have some
catch up to do in some areas.

Click to expand...

Jonas, after thinking about this last night, I thought it might be
easier to just patch the Java source files for the PEF program you
already used so as to read the raw data from CR2 files, uncompress
it, and pass it to the routines the guy already wrote. In that way
you don't need to download the DotNet 2.0 runtime (if you don't
have it), as if you could run this program, you should be able to
run the patched version.

Haven't used Java for a while, and have misplaced my compiler, but
am downloading a free free version. Shouldn't take too long after
that.

Click to expand...

Hi Gordon,

Well, I'm grateful for anything that can be done. At the same time I don't think the question is that important and worthy too much work. I also guess the answer is out there somewhere.

(Just thinking loud here There must be many that, like me, leaves the White Balance setting at AWB or Sunshine and then never touches it. Should there be a problem it should be well known. Hmm. Thinking about it we didn't know about the DS behaviour until recently... And it did shed some light on some things some of us think we have run into when post processing...)

Ok, if it's easy done and if you don't loose much time I'm interested in trying a similar tool for the 5D.

thank you,

Jonas

PS: I think I have the Net framework or what it is called loaded here. No idea what version it is, or how to check it. If the malykh tool is a Java thing it will of course work.