RAW vs White balance

The idea that WB may effect RAW files caught my attention. It would be great if I could reduce red channel saturation in tungsten light by changing white balance.

So I wanted to see it myself. Unfortunately I have a *istD and the histogram Java program is for those models that compact the Raw file by using the top 4 bits of the 16 bit word.

It had been a long time since I have been able to play with writing software, so I started a background project to implement the histogram routine in MS Excel QBA. This way I would have more flexibilty in analyzing the results.

It is hard to stop once you get going on a project like this ... so now I have an Excel spreadsheet that has QBA modules that will not onlly draw historams for the *istD but also create tone curves from a series of "white wall" images. I plot the mode of the values in the file. This is the most common value.

I did a white wall test early this morning before the sun came up so I could use pure tungsten light. The results are below.

The first curve is from images shot with white balance set to daylight and the second with white balance set to tungsten.

The differences between the curves is very small. What difference there is I atribute to changing light conditions -- the sun was coming out.

Clearly changing white balance will not reduce Red saturaton in tungsten light.





--
Ed Hannon
http://www.pbase.com/edhannon

edhannon said:

The idea that WB may effect RAW files caught my attention. It
would be great if I could reduce red channel saturation in tungsten
light by changing white balance.

So I wanted to see it myself. Unfortunately I have a *istD and the
histogram Java program is for those models that compact the Raw
file by using the top 4 bits of the 16 bit word.

It had been a long time since I have been able to play with writing
software, so I started a background project to implement the
histogram routine in MS Excel QBA. This way I would have more
flexibilty in analyzing the results.

It is hard to stop once you get going on a project like this ... so
now I have an Excel spreadsheet that has QBA modules that will not
onlly draw historams for the *istD but also create tone curves from
a series of "white wall" images. I plot the mode of the values in
the file. This is the most common value.

I did a white wall test early this morning before the sun came up
so I could use pure tungsten light. The results are below.

The first curve is from images shot with white balance set to
daylight and the second with white balance set to tungsten.

The differences between the curves is very small. What difference
there is I atribute to changing light conditions -- the sun was
coming out.

Clearly changing white balance will not reduce Red saturaton in
tungsten light.

Click to expand...

Great work, so the *ist D RAW file is WB independent!

--
Rob

Ed, it is addictive, isn't it?

I don't exactly see your conclusions in your results as what you have tone responses for a series of white wall images for raw outputs, going by your vertical scale of about a 12 bit range and I'm not sure what the horizontal scale means - perhaps exposure. Clearly, you were clipping at the upper end of the scale so we need to disregard those readings.

I see a definite difference at say the horizontal axis tick between 13 and 14, where the red channel reads something over 3000 in the top chart and about 2500 in the bottom one. That's gain adjustment.

Since I'm patching the histogram program for Jonas, maybe I should patch it for you, too. If I do, I think you are going to see the "typical" Pentax resonse to WB at extreme pre-set settings.

If this keeps up, maybe I should just patch the program to do this for uncompressed DNG and then it can be used by everybody just by converting their raws to DNG using the Adobe utility.

Regards, GordonBGood

edhannon said:

The idea that WB may effect RAW files caught my attention. It
would be great if I could reduce red channel saturation in tungsten
light by changing white balance.

So I wanted to see it myself. Unfortunately I have a *istD and the
histogram Java program is for those models that compact the Raw
file by using the top 4 bits of the 16 bit word.

It had been a long time since I have been able to play with writing
software, so I started a background project to implement the
histogram routine in MS Excel QBA. This way I would have more
flexibilty in analyzing the results.

It is hard to stop once you get going on a project like this ... so
now I have an Excel spreadsheet that has QBA modules that will not
onlly draw historams for the *istD but also create tone curves from
a series of "white wall" images. I plot the mode of the values in
the file. This is the most common value.

I did a white wall test early this morning before the sun came up
so I could use pure tungsten light. The results are below.

The first curve is from images shot with white balance set to
daylight and the second with white balance set to tungsten.

The differences between the curves is very small. What difference
there is I atribute to changing light conditions -- the sun was
coming out.

Clearly changing white balance will not reduce Red saturaton in
tungsten light.

Click to expand...

GordonBGood said:

If this keeps up, maybe I should just patch the program to do this
for uncompressed DNG and then it can be used by everybody just by
converting their raws to DNG using the Adobe utility.

Click to expand...

Gordon, that's a really neat idea with DNG. I was looking to write my own program for Canon CR2 files but DNG thing is certainly better. I guess a lot of dcraw.c code can be reused (with Dave's permission of course).

The idea of RAW not being WB neutral really bothers me about K10D. With my KM A2 and Rebel XT I always shoot RAW and do it on assumption of neutrality to WB. So far all my superficial tests show that CR2 files are WB neutral.

This WB thing is a real bummer and even made me question my decision to go with K10D. I'm beginning to wonder what else K10D RAW data is sensitive to? Tone curve (Natural, Bright), saturation and contrast?

--
http://www.pbase.com/klopus

What the tone curve represents is the relationship between the 12-bit raw value and the exposure (in 1/2 stop steps). I plot the mode (the histogram cell with the largest value).

I agree that there is a difference. However, if Pentax was really compensating for the color of light, I wouold expect a larger change. The tungsten setting should change the tone curve enough to make up the difference between tungsten and daylight color temp.

I plan to do the same test in daylight conditions. If Pentax is adjusting color temp to a degree that is useful then I would expect the tone curves of tungsten light with a tungsten setting to equal that of day light with a daylight setting.

To make it worth while to me to change white balance when using raw mode I would have to see a change in the clipping of the red channel. And I do not see that. It clips at the same EV with tungsten WB as it does with daylight WB.
--
Ed Hannon
http://www.pbase.com/edhannon

tapir said:

The idea of RAW not being WB neutral really bothers me about K10D.
With my KM A2 and Rebel XT I always shoot RAW and do it on
assumption of neutrality to WB. So far all my superficial tests
show that CR2 files are WB neutral.

Click to expand...

I don't have a problem with it, we may have got used to the thought of no signal preconditioning for WB but I'd suggest that if it provides us with more latitude for a given WB then it's a good thing. Then again I've always set WB in camera so it works perfectly for me.

--
Rob

tapir said:

GordonBGood said:

If this keeps up, maybe I should just patch the program to do this
for uncompressed DNG and then it can be used by everybody just by
converting their raws to DNG using the Adobe utility.

Click to expand...

Gordon, that's a really neat idea with DNG.

Click to expand...

I can only second that. If it's possible for you (Gordon) to do this anyone would be able to check their raw files at different ISO values and WB settings to see if they are altered (in a relative way) or not before being written to the memory card.

Jonas

edhannon said:

The idea that WB may effect RAW files caught my attention. It
would be great if I could reduce red channel saturation in tungsten
light by changing white balance.

So I wanted to see it myself. Unfortunately I have a *istD and the
histogram Java program is for those models that compact the Raw
file by using the top 4 bits of the 16 bit word.

It had been a long time since I have been able to play with writing
software, so I started a background project to implement the
histogram routine in MS Excel QBA. This way I would have more
flexibilty in analyzing the results.

It is hard to stop once you get going on a project like this ... so
now I have an Excel spreadsheet that has QBA modules that will not
onlly draw historams for the *istD but also create tone curves from
a series of "white wall" images. I plot the mode of the values in
the file. This is the most common value.

I did a white wall test early this morning before the sun came up
so I could use pure tungsten light. The results are below.

The first curve is from images shot with white balance set to
daylight and the second with white balance set to tungsten.

The differences between the curves is very small. What difference
there is I atribute to changing light conditions -- the sun was
coming out.

Clearly changing white balance will not reduce Red saturaton in
tungsten light.

Click to expand...

Hi Ed,

I read your post with interest. It seems as you are back to the old idea about using a light blue filter then to keep the reds within limits?

When plotting the data, where from do you get the numbers? I guess you run the file through a raw converter (unless I missed something essential in your post)?

Can you elaborate a little? I'm thinking of a situation where you tell the raw converter that you used the Tungsten setting for the picture. Then the converter adjusts accordingly and your data shows just what came out from the converter, not what is in the raw data stream? Judging from your earlier posts I don't think you missed anything but I'm still interested in what the workflow was, exactly, when you got these curves. If you got the time.

regards,

Jonas

I wrote a Quick Basic for Applications (QBA) module in Excel that is modeled on the Java program above.

My initial purpose was to have a histogram program that would work with the *istD file format.

Once I had it working I decided to add the capability to generate a tone curve from a sequence of white wall images. Actually they are a series of images of a sheet of 13x19 Expson matte printer paper. What I plot is the median value of each image. The values are in assending order by EV calulated from the F-stop and exposure.

I did one series with WB set to tungsten and one series with WB set to dayight. Both were illuminated by tungsten light. However, the sun had started to rise during the second series so the light is a little different for that one

I am quite pleased with how smooth the curves are and how little variation there is between each set of images.

I had hoped to do two series in bright dayllight today - but the Honey Do list was too long.
--
Ed Hannon
http://www.pbase.com/edhannon

I was able to get outside for a few minutes to take a series of "white wall" (Epson Heavtweight MAtte paper) tests in daylight shade. My honey do list kept me from getting out during the constant light portion of the day. The daylight series were taken about 3:30 local time with about 10 minutes between first and last.

The first two curves are from this afternoons test and the last are repeats of the early morning tungsten test that I posted before.

IMHO the differences in both cases are well within the range of light variation that I would expect and for me the issue is closed for *istD.

I do plan to further analyze the curves to determine how many stops I really do have between the meter reading and the clipping of the first channel. At some future date I may try to determine if there are blue and red filters that will balance out the curves in daylight and tungsten to give a little more DR without losing too much quality.

In Shade with WB set to Daylight



In shade with WB set to Tungsten



In room lit by tungsten with WB set to Daylight



In room lit by tungsten with WB set to Tungsten



--
Ed Hannon
http://www.pbase.com/edhannon

edhannon said:

IMHO the differences in both cases are well within the range of
light variation that I would expect and for me the issue is closed
for *istD.

Click to expand...

Why don't you shoot with extreme manual white balances? That should highlight any difference much more clearly.

--

The differences are too small to be anything but fly specs.

If the *istD was really doing WB in raw the tungsten WB in tungsten likght would look almost identical to the shade in daylight WB.
--
Ed Hannon
http://www.pbase.com/edhannon

edhannon said:

The differences are too small to be anything but fly specs.

If the *istD was really doing WB in raw the tungsten WB in tungsten
likght would look almost identical to the shade in daylight WB.

Click to expand...

You really should try your software with a custom WB - either correct, or intentionally off a colored object. The color from a given shot is a combination of the values in the raw file, and the color profile used to process it. One can implement WB in either place, and in the case of the Ds, it apears to be a combination of both. I noticed some of the preset WB types didn't show much difference in the raw file, but a custom WB always is different. So a preset daylight WB shot's raw values are quite different from the same shot with a manual WB done on a white target in daylight - though if you run the two images through your raw converter, they come out nearly identical.

tapir said:

GordonBGood said:

If this keeps up, maybe I should just patch the program to do this
for uncompressed DNG and then it can be used by everybody just by
converting their raws to DNG using the Adobe utility.

Click to expand...

Gordon, that's a really neat idea with DNG. I was looking to write
my own program for Canon CR2 files but DNG thing is certainly
better. I guess a lot of dcraw.c code can be reused (with Dave's
permission of course).

Click to expand...

I've written for all of CR2, PEF, and DNG, so far. Yes, Dave's code is a help, although it's not always as efficient as it could be as to speed. Most of Dave's code is Open Source so can be used if you really wanted to.

tapir said:

The idea of RAW not being WB neutral really bothers me about K10D.
With my KM A2 and Rebel XT I always shoot RAW and do it on
assumption of neutrality to WB. So far all my superficial tests
show that CR2 files are WB neutral.

Click to expand...

t isn't really a problem, as all Pentax do with some models is somewhat tune to gain ratios between channels to make them more wb neutral as per the guesses the photographer has made. In some ways, this is better than not adjusting and letting the channels clip where they may as other makers do.

tapir said:

This WB thing is a real bummer and even made me question my
decision to go with K10D. I'm beginning to wonder what else K10D
RAW data is sensitive to? Tone curve (Natural, Bright), saturation
and contrast?

Click to expand...

Ed seems to feel that raw isn't sensitive to WB for the K10D. Almost certainly it isn't senstive to those other things.

So Don't Panic ;-)

Regards, GordonBGood

edhannon said:

What the tone curve represents is the relationship between the
12-bit raw value and the exposure (in 1/2 stop steps). I plot the
mode (the histogram cell with the largest value).

Click to expand...

The only thing I don't understand is that, since this is done from raw data without any processing, why the presumably linear raw values don't change linearily with exposure. Normally if a particular channel is close to clipping to level 4095 at say exposure 8, then at half that exposure the same channel should be about 2048 or half the value. I don't seem to see that in your charts unless I'm looking at them correctly?

Regards, GordonBGood

Jonas B said:

tapir said:

GordonBGood said:

If this keeps up, maybe I should just patch the program to do this
for uncompressed DNG and then it can be used by everybody just by
converting their raws to DNG using the Adobe utility.

Click to expand...

Gordon, that's a really neat idea with DNG.

Click to expand...

I can only second that. If it's possible for you (Gordon) to do
this anyone would be able to check their raw files at different ISO
values and WB settings to see if they are altered (in a relative
way) or not before being written to the memory card.

Click to expand...

Jonas and tapir, working on it. But I decided to finish the PEF version first since no one can convert PEF's to DNG yet. Maybe within the next 6 hours.

Regards, GordonBGood

Ed, I took a break for lunch and of course the answer came to me: "stops" are log base 2 and therefore the plot of the linear response raw data comes out logarithmic.

Everything is about right within experimental error. Sorry for that.

Regards, GordonBGood

GordonBGood said:

edhannon said:

What the tone curve represents is the relationship between the
12-bit raw value and the exposure (in 1/2 stop steps). I plot the
mode (the histogram cell with the largest value).

Click to expand...

The only thing I don't understand is that, since this is done from
raw data without any processing, why the presumably linear raw
values don't change linearily with exposure. Normally if a
particular channel is close to clipping to level 4095 at say
exposure 8, then at half that exposure the same channel should be
about 2048 or half the value. I don't seem to see that in your
charts unless I'm looking at them correctly?

Click to expand...

You were right. The first curve below was with a manual WB shot at a deep red chair. The second shot with a light blue folder cover.

So, it looks like the *istD does not change the raw values in the preset WB modes but does in the manual mode.

However, I notice that it does not do what I had hoped it would - that is extend the DR. The highlights of at least one color still saturate at about 3 stops above meter reading (EV = 8). In fact with the light blue card I lost 1/2 stop.

The curves look as though you might find a color of card that would extend the highlights by about 1/2 stop. But not worth the effort in my opinion.





--
Ed Hannon
http://www.pbase.com/edhannon

Ed, can you please have a look at my question above? If I miss something obvious it is because I'm a little slow and can only ask you to bear with me.

Jonas

JonasB said:

When plotting the data, where from do you get the numbers?

Click to expand...

I wanted to use the RAW histogram program referenced above on my *istD. However, the *istD does not pack the 12-bit RAW data into two 16-bit words like the latter ones do. However, I do not have a Java compiler. So I could not just modify the souce code provided.

So I wrote a quick basic for applications (QBA) module in Excel that adapted the algorithm used in the Java source referenced above.

A side benefit of doing it in Excel QBA is that I ended up with the RAW histogram bin counts from the file in a spreadsheet. These are the counts of how many pixels have a specific value. There are 4096 bins, one for each of the possible values of the 12-bit RAW data.

This allows me to do statistical analysis on the RAW data before it is processed through any converter.

My first question was how many stops above the meter reading does the first channel saturate? To answer this I modified the QBA to read in multiple files and place the histogram bin data in sets of three columns.

I also compute several statistics for each file: average value , standard deviation, median value, how many pixels are 0 (pure black) and how many are 4095 (pure white)

I then set up a curve in Excel that plots the median value for each of the curves against the EV computed from the exposure and apperture.

Then I shoot a series of "white wall" images with exposures separated by 1/2 stops, run them through my Excel QBA and then sort them by EV.

This results in a tone curve for each of the three colors.

The curves I posted show that the first channel clips about 3-stops above the meter reading - regardless of the WB setting. I used a Minolta Spot Meter F that I have calibrated against my *istD and a 77mm limited to give a vaule of 90 when converted with default settings. This correlates well with the internal meter in daylight - but not in tungsten - that is an issue for a different post!

They also show that the *istD does not change the RAW values with preset WB settings but that it does with manual WB.

Ed Hannon
http://www.pbase.com/edhannon