RAW vs White balance

Started Nov 20, 2006 | Discussions
edhannon
edhannon Senior Member • Posts: 1,761
Re: RAW vs White balance

GordonBGood wrote:

Ed seems to feel that raw isn't sensitive to WB for the K10D.

I ran my QBA program against data from my *istD.

WIth a little proding from CurtisR I found that the *istD does not change the RAW data with changes in the PRESET WB but that it does with MANUAL WB.

-- hide signature --
 edhannon's gear list:edhannon's gear list
Pentax K10D Pentax K-3 Pentax smc DA 16-45mm F4 ED AL Pentax smc FA 77mm 1.8 Limited +1 more
Jonas B Forum Pro • Posts: 14,596
Re: Manual WB does impact *istD RAW

Ok. Thank you for explaining all that. It made an interesting read, and also well in line with the observations over the DS' behaviour.

So, a manual whibal against, say a pink or light red paper, could possibly hold back the red channel half a stop... Taken from memory it seems to be about the same effect a weak blue filter can provide. Of course, my memory isn't the best. Right now I even don't remember the name of the woman that "invented" the method. How stupid.
Well, half a stop isn't very much anyway.
It will be interesting to see if the K10D behaves the same way.

regards,

Jonas

CurtisR Regular Member • Posts: 263
Re: Manual WB does impact *istD RAW

Jonas B wrote:

Ok. Thank you for explaining all that. It made an interesting read,
and also well in line with the observations over the DS' behaviour.
So, a manual whibal against, say a pink or light red paper, could
possibly hold back the red channel half a stop... Taken from memory
it seems to be about the same effect a weak blue filter can
provide. Of course, my memory isn't the best. Right now I even
don't remember the name of the woman that "invented" the method.
How stupid.
Well, half a stop isn't very much anyway.

Well, I haven't really seen a limit on the number of stops of sensitivity are possible with this method, but it's still cool that you can mimic old-school analog filters by using colored paper. I mean, you can go down to the Home Depot and grab a whole massive set of filters that are optically perfect and induce no flare and just happen to fit every single lens you own and weigh practically nothing and you can use them to wipe your bum in a pinch. Oh yeah... For free!

It will be interesting to see if the K10D behaves the same way.

regards,

Jonas

ddd New Member • Posts: 19
New version of malykh-pef (K10D, *ist D are supported now)

From a Russian forum:

New version of malykh-pef:
http://malykh/temp/2006-12-11/malykh-pef-2006-12-11.zip

Now it supports nearly all PEF formats from available Pentax DSLRs (K10D, *ist D also).
You'll need JRE (Java Runtime Environment) 1.4 or higher to run this.

GordonBGood Veteran Member • Posts: 6,308
Re: Manual WB does impact *istD RAW

edhannon wrote:

You were right. The first curve below was with a manual WB shot at
a deep red chair. The second shot with a light blue folder cover.

So, it looks like the *istD does not change the raw values in the
preset WB modes but does in the manual mode.

Ed, I think that if you check your results carefully, that the R/G and B/G ratios are changing, but more slightly - something in the order of 25%.

However, I notice that it does not do what I had hoped it would -
that is extend the DR. The highlights of at least one color still
saturate at about 3 stops above meter reading (EV = 8). In fact
with the light blue card I lost 1/2 stop.

The curves look as though you might find a color of card that would
extend the highlights by about 1/2 stop. But not worth the effort
in my opinion.

Of course you don't extend the DR much because the green channel gain is left untouched; all the camera does is change the relative gain of the red and blue channels to optimize for the requested wb setting. In the case of wb presets and especially Manual WB, this effectively reduces the gain of the strongest channel to avoid that channel clipping sooner than expected, resulting in slight extra exposure headroom.

You may be interested in vortout's other thread related to the K10D, somewhat related to this one at:

http://forums.dpreview.com/forums/read.asp?forum=1036&message=21236953

Regards, GordonBGood

GordonBGood Veteran Member • Posts: 6,308
Re: RAW vs White balance

Jonas B wrote:

tapir wrote:

GordonBGood wrote:

If this keeps up, maybe I should just patch the program to do this
for uncompressed DNG and then it can be used by everybody just by
converting their raws to DNG using the Adobe utility.

Gordon, that's a really neat idea with DNG.

I can only second that. If it's possible for you (Gordon) to do
this anyone would be able to check their raw files at different ISO
values and WB settings to see if they are altered (in a relative
way) or not before being written to the memory card.

Jonas and tapir, I've sent the DNG version of the Java program to the e-mail addresses from your profiles. You're the "Beta testers". Let me know if there are problems.

Regards, GordonBGood

Dark Mist Contributing Member • Posts: 558
Re: I don't believe it unless you tell me...

I think you will find that the histogram is from the software's interpretation of the RAW data. Different software may give differing results, as would the camera's own histogram display.

I'm not sure that what we are seeing here means anything other than make sure you are happy with the results of your RAW conversion software.

I only saw this post after playing with Lightroom for a couple hours. The colour temperature reported in Lightroom differs from that reported by the Pentax software. The Adobe software shows the temp as being cooler. I can get the resulting TIFF files to look identical in colour balance.

tapir Veteran Member • Posts: 3,860
Re: RAW vs White balance

GordonBGood wrote:

Jonas and tapir, I've sent the DNG version of the Java program to
the e-mail addresses from your profiles. You're the "Beta
testers". Let me know if there are problems.

Gordon, wow, you're fast I'll be able to check it today evening at home with CR2's converted to DNG and will report.

Thanks again!

-- hide signature --
GordonBGood Veteran Member • Posts: 6,308
Re: RAW vs White balance

tapir wrote:

GordonBGood wrote:

Jonas and tapir, I've sent the DNG version of the Java program to
the e-mail addresses from your profiles. You're the "Beta
testers". Let me know if there are problems.

Gordon, wow, you're fast I'll be able to check it today evening
at home with CR2's converted to DNG and will report.

Jonas and tapir: Maybe a little too fast. See a discrepency in the histograms produced directly from DS/K100D PEF's and the DNG's produced from those files, although ACR produces exactly the same histogram with equivalent processing. Currently looking into it. Give me a few hours....

Regards, GordonBGood

GordonBGood Veteran Member • Posts: 6,308
Re: I don't believe it unless you tell me...

Dark Mist wrote:

I think you will find that the histogram is from the software's
interpretation of the RAW data. Different software may give
differing results, as would the camera's own histogram display.

I'm not sure that what we are seeing here means anything other than
make sure you are happy with the results of your RAW conversion
software.

I'm not sure you read through this whole thread; what we are talking about are histograms produced from the raw data with no processing whatsoever . So provided there aren't any programming errors, histograms from different software doing the same thing (ie. not processing) from the same image should produce identical histograms.

I only saw this post after playing with Lightroom for a couple
hours. The colour temperature reported in Lightroom differs from
that reported by the Pentax software. The Adobe software shows the
temp as being cooler. I can get the resulting TIFF files to look
identical in colour balance.

The reasons that Adobe software may show a different colour temperatures than Pentax software are many, as follows:

1. The colour temperature can be calculated based on a black body radiator and tuned for other types of illuminant power spectrum (tint), as Adobe does, or can be calculated based on a daylight illuminant power spectrum and tuned differently.

2. The base relative gain ratios of the red and blue channels to the green channel may be assumed incorrectly by Adobe.

3. The Auto WB (where it is done outside the camera) algorithm is certainly different for Adobe than for Pentax.

4. etc.

None of this processed colour wb/temperature have any bearing on the raw WB that is discussed in this thread. The OP discusses observed changes in the WB relative gains between the RGB channels due to in-camera WB setting in the recorded raw data before any processing.

Regards, GordonBGood

Jonas B Forum Pro • Posts: 14,596
Re: RAW vs White balance

GordonBGood wrote:

tapir wrote:

GordonBGood wrote:

Jonas and tapir, I've sent the DNG version of the Java program to
the e-mail addresses from your profiles. You're the "Beta
testers". Let me know if there are problems.

Gordon, wow, you're fast I'll be able to check it today evening
at home with CR2's converted to DNG and will report.

Jonas and tapir: Maybe a little too fast. See a discrepency in
the histograms produced directly from DS/K100D PEF's and the DNG's
produced from those files, although ACR produces exactly the same
histogram with equivalent processing. Currently looking into it.
Give me a few hours....

@anyone: Just in case you wonder: there are some emails exchanged between me, Gordon and tapir. It's not like Gordon works in a vacuum. When the software works with DNG files I hope somebody having a K10D can jump in and take some test pictures as well.

@Gordon: No sweat. Just take your time with the binary mysteries (or whatever it is when Java is involved).

Jonas

GordonBGood Veteran Member • Posts: 6,308
Re: RAW vs White balance

Jonas B wrote:

GordonBGood wrote:

Jonas and tapir: Maybe a little too fast. See a discrepency in
the histograms produced directly from DS/K100D PEF's and the DNG's
produced from those files, although ACR produces exactly the same
histogram with equivalent processing. Currently looking into it.
Give me a few hours....

Fixed it and sent copies to Jonas and tapir. Perhaps someone can post both the DNG_Analyser and the PEF_Analyser somewhere so anyone can do what testing pleases them.

Found two problems: forgot a pair of brackets in an byte data array address calculation for the Adobe type DNG's and had to move one statement outside a conditional clause in a couple of places for packed byte DNG's as from the K10D. All seems to work fine now.

So who cares as long as it's fixed

@anyone: Just in case you wonder: there are some emails exchanged
between me, Gordon and tapir. It's not like Gordon works in a
vacuum. When the software works with DNG files I hope somebody
having a K10D can jump in and take some test pictures as well.

Actually, they have a choice to generate DNG's or PEF's in camera, as my PEF_Analyser handles compressed PEF's from the K10D as well as large PEF's from the D or packed byte PEF's from the rest. I would think that in a month or so Adobe will release ACR 3.7, which will convert compressed PEF's from the K10D to DNG's.

Regards, GordonBGood

Dark Mist Contributing Member • Posts: 558
Re: I don't believe it unless you tell me...

Thanks for the explanation.

I did follow what the thread is about. I'm just unsure how the histogram gets generated from the RAW data. It seems to me that some software must be used to read the raw file and make the histogram. Even the histogram display on the camera's LCD is a software interpretation of the RAW data held on the memory card.

I can see what you are saying about the presets giving different readings. It's demonstrated in your histograms. I was just suggesting that the software used to display or convert the file and generate the histogram would make as much if not more difference. Your explanation about how Adobe measure colour temp would seem to suggest this is possible too.

All conversion software seems to use the header info in the RAW file to set a starting point for exposure, colour temp, contrast etc which can then be manipulated before converting. Of course if the histograms from a number of different conversion programs were identical that would prove the difference demonstrated is a result of how the camera records and saves the RAW data.

I don't pretend to have an expert understanding. I just like all possibilities to have been considered and at least steps taken to remove variables before accepting something new as fact.

GordonBGood Veteran Member • Posts: 6,308
Re: I don't believe it unless you tell me...

Dark Mist wrote:

I did follow what the thread is about. I'm just unsure how the
histogram gets generated from the RAW data. It seems to me that
some software must be used to read the raw file and make the
histogram. Even the histogram display on the camera's LCD is a
software interpretation of the RAW data held on the memory card.

The software works as follows:

1. Find the start of the raw data.

2. For all the rows and all the columns determine what colour channel the reading will be binned against from the Bayer array of the pattern such as:
RG
GB
repeated horizontally and vertically as for the Pentax cameras.

3. Uncompressed/Unpack/otherwise read the 12 bit data (again in the case of Pentax).

4. According to the value read, increase that bin number for that colour by one (all bins having been originally be initiallized to zero counts).

5. When all rows and columns have been read and binned, display all the bins (values) across the horizontal axis for each colour, with the colours being the logical or of the combinations of colours (ie additive colours) and the height proportional to the number if times the bin was increments.

That's a raw data colour histogram with no processing and that's what was produced here. If you read Java, the source code was provided with the program and one can see that this is what's done.

Regards, GordonBGood

Jonas B Forum Pro • Posts: 14,596
5D malykh-GordonBGood histos here

As the findings for the *ist series of cameras were quite interesting I felt I needed to see what my 5D does. Just for the sake of it I post a link to a gallery with histograms from a series of test pictures taken with the 5D.
http://www.imageevent.com/jonas_b

The result shows that the 5D:
should be handled with care when using ISO 50,

  • increases headroom for ISO 100-1600 compared to what happens at the ISO 50 setting,

  • does some bit shifting at ISO 3200 (and you can possible just as well use ISO 1600 and compensate when developing the raw file,

  • never changes the amplification between the channels regardless of what setting is used for the White Balance

  • Furthermore: I have found that the RGB histogram, as expected, is read from data processed according to the WhiBal setting which forces the photographer to do some thinking when evaluating the RGB histogram and setting the exposure. This is something I guess is valid for most, if not all cameras providing a RGB histogram for the photographer.

Some testing is always at place, in other words. I'm about to learn to use the 5D and this gives me some insights I don't want to be without when taking real pictures.

And yes, this post will appear in the appropiate forum as well.

Thank you Malykh (whoever you are), vortout and GordonBGood,

Jonas

tapir Veteran Member • Posts: 3,860
Similar with XT...

I'll upload XT histos today or tomorrow but here're my findings:

Jonas B wrote:

  • never changes the amplification between the channels regardless

of what setting is used for the White Balance

Same here. AWB, WB preset or manual on XT have no bearing on RAW color channel data. That's I believe how it should be!

If K10D along with *istD also defeats the reason and purpose of RAW I'll be very concerned and dissapointed especially given poor JPEG showing. I'm shooting RAW almost exclusively. Partial reason is ability to adjust WB in a most undestructive manner.

  • Furthermore: I have found that the RGB histogram, as expected, is

read from data processed according to the WhiBal setting which
forces the photographer to do some thinking when evaluating the RGB
histogram and setting the exposure. This is something I guess is
valid for most, if not all cameras providing a RGB histogram for
the photographer.

Yes, that's why lack of RGB histo on XT is dangerous.

Thank you Malykh (whoever you are), vortout and GordonBGood,

Many thanks also.

-- hide signature --
GordonBGood Veteran Member • Posts: 6,308
Re: Similar with XT...

tapir wrote:

If K10D along with *istD also defeats the reason and purpose of RAW
I'll be very concerned and dissapointed especially given poor JPEG
showing. I'm shooting RAW almost exclusively. Partial reason is
ability to adjust WB in a most undestructive manner.

Alec, if you think about it and just want a fixed gain between channels for the D's and K110/100D, just shoot with AWB or with Sunny preset, which will give you fixed gain ratios that are more or less as one would get from Canon. However, if you are in an extreme WB situation, Pentax have given you the option of choosing a preset WB or Manual WB that more closely matches the scene WB, and correspondingly adjusts gain ratios to give you a little more clipping in some colour channels. This is an advantage, not a disadvantage.

Something on which Jonas did not comment is that, although the Canon 5D does not change relative gains between the RGB channels, it does appear to reduce overall gain for all three channels to increase clipping headroom in extreme lighting situation presets similar to what Pentax does as explained in the paragraph above. In fact, the 5D's method then loses tonality in the channels that aren't "pushed" more than with Pentax's method above.

From tests done already, it appears that the K10D does not have the same behaviour as earlier cameras and gain ratios between channels appear to be fixed for all WB settings.

Regards, GordonBGood

tapir Veteran Member • Posts: 3,860
Re: Similar with XT...

GordonBGood wrote:

tapir wrote:

If K10D along with *istD also defeats the reason and purpose of RAW
I'll be very concerned and dissapointed especially given poor JPEG
showing. I'm shooting RAW almost exclusively. Partial reason is
ability to adjust WB in a most undestructive manner.

Alec, if you think about it and just want a fixed gain between
channels for the D's and K110/100D, just shoot with AWB or with
Sunny preset, which will give you fixed gain ratios that are more
or less as one would get from Canon. However, if you are in an
extreme WB situation, Pentax have given you the option of choosing
a preset WB or Manual WB that more closely matches the scene WB,
and correspondingly adjusts gain ratios to give you a little more
clipping in some colour channels. This is an advantage, not a
disadvantage.

Gordon, what you say makes a lot of sense. But what happens if I didn't correctly assess the light situation and have chosen a wrong preset or wrong non-gray/white card? With Pentax approach I may wind up with incorrectly gained and clipped channel. Exactly the thing to avoid and reason to use RAW (provided it WB-blind in Canon sense) under complex light situation.

-- hide signature --
GordonBGood Veteran Member • Posts: 6,308
Re: Similar with XT...

As follows:

tapir wrote:

GordonBGood wrote:

tapir wrote:

If K10D along with *istD also defeats the reason and purpose of RAW
I'll be very concerned and dissapointed especially given poor JPEG
showing. I'm shooting RAW almost exclusively. Partial reason is
ability to adjust WB in a most undestructive manner.

Alec, if you think about it and just want a fixed gain between
channels for the D's and K110/100D, just shoot with AWB or with
Sunny preset, which will give you fixed gain ratios that are more
or less as one would get from Canon. However, if you are in an
extreme WB situation, Pentax have given you the option of choosing
a preset WB or Manual WB that more closely matches the scene WB,
and correspondingly adjusts gain ratios to give you a little more
clipping in some colour channels. This is an advantage, not a
disadvantage.

Gordon, what you say makes a lot of sense. But what happens if I
didn't correctly assess the light situation and have chosen a wrong
preset or wrong non-gray/white card? With Pentax approach I may
wind up with incorrectly gained and clipped channel. Exactly the
thing to avoid and reason to use RAW (provided it WB-blind in Canon
sense) under complex light situation.

Alec, if you aren't sure how to asses whether a scene is "somewhat bluish" as in shady/cloudy, or "somewhat warmish" as in incandescent/firelight/candlelight, then just use AWB or Sunny or Flash presets (as being in the middle) and be no worse than if using a camera (ie Canon) that doesn't do this relative gain tweak. As to using coloured WB cards with Manual WB or incorrect presets to trick the camera into extending the headroom in some channels, one would have to educate themselves on how to use it effectively for particular shooting situations. I've provided the tools in the Java DNG_Analyser or the PEF_Analyser to see for yourself what particular settings do; the risk of doing it is up to you.

Again, note that the K10D appears not to do relative channel gain adjustments as do the former models.

Regards, GordonBGood

Dark Mist Contributing Member • Posts: 558
Re: I don't believe it unless you tell me...

Gordon

Thanks for explaining that for me. As I inferred before I'm a user not a programmer but I find that logic always goes a long way.

Your explanation makes perfect sense now.

Keyboard shortcuts:
FForum MMy threads