Why is high value lower when overexposed

Today I took my 5DS and took some test shots of a cloud to test out spot metering, a feature I'd not used before. I figured I'd aim the spot at the brightest portion of a cloud and adjust exposure until it showed +2 in the camera's meter. That was my starting point (in retrospect I should have used +1). From there, I took additional shots, increasing the exposure by 1/3 stop for each following shot.

Next, I opened the raw files in RawDigger, expecting the first couple of frames to not be blown, but the rest to be blown in increasingly larger portions of the frame.

What I actually found was this:

First frame (+2 EC): max value was 12686

+2 1/3: 12919

+2 2/3: 12989

+3: 12898

+3 1/3: 12939

+6ish: 12686

Additionally, I used the selection tool of raw digger to select areas of the frame and found that the +6ish frame to be exactly 12686 throughout areas that were overexposed.

So it seems like 12686 is the good max value. But then why am I seeing higher values when most of the frame is not clipped?

P. S. I'm inclined not to use the spot metering feature because the spot size is too large for how I was planning to use it.

--
Victor Engel

Victor Engel said:

Today I took my 5DS and took some test shots of a cloud to test out spot metering, a feature I'd not used before. I figured I'd aim the spot at the brightest portion of a cloud and adjust exposure until it showed +2 in the camera's meter. That was my starting point (in retrospect I should have used +1).

Click to expand...

Whitest clouds are +3 EV if spot-mertered.

Victor Engel said:

From there, I took additional shots, increasing the exposure by 1/3 stop for each following shot.

Next, I opened the raw files in RawDigger, expecting the first couple of frames to not be blown, but the rest to be blown in increasingly larger portions of the frame.

What I actually found was this:

First frame (+2 EC): max value was 12686

Click to expand...

What was the ISO setting on the camera? Was it a fractional one? What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Victor Engel said:

But then why am I seeing higher values when most of the frame is not clipped?

Click to expand...

It is the effect that is showing up in the case of strong overexposure of the image sensor, rather common with Canon cameras. When a pixel is overexposed the photodiode can no longer store all generated charges, and a "leak" occurs. When it starts, it may result in too much "leak", negative characteristic curve slope, a bit similar to Sabatier effect (solarization). "Leaks" also may result in larger than normal cross-talk.

Check how the ratio of red to green is progressing.

It is always easier to discuss those things when raw files are available for analysis.

--
http://www.libraw.org/

Iliah Borg said:

Iliah Borg said:

First frame (+2 EC): max value was 12686

Click to expand...

What was the ISO setting on the camera? Was it a fractional one? What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Click to expand...

He is not reporting the raw data numbers, black level about 2048 is subtracted. 12686+2048=14734 looks normal.

Iliah Borg said:

It is always easier to discuss those things when raw files are available for analysis.

Click to expand...

+1 to that.

Iliah Borg said:

Victor Engel said:

Today I took my 5DS and took some test shots of a cloud to test out spot metering, a feature I'd not used before. I figured I'd aim the spot at the brightest portion of a cloud and adjust exposure until it showed +2 in the camera's meter. That was my starting point (in retrospect I should have used +1).

Click to expand...

Whitest clouds are +3 EV if spot-mertered.

Click to expand...

That's why I thought I was OK starting at +2. But the spot was too big to include only the brightest part of the cloud, so that's probably why it clipped before +3.

Iliah Borg said:

Iliah Borg said:

From there, I took additional shots, increasing the exposure by 1/3 stop for each following shot.

Next, I opened the raw files in RawDigger, expecting the first couple of frames to not be blown, but the rest to be blown in increasingly larger portions of the frame.

What I actually found was this:

First frame (+2 EC): max value was 12686

Click to expand...

What was the ISO setting on the camera?

Click to expand...

ISO 100.

Iliah Borg said:

Was it a fractional one?

Click to expand...

ISO 100. I changed exposure, not ISO.

Iliah Borg said:

What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Click to expand...

Where do I find that?

Iliah Borg said:

Iliah Borg said:

But then why am I seeing higher values when most of the frame is not clipped?

Click to expand...

It is the effect that is showing up in the case of strong overexposure of the image sensor, rather common with Canon cameras. When a pixel is overexposed the photodiode can no longer store all generated charges, and a "leak" occurs. When it starts, it may result in too much "leak", negative characteristic curve slope, a bit similar to Sabatier effect (solarization). "Leaks" also may result in larger than normal cross-talk.

Click to expand...

I thought that it might be something like that - and if a large area is clipped, they all flood each other so they even out?

Iliah Borg said:

Check how the ratio of red to green is progressing.

Click to expand...

On the first exposure with clipping, red and blue don't clip. Only green clips.

Iliah Borg said:

It is always easier to discuss those things when raw files are available for analysis.

Click to expand...

Sure. I can post them in a bit.

Xasan said:

Xasan said:

Xasan said:

First frame (+2 EC): max value was 12686

Click to expand...

What was the ISO setting on the camera? Was it a fractional one? What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Click to expand...

He is not reporting the raw data numbers, black level about 2048 is subtracted. 12686+2048=14734 looks normal.

Xasan said:

It is always easier to discuss those things when raw files are available for analysis.

Click to expand...

+1 to that.

Click to expand...

True. Min values are also two digits, so obviously not raw values. Auto black level says 2047. I'm new to RawDigger, so I may simply not know where the appropriate settings are.

Victor Engel said:

Victor Engel said:

What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Click to expand...

Where do I find that?

Click to expand...

You can use exiftool, or open exif window in RD and scroll down or filter by the word 'level'

Raw files are here. I'll take them down in a couple days, because this just triggered the almost out of space message.

Xasan said:

Xasan said:

Xasan said:

What is recorded in tags 0x030e NormalWhiteLevel and 0x030f SpecularWhiteLevel?

For ISO 100 they should read something like 13954 / 14466

Click to expand...

Where do I find that?

Click to expand...

You can use exiftool, or open exif window in RD and scroll down or filter by the word 'level'

Click to expand...

Oh, OK. I thought I was supposed to find something in RawDigger.

Iliah Borg said:

What was the ISO setting on the camera? Was it a fractional one? What is recorded in tags

0x030e

Click to expand...

13,954

Iliah Borg said:

NormalWhiteLevel and

0x030f

Click to expand...

14,466

Iliah Borg said:

SpecularWhiteLevel?

Click to expand...

Victor Engel said:

Iliah Borg said:

What was the ISO setting on the camera? Was it a fractional one? What is recorded in tags

0x030e

Click to expand...

13,954

Victor Engel said:

NormalWhiteLevel and

0x030f

Click to expand...

14,466

Victor Engel said:

SpecularWhiteLevel?

Click to expand...

Click to expand...

Same as mine. Canon sets those a bit conservative, specular white level (raw data numbers, black level not subtracted) indicates guaranteed linearity margin (not more than 5% deviation from the linear behavior of the response curve).

RawDigger Preferences for this session:













Raw composite mode + OE indication (black level subtraction is off, from Preferences, second screenshot, this setting is also available if you press black level button on the lower bar, left corner):



Selection over the OE-highlighted area (press shift, hold it, press the left mouse button, drag, release when done):



Right-click inside the selection, from the menu choose Show histogram (mind the options I used on the histogram, including setting the higher limit, to minimize jumps):



As you can see the histogram "hits the wall", indicating highlight clipping.

Approximate number of clipped pixels is indicated on the main screen, to know the exact number you can save histogram as CSV and look at the last line, containing the senior bin, it reads (depending on the selection, of course - you will have different numbers, but the tendency is the same)

"14733","14733","2.8468","2.8468","0","6987","0","7177"

that is, G1 channel contains 6987 blown-out pixels, G2 - 7177.

Dragging the next file into RD window or onto the icon, to preserve the selection in place, you can verify that the real maximum stays in place, only noise happens past 14733.

The most exposed file reads for the bin 14733

"14733","14733","2.8468","2.8468","7997","49996","50284","50111"

but now the noise is rather high, and you will also see

"14801","14801","2.8534","2.8534","0","0","0","1" and some other 1s in other bins.

That is, for the practical purposes the maximum stays at 14,733 for this ISO (black level not subtracted), which corresponds to Canon's 14,466 containing some safety margin.

--

Relevant to my original question, I did not include the most exposed one in the shared folder. I've added it now (file number ends 66).

Victor Engel said:

Next, I opened the raw files in RawDigger, expecting the first couple of frames to not be blown, but the rest to be blown in increasingly larger portions of the frame.

What I actually found was this:

First frame (+2 EC): max value was 12686

+2 1/3: 12919

+2 2/3: 12989

+3: 12898

+3 1/3: 12939

+6ish: 12686

Additionally, I used the selection tool of raw digger to select areas of the frame and found that the +6ish frame to be exactly 12686 throughout areas that were overexposed.

So it seems like 12686 is the good max value. But then why am I seeing higher values when most of the frame is not clipped?

--
Victor Engel

Click to expand...

I have been fooled in the past by RawDigger's results. For example, a Sigma camera that saturates at around 8,000 can show fliers of over 10,000 in RawDigger. As to why those are not 12bit numbers, that's just the Sigma/Foveon system which is irrelevant here.

Main point is that RawDigger shows the max value literally, even if your capture has only one occurrence of that value and even if that one value is significantly higher than all others. I suggest that you set the histogram to EV levels and logarithmic counts.

From the histogram shape, it is easier to gauge the sensor exposure - especially for a cloud shot and I expect that you would be able get more logical numbers than the above list.

These comments are general because I am not a Canonista.

--
"What we've got hyah is Failyah to Communicate": 'Cool Hand Luke' 1967.

One of your screens is different from mine. What version are you using?











--
Victor Engel

Iliah Borg said:

Same as mine. Canon sets those a bit conservative, specular white level (raw data numbers, black level not subtracted) indicates guaranteed linearity margin (not more than 5% deviation from the linear behavior of the response curve).

Click to expand...

Out of interest, is it common for manufacturers to include non-linear response within range captured by the ADC?

Or to leave this safety margin (which presumably ends up being, what, something like 1/6EV of recoverability)?

Richard - dpreview.com

Richard Butler said:

is it common for manufacturers to include non-linear response within range captured by the ADC?

Click to expand...

Those values may enter the raw data in highlights, but are mostly ignored by the in-camera and recommended raw converters. Some Panasonic cameras at the lowest ISO setting have their linear maximums much lower than the maximums in raw:

DMC-FZ1000 iso 80 3277

AG-GH4 iso 100 2111

DMC-G7 iso 100 2111

DMC-GF7 iso 100 2111

DMC-GH4 iso 100 2111

DMC-GM5 iso 100 2111

DMC-GX8 iso 100 2111

DMC-GX85 iso 100 2111 (same as Lumix DMC-GX80 and Lumix DMC-GX7 Mark II)

DMC-LX100 iso 100 2111

DMC-GH3 iso 125 2626

DMC-GM1 iso 125 2626

DMC-GX7 iso 125 2626

Shadows are always non-linear to some extent because of flare and glare.

The worst case scenario (following CIPA DC-004, "Sensitivity of digital cameras") is that the linear portion is guaranteed for the scenes that can be captured within the range of 98% and 3.3% (the reference point is 18%, obviously); that is within +2.5EV .. -5EV from the metering point, very close to Kodak recommendations based on the studies made in '60s. That's a call for compliance with the exposure control system.

Richard Butler said:

Or to leave this safety margin (which presumably ends up being, what, something like 1/6EV of recoverability)?

Click to expand...

Some cameras have the recommended clipping point included in makernotes, some have it implicit. Ignoring those (explicit or implicit, while "explicit" often means non-documented, as makernotes format is not openly published) recommendations may result not only in the loss of linearity in highlights, but also in "white turning purple" (purple highlights, purple clouds) phenomenon (but that is not the only possible reason for this effect, the other being improper clipping of highlights in conversion after applying white balance).

Victor Engel said:

One of your screens is different from mine. What version are you using?

Click to expand...

Screenshots were made in build 538, today we have build 539.

Changes

• Camera support added:
  • Canon 200D, 6D Mark II
  • Fujifilm X-A10
  • Gitup GIT2P
  • Leica TL2
  • Nikon D7500
• CGATS export:
  • LinearMax=val added to DESCRIPTOR field if camera provides this data (note: black level is not subtracted)
  • WBAsShot=a,b,c,d added to DESCRIPTOR field if camera provides as-shot white balance
  • New checkbox 'Add DESCRIPTOR "DCP Profile"' in Samples export dialog. Will add this text string to CGATS output (for use with BasicColor Input profiler to create DNG profiles).
• New setting to Preferences - Histogram, 'Set linear histogram auto range'.
  • If one sets it to 'Camera data range', histogram opens (or is recalculated for a new file) with 0...camera max limits.
  • If set to 'Current frame data', linear scale range will reflect current frame min/max values
• Exiftool updated to 10.60
• Bugs fixed
  • image does not move on next/prev file.
  • OS X: fixed freeze if key is expired and RawDigger was run by dragging image on program icon