Is the E-M5 sensor response nonlinear?

[Iliah Borg]

It probably ain't that simple, but on the other hand, isn't it almost impossible to get such an almost perfect match by pure coincidence?

The max values are off compared to several camera samples I profiled. So I do not see a match. Noise analysis is absent, RawDigger mode is not stated...

Don't think it's supposed to be the sensors max values

"Green channel (average of G and G2)
3615 1.27 (about 40k pixels clipped)"

Reading, at 3615 saturation was reached, and sensor maxed out.

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[Iliah Borg]

Both Iliah Borg and bobn2 additionally suggested that the sensor response might be nonlinearly encoded as has been the case on some recent Sony cameras.

http://forums.dpreview.com/forums/readflat.asp?forum=1041&message=41965851&changemode=1

??

In case you have already forgotten our discussion in the other thread, here's the beginning of it:

Iliah Borg: The "brightest" stop is plagued with noise, so for such a scene you may want to keep in "ETTR-1".

Anders W: On what grounds do you consider the "brightest" stop plagued with noise? Because there is a partial risk of clipping? If so, then you aren't really in the brightest stop but a bit beyond.

Iliah Borg: Sensors are not linear close to saturation, shot noise is dominating in the lighter parts of the image; modern raw compression schemes use much less levels to represent lighter parts of the image because the information there is already unreliable.

Best case, you do not see the difference between nonlinearly encoded and not linear by nature. That happens....

Sure. And so what?

Nothing. As in "nothing will come out of nothing"

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[Iliah Borg]

AndersW reported "Ave" statistics.

Exactly. Average values are close without white balance. Field is grossly non-uniform. Lens is known to the camera, and it is possible the firmware is running compensations. I do not know what to make out of it.

On a side note the Oly samples Imaging Resources present seem to saturate at 3791 too at ISO 200.

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[Detail Man]

AndersW reported "Ave" statistics.

Exactly. Average values are close without white balance. Field is grossly non-uniform.

Lens is known to the camera, and it is possible the firmware is running compensations.

That's interesting. Some possibility of E-M5 RAW-level "corrections" based on lens characteristics ?

Is that something that you are aware of occuring on other Olympus (or any other brand) camera ?

 

[Detail Man]

AndersW reported "Ave" statistics.

Exactly. Average values are close without white balance. Field is grossly non-uniform.

Lens is known to the camera, and it is possible the firmware is running compensations.

That's interesting. Some possibility of E-M5 RAW-level "corrections" based on lens characteristics?

Is that something that you are aware of occuring on other Olympus (or any other brand) camera?

Actually, I now recall that DPReview reported vignetting corrections (that Panasonic has some other fancy name for) in the GX1 review appeared to affect the RAW files recorded. Not clear whether the lens communicated specific information to the GX1 body regarding it's characteristics - but that would seem to be highly probable. The G3 is also reported to have this functionality ...

 

[Anders W]

AndersW reported "Ave" statistics.

Exactly. Average values are close without white balance.

Close to what?

Field is grossly non-uniform.

Could you please quantify this statement?

Lens is known to the camera, and it is possible the firmware is running compensations.

Compensations for what? The nonlinearity of sensor response which is what I tried to test? If so, please demonstrate how a lens with minimal vignetting (that's why I chose the 45) well stopped down and used at the same aperture for all tests shots (f/5.6) could conceivably influence the outcome and in what way the camera could compensate so as to prevent us from seeing that fact.

I do not know what to make out of it.

That's your problem, not mine. The burden of evidence is now on your side. If you think you can do the test better than I did and, importantly, can show that doing it better gives significantly different results than mine, please do that rather than keep nagging.

On a side note the Oly samples Imaging Resources present seem to saturate at 3791 too at ISO 200.

So what?

 

[Timur Born]

"Max" stats appear to increase with ISO

Doesn't that just mean that stronger analog gain leads to increased noise in the gain circuitry and thus to higher maximum values after gain (sum of signal + added noise)?

By the way, at ISO 200 my E-M5 maxes out at 3791 on all channels.

 

[Detail Man]

"Max" stats appear to increase with ISO

Doesn't that just mean that stronger analog gain leads to increased noise in the gain circuitry and thus to higher maximum values after gain (sum of signal + added noise)?

Probably not likely. The (input-referred) SNR of typical analog amplification circuitry is typically on the order of 80 dB (according to our friend bobn2 ). That is around 13.39 EV below maximum ...

One (minor) factor is the black-level that the E-M5 embeds in the image-file meta-data.

Here is texinwien 's data reported for his E-M5:

Also, I do have Black Level set to Auto in RawDigger. Levels displayed for each ISO:

200 253
400 253
800 254
1600 252
3200 249
6400 244
12800 232
25600 209

From: http://forums.dpreview.com/forums/readflat.asp?forum=1041&thread=41814418

By the way, at ISO 200 my E-M5 maxes out at 3791 on all channels.

Sounds like a familiar number. Seems to be more common than Gakuranman 's measured 3811

 

[Anders W]

"Max" stats appear to increase with ISO

Doesn't that just mean that stronger analog gain leads to increased noise in the gain circuitry and thus to higher maximum values after gain (sum of signal + added noise)?

Something like that perhaps. But there's of course a conscious decision of what max value to allow involved. To my mind, it makes some sense to expand that value with higher ISOs to allow for increased noise around the true value. If you didn't do that, you'd have more and more clipping at the top end, artificially lowering the average for the extreme highlights.

By the way, at ISO 200 my E-M5 maxes out at 3791 on all channels.

I guess you use RawDigger to establish that. What's the black level set to and what firmware do you have on the camera?

As already indicated, I get 3815 on all four channels with the black level at 254 and with firmware 1.1 on the camera.

 

[texinwien]

As already indicated, I get 3815 on all four channels with the black level at 254 and with firmware 1.1 on the camera.

I get 3815, 3815, 3815, 3816 with black level at 254 and firmware 1.1 at ISO200.

In an earlier test, I saw values of 3822 / 3821 in all 4 channels, but that was much earlier, and I don't recall which version of firmware I was running. I also did not check the black level at that time.

--
http://www.flickr.com/photos/kurtmilam/

 

[Timur Born]

Probably not likely. The (input-referred) SNR of typical analog amplification circuitry is typically on the order of 80 dB (according to our friend bobn2 ). That is around 13.39 EV below maximum ...

Even at ISO 25600 where analog gain is really pushed to its limits?

One (minor) factor is the black-level that the E-M5 embeds in the image-file meta-data.

That's a difference of 44 levels between ISO 200 and 25600 only, though, while the maximum clipped raw levels are 132 levels apart in the measurements listed earlier.

 

[Timur Born]

Something like that perhaps. But there's of course a conscious decision of what max value to allow involved. To my mind, it makes some sense to expand that value with higher ISOs to allow for increased noise around the true value. If you didn't do that, you'd have more and more clipping at the top end, artificially lowering the average for the extreme highlights.

Just to make sure I get understand this all correctly: Where does that higher in value noise come from? Are single photo-sites averaged in their value over the exposure time period before a final value is written into the raw data?

If so then your explanation makes sense in that a shorter exposure time span could produce higher averages and thus needs additional headroom at higher ISO.

I guess you use RawDigger to establish that. What's the black level set to and what firmware do you have on the camera?

I used auto levels in RawDigger, reported black level is 254 at ISO 200.

Firmware is still on v1.0 because the camera fails to update on my Windows installation and I did not bother to try via OS X or a different computer again (and want to keep it in that conditions for Olympus support to take a look at).

 

[Iliah Borg]

AndersW reported "Ave" statistics.

Exactly. Average values are close without white balance. Field is grossly non-uniform.

Lens is known to the camera, and it is possible the firmware is running compensations.

That's interesting. Some possibility of E-M5 RAW-level "corrections" based on lens characteristics ?

Is that something that you are aware of occuring on other Olympus (or any other brand) camera ?

There is always a possibility that, say, digital vignetting is compensated before raw is recorded. Lots of this stuff happen with some brands, yes, I ran into it. I think it is always good to check for it.

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[Anders W]

Something like that perhaps. But there's of course a conscious decision of what max value to allow involved. To my mind, it makes some sense to expand that value with higher ISOs to allow for increased noise around the true value. If you didn't do that, you'd have more and more clipping at the top end, artificially lowering the average for the extreme highlights.

Just to make sure I get understand this all correctly: Where does that higher in value noise come from? Are single photo-sites averaged in their value over the exposure time period before a final value is written into the raw data?

No they are (ordinarily) not averaged over exposure time but read when the exposure is finished. What I mean is the following: Suppose the true value of a pixel is 3800. Due to noise, however, that number is just the average you would get across multiple exposures. The noise level is higher at higher ISOs. If we'd cut the noise at the same value as the true average (3800), we'd on average record lower values than 3800 for that pixel. To avoid that, it seems advisable to allow some "noise headroom" and to allow a bit more of it at higher ISOs than at lower.

If so then your explanation makes sense in that a shorter exposure time span could produce higher averages and thus needs additional headroom at higher ISO.

I guess you use RawDigger to establish that. What's the black level set to and what firmware do you have on the camera?

I used auto levels in RawDigger, reported black level is 254 at ISO 200.

Firmware is still on v1.0 because the camera fails to update on my Windows installation and I did not bother to try via OS X or a different computer again (and want to keep it in that conditions for Olympus support to take a look at).

OK. Then it might be that the firmware accounts for the difference in your case.

 

[Anders W]

It probably ain't that simple, but on the other hand, isn't it almost impossible to get such an almost perfect match by pure coincidence?

The max values are off compared to several camera samples I profiled. So I do not see a match. Noise analysis is absent, RawDigger mode is not stated...

Don't think it's supposed to be the sensors max values

"Green channel (average of G and G2)
3615 1.27 (about 40k pixels clipped)"

Reading, at 3615 saturation was reached, and sensor maxed out.

As usual, you got it wrong and everyone else got it right. As already stated, 3815 is the clipping point. 3615 is the average level recorded when about 40 k pixels per channel (about one percent) have reached the clipping point. The standard deviation of the distribution surrounding the average of 3615 is about 110. Got it?

 

[Anders W]

It probably ain't that simple, but on the other hand, isn't it almost impossible to get such an almost perfect match by pure coincidence?

The max values are off compared to several camera samples I profiled. So I do not see a match. Noise analysis is absent, RawDigger mode is not stated...

Don't think it's supposed to be the sensors max values

"Green channel (average of G and G2)
3615 1.27 (about 40k pixels clipped)"

Reading, at 3615 saturation was reached, and sensor maxed out.

As usual, you got it wrong and everyone else got it right. As already stated, 3815 is the clipping point. 3615 is the average level recorded when about 40 k pixels per channel (about one percent) have reached the clipping point. The standard deviation of the distribution surrounding the average of 3615 is about 110. Got it?

One might that that for a roughly normal distribution (as we have in this case) the average and standard deviation combined imply that more than 95 percent of the pixels are in the range between 3415 and 3815.

 

[bobn2]

Probably not likely. The (input-referred) SNR of typical analog amplification circuitry is typically on the order of 80 dB (according to our friend bobn2 ). That is around 13.39 EV below maximum ...

Even at ISO 25600 where analog gain is really pushed to its limits?

The SNR of a negative feedback controlled amplifier is relatively constant, being in the main determined by the noise in the first stage, before the feedback loop (which corrects noise just as it corrects distortion). The open loop voltage gain of an operational amplifier might be 100dB or more. The gain required going from 200 to 25600 is 42dB, leaving about 60dB of feedback to control the noise. Essentially, that will be a constant SNR. That is why the commonly held idea that 'ISO noise' is due to noise from the gain amplifiers is a myth.

--
Bob

 

[Iliah Borg]

It probably ain't that simple, but on the other hand, isn't it almost impossible to get such an almost perfect match by pure coincidence?

The max values are off compared to several camera samples I profiled. So I do not see a match. Noise analysis is absent, RawDigger mode is not stated...

Don't think it's supposed to be the sensors max values

"Green channel (average of G and G2)
3615 1.27 (about 40k pixels clipped)"

Reading, at 3615 saturation was reached, and sensor maxed out.

As usual, you got it wrong

ROTFLMAO

You do not know what you are talking about.

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[bobn2]

On course, the saturation level that you see depends on the the raw file full scale,

Which, for the cameras I profiled, was 3791 (black level subtracted) uniformly between all 4 channels.

What is unknown is Olympus use of the programmable variable gain amplifier in each columns (which does not necessarily go directly with ISO) and, if they use the 14 bit ADC, how they are mapping that to a 12 bit raw file, whether it is simple truncation (top, and/or bottom) or something more subtle.

Sometimes it is good to check that SNR at exp1 to SNR at exp2 is along with the linear model.

On a side note, Nikon D4 has a strong non-linearity at the highlight portion of the characteristic curve which is compensated by calibration of sensor assemblies. When this calibration is not done properly the camera is not very easy to use at daylight - one needs to underexpose 2 stops to get decent output.

Interesting information, Iliah. Calibration and correction by digital mapping is very, very common in the sensing world. Essentially, it allows use of cheap, not very linear sensors as precision instruments, and there are even machine learning based self calibration systems available. However, in production terms it is an expensive process. In the context of the D4, I'd guess its' not a case of using cheap sensors, but enabling the pushing of the sensor to its limits in pursuit of the highest performance. Explains in part the cost of the camera, though.

--
Bob

 

[Iliah Borg]

Calibration and correction by digital mapping is very, very common in the sensing world.

But of course. I was very much surprised by the effect of it, however, as I got my hands on files from 2 D4 cameras that were out of calibration initially. The files before and after differ like day and night, and even at low ISO with a test chart that encompasses only 7 stops of DR.

--
http://www.libraw.org/
http://www.RawDigger.com/