Is the E-M5 sensor response nonlinear?

[Anders W]

On a side note, fools are not qualified to make judgements on the foolishness of others.

Correct. But some fools are not capable of fully appreciating their own foolishness, which presumably explains why you keep making this mistake.

Of course I do not expect you to understand that, but maybe you can at least practice that simple rule?

Unlike you, I accept the conclusion only when the premise is valid.

 

[bobn2]

On a side note, fools are not qualified to make judgements on the foolishness of others.

Correct. But some fools are not capable of fully appreciating their own foolishness, which presumably explains why you keep making this mistake.

Of course I do not expect you to understand that, but maybe you can at least practice that simple rule?

Unlike you, I accept the conclusion only when the premise is valid.

Anders, you are generally a sensible fellow. Just don't let the adrenalin of 'entertainment' lead you into making a fool of yourself. You won't beat Iliah in one of these exchanges even in one of the rare cases where he is wrong - let alone in this case, when he is right.

I suggest you take a deep breath, forget the entertainment for a while, and try to reason out what he's saying.
Been there, got the T-shirt.
--
Bob

 

[Anders W]

On a side note, fools are not qualified to make judgements on the foolishness of others.

Correct. But some fools are not capable of fully appreciating their own foolishness, which presumably explains why you keep making this mistake.

Of course I do not expect you to understand that, but maybe you can at least practice that simple rule?

Unlike you, I accept the conclusion only when the premise is valid.

Anders, you are generally a sensible fellow. Just don't let the adrenalin of 'entertainment' lead you into making a fool of yourself.

I am actually moderately entertained by the "exchange" (if you can call it that) with Iliah. And in precisely what sense do you think I am making a fool of myself?

You won't beat Iliah in one of these exchanges even in one of the rare cases where he is wrong - let alone in this case, when he is right.

In precisely what way do you think he is right?

Iliah made a claim which I proved wrong. I have invited him to perform a better experiment and a better analysis that proves me wrong. He hasn't been able or willing to do that. End of story.

I suggest you take a deep breath, forget the entertainment for a while, and try to reason out what he's saying.

If he has something to say beyond semi-comprehensible one-liners, I'd be happy to listen. I have never found reason to place much confidence in people who are unable or unwilling to develop their thought in a paragraph or two of ordinary text meant to convince the other party rather than FUD him/her into silence by vague references to this or that technicality without even bothering to spell out their relevance.

 

[Timur Born]

If he has something to say beyond semi-comprehensible one-liners, I'd be happy to listen. I have never found reason to place much confidence in people who are unable or unwilling to develop their thought in a paragraph or two of ordinary text meant to convince the other party rather than FUD him/her into silence by vague references to this or that technicality without even bothering to spell out their relevance.

I agree on this and would really like to see/read some comprehensive explanation or demonstration of how the upper stop of E-M5 raw files behaves non-linear and what to look out for when working with it.

That doesn't mean that I am not very open to the possibility of non-linear parts in the raw data, but for clarification and education there has to be more than just a "it's like that because I measured it without publishing results". Or can anyone please provide a link where these results are available?

 

[Iliah Borg]

And in precisely what sense do you think I am making a fool of myself?

Not presenting the data but claiming you presented it; that is having no understanding what an experiment is and what the data is..
Misinterpreting trivial terms.
Not knowing all sensors are non-linear, to one extent or to another.

And the most interesting part - you are making funny noises ("Both Iliah Borg and bobn2 additionally suggested that the sensor response might be nonlinearly encoded" - while I clearly said "The output to raw from E-M5 is rather linear" - http://forums.dpreview.com/forums/read.asp?forum=1041&message=41972271 )

Iliah made a claim which I proved wrong.

No, you just proved you have no understanding of the issue at hand and that you can't learn.

Vast majority are shooting JPEGs. Your claim on the reasons Olympus cameras have certain metering (you said "The advantage of the approach chosen by Olympus is that photographers who shoot RAW run less risk of clipping the highlights by mistake") is hilarious. All cameramakers prefer to underexpose and clip the last stop in processing, losing about 0.7 to 0.5 stops of dynamic range. You refuse to understand why. Adobe are having silent exposure compensation, for the same amount. You refuse to understand why neither it is so, nor why they are not documenting this clearly.

But you are in an excellent company

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[bobn2]

Interesting, since it would make it possible to have more than 12 stops of pixel-level ('screen') DR, even though the raw files are 'just' 12-bit. Don't know how to check that (nonlinear encoding).

You can in any case. 12 bits can encode more than 12 stops of DR, it just can't do it completely. Remember noise is a statistical variation, it is not something that is measured in a single pixel. So, it's not the case that, say 0.25 bits of noise will be registered as 1 bit. If you average a lot of bits in which there is 0.25 bits of noise, you'll find that the average value is 0.25, composed of three times as many '0' bits as '1' bits, though obviously randomly arranged. However, you can't encode fully the brightness ranges that should be available between '1' and '0.25' in a pixel. You can in large areas, where the noise dithers the average to give a smooth(ish) tone showing less than 1 bit's worth of brightness.

Isn't that the same as saying that DR increases when downsampling an image?

No

--
Bob

 

[Detail Man]

Please read what I say above.

...

We are talking about NOISE, not maximum SIGNAL.

Yes, but I was talking about signal. I just was curious where the amplifier might be working within its own limits. When you have a (possible) gain range of 80 dB ...

I said that the SNR was on the order of 80 dB. The phrase "gain range" is used to denote a variable-gain device or system. If you are talking about absolute signal levels, then say it that way.

... and only need to amplify a signal range of 40 dB then it makes sense not to use an analog amplifier too close to its own ceiling.

I said that the SNR was on the order of 40 dB. SNR is a ratio of signal to noise, and does not address issues of absolute signal levels themselves. I find it to be a lot easier to answer your questions when you just ask questions - as opposed to making engineering recommendations.

That is about maximum SIGNAL. Linear is what is desired. It is not like a "tube power amplifier".

Surely not a tube power amplifier, but an analog amplifier nonetheless, which usually means that it may behave non-linear towards its upper limit. This is why I was curious how "hot" these amplifiers are run while they have so much room for playing it save.

That is the subject of this thread (linearity of measurement near the upper ranges of signal-level). Nobody on this thread has (in seriousness) suggested that system non-linearity is a good idea ..

 

[bobn2]

On a side note, fools are not qualified to make judgements on the foolishness of others.

Correct. But some fools are not capable of fully appreciating their own foolishness, which presumably explains why you keep making this mistake.

Of course I do not expect you to understand that, but maybe you can at least practice that simple rule?

Unlike you, I accept the conclusion only when the premise is valid.

Anders, you are generally a sensible fellow. Just don't let the adrenalin of 'entertainment' lead you into making a fool of yourself.

I am actually moderately entertained by the "exchange" (if you can call it that) with Iliah.

If entertainment is what you are after, there are more productive sources. With Iliah you end up with the bruises to show for it.

And in precisely what sense do you think I am making a fool of myself?

By engaging in 'entertainment' when there is knowledge to be gained. It's easy to do. You're convinced you're right, haven't quite got the supporting evidence or argument, come across someone who disagrees in a supercilious way that dents your ego a bit, and you end up taking wrong turnings in your riposte. Done it myself, many times. Here's a recent example:
http://forums.dpreview.com/forums/read.asp?forum=1018&message=41982258

I was right in this case, but went down a great big blind alley before I took the trouble to sit down, work it out and get to the heart of why he was wrong.

You won't beat Iliah in one of these exchanges even in one of the rare cases where he is wrong - let alone in this case, when he is right.

In precisely what way do you think he is right?

He is right about non-linearity of the sensor (including the pixel SF) close to saturation. He's right that raw 'saturation' is commonly well short of pixel 'saturation'. He's also very probably right (though I don't have any information of my own) that manufacturers are using non-linear coding to re-linearise sensor response (he gave the example of the D4). I can't remember that there were any other substantive issues, though once you get into 'entertainment' those issues start to get lost.

Iliah made a claim which I proved wrong.

Just backtracked through the thread. Can't see it. Once you get into 'entertainment' self perception of proof starts to get lost. Proof gives way to 'that showed him'.

I have invited him to perform a better experiment and a better analysis that proves me wrong. He hasn't been able or willing to do that. End of story.

And yet, he is right, and you know that he does have the experiment and analysis to back up what he says. He's just going to make you get there for yourself. Frustrating, isn't it?

I suggest you take a deep breath, forget the entertainment for a while, and try to reason out what he's saying.

If he has something to say beyond semi-comprehensible one-liners, I'd be happy to listen.

Iliah's pedagogic style. Doesn't lecture, just drops you cryptic one liners that you need to follow and dig under to do the learning for yourself. Since my pedagogic style is more of the lecturing variety, and that upsets people too, I wouldn't be one to say which causes more resentment. But the underlying point is the same, people who resent being taught don't learn.

I have never found reason to place much confidence in people who are unable or unwilling to develop their thought in a paragraph or two of ordinary text meant to convince the other party rather than FUD him/her into silence by vague references to this or that technicality without even bothering to spell out their relevance.

And generally that test will separate out the BS merchants from the real deal, but in this case, it hasn't.
--
Bob

 

[Timur Born]

An amplifier running with 40dB of negative feedback is linear right up until where it clips. That's the difference between 'transistor sound' and 'tube sound'.

I will have to (and do) take your word on that they are implemented and work that way in practice.

One question: are bandwidth limitations of the amplifier - and thus feedback loop - a problem for linearity of high frequency content in images?

One remark: AFAIK CDs are usually mastered at -0.3 to -0.5 dB in order to stay somewhat below the clipping point of DA converters in order to protect from non-linearities (distortion) that happens with many (inexpensive) converter/gain circuits in common CD players.

Frankly I never cared how these things are implemented on their lowest level, but now I wonder (and should maybe ask my contacts at RME Audio) how amplifier circuits inside and around are build up (closed vs. open loop). But that's pure curiosity and really not the topic of this thread.

 

[Detail Man]

An amplifier running with 40dB of negative feedback is linear right up until where it clips. That's the difference between 'transistor sound' and 'tube sound'.

I will have to (and do) take your word on that they are implemented and work that way in practice.

One question: are bandwidth limitations of the amplifier - and thus feedback loop - a problem for linearity of high frequency content in images?

The amplifiers that Bob is talking about relay the image-data from individual photo-sites to an A/D converter. Their own bandwidth limitations would affect all photo-sites in the same way.

In images, "spatial-frequency" relates to "cycles per image dimension" (the height or width of the image-frame, or sometimes a subdivision therein), so it is a different situation than the "cycles per second" (of a sound-pressure-level wave, or of some recorded or electronic signal) of audio-frequency signals. The latter is in the "time domain" (or the "frequency domain", describing time-domain signals), whereas the former is in the "spatial domain" (or the "spatial-frequency domain", describing spatial domain signals).

For example, the pattern of alternating lines and spaces used to test (vertical or horizontal) resolution of imaging systems is a "square wave" (with a 50% duty-cycle) in the spatial-domain. The units of the spatial-domain can be in distance itself (i.e., L/mm "lines per milliMeter", or LP/mm "line-pairs per milliMeter), or expressed with reference to an image-frame itself (i.e., L/H "lines per height", or L/W "lines per width", or LP/H "line-pairs per height", or LP/W "line-pairs per width".

The notations can be confusing. (As I understand it), "LPH" normally implies "lines per height" (and LPW implies "lines per width"), which are actually describing alternating lines and spaces. Thus there are only 1/2 as many actual "line-pairs" as their are (commonly called) "lines". The same is true for "LPM", which signifies "lines" (one-half of a "line-pair") per milliMeter .

One remark: AFAIK CDs are usually mastered at -0.3 to -0.5 dB in order to stay somewhat below the clipping point of DA converters in order to protect from non-linearities (distortion) that happens with many (inexpensive) converter/gain circuits in common CD players.

It is not the Digital to Analog converters that "get it wrong". When the DAC output is "AC-coupled" (using capacitive-coupling, as is pretty much always the case) in an audio signal-path, the square-wave response of a passive (resistor-capacitor) coupling network can (in the worst case) exceed the peak value of the square-wave by a full 6 dB (a linear factor of two). Thus, a full 6 dB of "headroom" would need to be left (relative to the maximum signal level that can be reproduced by any amplifier in the subsequent signal-path). As you know, such is rarely the case.

Frankly I never cared how these things are implemented on their lowest level, but now I wonder (and should maybe ask my contacts at RME Audio) how amplifier circuits inside and around are build up (closed vs. open loop). But that's pure curiosity and really not the topic of this thread.

Closed loop negative-feedback amplifiers have been widely used since the era of vacuum tube amplifiers. Transistors and FETs and operational amplifiers made up of such discrete semiconductors are capable of providing much higher open-loop voltage/current gains than vacuum tube configurations. When a feedback-loop is closed around that high-gain amplification, the net result is a much higher "loop-gain" (which is the "margin" of extra gain left after the open-loop-gain is divided by the closed-loop-gain).

When calculating the above types of "gains" using (logarithmic) deciBels, the calculations become sums and differences (which represent linear multiplications and divisions). That is the more common way that such things are described. 6 dB is (very close to) a linear factor of two, and is also the same as the weighting of a digital binary (M.S. Digit) "bit" in the value of a binary number.

 

[Timur Born]

It is not the Digital to Analog converters that "get it wrong".

From what I understood it is the DAC. Here one example: "Also, make sure you set any Out Ceiling control to a little under 0dBFS, to cope with the D-A converters on some CD players, which react badly to signals that peak at 0dBFS. I use a -0.3dB setting, but some mastering engineers drop as low as -3dB."

Of course there is some confusion around whether this ceiling is left for protecting against intersample-peaks or because of just running converters too hot at their limit. I guess it's mostly the first.

When the DAC output is "AC-coupled" (using capacitive-coupling, as is pretty much always the case) in an audio signal-path, the square-wave response of a passive (resistor-capacitor) coupling network can (in the worst case) exceed the peak value of the square-wave by a full 6 dB (a linear factor of two). Thus, a full 6 dB of "headroom" would need to be left (relative to the maximum signal level that can be reproduced by any amplifier in the subsequent signal-path). As you know, such is rarely the case.

More headroom would be a sane decision anyway, not only for not so good equipment, but that's not how the business is rolling for the last couple of years (Loudness Wars).

Closed loop negative-feedback amplifiers have been widely used since the era of vacuum tube amplifiers. Transistors and FETs and operational amplifiers made up of such discrete semiconductors are capable of providing much higher voltage/current gains than vacuum tube configurations.

You seem to get the impression that I am more fluent with tube amplifiers than transistors, but the opposite is the case. All of my equipment consists of opamps except for some tiny tubes in the preamp section of an mostly unused guitar amp.

Of course that doesn't mean that I know the exact internals of opamps at all (just never needed this kind of knowledge). And as you can see I sometimes like to gather some "drive-by" (half)knowledge out of curiosity.

 

[Detail Man]

It is not the Digital to Analog converters that "get it wrong".

From what I understood it is the DAC. Here one example: "Also, make sure you set any Out Ceiling control to a little under 0dBFS, to cope with the D-A converters on some CD players, which react badly to signals that peak at 0dBFS. I use a -0.3dB setting, but some mastering engineers drop as low as -3dB."

Of course there is some confusion around whether this ceiling is left for protecting against intersample-peaks or because of just running converters too hot at their limit. I guess it's mostly the first.

DACs are usually very accurate as that goes. Trust me, I am an audio designer. There are a whole lot of people who write a whole lot of things who do not have a lot of understanding of what they are talking about. The "audio world" is rife with such hearsay, and repetitions of inaccurate information. Not at all unlike this photography stuff, where few understand digital signal-processing.

0.3 dB is only around -3%. 3 dB (around -29%) is meaningful. 6 dB (-50%) would be the conservative approach. But it is rarely followed. Pretty much all commercial CD's that I have processed in Sony SoundForge are right at 0 dB level. They don't care if your cheapo sound reproduction system "clips". As you say, "louder is better" - and that is why the average levels are also pumped-up so high in modern production. AM, and now FM radio routinely employ additional processing that compresses the transmitted music even more so. It's no wonder than there is no sense of dynamics left, and not a hint of silence between the notes in almost all modern mixes ...

When the DAC output is "AC-coupled" (using capacitive-coupling, as is pretty much always the case) in an audio signal-path, the square-wave response of a passive (resistor-capacitor) coupling network can (in the worst case) exceed the peak value of the square-wave by a full 6 dB (a linear factor of two). Thus, a full 6 dB of "headroom" would need to be left (relative to the maximum signal level that can be reproduced by any amplifier in the subsequent signal-path). As you know, such is rarely the case.

More headroom would be a sane decision anyway, not only for not so good equipment, but that's not how the business is rolling for the last couple of years (Loudness Wars).

Closed loop negative-feedback amplifiers have been widely used since the era of vacuum tube amplifiers. Transistors and FETs and operational amplifiers made up of such discrete semiconductors are capable of providing much higher voltage/current gains than vacuum tube configurations.

You seem to get the impression that I am more fluent with tube amplifiers than transistors, but the opposite is the case. All of my equipment consists of opamps except for some tiny tubes in the preamp section of an mostly unused guitar amp.

Of course that doesn't mean that I know the exact internals of opamps at all (just never needed this kind of knowledge). And as you can see I sometimes like to gather some "drive-by" (half)knowledge out of curiosity.

I noticed that. I started with Op Amps, then learned discrete semiconductors, then learned tubes! You can trust what I tell you. I really do (where it comes to audio engineering) know some things.

 

[Timur Born]

I think one reason for the -0.3 dB advice is that for a time many CD players seemed to have employed the same widely available (not too expensive) DACs which all suffered from a common issue.

Anyway, the reason why I was curious about where within its limits the internals of a camera are run is mostly because in my experience it is the limits where problems show up (or rather it's more expensive to get the limits right). And even if by design it should work, more often than not things don't work in practice as they are intended to do on paper.

So with a huge headroom of 80 dB vs. a limited signal range of 40 dB it seems to make sense not to scrape at the limit, but play it save, simply because it is not necessary and may even save some dough and headaches.

 

[Detail Man]

I think one reason for the -0.3 dB advice is that for a time many CD players seemed to have employed the same widely available (not too expensive) DACs which all suffered from a common issue.

Huh, maybe. -0.3 dB is a mere -3.4 %. Whatever.

Anyway, the reason why I was curious about where within its limits the internals of a camera are run is mostly because in my experience it is the limits where problems show up (or rather it's more expensive to get the limits right). And even if by design it should work, more often than not things don't work in practice as they are intended to do on paper.

Well, the whole matter of understanding anything sounds rather futile if that is true. Why bother.

So with a huge headroom of 80 dB vs. a limited signal range of 40 dB it seems to make sense not to scrape at the limit, but play it save, simply because it is not necessary and may even save some dough and headaches.

Signal/Noise RATIO is not "headroom". It does represent a range (ratio) of signal levels over which a device or system can transmit signals without them being lost in noise. "Headroom" is a margin allowed for peak levels that may exceed some given "average" level of a signal. Different concept.

 

[bobn2]

An amplifier running with 40dB of negative feedback is linear right up until where it clips. That's the difference between 'transistor sound' and 'tube sound'.

I will have to (and do) take your word on that they are implemented and work that way in practice.

One question: are bandwidth limitations of the amplifier - and thus feedback loop - a problem for linearity of high frequency content in images?

One remark: AFAIK CDs are usually mastered at -0.3 to -0.5 dB in order to stay somewhat below the clipping point of DA converters in order to protect from non-linearities (distortion) that happens with many (inexpensive) converter/gain circuits in common CD players.

Frankly I never cared how these things are implemented on their lowest level, but now I wonder (and should maybe ask my contacts at RME Audio) how amplifier circuits inside and around are build up (closed vs. open loop). But that's pure curiosity and really not the topic of this thread.

I'll defer to DM for your explanation of this, he has much more experience of this area of engineering than do I.
--
Bob

 

[Detail Man]

An amplifier running with 40dB of negative feedback is linear right up until where it clips. That's the difference between 'transistor sound' and 'tube sound'.

I will have to (and do) take your word on that they are implemented and work that way in practice.

One question: are bandwidth limitations of the amplifier - and thus feedback loop - a problem for linearity of high frequency content in images?

One remark: AFAIK CDs are usually mastered at -0.3 to -0.5 dB in order to stay somewhat below the clipping point of DA converters in order to protect from non-linearities (distortion) that happens with many (inexpensive) converter/gain circuits in common CD players.

Frankly I never cared how these things are implemented on their lowest level, but now I wonder (and should maybe ask my contacts at RME Audio) how amplifier circuits inside and around are build up (closed vs. open loop). But that's pure curiosity and really not the topic of this thread.

I'll defer to DM for your explanation of this, he has much more experience of this area of engineering than do I.

Just when I was hoping that you might "pick up the gauntlet" at the podium, Obe-Wan KenBobby

 

[Timur Born]

Huh, maybe. -0.3 dB is a mere -3.4 %. Whatever.

Can be mere, can be a lot, depends on the breaking point. But it doesn't affect any of us anyway, so yes, whatever.

Well, the whole matter of understanding anything sounds rather futile if that is true. Why bother.

One reason is to get an idea of how these things are handled by developers/producers, how they "tick" and what tendencies they follow.

Signal/Noise RATIO is not "headroom". It does represent a range (ratio) of signal levels over which a device or system can transmit signals without them being lost in noise. "Headroom" is a margin allowed for peak levels that may exceed some given "average" level of a signal. Different concept.

Exchange headroom with range in my sentence, but in both cases you get the idea. The shoe is a lot bigger than the foot and my question was whether the foot is placed at the heel or the tip of the shoe. I didn't mean to cause so much writing troubles, it was a rather innocent question.

 

[noirdesir]

Iliah's pedagogic style. Doesn't lecture, just drops you cryptic one liners that you need to follow and dig under to do the learning for yourself. Since my pedagogic style is more of the lecturing variety, and that upsets people too, I wouldn't be one to say which causes more resentment. But the underlying point is the same, people who resent being taught don't learn.

Except that in 99.9% of all cases people don't learn anything from Iliah. If nobody in the thread understands what Iliah is referring to, than nobody learns anything. The result is that is very few believe that Iliah's main motivation is to further understanding (simply because he is definitely unsuccessful with this in the vast majority of cases) but rather to feel superior because all those poor souls fail to understand what he is referring to. The impression is really that if you are not able to figure something out yourself, you are not worthy of understanding it in Iliah's eyes.

In this particular case Anders has shown linear raw value behaviour (up to a point) and Iliah has claimed that this is the result of linearisation during raw file generation (which I am willing to believe because other pieces of data I have seen, though not in this thread). But that leaves 90% of Iliah's doubting questions unanswered by anybody, so basically nobody knows whether Iliah is really smart or just really arrogant.

 

[Iliah Borg]

He's also very probably right (though I don't have any information of my own) that manufacturers are using non-linear coding to re-linearise sensor response (he gave the example of the D4).

Here is a shot from D800, http://www.imaging-resource.com/PRODS/nikon-d800/nikon-d800THMB.HTM
One can open it in RawDigger and add samples on the grey patches, Alt-Click



Now let's look at samples table, Ctrl-L, or Menu-Windows-Samples



Transfer the data, say, to Excel (to do so, while in samples table, - Ctrl-A to select all rows and press Selected to Clipboard) and look at G2 channel



Do you also see SNR value in highlights being smaller than in midtones?

That is why I suggested that evaluating (non-)linearity without noise analysis is not enough; and unfortunately even when noise analysis does not show odd results we still may need to dig deeper.

--
http://www.libraw.org/
http://www.RawDigger.com/

 

[noirdesir]

Instead of just being dismissive of a previous post. Keep up the good work (seriously).

He's also very probably right (though I don't have any information of my own) that manufacturers are using non-linear coding to re-linearise sensor response (he gave the example of the D4).

Here is a shot from D800, http://www.imaging-resource.com/PRODS/nikon-d800/nikon-d800THMB.HTM
One can open it in RawDigger and add samples on the grey patches, Alt-Click



Now let's look at samples table, Ctrl-L, or Menu-Windows-Samples



Transfer the data, say, to Excel (to do so, while in samples table, - Ctrl-A to select all rows and press Selected to Clipboard) and look at G2 channel



Do you also see SNR value in highlights being smaller than in midtones?

That is why I suggested that evaluating (non-)linearity without noise analysis is not enough; and unfortunately even when noise analysis does not show odd results we still may need to dig deeper.