DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions
GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Mjankor wrote:

cptobvious wrote:

Mjankor wrote:

Ok, you seem to be under the impression that DxO's ISO actually means something like the general definition of ISO.

So, two cameras, Camera A which measures DXO ISO of 200, at its claimed ISO 200, and Camera B, which measures DXO ISO 100 at its claimed ISO 200.

What do you think will happen will happen if both cameras photograph the same scene, same shutter speed, iso200, same fstop?

I don't take DxOMark's findings as gospel, but at least they've made an attempt at objective comparisons, something that other review sites have not come close to doing: http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Detailed-computation-of-DxOMark-Sensor-normalization

http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Pushed-ISO-Let-s-make-it-clear

To answer your question, Camera B will underexpose by 1 stop. There are a couple ways I think reviews could address this:

1. Show the side-by-side photos as-is with the uneven exposure.

2. Raise the ISO of Camera B to match the exposure of Camera A (or better, set both to a pre-determined reference exposure).

Either would be more informative than varying aperture or shutter speed.

I thought you'd think that.

Here is an experiment I did earlier today. I tested the Oly E520, which has a DXO ISO of 211 at a camera stated ISO 200, against the OM-D, which has a DXO ISO of 107, at a camera stated ISO of 200. The difference between the two cameras should be extremely obvious, if DXO ISO has any relevance to the actual ISO.

I set the same settings in each camera, ISO200, 25mm, f3.4, 1/800.

The same lens was used for both shots and both cameras were in raw.

Then, in Aperture I grabbed the 1/800 shots from each camera, exported them out in Jpeg, downsized the OM-D file to match the E520, cut it in half and dropped it over the top.

As you can see, both cameras meter about the same and record photos of similar exposure. If anything the E520 is a touch underexposed compared to the OM-D.

Conclusion:

The OM-D definitely did not require a shutter speed twice as slow as the E520 to take a similarly exposed image. Ergo the hypothesis that DXO ISO is comparable to the camera ISO is wrong.

This was a great experiment, but didn't go far enough.  It does attempt to prove that even though the DxOMark rated raw ISO sensitivities are different by a factor of over two, the same exposure can produce the same output level.  What you failed to prove because such low ISO's were used is any difference in noise between the Olympus E520 and the E-M5 and the E-M5 on a similar viewing size as you show above.

The other thing you fail to document is that the recorded raw histogram in the raw files will be completely different as revealed by a program such as RawDigger, with the E-M5 having only about half the span of histogram compared to that of the E520, which is why DxOMark raw ISO value shows that the E-M5 has about half the raw sensitivity as the E520; your raw converter of choice, Apple Aperture, recognized from the models of the camera that it had to apply an exposure boost to the raw output of the E-M5 compared to the raw output of the E520 in order to get about the same standard output (or may have applied an automatic exposure function as many raw converters do).  When evaluating raw files, one needs to look at the raw histograms made available by such programs as RawDigger in order to show that this is done.

When a raw converter applies an exposure compensation in order to standardize output brightness, the effect is to increase the effective ISO sensitivity of the image with the darker raw histogram as compared to the one with the brighter raw histogram, in this case roughly doubling the effective ISO of the E-M5 as compared to the E520 and making the output ISO sensitivities essentially equal, but in the case of the E-M5 as compared to the E520 leaving the E-M5 still with a noise advantage with the E-M5 being able to be set to ISO 400 (with shutter speed increased to 1/1600 second with the same aperture for half the light striking the sensor) before noise levels would be the same as this camera set ISO 200 for the E520.

This is why DPR's statements to the effect that raw ISO sensitivities as measured by DxOMark don't matter for purposes of comparing noise for various raw camera ISO settings as long as output brightness is the same as the exposure boost made in equalizing the RGB output will compensate for any differences in raw ISO sensitivity as in what is measured by DxOMark - as long as illumination levels are consistent, which DPR state are for purposes of ISO noise testing.

DPR are said to determine differences in JPEG ISO sensitivity between cameras, which for most modern cameras is pretty consistent within about a sixth to no more than a third of a stop, then do the rest of the testing based on that as a standard.  If their illumination sources are consistent between camera models, then each camera should be getting the same amount of light for each ISO sensitivity setting and if final image output levels from raw are about the same than any differences between raw sensitivity should be compensated, whether the brightness levels are adjusted manually or automatically in the raw conversion process.

Regards, GordonBGood

GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jeff Charles wrote:
I have read a number of Andy Westlake's posts, and my understanding from them is that DPR adjusts output brightness by varying shutter speed, i.e. exposure. Here's what he wrote in a post about a year ago:
The ISO testing uses critically-controlled lighting. Once you know the results of the that, nothing else needs to be shot under 'critically' controlled lighting (i.e. to a fraction of an EV) , it just needs to be shot to a controlled output brightness. Then, by the very definition of ISO, it doesn't matter whether the light level is slightly different and you compensate by changing the shutter speed - if the grey patches are rendered at the correct output brightness, you've got the same exposure.
He's right that changing shutter speed is a valid way to adjust for different light levels, but he's not right that "if the grey patches are rendered at the correct output brightness, you've got the same exposure." Output brightness is a function of processing, not exposure.

Jeff, there seems to be some clarification needed as to how output brightness is used for purposes of adjust raw conversions; you are correct that it enters into the evaluation equations. For instance, for the purposes of comparisons of the E-M5 to any other camera, if brightness compensations are used to brighten the raw conversion output than the effect is to increase the effective ISO sensitivity.

Also, I haven't read a post by any DPR staff member that says they adjust output brightness in ACR. Check this post from Richard Butler, in which he dismisses my suggestion that they do so.

The problem with the use of Adobe Camera Raw (ACR) as a raw conversion "standard" is that it is known to automatically apply differing conversion parameters by model so that comparison of the results is essentially useless.  For instance, we know that the raw histograms from the E-M5 will have about half the width of more typical raw histograms, yet when passed through ACR using default settings (even without automatic exposure compensation), the outputs come to to look to have the same brightness - ACR's defaults settings are so as to produce about the same output brightness as for Out Of Camera (OOC) JPEG's.

There are almost certainly other differences in processing applied by camera model or errors in setting equivalent exposures.  For instance, in the E-M5 review where raw noise is compared to that of the Sony NEX-7, there must be a reason for why the grey noise is plotted to be so much higher than that of the EM-5 which isn't there according to DxOMark (which I trust), or for that matter differences in grey noise between the Nikon D7000 and the Pentax K-5 as to raw grey noise which we know should be almost identical for a given ISO sensitivity.

For these discrepancies, I find that I can't really use DPR reviews for comparisons of raw noise.

Regards, GordonBGood

Jack Hogan Veteran Member • Posts: 6,983
Good Show

tko wrote:

Suppose instead of ISO you have a knob labeled "noise" or "gain." How would you measure performance? You'd set a standard exposure, set the "gain" knob for the proper intensity (RAW file values), and measure the resulting noise.

Very well said. That's exactly what DxO ISO allows us to do: it tells us at what camera-specific value to set the "gain" (in-camera ISO) knob so that every camera receives the same 'proper intensity (Raw values)'.

Still I guess the current method works pretty well until you get the odd player out and everyone gets excited.

Yes. If everybody labelled their "gain" knob with the same SI units threads like this would not exist.

Jack

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

GordonBGood wrote:

Mjankor wrote:

cptobvious wrote:

Mjankor wrote:

Ok, you seem to be under the impression that DxO's ISO actually means something like the general definition of ISO.

So, two cameras, Camera A which measures DXO ISO of 200, at its claimed ISO 200, and Camera B, which measures DXO ISO 100 at its claimed ISO 200.

What do you think will happen will happen if both cameras photograph the same scene, same shutter speed, iso200, same fstop?

I don't take DxOMark's findings as gospel, but at least they've made an attempt at objective comparisons, something that other review sites have not come close to doing: http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Detailed-computation-of-DxOMark-Sensor-normalization

http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Pushed-ISO-Let-s-make-it-clear

To answer your question, Camera B will underexpose by 1 stop. There are a couple ways I think reviews could address this:

1. Show the side-by-side photos as-is with the uneven exposure.

2. Raise the ISO of Camera B to match the exposure of Camera A (or better, set both to a pre-determined reference exposure).

Either would be more informative than varying aperture or shutter speed.

I thought you'd think that.

Here is an experiment I did earlier today. I tested the Oly E520, which has a DXO ISO of 211 at a camera stated ISO 200, against the OM-D, which has a DXO ISO of 107, at a camera stated ISO of 200. The difference between the two cameras should be extremely obvious, if DXO ISO has any relevance to the actual ISO.

I set the same settings in each camera, ISO200, 25mm, f3.4, 1/800.

The same lens was used for both shots and both cameras were in raw.

Then, in Aperture I grabbed the 1/800 shots from each camera, exported them out in Jpeg, downsized the OM-D file to match the E520, cut it in half and dropped it over the top.

As you can see, both cameras meter about the same and record photos of similar exposure. If anything the E520 is a touch underexposed compared to the OM-D.

Conclusion:

The OM-D definitely did not require a shutter speed twice as slow as the E520 to take a similarly exposed image. Ergo the hypothesis that DXO ISO is comparable to the camera ISO is wrong.

This was a great experiment, but didn't go far enough. It does attempt to prove that even though the DxOMark rated raw ISO sensitivities are different by a factor of over two, the same exposure can produce the same output level. What you failed to prove because such low ISO's were used is any difference in noise between the Olympus E520 and the E-M5 and the E-M5 on a similar viewing size as you show above.

I didn't see much point comparing the noise level between the E-M5 and the E520 as they are several generations apart. This experiment was designed to demonstrate that for a given scene and camera ISO, the metered exposure would be about the same and the final image file would be about the same brightness.

The other thing you fail to document is that the recorded raw histogram in the raw files will be completely different as revealed by a program such as RawDigger, with the E-M5 having only about half the span of histogram compared to that of the E520, which is why DxOMark raw ISO value shows that the E-M5 has about half the raw sensitivity as the E520; your raw converter of choice, Apple Aperture, recognized from the models of the camera that it had to apply an exposure boost to the raw output of the E-M5 compared to the raw output of the E520 in order to get about the same standard output (or may have applied an automatic exposure function as many raw converters do). When evaluating raw files, one needs to look at the raw histograms made available by such programs as RawDigger in order to show that this is done.

Agreed. I had a hunch the only way I'd find this "1 stop difference" is through the use of something like RawDigger.

When a raw converter applies an exposure compensation in order to standardize output brightness, the effect is to increase the effective ISO sensitivity of the image with the darker raw histogram as compared to the one with the brighter raw histogram, in this case roughly doubling the effective ISO of the E-M5 as compared to the E520 and making the output ISO sensitivities essentially equal, but in the case of the E-M5 as compared to the E520 leaving the E-M5 still with a noise advantage with the E-M5 being able to be set to ISO 400 (with shutter speed increased to 1/1600 second with the same aperture for half the light striking the sensor) before noise levels would be the same as this camera set ISO 200 for the E520.

Agreed. Although in actual usage the IQ between these two cameras in this case is substantially larger than 1 stop. (eg: manufacturers ISO 800 on the E520 results in images with similar noise to the E-M5 at around manufacturers ISO of 3200-6400)

This is why DPR's statements to the effect that raw ISO sensitivities as measured by DxOMark don't matter for purposes of comparing noise for various raw camera ISO settings as long as output brightness is the same as the exposure boost made in equalizing the RGB output will compensate for any differences in raw ISO sensitivity as in what is measured by DxOMark - as long as illumination levels are consistent, which DPR state are for purposes of ISO noise testing.

DPR are said to determine differences in JPEG ISO sensitivity between cameras, which for most modern cameras is pretty consistent within about a sixth to no more than a third of a stop, then do the rest of the testing based on that as a standard. If their illumination sources are consistent between camera models, then each camera should be getting the same amount of light for each ISO sensitivity setting and if final image output levels from raw are about the same than any differences between raw sensitivity should be compensated, whether the brightness levels are adjusted manually or automatically in the raw conversion process.

Regards, GordonBGood

Agreed.

My thoughts are that while the OM-D may well underexpose and then gain up it's raw file this is not at all relevant to most of us as the process is pretty well invisible. The camera meters for the correct manufacturers ISO, and generates jpegs and raws at the correct output brightness level in most software. This allows us to compare it against other cameras without issue.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Mjankor wrote:

Nobody's stopping you from doing that, in fact you can check DPR's 'comparometer' as per my previous post. Too bad that if one manufacturer calls ISO 400 what another calls ISO 1600 you'd be unknowingly coming to the completely wrong conclusion... alas, for some ignorance is bliss

Jack

I'd like you to explain this.

Perhaps an example would help. Suppose one wanted to compare the noise performance of two different cameras in the kind of conditions that required ISO 200 in order to satisfy his artistic constraints for blown highlights, DOF and blur - and came across this comparison on a fictitious but nevertheless well respected site:

What erroneous conclusion would he reach?

Jack
PS Did I mention that the current standard basically allows manufacturers to label in-camera ISOs as they please? 400, 200, 100 or whatever they like? That's the EM5 at 25k, by the way, but I, the manufacturer, decided that today it felt more like 200  Clearer now?

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Jack Hogan wrote:

Mjankor wrote:

Nobody's stopping you from doing that, in fact you can check DPR's 'comparometer' as per my previous post. Too bad that if one manufacturer calls ISO 400 what another calls ISO 1600 you'd be unknowingly coming to the completely wrong conclusion... alas, for some ignorance is bliss

Jack

I'd like you to explain this.

Perhaps an example would help. Suppose one wanted to compare the noise performance of two different cameras in the kind of conditions that required ISO 200 in order to satisfy his artistic constraints for blown highlights, DOF and blur - and came across this comparison on a fictitious but nevertheless well respected site:

What erroneous conclusion would he reach?

Jack
PS Did I mention that the current standard basically allows manufacturers to label in-camera ISOs as they please? 400, 200, 100 or whatever they like? That's the EM5 at 25k, by the way, but I, the manufacturer, decided that today it felt more like 200 Clearer now?

Oh, right, so in your example, we can throw exposure out the window.

Now you're proposing testing 1 camera with 1/100 exposure time, against one with 1/4000 exposure time. Why?

In short, no, it's not clearer.

The OM-D meters like other cameras, and has a final image brightness like other cameras. It can use the same exposure as other cameras for a given final image brightness. Not sure what you think is going on, but it doesn't appear to align with real world tests.

The OM-D's manufacturers ISO rating is pretty well in line with most other cameras. Maybe you can present us with an example of a camera where the manufacturer has declared their ISO values off by a significant amount.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
Goodness tex, you've been busy

I don't even know where to begin, so I won't because something tells me Iìd be wasting my time, but for mere fun I'll take a crack at this gem

texinwien wrote:

Jack Hogan wrote: Because of this fact it is not possible to measure the performance of camera A at a set in-camera ISO (let's call it Subjective S) and expect to meaningfully compare its results to those from Camera B at the same ISO setting.

Please provide a concise example in support of this statement. To be clear, I am challenging the validity of this statement and asking you to provide proof for it.

Here is my concise example:

Fictitious example of why arbitrary in-camera ISO labelling is not conducive to meaninglful comparisons

Cheers,

Jack

ultimitsu
ultimitsu Veteran Member • Posts: 6,650
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

GordonBGood wrote:

In summary, the differing exposures of the cameras means very little to the raw image quality/noise comparisons when both have been post processed to about the same output brightness levels other than if the required exposures started to get down to a tenth of a second or slower where differing amount of hot pixels would start to make a difference.

Again, the secondary conclusion is that much too much concern is being made about this and much more than it deserves.

With respect, This whole analysis is really erroneous. It really is elemental knowledge that the more exposure the more light thus the less noise there will be. All images above base iso need to be "brightened", it is done by in camera hardware amplification, or software gain, or post processing in one's computer. But none of that really matters because in the end of the day you are comparing which camera does better in low light.

OMD mjust not receive more exposure in order to have a meaningful comparison.

ultimitsu
ultimitsu Veteran Member • Posts: 6,650
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Mjankor wrote:

Oh, right, so in your example, we can throw exposure out the window.

It has been thrown already, OMD got 2/3 stop more exposure than G3, Jack just exaggerate that a bit more to make the point more clear.

Now you're proposing testing 1 camera with 1/100 exposure time, against one with 1/4000 exposure time. Why?

Exactly, if 1/100 vs 1/4000 is not right, what makes 1/100 vs 1/160 right?

The OM-D meters like other cameras, and has a final image brightness like other cameras.

Maybe maybe not, but DPR ought do the same exposure to OMD and other m4/3 cameras for a usable comparison.

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

ultimitsu wrote:

GordonBGood wrote:

In summary, the differing exposures of the cameras means very little to the raw image quality/noise comparisons when both have been post processed to about the same output brightness levels other than if the required exposures started to get down to a tenth of a second or slower where differing amount of hot pixels would start to make a difference.

Again, the secondary conclusion is that much too much concern is being made about this and much more than it deserves.

With respect, This whole analysis is really erroneous. It really is elemental knowledge that the more exposure the more light thus the less noise there will be. All images above base iso need to be "brightened", it is done by in camera hardware amplification, or software gain, or post processing in one's computer. But none of that really matters because in the end of the day you are comparing which camera does better in low light.

OMD mjust not receive more exposure in order to have a meaningful comparison.

Agreed, but as the OM-D uses the same exposure as most other cameras I don't see the problem.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
texinwien Veteran Member • Posts: 3,326
do your homework

olliess wrote:

texinwien wrote:

olliess wrote:

...From the EXIF data it looks as if both shots were taken with the same 50 mm Olympus lens at f/6.3, but the OM-D shot was exposed at 1/800 while the G3 shot was exposed at 1/1300, which is about +2/3 stops more exposure for the OM-D. This would agree with the subjective differences in brightness (and suggest that you would need to dial in +2/3 stop on the G3 to make the shutter speeds match). But what good is this as a "comparison" of images at fixed ISO, e.g., 3200?

Although you would think that to be the case, DPReview has addressed this in great detail and with absolute certainty - the EXIFs on these shots cannot be used to determine relative exposure levels, due to DPReview's testing regime.

So what can we determine from this comparison, if anything?

Everything, basically. Once you understand DPReview's testing regime, you'll understand that.

Read and digest the following 4 posts from Mr. Westlake, them get back to me:

http://forums.dpreview.com/forums/post/40933688

http://forums.dpreview.com/forums/post/40944061

http://forums.dpreview.com/forums/post/40947308

http://forums.dpreview.com/forums/post/40947640

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

ultimitsu wrote:

Mjankor wrote:

Oh, right, so in your example, we can throw exposure out the window.

It has been thrown already, OMD got 2/3 stop more exposure than G3, Jack just exaggerate that a bit more to make the point more clear.

For a different scene brightness. DPR specifically state that the scene is not necessarily the same brightness for each test. If it's darker, you need a longer exposure.

Now you're proposing testing 1 camera with 1/100 exposure time, against one with 1/4000 exposure time. Why?

Exactly, if 1/100 vs 1/4000 is not right, what makes 1/100 vs 1/160 right?

The OM-D meters like other cameras, and has a final image brightness like other cameras.

Maybe maybe not, but DPR ought do the same exposure to OMD and other m4/3 cameras for a usable comparison.

False premise. The studio scene is not shot under the same lighting conditions each time. The ISO tests are the ones you're interested in. (They are shot with controlled lighting).

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Mjankor wrote:

Oh, right, so in your example, we can throw exposure out the window.

Now you're proposing testing 1 camera with 1/100 exposure time, against one with 1/4000 exposure time. Why?

In short, no, it's not clearer.

You seem to assume that changing ISO in-camera means changing Exposure .  In the context of this discussion, that is that of an intermediate, IQ-aware photographer wishing to compare the SNR performance of two similar cameras, that is definitely not the case.  Assuming the same scene, lens and sensor size - f/number and shutter speed (that is Exposure) are fixed and the same, because they are defined by your artistic requirements for DOF and Blur.  So set them as needed, the same on both cameras, and forget them.  Now the question of ISO comes up.

If one camera labelled in-camera ISO settings 1/2, 1, 2, 4, 8 etc. and the other A, B, C, D etc. and you wanted to compare fairly each camera's SNR performance capturing the very same scene subject to the very same artistic constraints, how would you go about choosing the appropriate in-camera ISO setting for both, knowing that the in-camera rendering engine (or raw converter in PP) can make the output image look as bright as one wants??

ISO only.  Exposure (f/n and ss) stays fixed, say at f/6.3 1/800, and the same on both cameras.

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

Mjankor wrote:

Oh, right, so in your example, we can throw exposure out the window.

Now you're proposing testing 1 camera with 1/100 exposure time, against one with 1/4000 exposure time. Why?

In short, no, it's not clearer.

You seem to assume that changing ISO in-camera means changing Exposure . In the context of this discussion, that is that of an intermediate, IQ-aware photographer wishing to compare the SNR performance of two similar cameras, that is definitely not the case. Assuming the same scene, lens and sensor size - f/number and shutter speed (that is Exposure) are fixed and the same, because they are defined by your artistic requirements for DOF and Blur. So set them as needed, the same on both cameras, and forget them. Now the question of ISO comes up

No. The example you gave had different exposures. The OM-D at at 25k used a shutter of 1/4000, the S110 used a shutter value of 1/100. (I ignored the aperture, I'm afraid)

If one camera labelled in-camera ISO settings 1/2, 1, 2, 4, 8 etc. and the other A, B, C, D etc. and you wanted to compare fairly each camera's SNR performance capturing the very same scene subject to the very same artistic constraints, how would you go about choosing the appropriate in-camera ISO setting for both, knowing that the in-camera rendering engine (or raw converter in PP) can make the output image look as bright as one wants??

ISO only. Exposure (f/n and ss) stays fixed, say at f/6.3 1/800, and the same on both cameras.

Well, presumably a nice easy way to do so would be to stop through the ISO values and see which one gives a nice, well exposed final image. Then you could figure out if ISO 2 = ISO B, for example.

Or, you could shoot both cameras in base ISO and see how many stops of normalisation are required to get a nice well exposed image.

So how does this apply to the OM-D?

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
The heart of the matter

Mjankor wrote:

Jack Hogan wrote: If one camera labelled in-camera ISO settings 1/2, 1, 2, 4, 8 etc. and the other A, B, C, D etc. and you wanted to compare fairly each camera's SNR performance capturing the very same scene subject to the very same artistic constraints, how would you go about choosing the appropriate in-camera ISO setting for both, knowing that the in-camera rendering engine (or raw converter in PP) can make the output image look as bright as one wants?

ISO only. Exposure (f/n and ss) stays fixed, say at f/6.3 1/800, and the same on both cameras.

Well, presumably a nice easy way to do so would be to stop through the ISO values and see which one gives a nice, well exposed final image. Then you could figure out if ISO 2 = ISO B, for example.

Not really, but I am glad that we are getting to the heart of the matter. The fact is, both captures are and remain perfectly exposed independently of in-camera ISO settings: we chose that very Exposure to give the artistic results that we wanted, and since both cameras were set up with the same f/n and ss, they are both perfectly exposed according to our photographic wishes.  Exposure is fully defined by shutter speed and f/number only, all other things being equal.  But you knew that already.

Perhaps instead of 'well exposed' you meant to refer to a final image of 'pleasing tonal balance and brightness'. Unfortunately this is immaterial to our quest, because one can make an output as bright and tonally balanced as one wants in-camera or in-pp from any capture - sometimes at the expense of IQ (SNR for this thread). So we do not really care about image brightness other than to ensure that both outputs are about equally bright for comparison purposes. And I can confirm that both outputs have been made equally pleasingly bright, independently of the in-camera ISO chosen.

Clearly if an inappropriate ISO is chosen, one may look much noisier than the other.  Or perhaps one my clip desirable highlights, negating the artistic intent of our original Exposure perfect choices. But as far as brightness is concerned, not a problem.

So since we can choose any ISO we want while keeping Exposure perfect and unchanged, how do we go about choosing in-camera ISOs to make sure that we are comparing the two camera's performance fairly, in the situation described above?

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

Mjankor wrote:

Jack Hogan wrote: If one camera labelled in-camera ISO settings 1/2, 1, 2, 4, 8 etc. and the other A, B, C, D etc. and you wanted to compare fairly each camera's SNR performance capturing the very same scene subject to the very same artistic constraints, how would you go about choosing the appropriate in-camera ISO setting for both, knowing that the in-camera rendering engine (or raw converter in PP) can make the output image look as bright as one wants?

ISO only. Exposure (f/n and ss) stays fixed, say at f/6.3 1/800, and the same on both cameras.

Well, presumably a nice easy way to do so would be to stop through the ISO values and see which one gives a nice, well exposed final image. Then you could figure out if ISO 2 = ISO B, for example.

We are getting to the heart of the matter. The fact is, both captures are and remain perfectly exposed independently of in-camera ISO settings: we chose that very exposure to give the artistic results that we wanted, and since both cameras were set up with the same f/n and ss, they are both perfectly exposed according to our photographic wishes. you see, exposure is fully defined by shutter speed and f/nuymber, all other things being equal.

But perhaps instead of 'well exposed' you meant to refer to a final image of 'pleasing tonal balance and brightness'. Unfortunately this is immaterial to our quest, because one can make a capture as bright and tonally balanced as one wants in-camera or in-pp - sometimes at the expense of IQ (SNR for this thread). So we do not really care about image brightness other than to ensure that both outputs are about equally bright for comparison purposes. And I can confirm that both of them can be made equally pleasingly bright, independently of the in-camera ISO chosen. Clearly if an inappropriate ISO is chosen, one may look much worse than the other.

So since we can choose any ISO we want while keeping Exposure perfect and unchanged, how do we go about choosing in-camera ISOs to make sure that we are comparing the two camera's noise performance fairly, in the situation described above?

Choose two ISO's that provide a similar 'pleasing tonal balance and brightness', as I stated above.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
The Heart of the Matter

Mjankor wrote:

Choose two ISO's that provide a similar 'pleasing tonal balance and brightness', as I stated above.

No, because pleasing brightness can always be achieved (in-camera or in-pp) independently of ISO - but at the expense of noise and highlight clipping. So if you raise the ISO too much you may have an equally pleasingly bright output image but with unwanted clipping. If too little you may have an equally and pleasingly bright output image that's way too noisy. So what do you do?

The answer rests in selecting the in-camera ISO for each camera that results in highlights clipping in the recorded data at the same time. That way you have the same exposure, you record the same scene tonal range and, after you have ensured that both output images are about equally bright, you can finally take a look at them and fairly decide which looks cleaner.

So how do you go about choosing what in-camera ISO results in the same scene tonal range being recorded? You select the in-camera ISO for each camera that results in the same tone from the scene getting recorded at the same level of full scale in the Raw data of each.

Jack
PS You don't have to do it, DxO does it for us. And DxO tells us that to record the same tonal information from the same scene for a given exposure and equally bright and balanced outputs, the EM5 should be set to ISO 3200 and the G3 to ISO 1600. And then you look at the real world apples-to-apples results

Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

Mjankor wrote:

Choose two ISO's that provide a similar 'pleasing tonal balance and brightness', as I stated above.

No, because pleasing brightness can always be achieved (in-camera or in-pp) independently of ISO - but at the expense of noise and highlight clipping. So if you raise the ISO too much you may have an equally pleasingly bright output image but with unwanted clipping. If too little you may have an equally and pleasingly bright output image that's way too noisy. So what do you do?

The answer rests in selecting the in-camera ISO for each camera that results in highlights clipping in the recorded data at the same time. That way you have the same exposure, you record the same scene tonal range and, after you have ensured that both output images are about equally bright, you can finally take a look at them and decide which looks cleaner.

So how do you go about choosing what in-camera ISO results in the same scene tonal range being recorded? You select the in-camera ISO for each camera that results in the same tone from the scene getting recorded at the same level in the Raw data of each.

Jack
PS You don't have to do it, DxO does it for us

Ok. You've got the DXO data for my two cameras.

Give me the testing specifications and I'll run a test.

So, you're idea of measuring the difference between the G3 and the OM-D is to:

Set exposure the same.

Set OM-D camera ISO 1 stop higher than G3. 3200, instead of 1600.

In post processing, normalise the two images by shifting the OM-D's image back down about 1 stop or the G3's image up about 1 stop.

Or are you under the impression that they won't need to be normalised?

PS. Your real world, apples to apples results can be achieved (and probably have been achieved) by using 1 stop less exposure on the OM-D, than on the G3 (Actually about 2/3s due to the differences between those cameras you noticed earlier). eg: The OM-D would have used 1/320, the G3 would have used 1:200 (or thereabouts).

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Martin.au
Martin.au Forum Pro • Posts: 13,273
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Mjankor wrote:

Jack Hogan wrote:

Mjankor wrote:

Choose two ISO's that provide a similar 'pleasing tonal balance and brightness', as I stated above.

No, because pleasing brightness can always be achieved (in-camera or in-pp) independently of ISO - but at the expense of noise and highlight clipping. So if you raise the ISO too much you may have an equally pleasingly bright output image but with unwanted clipping. If too little you may have an equally and pleasingly bright output image that's way too noisy. So what do you do?

The answer rests in selecting the in-camera ISO for each camera that results in highlights clipping in the recorded data at the same time. That way you have the same exposure, you record the same scene tonal range and, after you have ensured that both output images are about equally bright, you can finally take a look at them and decide which looks cleaner.

So how do you go about choosing what in-camera ISO results in the same scene tonal range being recorded? You select the in-camera ISO for each camera that results in the same tone from the scene getting recorded at the same level in the Raw data of each.

Jack
PS You don't have to do it, DxO does it for us

Ok. You've got the DXO data for my two cameras.

Give me the testing specifications and I'll run a test.

So, you're idea of measuring the difference between the G3 and the OM-D is to:

Set exposure the same.

Set OM-D camera ISO 1 stop higher than G3. 3200, instead of 1600.

In post processing, normalise the two images by shifting the OM-D's image back down about 1 stop or the G3's image up about 1 stop.

Or are you under the impression that they won't need to be normalised?

PS. Your real world, apples to apples results can be achieved (and probably have been achieved) by using 1 stop less exposure on the OM-D, than on the G3 (Actually about 2/3s due to the differences between those cameras you noticed earlier). eg: The OM-D would have used 1/320, the G3 would have used 1:200 (or thereabouts).

Oh, wait a minute. I just spotted you've been using the studio scene from DPR for your example, in which case you cannot expect similar shutter speeds for an equal exposure as the lighting level is not well controlled.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
Jack Hogan Veteran Member • Posts: 6,983
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Mjankor wrote:

Jack Hogan wrote:

Mjankor wrote:

Choose two ISO's that provide a similar 'pleasing tonal balance and brightness', as I stated above.

No, because pleasing brightness can always be achieved (in-camera or in-pp) independently of ISO - but at the expense of noise and highlight clipping. So if you raise the ISO too much you may have an equally pleasingly bright output image but with unwanted clipping. If too little you may have an equally and pleasingly bright output image that's way too noisy. So what do you do?

The answer rests in selecting the in-camera ISO for each camera that results in highlights clipping in the recorded data at the same time. That way you have the same exposure, you record the same scene tonal range and, after you have ensured that both output images are about equally bright, you can finally take a look at them and decide which looks cleaner.

So how do you go about choosing what in-camera ISO results in the same scene tonal range being recorded? You select the in-camera ISO for each camera that results in the same tone from the scene getting recorded at the same level in the Raw data of each.

Jack
PS You don't have to do it, DxO does it for us

Ok. You've got the DXO data for my two cameras.

Give me the testing specifications and I'll run a test.

Been there, done that. On the other hand you may be interested in these fair, controlled, professional noise performance results from DxOmark.com. The EM5's Exmor sensor is about 2/3 of a stop better than the E520's 5 year old technology, and at least we know the comparison is valid.

So, you're idea of measuring the difference between the G3 and the OM-D is to:

Set exposure the same.

Set OM-D camera ISO 1 stop higher than G3. 3200, instead of 1600.

In post processing, normalise the two images by shifting the OM-D's image back down about 1 stop or the G3's image up about 1 stop.

Or are you under the impression that they won't need to be normalised?

Don't know, don't particularly care, it's immaterial to this discussion and it depends on the in-camera processing/raw-converter and profiles used.

PS. Your real world, apples to apples results can be achieved (and probably have been achieved) by using 1 stop less exposure on the OM-D, than on the G3 (Actually about 2/3s due to the differences between those cameras you noticed earlier). eg: The OM-D would have used 1/320, the G3 would have used 1:200 (or thereabouts).

If everything else is the same that seems rarther unfair to the OM-D, but to each his own

Keyboard shortcuts:
FForum MMy threads