DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions
ultimitsu
ultimitsu Veteran Member • Posts: 6,650
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Just so we do not waste time beat around the bush. In iso 6400 test images, Pana G3 and OMD  used the same lens (oly 50mm prime), both shot at F6.3, G3 1/2500, OMD 1/1600. that is a very significant 2/3 stop more light for OMD.

I know DPR insist that they cannot ensure lighting is the same for every shot. but I would guess the difference is in any case minimal. and the variation of exposure in DPR's images coincide with Imaging Resource too.

GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
4

cptobvious wrote:

GordonBGood wrote:

cptobvious wrote:

Yes, the ISO standard is silly, that's why comparisons based on a single testing methodology is needed.

Yes, the as the ISO standard only applies to output formats and isn't all that exact even at that, DxOMark had to come up with their own sensitivity "standard: for the purposes of comparing linear raw sensor outputs, and there is nothing wrong with their "standard" for its intended purpose.

However, your concern about the comparison between the raw sensitivity and the camera ISO isn't really a concern as DxOMark shows that the actual Olympus OM-D E-M5 sensor performs normally for the size of the sensor as to performance so that after making whatever post processing adjustments are necessary for the actual raw sensitivity, the image quality is in line with a rating based on sensor size, while the Fuji X100 does have a slight amount more noise in the bright tones than other APS-C sensors; the overall DxOMark sensor scores for these cameras are mostly down graded because the sensors do not use as low a real sensitivity as some other camera sensors do and that the digital data acquisition circuits put a cap on Dynamic Range (DR) at about the camera measured ISO 400 sensitivity for the Olympus.

In short, for real equivalent use, although the Olympus E-M5 does have about a stop less high ISO sensitivity usability due to its smaller sensor, it's real limit is more due a limited DR as compared to the best sensors for low ISO sensitivity use (but not everyone needs or uses this) and the Fuji X100 has this same limit with in a additional slightly less efficiency in photon gathering efficiency per unit area than the best sensors; other than the difference in maximum DR, there is only about a half a stop in difference in high ISO use between these cameras and the best sensors, which won't be a major factor in choice of cameras - other factors such as user interface and usability or lens selection will weigh much more.

Regards, GordonBGood

Your points are well taken. But my concern with the E-M5's ISO calculation methodology is that it has misled reviewers to claim that it has the same noise performance as larger sensors that obviously perform better but had their ISOs calculated more conservatively. Some examples:

http://www.imaging-resource.com/PRODS/omd-em5/omd-em5RAW.HTM (scroll down for D7000 comparison)

http://www.stevehuffphoto.com/2012/05/31/crazy-comparison-the-olympus-om-d-e-m5-vs-nikon-d800-for-high-iso/ (claiming the E-M5 holds its own in ISO performance against the D800)

http://www.cameralabs.com/reviews/Olympus_OM-D_E-M5/Olympus_E-M5_vs_Sony_NEX-7_vs_Nikon_D3200_noise_RAW.shtml

http://dslr-check.at.webry.info/201204/article_10.html (Japanese site)

http://www.pcmag.com/article2/0,2817,2403444,00.asp (claims better noise performance than the NEX-7)

Even DPReview's studio shot comparison tool uses different aperture settings:

http://www.dpreview.com/reviews/studio-compare#baseDir=%2Freviews_data&cameraDataSubdir=boxshot&indexFileName=boxshotindex.xml&presetsFileName=boxshotpresets.xml&showDescriptions=false&headerTitle=Studio%20scene&headerSubTitle=Standard%20studio%20scene%20comparison&masterCamera=oly_em5&masterSample=p1010013.acr&slotsCount=4&slot0Camera=oly_em5&slot0Sample=p1010013.acr&slot0DisableCameraSelection=true&slot0DisableSampleSelection=true&slot0LinkWithMaster=true&slot1Camera=nikon_d7000&slot1Sample=dsc1_1891.acr&slot2Camera=sony_nex5n&slot2Sample=dsc00790.acr&slot3Camera=pentax_k5&slot3Sample=imgp3870.acr&x=-0.18003992015968062&y=-0.8257205072370949&extraCameraCount=0

I have seen posted on reviews and on forums numerous times that the E-M5's sensor is as good or even better than the Sony Exmor APS-C equivalents at noise performance. The links I just pulled up are from just one search on Google in 10 minutes. To me this is just dishonest on the part of Olympus, and is inevitably going to lead to a 'race to the bottom' in future generation of cameras by manufacturers playing looser with their reported ISOs.

The DxOMark results for the Olympus OM-D E-M5 show that the raw sensor response is about a half stop worse than the best APS-C sensors for equivalent raw sensor sensitivity for typical none wide Dynamic Range (DR) use, which I believe to be correct.

Comparing images on the basis of JPEG output with all the variables of various ways of processing including differing sharpening and Noise Reduction (NR) methods and amounts, many of which cannot be entirely disabled, is always a very "iffy" and subjective comparison.  I know that is why you have dug up all these reviews that compare conversions from raw sensor data, but even those can be skewed by the raw converter used as some such as Adobe Camera Raw (ACR) as used by DPR has custom processing applied by camera model so that the results can be affected. But many of these reviews don't really prove your point, as follows (in reverse order):

  1. DPR has admitted that the studio light levels aren't always consistent across different cameras nor have attempt been made in the past to keep consistent aperture and shutter speed settings, unlike as Image-Resource sample studio images do.  Even at that, the E-M5 results as per the posted comparison link is not better than the Nikon D7000 nor the Pentax K-5 at ISO 3200 and is likely about a half stop worse just as DxOMark would indicate.  The E-M5 does look better than the Sony NEX-5N but that may be due to the reasons as stated as the Sony is less exposed by about 2/3's of a stop.
  2. The unsupported-by-images statement by PCMag that the E-M5 has less noise than the Sony NEX-7 is true on a per pixel basis given that the NEX-7 has 50% more of them.
  3. The Japanese website/blog showing that the E-M5 has close to the same noise as that of the Sony NEX-7 is probably correct according to DxOMark as it shows the objects are about the same size, meaning that the NEX-7 image has likely been down sampled by some unknown algorithm of the SilkyPix raw converter program used.
  4. CameraLabs use ACR for their comparison, with the usual provisos:  it is known to use different processing depending on camera model and the 24 Megapixel images have been sized to match the 16 Megapixel output of the E-M5.  The results don't really prove anything other than that the cameras are quite close in image quality for a given viewing size.
  5. The Steve Huff article proves nothing other than that the per photosite noise of the E-M5 has a similar level to that of the Niikon D800 when each is set to the same camera ISO sensitivity, which comes as no surprise given that DxOMark also predicts this for the two.
  6. The best and most consistent comparison is that of the Imaging-Resource (IR) link but again there are no surprises revealed in that DxOMark results predict that the E-M5 has better image quality that any of the cameras compared:  The Olympus E-P3, the Panasonic G3, and the Samsung NX200.  Looking at the exposure data for the sample images we see that the E-M5 is exposed by about 20% more than the Samsung NX200, but that the raw histogram still shows the raw sensor data for the E-M5 to be still less exposed than that of the NX200.

Now, other than for the IR comparison (which also uses the open-source dcraw converter as a verifiable camera-model-neutral application), it is difficult to dig out the exposures used for the other reviews to verify your clams that ISO settings are distorted excluding the DPR comparisons where they state themselves that there is every indication to believe that metered ISO sensitivity is standard as stated by the camera for output images within a small number of stops; there are no indications that there is any "foolery" done by Olympus other than to underexpose the raw captures for  more "raw highlight headroom" with metering set to be slightly compensate this by slight over exposure.  Raw-Digger examination of DPR and IR raw example files confirm this:  the E-M5 raw captures have more "highlight headroom" than comparable cameras even though they meter to expose slightly more.

However, every raw comparison you have cited has adjusted exposure in raw conversion and thus effective ISO sensitivity in order to compensate for the dark raw histogram else the IR comparison would show the EM-5 as much darker than the result from the Samsung NX200, for example.  Thus, the reviews use of the raw conversion programs (or in many cases automatic compensation applied by those programs) has done pretty much what the scaling and shifting of the DxOMark results due to their determination of real raw sensor sensitivity does.  In other words, raw conversion whether by camera or by raw converter application pretty much cancels out the effect of underexposure of the raw captures.

The conclusion remains:  DxOMark results stand that the Olympus OM-D E-M5 has a pretty good sensor but not quite as good as the best of the new APS-C sensors (or 35 mm. sensors) when compared on a per image size viewed basis.  None of the reviews to which you've linked refute this and only take a slightly different approach at their comparisons which don't make the conclusion as clear.  An alternate summary could be that you are wasting everyone's time in constantly harping that the E-M5 isn't as good a camera as many believe it to be based on this; it likely doesn't have quite the image quality of the best competing APS-C sensors, but that has little or nothing to do with Olympus's use of the sensor or their choice of metering sensitivity.

Regards, GordonBGood

GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

edhannon wrote:

Gordon, not sure I see the usefullness of ISO rating, either for comparison between cameras or use in a camera I use.

IMHO better measures are the number of stops saturation is above middle grey meter reaading and the number of stops a SNR of 0 is below meter reading.

That gives the useable dynamic range and whether the camera under/over exposes.

For day to day use those numbers allow you to adjusrt exposure for difficult subject brightness ranges.

One complication is that the saturation number varies based on quality of light (sunny, cloudy, shade, tungsten) and type of lens (new Pentax, M42, other manufactures).

Comments?

Ed, it is true that as ISO sensitivity ratings mostly only apply to image output and as a comparison tool they mostly only apply to JPEG's as output from the camera if the ISO sensitivity ratings are accurate; however, we know that there are so many unknown variables that apply to JPEG's output from the camera as in contrast curves applied, colour transformations, sharpening amounts and types used, noise reduction amounts and types, etcetera, that such comparisons are very difficult to make.

Your definition of usable Dynamic Range (DR) is of course the same as DxOMark has defined to which you add the metered grey point into that range.  As a raw shooter you are correct that this is all you really need, other than a further compensation due to the colour of the illumination and/or the subject as you have noted.

And that would likely be all that we would ever have to know if our cameras were set to work tbased on those principles.  Unfortunately, we still have this control known as ISO sensitivity which doesn't really do all that much for the best of modern cameras other than to throw away "highlight headroom" but which control we are forced to use in order to get useful image thumbnails and previews to evaluate focus and framing and even to make our automatic metering work for situations where we want to use only the lower part of our sensors raw exposure range.

I think that you have nailed why even the raw noise comparisons of DPR reviews are less useful than those of DxOMark:  DPR makes no compensation for real raw sensor exposures whereas DxOMark does this "DxO ISO sensitivity" compensation.

I think discussions such as in this thread and those related ones all comes about because of misunderstands one major point and a related side effect of poor data acquisition electronics:  that all digital sensors only have one real ISO sensitivity with all other sensitivities obtained just by using sub ranges of that full range, and secondarily that the main obstacle to true ISO-less use of such sensors is the poor electronics that injects noise such that the full sensor DR at lowest ISO sensitivities cannot be used.

Regards, GordonBGood

GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

ultimitsu wrote:

Just so we do not waste time beat around the bush. In iso 6400 test images, Pana G3 and OMD used the same lens (oly 50mm prime), both shot at F6.3, G3 1/2500, OMD 1/1600. that is a very significant 2/3 stop more light for OMD.

I know DPR insist that they cannot ensure lighting is the same for every shot. but I would guess the difference is in any case minimal. and the variation of exposure in DPR's images coincide with Imaging Resource too.

Let's assume that the light illumination was exactly the same and see where that takes us:

  1. I use the ISO 3200 results as I already have downloaded them and there is less chance of compulsory raw Noise Reduction (NR) being used at ISO 3200 than at ISO 6400 according to DxOMark.
  2. I think both of us will accept that there is still a significant difference in exposure, both at ISO 3200 and f/6.3 with 1/800 and 1/1300 seconds, respectively, for the E-M5 and the Panasonic G-3, for an extra exposure of about two thirds of a stop.
  3. Things are even worse than that, as the EM-5 has a peak green histgram of about 1400/4093 whereas the Panasonic G3 has a peak green histogram of 2400 for another required extra exposure of two thirds of a stop in order to make the raw histograms about equivalent for a total extra exposure of about one and one third more exposure of the Olympus E-M5.
  4. Now the comparison shows both the E-M5 and the G3 to have about the same brightness after raw conversion, which means that the raw converter has boosted the raw histogram by about two thirds of a stop by digital multiplication, meaning that the actual real sensor ISO sensitivity as used in the RGB comparison image is about ISO 5500 in camera reference terms.
  5. However, DxOMark tells us that the real raw sensitivity of the sensor is less than half of the camera relative ones at 1489 for ISO 3200, meaning that the ISO sensitivity as used in DxOMark terms is about ISO 2560.
  6. For the Panasonic G3, DxOMark tells us that a camera ISO of 3200 is really a raw referenced DxOMark sensitivity of about ISO 2950.
  7. Thus, a little bit of the good showing of the E-M5 versus the G3 is due to a slightly lower effective raw ISO, but it isn't huge and we can still fight about slight differences in colour rendering, contrast, brightness, and sharpening as applied by ACR to the different camera models.
  8. Once we have compensated for all those variations, we will come to a conclusion just as DxOMark:  on a per pixel basis, the E-M5 and the G3 have very similar image quality as to noise for a given real raw sensor sensitivity, other than the much greater noise of the G3 in the deep shadows = decreased Dynamic Range (DR) as can be seen in the deep shadows as revealed in the DPR 100% pixel for pixel views.

In summary, the differing exposures of the cameras means very little to the raw image quality/noise comparisons when both have been post processed to about the same output brightness levels other than if the required exposures started to get down to a tenth of a second or slower where differing amount of hot pixels would start to make a difference.

Again, the secondary conclusion is that much too much concern is being made about this and much more than it deserves.

Regards, GordonBGood

Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs
3

gwlaw99 wrote:

What is baffling about this topic in general is that no one has developed a standardized way to test sensors so that consumers can make apples to apples comparisons.

But there is, as far as interchangeable lens DSC sensors are concerned*: it's called ISO Saturation Speed (Ssat) as defined in 1998 by standard ISO 12232.

Remarkably, given the tone of this thread, it's exactly the 'ISO' that DxOmark provides us with

Jack
*It does not of course have much to do with what is labelled 'ISO' on a camera set today because of an aberration called REI.  Sadly in 2006 the standard above was amended to add a couple of additional methods for defining 'ISO', including Recommended Exposure Index, which is arbitrarily determined by a camera vendor so that exposure will generate a picture of 'adequate' output level - 'adequate' and 'output' remaining undefined and at the whim of each vendor. Tsk, tsk.

edhannon
edhannon Senior Member • Posts: 1,761
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Gordon, you are right on about color adjustments.

Since I retired I have been enrolled in the local college's Associates in Science program in photography. First two courses were film. However, last semester finally got to digital.

Professor insisted that we shoot in raw at lowest ISO for every assignment.  Primary reason was that adaquate color balance to get a good grade on an image could not be done in JPEG - needed to be adjusted manually in Lightroom on a calibrated monitor.

Assignments were submitted in JPEG or TIFF format.  At first a couple of the students tried to short cut and do the assignment in JPEG. In the assignment critiques he spotted the ones done in JPEG every time.

-- hide signature --
 edhannon's gear list:edhannon's gear list
Pentax K10D Pentax K-3 Pentax smc DA 16-45mm F4 ED AL Pentax smc FA 77mm 1.8 Limited +1 more
edhannon
edhannon Senior Member • Posts: 1,761
ISO vs "pushing"

Gordan, your reply got me to thinking about ISO and digital.  In particular, the question of whether, when faced with a low light situation, if 'tis best to change ISO or to do like we did in the film days - expose as though we had higher ISO in the camera and push develop?

I have never been really satisfied with the testing to answer this.  Concerned that too many variables changed between the two comparison images - noise reduction, ettc.

So for my question: would it make sense to measure this way:

1. Create a tone curve for at the base ISO by taking a succesion of images of a grey card at varying exposures and plotting the result. I still have my program that plots this from the origianl raw files. It also identifies the the SNR 0 point.

2. Create another tone curve at a higher ISO (say 800)

3. Compare number of stops from metered grey to SNR of 0 for both. Subtract the number of stops difference between ISO 100 and 800 (3 stops) from the base ISO one

If the higher ISO has more stops than the adjusted base ISO then increasing the ISO is best.  If the adjusted base has more stops then "push" developing in Lightroom/ACR is best.  If they are about the same do what ever is easier.

Comments?

Ed Hannon
http://www.pbase.com/edhannon

 edhannon's gear list:edhannon's gear list
Pentax K10D Pentax K-3 Pentax smc DA 16-45mm F4 ED AL Pentax smc FA 77mm 1.8 Limited +1 more
kencurtis16
kencurtis16 New Member • Posts: 1
Re: DxOMark's measured ISOs vs. manufacturer ISOs

I think that was a vintage invasion  but still reliable.

-- hide signature --

INSTANT VIDEOS FROM THE PHOTOS & MUSIC  YOU LOVE!

Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs
3

GordonBGood wrote:

Let's assume that the light illumination was exactly the same and see where that takes us:

  1. I use the ISO 3200 results...

In summary, the differing exposures of the cameras means very little to the raw image quality/noise comparisons when both have been post processed to about the same output brightness levels other than if the required exposures started to get down to a tenth of a second or slower where differing amount of hot pixels would start to make a difference.

Hi GB,

I think the point is that, to stay within the boundaries of this thread, it is misleading to compare sensors at the same in-camera ISO setting. For instance if a hapless potential buyer were to compare the two images mentioned above in supposedly 'similar' conditions, the EM5 looks leaps and bounds better than the G3 in terms of noise performance. Incredible, wow, bravo Olympus!

Noise Performance at the same in-camera ISO settings of 3200

Convinced after such an authoritative comparison our hapless buyer goes out and gets an EM5. He is however disappointed to find out that, in the same field conditions, its noise performance is not really significantly better than his friend's much cheaper G3. How is this possible, DPR's comparison showed that at the same in-camera ISO his EM5 should blow the G3 out of the water!?!

Ah, but in-camera ISOs mean nothing these days of REI. Might as well give them a name: 1/2, base, 2x, 4x etc. To compare apples-to-apples the performance of two sensors (the basis of camera set IQ), one first needs to set them up so that they respond similarly to light. That's easy because DSC sensors being linear, one only needs to make sure that they clip at the same Exposure and ETTR. Luckily, DxO measures saturation exposure of every sensor for us. So our hapless EM5 owner, having evaluated his scene, sets ISO in both cameras up so that their sensors will both saturate at the same Hsat = 0.0525 lx-s, which just so happens to correspond to an in-camera setting of ISO 3200 for the EM5 and ISO 1600 for the G3 (or Ssat of 1489 and 1481 respectively). And this is the response he gets:

Noise performance when in-camera ISO is chosen so that both *sensors/ADCs* will provide the same mean outputl when presented with the same Exposure

Whoa, un-wow, right? The G3 is not so bad after all, in fact*... How dumb was he to compare cameras on the basis of in-camera ISO settings! If he had known he may well have chosen otherwise. He wonders whether the marketing folks at Olympus had anything to do with the ISO labelling and why the competent folks at DPR did not warn him about that, but with his new little jewel in his hands he soon forgets about the whole affair and goes out to take some outstanding pictures .-)

Again, the secondary conclusion is that much too much concern is being made about this and much more than it deserves.

Indeed, especially if one understands the above.

Cheers,
Jack
*All right, all right - so the lighting or the reflected meter constant k is slightly different between the two, resulting in shutter speeds 1/3 of a stop apart, but you get my point.

Andrew Westlake Senior Member • Posts: 2,928
Re: DxOMark's measured ISOs vs. manufacturer ISOs
3

Jack Hogan wrote:

The OP provides evidence that the EM5 is an outlier

The OP is using DxO's 'measured ISO' to reinterpret tests that are calibrated to a different ISO definition (ISO 12232:2006 Standard Output Sensitivity). This is entirely erroneous.

-- hide signature --

Andy Westlake
dpreview.com

 Andrew Westlake's gear list:Andrew Westlake's gear list
Fujifilm FinePix F30 Zoom Olympus XZ-2 iHS Olympus Stylus Verve S Panasonic Lumix DMC-GF1 Olympus OM-D E-M5 +11 more
Martin.au
Martin.au Forum Pro • Posts: 13,250
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

GordonBGood wrote:

Let's assume that the light illumination was exactly the same and see where that takes us:

  1. I use the ISO 3200 results...

In summary, the differing exposures of the cameras means very little to the raw image quality/noise comparisons when both have been post processed to about the same output brightness levels other than if the required exposures started to get down to a tenth of a second or slower where differing amount of hot pixels would start to make a difference.

Hi GB,

I think the point is that, to stay within the boundaries of this thread, it is misleading to compare sensors at the same in-camera ISO setting. For instance if a hapless potential buyer were to compare the two images mentioned above in supposedly 'similar' conditions, the EM5 looks leaps and bounds better than the G3 in terms of noise performance. Incredible, wow, bravo Olympus!

Noise Performance at the same in-camera ISO settings of 3200

Convinced after such an authoritative comparison our hapless buyer goes out and gets an EM5. He is however disappointed to find out that, in the same field conditions, its noise performance is not really significantly better than his friend's much cheaper G3. How is this possible, DPR's comparison showed that at the same in-camera ISO his EM5 should blow the G3 out of the water!?!

Ah, but in-camera ISOs mean nothing these days of REI. Might as well give them a name: 1/2, base, 2x, 4x etc. To compare apples-to-apples the performance of two sensors (the basis of camera set IQ), one first needs to set them up so that they respond similarly to light. That's easy because DSC sensors being linear, one only needs to make sure that they clip at the same Exposure and ETTR. Luckily, DxO measures saturation exposure of every sensor for us. So our hapless EM5 owner, having evaluated his scene, sets ISO in both cameras up so that their sensors will both saturate at the same Hsat = 0.0525 lx-s, which just so happens to correspond to an in-camera setting of ISO 3200 for the EM5 and ISO 1600 for the G3 (or Ssat of 1489 and 1481 respectively). And this is the response he gets:

Noise performance when in-camera ISO is chosen so that both *sensors/ADCs* will provide the same mean outputl when presented with the same Exposure

Whoa, un-wow, right? The G3 is not so bad after all, in fact*... How dumb was he to compare cameras on the basis of in-camera ISO settings! If he had known he may well have chosen otherwise. He wonders whether the marketing folks at Olympus had anything to do with the ISO labelling and why the competent folks at DPR did not warn him about that, but with his new little jewel in his hands he soon forgets about the whole affair and goes out to take some outstanding pictures .-)

Again, the secondary conclusion is that much too much concern is being made about this and much more than it deserves.

Indeed, especially if one understands the above.

Cheers,
Jack
*All right, all right - so the lighting or the reflected meter constant k is slightly different between the two, resulting in shutter speeds 1/3 of a stop apart, but you get my point.

Can you please include some hypothetical shutter, iso, etc numbers for these examples.

For example, the first two at ISO 3200 would be:

OMD - 1/100, ISO 3200, f5

G3 1/100, ISO 3200, f5.

And for the second

OMD - 1/100, ISO3200, f5

G3 - 1/50, ISO 1600, f5

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 60mm 1:2.8 Macro Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Panasonic Lumix G Fisheye 8mm F3.5 +7 more
GordonBGood Veteran Member • Posts: 6,308
Re: ISO vs "pushing"
1

edhannon wrote:

Gordon, your reply got me to thinking about ISO and digital. In particular, the question of whether, when faced with a low light situation, if 'tis best to change ISO or to do like we did in the film days - expose as though we had higher ISO in the camera and push develop?

I have never been really satisfied with the testing to answer this. Concerned that too many variables changed between the two comparison images - noise reduction, ettc.

So for my question: would it make sense to measure this way:

1. Create a tone curve for at the base ISO by taking a succesion of images of a grey card at varying exposures and plotting the result. I still have my program that plots this from the origianl raw files. It also identifies the the SNR 0 point.

2. Create another tone curve at a higher ISO (say 800)

3. Compare number of stops from metered grey to SNR of 0 for both. Subtract the number of stops difference between ISO 100 and 800 (3 stops) from the base ISO one

If the higher ISO has more stops than the adjusted base ISO then increasing the ISO is best. If the adjusted base has more stops then "push" developing in Lightroom/ACR is best. If they are about the same do what ever is easier.

Comments?

Ed, I think the question of the differences in image quality between underexposing a low ISO shot and "push processing" or to advance the ISO sensitivity of the camera has been fairly well answered but it depends on the camera.

Remember that "push processing" produces the the same sensitivity gain as the camera applies in camera to boost ISO sensitivity other than that the lower ISO's are generally boosted in the camera by analogue gain whereas higher ISO sensitivity gains are achieved the same as "push processing" by just a digital multiplication - the effect is the same and the reason analogue amplification is necessary is due to limitations of the data acquisition circuitry as to injecting noise between the sensor and the digitization.

For cameras that effectively do not inject noise (or use circuit means to cancel it out), digital multiplication has the exact same effect as analogue amplification as long as there are sufficient bits used in the digital capture to express all of the usable quantization levels in the signal above noise; however, for those cameras that have injected noise between the sensor and the digitization, analogue amplification works better for the first few stops in sensitivity gain as this amplification amplifies the signal without amplifying the injected noise as digital multiplication would.  Examples of this are pretty well all Canon DSLR's and are clearly seen in the DxOMark Dynamic Range (DR) charts as a nonlinear response with decreasing ISO sensitivity as they tend to reach a DR level limited by the injected noise.

The test that you are proposing essentially just repeats the testing done by DxOMark in determining these DR charts, and your results should be just about exactly the same as long as you allow for all the factors for which they allow such as that the very lowest level signals may be clipped to zero with the negative excursions of noise "blocked" if the raw data format has already applied black level compensation to offset black to level zero rather than some positive offset as is the normal output from the sensor.

So yes, you could do the testing but it is much easier to just refer to the DxOMark DR charts where the work is already done for you.

The test as you propose also has some extra variables that can cause some concern - for instance in order to test a camera with a 14 stop DR one would need to use shutter speeds for a given aperture over this same 14 stop range so that for a typical camera with a 1/4000 second fastest exposure time, one would need a four second exposure time; in order to avoid these slow exposure times below about a tenth of a second, one would need to use aperture as part of the exposure control which would add another variable.  DxOMark does this testing using single exposures taken of a type of step wedge so that only one exposure needs to be taken per ISO sensitivity.

Regards, GordonBGood

texinwien Veteran Member • Posts: 3,326
I wish I could favorite this 1,000 times!

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

I'm seriously laughing over here.

True, concise and hilarious. Touché!

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
texinwien Veteran Member • Posts: 3,326
I don't like to say "I told you so"

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

So I won't.

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
OP cptobvious Contributing Member • Posts: 810
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

GordonBGood wrote:

The DxOMark results for the Olympus OM-D E-M5 show that the raw sensor response is about a half stop worse than the best APS-C sensors for equivalent raw sensor sensitivity for typical none wide Dynamic Range (DR) use, which I believe to be correct.

Comparing images on the basis of JPEG output with all the variables of various ways of processing including differing sharpening and Noise Reduction (NR) methods and amounts, many of which cannot be entirely disabled, is always a very "iffy" and subjective comparison. I know that is why you have dug up all these reviews that compare conversions from raw sensor data, but even those can be skewed by the raw converter used as some such as Adobe Camera Raw (ACR) as used by DPR has custom processing applied by camera model so that the results can be affected. But many of these reviews don't really prove your point, as follows (in reverse order):

  1. DPR has admitted that the studio light levels aren't always consistent across different cameras nor have attempt been made in the past to keep consistent aperture and shutter speed settings, unlike as Image-Resource sample studio images do. Even at that, the E-M5 results as per the posted comparison link is not better than the Nikon D7000 nor the Pentax K-5 at ISO 3200 and is likely about a half stop worse just as DxOMark would indicate. The E-M5 does look better than the Sony NEX-5N but that may be due to the reasons as stated as the Sony is less exposed by about 2/3's of a stop.
  2. The unsupported-by-images statement by PCMag that the E-M5 has less noise than the Sony NEX-7 is true on a per pixel basis given that the NEX-7 has 50% more of them.
  3. The Japanese website/blog showing that the E-M5 has close to the same noise as that of the Sony NEX-7 is probably correct according to DxOMark as it shows the objects are about the same size, meaning that the NEX-7 image has likely been down sampled by some unknown algorithm of the SilkyPix raw converter program used.
  4. CameraLabs use ACR for their comparison, with the usual provisos: it is known to use different processing depending on camera model and the 24 Megapixel images have been sized to match the 16 Megapixel output of the E-M5. The results don't really prove anything other than that the cameras are quite close in image quality for a given viewing size.
  5. The Steve Huff article proves nothing other than that the per photosite noise of the E-M5 has a similar level to that of the Niikon D800 when each is set to the same camera ISO sensitivity, which comes as no surprise given that DxOMark also predicts this for the two.
  6. The best and most consistent comparison is that of the Imaging-Resource (IR) link but again there are no surprises revealed in that DxOMark results predict that the E-M5 has better image quality that any of the cameras compared: The Olympus E-P3, the Panasonic G3, and the Samsung NX200. Looking at the exposure data for the sample images we see that the E-M5 is exposed by about 20% more than the Samsung NX200, but that the raw histogram still shows the raw sensor data for the E-M5 to be still less exposed than that of the NX200.

According to DxOMark the E-M5 actually has the smallest pixel pitch among it and the NEX-7 and D800:

OM-D E-M5: 3.73 micrometers

NEX-7: 3.9 micrometers

D800: 4.7 micrometers

You can't just say a sensor has more noise because it has more pixels, without taking into account sensor size.

Assuming the 100% crops are the same proportion relative to the original image, a crop taken from a larger sensor will be from a larger exposed surface area than a smaller sensor. The E-M5 crops have the same or smaller pixels at a higher relative magnification so we should be seeing more noise.

I think the CameraLabs review says it all - they said they shot in aperture priority mode. None of the other reviews state how they exposed the image, but I suspect they did the same by varying shutter speed and/or aperture. By changing these variables they introduced confounding variables that makes high ISO comparisons essentially meaningless.

Martin.au
Martin.au Forum Pro • Posts: 13,250
Re: DxOMark's measured ISOs vs. manufacturer ISOs

cptobvious wrote:

GordonBGood wrote:

The DxOMark results for the Olympus OM-D E-M5 show that the raw sensor response is about a half stop worse than the best APS-C sensors for equivalent raw sensor sensitivity for typical none wide Dynamic Range (DR) use, which I believe to be correct.

Comparing images on the basis of JPEG output with all the variables of various ways of processing including differing sharpening and Noise Reduction (NR) methods and amounts, many of which cannot be entirely disabled, is always a very "iffy" and subjective comparison. I know that is why you have dug up all these reviews that compare conversions from raw sensor data, but even those can be skewed by the raw converter used as some such as Adobe Camera Raw (ACR) as used by DPR has custom processing applied by camera model so that the results can be affected. But many of these reviews don't really prove your point, as follows (in reverse order):

  1. DPR has admitted that the studio light levels aren't always consistent across different cameras nor have attempt been made in the past to keep consistent aperture and shutter speed settings, unlike as Image-Resource sample studio images do. Even at that, the E-M5 results as per the posted comparison link is not better than the Nikon D7000 nor the Pentax K-5 at ISO 3200 and is likely about a half stop worse just as DxOMark would indicate. The E-M5 does look better than the Sony NEX-5N but that may be due to the reasons as stated as the Sony is less exposed by about 2/3's of a stop.
  2. The unsupported-by-images statement by PCMag that the E-M5 has less noise than the Sony NEX-7 is true on a per pixel basis given that the NEX-7 has 50% more of them.
  3. The Japanese website/blog showing that the E-M5 has close to the same noise as that of the Sony NEX-7 is probably correct according to DxOMark as it shows the objects are about the same size, meaning that the NEX-7 image has likely been down sampled by some unknown algorithm of the SilkyPix raw converter program used.
  4. CameraLabs use ACR for their comparison, with the usual provisos: it is known to use different processing depending on camera model and the 24 Megapixel images have been sized to match the 16 Megapixel output of the E-M5. The results don't really prove anything other than that the cameras are quite close in image quality for a given viewing size.
  5. The Steve Huff article proves nothing other than that the per photosite noise of the E-M5 has a similar level to that of the Niikon D800 when each is set to the same camera ISO sensitivity, which comes as no surprise given that DxOMark also predicts this for the two.
  6. The best and most consistent comparison is that of the Imaging-Resource (IR) link but again there are no surprises revealed in that DxOMark results predict that the E-M5 has better image quality that any of the cameras compared: The Olympus E-P3, the Panasonic G3, and the Samsung NX200. Looking at the exposure data for the sample images we see that the E-M5 is exposed by about 20% more than the Samsung NX200, but that the raw histogram still shows the raw sensor data for the E-M5 to be still less exposed than that of the NX200.

According to DxOMark the E-M5 actually has the smallest pixel pitch among it and the NEX-7 and D800:

OM-D E-M5: 3.73 micrometers

NEX-7: 3.9 micrometers

D800: 4.7 micrometers

You can't just say a sensor has more noise because it has more pixels, without taking into account sensor size.

Assuming the 100% crops are the same proportion relative to the original image, a crop taken from a larger sensor will be from a larger exposed surface area than a smaller sensor. The E-M5 crops have the same or smaller pixels at a higher relative magnification so we should be seeing more noise.

I think the CameraLabs review says it all - they said they shot in aperture priority mode. None of the other reviews state how they exposed the image, but I suspect they did the same by varying shutter speed and/or aperture. By changing these variables they introduced confounding variables that makes high ISO comparisons essentially meaningless.

Ok, you seem to be under the impression that DxO's ISO actually means something like the general definition of ISO.

So, two cameras, Camera A which measures DXO ISO of 200, at its claimed ISO 200, and Camera B, which measures DXO ISO 100 at its claimed ISO 200.

What do you think will happen will happen if both cameras photograph the same scene, same shutter speed, iso200, same fstop?

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 60mm 1:2.8 Macro Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Panasonic Lumix G Fisheye 8mm F3.5 +7 more
OP cptobvious Contributing Member • Posts: 810
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Mjankor wrote:

Ok, you seem to be under the impression that DxO's ISO actually means something like the general definition of ISO.

So, two cameras, Camera A which measures DXO ISO of 200, at its claimed ISO 200, and Camera B, which measures DXO ISO 100 at its claimed ISO 200.

What do you think will happen will happen if both cameras photograph the same scene, same shutter speed, iso200, same fstop?

I don't take DxOMark's findings as gospel, but at least they've made an attempt at objective comparisons, something that other review sites have not come close to doing: http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Detailed-computation-of-DxOMark-Sensor-normalization

http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Pushed-ISO-Let-s-make-it-clear

http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/ISO-sensitivity

To answer your question, Camera B will underexpose by 1 stop. There are a couple ways I think reviews could address this:

1. Show the side-by-side photos as-is with the uneven exposure.

2. Raise the ISO of Camera B to match the exposure of Camera A (or better, set both to a pre-determined reference exposure).

Either would be more informative than varying aperture or shutter speed.

Martin.au
Martin.au Forum Pro • Posts: 13,250
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

cptobvious wrote:

Mjankor wrote:

Ok, you seem to be under the impression that DxO's ISO actually means something like the general definition of ISO.

So, two cameras, Camera A which measures DXO ISO of 200, at its claimed ISO 200, and Camera B, which measures DXO ISO 100 at its claimed ISO 200.

What do you think will happen will happen if both cameras photograph the same scene, same shutter speed, iso200, same fstop?

I don't take DxOMark's findings as gospel, but at least they've made an attempt at objective comparisons, something that other review sites have not come close to doing: http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Detailed-computation-of-DxOMark-Sensor-normalization

http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Pushed-ISO-Let-s-make-it-clear

To answer your question, Camera B will underexpose by 1 stop. There are a couple ways I think reviews could address this:

1. Show the side-by-side photos as-is with the uneven exposure.

2. Raise the ISO of Camera B to match the exposure of Camera A (or better, set both to a pre-determined reference exposure).

Either would be more informative than varying aperture or shutter speed.

I thought you'd think that.

Here is an experiment I did earlier today. I tested the Oly E520, which has a DXO ISO of 211 at a camera stated ISO 200, against the OM-D, which has a DXO ISO of 107, at a camera stated ISO of 200. The difference between the two cameras should be extremely obvious, if DXO ISO has any relevance to the actual ISO.

I set the same settings in each camera, ISO200, 25mm, f3.4, 1/800.

The same lens was used for both shots and both cameras were in raw.

Then, in Aperture I grabbed the 1/800 shots from each camera, exported them out in Jpeg, downsized the OM-D file to match the E520, cut it in half and dropped it over the top.

As you can see, both cameras meter about the same and record photos of similar exposure. If anything the E520 is a touch underexposed compared to the OM-D.

Conclusion:

The OM-D definitely did not require a shutter speed twice as slow as the E520 to take a similarly exposed image. Ergo the hypothesis that DXO ISO is comparable to the camera ISO is wrong.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 60mm 1:2.8 Macro Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Panasonic Lumix G Fisheye 8mm F3.5 +7 more
Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Mjankor wrote:

Jack Hogan: So our hapless EM5 owner, having evaluated his scene, sets ISO in both cameras up so that their sensors will both saturate at the same Hsat = 0.0525 lx-s, which just so happens to correspond to an in-camera setting of ISO 3200 for the EM5 and ISO 1600 for the G3 (or Ssat of 1489 and 1481 respectively). And this is the response he gets:

Noise performance when in-camera ISO is chosen so that both *sensors/ADCs* will provide the same mean outputl when presented with the same Exposure

Whoa, un-wow, right? The G3 is not so bad after all

Can you please include some hypothetical shutter, iso, etc numbers for these examples.

For example, the first two at ISO 3200 would be:

OMD - 1/100, ISO 3200, f5

G3 1/100, ISO 3200, f5.

And for the second

OMD - 1/100, ISO3200, f5

G3 - 1/50, ISO 1600, f5

In my simplified example above both cameras use the same lens and both sensors are set up* to clip the same scene highlights at the same Exposure (Hsat). Since DSC sensor performance below saturation is linear, that means that the mean Raw value produced by either sensor/ADC as a fraction of full scale will be about the same for any scene Exposure below this value - resulting in an apples to apples, objective comparison of the foundation of IQ: 'for a given amount of light (Exposure) from the scene, camera A produces a SNR of x and camera B a SNR of y'.

Exposure is fully defined by shutter speed and f/number only. So to make sensor Exposure the same one can use the same shutter speed and f/number in both cameras, or any other combination that keeps the Exposure Value constant. In the apples to apples image above with the same lens, the settings were

EM5 f/6.3 1/800s
G3 f/6.3 1/640s

The two shutter speeds should be identical but they are in fact 0.3 stops apart because of differences in metering - see the PS of my previous message.

*In-camera ISO settings do not matter for this comparison and are simply a consequence of having set up both cameras' sensors/ADCs to respond similarly to scene light, so that they will provide the same mean output level for a given input. If you are curious, that meant 3200 for the EM5 vs 1600 for the G3.

Cheers,
Jack

TrojMacReady
TrojMacReady Veteran Member • Posts: 8,729
to the OP: Overlooking a major factor.
2

If what you say were true (that the DXO measurements are the final word on how ISO's should relate to exposures and who's cheating and who isn't), the visible exposure from the X100 wouldn't change above ISO 1600. Because the DXO graph shows a horizontal line from ISO 1600 onwards.

But it does. So that should raise some new questions about the validity of your conclusions. Some of which are answered here.

In short, there's a big factor being overlooked, digital gain during conversion (through curves), be it in camera or the converters on our computers (based on metadata or profiling). In that sense, labeling dumb binary data with image exposure ISO's, causes more confusion than that it answers things.

Keyboard shortcuts:
FForum MMy threads