DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions thread
texinwien Veteran Member • Posts: 3,326
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Jeff Charles wrote:

texinwien wrote:

Jeff Charles wrote:

texinwien wrote:

This is one reasonable way to compare noise between two cameras, by taking pictures of the same test scene, ensuring that both get the same exposure, using matching camera ISO settings (aka exposure indices).

By the "same exposure", I assume you mean the same aperture and shutter speed, or at least an equivalent combination, on each camera.

Almost. It is also possible, in a studio setting, to adjust the illuminance, which, if you will read up on the earlier discussions of this topic, DPReview sometimes does.

If they use an external meter for each session, and set the camera accordingly, changing illumination is fine.

As long as scene illumination stays the same that's the essence of a valid test.

Correct.

However, to complete the test, we have to equalize output brightness between the two images, either by adjusting camera ISOs, or more likely, by boosting the brightness of the darker image on the computer.

Again, DPReview has addressed this. Where necessary, they have adjusted the ACR brightness or exposure slider to equalize brightness.

I have read a number of Andy Westlake's posts, and my understanding from them is that DPR adjusts output brightness by varying shutter speed, i.e. exposure.

Andy Westlake: The studio comparison shots are not ISO tests - they've not necessarily run under strictly-controlled lighting, for entirely pragmatic reasons. We've also recently changed the entire lighting setup to one that's sensibly dimmable, which means we don't run out of shutter speeds when testing cameras that can run at ISO 102800, and therefore don't have to mess around with ND gels over the lights. This has changed the base light level - and it simply doesn't matter given how our testing regime works.

This is also not the only official DPReview statement where adjusting the lighting is mentioned - that is, it is sometimes not possible to dial in exposure as finely as they'd like (with 1/3EV stops), in which case they can slightly adjust the light levels in order to make even finer adjustments to the exposure.

Further more, what I described in the last sentence is also a practice that imagine-resource.com uses, adjusting scene illuminance (by sliding their studio lights closer or farther away from the test scene) in order to effect finer adjustments to exposure than the 1/3EV adjustments allowed on most cameras - in other words, this seems to be an accepted practice for more than one studio that performs camera tests.

Here's what he wrote in a post about a year ago:

The ISO testing uses critically-controlled lighting. Once you know the results of the that, nothing else needs to be shot under 'critically' controlled lighting (i.e. to a fraction of an EV) , it just needs to be shot to a controlled output brightness. Then, by the very definition of ISO, it doesn't matter whether the light level is slightly different and you compensate by changing the shutter speed - if the grey patches are rendered at the correct output brightness, you've got the same exposure.

He's right that changing shutter speed is a valid way to adjust for different light levels, but he's not right that "if the grey patches are rendered at the correct output brightness, you've got the same exposure."

He is if you realize, first, that DPReview first measures the camera's ISO settings using the SOS method, takes the ISO test samples using their findings as the basis. I understand this is complex, but if you work your way backwards, you'll see that they have collected the required information that allows them to precisely control exposure based on how those grey patches are rendered in the OOC JPEGs.

Output brightness is a function of processing, not exposure.

It's a function of processing a RAW that was generated from an exposure. Which, by the way, is what the SOS method for determining a camera's exposure index, defined in ISO 12232:2006, is based on, i.e., it is based on measuring the output brightness of an 18% grey patch in a camera's OOC JPEG.

Also, I haven't read a post by any DPR staff member that says they adjust output brightness in ACR. Check this post from Richard Butler, in which he dismisses my suggestion that they do so.

Andy WestlakeOur RAW conversions are brightness-matched to the camera JPEGs so the comparison makes visual sense between cameras. This is Adobe's policy anyway - after all, it's what most users expect - so we don't generally have to adjust the output sliders from default positions, but occasionally we'll tweak the brightness control to get the best possible match.

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
Post (hide subjects) Posted by
tko
tko
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow