Olympus OM-D (EM-5) comparison samples are now.. (continued)

Continued to answer Andy Westlake:

Raist3d wrote:
Andy Westlake wrote:
A> How does the lighting intensity affect the high ISO noise?

R> If the camera compensates for the exposure with different shutter speed it sure can! A capture of the same final intensity under a longer exposure accumulates more noise.

A> Can you show me a practical example, using shutter speeds similar to those we're using here? Reality is that while this can theoretically be an issue for exposures longer than a second or two, we're nowhere near that in these tests. It can also be effectively compensated by dark-frame subtraction, AKA long exposure noise reduction, anyway.

I don't agree with you we need to get to longer than a second or two to see the difference particularly at high ISO and bad light, but I will agree with your point that at very fast shutter speeds like the ones shown it should not matter. Asking the general question, I would say it can certainly affect it.

Now- I did miss that you had mentioned elsewhere that for this test, the light intensity between the EM5 test and many other cameras was changed. That I was missing so now it makes more sense. But that said, it would be good if the test mentioned this somewhere, because it sure seems a bit odd when you see the exif.

This all said, going by the assumption that the manufacturers are conforming all good and peachy with the ISO to me is a bit of a big assumption, when in the past there have been cameras measured by this very website that certainly did not. So I still say it would be a good idea to always verify that.

R> This gets compounded in the real world where high intensity light usually is more full spectrum light vs lower light which is uially candle or tungsten or sodium light but at least this latter part shouldn't be affected in the studio.

A> Arguments about the white balance of artifical light sources are irrelevent in this case (and it's white balance, rather than intensity or long shutter speeds, that has most effect). But if you want to see what the E-M5 can do at high ISO under low-intensity artificial light, we've previously published samples to give an idea:
http://www.dpreview.com/news/2012/03/09/olympus-omd-em5-iso-series

This is a different scene, granted, but it does have nice big swathes of dark background to help assess the shadow noise and potential banding etc.

I saw that test, and while I like that you guys set it up with tungsten light which is tough on sensors, I did see a bit high of shutter speeds suggesting more light than what I find is a low light condition in the real world (at least indoors, street night life and wedding reception at night). Not too bad on that end though, I have seen farse worse there.

I do agree it helps see banding.

--

Raist3d/Ricardo (Photographer, software dev.)- "You are taking life too seriously if it bugs you in some way that a guy quotes himself in the .sig quote" - Ricardo

Member said:

Raist3d wrote:

Now- I did miss that you had mentioned elsewhere that for this test, the light intensity between the EM5 test and many other cameras was changed.

Click to expand...

If DPReview changes the light intensity between cameras when testing in studio then what is the purpose of the studio shot comparison widget?
--
Panagiotis

Raist3d said:

Continued to answer Andy Westlake:

Raist3d wrote:
Andy Westlake wrote:

A> How does the lighting intensity affect the high ISO noise?

R> If the camera compensates for the exposure with different shutter speed it sure can! A capture of the same final intensity under a longer exposure accumulates more noise.

A> Can you show me a practical example, using shutter speeds similar to those we're using here? Reality is that while this can theoretically be an issue for exposures longer than a second or two, we're nowhere near that in these tests. It can also be effectively compensated by dark-frame subtraction, AKA long exposure noise reduction, anyway.

I don't agree with you we need to get to longer than a second or two to see the difference particularly at high ISO and bad light, but I will agree with your point that at very fast shutter speeds like the ones shown it should not matter. Asking the general question, I would say it can certainly affect it.

Now- I did miss that you had mentioned elsewhere that for this test, the light intensity between the EM5 test and many other cameras was changed. That I was missing so now it makes more sense. But that said, it would be good if the test mentioned this somewhere, because it sure seems a bit odd when you see the exif.

This all said, going by the assumption that the manufacturers are conforming all good and peachy with the ISO to me is a bit of a big assumption, when in the past there have been cameras measured by this very website that certainly did not. So I still say it would be a good idea to always verify that.

R> This gets compounded in the real world where high intensity light usually is more full spectrum light vs lower light which is uially candle or tungsten or sodium light but at least this latter part shouldn't be affected in the studio.

A> Arguments about the white balance of artifical light sources are irrelevent in this case (and it's white balance, rather than intensity or long shutter speeds, that has most effect). But if you want to see what the E-M5 can do at high ISO under low-intensity artificial light, we've previously published samples to give an idea:
http://www.dpreview.com/news/2012/03/09/olympus-omd-em5-iso-series

This is a different scene, granted, but it does have nice big swathes of dark background to help assess the shadow noise and potential banding etc.

I saw that test, and while I like that you guys set it up with tungsten light which is tough on sensors, I did see a bit high of shutter speeds suggesting more light than what I find is a low light condition in the real world (at least indoors, street night life and wedding reception at night). Not too bad on that end though, I have seen farse worse there.

I do agree it helps see banding.

Click to expand...

Is all this for the black version or the silver ???

Danny.
...........................
m4/3 macro
http://www.macrophotos.com/g2macro

m4/3 feathered flying gadgets
http://www.macrophotos.com/avian/avian.html

Worry about the image that comes out of the box, rather than the box itself.

panos_m said:

panos_m said:

Raist3d wrote:

Now- I did miss that you had mentioned elsewhere that for this test, the light intensity between the EM5 test and many other cameras was changed.

Click to expand...

If DPReview changes the light intensity between cameras when testing in studio then what is the purpose of the studio shot comparison widget?
--
Panagiotis

Click to expand...

--
Simplicity is the ultimate sophistication.

God is the tangential point between zero and infinity.

Imagination is more important than knowledge.

God always take the simplest way.

• http://aleo-photo.org

• http://blog.aleo-photo.org

panos_m said:

panos_m said:

Raist3d wrote:

Now- I did miss that you had mentioned elsewhere that for this test, the light intensity between the EM5 test and many other cameras was changed.

Click to expand...

If DPReview changes the light intensity between cameras when testing in studio then what is the purpose of the studio shot comparison widget?

Click to expand...

While I agree the intensity should not change, Andy's point was that even though it did change and caused the 1 stop difference I found, at the shutter speeds the cameras where shooting (at least say ISO 3200- 1/1600 or 1/3200 for ISO 6400 I think it was), it is a very high speed which would make the extra noise due to lower light irrelevant.

I think what he answered is reasonable even though I still think they should use the same and not assume all manufacturers will comply with the ISO standard number- as they already apparently found out with the EM5 as it seems to run "1/3rd ev behind."

This is why I rather see DXo's scores, since they normalize sensor sensitivity and I also trust more a company that writes and ships real world professional raw converters in measuring things like these.

panos_m said:

--
Panagiotis

Click to expand...

--

Raist3d/Ricardo (Photographer, software dev.)- "You are taking life too seriously if it bugs you in some way that a guy quotes himself in the .sig quote" - Ricardo

--

Raist3d/Ricardo (Photographer, software dev.)- "You are taking life too seriously if it bugs you in some way that a guy quotes himself in the .sig quote" - Ricardo

panos_m said:

If DPReview changes the light intensity between cameras when testing in studio then what is the purpose of the studio shot comparison widget?

Click to expand...

You seem to have entirely misunderstood what I've written. Let's try again.

We don't randomly change the light levels between cameras on a whim. But we have changed the lighting setup recently to one that gives us more control, and lets us dim the lights when testing cameras with very high ISO settings. For the purpose of the test, this simply doesn't matter - it doesn't have to be done at a fixed light level because, by a happy accident of design, cameras can compensate for differing light levels by using different shutter speeds.

The purpose of the studio comparison widget is to compare how cameras behave in terms of noise and noise reduction at different ISOs. To this end, the exposure is tightly controlled between cameras, based essentially on the SOS definition of ISO 12232:2006. This is the definition used used by camera manufacturers. DxOMark uses a different measure which may, or may not, give the same answer, depending on the camera.

For the purposes of our testing ISO defines the relationship betwen exposure (as a function of light level, shutter speed and aperture) and JPEG image brightness. If two cameras have the same SOS ISO calibration, and output JPEGs of the same brightness, they must have received the same exposure, regardless of the actual light level. Any differences in light levels can be compensated by changing the shutter speed. This is how cameras work. Our tests are based on setting the exposures so that specific reference gray patches are white-balanced and rendered at the same brighness from every camera (as far as possible).

Shutter speeds have absolutely minimal impact on noise. If you shoot the same scene at ISO100 1/125sec F16 and ISO100 1/2000sec F4 you really won't see any difference in the noise. To me this is so self evident from everyday shooting I'm actually quite surprised to find myself typing it here. Then again, I did spend several years of my life testing lenses, so I guess have a bit more experience of this than most.

DxOMark's ISO measurements are fine in their own way - they provide a logical framework for DxOmark's own particular method of RAW data comparison. They just bear no relation to our tests based on SOS ISO - the two simply don't intersect. You can't say our ISO tests are wrong because DxO's are different, and you can't reinterpret our data based on DxOMark's measurements. The two testing regimes just use entirely different methodologies.

--
Andy Westlake
dpreview.com

Andy,

you are using multi segment exposure for all your tests. While a camera might be calibrated, the multi segment exposure algorithm probably isn't. You'd have to switch to spot metering to get more reliable results regarding the relationship between ISO, F-stop and exposure time.

To me, your test setup has a rather pragmatic and practical approach, which i actually prefer over the theoretical one provided by dxo. At least when comparing test shots here i generally get a good idea of what a camera will produce when i#m shooting with it.

Sabine

snoack said:

you are using multi segment exposure for all your tests.

Click to expand...

Actually, we're not. We use manual exposure to set specific grey patches to the same brightness. The camera's metering is not involved in this process at all.

--
Andy Westlake
dpreview.com

nzmacro said:

Is all this for the black version or the silver ???

Click to expand...

Actually, we're using the top-secret pink version that has a reduced-height 'hump'

--
Andy Westlake
dpreview.com

Raist3d said:

I don't agree with you we need to get to longer than a second or two to see the difference particularly at high ISO and bad light, but I will agree with your point that at very fast shutter speeds like the ones shown it should not matter. Asking the general question, I would say it can certainly affect it.

Click to expand...

So, if I'm reading this correctly, you agree that it doesn't matter if we adjust shutter speeds to compensate for any changes in the intensity of the lighting in this test.

--
Andy Westlake
dpreview.com

Hi Andy!

Thanks a lot for giving us two parts os the upcoming review in advance.

You really have to have a planty of patience with some people here, they really can't stand being wrong. It has nothing to do with your test protocol or how the test was conducted but with the results not conforming to their picture of how the things must be. Their "truth" has to preveil even by childish means like present questioning of your testing procedure and legitimity of if. We will se at least ten more threads with the same goal with EM-5 qualities being questioned and deminished just so those persons can make a statement and preserve their delusional picture of being some kind of gurus while in reality not being anything but an "average Joe" at best.

Thanks once again for your valuable inputs and I am looking for a full review.

Regards
Haris

Andy Westlake said:

Shutter speeds have absolutely minimal impact on noise. If you shoot the same scene at ISO100 1/125sec F16 and ISO100 1/2000sec F4 you really won't see any difference in the noise. To me this is so self evident from everyday shooting I'm actually quite surprised to find myself typing it here. Then again, I did spend several years of my life testing lenses, so I guess have a bit more experience of this than most.

Click to expand...

So if I understand it correctly the only two variables that you try to keep constant between cameras tested in your studio setup is the rendered middle gray value of the OOC jpeg at standard settings (factory jpeg settings out of the box) and the manufacturer stated iso.

Or stated a little differently you try to align the exposures between cameras based on the rendered middle gray value of the OOC jpeg at standard settings (factory jpeg settings out of the box) at the manufacturer stated iso.

Am I right?

Andrew Westlake said:

Andy Westlake wrote:

panos_m said:

If DPReview changes the light intensity between cameras when testing in studio then what is the purpose of the studio shot comparison widget?

Click to expand...

You seem to have entirely misunderstood what I've written. Let's try again.

We don't randomly change the light levels between cameras on a whim. But we have changed the lighting setup recently to one that gives us more control, and lets us dim the lights when testing cameras with very high ISO settings. For the purpose of the test, this simply doesn't matter - it doesn't have to be done at a fixed light level because, by a happy accident of design, cameras can compensate for differing light levels by using different shutter speeds.

The purpose of the studio comparison widget is to compare how cameras behave in terms of noise and noise reduction at different ISOs. To this end, the exposure is tightly controlled between cameras, based essentially on the SOS definition of ISO 12232:2006. This is the definition used used by camera manufacturers. DxOMark uses a different measure which may, or may not, give the same answer, depending on the camera.

For the purposes of our testing ISO defines the relationship betwen exposure (as a function of light level, shutter speed and aperture) and JPEG image brightness. If two cameras have the same SOS ISO calibration, and output JPEGs of the same brightness, they must have received the same exposure, regardless of the actual light level. Any differences in light levels can be compensated by changing the shutter speed. This is how cameras work. Our tests are based on setting the exposures so that specific reference gray patches are white-balanced and rendered at the same brighness from every camera (as far as possible).

Shutter speeds have absolutely minimal impact on noise. If you shoot the same scene at ISO100 1/125sec F16 and ISO100 1/2000sec F4 you really won't see any difference in the noise. To me this is so self evident from everyday shooting I'm actually quite surprised to find myself typing it here. Then again, I did spend several years of my life testing lenses, so I guess have a bit more experience of this than most.

DxOMark's ISO measurements are fine in their own way - they provide a logical framework for DxOmark's own particular method of RAW data comparison. They just bear no relation to our tests based on SOS ISO - the two simply don't intersect. You can't say our ISO tests are wrong because DxO's are different, and you can't reinterpret our data based on DxOMark's measurements. The two testing regimes just use entirely different methodologies.

--
Andy Westlake
dpreview.com

Click to expand...

--
Panagiotis

... or not as the case may be...

Andy Westlake said:

cameras can compensate for differing light levels by using different shutter speeds.

Click to expand...

Ah. But therein lies the issue. Cameras might be able to, but often, photographers cannot!

Here is a hypothetical (albeit very common) scenario which I'm sure you're aware of, but I need to write it down to illustrate the point. You are photographing some lowish light sport event. Your 300/2.8 lens is at its widest because you want as much light as possible and you also want as much isolation as you can get. You've decided you want to freeze some fast action and so 500th of a second is the figure your experience tells you will just about let you get away with your 'freeze'. So your aperture AND shutter speed are fixed. There is nowhere to go except change the ISO so this is how you shoot.

Now, after reviewing the shots, you decide you might want to buy a new camera because you want better noise performance. You now need to review some identical side by side comparisons from different cameras, so really these need to be taken under the same conditions, which means the same source light intensity, the same aperture and the same shutter speed. Only then does side by side camera ISO comparison make real sense if you're trying to decide on which camera to buy based on its noise v ISO performance.

Andrew Westlake said:

The purpose of the studio comparison widget is to compare how cameras behave in terms of noise and noise reduction at different ISOs. To this end, the exposure is tightly controlled between cameras, based essentially on the SOS definition of ISO 12232:2006. This is the definition used used by camera manufacturers. DxOMark uses a different measure which may, or may not, give the same answer, depending on the camera.

Click to expand...

Well, your comparison test shots do indeed provide an excellent and invaluable resource when looking at how the manufacturers settings change the image at different settings. However at the moment, it seems to me as if yourselves and DxO need to be viewed as complimentary testing procedures, rather than mutually exclusive or competing ones because the manufacturers themselves are implementing what most photographers think of as ISO in different ways.

Your extensive samples provide a range of test shots which show how what the output will look like when you set the camera to various settings such as different ISO and shutter speeds as calibrated by the manufacturer, however one then needs to cross reference these to DxO figures to be able to work out which of your test images need to be selected, if one is comparing images across camera models (and then it becomes problematic if the difference in manufacturers quoted ISO values don't fall onto a clearly defined one stop boundary - which they invariably don't!)

What many people are asking for is a truly fixed set of parameters. Fixed lighting. Fixed aperture. Fixed shutter speed. The same for all models. RAW shooters can then take these files and play as many equivalence games as they want to see how the files come out when processed. But you need as level a playing field as possible to start with.

At the moment, it seems you can't look at an ISO6400 file from one camera, compare it with an ISO6400 image from a different model, and simply by looking at the test results, work out how the cameras are actually going to perform in real life when you have to take shutter speed into account, which for many people, is quite often!

Cheers
Ga.

If you want to create a tool for camera's comparison at same high iso, light HAVE to be exactly the same

In a setup with exactly the same lightining, if a camera require double shutter timing at the same iso as other, there's no need to compare those camera's at same iso

" Is all this for the black version or the silver ??? "

Danny.

Why does this feel sooooo appropriate ?

Vjim

I've been following the discussion over the two threads and your replies have cleared up a lot of things - thank you for taking the time Andy!

But I too am a little curious - is there no way to test how different cameras respond to a different lighting situations? I'd be very interested in a series of tests that show how a camera chooses the shutter speed and ISO for a given aperture. One that consistently chooses a slower shutter speed or higher ISO for a given lighting situation would be more problematic in real life automated shooting.
--
Hompage: http://gakuranman.com
Haikyo: http://gakuranman.com/category/haikyo-ruins/

Andy Westlake said:

nzmacro said:

Is all this for the black version or the silver ???

Click to expand...

Actually, we're using the top-secret pink version that has a reduced-height 'hump'

Click to expand...

Awwww, I like the pink cameras with larger humps...in all the right places.

--
Rick Krejci
http://www.ricksastro.com

Big Ga said:

Andy Westlake said:

cameras can compensate for differing light levels by using different shutter speeds.

Click to expand...

Ah. But therein lies the issue. Cameras might be able to, but often, photographers cannot!

Here is a hypothetical (albeit very common) scenario which I'm sure you're aware of, but I need to write it down to illustrate the point. You are photographing some lowish light sport event. Your 300/2.8 lens is at its widest because you want as much light as possible and you also want as much isolation as you can get. You've decided you want to freeze some fast action and so 500th of a second is the figure your experience tells you will just about let you get away with your 'freeze'. So your aperture AND shutter speed are fixed. There is nowhere to go except change the ISO so this is how you shoot.

Now, after reviewing the shots, you decide you might want to buy a new camera because you want better noise performance. You now need to review some identical side by side comparisons from different cameras, so really these need to be taken under the same conditions, which means the same source light intensity, the same aperture and the same shutter speed. Only then does side by side camera ISO comparison make real sense if you're trying to decide on which camera to buy based on its noise v ISO performance.

Click to expand...

Let's expand on this...the next time you go, you find they decided to change their lighting system and it's now metered to be twice as bright, so you choose to up your shutter by a stop to compensate and still use the same ISO. This should yield exactly the same noise, shouldn't it?

I think that was Andy's point. Even though their lighting may change slightly, that shouldn't affect the results at all.

--
Rick Krejci
http://www.ricksastro.com

Andy Westlake said:

snoack said:

you are using multi segment exposure for all your tests.

Click to expand...

Actually, we're not. We use manual exposure to set specific grey patches to the same brightness. The camera's metering is not involved in this process at all.

Click to expand...

Let's take this to the extreme...say Camera A requires 4 stops longer shutter speed to attain this brightness level under the same lighting as camera B when these cameras are set to the same ISO. And let's assume noise performance is 100% related to how much light hits the sensor and that A and B have the same sensor technology, the only difference is how the ISO is rated.

In your comparison tool, will camera A be given a 4 stop advantage due to the longer shutter speed? I know you address the ISO sensitivity elsewhere in a review, so it would come out, but I'm just talking about the comparison tool only.

--
Rick Krejci
http://www.ricksastro.com