D5 wrongly reviewed by DXOmark

Started Jul 1, 2016 | Discussions
eno2 Regular Member • Posts: 223
D5 wrongly reviewed by DXOmark
11

Hi everyone,

I don't know what happened to DXO lately but I'm starting to see wrong evaluation of cameras. In this thread I will show the example of Nikon D5.

If you compare it on DXO to the new Canon 1Dx mk 2 (a fantastic camera nonetheless), you'll get the impression that the Canon is superior both in DR at lower ISO's (I have no doubt about that) and also regarding noise. Then, if you try comparing samples from both cameras at different ISO values(from any review sites), the truth is quite different, the Nikon is a little bit better than Canon until ISO 52.000 and a lot better beyond (probably with one ISO stop). The funny thing is DXO states that at ISO 200K, the D5 is one full stop worst than 1Dxmk2 and the reality shows exact the opposite.

2 ISO stops deviation from their measurement is quite severe I'd say and their reputation must answer to those false value.

Below I attached two examples taken from Dpreview and compared them with DXO measurements.

What are your thoughts about this?

Canon EOS-1D Nikon D5
If you believe there are incorrect tags, please send us this post using our feedback form.
Eric Calabros
Eric Calabros Regular Member • Posts: 186
Re: D5 wrongly reviewed by DXOmark
4

Its not first time that their measured number doesn't represent real life performance. in ISO 3200-12800 range, there is only one sensor that performs better than D5:  Sony A7s, with %60 less pixels.

hikerdoc Senior Member • Posts: 2,739
Re: D5 wrongly reviewed by DXOmark
5

Eric Calabros wrote:

Its not first time that their measured number doesn't represent real life performance.

That is what our colleagues in the Canon camp have claimed for years!

Many of them are now proponents of DXO results.

In real life performance either of these machines has capabilities far exceeding my real life performance.

D

 hikerdoc's gear list:hikerdoc's gear list
Apple iPhone 8
whumber
whumber Veteran Member • Posts: 3,307
Re: D5 wrongly reviewed by DXOmark
3

eno2 wrote:

Hi everyone,

I don't know what happened to DXO lately but I'm starting to see wrong evaluation of cameras. In this thread I will show the example of Nikon D5.

If you compare it on DXO to the new Canon 1Dx mk 2 (a fantastic camera nonetheless), you'll get the impression that the Canon is superior both in DR at lower ISO's (I have no doubt about that) and also regarding noise. Then, if you try comparing samples from both cameras at different ISO values(from any review sites), the truth is quite different, the Nikon is a little bit better than Canon until ISO 52.000 and a lot better beyond (probably with one ISO stop). The funny thing is DXO states that at ISO 200K, the D5 is one full stop worst than 1Dxmk2 and the reality shows exact the opposite.

2 ISO stops deviation from their measurement is quite severe I'd say and their reputation must answer to those false value.

Below I attached two examples taken from Dpreview and compared them with DXO measurements.

What are your thoughts about this?

Dynamic range really does a better job predicting high ISO performance than SNR since we tend to notice the noise more in shadow regions.  If you look at the DR plot for the D5 vs 1DXII then I t hink you'll find that the DxO measurements make more sense.  The single number scores, especially the ISO/sports score, are basically worthless.

 whumber's gear list:whumber's gear list
Fujifilm X-T1 Olympus OM-D E-M1X Olympus E-M1 III Olympus M.Zuiko Digital ED 60mm F2.8 Macro Olympus M.Zuiko Digital ED 75mm F1.8 +9 more
OP eno2 Regular Member • Posts: 223
Re: D5 wrongly reviewed by DXOmark
1

The dynamic range over ISO 3200 is greater on D5 compared to any other camera, your argument is flawed. The only DR disadvantage D5 has, is around lower ISO values.

whumber
whumber Veteran Member • Posts: 3,307
Re: D5 wrongly reviewed by DXOmark
7

eno2 wrote:

The dynamic range over ISO 3200 is greater on D5 compared to any other camera, your argument is flawed. The only DR disadvantage D5 has, is around lower ISO values.

Uh, that was exactly my point.  The D5 has greater dynamic range at higher ISO which corresponds to the high ISO images looking better.

 whumber's gear list:whumber's gear list
Fujifilm X-T1 Olympus OM-D E-M1X Olympus E-M1 III Olympus M.Zuiko Digital ED 60mm F2.8 Macro Olympus M.Zuiko Digital ED 75mm F1.8 +9 more
OP eno2 Regular Member • Posts: 223
Re: D5 wrongly reviewed by DXOmark
2

I understood something else, my bad.

EvilTed Junior Member • Posts: 31
Re: D5 wrongly reviewed by DXOmark
2

Actually, no.

Looking at the Photographic Dynamic Range of both D5 and A7s shows that the A7s is "marginally" better from ISO 102400 to 409600.

The D5 is noticeably better from ISO 2500.

The A7r II is the model that is better than the D5 but that is only between ISO 30,000 and 102400.

Having tested all three cameras in next to no light, I think that the D5 is the better of the 3.

 EvilTed's gear list:EvilTed's gear list
Nikon D5 Sony a9
rhlpetrus Forum Pro • Posts: 26,131
DR vs SNR and high ISO
1

whumber wrote:

eno2 wrote:

Hi everyone,

I don't know what happened to DXO lately but I'm starting to see wrong evaluation of cameras. In this thread I will show the example of Nikon D5.

If you compare it on DXO to the new Canon 1Dx mk 2 (a fantastic camera nonetheless), you'll get the impression that the Canon is superior both in DR at lower ISO's (I have no doubt about that) and also regarding noise. Then, if you try comparing samples from both cameras at different ISO values(from any review sites), the truth is quite different, the Nikon is a little bit better than Canon until ISO 52.000 and a lot better beyond (probably with one ISO stop). The funny thing is DXO states that at ISO 200K, the D5 is one full stop worst than 1Dxmk2 and the reality shows exact the opposite.

2 ISO stops deviation from their measurement is quite severe I'd say and their reputation must answer to those false value.

Below I attached two examples taken from Dpreview and compared them with DXO measurements.

What are your thoughts about this?

Dynamic range really does a better job predicting high ISO performance than SNR since we tend to notice the noise more in shadow regions. If you look at the DR plot for the D5 vs 1DXII then I t hink you'll find that the DxO measurements make more sense. The single number scores, especially the ISO/sports score, are basically worthless.

I have mentioned this before, but I'm still waiting for the technorati here to explain why DR is a better predictor than SNR for high ISO visual noise perception. The issue is that DxO, despite using both curves for establishing the high ISO score, in principle, ends up using the SNR 18% curve, which reaches the cutpoint earlier, and that's what determines their high ISO score.

Maybe, if one uses all SNR curves carefully, considering lower saturation curves as well, one might get a better predictor of high ISO performance.

Anyway, for the first time in DxO, the high ISO cutpoints for D500 and D5 are lower than one would expect from their visual results (and from their DR curves).

-- hide signature --

Renato.
http://www.flickr.com/photos/rhlpedrosa/
OnExposure member
http://www.onexposure.net/
Good shooting and good luck
(after Ed Murrow)

 rhlpetrus's gear list:rhlpetrus's gear list
Leica D-Lux (Typ 109) Nikon D7000 Nikon 1 V1 Nikon AF-S DX Nikkor 12-24mm f/4G ED-IF Nikon AF Nikkor 35mm f/2D +4 more
MoreorLess Veteran Member • Posts: 4,747
Re: D5 wrongly reviewed by DXOmark
1

whumber wrote:

eno2 wrote:

The dynamic range over ISO 3200 is greater on D5 compared to any other camera, your argument is flawed. The only DR disadvantage D5 has, is around lower ISO values.

Uh, that was exactly my point. The D5 has greater dynamic range at higher ISO which corresponds to the high ISO images looking better.

Pretty much, people are used to viewing standard image noise as the main factor in out of camera IQ with DR only being an issue when pushing images in post but when your getting to higher ISO values DR is lowering enough for it to damage basic output much more noticeably.

Looking at the DxO numbers I think they match the DPR studio scene very well up to around ISO 50,000. The D5 starts to show cleaner dark areas of the image(such as the grey background) at around ISO 3200 and is about 2/3rds of a stop better by ISO 6400. The OP is correct though that the numbers don't match well by ISO 50,000 where DxO show the Canon equalling the D5 in DR and pulling away in basic noise when its clear the D5 is performing better.

MrScrooge Contributing Member • Posts: 746
Re: D5 wrongly reviewed by DXOmark

DPReview samples are generated from ACR.

I think that DxO use Rawdigger or something similar to extract a final SNR figure from the original .raw using MATH (agh!) rather than from the final results after conversion as we see them in the samples above ^

But I'll just iterate right now that I'm not 100% certain on this and I'll happily stand corrected if someone has better info than I regarding DxO's testing methodology.

MoreorLess Veteran Member • Posts: 4,747
Re: D5 wrongly reviewed by DXOmark

MrScrooge wrote:

DPReview samples are generated from ACR.

I think that DxO use Rawdigger or something similar to extract a final SNR figure from the original .raw using MATH (agh!) rather than from the final results after conversion as we see them in the samples above ^

But I'll just iterate right now that I'm not 100% certain on this and I'll happily stand corrected if someone has better info than I regarding DxO's testing methodology.

Honestly my view on DxO most of the time is that there raw numbers do match output pretty well, the issue is more in the methodology of giving out headline scores which is questionable, most obviously only counting base ISO DR.

Iliah Borg Forum Pro • Posts: 27,694
Re: D5 wrongly reviewed by DXOmark
6

my view on DxO most of the time is that there raw numbers do match output pretty well

But sometimes they are wonderfully off: http://www.dxomark.com/Cameras/Nikon/D500---Measurements - press Full SNR (log) and you will see rather odd things.

-- hide signature --
crewchief227
crewchief227 Regular Member • Posts: 432
Re: D5 wrongly reviewed by DXOmark

whumber wrote:

eno2 wrote:

The dynamic range over ISO 3200 is greater on D5 compared to any other camera, your argument is flawed. The only DR disadvantage D5 has, is around lower ISO values.

Uh, that was exactly my point. The D5 has greater dynamic range at higher ISO which corresponds to the high ISO images looking better.

And that is why I have a D5 to compliment my D810.  The D810 is a low ISO DR monster and the D5 is for speed (where most likely you'll be pushing up the ISO) and low light at like wedding receptions.  If you compare the high ISO images to the D810 and the D810's low ISO DR you will be equally amazed at these given cameras intended sweet spot. LOL

-- hide signature --

Have you had your shot today?

 crewchief227's gear list:crewchief227's gear list
Nikon D810 Nikon D5 Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF-S Nikkor 50mm F1.4G Tokina AT-X Pro 100mm f/2.8 Macro +5 more
Marianne Oelund Veteran Member • Posts: 7,788
Re: D5 wrongly reviewed by DXOmark
9

MrScrooge wrote:

DPReview samples are generated from ACR.

I think that DxO use Rawdigger or something similar to extract a final SNR figure from the original .raw using MATH (agh!)

That's what I do, and DxO's numbers do not match mine.  It's as if they were testing a defective D5.

There is no way I can massage the data I get from my D3s, D4 and D5 to say that the D5 should have a lower sports ISO score, if that score is truly for the resolution-normalized case.

-- hide signature --

Source credit: Prov 2:6
- Marianne

Iliah Borg Forum Pro • Posts: 27,694
Re: D5 wrongly reviewed by DXOmark
1

There is no way I can massage the data I get from my D3s, D4 and D5 to say that the D5 should have a lower sports ISO score, if that score is truly for the resolution-normalized case.

Yes (though I tried with D4s, not D4).

-- hide signature --
(unknown member) Veteran Member • Posts: 9,762
Re: D5 wrongly reviewed by DXOmark

Eric Calabros wrote:

Its not first time that their measured number doesn't represent real life performance. in ISO 3200-12800 range, there is only one sensor that performs better than D5: Sony A7s, with %60 less pixels.

Wow that's not true.  There are several Sony sensors that perform better than the Sony sensor in the D5 when it comes to DR and noise.  D600, D750, D800, D800E, D810, D4, D4s, DF, and D3X.

whumber
whumber Veteran Member • Posts: 3,307
Re: D5 wrongly reviewed by DXOmark
2

Negative287 wrote:

Eric Calabros wrote:

Its not first time that their measured number doesn't represent real life performance. in ISO 3200-12800 range, there is only one sensor that performs better than D5: Sony A7s, with %60 less pixels.

Wow that's not true. There are several Sony sensors that perform better than the Sony sensor in the D5 when it comes to DR and noise. D600, D750, D800, D800E, D810, D4, D4s, DF, and D3X.

First, all reports indicate that the D5 sensor is not made by Sony.  Second, the D5 beats all of the cameras you've listed in the indicated ISO range.  So yes, "Wow that's not true".

 whumber's gear list:whumber's gear list
Fujifilm X-T1 Olympus OM-D E-M1X Olympus E-M1 III Olympus M.Zuiko Digital ED 60mm F2.8 Macro Olympus M.Zuiko Digital ED 75mm F1.8 +9 more
(unknown member) Veteran Member • Posts: 9,762
Re: D5 wrongly reviewed by DXOmark

whumber wrote:

Negative287 wrote:

Eric Calabros wrote:

Its not first time that their measured number doesn't represent real life performance. in ISO 3200-12800 range, there is only one sensor that performs better than D5: Sony A7s, with %60 less pixels.

Wow that's not true. There are several Sony sensors that perform better than the Sony sensor in the D5 when it comes to DR and noise. D600, D750, D800, D800E, D810, D4, D4s, DF, and D3X.

First, all reports indicate that the D5 sensor is not made by Sony. Second, the D5 beats all of the cameras you've listed in the indicated ISO range. So yes, "Wow that's not true".

No it doesn't, not even the D600! You're wrong about that and I'm not saying the D5 is bad, just not Nikons best in every area even if it is their highest ISO capable body.

If it isn't made by Sony then who else has a FF sensor this advanced?

whumber
whumber Veteran Member • Posts: 3,307
Re: D5 wrongly reviewed by DXOmark
4

Negative287 wrote:

whumber wrote:

Negative287 wrote:

Eric Calabros wrote:

Its not first time that their measured number doesn't represent real life performance. in ISO 3200-12800 range, there is only one sensor that performs better than D5: Sony A7s, with %60 less pixels.

Wow that's not true. There are several Sony sensors that perform better than the Sony sensor in the D5 when it comes to DR and noise. D600, D750, D800, D800E, D810, D4, D4s, DF, and D3X.

First, all reports indicate that the D5 sensor is not made by Sony. Second, the D5 beats all of the cameras you've listed in the indicated ISO range. So yes, "Wow that's not true".

No it doesn't, not even the D600! You're wrong about that and I'm not saying the D5 is bad, just not Nikons best in every area even if it is their highest ISO capable body.

Have you even bothered to look at the actual measurements or are you just going by the single number scores?

Well look there, turns out the D5 slaughters the D750 and D600 for high ISO.

If it isn't made by Sony then who else has a FF sensor this advanced?

Well Renesas manufactured the sensor in the D4/Df/D4s so that would be a good place to start.

 whumber's gear list:whumber's gear list
Fujifilm X-T1 Olympus OM-D E-M1X Olympus E-M1 III Olympus M.Zuiko Digital ED 60mm F2.8 Macro Olympus M.Zuiko Digital ED 75mm F1.8 +9 more
Keyboard shortcuts:
FForum MMy threads