what is your take on DxOMark tests usefulness?

How do they translate to real life, to you? I'm not questioning their credibility which I have no reason to doubt, but rather if they influence your lens purchase decisions and to what extent and if you feel the results are compatible with your own experiences.

Here is an example, Zeiss Planar 85mm f/1.4 vs Nikkor 85mm f /1.4G on a nikon d810 body:

https://www.dxomark.com/Lenses/Comp...KKOR-85mm-f14G-on-Nikon-D810__338_963_388_963

The nikkor has a score of 42, the zeiss gets 31. It is a big gap. I happen to own both (selling the nikkor currently) and I don't see a difference except in caracter and subjective atributes like bokeh. I'm still going to conduct more tests, i'm curious on this one.

The "angry photographer" has a youtube review praising the zeiss over the nikkor 85mm f1.4g and he usually (in my opinion) is very accurate and credible. In all the forums and stuff I've read (even before deciding getting the zeiss) people seem to say both are optically great and talk about the obvious manual vs auto focus and subjective atributes.

So I'm kind of curious of how such a big score gap doesn't reflect into real life perceptions.

I ignore them. I used to study them sometime ago - but then I realised that I could find no relevance whatsoever to what I do with the gear. I couldn't translate it into a practical application for me.

I have the feeling that they examine minute differences between equipment, differences that are mostly undetectable in practical terms. And they then blow those minute differences up on an arbitrary scale, making them look significant.

A couple of proverbs about statistics come to mind...

Regards, Mike

--
Wait and see...
I hardly ever speak for anybody but myself. In the cases where I do mean to speak generally the statements are likely to be marked as such.

What DxOMark does is obsessing over stats, and it's not very accurate since every lens copy is different. Photozone will say lens A is sharpest at f/8.0, DxOMark says the same lens on the same camera is sharpest at f/5.6, no lens is the same.

If you're a person obsessed with numbers, with af micro-adjustment, someone who returns 10 copies of the same lens until they get the best one. Then by all means, visit DxOMark. But if you think photography is meant to be fun and enjoyable, don't.

lb77 said:

How do they translate to real life, to you? I'm not questioning their credibility which I have no reason to doubt, but rather if they influence your lens purchase decisions and to what extent and if you feel the results are compatible with your own experiences.

Here is an example, Zeiss Planar 85mm f/1.4 vs Nikkor 85mm f /1.4G on a nikon d810 body:

https://www.dxomark.com/Lenses/Comp...KKOR-85mm-f14G-on-Nikon-D810__338_963_388_963

The nikkor has a score of 42, the zeiss gets 31. It is a big gap. I happen to own both (selling the nikkor currently) and I don't see a difference except in caracter and subjective atributes like bokeh. I'm still going to conduct more tests, i'm curious on this one.

The "angry photographer" has a youtube review praising the zeiss over the nikkor 85mm f1.4g and he usually (in my opinion) is very accurate and credible. In all the forums and stuff I've read (even before deciding getting the zeiss) people seem to say both are optically great and talk about the obvious manual vs auto focus and subjective atributes.

So I'm kind of curious of how such a big score gap doesn't reflect into real life perceptions.

Click to expand...

Dxomark does very scientific testing.... things you may not see with the naked eye. In this case we are talking sharpness. To know definitively if that test is accurate in regards to the lenses you own you should also do a scientific test. Take the same shot with both lenses. Use a tripod. Make sure you shoot with mirror up to remove camera shake. Load them into lightroom and look at both zoomed in. Go past 1:1.

Angry photographer knows his gear but have you seen how crappy his pictures are? I have no clue how so many people decided to follow him on youtube. Do some digging and you can see all his terrible photos on flickr.

Actually, they don't do very scientific testing. However, their techniques are very useful, effective, and appropriate to both their purpose (raw development), and the field of photography.

lb77 said:

How do they translate to real life, to you? I'm not questioning their credibility which I have no reason to doubt, but rather if they influence your lens purchase decisions and to what extent and if you feel the results are compatible with your own experiences.

Here is an example, Zeiss Planar 85mm f/1.4 vs Nikkor 85mm f /1.4G on a nikon d810 body:

https://www.dxomark.com/Lenses/Comp...KKOR-85mm-f14G-on-Nikon-D810__338_963_388_963

The nikkor has a score of 42, the zeiss gets 31. It is a big gap. I happen to own both (selling the nikkor currently) and I don't see a difference except in caracter and subjective atributes like bokeh. I'm still going to conduct more tests, i'm curious on this one.

The "angry photographer" has a youtube review praising the zeiss over the nikkor 85mm f1.4g and he usually (in my opinion) is very accurate and credible. In all the forums and stuff I've read (even before deciding getting the zeiss) people seem to say both are optically great and talk about the obvious manual vs auto focus and subjective atributes.

So I'm kind of curious of how such a big score gap doesn't reflect into real life perceptions.

Click to expand...

I think DxO Mark is some of the most useful lens testing information out there, because they do what other websites don't - they compare a vast number of lens/body combinations, including older lenses on the newest bodies and the newest lenses on older bodies. In most cases this allows an "apples to apples" comparison of a given pair of lenses on the SAME camera (assuming the same lens mount, of course), which eliminates camera differences from lens comparisons.

As for the "scores," that's not the useful part of DxO Mark; look to the details. If it's sharpness you're comparing, look to the "sharpness" tab and compare "profiles," at all the different aperture and focal length (for zooms) settings. The big differentiator for the lenses you're talking about in their testing seems to be the wide open/larger apertures. If you're using these lenses stopped down (f5.6 or higher f#s), there is little to differentiate them.

Bottom line though is the same as any lens test; "tests" are typically done with a single copy of a lens, which means the results do not account for sample variation. "Tests" are typically done at relatively short focus distances, and therefore may not be indicative of the relative lens quality under different shooting conditions (i.e., at/near "infinity"). "Tests" are "testing" flat-field performance, which penalizes field curvature that may have little effect on (or in some cases, even improve) the results in real world shooting, since the real world is three-dimensional. So basically, take any "lens test" with a grain, or perhaps even a handful, of salt.

Martin.au said:

Actually, they don't do very scientific testing. However, their techniques are very useful, effective, and appropriate to both their purpose (raw development), and the field of photography.

Click to expand...

How so?

I ignore the scores, I mostly just look about sharpness across the frame, light fall off and distortion charts.

Mike CH said:

I ignore them.

Click to expand...

+1

AwesomeIan said:

Dxomark does very scientific testing....

Click to expand...

No, they don't. Far from it, actually.

Scientific testing would - by necessity - include full disclosure of methods as well as repeatability by other testers.

Both areas where DxO are lacking; the first severely and the second, well...

Do not forget - they also sell a product.

AwesomeIan said:

things you may not see with the naked eye. In this case we are talking sharpness.

Click to expand...

Photography is about producing a human visible result. So if you can't see it with your naked eye - what do you want it for?

AwesomeIan said:

To know definitively if that test is accurate in regards to the lenses you own you should also do a scientific test. Take the same shot with both lenses. Use a tripod. Make sure you shoot with mirror up to remove camera shake. Load them into lightroom and look at both zoomed in. Go past 1:1.

Click to expand...

Hmmm...
Regards, Mike

"How do they translate to real life, to you? ... if they influence your lens purchase decisions .... you feel the results are compatible with your own experiences."

At one time I spent quite of bit of time looking at DXO scores and details.

But I found too many instances where there was a big difference between what DXO said versus what my experience. Others have noted the same. So now I ignore DXO. But I do pay a lot of attention to comments on the forums here. I don't ever buy a lens now without having done a search in the appropriate forum here on dpreview.

Besides that. real life users bring up lots of things DXO doesn't test for. Like glare, AF speed, noise, build quality, smoothness of adjustment, stiffness or looseness of adjustment rings, effectiveness of image stabilization and other various things.

One does have to be aware that different people have different expectations and needs from lenses. But if you read comments from a number of people or post questions you can learn a lot.

Mike CH said:

AwesomeIan said:

Dxomark does very scientific testing....

Click to expand...

No, they don't. Far from it, actually.

Scientific testing would - by necessity - include full disclosure of methods as well as repeatability by other testers.

Both areas where DxO are lacking; the first severely and the second, well...

Do not forget - they also sell a product.

Mike CH said:

things you may not see with the naked eye. In this case we are talking sharpness.

Click to expand...

Photography is about producing a human visible result. So if you can't see it with your naked eye - what do you want it for?

Mike CH said:

To know definitively if that test is accurate in regards to the lenses you own you should also do a scientific test. Take the same shot with both lenses. Use a tripod. Make sure you shoot with mirror up to remove camera shake. Load them into lightroom and look at both zoomed in. Go past 1:1.

Click to expand...

Hmmm...

Regards, Mike

--
Wait and see...
I hardly ever speak for anybody but myself. In the cases where I do mean to speak generally the statements are likely to be marked as such.

Click to expand...

IMO dxomark is the most scientific measurement that is currently available. For the most part I have to agree with most of the findings they publish (i.e. sharpness). You can read all about how they do this here:

https://www.dxomark.com/About/In-depth-measurements/Measurements/Sharpness

"naked eye" test is subjective. The size of the screen is a big variable. Say you want to produce a large print of a photograph. In this case looking at a picute on a 17" monitor wouldn't provide the same view as a large print without looking 1:1 to make sure the noise and sharpness are acceptable.

--
Blog
http://iangeglia.wix.com/mysite

Since I'm new to photography, and still buying gear and trying to figure out what I want, the DxO tests are useful to me.

What else is there out there, that's like DxO?

I have about decided that at the very dawn of digital cameras, that every DSLR with every DSLR lens was an acceptably useful piece of gear.

During the last decade or so, if you'd like better than average, and acceptable, cameras and lenses, then DxO is the only source for unbiased, scientific, testing.

There's another way to do it, as well.

You get what you pay for. If you'd like better, bring more money.

But one of the reasons you get what you pay for, is that the makers know that DxO is out there and they can't get away for long selling cameras and lenses that don't rack up higher DxO scores.

That's my ill informed, half guess and by golly, take on the subject.

They are what they are.

Anyone with any real experience will use their eyes in the end. Crazy stuff, doing that with a visual medium and all.....

AwesomeIan said:

IMO dxomark is the most scientific measurement that is currently available.

Click to expand...

Maybe in your opinion - but that doesn't that what they do has little to do with any kind of science.

If the methods and the calculations aren't fully disclosed, it isn't science - it's snake oil.

If you want to see something more sciency, look at R. Cicalas work over at LensRentals.

That does not mean that what they do can't be interesting or of use to some people.
Regards, Mike

Tuloom said:

Crazy stuff, doing that with a visual medium and all.....

Click to expand...

Yeah

Regards, Mike

AwesomeIan said:

IMO dxomark is the most scientific measurement that is currently available. For the most part I have to agree with most of the findings they publish (i.e. sharpness). You can read all about how they do this here:

https://www.dxomark.com/About/In-depth-measurements/Measurements/Sharpness

"naked eye" test is subjective.

Click to expand...

... vs. their handwaving definition of acutance and the mysterious P-mpix?

Mike CH said:

AwesomeIan said:

IMO dxomark is the most scientific measurement that is currently available.

Click to expand...

Maybe in your opinion - but that doesn't that what they do has little to do with any kind of science.

If the methods and the calculations aren't fully disclosed, it isn't science - it's snake oil.

If you want to see something more sciency, look at R. Cicalas work over at LensRentals.

Click to expand...

Roger does great work, but hardly does a comprehensive testing (in terms of which lenses), thereby limiting its usefulness somewhat.

Mike CH said:

That does not mean that what they do can't be interesting or of use to some people.

Regards, Mike

--
Wait and see...
I hardly ever speak for anybody but myself. In the cases where I do mean to speak generally the statements are likely to be marked as such.

Click to expand...

PK24X36NOW said:

Mike CH said:

AwesomeIan said:

IMO dxomark is the most scientific measurement that is currently available.

Click to expand...

Maybe in your opinion - but that doesn't that what they do has little to do with any kind of science.

If the methods and the calculations aren't fully disclosed, it isn't science - it's snake oil.

If you want to see something more sciency, look at R. Cicalas work over at LensRentals.

Click to expand...

Roger does great work, but hardly does a comprehensive testing (in terms of which lenses), thereby limiting its usefulness somewhat.

Click to expand...

True enough, but I'll take his 'mutterings' about a lens any day over just about anybody else.
Regards, Mike

PK24X36NOW said:

Mike CH said:

AwesomeIan said:

IMO dxomark is the most scientific measurement that is currently available.

Click to expand...

Maybe in your opinion - but that doesn't that what they do has little to do with any kind of science.

If the methods and the calculations aren't fully disclosed, it isn't science - it's snake oil.

If you want to see something more sciency, look at R. Cicalas work over at LensRentals.

Click to expand...

Roger does great work, but hardly does a comprehensive testing (in terms of which lenses), thereby limiting its usefulness somewhat.

Click to expand...

We've done pretty much every full-frame lens available for EF, F, or FE mount -- the total is up to 204 models.

PK24X36NOW said:

PK24X36NOW said:

That does not mean that what they do can't be interesting or of use to some people.

Regards, Mike

--
Wait and see...
I hardly ever speak for anybody but myself. In the cases where I do mean to speak generally the statements are likely to be marked as such.

Click to expand...

Click to expand...