Nikon Z 24-200 f/4-6.3 compared to Canon RF 70-200 f/4

Started 8 months ago | Discussions thread
kenw Veteran Member • Posts: 6,926
Re: an egregious example

peripheralfocus wrote:

kenw wrote:

Your faith is misplaced. Try this one:

And compare to this:

Same lens on two cameras using the same 16MP sensor. And yet DXO gives a sharpness figure of merit of 13MP on one and 5MP on the other.

I didn't know about this example. That's a puzzler; if those are really their measured results, I'm amazed they didn't stop and say to themselves, "something is wrong here". If it's just a misprint or mistranscription of data, then I can see how they might not have noticed it.

The best I could deduce was the issue is that they attempt to provide results for bodies they never actually tested on.  For instance for the above lens the claim it has been tested on 19 different bodies which is of course not true.  It appears they test on one body and then attempt to normalize the results to other bodies.  They never document this method.  Worse, they never tell you which body they actually did the real test on.  Worse still quiet obviously over time which body they actually test a given system's lenses on changes but they never tell us.  And as clearly demonstrated here their attempt at normalizing results across different bodies does not work properly.

So it leaves us with a huge database which is completely unreliable for evaluating not just lenses between systems but even lenses within systems!  For other sites they actually tell us what body they did their test on and if we see they've changed the body we immediately know to be careful about comparing results.  DxO on the other hand hides this information making it impossible to reliably use their test data.

So I fully suspect when they actually test a lens they do it carefully and get what is probably quality data.  Unfortunately it is impossible to tell exactly which data set is that quality data and which is their ham-fisted normalized data.

To that end I find their testing useful for giving an overview of lens performance such as whether it has soft corners at particular focal lengths and apertures.  And like with the other sites I can usually take out some of the uncertainty in their numbers by using the center resolution results from small apertures.  For example, "oh look this lens at F/4 has corners that are worse than diffraction at F/16."

But sadly it means their data is actually less useful for comparing lenses than almost any of the other sites.  We have no idea what mistakes in normalizing the data and presenting it on the website have been made and no way to verify we are making valid comparisons between lenses.  Likely the differences are usually fairly small but again it is just a black box so never can tell.

And yes this has been pointed out to them and they don't respond or fix it.  Not surprising as their test data site is not revenue generating in any way for them so they aren't likely to spend much effort making corrections to something that is really just a polite favor to the community to begin with for which they hope to get some advertising.

For 15+ years, I did camera testing for publication, and, of course, there were times when I got results that seemed really unlikely. Never published them without going back and validating the tests in multiple ways. Almost always, it turned out that I had made a mistake or I had a defective copy of the equipment I was testing.

I only test cameras and lenses for myself, but I do test a lot of other complicated systems for my work, and what you say completely matches my experience.  In fact I usually refuse to consider any unusual test data someone shows me unless they can repeat it and get the same result.  I have to do this with myself when testing lenses - it is so easy to make a mistake in the test and actually even repeat the mistake.  If something appears funny I usually don't just repeat the same test but use an entirely different test setup to try to reproduce.

-- hide signature --

Ken W
See profile for equipment list

 kenw's gear list:kenw's gear list
Panasonic Lumix DMC-GM1 Nikon Z7 Panasonic Leica DG Summilux 15mm F1.7 ASPH Nikon Z 14-30mm F4 Nikon Z 24-200mm F4-6.3 VR +42 more
Post (hide subjects) Posted by
(unknown member)
(unknown member)
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow