Why the religious belief in DxO ratings?

Frasier Krane · Oct 3, 2015

OzRay said:
Henry Richardson said:

OzRay said:

There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements.

Click to expand...

Can you post a few links where people did that? I don't recall seeing it. Thanks.

--
Henry Richardson
http://www.bakubo.com

Click to expand...

Well, no, because my comments come from a collection of references to DxOMark scores. Unfortunately, I can't assist with your recollections. However, the recent thread about the 30mm is somewhat illustrative of my point.

--
Thoughts, Musings, Ideas and Images from South Gippsland
http://australianimage.com.au/wordpress/

Are you referring to my post in that thread? If you are, then you're mistaken in your conclusions.

I was simply responding to a post in that thread that stated that DxO didn't like the lens much. So, in response, I posted a pic demonstrating the overall ranking of this lens compared to other Panasonic lenses.

Would it surprise you to know that I personally think the overall scoring on DxO is next to useless? But I do look at the acutance measurement of lenses.

The reason I posted that was simply to demonstrate that there was no reason to claim that DxO didn't like the lens. I almost never look at the final scores or the overall rankings, as these seem to have very little basis in reality. From my experience, the final scores don't correlate well with real world results Frankly, I have no clue how they calculate them, and I've seen lenses that do better on virtually all individual tests have a final score lower than lenses that did worse on the individual tests.

But the acutance charts do seem to have some correlation with real world results, so I look mainly at those.

jim stirling · Oct 3, 2015

OzRay said:
There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements. As soon as the rubber hits the road, the results become meaningless.

Much, much, better are actual results from users with lenses/cameras in real world use, all the better if they are actual photography enthusiasts and not journos that are writing for some photography mag or the like. Again with the car analogy, motoring journos are not the ideal source for information on cars that might suit your needs.

Do people really consider their lens/camera selections on DxO results?

--
Thoughts, Musings, Ideas and Images from South Gippsland
http://australianimage.com.au/wordpress/

I think DXO data is excellent the headline score not so much. The main group of people who have problems with it are almost always users of smaller sensor cameras who seem surprised when a larger sensor of the same tech generation does better than their favourite camera.

Larger sensors are less demanding on lenses so the combination of a decent FF lens on a current gen sensor will easily outperform even the best mFT lens+ camera combinations. This does not mean that mFT lenses are poorer quite the contrary most are very good and several are excellent. For me where DXO comes into its own is not in the tedious cross format comparisons a venerable favourite topic here in the mFT forum, but rather comparing lenses within a format.

Bottom line is there are spectacular images being made with all cameras regardless of sensor size. If your shooting involves specific niche subjects such as high speed sports in low light or you need ultra-high ISO , or demand ever higher MP counts or you want better DOF control then there are cameras aplenty to choose from. Personally I want the low ISO image quality of the Phase one IQ180 in a body the size of an mFT camera and at the same price please

. There is always something bigger and better out there the thing is do you really need it?

jim stirling · Oct 3, 2015

Martin.au said:
Isn't the modern version basically the RX10?

Or does the quirky design achieve something different/better?

Well the R1 was an APS camera a modern APS sensor in a body sized like the R1 with the same quality of lens could be a real knock-out. Though as an FZ1000 owner the 1" Sony sensor is surprisingly good. I liked a lot about the R1 though its AF performance certainly wasn't one of them

LMNCT · Oct 3, 2015

When evaluating the M4/3 cameras, there should be no comparison to FF or APS. Each group should be judged within its own. Comparisons are not useful and pointless. The M3/4 sensor group does what I want it to do and some of them do a bit better than others. Simple enough.

jim stirling · Oct 3, 2015

LMNCT said:
When evaluating the M4/3 cameras, there should be no comparison to FF or APS. Each group should be judged within its own. Comparisons are not useful and pointless. The M3/4 sensor group does what I want it to do and some of them do a bit better than others. Simple enough.

I agree the best use of DXO is to compare lenses within a system if for example you are looking for a 50mm lens for a D810 it is irrelevant how a 25mm does on mFT or a 35mm on APS

MrFresno · Oct 3, 2015

LMNCT said:
When evaluating the M4/3 cameras, there should be no comparison to FF or APS. Each group should be judged within its own. Comparisons are not useful and pointless. The M3/4 sensor group does what I want it to do and some of them do a bit better than others. Simple enough.

Really? What if I am shopping for a new system and I have it narrowed down to the E-M1, X-T1, and A7ii? Would a comparison be useless and pointless? How else would I decide between these cameras if not to compare them?

cainn24 · Oct 3, 2015

John King said:
It appears to me that you have extended not the slightest respect towards others with different views from yours in either this thread, or any other. I hope that I have not been personally vindictive towards anyone in this thread. Perhaps you could ask yourself the same question?

I'm really not sure what you are talking about here.

If you feel that I have made a habit of being discourteous and vindictive feel free to try to establish such a pattern of behaviour and bring it to the attention of our moderators.

John King said:
Generally, you get what you give in life, but I, for one try not to treat these things as a zero sum game, infinitely preferring the outcomes attendant in non-zero sum games. However, I see that DPR is still a combat zone.

The only points of contention here are the claims you have made about the veracity of the results that DxO Labs publish, and the supposed degree of obfuscation they engage in with respect to how they arrive at them, which you have still not seen fit to even begin to evidence.

In my opinion it is not only permissible to challenge such claims, there is something of an obligation connected to doing so given how large the audience tends to be in these forums. Misinformation simply doesn't serve anybody.

John King said:
BTW, I did read the entirety of that link. DxOMark state lots of 'Mom and apple pie' stuff, but there are a lot of important and contestable items that are either missing or glossed over. Just like any good snake oil salesman of times gone by ... The training I have had in statistics and other areas helps me to look for what's not said as well as to evaluate critically what is said.

Please provide an example of what is missing and I'll see if I can find the information elsewhere on their website as they have several pages devoted to their testing methodology and their DxO Analyzer software. A few other camera review sites are using the package as well so some clarification may also be found externally.

Moreover given that the most comprehensive information would likely come as a result of acquiring and using the package itself, which you presumably haven't done, how can you persist with your implied characterization of it all as "snake oil"?

Sure, I don't have it either. But I can and do fall back on this one simple truth: the results that DxO Labs publish correlate with what can be learned by properly examining a broad range of image samples produced by multiple camera models, and you can certainly do that yourself.

John King said:
Let's leave it at that, shall we? Vituperation and personal attacks merely make this place more unpleasant than it normally is.

I don't really see much in the way of personal attacks here. Certainly not any sort of abuse. What I am actually doing is repeatedly challenging you to evidence your claims, which I think is perfectly reasonable.

Cani · Oct 3, 2015

OzRay said:
There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements. As soon as the rubber hits the road, the results become meaningless.

Much, much, better are actual results from users with lenses/cameras in real world use, all the better if they are actual photography enthusiasts and not journos that are writing for some photography mag or the like. Again with the car analogy, motoring journos are not the ideal source for information on cars that might suit your needs.

Do people really consider their lens/camera selections on DxO results?

peppermonkey · Oct 3, 2015

OzRay said:
OzRay said:

The fact that people quote DxOMark is sufficient in itself that it's used as some form of justification.

Click to expand...

... Ever quote dxo?

I mean, if the mere act of linking one review from dxo means you are using dxo as the be all and end all of reviews, then quoting any review would be the same thing. So if we want to not use a review as our bible, we can't quote ANY reviews...

Unless you truly believe that absolutely everything in dxo is absolute trash, people quoting a part of some particular review can have meaningful significance depending on context and meaning of whatever the OP is saying.

Poopster · Oct 3, 2015

OzRay said:
There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements. As soon as the rubber hits the road, the results become meaningless.

Much, much, better are actual results from users with lenses/cameras in real world use, all the better if they are actual photography enthusiasts and not journos that are writing for some photography mag or the like. Again with the car analogy, motoring journos are not the ideal source for information on cars that might suit your needs.

Do people really consider their lens/camera selections on DxO results?

Eric Nepean · Oct 3, 2015

I have not observed any religious belief in DxO ratings.

Day Hiker · Oct 3, 2015

Like dpReview in general, DxO is entertainment for gearheads, first and foremost. There may be useful information there if you know what you are looking to find. In the end, though, it is entertainment at the product level. Anyone serious about real photography will not be spending much time here or over at DxO.

Full disclosure: I'm probably closer to being a gearhead than a real photographer. ;-)

Jim Pilcher
Summit County, Colorado, USA
Life is good in the woods

jkrumm · Oct 3, 2015

I don't mind when a review site like DPR applies an opinion score to a review, but when a company like DXO tests for various results and then does the same thing with an indecipherable "score" it irritates me. But I still use them. I like their graphs of sensor tests. And they seem pretty consistent with their testing results. Their lens testing is near useless if you look at their scores once again since it is sensor resolution based as much as anything. I prefer Lenstip for lens testing, and of course Lens Rentals (I wish they would do some M43 lens testing).

rueil · Oct 3, 2015

OzRay said:
Do people really consider their lens/camera selections on DxO results?

Moti · Oct 3, 2015

zxaar said:
OzRay said:

zxaar said:

Just to complain that a camera sensor testing site does not take into account human element is pretty stupid.

Click to expand...

Actually, a camera testing site (one that tests lenses and cameras) that doesn't take into account the human element is pretty stupid.

Click to expand...

So smarty pants do you have any idea how to do it?

Or you are just stupid words and no essence.

Yes. Take the camera out, take a series of real world photographs and use your eyes to evaluate the results.. This is the best way a photography gear should be tested and evaluated.

There is no better way.

Moti

obsolescence · Oct 3, 2015

Four things to consider: 1) sample variation, which AFAIK DXOmark cannot factor in; 2) camera body used for testing, incl. IBIS; 3) software corrections which can compensate for lens deficiencies; 4) build quality to maintain performance over time and through harsh conditions.

Lenses no longer can be evaluated in isolation based on the performance of one sample on one camera body.

As Roger Cicala of LensRentals has explained, there is often considerable variation in samples, and some brands have better QC than others. It can be worth the extra cost to buy a well reputed brand based on this factor alone.

Manufacturers increasingly design lenses assuming software correction in order to reduce size and weight, which are factors that significantly affect a photographer's handling and productivity. A highly responsive IBIS system also makes a huge difference in real world performance, especially with longer lenses and low light.

Build quality and ruggedness enable lenses to hold up over time and through heavy use, which may be important because life's a beach.

I buy lenses for the long haul. I am especially fond of the older SHG Olympus ZD lenses for Four Thirds that are optically uncompromised designs. They don't have the compactness and superior functionality of the newer ones designed for mirrorless cameras, so I have to accept that compromise, but they're fine for the kind of work I do. AFAIK these lenses are have not been reviewed by DXOmark, but if they were, I think they would stand at or near the top of the scale.

Newer lens designs can only prove their quality and durability through actual use, not through lab testing alone.

Fri13 · Oct 3, 2015

How many times it is required to show how a DXO statics are false? Same objective on different camera with a same sensor results different lens resolution.... Or a same camera with a different objective results to different ISO performance etc?

A DXO gives a huge difference but a real world sampling gives results without difference or results other way around no matter how many units would be tested?

The DXO is totally just a statics that work in the world of DXO but not even between their own results.

bikerhiker · Oct 3, 2015

OzRay said:
There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements. As soon as the rubber hits the road, the results become meaningless.

Much, much, better are actual results from users with lenses/cameras in real world use, all the better if they are actual photography enthusiasts and not journos that are writing for some photography mag or the like. Again with the car analogy, motoring journos are not the ideal source for information on cars that might suit your needs.

Do people really consider their lens/camera selections on DxO results?

Osvaldo Cristo · Oct 3, 2015

OzRay said:
There appears to be an insatiable urge to use DxOMark ratings as some irrefutable bible of camera/lens capability/quality, when in reality the results are little more than the equivalent of fuel economy ratings for cars worked out from relevant country EPA test requirements. As soon as the rubber hits the road, the results become meaningless.

I agree you it is very low value as an absolute measurement but it is useful when comparing two different cameras/sensors/lenses when they maintain the same criterions to evaluate them.

Usually user´s opinions are strongly biased towards their own preferences at such point that sometimes it is completely ridiculous except for the guy that is stating his/her opinion.

No - they aren´t perfect but I think there is some value on their figures.

OzRay said:
Much, much, better are actual results from users with lenses/cameras in real world use, all the better if they are actual photography enthusiasts and not journos that are writing for some photography mag or the like. Again with the car analogy, motoring journos are not the ideal source for information on cars that might suit your needs.

Yes and no. You forgot to consider the personal skills and techniques from each one: there is a very huge variation. Some Photographers do not know even the basics on camera handling, some are so meticulous like a Scientist - most are between these extremes.

I knew an old color expert that got consistently extremely good image quality from the crap BarneyScan even when compared against venerable Hell and Crosfield mammoth scanners on the beginning of 1990s. Does his results turned BarneyScan better? Absolutely!

OzRay said:
Do people really consider their lens/camera selections on DxO results?

Yes, I do. Of course they aren´t the single information source I will look for my own decision.

Regards,

--
O.Cristo - An Amateur Photographer
Opinions of men are almost as various as their faces - so many men so many minds. B. Franklin

bclaff · Oct 3, 2015

John King said:
While I appreciate scientific investigation, my own main objection to DxOMark is that they keep their methodology secret.

This is a common complaint but DxOMark does in fact provide quite a bit of information.

See DxOMark image quality testing protocols for lenses and sensors.

Why the religious belief in DxO ratings?

Forum Enthusiast

Veteran Member

Veteran Member

Veteran Member

Veteran Member

Veteran Member

Veteran Member

Forum Enthusiast

Veteran Member

Well-known member

Veteran Member

Forum Pro

jkrumm

Guest

Member

Veteran Member

Leading Member

Veteran Member

Forum Enthusiast

Veteran Member

Forum Pro

Similar threads

Keyboard shortcuts