Why DxOmark claims are rubbish

Started Mar 31, 2012 | Discussions
Shop cameras & lenses ▾
Nonacnokin Forum Member • Posts: 54
Why DxOmark claims are rubbish

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

Nikon D800
If you believe there are incorrect tags, please send us this post using our feedback form.
Blade93012 Regular Member • Posts: 181
Re: Why DxOmark claims are rubbish

Excerpt from "Hammerforum.com" ( http://www.lensrentals.com/blog/2012/03/hammerforum-com )

Whacker: H2O just released their latest Impact Index and the Stilleto rated 92.745, the highest impact per oblique force applied they’ve measured (except for jackhammers).

BigBanger: I don’t trust anything H20 measures, they’re numbers are all crap and don’t reflect real-world hammering. Besides, they down score everything to be equivalent to a 6 ounce jewelers hammer, which makes no sense. I prefer a ‘hand’s on’ review. Maxwell’s reviews over at SilverHammer.com really let you know how a hammer does in the real world.

SRHelicity New Member • Posts: 9
Re: Why DxOmark claims are rubbish

I don't think there's anything wrong with DxO's measurements, assuming you don't think they're fraudulent and/or completely made-up. The results are what they are.

The problem is that people don't use the results properly, and/or some people generalize the results to infer things that the tests don't test. The scores do NOT indicate which camera is "best". Heck, what in the world does "best" even mean? Best for me is going to be much different than what's best for a portrait photographer and what's best for a sports photographer who uses his equipment to make a living. The scores having nothing to do with fps, AF performance, in-camera features (HDR, bracketing, etc.), or other very important factors that one should considering when figuring out which camera is "best" for him or her. The entire package can affect the final output; the quality of a photo can be significantly affected by things beyond sensor performance.

Millions of amazing photographs have been taken in the past using equipment much less sophisticated (with greater noise, less resolution, lower DR, etc.) than what people are arguing over today (D800 vs. 5D3). Having good equipment can help increase your odds of capturing some great photos, but the actual camera is less important than what's behind the camera.

Randplaty Contributing Member • Posts: 504
Re: Why DxOmark claims are rubbish

I think that their measurements are accurate but I agree with you that they don't necessarily correspond to reality and that they don't necessarily equate with a better sensor. For example, I trust their dynamic range score but the score doesn't necessarily tell me the quality of the DR. Plus I typically need more highlight headroom than shadow detail but DR is just one number and is not broken down.

Also their high ISO cutoff is pretty arbitrary. Most users use much higher ISO figures than are indicated by DxOmark. Thus if a camera had poor ISO noise at 1600, but then the noise leveled off quickly and it had really low noise for 3200 and above compared to other sensors, DxOmark would not reflect that because they don't publish a graph. They only publish a single number. But high ISO noise cannot be represented by a single number.
--
http://www.orangeturtlephotography.com

mike_2008 Veteran Member • Posts: 5,635
Re: Why DxOmark claims are rubbish

In my experience there is a direct correlation between the perceived veracity of dxomark results in a dpr gear forum and the current ranking of of that gear on dxomark.

-- hide signature --

There are none so blind as those who will not see

 mike_2008's gear list:mike_2008's gear list
Canon EOS 5D Sony SLT-A55 Sony SLT-A77 Canon EF 50mm f/1.8 II Canon EF 17-40mm f/4.0L USM +5 more
Nonacnokin OP Forum Member • Posts: 54
Re: Why DxOmark claims are rubbish

Please note I did not say anything about their measurements. I am saying their CLAIMS are rubbish i.e. that DxOmark is a sensor score.

iwannabesedated Senior Member • Posts: 1,447
Re: Why DxOmark claims are rubbish

I take it some DxOMarks of interest to this forum are about to be released? -iwbs

 iwannabesedated's gear list:iwannabesedated's gear list
Nikon D800E Nikon D810 Nikon D300S Nikon D500
BadBeta Regular Member • Posts: 454
Re: Why DxOmark claims are rubbish

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

They have been publishing their figures for years, not just lately. Self-promoting? Where?

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

That is actually fine with me. The RAW is what I get out, thus I'm fine with them measuring exactly that. Measuring what I get is more useful than measuring some sensor characteristic which then may or may not propagate into the RAW file - which is what I use and care about. I have no intention of ripping out the sensors and hooking it up to some backyard electronics, so I don't really care what it does. I care about what I get.

Now, that said, I do want a raw file to be exactly that, without any NR or other processing in the camera.

I don't know the DXO measuring process enough to comment on that, but it does seem to have some offbeat consequences. Either way I just treat it like an indication.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There is an idiot out there for any crazy though. Resolution alone will certainly not make anything equal to medium format. Larger sensors catch more photons, get more information, and that advantage will always be there. If medium format use similar sensor technology they should always be ahead of 35mm. I think we actually agree on something!

 BadBeta's gear list:BadBeta's gear list
Ricoh GR Digital II Ricoh GR Digital III Canon EOS 5D Canon EOS 5D Mark II Canon EF 16-35mm f/2.8L II USM +7 more
webfrasse Veteran Member • Posts: 3,858
Re: Why DxOmark claims are rubbish

Is there any other way of measuring the output from the sensor that makes sense then and how would I apply that info aside from using the sensor in a camera...build one myself?

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

-- hide signature --

Mikael

Tinu_ch Contributing Member • Posts: 660
Re: Why DxOmark claims are rubbish

BadBeta wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

They have been publishing their figures for years, not just lately. Self-promoting? Where?

DxO makes a living by selling software. DxO Optics Pro, DxO FilmPack, DxO Analyzer (used by many reviewing sites ...) and a swarm of "embedded imaging solutions" for Smartphones.

They have to analyze RAW files to keep their RAW converter up to date anyway. The same with lenses to enable their software to reduce CA, distortion and many other things. Publishing their (basic) findings on the net for free is good publicity. Most photographers are like me and can't interpret correctly the data as is. That's why DxO translates it into "ratings" which get misinterpreted as well.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

That is actually fine with me. The RAW is what I get out, thus I'm fine with them measuring exactly that.

Here I am completely with you. I am interested in the image file I can get at.
But some questions remain:

How closely does the measurement relate to the image characteristics you are looking for?

How much can "cooking" the RAW improve certain characteristics while compromizing others that are not measured?

How important are the different image characteristics for you and which compromize do you prefer?

Now, that said, I do want a raw file to be exactly that, without any NR or other processing in the camera.

There is nothing like an untainted RAW file. As far as I know, with CMOS there is a lot going on on chip already. And there may be potential for further wizardry which would be less efficient or even impossible to do after the read-out. We are going to see more of that and we will have to live with the advatages and trade-offs the engineers decide on.

Tinu

-- hide signature --

If the text above reads like real English, it must be a quotation
Some of my pictures: http://www.pbase.com/tinu

 Tinu_ch's gear list:Tinu_ch's gear list
Sony RX100 Canon EOS 5D Mark II Canon EOS 7D Canon EOS-1D Mark IV Canon EF 500mm f/4.0L IS USM +7 more
ukat123 Contributing Member • Posts: 761
Re: Why DxOmark claims are rubbish

Again this sense of insecurity on the Canon forum. Canon's sensor technology cannot compete with the R&D association Nikon/Sony at this time. Get over it. It does not mean you cannot take pictures with your Canon gear though. So, enjoy what you have and stop pretending.

boels069 Contributing Member • Posts: 527
Re: Why DxOmark claims are rubbish

webfrasse wrote:

Is there any other way of measuring the output from the sensor that makes sense then and how would I apply that info aside from using the sensor in a camera...build one myself?

Images of a (complex) grey and/or color scale?

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite +1 more
Bernie Ess Veteran Member • Posts: 7,004
The real problem with DxOmark is this:

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Well, they just do their stuff, and some real experts here (as opposed to the "sofa experts") think that Dxo offers - overall - the most valid and systematic data set for a given camera.

The problem that many users here (on Canon SLR talk) seem to have is obviously that Canon regulary comes out second or third in terms of sensor performance. They do not get the same noise/ shadow performance out of the sensors as Sony/ Nikon does.

Would the result be that Canon comes out first, DxO would be considered the No.1 source of reliable data on the web in this forum, I am absolutely sure of that.

So the reasons for your difficulty with DxO are not that complicated, after all. I feel neither identified to Nikon nor to Canon, I have stuff by both platforms which sometimes is more expensive, but all in all very relaxing.

And, by the way, I consider the 5dIII to be grossly overpriced and to be a deception given the long time of development. After 4 years, this seems essentially a standstill. I keep my 5dII till Canon comes up with something more convincing. If they manage to do so....

Bernie
--

'All the technique in the world doesn’t compensate for the inability to notice.' (Elliot Erwitt)

boels069 Contributing Member • Posts: 527
Re: Why DxOmark claims are rubbish

ukat123 wrote:

Again this sense of insecurity on the Canon forum. Canon's sensor technology cannot compete with the R&D association Nikon/Sony at this time. Get over it. It does not mean you cannot take pictures with your Canon gear though. So, enjoy what you have and stop pretending.

For me it has nothing to do with insecurity.

I'm just curious and "need" to know the differences and where in the process from sensel to file they occur.

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite +1 more
dominikov Senior Member • Posts: 1,099
Re: Why DxOmark claims are rubbish

Nonacnokin wrote:

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There are two Phase One cameras, 65mp and 80mp, ranked ahead of it on Portrait Color Depth.

Color Depth or Color Sensitivity indicates both the breadth of the color spectrum and the degree of fine separation between colors that a camera can distinguish. The higher the color sensitivity, the more color nuances can be distinguished, which is of particular importance for portrait and studio photographers.

The D800 is ahead on DR by almost a stop but then what makes you think Sony isn't producing superior sensors to Kodak , a company in decline which has recently filed for bankruptcy?

As for the Low-light ISO category, well that's a no brainer. The Kodak CCD sensors as expected get beaten convincingly. Those two Phase Ones are ranked 18th and 29th, respectively.

So there's your higher score for the D800.

 dominikov's gear list:dominikov's gear list
Nikon D700 Nikon D800E Nikon AF-S Nikkor 70-200mm f/2.8G ED VR Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF Nikkor 50mm f/1.4D +4 more
DotCom Editor Veteran Member • Posts: 6,813
No one has ever asked

When selling large 40x60-inch museum-quality archival pigment prints to my clients, would you believe not one has ever asked, "Say, what was the DxOmark for the sensor you used to make this photo?"

Lab numbers don't matter; it's real-world results that do.

 DotCom Editor's gear list:DotCom Editor's gear list
Epson Stylus Pro 3880 +2 more
marike6 Senior Member • Posts: 5,071
Re: No one has ever asked

I'm wondering if the OP would be trying to discredit DxOMark if his camera of choice were on top. I'm guessing no.

 marike6's gear list:marike6's gear list
Nikon Coolpix P330 Panasonic Lumix DMC-GH2 Nikon D800 Fujifilm X-E1 Panasonic Lumix G 20mm F1.7 ASPH +7 more
boels069 Contributing Member • Posts: 527
Re: No one has ever asked

marike6 wrote:

I'm wondering if the OP would be trying to discredit DxOMark if his camera of choice were on top. I'm guessing no.

Read the fine print:
http://www.dxomark.com/index.php/About/Sensor-scores

There's nothing wrong with the scores (after putting them in perspective).

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite +1 more
Causio Regular Member • Posts: 234
Raw mark

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There's one flaw in your reasoning, that is: perhaps it would be more appropriate to call it "Raw file mark" instead of "sensor", but how do you think photographers get their ""measurable"" IQ, by plugging cables directly to the sensor or from the raw files?

Luc

-- hide signature --
 Causio's gear list:Causio's gear list
Sony RX100 II Canon EOS 5D Mark III Canon EF 100mm f/2.0 USM Canon EF 70-200mm f/2.8L IS USM Canon TS-E 45mm f/2.8 +16 more
Mikael Risedal
Mikael Risedal Veteran Member • Posts: 4,275
Re: Why DxOmark claims are rubbish, WHY tell me WHY

As long as the canons sensors data is not in the top? If Canon can manage the readout = low read out noise and improve DR, QE then you would probably think that DXO was great test site , but what you and others denial is that Canon has a old sensor line and Canon is passed a long time ago by Sony, Pansonic, Omnivision, Aptina the list could be very long.

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

Keyboard shortcuts:
FForum MMy threads