Why DxOmark claims are rubbish

Started Mar 31, 2012 | Discussions
Nonacnokin
Forum MemberPosts: 54
Like?
Why DxOmark claims are rubbish
Mar 31, 2012

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

Nikon D800
If you believe there are incorrect tags, please send us this post using our feedback form.
Blade93012
Regular MemberPosts: 181
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

Excerpt from "Hammerforum.com" ( http://www.lensrentals.com/blog/2012/03/hammerforum-com )

Whacker: H2O just released their latest Impact Index and the Stilleto rated 92.745, the highest impact per oblique force applied they’ve measured (except for jackhammers).

BigBanger: I don’t trust anything H20 measures, they’re numbers are all crap and don’t reflect real-world hammering. Besides, they down score everything to be equivalent to a 6 ounce jewelers hammer, which makes no sense. I prefer a ‘hand’s on’ review. Maxwell’s reviews over at SilverHammer.com really let you know how a hammer does in the real world.

Reply   Reply with quote   Complain
SRHelicity
New MemberPosts: 9
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

I don't think there's anything wrong with DxO's measurements, assuming you don't think they're fraudulent and/or completely made-up. The results are what they are.

The problem is that people don't use the results properly, and/or some people generalize the results to infer things that the tests don't test. The scores do NOT indicate which camera is "best". Heck, what in the world does "best" even mean? Best for me is going to be much different than what's best for a portrait photographer and what's best for a sports photographer who uses his equipment to make a living. The scores having nothing to do with fps, AF performance, in-camera features (HDR, bracketing, etc.), or other very important factors that one should considering when figuring out which camera is "best" for him or her. The entire package can affect the final output; the quality of a photo can be significantly affected by things beyond sensor performance.

Millions of amazing photographs have been taken in the past using equipment much less sophisticated (with greater noise, less resolution, lower DR, etc.) than what people are arguing over today (D800 vs. 5D3). Having good equipment can help increase your odds of capturing some great photos, but the actual camera is less important than what's behind the camera.

Reply   Reply with quote   Complain
Randplaty
Contributing MemberPosts: 504
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

I think that their measurements are accurate but I agree with you that they don't necessarily correspond to reality and that they don't necessarily equate with a better sensor. For example, I trust their dynamic range score but the score doesn't necessarily tell me the quality of the DR. Plus I typically need more highlight headroom than shadow detail but DR is just one number and is not broken down.

Also their high ISO cutoff is pretty arbitrary. Most users use much higher ISO figures than are indicated by DxOmark. Thus if a camera had poor ISO noise at 1600, but then the noise leveled off quickly and it had really low noise for 3200 and above compared to other sensors, DxOmark would not reflect that because they don't publish a graph. They only publish a single number. But high ISO noise cannot be represented by a single number.
--
http://www.orangeturtlephotography.com

Reply   Reply with quote   Complain
mike_2008
Veteran MemberPosts: 5,635Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

In my experience there is a direct correlation between the perceived veracity of dxomark results in a dpr gear forum and the current ranking of of that gear on dxomark.

-- hide signature --

There are none so blind as those who will not see

 mike_2008's gear list:mike_2008's gear list
Canon EOS 5D Sony SLT-A55 Sony SLT-A77 Canon EF 50mm f/1.8 II Canon EF 17-40mm f/4.0L USM +5 more
Reply   Reply with quote   Complain
Nonacnokin
Forum MemberPosts: 54
Like?
Re: Why DxOmark claims are rubbish
In reply to Randplaty, Mar 31, 2012

Please note I did not say anything about their measurements. I am saying their CLAIMS are rubbish i.e. that DxOmark is a sensor score.

Reply   Reply with quote   Complain
iwannabesedated
Contributing MemberPosts: 969Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

I take it some DxOMarks of interest to this forum are about to be released? -iwbs

 iwannabesedated's gear list:iwannabesedated's gear list
Nikon D800 Nikon D800E Nikon D810 Sony RX100 II Nikon D200 +2 more
Reply   Reply with quote   Complain
BadBeta
Regular MemberPosts: 454Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

They have been publishing their figures for years, not just lately. Self-promoting? Where?

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

That is actually fine with me. The RAW is what I get out, thus I'm fine with them measuring exactly that. Measuring what I get is more useful than measuring some sensor characteristic which then may or may not propagate into the RAW file - which is what I use and care about. I have no intention of ripping out the sensors and hooking it up to some backyard electronics, so I don't really care what it does. I care about what I get.

Now, that said, I do want a raw file to be exactly that, without any NR or other processing in the camera.

I don't know the DXO measuring process enough to comment on that, but it does seem to have some offbeat consequences. Either way I just treat it like an indication.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There is an idiot out there for any crazy though. Resolution alone will certainly not make anything equal to medium format. Larger sensors catch more photons, get more information, and that advantage will always be there. If medium format use similar sensor technology they should always be ahead of 35mm. I think we actually agree on something!

 BadBeta's gear list:BadBeta's gear list
Ricoh GR Digital II Ricoh GR Digital III Canon EOS 5D Canon EOS 5D Mark II Canon EF 16-35mm f/2.8L II USM +7 more
Reply   Reply with quote   Complain
webfrasse
Veteran MemberPosts: 3,762
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

Is there any other way of measuring the output from the sensor that makes sense then and how would I apply that info aside from using the sensor in a camera...build one myself?

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

-- hide signature --

Mikael

Reply   Reply with quote   Complain
Tinu_ch
Contributing MemberPosts: 659Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to BadBeta, Mar 31, 2012

BadBeta wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

They have been publishing their figures for years, not just lately. Self-promoting? Where?

DxO makes a living by selling software. DxO Optics Pro, DxO FilmPack, DxO Analyzer (used by many reviewing sites ...) and a swarm of "embedded imaging solutions" for Smartphones.

They have to analyze RAW files to keep their RAW converter up to date anyway. The same with lenses to enable their software to reduce CA, distortion and many other things. Publishing their (basic) findings on the net for free is good publicity. Most photographers are like me and can't interpret correctly the data as is. That's why DxO translates it into "ratings" which get misinterpreted as well.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

That is actually fine with me. The RAW is what I get out, thus I'm fine with them measuring exactly that.

Here I am completely with you. I am interested in the image file I can get at.
But some questions remain:

How closely does the measurement relate to the image characteristics you are looking for?

How much can "cooking" the RAW improve certain characteristics while compromizing others that are not measured?

How important are the different image characteristics for you and which compromize do you prefer?

Now, that said, I do want a raw file to be exactly that, without any NR or other processing in the camera.

There is nothing like an untainted RAW file. As far as I know, with CMOS there is a lot going on on chip already. And there may be potential for further wizardry which would be less efficient or even impossible to do after the read-out. We are going to see more of that and we will have to live with the advatages and trade-offs the engineers decide on.

Tinu

-- hide signature --

If the text above reads like real English, it must be a quotation
Some of my pictures: http://www.pbase.com/tinu

 Tinu_ch's gear list:Tinu_ch's gear list
Sony RX100 Canon EOS 5D Mark II Canon EOS 7D Canon EOS-1D Mark IV Canon EF 500mm f/4.0L IS USM +7 more
Reply   Reply with quote   Complain
ukat123
Contributing MemberPosts: 761
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

Again this sense of insecurity on the Canon forum. Canon's sensor technology cannot compete with the R&D association Nikon/Sony at this time. Get over it. It does not mean you cannot take pictures with your Canon gear though. So, enjoy what you have and stop pretending.

Reply   Reply with quote   Complain
boels069
Regular MemberPosts: 462Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to webfrasse, Mar 31, 2012

webfrasse wrote:

Is there any other way of measuring the output from the sensor that makes sense then and how would I apply that info aside from using the sensor in a camera...build one myself?

Images of a (complex) grey and/or color scale?

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite
Reply   Reply with quote   Complain
Bernie Ess
Veteran MemberPosts: 6,920Gear list
Like?
The real problem with DxOmark is this:
In reply to Nonacnokin, Mar 31, 2012

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Well, they just do their stuff, and some real experts here (as opposed to the "sofa experts") think that Dxo offers - overall - the most valid and systematic data set for a given camera.

The problem that many users here (on Canon SLR talk) seem to have is obviously that Canon regulary comes out second or third in terms of sensor performance. They do not get the same noise/ shadow performance out of the sensors as Sony/ Nikon does.

Would the result be that Canon comes out first, DxO would be considered the No.1 source of reliable data on the web in this forum, I am absolutely sure of that.

So the reasons for your difficulty with DxO are not that complicated, after all. I feel neither identified to Nikon nor to Canon, I have stuff by both platforms which sometimes is more expensive, but all in all very relaxing.

And, by the way, I consider the 5dIII to be grossly overpriced and to be a deception given the long time of development. After 4 years, this seems essentially a standstill. I keep my 5dII till Canon comes up with something more convincing. If they manage to do so....

Bernie
--

'All the technique in the world doesn’t compensate for the inability to notice.' (Elliot Erwitt)

Reply   Reply with quote   Complain
boels069
Regular MemberPosts: 462Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to ukat123, Mar 31, 2012

ukat123 wrote:

Again this sense of insecurity on the Canon forum. Canon's sensor technology cannot compete with the R&D association Nikon/Sony at this time. Get over it. It does not mean you cannot take pictures with your Canon gear though. So, enjoy what you have and stop pretending.

For me it has nothing to do with insecurity.

I'm just curious and "need" to know the differences and where in the process from sensel to file they occur.

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite
Reply   Reply with quote   Complain
dominikov
Senior MemberPosts: 1,067Gear list
Like?
Re: Why DxOmark claims are rubbish
In reply to Nonacnokin, Mar 31, 2012

Nonacnokin wrote:

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There are two Phase One cameras, 65mp and 80mp, ranked ahead of it on Portrait Color Depth.

Color Depth or Color Sensitivity indicates both the breadth of the color spectrum and the degree of fine separation between colors that a camera can distinguish. The higher the color sensitivity, the more color nuances can be distinguished, which is of particular importance for portrait and studio photographers.

The D800 is ahead on DR by almost a stop but then what makes you think Sony isn't producing superior sensors to Kodak , a company in decline which has recently filed for bankruptcy?

As for the Low-light ISO category, well that's a no brainer. The Kodak CCD sensors as expected get beaten convincingly. Those two Phase Ones are ranked 18th and 29th, respectively.

So there's your higher score for the D800.

 dominikov's gear list:dominikov's gear list
Nikon D700 Nikon D800E Nikon AF-S Nikkor 70-200mm f/2.8G ED VR Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF Nikkor 50mm f/1.4D +4 more
Reply   Reply with quote   Complain
DotCom Editor
Veteran MemberPosts: 6,564Gear list
Like?
No one has ever asked
In reply to Nonacnokin, Mar 31, 2012

When selling large 40x60-inch museum-quality archival pigment prints to my clients, would you believe not one has ever asked, "Say, what was the DxOmark for the sensor you used to make this photo?"

Lab numbers don't matter; it's real-world results that do.

Reply   Reply with quote   Complain
marike6
Senior MemberPosts: 5,070Gear list
Like?
Re: No one has ever asked
In reply to DotCom Editor, Mar 31, 2012

I'm wondering if the OP would be trying to discredit DxOMark if his camera of choice were on top. I'm guessing no.

 marike6's gear list:marike6's gear list
Nikon Coolpix P330 Panasonic Lumix DMC-GH2 Nikon D800 Fujifilm X-E1 Panasonic Lumix G 20mm F1.7 ASPH +7 more
Reply   Reply with quote   Complain
boels069
Regular MemberPosts: 462Gear list
Like?
Re: No one has ever asked
In reply to marike6, Mar 31, 2012

marike6 wrote:

I'm wondering if the OP would be trying to discredit DxOMark if his camera of choice were on top. I'm guessing no.

Read the fine print:
http://www.dxomark.com/index.php/About/Sensor-scores

There's nothing wrong with the scores (after putting them in perspective).

 boels069's gear list:boels069's gear list
Canon EOS 5D Mark II Canon EF 100mm f/2.8L Macro IS USM Canon EF 24-105mm f/4L IS USM Canon EF 70-200mm f/4L IS USM DxO Optics Pro Elite
Reply   Reply with quote   Complain
Causio
Regular MemberPosts: 220Gear list
Like?
Raw mark
In reply to Nonacnokin, Mar 31, 2012

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

There's one flaw in your reasoning, that is: perhaps it would be more appropriate to call it "Raw file mark" instead of "sensor", but how do you think photographers get their ""measurable"" IQ, by plugging cables directly to the sensor or from the raw files?

Luc

-- hide signature --
 Causio's gear list:Causio's gear list
Sony RX100 II Canon EOS 5D Mark III Canon EF 100mm f/2.0 USM Canon EF 70-200mm f/2.8L IS USM Canon TS-E 45mm f/2.8 +16 more
Reply   Reply with quote   Complain
Mikael Risedal
Veteran MemberPosts: 3,162
Like?
Re: Why DxOmark claims are rubbish, WHY tell me WHY
In reply to Nonacnokin, Mar 31, 2012

As long as the canons sensors data is not in the top? If Canon can manage the readout = low read out noise and improve DR, QE then you would probably think that DXO was great test site , but what you and others denial is that Canon has a old sensor line and Canon is passed a long time ago by Sony, Pansonic, Omnivision, Aptina the list could be very long.

Nonacnokin wrote:

DxO has been doing a lot of self promotion lately claiming this sensor is better than that sensor and blah blah.

Let's get something straight. DxO is not measuring sensor performance directly. It is measuring raw files constructed by the manufacturer. It then measures some quantitative characteristics that purportedly are proxies for image quality, which are tenuous at best. It then provides an arbitrarily weighted overall score. That score has little to do with the sensor itself notwithstanding its promotion of it as a "sensor score".

Ultimately, it is a score on how a manufacturer constructs its raw file. Therefore, NR applied across the whole range of iso will produce a very good noise score. This is irrespective of data integrity or whether there are false detail. That is why there are wide ranging scores for the exact same sensor. Acuity, micro contrast, data integrity, detail, resolution are deemed irrelevant for IQ according to DxO.

Because of DxO's constant self promotion, idiots out there think the IQ from a D800 is better than IQ from Phase One and Hassie.

These are lemmings that follow the piper off the cliff blindly.

Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads