Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?

Started Aug 10, 2015 | Discussions
Tian6869 New Member • Posts: 22
Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?
5

So, I have always used "Sports (Low-Light ISO)" on DxOMark to get a basic idea on noise level in ISO in cameras.

I think they are very good.

However, recently DxOMark release DxO One and heavily advertisement on their sites (e.g pop up, side Ads and Headline news), so I took a look and found found out that

DxOMark rated their own camera sensor Sports (Low-Light ISO): 1657 ISO

1 inch sensor, 20.2 MP has Sports (Low-Light ISO): 1657 ISO!!!!

So, I was shocked.

I always thought the bigger the sensor, more light, the smaller the sensor, less light.

DxOMark One only has 1 inch sensor.

Looking at other 1 inch sensor"

Canon G7X Sports (Low-Light ISO): 556 ISO,

Panasonic LX100 Sports (Low-Light ISO): 553 ISO,

Panasonic FZ1000 Sports (Low-Light ISO): 517 ISO,

Sony RX100M3 Sports (Low-Light ISO): 495 ISO,

Nikon 1 J5 Sports (Low-Light ISO): 479 ISO,

Nikon 1 J4 Sports (Low-Light ISO): 426 ISO.

Note sure who made DxOMark One's sensor, but DxOMark rated all other major manufacturers way less than their own.

I mean, not by little, but almost 3 TIMES (e.g. 556 vs 1657)

Now, lets looks at M43

Olympus OM-D E-M5 Mark II Sports (Low-Light ISO): 896 ISO,

Olympus E-PL7 Sports (Low-Light ISO): 873 ISO,

Panasonic GX7 Sports (Low-Light ISO): 718 ISO,

Panasonic GM5 Sports (Low-Light ISO): 712 ISO,

Now, lets looks at APS-C

Nikon D5500 Sports (Low-Light ISO): 1438 ISO

Nikon D7200 Sports (Low-Light ISO): 1333 ISO

Sony A5100 Sports (Low-Light ISO): 1347 ISO

Canon EOS 7D Mark II Sports (Low-Light ISO): 1082 ISO

As you can see,

IF DxOMark DxO One with 1 inch sensor, 20.2 MP and has Sports (Low-Light ISO): 1657 ISO is correct.

Then, such sensor will beat Nikon D5500 APS-C sensor.

But, can it be true, that there is a 1 inch sensor that can beat the APC-C sensor which collect more light?

ChristianHass Veteran Member • Posts: 3,167
Re: Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?
8

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

-- hide signature --
 ChristianHass's gear list:ChristianHass's gear list
Sigma DP2 Merrill Sony RX100 VI Fujifilm XF10 Sony a7R II
GoneMirrorless Senior Member • Posts: 1,069
Wow, DxO is credibly misleading
3

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

I remember the days when they withheld their E-M5 ratings for about 6 months because it scored comparably to APSC cameras (sponsors).   They kept giving excuses like vacation and inquires to manufacturer while refusing publish their completed results.

And like you pointed out they usually refuse to test cameras with special settings like these.  You won't see any Sony multiframe low light mode results, and you won't see the E-M5ii his-res mode results.

-- hide signature --

Cameras with mirrors - Just when I thought I was out, they pull me back in

 GoneMirrorless's gear list:GoneMirrorless's gear list
Sony a77 II
OP Tian6869 New Member • Posts: 22
Re: Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?
2

That is exactly what I am trying to point out.

There are two ratings "DxO ONE SuperRAW Plus" and "DxO ONE"

DxO ONE SuperRAW Plus is when Multi-frame noise reduction mode is one.

However, Multi-frame noise reduction mode is not new. It can be done either within the cameras or in your computer.

For example, Sony RX100 also has such function (Multi-frame noise reduction mode) that it can take a fast burst of shots (4-5 shots) and composites them to produce a single image with less noise.

To be fair, Sensor rating should be based on the same method/testing (e.g. original single shot) as they are rating every other cameras.

That means if they want to use "superraw" mode and get 1657, then to be fair, at least they should rate ALL camera like that (or at least cameras which has such function buildin).

BUT they are not.

They ignored other cameras which also has such function, and not rating them under Multi-frame noise reduction mode, and only rated their own device with Multi-frame noise reduction mode.

I think if they rated RX100M3/RX100M4 with Multi-frame noise reduction mode, it will go much higher then DxO One.

BUT they are not doing that.

cainn24 Veteran Member • Posts: 4,892
No need to throw the baby out with the bathwater
11

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

J A C S
J A C S Forum Pro • Posts: 17,813
Re: No need to throw the baby out with the bathwater
1

cainn24 wrote:

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

I believe that their sensor measurements are trustworthy, as well even though they are a bit stubborn and do not reveal enough details. But once a cheater... How do we know where dishonesty stops?

va2k0r
va2k0r Regular Member • Posts: 454
Re: No need to throw the baby out with the bathwater
2

Super raw is, well, raw, while multi shot noise reductions is JPG only.
I would like to see that feature in every camera, thx.

 va2k0r's gear list:va2k0r's gear list
Nikon D3300 Sigma 17-50mm F2.8 EX DC OS HSM Nikon 85mm F1.8G Sigma 8-16mm F4.5-5.6 DC HSM +1 more
cainn24 Veteran Member • Posts: 4,892
Re: No need to throw the baby out with the bathwater
2

J A C S wrote:

cainn24 wrote:

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

I believe that their sensor measurements are trustworthy, as well even though they are a bit stubborn and do not reveal enough details. But once a cheater... How do we know where dishonesty stops?

I personally wouldn't go so far as to call them dishonest. It's not like they have actually fabricated any results. In fact they haven't even neglected to expose the true performance characteristics of the sensor in the DxO One product in single-shot mode. The only devious thing they have done is to give it a separate rating based on a shooting mode that some other cameras essentially have as well, but without showing those cameras (and their users) the same courtesy.

Like I said I think that's unforgivable, and I also think it has the potential to mislead people. But so long as they do in fact tell the truth about how they arrived at their numbers, which they have in fact done from the very beginning, I don't think it calls into question the veracity of any previously published results, or any results they might post in the future.

But we are, of course, all free to react to this however we see fit. In my opinion DxO Labs probably deserve a bit of a backlash.

cainn24 Veteran Member • Posts: 4,892
Re: No need to throw the baby out with the bathwater
2

va2k0r wrote:

Super raw is, well, raw, while multi shot noise reductions is JPG only.
I would like to see that feature in every camera, thx.

Sure, trying to accurately determine sensor performance characteristics by analysing JPEG output is just about impossible.  But there are other cameras that output RAW data in their "special" modes as well, like the E-M5 Mark II for example.

walkaround Senior Member • Posts: 2,551
Answer: NO
5

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

They have always been a commercial software vendor (and now camera maker, LOL), shamelessly creating the appearance of a scientific process, and deliberately posting controversial "results" in order to drive traffic to their site. Their pseudo-scientific testing and scoring process has always been a black box, to spite vaguely worded blurbs on their site that people love to link to.

Bottom line: this whole "DXO One" fiasco proves beyond a doubt that DXO has zero integrity, and everything they post on their site is suspect. And we'll probably never know the full extent of their financial relationship with Sony and Nikon, although it is clear that one exists.

cainn24 Veteran Member • Posts: 4,892
Re: Answer: NO
2

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

va2k0r
va2k0r Regular Member • Posts: 454
Re: Answer: NO
1

Now I'd really like to see a RX100 competitor from DXO ...

 va2k0r's gear list:va2k0r's gear list
Nikon D3300 Sigma 17-50mm F2.8 EX DC OS HSM Nikon 85mm F1.8G Sigma 8-16mm F4.5-5.6 DC HSM +1 more
walkaround Senior Member • Posts: 2,551
Re: Answer: NO
3

cainn24 wrote:

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

I see. So you have tested camera "x" and camera "y", and visually assessed that indeed one is a "92" and the other is an "88"?

Or something... it's all so fascinating.

cainn24 Veteran Member • Posts: 4,892
Re: Answer: NO
2

walkaround wrote:

cainn24 wrote:

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

I see. So you have tested camera "x" and camera "y", and visually assessed that indeed one is a "92" and the other is an "88"?

Or something... it's all so fascinating.

I didn't say that. I don't deny that the overall score can be a little bit useless because it seems to be weighted in favour of the camera that can produce the best results in the most ideal circumstance rather than being a reflection of performance in challenging conditions. What I said was that I have rarely encountered an individual measurement [SNR, DR etc] that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

People are best served by DxO Labs not by obsessing over the overall score but by knowing what specific performance characteristics are most important to them and then clicking through to the relevant sections within the measurements tab.

OP Tian6869 New Member • Posts: 22
Re: No need to throw the baby out with the bathwater
1

agree.

I am still trusted the site's review and ratings, but will not trust them as before.

I was only use it as my primary/stander source for sensor rating.

But now I will use it in combination of other sites.

nuke12 Veteran Member • Posts: 3,432
Re: Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?
1

Why would anyone trust a single source? It's just one of my stops when I'm looking at a camera or lens. I don't care what they did with the DxO One because I'm not buying one. I look at what I'm interested in and see if other sources are saying basically the same.

-- hide signature --

I'm a photo hacker. I use my expensive equipment to destroy anything in front of my camera. This is a special skill that can never be realized by low life photographers. A nurtured skill since the 1970's.

parcheesi Junior Member • Posts: 37
Re: Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?
3

You can choose to ignore the "Super Raw" settings. They've always had an asterisk by the ratings related to "Super Raw", so I don't think they're being dishonest. I've never felt mislead, and I've judged the sensor based on what I think the expected usage will be.

It's also important to recognize that the rating is based on a sensor+lens combination. Fixed focal length, custom designed lens to match the sensor should give a better result for the same sensor. You give up zoom though, and they've never been dishonest about that either.

The hand-wringing around DxO has been really strange to watch. Yes, they are publishing a score software interpolated "RAW" that they don't offer to anyone else with the same feature. Yes, the camera is $600. Yes, it is only for Apple phones. They've been up-front about all of that information, and if you find that distasteful or not up your alley, you're free to ignore their results (whether that be globally or just for that one camera).

I'm replacing my x100 with the DxO one, in part because I feel it will better fit a niche that my x100 currently fills. Fixed-focal length camera with a larger sensor and fast lens for travel. I'm walking into the purchase with my eyes wide open, and have not been hoodwinked by anyone. I was originally consider the Olympus Air, and in my own comparison I think the lightning connector is a better solution than fiddly WiFi+Bluetooth.

That's just my opinion though, and your mileage may vary.

perry rhodan
perry rhodan Senior Member • Posts: 1,895
DxO optics pro SW is one of the best for easy PP with very nice results, but ..
1

thats it. Period. They started with the SW DXO OP. They used extensive measures to all kinds of sensors and lenses. After this............  they cleverly started using their measured values to enhance the sales model.

-- hide signature --

Regards Perry

TrojMacReady
TrojMacReady Veteran Member • Posts: 8,729
One difference though.
2

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

walkaround Senior Member • Posts: 2,551
They don't test raw "data"
2

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc. They use raw files straight from the camera, and do in fact convert them to "images":

"As we do not have access to intermediate outputs on the sensor, DxOMark measures RAW images — the very same images that can be accessed by photographers who use cameras that shoot in RAW."

Keyboard shortcuts:
FForum MMy threads