Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?

That is exactly what I am trying to point out.

There are two ratings "DxO ONE SuperRAW Plus" and "DxO ONE"

DxO ONE SuperRAW Plus is when Multi-frame noise reduction mode is one.

However, Multi-frame noise reduction mode is not new. It can be done either within the cameras or in your computer.

For example, Sony RX100 also has such function (Multi-frame noise reduction mode) that it can take a fast burst of shots (4-5 shots) and composites them to produce a single image with less noise.

To be fair, Sensor rating should be based on the same method/testing (e.g. original single shot) as they are rating every other cameras.

That means if they want to use "superraw" mode and get 1657, then to be fair, at least they should rate ALL camera like that (or at least cameras which has such function buildin).

BUT they are not.

They ignored other cameras which also has such function, and not rating them under Multi-frame noise reduction mode, and only rated their own device with Multi-frame noise reduction mode.

I think if they rated RX100M3/RX100M4 with Multi-frame noise reduction mode, it will go much higher then DxO One.

BUT they are not doing that.

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

cainn24 wrote:

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

I believe that their sensor measurements are trustworthy, as well even though they are a bit stubborn and do not reveal enough details. But once a cheater... How do we know where dishonesty stops?

J A C S wrote:

cainn24 wrote:

DxO Labs have an extremely useful database of sensor performance characteristics. That hasn't changed. They may be leveraging their position to help make their own product successful, and they may be going about it in an unforgivably inconsistent and borderline shameless manner, but for those who know what they are looking for, how to look for it, and how to interpret it, the site is just as useful as it has always been.

I believe that their sensor measurements are trustworthy, as well even though they are a bit stubborn and do not reveal enough details. But once a cheater... How do we know where dishonesty stops?

I personally wouldn't go so far as to call them dishonest. It's not like they have actually fabricated any results. In fact they haven't even neglected to expose the true performance characteristics of the sensor in the DxO One product in single-shot mode. The only devious thing they have done is to give it a separate rating based on a shooting mode that some other cameras essentially have as well, but without showing those cameras (and their users) the same courtesy.

Like I said I think that's unforgivable, and I also think it has the potential to mislead people. But so long as they do in fact tell the truth about how they arrived at their numbers, which they have in fact done from the very beginning, I don't think it calls into question the veracity of any previously published results, or any results they might post in the future.

But we are, of course, all free to react to this however we see fit. In my opinion DxO Labs probably deserve a bit of a backlash.

va2k0r wrote:

Super raw is, well, raw, while multi shot noise reductions is JPG only.
I would like to see that feature in every camera, thx.

Sure, trying to accurately determine sensor performance characteristics by analysing JPEG output is just about impossible.  But there are other cameras that output RAW data in their "special" modes as well, like the E-M5 Mark II for example.

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

They have always been a commercial software vendor (and now camera maker, LOL), shamelessly creating the appearance of a scientific process, and deliberately posting controversial "results" in order to drive traffic to their site. Their pseudo-scientific testing and scoring process has always been a black box, to spite vaguely worded blurbs on their site that people love to link to.

Bottom line: this whole "DXO One" fiasco proves beyond a doubt that DXO has zero integrity, and everything they post on their site is suspect. And we'll probably never know the full extent of their financial relationship with Sony and Nikon, although it is clear that one exists.

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

cainn24 wrote:

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

I see. So you have tested camera "x" and camera "y", and visually assessed that indeed one is a "92" and the other is an "88"?

Or something... it's all so fascinating.

walkaround wrote:

cainn24 wrote:

walkaround wrote:

Answer: no.

We couldn't trust them before the DXO One, and we certainly can't trust them now.

Yet I have rarely encountered an individual measurement that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

I see. So you have tested camera "x" and camera "y", and visually assessed that indeed one is a "92" and the other is an "88"?

Or something... it's all so fascinating.

I didn't say that. I don't deny that the overall score can be a little bit useless because it seems to be weighted in favour of the camera that can produce the best results in the most ideal circumstance rather than being a reflection of performance in challenging conditions. What I said was that I have rarely encountered an individual measurement [SNR, DR etc] that doesn't gel with a visual analysis of actual RAW output from the cameras in question.

People are best served by DxO Labs not by obsessing over the overall score but by knowing what specific performance characteristics are most important to them and then clicking through to the relevant sections within the measurements tab.

agree.

I am still trusted the site's review and ratings, but will not trust them as before.

I was only use it as my primary/stander source for sensor rating.

But now I will use it in combination of other sites.

Why would anyone trust a single source? It's just one of my stops when I'm looking at a camera or lens. I don't care what they did with the DxO One because I'm not buying one. I look at what I'm interested in and see if other sources are saying basically the same.

You can choose to ignore the "Super Raw" settings. They've always had an asterisk by the ratings related to "Super Raw", so I don't think they're being dishonest. I've never felt mislead, and I've judged the sensor based on what I think the expected usage will be.

It's also important to recognize that the rating is based on a sensor+lens combination. Fixed focal length, custom designed lens to match the sensor should give a better result for the same sensor. You give up zoom though, and they've never been dishonest about that either.

The hand-wringing around DxO has been really strange to watch. Yes, they are publishing a score software interpolated "RAW" that they don't offer to anyone else with the same feature. Yes, the camera is $600. Yes, it is only for Apple phones. They've been up-front about all of that information, and if you find that distasteful or not up your alley, you're free to ignore their results (whether that be globally or just for that one camera).

I'm replacing my x100 with the DxO one, in part because I feel it will better fit a niche that my x100 currently fills. Fixed-focal length camera with a larger sensor and fast lens for travel. I'm walking into the purchase with my eyes wide open, and have not been hoodwinked by anyone. I was originally consider the Olympus Air, and in my own comparison I think the lightning connector is a better solution than fiddly WiFi+Bluetooth.

That's just my opinion though, and your mileage may vary.

thats it. Period. They started with the SW DXO OP. They used extensive measures to all kinds of sensors and lenses. After this............  they cleverly started using their measured values to enhance the sales model.

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc. They use raw files straight from the camera, and do in fact convert them to "images":

"As we do not have access to intermediate outputs on the sensor, DxOMark measures RAW images — the very same images that can be accessed by photographers who use cameras that shoot in RAW."