DxOMark Mobile testing protocol now considers bokeh simulation, low-light, motion and zoom

it seems that would yield a qualitative opinion rather than a quantitative value or number.

May as well have a test in which the metric is "do I like the picture?"

Sep 13, 2017*permalink

ikfoto

BadScience,
I can easily imagine how to. A series of shots, at different FLs and apertures, of tiny red LEDs placed at different distances and positions across the frame, with the lens focused on the closest. It remains to figure out the right distances. And let people judge for themselves.

A simple version can be seen at
https://www.lenstip.com/468.7-Lens_review-Panasonic_Lumix_G_25_mm_f_1.7_ASPH._Coma__astigmatism_and_bokeh.html

But you obviously have no clue, your nickname speaks for itself.

Sep 14, 2017permalink

"And let people judge for themselves."

what's the difference between this and

""do I like the picture?"

what's the metric? What *number* are you comparing? What *number* is better than another?

Sep 14, 2017permalink

ikfoto

What the heck do you need a number for? To plug it into the the overall DXO mark? It already has many weighted variables. But if you insist, I can see several approaches.

1. It's not that hard to come up with equations for the roundness and radial fall-off smoothness, when the LEDs appear on a black background, as shown on Lenstip.

2. DXO could use a panel of experts to assign a number.

3. A combination of the above.

4. Whatever the methodology, bokeh as a separate measure, not to skew the overall mark, since not everyone pays attention to bokeh. That's why "let people judge for themselves", as a simple but effective way, as a supplement for the most demanding users, with numbers or without.

Sep 15, 2017permalink

ikfoto: "What the heck do you need a number for? "

If you measure something, the measurement is a number.

It's quite simple.

otherwise you are left with opinion

Sep 15, 2017permalink

Expat Nomad

How do the test scores from phones compare to their other rating system for cameras? I do wish they'd rate equivalently in a single scale.

Sep 12, 2017permalink

They don't rate raw files from smartphones. Therefore it's not comparable.

Sep 12, 2017permalink

Many phones have RAW so they could, and should. If phone want to compete with cameras they should compete with cameras.

Sep 13, 2017permalink

Melchiorum

If phones "should compete with cameras", this should work both ways. How about we also compare size, connectivity, third-party app support, accessibility, ease of use, rate of innovation and so on?

Comparing phones to dedicated photography equipment is as crazy, as criticizing Nikon D850 for not being able to put in into one's pocket or post a picture taken with it to Facebook by just tapping a screen a couple of times.

They are different devices with different purpose with some functionality being similar.

Sep 13, 2017permalink

@Melchiorum

You know if you carry on making sober and rational comments on such things you'll eventually be barred from the internet.

Sep 13, 2017permalink

expat - that would be like comparing a F1 car's ability to drive across a ploughed field or a tractor's ability to negotiate Silverstone at 200 mph.

They are both vehicles, with four wheels, and an engine, that conveys a passenger from A-B. But they are different beasts.

Okay, this is an extreme example, but you get the drift

Sep 13, 2017*permalink

@Melchiorium, Totally disagree. That makes no sense at all. When car magazines run acceleration, braking and cornering tests, all cars get the same standard testing. It doesn't matter if it is a Corvette or Spark or Silverado. How else would you know how much better the Corvette is if you don't run the same tests?

Phones are getting the same sized sensors at many compact cameras, and some shoot RAW, which some compact cameras can't do. Why again shouldn't they run through the same test procedure?

Sep 14, 2017*permalink

@Michael1000

Not sure what the cars you refer to are - they sound like brands of ready meal to me. If you're going to assess a Ferrari F40 on luggage space, people carrying and negotiating of traffic humps on the school run and a Citroen Picasso [that's a people carrier] on its ability to hug a corner at 120mph and do doughnuts, you might conclude both are crap cars. They do different jobs, designed for different purposes. Which is 'best' depends on what your needs are. If you tested and scored them in exactly the same way it might be meaningful. But only to someone VERY strange.

Sep 14, 2017permalink

F40? What year is this?

Go look at actual magazine reviews from major publications, such as Motor Trend, Car and Driver, and Road and Track. The cars still get put their their same standardized tests. They have been doing this since the beginning of time, yes, even when the F40 was built. Yes, they even record the luggage space. I'm not talking about a head-to-head. Of course, they are not going to put some POS Citroen against a Ferrari. I'm talking about individual car reviews, with standardized tests.

I don't understand all the resistance to standardized tests. Why can't I get the dynamic range, ISO Invariance, Color Sensitivity, etc. for a phone camera as is done with other cameras. There is no reason why this can't be done. Of course, the performance will be less than most DSLRs, but it will be close to come compacts. At least you have comparison data. What do we have now? Nothing. It's meaningless without the same tests.

Sep 14, 2017*permalink

So does that mean with DSLRs they perform a test on how the second camera performs? How all the sensors work in conjunction? How effective the panoramic stitching is? Not to mention how quickly they post to social media, how well they geotag and so on. Discounting the fact they do different things, they also do the same things in often significantly different ways. So ultimately it becomes impractical to do identical tests and meaningful specific comparisons with absolute impartiality. The same would be true of things like the L16 camera - it's a different paradigm. You can do SOME useful comparisons, but in the final analysis you need to recognise what you're testing are fundamentally different devices.

Sep 15, 2017*permalink

I think in the future as all imaging devices become more essentially computational than mechanical, many of the current tests will start to look obsolete. So I think it's likely tests will move more toward the assessments undertaken with smartphones than the current ones used to measure traditional imaging devices. Without being able to precisely anticipate what the technology will be and precisely how it will work, it becomes ever more difficult to design a standard testing protocol.

DPReview now tests far more things today than when Phil Askey started the site many years ago. Much as I fondly remember his stewardship of this place, it's had to move with the times.

Sep 15, 2017permalink

I don't know why this is so difficult for people to understand. We are talking about objective image tests, think measurements, not how well it posts to social media.

Sep 16, 2017permalink

Expat Nomad

Duplicate. 404.

Sep 12, 2017*permalink

sirhawkeye64

IMO, Low light, motion and would be useful.... zoom and bokeh, probably not so much But that's just me. but honestly, the best test that will really tell the truth is YOU, the person using the phone/camera. What DxO thinks may or may not fit your style or preferences and usually those tests are in a controlled environment. I'd rather do my tests myself, camera/phone in hand and then compare.

Sep 12, 2017*permalink

Their "Zoom" rating is very misleading.
It should be called "Tele" instead. Because apparently they don't compare the sharpness at 2x ,3x, zoom. Instead they compare the sharpness at 50mm and at 85mm: "We compare zooms by setting the camera zoom to a 50mm and 85mm full-frame equivalent field of view by using whatever zoom method the camera supports (optical, digital, or hybrid) so that the full target is framed."
https://www.dxomark.com/shoot-apple-iphone-7-versus-apple-iphone-7-plus-using-new-test-protocols/
So of course a 30mm LG G6 or a 28mm iPhone 7 should be sharper at 50mm than a 26mm Google Pixel. But this does not say which one is better at 2x digital/hybrid zoom. Such a tele rating is interesting indeed, but the term "Zoom" is misleading.
They also have a "Bokeh" rating which should be rather called "Portrait mode" because it includes factors like detail and perspective distortion (see link).
Very misleading. Deception is the opposite of science.

Sep 12, 2017permalink

Hannu108

Where are the camera sensor tests...? Latest camera sensor review in June!

Sep 12, 2017permalink

panther fan

I'm going against the trend and say the Dxomark is often the only source that you have for comparing all the different lenses, cameras and phones out there. You can trash talk them all you want but they offer reliable data and there is simply no alternative.
So thank you Dxomark for your work.

Sep 12, 2017permalink

vadims

> there is simply no alternative

You are right, they're about the only game in town. That does not make them any better than they are though.

That is not to say they are "bad" (let alone as bad as some are portraying them...), but they do have their problems.

Oh, and DxO One was a mistake, it irritated many people and DxO is now reaping the consequences. No-one but themselves to blame.

Sep 12, 2017permalink

So you think they worked reliably although they nearly never published any full resolution samples in order to explain their results? You think they are reliable although they cooperated with certain smartphone companies like Oneplus and although they want to sell their own camera products (and therefore can have a conflict of interest)? You think they are reliable although they occasionally publish a rating on the same day as the announcement of the product? You think they are reliable although the Xperia Z5 or HTC 10 were rated as the best smartphone cameras when they were released?
And even if they were reliable, it would be ridiculous to compare smartphone cameras with these scores and Dxomark itself delivers the proof: Dxomark has changed the way it tests phones and now the U11 is not better anymore. If the previous scores made sense, then there wouldn't have been a need to change it. So obviously the scores are useless.

Sep 12, 2017permalink

Sep 12, 2017*permalink

kuklukklak

yeah right. As long as they rank your brand higher they will still be reliable. Haha

Sep 12, 2017permalink

https://support.dxo.com/hc/en-us/articles/206501907-How-is-it-possible-for-the-DxO-ONE-to-capture-DSLR-quality-images-

When you download their full resolution camera samples you even see that they even don't compare the phones under the same conditions. For instance they say 5 Lux, but when you compare the images from the two smartphones, you'll see for instance that some things are illuminated differently (light coming from a different direction) than in the sample from the other phone. They also don't compare the phones at the same time (for instance they compare samples from 2016 with samples from 2017) although the phone cameras change with updates. They even seem to occasionally rate pre-production units. Seems very amateurish.

Sep 13, 2017*permalink

c45

DxOMark is a joke and a scam.

Sep 12, 2017permalink

Bambi24

DxO, the site that says their own DxO One camera with a puny 1'' inch sensor takes the same quality images as a DSLR.

Why does anyone take them serious.

Sep 12, 2017*permalink

don-bosco

This company is a joke! They don't worth anyone's time really. However Dpreview cannot breath without mentioning their name.

Sep 12, 2017permalink

don-bosco

This company is a joke! They don't worth anyone's time really. However Dpreview cannot breath without mentioning their name.

Sep 12, 2017permalink

panther fan

No they don't. Read BOTH tests. They say it just performs like every one inch camera but can merge four photos to get better results. Olympus and Pentax offer similar tech and it works. Just not for moving subjects.

Sep 12, 2017permalink

big aperture

you can do it with every camera if you work in photoshop

Sep 12, 2017permalink

weixing

Hi,
After they merge and process (include NR) the four images, they still call it RAW... ok, may be they don't call it RAW, they call it SuperRAW, but they still use that to compare to a DSLR single shot RAW and call the IQ far exceeds that of most high-end digital cameras...
This is like compare a RAW meat to a cooked meat prepare by a top chef and declare the winner is the cooked meat...

Hmm... if that's the case, after photoshop, NR and export to TIFF, mine is call TIFFRAW.... I think it'll score far better than DxO ONE... what a joke... ha ha ha :-P

Have a nice day.

Sep 12, 2017*permalink

big aperture

they use marketing terms (like super-raw etc) to push this dxo camera..i am not buying it ofc but some might people might buy it.

Sep 12, 2017permalink