DxOMark Mobile testing protocol now considers bokeh simulation, low-light, motion and zoom

it seems that would yield a qualitative opinion rather than a quantitative value or number.

May as well have a test in which the metric is "do I like the picture?"

Sep 13, 2017*permalink

ikfoto

BadScience,
I can easily imagine how to. A series of shots, at different FLs and apertures, of tiny red LEDs placed at different distances and positions across the frame, with the lens focused on the closest. It remains to figure out the right distances. And let people judge for themselves.

A simple version can be seen at
https://www.lenstip.com/468.7-Lens_review-Panasonic_Lumix_G_25_mm_f_1.7_ASPH._Coma__astigmatism_and_bokeh.html

But you obviously have no clue, your nickname speaks for itself.

Sep 14, 2017permalink

"And let people judge for themselves."

what's the difference between this and

""do I like the picture?"

what's the metric? What *number* are you comparing? What *number* is better than another?

Sep 14, 2017permalink

ikfoto

What the heck do you need a number for? To plug it into the the overall DXO mark? It already has many weighted variables. But if you insist, I can see several approaches.

1. It's not that hard to come up with equations for the roundness and radial fall-off smoothness, when the LEDs appear on a black background, as shown on Lenstip.

2. DXO could use a panel of experts to assign a number.

3. A combination of the above.

4. Whatever the methodology, bokeh as a separate measure, not to skew the overall mark, since not everyone pays attention to bokeh. That's why "let people judge for themselves", as a simple but effective way, as a supplement for the most demanding users, with numbers or without.

Sep 15, 2017permalink

ikfoto: "What the heck do you need a number for? "

If you measure something, the measurement is a number.

It's quite simple.

otherwise you are left with opinion

Sep 15, 2017permalink

Expat Nomad

How do the test scores from phones compare to their other rating system for cameras? I do wish they'd rate equivalently in a single scale.

Sep 12, 2017permalink

(unknown member)

They don't rate raw files from smartphones. Therefore it's not comparable.

Sep 12, 2017permalink

Many phones have RAW so they could, and should. If phone want to compete with cameras they should compete with cameras.

Sep 13, 2017permalink

Melchiorum

If phones "should compete with cameras", this should work both ways. How about we also compare size, connectivity, third-party app support, accessibility, ease of use, rate of innovation and so on?

Comparing phones to dedicated photography equipment is as crazy, as criticizing Nikon D850 for not being able to put in into one's pocket or post a picture taken with it to Facebook by just tapping a screen a couple of times.

They are different devices with different purpose with some functionality being similar.

Sep 13, 2017permalink

@Melchiorum

You know if you carry on making sober and rational comments on such things you'll eventually be barred from the internet.

Sep 13, 2017permalink

expat - that would be like comparing a F1 car's ability to drive across a ploughed field or a tractor's ability to negotiate Silverstone at 200 mph.

They are both vehicles, with four wheels, and an engine, that conveys a passenger from A-B. But they are different beasts.

Okay, this is an extreme example, but you get the drift

Sep 13, 2017*permalink

@Melchiorium, Totally disagree. That makes no sense at all. When car magazines run acceleration, braking and cornering tests, all cars get the same standard testing. It doesn't matter if it is a Corvette or Spark or Silverado. How else would you know how much better the Corvette is if you don't run the same tests?

Phones are getting the same sized sensors at many compact cameras, and some shoot RAW, which some compact cameras can't do. Why again shouldn't they run through the same test procedure?

Sep 14, 2017*permalink

@Michael1000

Not sure what the cars you refer to are - they sound like brands of ready meal to me. If you're going to assess a Ferrari F40 on luggage space, people carrying and negotiating of traffic humps on the school run and a Citroen Picasso [that's a people carrier] on its ability to hug a corner at 120mph and do doughnuts, you might conclude both are crap cars. They do different jobs, designed for different purposes. Which is 'best' depends on what your needs are. If you tested and scored them in exactly the same way it might be meaningful. But only to someone VERY strange.

Sep 14, 2017permalink

F40? What year is this?

Go look at actual magazine reviews from major publications, such as Motor Trend, Car and Driver, and Road and Track. The cars still get put their their same standardized tests. They have been doing this since the beginning of time, yes, even when the F40 was built. Yes, they even record the luggage space. I'm not talking about a head-to-head. Of course, they are not going to put some POS Citroen against a Ferrari. I'm talking about individual car reviews, with standardized tests.

I don't understand all the resistance to standardized tests. Why can't I get the dynamic range, ISO Invariance, Color Sensitivity, etc. for a phone camera as is done with other cameras. There is no reason why this can't be done. Of course, the performance will be less than most DSLRs, but it will be close to come compacts. At least you have comparison data. What do we have now? Nothing. It's meaningless without the same tests.

Sep 14, 2017*permalink

So does that mean with DSLRs they perform a test on how the second camera performs? How all the sensors work in conjunction? How effective the panoramic stitching is? Not to mention how quickly they post to social media, how well they geotag and so on. Discounting the fact they do different things, they also do the same things in often significantly different ways. So ultimately it becomes impractical to do identical tests and meaningful specific comparisons with absolute impartiality. The same would be true of things like the L16 camera - it's a different paradigm. You can do SOME useful comparisons, but in the final analysis you need to recognise what you're testing are fundamentally different devices.

Sep 15, 2017*permalink

I think in the future as all imaging devices become more essentially computational than mechanical, many of the current tests will start to look obsolete. So I think it's likely tests will move more toward the assessments undertaken with smartphones than the current ones used to measure traditional imaging devices. Without being able to precisely anticipate what the technology will be and precisely how it will work, it becomes ever more difficult to design a standard testing protocol.

DPReview now tests far more things today than when Phil Askey started the site many years ago. Much as I fondly remember his stewardship of this place, it's had to move with the times.

Sep 15, 2017permalink