DXOMARK.com, did you notice

Started Jan 28, 2014 | Discussions thread
tt321 Forum Pro • Posts: 11,614
Re: DXO scores are like MTF charts

agentul wrote:

tt321 wrote:

agentul wrote:

DonSC wrote:

There simply isn't any perfect way to make the comparison.

I think they try their best. It's just not that easy.

the best way is to look at some actual photos (real life and test scenes). in the end, that is what you are buying the camera to do.

In order for this to work, ultimately one needs to do one's own testing, on scenes that one is interested in and explore issues one cares most about, and this is not realistic for most people, especially in the current commercial environment where population centres of substantial size (even over 1M inhabitants) sometimes don't have even one comprehensive camera shop or camera+lens rental facility and most people have to or prefer to rely on web retailers.

the internet has many review sites and blogs. it does take a while to look at them, but the advantage is that you can spread out the process across a month or two, so that you can get a better idea. doing one's own testing probably works for people who are professional photographers and can set aside dedicated time for this. as i said, test photos don't tell the whole story, nor do synthetic benchmarks of any kind. i prefer to combine everything along with the little technical understanding of cameras that i have in order to arrive to a conclusion. at least that's what i did when i was shopping for a camera.

i see that there are people for which DxO is the final word in camera performance. allow me to share a similar story: there are a few synthetic computer benchmarking tools out there, designed initially to simulate certain workloads such as archiving and gaming performance. one thing that has been noticed over the years in reviews is that some of these programs simulate conditions that are not really often encountered in real life. one such example is 3D Mark, which has grown to be more demanding than many high end modern games. this is one reason why CPU and video card reviews use real life games and programs - most people want to know how fast their game or video encoding will be, even with mid-level hardware. having 20% less points in a synthetic benchmark doesn't necessarily mean that you're getting sub-par hardware. it may mean that you're getting enough performance for your needs at a lower price.the other reason computer hardware reviews have moved away from synthetic benchmark programs is that some manufacturers have been caught cheating - optimizing drivers for the synthetic benchmarks used in reviews, while real life performance was lagging.

also, i recall some embarrassing articles back in 2008 when the X58 chipset was launched, and motherboard manufacturers were rushing to put out models that would be faster than the competition in terms of RAM performance. a closer look by AnandTech revealed that the BIOS did not match what was written on paper: even though a model was supposed to support more than 20GB of RAM, in real life it wasn't working properly with anything above the 3 or 6 GB used in reviews at the time.

another example would be how certain car manufacturers have been caught cheating at fuel efficiency tests by optimizing the ECUs to detect the test cycle and reduce fuel consumption to levels below what would be encountered in real life scenarios.

bottom line: synthetic benchmarks are useful, but the numbers should always be taken with a grain of salt.

I agree with most of what you said, but synthetic benchmarks serve other purposes as well, such as providing a stable and well-defined standard for a common discussion to be based on. A lot of academic work make huge uses of these benchmarks and that's a widely accepted practice so long as the methodology is supportable.

Your objections are not with the benchmarks themselves, but the way they have been misused.

On the other hand, it's probably very difficult and not worth the trouble to design a lens that is brilliant at MTF-50 but otherwise crappy in picture taking as everyone would see with their own eyes from the results, even if this is possible. Fuel economy is somewhat different as a lot of people don't measure how much they are getting out of their fuel with any precision at all and sometimes even large differences are unnoticed.

Post (hide subjects) Posted by
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow