DxOmark -- why "only" 71 points?

[noirdesir]

Further, I encourage you to perform the following experiment.

I encourage myself to trust my judgement and wait for somebody doing quantitative measurements instead of relying on visual judgements which I only trust when integrated over a large enough sample (and I consider one test scene and one person, ie, myself very far from large enough) in particular when said sample is imperfect in regard to exposure consistency.

I have done this exercise over a large enough sample to trust my observations. And as far as persons and their perceptions are concerned, my own are the ones I primarily care about. I want my pictures to look good to myself.

I of course get satisfaction from my own photos, but I get more satisfaction if other people like them.

OK. Your choice. But where do you find good data on how the average perception of the particular people to whom you show your pictures correlate with numbers like those produced by DxO.

If you have any more representative information than DxO, I'll be all ear.

And even if you had such information, how would it help you when you PP your images (as I guess you do). I for the most part rely on my own eyes here

I just try use representative data as much as possible and keep my own subjective contribution as small as possible. This means I choose the best camera, the best lenses, the best raw converter, the best processing techniques based on representative data. The better my basis is, the less my own subjective decisions will let me deviate from the optimal results.

And I actually went down that road and compared the full SNR curves from DxO from different cameras to replicate Marianne's graphs for other cameras. But they are a pain to use as one cannot access the numbers directly which is necessary to scale them properly, ie, one either scales images of curves and pastes them together or manually reads the values out of the graph.

Yes, doing the exercise I proposed is much easier.

But much less precise and much more biased.

In what ways would it be less precise and more biased. As far as I can see, there is neither less precision nor more bias built into the procedure I proposed. It's just a way of checking which of two alternative DxO measures are better attuned to your perceptions.

Dpreview does not keep the exposure constant between different cameras, that makes it less precise. It is biased because it is one test scene. One test scene will never be fully representative. Plus it is using one raw converter that biases it further. Of course I could download raw files where available and process them in my raw converter, that would make it more precise but raw converters evolve (including my skills in exploiting their capabilities) plus I am not just interested in my images but also in giving advice to others and I don't know what raw converter they use.

 

[Anders W]

Further, I encourage you to perform the following experiment.

I encourage myself to trust my judgement and wait for somebody doing quantitative measurements instead of relying on visual judgements which I only trust when integrated over a large enough sample (and I consider one test scene and one person, ie, myself very far from large enough) in particular when said sample is imperfect in regard to exposure consistency.

I have done this exercise over a large enough sample to trust my observations. And as far as persons and their perceptions are concerned, my own are the ones I primarily care about. I want my pictures to look good to myself.

I of course get satisfaction from my own photos, but I get more satisfaction if other people like them.

OK. Your choice. But where do you find good data on how the average perception of the particular people to whom you show your pictures correlate with numbers like those produced by DxO.

If you have any more representative information than DxO, I'll be all ear.

I am afraid I don't follow you here. How can the objective (non-perceptual) information you get from DxO tell you how their various measures correlate with your own perceptions or those you show your pictures to?

And even if you had such information, how would it help you when you PP your images (as I guess you do). I for the most part rely on my own eyes here

I just try use representative data as much as possible and keep my own subjective contribution as small as possible. This means I choose the best camera, the best lenses, the best raw converter, the best processing techniques based on representative data. The better my basis is, the less my own subjective decisions will let me deviate from the optimal results.

And I actually went down that road and compared the full SNR curves from DxO from different cameras to replicate Marianne's graphs for other cameras. But they are a pain to use as one cannot access the numbers directly which is necessary to scale them properly, ie, one either scales images of curves and pastes them together or manually reads the values out of the graph.

Yes, doing the exercise I proposed is much easier.

But much less precise and much more biased.

In what ways would it be less precise and more biased. As far as I can see, there is neither less precision nor more bias built into the procedure I proposed. It's just a way of checking which of two alternative DxO measures are better attuned to your perceptions.

Dpreview does not keep the exposure constant between different cameras, that makes it less precise.

Yes, they do.

It is biased because it is one test scene. One test scene will never be fully representative. Plus it is using one raw converter that biases it further. Of course I could download raw files where available and process them in my raw converter, that would make it more precise but raw converters evolve (including my skills in exploiting their capabilities) plus I am not just interested in my images but also in giving advice to others and I don't know what raw converter they use.

As long as the scene lets us see noise across the entire spectrum (from deep shadows to strong highlights), it will be fully representative as far as noise is concerned. And it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

 

[Detail Man]

Dpreview does not keep the exposure constant between different cameras, that makes it less precise.

Yes, they do.

Even though it seems that individual camera-metering systems are not involved, is it not true that individual (and varying) camera RGB tone-curve transfer-functions are involved ? Or have I perhaps mis-understood what has been previously stated about DPReview's ISO/SCT test-shot techniques?

... it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

Would it not be a much better method if DPReview were to use a RAW processor with known and controllable characteristics - as opposed to inextricably intertwining results within the proprietary (and thus not truly knowable) elements of potential Black Point shifting, potential silent RGB Tone Curve transfer-function characteristics, as well as possible silent Noise Reduction, taking place ?

One could argue that RAW Therapee is always evolving in form - but the very same thing can (also) be said regarding Lightroom / Camera RAW (where changes are not disclosed, or known)

 

[Anders W]

Dpreview does not keep the exposure constant between different cameras, that makes it less precise.

Yes, they do.

Even though it seems that individual camera-metering systems are not involved, is it not true that individual (and varying) camera RGB tone-curves transfer-functions are involved ? Or have I perhaps mis-understood what has been previously stated about DPReview's SCT test-shot techniques ?

It's not involved in a way that would run into trouble with my claim. The only real reservation here is that DPR reports in each review (Noise and NR section) whether the camera deviates from the ISO standard. If it does, then it will be given correspondingly less or more exposure than cameras that do not deviate, which you can then take into account in your comparisons. It is rather unusual for cameras to deviate from the ISO standard by more than the threshold set by DPR (1/6 EV).

The only other reservation is the one we must make whenever someone reports something: That DPR actually do what they say they do.

... it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

Would it not be a much better thing if DPReview was to use a RAW processor with known and controllable characteristics - as opposed to inextricably intertwining results within the proprietary (and thus not knowable) elements of potential Black Point shifting, potential "silent" RGB Tone Curve transfer-function characteristics, as well as possible "silent" Noise Reduction taking place ?

One could argue that RAW Therapee is always evolving in form - but the very same thing can (also) be said regarding Lightroom / Camera RAW (where changes are not disclosed, or known)

This is a thorny issue because just using exactly the same procedures does not guarantee comparable results. Different choice of black-point offset by different manufactures would imply different choice of black-point in the jpeg conversion (did you try to see what happens if you set them identically for the GH2 and the E-M5 ). Different CFAs would imply different multipliers to achieve the same WB. The same tone curve mapping cannot be applied if the RAW information is differently encoded (e.g., linearly relative to the light level in some cases and non-linearly in others). And non-Bayer sensors (e.g., Fuji) cannot be demosaiced in the same way as conventional Bayer sensors. And so on.

As to the question of whether no NR is no NR in ACR, I am not particularly worried.

 

[Chez Wimpy]

But what you say about RRS bracket and spare batteries surprise me a bit. Why would it be more difficult to get those in Japan than elsewhere? If anything, I would have guessed the opposite.

I had serious "issues" with RRS with an international purchase of brackets for my Canon cameras before. Long story, but the trip(s) to Japan, CC, refunds, ect was not without substantial headaches. Batteries, that is another ebay/US domestic prices for 3rd party as opposed to shopping Oly in Japan. Did very well with the 5D2 chipped batteries. The Japanese post office will also not "air" any batteries at all, which is a problem over several legs of the outlying-islands trip. I still don't have any extra batteries for my GH2 either, which could be a problem this winter.

Even though you don't have the E-M5, I'd be very interested in seeing directly comparable samples from what you have, especially the G1 and the GH2.

I only have LED or fluorescent light (compact / reg) but if that is good enough, and you want to see via the 45/1.8, I will see what I can do.

--
-CW

 

[boggis the cat]

In any case:

• fact is that DxO do not publish margins of error,

like every other consumer test site.

As I have also pointed out, Bob, the lack of uncertainties ("margins of error") makes comparisons and reuse of their data problematic.

When you are looking at data to 0.1 EV resolution then the uncertainties should really back that up. If not, then you have implicit error bars of unknown magnitude in all of the DxO data (and your sensorgen information).

and that these margins are likely at least 10 per cent

on what do you base that estimate? What is its margin of error.

That would be about my guesstimate, too, assuming a tightly controlled test setup. Somewhere between 5 and 20 percent.

We also have little idea of "native" variability for these cameras. Again, my guesstimate would be around 0.2 EV or lower (for higher-end equipment).

and thus most of the score differences they highlight and so skillfully make buzz about, are meaningless

Not meaningless, just that they have a margin of error. Different thing.

Lack of that data makes the results not strictly comparable between cameras -- and that is the point, right?

• fact is that DxO's black box is not credible. Suffice it to remember that when they first introduced mid format cameras in their rankings, they came out quite low. When photographers protested that this was ridiculous and certainly did not reflect real life, DxO just tweaked the black box so they would come up on top. Ridiculous.

Your statement is ridiculous. The MF backs that scored low still score low. New MF backs have scored better. Look just a little and it's easy to see why they didn't score well, in proportion to their size with respect to state of the art CMOS FF sensors. Much higher read noise, much lower QE.

I recall reading about DxO "tweaking" results before.

Then they have the Canon EF 70-200mm f/2.8L IS II USM erroneous results, that they did revisit after enough complaints -- but did they really believe that a new design much-lauded by every other review site was worse than the original?

• fact is that both Adobe and DxO have repeatedly proved behind the new camera curve, and not providing the consistent processing you mention

Wouldn't know, don't use either, but if that statement is as reliable as most of your 'facts', I'll discount it.

Adobe were certainly behind on demosaicing. This was one of the odd things I discovered when looking into "moire" issues on the E-5. (It turns out that most "moire" is attributable to lousy demosaicing; and not only with the E-5 or Panasonic sensors.)

Good for DxO if they're managing to create all that excitement with their largely phoney rankings - but these have very little to do with actual photography.

A juvenile position to take. The fact there is no published margin of error would make all consumer camera and lens tests phony. The idea that SNR or DR or colour depth have little to do with actual photography is silly.

That depends on how high that uncertainty is.

If you have a "14.0 EV" camera and a "13.0 EV" camera, but your uncertainties for each result are 0.5 EV, then the chances are that the actual delta is 0.7-1.3 EV (this depends on the distribution, or type of curve e.g. Gaussian or rectangular etc.).

Another case may be if you have a "13.2 +/- 0.5 EV" camera and a "13.0 EV +/- 0.1 EV" camera then you may wish to buy the camera with the tighter uncertainty to be surer of getting at least 12.9 EV.

Back to the "juvenile" comment - you should perhaps be more careful, because perhaps the person you try to slightly insult, has more business experience, more experience of regularly meeting CEOs around the world, more experience investing in various sectors, than you ever will?

In this case, I very much doubt it. The way you write and the quality of your analysis speaks for itself. Juvenile.

Well, your view of DxOMark represents a conflict of interest. Overlooking the qualitative component of the "magic numbers" -- the uncertainties -- is an odd thing for someone with a science background to do.

 

[Anders W]

But what you say about RRS bracket and spare batteries surprise me a bit. Why would it be more difficult to get those in Japan than elsewhere? If anything, I would have guessed the opposite.

I had serious "issues" with RRS with an international purchase of brackets for my Canon cameras before. Long story, but the trip(s) to Japan, CC, refunds, ect was not without substantial headaches. Batteries, that is another ebay/US domestic prices for 3rd party as opposed to shopping Oly in Japan. Did very well with the 5D2 chipped batteries. The Japanese post office will also not "air" any batteries at all, which is a problem over several legs of the outlying-islands trip. I still don't have any extra batteries for my GH2 either, which could be a problem this winter.

Yes. I forgot that your location along with the restrictions on transporting batteries by air might give rise to special problems. I encountered that restriction myself once when I considered importing some very special batteries for my car from the US. On the other hand, it strikes me now that I had no trouble getting a spare battery for my G1 from Hongkong, which I am sure came by air. So the restriction is implemented differently in different places I suppose.

Even though you don't have the E-M5, I'd be very interested in seeing directly comparable samples from what you have, especially the G1 and the GH2.

I only have LED or fluorescent light (compact / reg) but if that is good enough, and you want to see via the 45/1.8, I will see what I can do.

Fluorescent will do nicely so I'd be much obliged if you could try that. I can easily check for any difference between halogen (which I used for the test shots I posted) and fluorescent with my own two cameras. Haven't gotten around to trying LED bulbs in our apartment yet. Still quite expensive over here although the prices are of course coming down.

 

[bobn2]

As I have also pointed out, Bob, the lack of uncertainties ("margins of error") makes comparisons and reuse of their data problematic.

Sure, but no more problematic than any other comparison site - they none of them publish error margins. One would not expect them to, it is not part of common practice for consumer information sites. So we get left with the situation where DxO seems to be held uniquely culpable for not publishing error margins sites such as DPR, which also do not publish error margins (and moreover have quite glaring procedural faults in their tests). It seems inconsistent that DxO is picked out.

When you are looking at data to 0.1 EV resolution then the uncertainties should really back that up. If not, then you have implicit error bars of unknown magnitude in all of the DxO data (and your sensorgen information).

And every other spurce of data. That includes that coming from people like Emil Martinec and Bill Claff.

That would be about my guesstimate, too, assuming a tightly controlled test setup. Somewhere between 5 and 20 percent.

If you want a guesstimate, that seems way too large. In the case of the sensor measurements, most of them relate to the SNR at some illumination level. A laboratory illuminant should be controllable to a fraction of a percent. If the SNR is taken using a 100 pixel square, that is a sample size of 10,000 - which will give a confidence interval for the SD (noise) of a couple of percent. So my guesstimate would be something smaller than 5%, which is 0.07 stops.

We also have little idea of "native" variability for these cameras. Again, my guesstimate would be around 0.2 EV or lower (for higher-end equipment).

You mean sample to sample variation. I have no idea - gain that is a weakness for all these sites that rely on single sample tests, which is all of them.

I recall reading about DxO "tweaking" results before.

That doesn't make it true. Many people wish to discredit DPR for their own reasons.

Then they have the Canon EF 70-200mm f/2.8L IS II USM erroneous results, that they did revisit after enough complaints -- but did they really believe that a new design much-lauded by every other review site was worse than the original?

They haven't changed their view. they still give the MkII a DxO mark of 17 and the original 21. So the story that they 'revisited' is bunkum. The DxO overall scores should be taken with a pinch of salt anyway - as they say 'The DxOMark Score is the performance of a lens (with a camera body) for its best focal length and aperture combination. It does not show how the lens behaves over its entire focal range.' so it is quite possible that the MkII is better because it has a more balanced performance over the whole range. Of course, Canon fanboys, like Olympus and any other fanboys don't bother to read or think about what the numbers mean - if they don't see their pet product performing best, they just throw the toys out of the pram.

That depends on how high that uncertainty is.

The idea that SNR, DR or colour depth have little to do with actual photography is independent of the quality of the measurements of those things.

If you have a "14.0 EV" camera and a "13.0 EV" camera, but your uncertainties for each result are 0.5 EV, then the chances are that the actual delta is 0.7-1.3 EV (this depends on the distribution, or type of curve e.g. Gaussian or rectangular etc.).

Another case may be if you have a "13.2 +/- 0.5 EV" camera and a "13.0 EV +/- 0.1 EV" camera then you may wish to buy the camera with the tighter uncertainty to be surer of getting at least 12.9 EV.

Sure, but you have now thrown out every single photographic test site, not just DxO. They have their uses, you just need to be aware of their limitations. Since none of the publish the error margins, there is no way you could make that decision. In any case, it is juvenile to get obsessive about ranking cameras. Really all that matters to an individual is whether a camera (or lens) is right for them. These sites offer some reassurance that you are not getting an absolute dog.

In this case, I very much doubt it. The way you write and the quality of your analysis speaks for itself. Juvenile.

Well, your view of DxOMark represents a conflict of interest.

We've been doing quite well recently. If we want to continue to do so, you need to understand how offensive comments like that are, especially based on your previous posturing on the issue. There is no conflict of interest. I make no commercial gain from Sensorgen, nor does my professional reputation depend on it. I produced it purely because people were asking for such a thing and I could do it. It costs me hardly anything to run (one reason why the server is so slow) and its sufficient for me that it is useful for people who want to use it. If you don't, then don't. DxO is convenient in that it is the largest and best constructed source of data. No available source of data publishes error margins, and nor am I going to go into testing the cameras for myself (in that case, it certainly would have to go commercial).

Overlooking the qualitative component of the "magic numbers" -- the uncertainties -- is an odd thing for someone with a science background to do.

I'm in the same camp as Emil Martinec there:
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/

He's a better scientist than I was or will ever be, so I'm not too worried. I'm certainly not at all worried about what you think.
--
Bob

 

[Leo]

DXO tests should be considered as comparative-the same procedures, equipment and conditions. This is a very important and great value for consumers and manufacturers.

The other way would be to purchase several cameras and compare then keep one and return the other . This site as all other post comparative reviews also. DXO is technical and pro - I always consider them, this site, imaging resource and DCResource and Steve's and many other
Leo

 

[sgoldswo]

+1. The E-M5 is undoubtedly better at ISO 1600 than the NEX 7. Equally the NEX-5N sensor is superior to the E-M5 at high iso until you factor in the effects of image stabilisation. Given the paucity and slowness of most stabilised lenses for NEX most of the time in real world use you'll actually get better results from the E-M5.

There are some other odd features going on as well in the DXO mark results. For example, the G5 shares the same sensor as the GH2 but apparently has worse high iso performance despite the tweaks made in the G5 on the image processing side having been designed to improve high iso performance (amongst other things).

Finally, kep in mind DxO scores don't tell the full story. They are mostly useful to compare to eachother from one reference point, and that the importance of the categorical scores may matter differently for different uses.

For example, DxO rates the GH1 as being better than the GH2. Most practical usage, however, will show the contrary. This is most evident in the ISO score, where the GH1 beats the GH2 according to DxO, but by virtually all usage accounts, the GH2 outperforms the GH1. The OM-D scores lower than the NEX-7 for low light, but even accounting for underexposure on the OM-D, it seems to slightly but noticeably outperform the 7 in all of my trials.

Then there are other things not factored. For instance the OM-D may be worse than a 5N in low light, but having very effective IBIS means that with static subject, your images will often come out cleaner than those of a 5N with non-stabilized lenses.

DxO is a useful tool, but not the full story, as they acknowledge themselves.

 

[Anders W]

+1. The E-M5 is undoubtedly better at ISO 1600 than the NEX 7. Equally the NEX-5N sensor is superior to the E-M5 at high iso until you factor in the effects of image stabilisation. Given the paucity and slowness of most stabilised lenses for NEX most of the time in real world use you'll actually get better results from the E-M5.

There are some other odd features going on as well in the DXO mark results. For example, the G5 shares the same sensor as the GH2 but apparently has worse high iso performance despite the tweaks made in the G5 on the image processing side having been designed to improve high iso performance (amongst other things).

That depends on where you look. There's a trifling difference in the "sports" score in favor of the GH2. But for reasons that I have explained at some length earlier in this thread, that's a score I don't even look at. If you look at the DR curve (not just the DR score which is for base ISO only), the G5 does somehwat better than the GH2 and if this difference is real, you can be expected to see slightly less high-ISO shadow noise with the G5 than with the GH2.

Finally, kep in mind DxO scores don't tell the full story. They are mostly useful to compare to eachother from one reference point, and that the importance of the categorical scores may matter differently for different uses.

For example, DxO rates the GH1 as being better than the GH2. Most practical usage, however, will show the contrary. This is most evident in the ISO score, where the GH1 beats the GH2 according to DxO, but by virtually all usage accounts, the GH2 outperforms the GH1. The OM-D scores lower than the NEX-7 for low light, but even accounting for underexposure on the OM-D, it seems to slightly but noticeably outperform the 7 in all of my trials.

Then there are other things not factored. For instance the OM-D may be worse than a 5N in low light, but having very effective IBIS means that with static subject, your images will often come out cleaner than those of a 5N with non-stabilized lenses.

DxO is a useful tool, but not the full story, as they acknowledge themselves.

 

[exdeejjjaaaa]

And it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

not the same version of raw converter and we all know that the way ACR works changed significantly... like, for example, old versions had mandatory, behind the scenes, luminance NR applied even in UI you put all sliders to zero...

 

[Anders W]

And it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

not the same version of raw converter and we all know that the way ACR works changed significantly...

like, for example, old versions had mandatory, behind the scenes, luminance NR applied even in UI you put all sliders to zero...

Do you have a link to the place where this is demonstrated?

 

[Detail Man]

And it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

not the same version of raw converter and we all know that the way ACR works changed significantly...

like, for example, old versions had mandatory, behind the scenes, luminance NR applied even in UI you put all sliders to zero...

Do you have a link to the place where this is demonstrated?

When Lightroom 3.x / Camera RAW 6.x were released (in the Summer of 2010), there was a fair amount of discussion on various forums regarding differences between the amount of image-noise (and the flip-side of that being resolution of fine details) visible when the Noise Reduction controls (primarily Luminance, but to some extent Color, as well) were set to minimum values. My general impression from reading posters who seemed to have something of a head on their shoulders was that the changes did indeed appear to clearly reveal that Adobe had been applying some amount of "silent" NR in Lightroom 2.x (with some change in that relationship being noted to have occured in Version 2.8 if I remember right). Most posters appreciated the benefits of increased user control

Of course, the question of how much "silent" NR Adobe may have (still) been applying when NR (primarily Luminance) is set to Zero arose - and (not surprisingly), nobody could say for sure (as everything is relative where is comes to a proprietary application, and issues surrounding de-mosaicing also affect resolution of fine-details). Adobe's Eric Chan has recently stated that LR-4.x/CR-7.x uses exactly the same NR and Sharpening algorithms/controls as the previous LR-3.x/CR-6.x versions. That's life in the "secret sauce" lane. Nothing is (or will be made) certain

Note: It has been a while since, but I think that the most interesting and valuable discussions that I read were on the Luminous Landscape Forum. Perhaps some content on the Adobe Forum, too.

 

[Anders W]

And it's a good thing that they use the same RAW converter across the board so that this factor is kept constant.

not the same version of raw converter and we all know that the way ACR works changed significantly...

like, for example, old versions had mandatory, behind the scenes, luminance NR applied even in UI you put all sliders to zero...

Do you have a link to the place where this is demonstrated?

When Lightroom 3.x / Camera RAW 6.x were released (in the Summer of 2010), there was a fair amount of discussion on various forums regarding differences between the amount of image-noise (and the flip-side of that being resolution of fine details) visible when the Noise Reduction controls (primarily Luminance, but to some extent Color, as well) were set to minimum values. My general impression from reading posters who seemed to have something of a head on their shoulders was that the changes did indeed appear to clearly reveal that Adobe had been applying some amount of "silent" NR in Lightroom 2.x (with some change in that relationship being noted to have occured in Version 2.8 if I remember right). Most posters appreciated the benefits of increased user control

Of course, the question of how much "silent" NR Adobe may have (still) been applying when NR (primarily Luminance) is set to Zero arose - and (not surprisingly), nobody could say for sure (as everything is relative where is comes to a proprietary application, and issues surrounding de-mosaicing also affect resolution of fine-details). Adobe's Eric Chan has recently stated that LR-4.x/CR-7.x uses exactly the same NR and Sharpening algorithms/controls as the previous LR-3.x/CR-6.x versions. That's life in the "secret sauce" lane. Nothing is (or will be made) certain

Note: It has been a while since, but I think that the most interesting and valuable discussions that I read were on the Luminous Landscape Forum. Perhaps some content on the Adobe Forum, too.

Thanks for the info DM. Well, if 2.x had some minimal NR left even at zero settings, and that actually made some perceptible difference, then the old cameras should look better than expected, e.g., relative to the DxO figures. But I can't say that's my impression although I have done some fairly systematic over-time comparisons in order to see how sensors get visibly better and relate that to the info we have on how they get measureably better.

As to the "secret sauce", yes I see the downsides of using commercial software. On the other hand, there are downsides to using the open-source stuff too. DPR would be charged with the burden of developing optimal profiles for each new camera that would do justice to each of them while still maintaining comparability. A rather difficult and time-consuming task I am afraid.

 

[Leo]

The 4/3 sensor technology should be better to match a bigger sensor. The 4/3 should be a front runner, I think, If the sensor pixel count would be equal 1 then the bigger get more photons. There may be a new technology 1 photon = 2 electrons used to build 4/3 sensor and will be a winner for the time the new technology spread. Am I wrong?
Leo

 

[boggis the cat]

As I have also pointed out, Bob, the lack of uncertainties ("margins of error") makes comparisons and reuse of their data problematic.

Sure, but no more problematic than any other comparison site - they none of them publish error margins. One would not expect them to, it is not part of common practice for consumer information sites. So we get left with the situation where DxO seems to be held uniquely culpable for not publishing error margins sites such as DPR, which also do not publish error margins (and moreover have quite glaring procedural faults in their tests). It seems inconsistent that DxO is picked out.

This is not inconsistent because DxOMrk make unique claims:

DxOMark is the trusted industry standard for camera and lens independent image quality measurements and ratings. For years we have established our reputation for the best:

• Rigorous hardware testing
• Industry-grade laboratory tools
• Database of thousands of camera-lens test results

DxOMark consists of a comprehensive RAW-based image quality measurement database and a set of scores used to evaluate and compare digital cameras and lenses.

http://www.dxomark.com/index.php/About/What-is-DxOMark2

DxOMark means objective, independent, comprehensive image quality data

When looking for digital camera image quality evaluation data and benchmarks, you want to be sure that the provided measurements have been openly and objectively performed by a completely independent testing operation that uses state-of-the-art equipment and follows industry-standard methods.

http://www.dxomark.com/index.php/About/What-is-DxOMark2/Test-result-reliability

So they are claiming (and most people are accepting) that their data is uniquely rigorous -- or "correct" to put that in layman's terms.

I am not aware of any other review website making such claims.

When you are looking at data to 0.1 EV resolution then the uncertainties should really back that up. If not, then you have implicit error bars of unknown magnitude in all of the DxO data (and your sensorgen information).

And every other spurce of data. That includes that coming from people like Emil Martinec and Bill Claff.

Sure. What do you think that this indicates?

If you look at the charts that DxOMark produce they have data points to 0.01 EV resolution but still no indication of probable uncertainty.

That would be about my guesstimate, too, assuming a tightly controlled test setup. Somewhere between 5 and 20 percent.

If you want a guesstimate, that seems way too large. In the case of the sensor measurements, most of them relate to the SNR at some illumination level. A laboratory illuminant should be controllable to a fraction of a percent. If the SNR is taken using a 100 pixel square, that is a sample size of 10,000 - which will give a confidence interval for the SD (noise) of a couple of percent. So my guesstimate would be something smaller than 5%, which is 0.07 stops.

Well, we simply don't know because that sort of information is not available.

We also have little idea of "native" variability for these cameras. Again, my guesstimate would be around 0.2 EV or lower (for higher-end equipment).

You mean sample to sample variation. I have no idea - gain that is a weakness for all these sites that rely on single sample tests, which is all of them.

I mention this as it is a confounding factor for even the most rigorous (and correctly reported) test results. My assumption would be that manufacturer's take more care and have tighter criteria with more expensive equipment aimed at professional users.

I recall reading about DxO "tweaking" results before.

That doesn't make it true. Many people wish to discredit DPR for their own reasons.

Then they have the Canon EF 70-200mm f/2.8L IS II USM erroneous results, that they did revisit after enough complaints -- but did they really believe that a new design much-lauded by every other review site was worse than the original?

They haven't changed their view. they still give the MkII a DxO mark of 17 and the original 21. So the story that they 'revisited' is bunkum. The DxO overall scores should be taken with a pinch of salt anyway - as they say 'The DxOMark Score is the performance of a lens (with a camera body) for its best focal length and aperture combination. It does not show how the lens behaves over its entire focal range.' so it is quite possible that the MkII is better because it has a more balanced performance over the whole range. Of course, Canon fanboys, like Olympus and any other fanboys don't bother to read or think about what the numbers mean - if they don't see their pet product performing best, they just throw the toys out of the pram.

Discussion about lens ratings here:
http://www.fredmiranda.com/forum/topic/1106506

That depends on how high that uncertainty is.

The idea that SNR, DR or colour depth have little to do with actual photography is independent of the quality of the measurements of those things.

That appears to be beside the point.

continued ...

 

[boggis the cat]

continued ...

If you have a "14.0 EV" camera and a "13.0 EV" camera, but your uncertainties for each result are 0.5 EV, then the chances are that the actual delta is 0.7-1.3 EV (this depends on the distribution, or type of curve e.g. Gaussian or rectangular etc.).

Another case may be if you have a "13.2 +/- 0.5 EV" camera and a "13.0 EV +/- 0.1 EV" camera then you may wish to buy the camera with the tighter uncertainty to be surer of getting at least 12.9 EV.

Sure, but you have now thrown out every single photographic test site, not just DxO. They have their uses, you just need to be aware of their limitations. Since none of the publish the error margins, there is no way you could make that decision.

The same approach should be taken with DxOMark data as with any other evaluations -- if DxOMark is an outlier then disregard their data. Of course, we can't necessarily recognise an outlier due to how they present their data. While it is possible to look at several test shots and decide that "reviewer A" botched the shot, this is not possible with DxOMark's data due to no qualitative information being available to the end user.

In any case, it is juvenile to get obsessive about ranking cameras. Really all that matters to an individual is whether a camera (or lens) is right for them. These sites offer some reassurance that you are not getting an absolute dog.

IMO, most review sites offer more useful information to a potential buyer. Whether a camera achieves a marginally higher number than another is not necessarily of any consequence.

And to address your anticipated criticism -- the fact that DxOMark do not offer rounded reviews makes it all the more important that the data they do provide is able to be qualified.

Well, your view of DxOMark represents a conflict of interest.

We've been doing quite well recently. If we want to continue to do so, you need to understand how offensive comments like that are, especially based on your previous posturing on the issue.

'How well we do' in conversation is dependent upon the rationality of your approach, from my assessment.

There is no conflict of interest. I make no commercial gain from Sensorgen, nor does my professional reputation depend on it. I produced it purely because people were asking for such a thing and I could do it. It costs me hardly anything to run (one reason why the server is so slow) and its sufficient for me that it is useful for people who want to use it. If you don't, then don't. DxO is convenient in that it is the largest and best constructed source of data. No available source of data publishes error margins, and nor am I going to go into testing the cameras for myself (in that case, it certainly would have to go commercial).

There is an obvious conflict of interest if you are taking a partisan position on data that you know is flawed . Claiming that the flaws do not matter while not mentioning the fact in regard to your own re-purposed variants appears to be a deliberate oversight.

Overlooking the qualitative component of the "magic numbers" -- the uncertainties -- is an odd thing for someone with a science background to do.

I'm in the same camp as Emil Martinec there:
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/
He's a better scientist than I was or will ever be, so I'm not too worried.

"Appeal to authority" is a logical fallacy -- and in this case does not change the facts of DxOMark's unknown data quality.

I'm certainly not at all worried about what you think.

But then I am not using this data, nor presenting it to others in a modified form.