How much weight do you give DXOMark reviews?

Reilly Diefenbach said:

Bodies yes, lenses not at all. DXO does a composite voodoo score for lenses which is not useful to me. Cameralabs is far better for lenses, testing using real world subjects at different distances.

Click to expand...

I fully agree.

Best Regards,
Renato
'The world is going to pieces and people like Adams and Weston are photographing rocks.' Henri Cartier Bresson, in the 1930's

I rate them pretty high. Of course it's not perfect, but I appreciate the methodology that lets one compare.

javiair said:

Just asking how much do you give the reviews from a site like DXO and photozone? I know that the final decision is mine however I do like an unbiased opinion on camera equipment that Im looking to buy.

I have noticed that when a score is low for a camera that one likes the numbers are biased or scewed or didnt factor in all the numbers. An Example would be the Canon 6d or the 5d m3 vs the Nikon D610. The scores are not even close. But Im sure one could argue that the Canons are still far better despite the stats or say that DXOmark has to be taken with a grain of salt.

Click to expand...

the Score is meaningless since it weights things and is only for bragging rights

javiair said:

What do you guys think about sites like DXO or photozone that gives stats and graphs.. Are they legit scores or still very subjective?

Click to expand...

Stats and grapps are important and dxo is seemingly consistent across brands in measuring

and since they measure the ISO versus the stated ISO it points out the manufacturers who 'cheat'

Unfortunately, they only test one unit and some people make a big thing about small differences which are likely just sample variation

I find their reviews to be vapid and nearly useless without the guts to be honestly critical about anything.

However, their actual data can be very useful. One of my favorite pieces of data is the Field Map section under the Sharpness metrics for a given lens + body combo. This is a great way to find out where you can count on high/low sharpness across the frame for different settings of the lens. This is not nearly as consistent and linear as many people assume and can be the big difference between two lenses.

For example, the Nikkor 35mm f/1.4G and the Sigma 35mm ƒ/1.4 DG seem to be right on par with one another in photozone.de tests. But, the real story is in the field map data at DxO that reveals that the measurements seem similar due to the places where photozone takes them in the frame. The Sigma in this comparison has a much larger high resolution area when shot at large apertures than the Nikkor which makes it far more useful as most photographers avoid sticking their subject matter dead center in the frame.

Another interesting piece of data is the field map on the Nikkor 24-70. You can see that it goes down then back up in sharpness at a point while moving away from the center.

There is info about CA and other things as measured all over the frame as well. There is a ton of interesting data if you dig in and if you care. If you don't care about this data... then I don't know why you're even looking at lab tests.

ZTJ said:

I find their reviews to be vapid and nearly useless without the guts to be honestly critical about anything.

However, their actual data can be very useful. One of my favorite pieces of data is the Field Map section under the Sharpness metrics for a given lens + body combo. This is a great way to find out where you can count on high/low sharpness across the frame for different settings of the lens. This is not nearly as consistent and linear as many people assume and can be the big difference between two lenses.

For example, the Nikkor 35mm f/1.4G and the Sigma 35mm ƒ/1.4 DG seem to be right on par with one another in photozone.de tests. But, the real story is in the field map data at DxO that reveals that the measurements seem similar due to the places where photozone takes them in the frame. The Sigma in this comparison has a much larger high resolution area when shot at large apertures than the Nikkor which makes it far more useful as most photographers avoid sticking their subject matter dead center in the frame.

Click to expand...

Interesting comment and well spotted!

A possible explanation is that the Nikon 35 F1.4G is designed to render literally "around" the more central subject and so "draw" a more pleasing image that "pops" out from the background, especially wide open or near to it. The penalty, of course, is the lack of "sharpness across the frame".

The very sharp Sigma F1.4 ART is designed to produce a flatter more even image across the frame and so can record more "faithfully and sharply" across more of the image. The downside of this approach is usually a "flatter, more two dimensional" image.

These are two very different and deliberate design goals and neither is better nor worse than the other IMHO.There is no absolute "best" lens made and due, to the optical trade-offs involved, there never will be with known technology. Its like marriage, I guess... its about compromise ;-)

What DxOMark doesn't tell you is that they do their testing, most likely, around 40X the focal length of the lens; so for a 50mm lens that testing is done, most likely, of a flat, two dimensional test chart of some kind at a distance of around 2 metres. And they use some sort of "pattern" to shoot in different parts of the image and that data is fed into the lens profiles for their acclaimed software.

If you are listening, Monsieur / Madam DxOMark, please reveal to us your full testing regimen and we will need to guess no more! Other sites do it, why don't you... even McDonalds reveal the ingredients in their "special sauce"... and the calories too! :-O

What DxOMark's, and most others, testing methodology does not tell us is that the excellent Siggy 50 ART loses its "sharpness" advantage at distances of around 5 metres and beyond compared to some others...

But, isn't it just great that armed with this sort of information we can then choose the lens(es) that is the closest fit for our specific needs? For example, I'm a very rank amateur these days but I own 2 x "50s" and 2 x "35s", not to mention the number of zooms I have that can give me those focal lengths...

ZTJ said:

Another interesting piece of data is the field map on the Nikkor 24-70. You can see that it goes down then back up in sharpness at a point while moving away from the center.

There is info about CA and other things as measured all over the frame as well. There is a ton of interesting data if you dig in and if you care. If you don't care about this data... then I don't know why you're even looking at lab tests.

Click to expand...

javiair said:

Just asking how much do you give the reviews from a site like DXO and photozone?

Click to expand...

How much weight I give to them? Zero. Never looked at them before I made up my mind for any camera. Would NEVER make a decision based on their conclusions and tests, as a matter of fact I really don't care if another person or another "test" site likes or dislikes whatever I have. Why bother? Who is a better judge of what fits my needs then I am?

What are you going to do if you'd buy what they recommend you to and it turns out to be not as you expected? Sue them? Equally, what are you going to do if your experience is more positive and totally opposite to their conclusions?

They use a value they refer to as perceived sharpness. How can perceived sharpness be used to empirically compare two lenses? It can't.

There process is very flawed for zoom lenses as they will take a zoom lens and do their evaluation at the middle of the zoom range. They should know that a zoom lens provides the lowest IQ at the widest angles so evaluating a 24-120mm zoom at 50mm is really a useless exercise.

None. Period.

javiair said:

Just asking how much do you give the reviews from a site like DXO and photozone? I know that the final decision is mine however I do like an unbiased opinion on camera equipment that Im looking to buy.

I have noticed that when a score is low for a camera that one likes the numbers are biased or scewed or didnt factor in all the numbers. An Example would be the Canon 6d or the 5d m3 vs the Nikon D610. The scores are not even close. But Im sure one could argue that the Canons are still far better despite the stats or say that DXOmark has to be taken with a grain of salt.

What do you guys think about sites like DXO or photozone that gives stats and graphs.. Are they legit scores or still very subjective?

Click to expand...

For cameras, a review only serves to confirm my decision to buy it. If the review doesn't confirm it, it really doesn't matter. For lenses, I do read reviews, and allow them to influence my buying decisions.

Example: Recently purchased the Sigma 120-300 2.8 Sports, based on DXO's rating on a D600 and D610. I also read the review on photozone.de. On a D610, the Sigma, was rated very highly and was one of the telephoto zooms recommended for that camera. Because it fit my other needs (already decided on as appropriate), I purchased it. I have the lens for a week now. Its performance confirms what I've read about it.

Example 2: The Nikon D4s just came out. It specifications, are exactly what I want in a Camera for sports. No need to look at a review for that decision.

In deciding that a camera is "Legit, " after reading a review that doesn't give it a great score, one has to look at the reader's investment in the system. If he has an extensive investment in Canon, he may justify his decision to buy Canon over Nikon, because of that investment.

It's one source that I depend on to get an objective eexamination of camera equipment. Dxo uses scientific methods of observation to come up with their numbers. I think it's about as good as one could expect from a laboratory. For lenses, I use SLRgear because their charts give me a better view of how the lens works over the various zoom ranges and f/stops. If you agree with their methodology for one brand, then you must agree for all brands because the method is the same.

None other than general direction - e.g. I see a lens mentioned that sounds as though it may be of interest. If every person who reviewed it said it was pants, I would probably not bother to investigate more.

Certainly I would never buy or not buy just because some stranger said yea or nay.

...?

Please take a look at this, a real life test with the D800 vs MKIII. You can take it for the D610 as the sensor has a lot of DR as well:

http://www.fredmiranda.com/5DIII-D800/index_controlled-tests.html

--

www.ramonvaquero.com

Slapbass said:

...?

Please take a look at this, a real life test with the D800 vs MKIII. You can take it for the D610 as the sensor has a lot of DR as well:

http://www.fredmiranda.com/5DIII-D800/index_controlled-tests.html

--

www.ramonvaquero.com

Click to expand...

Oh dear....!

Numbers don't take pictures and don't make the picture either.

Some so called top scoring lenses perform well but sometimes have a lack of '' magicness '' and are simply plain boring.

Sharpness is only one part of the equation, sometimes, the way the light travel thru the lens is what makes for a magical picture regardless it scores good or not.

just my 2 cents

[No message]

I ask this mainly for the lenses and reviews, especially the Nikkor 28-300VR lens for the D610.

In photozone.de, DXO, and Camera labs, the lens is getting panned saying that its not sharp at all and I read on one site asking "why would anyone want to own this lens?" As an amature I do listen to what the photographers, who make their living off of taking pictures, say. I do also realize it is subjective but when most sites say its not good enough to make money, then I will give concideration to that.

However on the other side I do see that photographers love the 28-300mm VR, they say that despite the reviews it is sharp, not as sharp as a prime or the kit lens but it does a fine job.

I just hate changing lenses (who doesnt hate it) and this lens is very attractive.. but is it worth the $1000 or will I just have to spend another $1300 later on down the road if the chance to make money presents itself? And get the 70-200mm F4 which has high marks.

I have learned so much from everyone here in these forums everyone is just a wealth of knowlege and I look forward to continue to learn techniques, styles, lighting, but for now, just asking on purchasing.

Any test which is based on a 1000:1 contrast target, as many are, is of academic rather than practical use.

It is extremely rare to encounter a 1000:1 contrast subject in the real photographic world.

Most photographers do not shoot only 4x3 foot subjects, which is close to what many 1000:1 targets are.

Results at a more common 125:1 contrast, or in extreme close up, or at long distances can vary significantly independent of the 1000:1 3x2 foot test performance.

Most tests do not provide enough information to make an informed decision though they are of some value in making a decision.

If in doubt and it is expensive my advice is hire and compare it to what you already use.

Leonard,

Thank you for the reply... I had to re-read it a few times but I think I understood what you were saying. From what you said.... testing is done under optimal conditions with a perfect subject..where in real life it is not..

javiair said:

From what you said.... testing is done under optimal conditions with a perfect subject..where in real life it is not.

Click to expand...

It depends what you mean by "optimal" conditions. To me there is no point in relying solely on "optimal" if it is never encountered in real life photography.

In days have film when we got 1000:1 and 1.6:1 figures they differed significantly between films. The chances of either a 1000:1 or 1.6:1 subject in real life photography is close to nil.

Kodachome was never top at 1000:1 but at 60:1it beat anything else. This made Kodachome an outstanding choice in overcast lighting.

What I want to know is how good is a lens is at different focus distances and at different contrast levels.

No site seems to do all of this

Digressing slightly, in the UK I get the chance to shoot with any Nikon lens I want every 3-6 months. This helps narrow down whether a new lens is good for me or not.

Off topic 90% of the work I produce is used at image sizes where the sort of differences being considered in this thread are irrelevant because the differences are too small for the human eye to perceive. I know my Nikon 24-70 is not brilliant in the corners of FX at 24 mm f2.8 if I make a 30 inch wide print, but it is easily good enough at these settings for 99% of PJ work.

Does it matter that the image below originated about 25 years before D&) existed?







--
Leonard Shepherd
Producing good quality photographs, or being good at sport or art, involves a little more than buying appropriate equipment. Practice, some learning and perhaps natural talent often play a bigger role than the equipment in your hands.

Cliff Fujii said:

It's one source that I depend on to get an objective eexamination of camera equipment. Dxo uses scientific methods of observation to come up with their numbers. I think it's about as good as one could expect from a laboratory. For lenses, I use SLRgear because their charts give me a better view of how the lens works over the various zoom ranges and f/stops. If you agree with their methodology for one brand, then you must agree for all brands because the method is the same.

--
Cliff

Click to expand...

Cliff, you make a good point. On the other hand, two wrongs do not make a right and if the methodology is flawed... :-(

I also read DxOmark and value their camera sensor scores very highly. IMHO, this aspect is the best in the business.

When it comes to lenses, unlike other sites, DxOMark seem to use a "special formula / algorithm / sauce" to compute their "scores". No one knows for sure (maybe not even DxOMark) exactly what "weighting" is placed on "what aspect" of a lens performance.Their observations are very likely "scientific" but their computations are just as likely "voodoo logic"... we just do not know! :-( They are not "transparent" with us.

So how can I make an informed decision when I don't even know how precisely they arrived at their conclusions? I guess I could just follow their advice blindly, I suppose...

Actually, I study at least 6 of the "better, IMHO" review sites, look at a lot of pictures and consult widely. Its tough but everyone probably has a slightly different need and that is what must be satisfied in the end. As far as I'm concerned, when it comes to choosing my lenses, it is all about me! :-D