Dear DPreview either your lens reviews are wrong...

Started Oct 24, 2010 | Discussions
nugat Contributing Member • Posts: 699
Dear DPreview either your lens reviews are wrong...
Scales USA Veteran Member • Posts: 3,121
Re: Dear DPreview either your lens reviews are wrong...

So you go around referring to ignored posts you made in another forum by adding links to them. Wanting attention? If you have a question, use the contact DPR link.

 Scales USA's gear list:Scales USA's gear list
Fujifilm MX-700 Canon G1 X II Canon EOS R
OP nugat Contributing Member • Posts: 699
Re: Dear DPreview either your lens reviews are wrong...

This is a serious site owned by a serious investor (Amazon).

I ask some questions about the integrity of their tests that influence purchasing decisions of many people.

If there is something factually wrong with my inquieries it is very easy to point out.
Eg.

Why Nyquist lines in the resolution charts disappeared, where should they be, and why lp/ph were changed to lw/ph? Why Imatest references disappeared? Why resolution tests of camera bodies differ significantly from those of lenses?
Is it really so difficult to answer?

Thank you in advance. Not looking for attention, this is the last question I have about that.

Scales USA wrote:

So you go around referring to ignored posts you made in another forum by adding links to them. Wanting attention? If you have a question, use the contact DPR link.

foot Veteran Member • Posts: 3,878
one users opinion

the dpreview "help" pages give a good overview on how to use the lens test widgets. The help pages can be accessed via the "help" button on the bottom of the widget, but i'll link to it here:
http://www.dpreview.com/learn/?/Glossary/Lens_Reviews/introduction_01.htm

i did surf around them and didn't see a reference to imatest. In the past i've tried reading the imatest pages but frankly i found them very confusing

this site seems to have more understandable explanations, plus it mentions how to test your own lens, using some free software. That sounds cool, i haven't tried it yet
http://www.normankoren.com/Tutorials/MTF.html

the dpreview lens test widget seems thoughtfully designed. Since i never could understand imatest i find the information it displays much more usefull

 foot's gear list:foot's gear list
Sigma dp2 Quattro
OP nugat Contributing Member • Posts: 699
Re: one users opinion

DPreview use Imatest AFAIK. But since so much changed recently in their tests maybe this factor too. Would be good to know.

Steen Bay Veteran Member • Posts: 7,418
Re: Dear DPreview either your lens reviews are wrong...

nugat wrote:

This is a serious site owned by a serious investor (Amazon).

I ask some questions about the integrity of their tests that influence purchasing decisions of many people.

If there is something factually wrong with my inquieries it is very easy to point out.
Eg.

Why Nyquist lines in the resolution charts disappeared, where should they be, and why lp/ph were changed to lw/ph? Why Imatest references disappeared? Why resolution tests of camera bodies differ significantly from those of lenses?
Is it really so difficult to answer?

Thank you in advance. Not looking for attention, this is the last question I have about that.

Good questions. I was wondering about the same thing in this subthread :

http://forums.dpreview.com/forums/read.asp?forum=1022&message=36716790

OP nugat Contributing Member • Posts: 699
Re: Dear DPreview either your lens reviews are wrong...

It's good to know somebody else noticed.

Resolution tests' results vary from site to site, because there is no standardized method. As long as the results are homogenous within a site that's understandable, we get at least a relative ranking of glass. But here something happened that changed the paradigm. Units changed , Nyquist disappeared, the resolution trumpets from camera test are not in line with the "widgets". In a while those trumpets perhaps will disappear too. Until some editor cares to explain what is going on, the resolution tests conducted by DPreview cannot be considered reliable.

Steen Bay wrote:

nugat wrote:

This is a serious site owned by a serious investor (Amazon).

I ask some questions about the integrity of their tests that influence purchasing decisions of many people.

If there is something factually wrong with my inquieries it is very easy to point out.
Eg.

Why Nyquist lines in the resolution charts disappeared, where should they be, and why lp/ph were changed to lw/ph? Why Imatest references disappeared? Why resolution tests of camera bodies differ significantly from those of lenses?
Is it really so difficult to answer?

Thank you in advance. Not looking for attention, this is the last question I have about that.

Good questions. I was wondering about the same thing in this subthread :

http://forums.dpreview.com/forums/read.asp?forum=1022&message=36716790

Andrew Westlake Senior Member • Posts: 2,928
Re: Dear DPreview either your lens reviews are wrong...

http://www.dpreview.com/learn/?/Glossary/Lens_Reviews/frequently_asked_questions_01.htm

In short:

1) We recently realised that we'd been labelling the MTF scale of the lens widget incorrectly - we were calculating MTF50 in lw/ph, not lp/ph (as the graphs were labeled).* We've corrected the scale, but as a direct consequence the Nyquist line has disappeared.

2) Camera reviews measure the point at which lines on the test chart blur into each other, which isn't the same thing as the MTF50 numbers in the lens reviews, but instead more like MTF10. This is fully expected to give different numbers.

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

-- hide signature --

Andy Westlake
dpreview.com

 Andrew Westlake's gear list:Andrew Westlake's gear list
Fujifilm FinePix F30 Zoom Olympus Stylus Verve S Olympus XZ-2 iHS Canon EOS 5D Panasonic Lumix DMC-GF1 +11 more
J A K Forum Pro • Posts: 15,833
I always thought you were a classy guy.

Andy Westlake wrote:

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

The civil, humble, no excuse reply above confirms my high thoughts of you. It's too bad some of your fellow DPR co-workers are not more like you when errors are pointed out.

Do you suppose DPR will ever apologize for publishing camera reviews using per-pixel comparisons (as versus per-picture)?

Regards,

Joe Kurkjian

Galleries: http://www.pbase.com/jkurkjia

-- hide signature --

SEARCHING FOR A BETTER SELF PORTRAIT

Steen Bay Veteran Member • Posts: 7,418
Re: I always thought you were a classy guy.

J A K wrote:

Andy Westlake wrote:

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

The civil, humble, no excuse reply above confirms my high thoughts of you.

Absolutely.

Do you suppose DPR will ever apologize for publishing camera reviews using per-pixel comparisons (as versus per-picture)?

That a bit more difficult. Should we compare the 'normalized' noise and detail in images downsampled to for example 8 MP instead?

OP nugat Contributing Member • Posts: 699
Re: Dear DPreview either your lens reviews are wrong...

Thank you.

Now your figures are more in line with sites that show 40lp/mm at MTF50 as good performance lens.

Not that this method is any better (for the lack of agreed testing standards) but at least shows consequence.

Andy Westlake wrote:

http://www.dpreview.com/learn/?/Glossary/Lens_Reviews/frequently_asked_questions_01.htm

In short:

1) We recently realised that we'd been labelling the MTF scale of the lens widget incorrectly - we were calculating MTF50 in lw/ph, not lp/ph (as the graphs were labeled).* We've corrected the scale, but as a direct consequence the Nyquist line has disappeared.

2) Camera reviews measure the point at which lines on the test chart blur into each other, which isn't the same thing as the MTF50 numbers in the lens reviews, but instead more like MTF10. This is fully expected to give different numbers.

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

KerryBE Regular Member • Posts: 120
Re: I always thought you were a classy guy.

Regarding the per pixel vs per picture. Is one wrong or just preferrable? Are they mislabelling something? Seeking to understand your point better.

-- hide signature --

Kerry Erington

 KerryBE's gear list:KerryBE's gear list
Sony RX100 Canon EOS 5D Mark II Sony a6000 Sony a7R II Canon EF 17-40mm f/4.0L USM +11 more
bobn2
bobn2 Forum Pro • Posts: 61,165
Re: I always thought you were a classy guy.

Kerry B Erington wrote:

Regarding the per pixel vs per picture. Is one wrong or just preferrable? Are they mislabelling something? Seeking to understand your point better.

IMO, the publishing of 100% crops was honest, in that they said they were what they were - however, 100% crops are not a very good aid for the visual comparison of cameras doing the same job (i.e. making equal sized images).

The problem was more in the supporting narratives, which would often suggest that a camera was 'noisier' because it looked 'noisier' when compared in a 100% crop. This came to a head in the infamous 50D review, which Joe did some good work debunking.
I think it is rare to see the same mistake in the narrative now.

I think it would be good if DPR could include (maybe in addition to the 100% crop) a crop sampled to some standardised pixel count. one useful one would be a 100% crop from a virtual A3 print (suppose now DPR's a US site it would have to be some inch paper size), which is an the largest 1:1:414 section of the sensor, resampled to 18MP, using some standard and declared resampling method (I suggest Lanczos with appropriate antialiasing pre blur if downsampling). While this wouldn't be the same as everyone would get from an A3 print, where the resampling would be done by their own print drivers, it would serve as a common reference point for both noise and detail in a print that size. (pixinfo.com used to do a nice job, unfortunately, they seem to have gon AWOL)
--
Bob

Lee Jay Forum Pro • Posts: 54,147
Re: Dear DPreview either your lens reviews are wrong...

Andy Westlake wrote:

2) Camera reviews measure the point at which lines on the test chart blur into each other, which isn't the same thing as the MTF50 numbers in the lens reviews, but instead more like MTF10. This is fully expected to give different numbers.

How about publishing MTF10 (or MTF9, which is a special case) too?

-- hide signature --

Lee Jay
(see profile for equipment)

 Lee Jay's gear list:Lee Jay's gear list
Canon IXUS 310 HS Canon PowerShot SX50 HS Canon EOS 7D Mark II Canon EF 70-200mm F2.8L IS II USM Canon EF 8-15mm f/4L Fisheye USM +23 more
Izu Contributing Member • Posts: 662
Re: Dear DPreview either your lens reviews are wrong...

Andy Westlake wrote:

http://www.dpreview.com/learn/?/Glossary/Lens_Reviews/frequently_asked_questions_01.htm

In short:

Is there any official statement about this "embarrassing error"?

bobn2
bobn2 Forum Pro • Posts: 61,165
Re: Dear DPreview either your lens reviews are wrong...

Andy Westlake wrote:

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

were you the one that got left behind in London as a punishment?

-- hide signature --

Bob

omr Senior Member • Posts: 1,118
Re: Dear DPreview either your lens reviews are wrong...

Izu wrote:

Is there any official statement about this "embarrassing error"?

See the 5th FAQ item here:

http://www.dpreview.com/learn/?/Glossary/Lens_Reviews/frequently_asked_questions_01.htm

-- hide signature --

omr

Izu Contributing Member • Posts: 662
Re: Dear DPreview either your lens reviews are wrong...

omr wrote:

See the 5th FAQ item here:

Thanks!

Steen Bay Veteran Member • Posts: 7,418
Re: Dear DPreview either your lens reviews are wrong...

bobn2 wrote:

Andy Westlake wrote:

*Obviously this is a rather embarrassing error, and we'd like to apologise to readers for any confusion caused.

were you the one that got left behind in London as a punishment?

Or could it be that this whole Seatte thing was the punishment?

Prognathous Veteran Member • Posts: 9,258
Re: Dear DPreview either your lens reviews are wrong...

Andy Westlake wrote:

1) We recently realised that we'd been labelling the MTF scale of the lens widget incorrectly - we were calculating MTF50 in lw/ph, not lp/ph (as the graphs were labeled).* We've corrected the scale, but as a direct consequence the Nyquist line has disappeared.

Is the Nyquist line going to make a comeback? It was useful to estimate the resolving power of the lens vs. the sensor limitation. As it is now, it's more difficult to tell how good lenses really are.

Prog.
--
http://www.flickr.com/photos/oren_b

 Prognathous's gear list:Prognathous's gear list
Sony a77 II Tamron SP AF 60mm F2 Di II LD IF Macro Tamron 18-270mm F3.5-6.3 Di II PZD Sony DT 16-50mm F2.8 SSM Olympus Stylus 1s +13 more
Keyboard shortcuts:
FForum MMy threads