Previous news story Next news story

Roger Cicala looks at lens, camera and autofocus variation

Oct 5, 2011 at 20:00:19 GMT

OTHER NEWS LENSRENTALS

Pin It

Print view

Roger Cicala of LensRentals has written an article about lens and camera sample variation. The latest article builds on the several excellent pieces he's already written by subjecting a series of lenses to studio testing. As one of the few independent people in a position to test and assess sample variation with a reasonable sample size, it's well worth a read. As with all his posts, it does a good job of explaining and demonstrating the inherent variability and inconsistencies in lens behavior. It also addresses the reasonable desire (and sometimes unreasonable behavior) of photographers to get a 'good copy' of a lens.

Click here to read Roger Cicala's 'Notes on lens and camera variation'

And, for a little more background, we'd also recommend the articles that led up to this latest one:

'"This lens is soft" and other myths' - which looks at reasons why users may conclude their lens is faulty, even though it isn't.

'"This lens is soft" and other facts' - which looks at some of the reasons behind poor lens/camera results.

Comments

Total comments: 43

thought the comment below about Canon's calibration software was interesting.

Seems like one should ask any camera dealer to include calibration in the sale.

I also wonder if the same sort of calibration software is available to Sony's service centers??

0 upvotes

LOVE THIS REVIEW: Just right, not too much, nor too little!

Been spending the week shooting birds (some rare ones) together with the wife, me armed with my K-x (the K-5 is dead, and off to the doctor), and the wife with her Pentax K-5 (never a problem) and her Pentax 55-300, me with my Sigma 150-500. I do love this lens, even if its colours are a bit subdued!

Who got the most perfectly focused shots, who got the most keepers: Definitely my wife!

When it comes to compacts I was armed with a Sony DSC-HX9V and a Olympus XZ-1, and the wife a Sony DSC-TX5. In good light my Sony won hands down, in low light the Olympus. In really good light the Sony 9V was even better than the K-5, a bit of a surprise, but far harder to aim right. Pentax K-x, and Sigma 150-500, was a close to 100% miss affaire (against a sky, or other non-descript background), while the Pentax DA55-300 was a near 100% success affaire! With a textured background the difference wasn't that big, I'm happy to add.

Comment edited 2 minutes after posting

0 upvotes

This guy is exquisite.
Bookmarked.

0 upvotes

This is a link to 3x 35L lenses of cropped handheld shots using liveview 10x of some text. My lens I felt was faulty in comparison to the 2 new lenses I tested at my local progear store. I sent my lens in to Canon & the lens has been repaired (not sure what was done) & is as perfect as I can hope for. My lens was fuzzy from f1.4-f2.0. I tried calibrating it but nothing was sharp. I confirmed this when I tested the 2 new lenses. I am over the moon that it was fixed.
http://www.flickr.com/photos/dves/sets/72157627838484518/detail/
I haven't reshot with my lens fixed yet, but will be visiting my local store to let them know.

0 upvotes

I have been a quality engineer/manager in the military electronics field for many years, and a part-time pro photographer since I was a kid. Mr. Cicala's tests and conclusions have sound roots in the theory and practice of statistical process control. In his welcomed articles, I have not seen any hint that he is trying to elude his business responsibility. I am very grateful that he took the time to go through these experiments and shed some light on subjects that puzzle many advanced photographers. Unfortunately, natural variation is built into the fabric of the universe. Photographers with a need for very tight tolerances should be aware of this and take the necessary steps to get their equipment properly calibrated, or do it themselves. What annoys me is the fact that most producers of photographic gear do not publish their manufacturing tolerances.

Comment edited 2 times, last edit 5 minutes after posting

3 upvotes

You and the fellow below have the wrong impression (I was forced to severely edit my post, by the way, due to length restrictions and posted the first part of my edited thoughts before I figured out that I could reply to myself to create a Part 2 - a lot was left out). I sincerely believe that Mr Cicala is a brilliant business man who has found a brilliant way to mitigate what must be a true customer service headache when dealing in a product whose sample variation that IMO can be sensed quite easily by experienced users. I have no problem with his testing either. That shows he is conscientious. In fact, I expect a person in his line of work to test his stock regularly. However, his "conclusions" and implications are suspect on its face simply because of a clear conflict of interest which would place restrictions on many proceedings in our society. It is incumbent upon the reader of this article to acquire the Big Picture independently.

Comment edited 4 times, last edit 11 minutes after posting

0 upvotes

Part 2

But surely, a person with your lengthy experience in photography is no stranger to the concept of sample variation so what makes Mr. Cicala's articles particularly newsworthy? Heck anyone who has been a consumer for more than 3 days knows that sample variation is a fact of life. Again what is newsworthy here?

I mildly disagree with the implications made in the articles (he might as well adopt the old Canon defense that a given lens operates within spec) but what I really disagree with are the conclusions drawn by readers, many of whom seem to be of that species of fanboy who are annoyed when someone posts to the forum that they have a bad copy of said fanboy's favorite lens. Mr Cicala's article helps to debunk that idea. However, his articles do not present any real evidence to absolve a lens that is producing poor results.

Comment edited 7 minutes after posting

0 upvotes

Part 3

What Mr. Cicala does not have when he tests a lens is your camera and if that combo consisting of your camera and his lens does not perform as well as it could, then it does not perform as well as it could. It is what it is. Another lens is in order especially if a customer does not have MA in his camera. A better situation for discriminating customers is to rent the lens and a camera the lens has been tested on.

0 upvotes

Quote: "So for those of you who spend your time worried about getting “the sharpest possible lens”, unfortunately sharpness is rather a fuzzy concept."

I think that your tone is somewhat aggressive towards Mr Cicala. It is reasonable for his staff to weed out the lenses that are complete out lyers and return them to the Manufacturer, but the variation seen in the other samples are unlikely to be an issue for most Professional Photographers (Mr Cicala's major customer base). These tests were carried out in a controlled environment with ideal lighting conditions. In the real world out there, many other factors will play a part in peceived "Sharpness", not least the application of good technique and usable shutter speeds. In an ideal world, every lens of a specific type and from the same Manufacturer would be identical in performance. The reality of life is that this is impossible to achieve in a Manufacturing environment, especially for mass produced items such as these.

0 upvotes

Ulterior motive. Mr Cicala rents lenses. Judging from the responses, he has been quite successful in planting the seed that most lens variations a customer may encounter with his stock ain't his fault.

Before MFA, the only choices were to exchange/return a lens or send it to the manufacturer for calibration (preferably with camera). Of course, a person buying a lens had/has this luxury. With a lens rental, one usually doesn't have time for an exchange.

There is still a significant base of cameras out there without MFA or Live View which, by the way, if one does have these features, the suggestion seeks to shift a greater burden onto the rental customer to adapt to a misfiring camera/lens combo. I would think that most folks renting a lens would naturally want the sharpest "variant" available and since the only way to find this out is to mount it onto his camera, Mr Cicala should expect exchanges. Rental customers should not be made to feel that he/she are not entitled to the best.

0 upvotes

Part 2

By the way, I have not nor do I have current plans to rent a lens. But, Mr. Cicala, if the situation ever does arise, expect a request for an exchange if the lens isn't sharp or it front/back focuses. It may not be your fault, but it is your responsibility to provide me with a lens that works properly. Since you wrote the articles and have become the defacto expert on lens variation and in such capacity declared that every lens has a variation relative to the next, you have no defense against a claim that your lens ain't playing nice with a given cam. If your rental contract shifts that responsibility to the rentor, expect no future business from me for real.

1 upvote

Rick,
I completely agree with you, and if you become a customer you will see that is exactly our policy - if the equipment doesn't function properly it is replaced or refunded.

Regards,

Roger Cicala

1 upvote

Awesome!

1 upvote

Great read, as it always is when it comes to Roger's articles.

1 upvote

Love these articles. Best I have seen since college!! His findings and reporting seem impeccable and inline with my education and experience.
Nice to have a level head with the ability to technically and objectively evaluate across the board physical, financial and mechanical limitations in this field!!!
Bravo Roger Cicala

2 upvotes

This article has some interesting data on autofocus variability. Using a mirrorless camera with EVF, one of the capabilities I most appreciate is the ability to magnify the live view image. This enables me to ensure correct focus for static subjects. Rather than chasing after the best copies of a lens, photographers ought to be ensuring the subject is in focus, rather than relying on their AF.

1 upvote

I have always been skeptical of all the reported "bad copies" of lenses in the past. I found it hard to believe that an industry that makes instruments of such high precision, would have so many out-of-tolerance products going out the door.

Roger's article certainly has opened my eyes. Unfortunately, in this day of mail order, with fewer and fewer actual local camera stores to deal with, when you get a lemon you have to embark on a journey of shipping back and forth, with less guarantee, in my opinion that the replacement product will be any better than the original. I can't speak from experience having never returned a lens. I did send a P7000 in for warranty work, and while the repair facility was as prompt and responsive as it was able to be, it was still harder than it would have been to drive down to the local store and get the matter resolved on the spot.

0 upvotes

Interesting and informative article. As the Author points out, many long standing Photographers are aware of this issue. Anecdotally, I shoot a lot of architectural images and spent a lot of time back in 2008 working up a decent methodology for exploiting the technique of Tone Mapping for interiors in old buildings. At that time I did not have a camera with Live View and had to rely a lot more on the use of Autofocus while shooting the multiple frames necessary to produce a decent Tone Mapped image. I quickly discovered that if I used the Autofocus but then switched the camera to Manual before taking the multiple frames needed, that I could achieve a superior end result as far as the Tone Mapped image was concerned. I think that most Autofocus systems tend to "Hunt", especially in situations with poor lighting such as the inside of churches etc. Steve

0 upvotes

Kinda of confirms what I think many have deduced from owning or testing different copies of a lens on a particular body....what it all amounts to IMHO is not that this is the "state of the ability" of modern manufacturing but rather indicative of the void of decent quality control in an industry that is more and more bottom line price driven by consumers obsessed with only megapixels.

0 upvotes

Lenses were tested after return from rental - maybe an average of 15 rentals and 30 transits. Variations with new lenses should be much lower.
The resolution variation (excluding the defective lens) is about 7.5% for the original Canon and about 20% for the later Canon and Zeiss.
In the AF link there is reference to a point light source and the 1970 patent. Things move on in a 40 year period.
Chuck Westfall writing as head of Canon USA said "The nature of the AF sensors used by EOS digital SLR’s as well as those from other manufacturers is that *they perform most reliably when the entire length of the focusing area sees readable detail*" ie. not so good with a point source.
LensRentals to not provide a link to their AF target. Without this it is not possible to deduce whether they are using an AF target likely to result in reliable AF with a wide range of camera bodies or some mis focus issues.
Summing up - interesting information - but safe conclusions cannot be made.

0 upvotes

There are indeed many questions unanswered. For example, they mostly tested lens resolution at the center as well overall average, but they have left out issues such as decentering etc.

Nevertheless, the conclusions are sound: AF does vary from camera to camera, from lens to lens and quality also varies from camera to camera, from lens to lens. Many people assume that AA filters do not show much variation from copy of a camera to another. But their results show otherwise.

There is nothing wrong with their general conclusions. If anything, they have probably erred on the conservative side. And the problem may be more pronounced than what they demonstrated. :)

1 upvote

Interesting reading indeed. And very healthy presentation of the data.

0 upvotes

great article. I ended up reading more of his articles.

0 upvotes

I'd like to see a 100% crop of an avderage lens and a bad lens to know exactly what to look for. I'm a newbie and it would be useful as I add lenses to my kit.

0 upvotes

I think the general moral of the story is.... you will know when you have a bad lens so don't worry about it too much.

3 upvotes

Excellent, excellent article. The article's description of those who wanted the top 3 copies of a particular lense reminds me of how at Steinway Piano’s hq in NYC, there’s a special room where the world’s great pianists get to hand pick the top pianos (out of several $100,000+ pianos!) to get the one that’s sounds just ‘right’.

When I first read about Fujifilm’s choice of a non-interchangeable fixed lens in their x100, they justified it by stating that the camera’s sensor was customized just for that lens in order to allow light to strike the sensor at an angle that is as perpendicular as possible (more important for digital sensors than film sensors). Were Fujifilm’s engineers possible trying to avoid the performance variations that might exist with interchangeable lenses as described in the article?

1 upvote

off topic, but is not that logo for LensRentals NSFW? Inquiring minds want to know ...

0 upvotes

Interesting article and a useful website. Thanks for posting the link.

0 upvotes

Although many of us have observed the variations in focus accuracy of lenses and bodies, it is nice to have some data to confirm that it happens. The testing roger does is very sensitive, as it should be, so results should and do vary. The important thing to note, is that they do not usually vary enough to be a concern, once a lens and autofocus are fine tuned to a camera.

One thing i'd be interested in seeing is how AF accuracy varies with the distance to the subject, since that sometimes seems to be a bigger headache and can't be cured by autofocus micro adjustment.

2 upvotes

This should be made compulsory reading for anyone considering posting on the forums so that it might reduce the amount of inane chatter and kvetching about "soft" lenses and the endless ramblings of the "sharpness" obsessives.

3 upvotes

I read Roger Cicala's blog posts regularly. They are always well-written, informative and entertaining.

2 upvotes

OK DSLR manufacturers please get rid of the source of error by using the same image sensor as AF sensor like the new Nikon J1 series.

Or give us the micro-adjustment feature on all models. It is a software feature. Don't hide away from us.

0 upvotes

Excellent research methodology. Great report.

2 upvotes

No error bars.

The data points are openly cherry-picked: the best of a subset of test results are kept for each lens.

The resolution measurement is critically dependent on focus accuracy and alignment.

The spread of values obtained for non-defective lenses is close to or within the experimental error, actually the author admitted as much when he noted that resolution figures for the best measuring lenses could not be reproduced.

For all practical purposes, you either have a defective lens or a working one.

1 upvote

I agree, this is a great idea and from someone in a position to make a reasonable assesment. But whenever you look at reviews you have to consider the source. One thing this article doesn't say, but has always been the case, is that most people don't know the difference. And then there's the fact that the majority of SLRs sold have smaller sensors and those folks never even see the corners. When I worked in the photo industry, reviewers did not get hand picked samples (well, rarely). However, if they got a bad test result, they would just try additional lenses until they got a good result. As a footnote, the info on AF may explain why Cosina/Voightlander lenses and PC lenses perform so well--they are always properly focused.

0 upvotes

In the not too distant past professional photographers would ask the lens manufacturer's representative for the sharpest lens they had in stock. They would accomodate this service to a select few photographers. What do you think happens when a new lens is released and tested by a pro photographer? They get the best of the batch. When Nikon supplied NASA cameras and lenses for the space shuttle I'm sure they went over everything for optimal quality.

1 upvote

but that is NASA.

0 upvotes

Very cool article indeed.

Would be interesting to test CDAF as well.

Confirms what most already know - if you have a few seconds to shoot, manual focus is king.

0 upvotes

This confirms a fewthings for me, and one of the unsung reasons why i'm so happy with m43 - CDAF - the focus is in a different league than the PDAF on my old 40D.

Once the issue of better CAF is solved, I wonder if Canikon will move to a similar system, after all, the returns and services must cost them a fortune, else why would they have included micro-adjust.

0 upvotes

This article is the most intelligent and useful article ever to appear about photographic equipment on dpreview or possibly anywhere else. Instead of presenting uninformed opinion as fact, Roger Cicala, has done some systematic testing of lens and camera body variabillity, individually and in combination. Here's someone who is worth listening to, a person with a very fine scientific mind. He presents his data so you draw you own interpretations. Also note that he does not go beyond his data, and when he speculates he says so.

With this short article he has raised the average IQ of dpreview posters by at least 30 points. We desperately need more articles of this caliber.

9 upvotes

And here I thought that my auto focus problems at f/1.4 was all my fault. This explains a lot :)

0 upvotes

Nice article that confirms what many people who know something about mass-produced items already knew. Nice to see some example data that shows how these things work and what's involved in performing a valid test of lens performance.

0 upvotes

After dealing with a very poor lens Canon 35 f1.4L & 3 visits for repair - the lens is now spot on. I found this article answers a lot of my questions around calibrating which appears so unrepeatable in real world use that I don't bother. There is a reason why F8 was always a pro shooters minimum aperture for studio work. Sports is something else & it would be very frustrating if focus was always missed. Still when you add a bit of distance to f2.8 the depth of field is not that small!

1 upvote

Total comments: 43


Roger Cicala compares Zeiss and Canon 135mm f/2 lenses Apr 29, 2013	Roger Cicala investigates Canon's AF marketing claims Aug 1, 2012	Roger Cicala looks at the Canon 40mm F2.8 STM pancake lens Jun 21, 2012	Grizzly bear tests limits of rented lens insurance cover May 21, 2012

	Pentax Efina Jun 3, 2013
	Olympus PEN E-P5 May 10, 2013
	Samsung NX2000 May 1, 2013
	Sony Cyber-shot DSC-HX50V Apr 24, 2013
	Panasonic Lumix DMC-G6 Apr 24, 2013
	Panasonic Lumix DMC-LF1 Apr 24, 2013
	Ricoh GR Apr 17, 2013
	Samsung NX1100 Apr 11, 2013
	Panasonic Lumix DMC-GF6 Apr 9, 2013
	Fujifilm FinePix S8400W Mar 22, 2013
	Fujifilm FinePix XP200 Mar 22, 2013

Sobering Sales Numbers for Mirrorless	150
Hmmm AF fine tuning : Bullsh#t	82
New Pentax K-50 leaked	77
Mirrorless & DSLR Jan-April shipment & ...	47
How to turn my Canon PowerShot S3 IS into a ...	44
Thom Hogan: DSLRs going obsolete	42
K5(II) SD Card Door Coming Open	42
Sigma 60/2.8... sigma be damned (for its ...	41
Help me create the best 6D system on a $3500 ...	39
Still no signs of an Olympus f2.8 zoom?	39
Selecting Camera	39
electronic first curtain shutter - on or off?	37
USPS... The worst mail servie in the world	36
Replacing Olympus OM-D with Fuji XE-1 or Xpro-1	33
micro 4/3s , Owls IF	32
Done Deal - Switched to Fuji XE-1 from ...	32
Too many pixels ? so why not reducing it ...	31
What's Happening to Sony Stores?	31
Focal reducers, why complexity for the sake of ...	30
32mm f/1.2 to start shipping on June 13 !	29

Your email address:
Your name: