Variation Facts and Fallacies

What is normal variation?

When we at Lensrentals started using computerized analysis (we use the Imatest package) to assess the MTF of large numbers of lenses, it became obvious that there is sharpness variation among copies. The example below shows the groupings for several different 100mm lenses: six samples of the Carl Zeiss Makro-Planar T* 2/100, eight samples of the Canon EF 100mm f/2.8 Macro USM, and twenty-four samples of the Canon EF 100mm f/2.8L Macro IS USM.

This graph plots center sharpness against a sharpness figure for the whole lens (measured in line pairs per picture height for unsharpened RAW images)

Two things should be apparent from this chart. First, there clearly is variation between the different copies of each lens. On average the Canon 100mm f2.8 IS L lenses are the sharpest, but some copies are sharper than others. And some copies of the other two lenses are sharper again. You can see how this might lead two different reviewers to hold slightly different opinions on which 100mm macro lens is the best.

Second, a true 'bad lens' is truly an outlier, and you can see a bad one way down on the lower left. The difference between a soft or bad copy and the main group is very large. The copy-to-copy variation that occurs between the other lenses is really minor. If you want to know how bad that bad copy is: our techs could identify it looking at JPEGs at 50% on a computer screen, but they'd be unlikely to spot it by looking at a web-sized image.

A second point about copy-to-copy variation must be made: the data above were for all of those lenses on one single camera body. As I've already explained, when you change the body the results change - the overall pattern will look the same, but each lens result will be slightly different. The sharpest lens on test camera A may well not be the sharpest on test camera B.

The example below shows two things. The green triangles represent multiple different samples of a lens shot on the same camera (notice that the range on both axis has been changed to illustrate the difference more clearly). We then took one of those lenses (circled) and shot it on 11 other camera bodies - the performance is represented here by the blue and red squares. As you can see, there is significant variation in the results from the different cameras.

A single lens (circled) was re-tested on eleven camera bodies (red boxes and blue diamonds)
from two different serial number series. All were brand new from box.

As suggested by our name, Lensrentals is a lens rental house, and as such I'd like to clarify one point: the variations that Ive been talking about don't occur because we're looking at used lenses. When we first started testing lenses, we made sure to carefully compare brand new lenses with our stock lenses to make sure that our quality assurance was keeping the stock lenses in good shape.

We buy new lenses in significant quantities; so we usually start testing a given lens by examining a dozen new copies right out of the box, then we start testing our rental copies. The graph below compares a group of new-out-of-the-box lenses with a group from rental stock to give an example. But in the graph immediately above, all of the camera bodies were new stock. 

We’ve only found one lens so far that did show a difference with age, but results from that aren’t used in any of these illustrations (I have discussed this in detail in this article).

 Focusing Variation

To give some idea of how small the difference between various lenses really is, we can examine the variation of a single camera's autofocus with a single lens. It's reasonable to assume that you shoot dozens of images with your camera and depend on autofocus to be accurate. You never notice, unless you look very carefully, that if you automatically focus on the same shot several times the camera focuses slightly differently each time. That’s why for testing purposes the data points I’ve shown were obtained using the best possible live-view manual focus for each lens.

But let’s look at the graph below, which consists of multiple autofocus shots made with one lens on one camera. First, I let the camera autofocus the lens once, turned off autofocus, and took 4 consecutive shots, represented by the blue diamonds in the graph. In theory they should be identical, since nothing was changed. In reality, they are just a little different, which demonstrates the amount of shot-to-shot variation in the testing method - very little, in short.

Without changing anything, I then moved the focus ring to one extreme or the other, let the camera autofocus again, took a single shot, and repeated that process six times. Those six shots are the red boxes. You can see the camera autofocuses the shot a bit differently each time (and this is with center focus point on a star chart: a near ideal situation). Finally, I focused the lens using live-view magnified focusing four consecutive times (represented here by the green triangles).

A single lens shot repeatedly with no changes (blue), manually focused 4 separate
times (green), and autofocused 6 times (red).

The scale in this series is changed again to illustrate the point clearly. It should be obvious that the variation of the camera’s autofocus system on repeated shots with a single lens is about 1/3 as great as the variation between different lenses. Autofocus microadjustment to the camera would improve the average AF performance to be nearly as good as the live-view focusing but it wouldn’t change the shot-to-shot variation (it would shift the red cluster towards the green results but it would remain just as spread out).

So How Big is This Variation in Real Life?

That’s really the point of all this. The variation I’ve described is easy to detect using modern computerized optical analysis. And we’ve seen that the variation between a truly bad lens and the group of acceptable lenses is very large. But how big is the variation between the different acceptable copies?

The best answer I can give is probably that it might be detectable by pixel-peepers, but not working photographers, at least hardly ever. Lensrentals has six full-time pixel-peepers on our staff (the inspection technicians) who are armed with a large array of well-lit charts to analyze sharpness, aberration, back and front focus, you name it. They can pick out a bad lens fairly easily, but can’t tell the difference between the best and worst images in the groups above.

But there’s a more scientific way to look at things than 'our techs said so.' The Subjective Quality Factor (SQF) is a measurement developed by Ed Granger and K.N. Cupery in the 1970s for Kodak and used by Popular Photographer for their lens reviews. Basically, SQF uses a mathematical formula, taking the MTF data from the lens (which we get from these tests) to predict with good accuracy how sharp a print would be perceived at various sizes and distances.

I’m not going into detail about SQF (for a more thorough discussion, see Bob Atkins' excellent article or the references below). The important part is that several experts have shown an SQF difference of less than 5 for a reasonably sized print is basically not detectable by human vision.

It’s a simple matter to have the computer calculate the SQF from the data we’ve already obtained in our testing. We arbitrarily calculate it for 8 X 10 inch print size, but the SQF difference would be the same for any reasonably sized print.

The graph below is for the same lens that I used for the autofocus example above, plus I added several other copies of that lens tested on the same camera. Then I had the computer calculate the SQF data for the best and worst copies. 

SQF for 8 X 10 inch print taken from selected MTF points.

As you can see the variation in SQF is less than 5, meaning theoretically, you wouldn’t be able to detect the difference in a print. The difference is real and can be detected by Imatest, or by a careful pixel-peeper armed with a few test charts, a large monitor, and too much time on their hands. But it wouldn't be significant enough to make an obvious difference in a print.

So What’s the Point of All This?

The main points are fairly straightforward:

1) Every lens and every camera exhibits slight variations relative to its twins that are detectable, but rarely significant.

2) Variations that wouldn’t make the slightest difference in a print may seem quite different when the numbers are presented in a lens review. And, just because one copy of lens X is sharper than one copy of lens Y, doesn’t mean they all are, or that they all will be in your camera.

3) Occasionally, an acceptable lens mounted to an acceptable camera combine their variations in a way that makes them unacceptable together. The lens may be fine with a different camera, and the camera fine with a different copy of the lens. 

4) Really bad, soft, out-of-acceptable range lenses do occur. They are fairly rare though and easy to detect.

5) Camera autofocus is more variable and less accurate than you think.* 

* Before you go all Major League Fanboy about the superiority of your camera’s autofocus system: autofocus variation exists in every camera from every brand we’ve tested. Want to prove it? Put a wide aperture prime on your camera, mount it to a tripod, and focus on something in the middle distance. Now move the focus ring to infinity, let the camera autofocus and note exactly where it ends up on the distance scale. Now turn the focus ring to near focus, let the camera autofocus on the subject at middle distance, and note the number on the distance scale. They will be slightly different.

When you buy a lens, and assuming your camera allows you to, you should microfocus adjust it. If you do it properly, using a sensible focus distance, it really does make a difference. Then do some very basic tests to make sure it functions properly, and go take some pictures. If you like the pictures it makes, then keep it.

Oddly enough, the conclusion I’ve reached from several years of dedicated pixel-peeping and lens analysis is this: Trying to find exactly the sharpest copy of the sharpest lens is a fool’s errand: you’ll be looking for something that doesn’t exist.

Roger Cicala, Lensrentals.com 


For more of Roger's thoughts on lens and camera variation - and much, much more, visit his blog

The views and opinions expressed in this article are those of the author and do not necessarily reflect the views and opinions held by dpreview.com or any affiliated companies.

Comments

Total comments: 231
123
CMDPHOTOS
By CMDPHOTOS (Oct 19, 2012)

Great article! However, I do have one problem. At what point did we decide that a $1500-$30000 lens shouldn't be "almost" perfect?I am sure there are those out there that think even $1500 is not a lot of money, but I'm not one of them! What ever happened to pride in workmanship or QC? Not to mention when did it become acceptable to spend this kind of money for a crap shoot (pardon the pun)? What if you can't afford to buy 3 different $5000 lenses to even try to hit a good lens? Are the tolerances displayed here something that should be acceptable? Yes, I do think I should be able to feel comfortable that my "only" $1500 lens will come to me in usable condition. If they can make one, they should be able to make sure that the odds are better than they apparently have. Buy the way, I would be willing to pay a reasonable price to have a new lens shipped to you first for a bench test before coming to me. No offense, but it is sad that I would have to go to this extreme to begin with.

1 upvote
ManVan123
By ManVan123 (Sep 29, 2012)

I Loved this articole !!!! Nice one.

0 upvotes
mikeon
By mikeon (Oct 18, 2012)

article

0 upvotes
Borek Lupomesky
By Borek Lupomesky (Jan 14, 2012)

I have EF 200/2.8L lens that's visibly softer on one side (but only at 100% pixels maginification). I have 5DMarkII that has slight magenta cast on the right side of image (but you need to shoot something really uniform and white to notice it). Well, nothing is perfect it seems, and I really decided not to worry about this too much.

0 upvotes
shahid11235
By shahid11235 (Jan 13, 2012)

"Tollerance value" is certainly an issue.. and NOTHING is "absolutely perfect", I agree. And if the manufacturers set a high precision level to minimize these types of flaw, then production cost must will increase which'll ultimately reflect an impact on price. This article might make some "perfectionists" understand why they can't get a "perfect" body or lens.

Another issue, one sample out of ten or a hundred is tested (I assume one is chosen randomly from a batch) for QC.. That's acceptable for a 150$ point-n-shoot camera, but for an 1200$ DSLR body or 1k$ lens? That doesn't make any sense.

Why don't you guys on dpreview run a test on various brands about their quality control and publish a detailed review on that? I think that'll help lots of photographers save both their time and money as well as energy?

Thanx :)

2 upvotes
Droppingin
By Droppingin (Dec 12, 2011)

My laptop ran out batteries before I finished the end

I made three posts: the first is (Dec 12, 2011 at 21:48:37 GMT) two posts down should be read first.

I just wanted to say that I am not attacking the author but my goal was to fill in the blanks or information that completed the picture of goes into the process control. Maybe because of background in the mfg'ing business the article, to me, painted a picture where things were not shown as controlled as they truly are. So I hope this balances it.

I did leave out in-process qc and final qc. It is one Hell of a lot more than testing samples these days. Specially when it comes to electronic equipment including things like lenses.

Crap, I just checked and one of my posts was lost and cannot remember what I said; probably said enough already.

Comment edited 3 times, last edit 10 minutes after posting
2 upvotes
Droppingin
By Droppingin (Dec 12, 2011)

If their is a change in supplier, the whole process is done again.

In the mfg' process, highly accurate and constrained jigs and fixtures are used to assemble parts such that the absolute minimum operator error impact is allowable. Knowing Canon, Nikon and others use robotic and dedicated automated machines to the critical assembly work.

With respect to screws, these are not critical items because if joining two parts are critical, an indexing tool pin is used which can be accurate to +/-0.001" or better depending on what is required. Screws simply help hold things together, not to locate two critical items. I am not saying the author made this claim but some folks may interpret his statements as such.

The modern machining processes are very accurate. Modern mfg'ing uses something called True Position analysis which when used properly and throughout an assembly and its subassemblies can produce amzaing results that surpass the traditional bias tolerancing. More in next post.

0 upvotes
Droppingin
By Droppingin (Dec 12, 2011)

I know I am not the only one that is/was a Mechanical engineer working electronics manufacturing or that we have the market corner on opinions on this topic. But here comes the however.

Generally speaking everything in the article is reasonably accurate but it does give rather bleak picture on the state of manufacturing and tolerance analysis.

I assume Canon, Nikon, and others have state of the art mfg'ing capability. They most likely use computer aided design software capable of sophisticated tolerance analysis before any material is cut or formed. That is, they have a good projection where there the problem areas will be.

All suppliers parts have to meet new part/supplier qualification programs that include: source(this means inspecting the supplier's plant) inspections, source quality control inspections, incoming part qualification programs, and ongoing parts-receivable inspections. I will finish in next post.

0 upvotes
Reactive
By Reactive (Dec 6, 2011)

What a superb article! I wonder if a photgrapher inspects and adjusts his masterpiece in Photoshop (i.e. pixel-peeping), then does the sharpening or automatic lens compensation made in Photoshop immediately outweigh and override any small errors made visible by a particular lens/camera combination?

It might be ideal to capture a perfectly focused shot 'in camera', but does software sharpening make the issue of tolerances largely irrelvant for most practical purposes (i.e. the perceived visual sharpness)?

Comment edited 59 seconds after posting
2 upvotes
fvdbergh2501
By fvdbergh2501 (Dec 6, 2011)

In theory you can undo blurring perfectly, but the conditions required for this do not occur in actual images (you need the PSF of the imaging system, zero noise, no quantization error, etc.). In practice this means that software sharpening introduces artifacts, which may or may not bother you.

A related issue is that the tolerance problems may introduce systematic bias (like Roger described above), which shows up as front- or back-focus. This means that the sharpest point of focus in your image may not be where you want it to be (imagine a portrait with the sharpest focus on the tip of the nose). You can only repair such a defect in software by selectively sharpening and blurring just the right areas of the photo, something that would become tiresome very quickly.

0 upvotes
S Hendrik Seelen
By S Hendrik Seelen (Dec 1, 2011)

@Roger
As you report live-view AF to be more precise, i.e. with less variation to it, I wonder what your data is on contrast- versus phase-based AF cameras--anecdotal and statistical. Do you get less complaints from four-thirds users than from dslr users? Did you do any experiments on 4/3's lens focussing? Are there any general "truths" to be gleaned on the difference between the systems?

0 upvotes
fvdbergh2501
By fvdbergh2501 (Dec 1, 2011)

If you want to evaluate the sharpness of your own lenses, or you wish to calibrate the AF fine tuning of your DSLR body, you should try MTF Mapper

http://sourceforge.net/projects/mtfmapper/

This utility is totally free, and source code is provided. Or you could buy Imatest for $300-$5000 :)

Please read the user documentation thoroughly, though.

I would appreciate any feedback, and welcome all discussion!

2 upvotes
Irakly Shanidze
By Irakly Shanidze (Dec 1, 2011)

Excellent write-up.

0 upvotes
ennemkay
By ennemkay (Dec 1, 2011)

very interesting analysis and presentation of data.

0 upvotes
Frank C.
By Frank C. (Nov 30, 2011)

silly pointless article, this read isn't going to change people's behavior one iota, his time would have been better spent reviewing some photo gear

Comment edited 4 times, last edit 4 minutes after posting
0 upvotes
Mr  Roygbiv rainbow
By Mr Roygbiv rainbow (Nov 30, 2011)

Great ! I am now paranoid about the nano mearsurements of machined surfaces that can effect optical alignment of lens/camera fit of my next purchase !

0 upvotes
Jim Lowell
By Jim Lowell (Dec 1, 2011)

Hey Frank C..
The article is interesting and does explain some things about why every lens is not perfect.

2 upvotes
PaulBearer
By PaulBearer (Dec 1, 2011)

Less pointless than at least one comment...

9 upvotes
HeezDeadJim
By HeezDeadJim (Dec 6, 2011)

I hope you don't watch movies, then. Watching them and then discussing what problems you had with friends and family over said movie isn't going to change one iota of why family member B thinks it's still a great movie. Instead of watching pointless movies (any movie) could be better spent taking photos. Because we all know that addressing/discussing certain topics is not worth ANY time than it being spent on being productive. Sleep is overrated.

2 upvotes
Jim Lowell
By Jim Lowell (Nov 30, 2011)

Good eye-opener article. I guess metal lens mounts on lenses are better than plastic ones to lessen wear and tear and maintain some accuracy over usage and time. Live and learn!

0 upvotes
AJs Dad
By AJs Dad (Nov 30, 2011)

Really good article and just about what I would expect from Roger (I've been a LensRentals customer for quite a few years). Lots of straight talk and unbiased opinion based in real world experience.

4 upvotes
AbrasiveReducer
By AbrasiveReducer (Nov 29, 2011)

Hi Roger - Since we're talking about pixel peeping and focus fine tuning, can you explain why you say the 28-75 Tamron is "...the best lens nobody knows about, and possibly the best bang-for-the-buck of any camera lens, period. Its nearly as sharp as the Nikon 24-70 , particularly on DX cameras, but much smaller, lighter, and cheaper...." I realize your remark about DX hints that the Tamron has soft corners but this seems like reading between the lines. Variations? I've had 4 of these lenses, with and without the motor and none has been particularly good. Granted, the alternative costs a lot more but the difference is equally dramatic; the 24-70 has sharp corners on FX.

0 upvotes
RCicala
By RCicala (Nov 30, 2011)

My take on the Tammie is several years old, written when the vast majority of photogs were shooting crop bodies (as opposed to less of a vast majority today, I guess). It is weak in the corners on an FX but on a DX I feel it's a bargain: 80% of the performance for 30% of the price. But more than that I was coming from the other direction: in the under $500 lens category there's this and the Tamron 17-50 as the f/2.8 zoom entries. Both are decent lenses at an amazing price, but neither is going to compete with a $1,500 lens wide open.

Roger

Comment edited 1 minute after posting
0 upvotes
taotoo
By taotoo (Nov 29, 2011)

Very informative article (as were your previous ones), but a couple of queries:

"If they are out-of-spec and get by quality control, then someone, somewhere gets a truly bad camera or lens. It doesn’t happen frequently, but it happens. When it does, it isn't a subtle call; it's very obvious the lens is bad."

This doesn't stand up to reason - it suggests cameras/lenses SLIGHTLY out of tolerance always get caught by quality control.

"The reality though is that the only sensible definition of 'best' is 'best with the camera body you are using,'."

And best at your chosen focal length, aperture, focus distance etc...

0 upvotes
RCicala
By RCicala (Nov 30, 2011)

I didn't mean to infer that slightly out of tolerance lenses get caught. Poor wording on my part. Rather I was suggesting that an occasionally awful lens does get by. Not often, but sometimes.

And good point about focal length, aperture, etc.

Roger

0 upvotes
taotoo
By taotoo (Dec 1, 2011)

Then I wonder why your data doesn't show any slightly out of tolerance lenses?

Assuming your interpretation of your data is correct, then it backs up your statement - that when there's an error the error is obvious.

But common sense suggests that it is more likely that a highly out of tolerance lens would be caught by QC than a slightly out of tolerance lens, or at least that there would be a significant number of slightly out of tolerance lenses appearing.

Maybe there is some explanation e.g perhaps when a lens receives a knock during delivery the alignment inevitably moves out by a large amount (and never a small amount).

0 upvotes
Summit
By Summit (Nov 29, 2011)

Thank you for providing data! Perhaps consider statistical analysis? (for fun)

0 upvotes
ljgude
By ljgude (Nov 29, 2011)

Great article that confirms much of what I have picked up over the years. I own a Kodak Rapid Rectilinear from about 1900 and one of its 'perfected' descendants - a Zeiss Tessar from the 30s. Yet Stieglitz, Adams, Weston all shot with lenses of that general quality. I shot Nikons in the 60s and 70s and had real need of a short zoom, but was plagued by the softness of Nikons 43-86 zoom until one day a student showed me a sharp 8 by10 he claimed he shot with the 43-86. I tested it; it was sharp! Here is a reverse example of the thrust of the article and clear evidence that things have improved. I recently reentered photography with a 600D and was wowed by the sharpness of the 18-55 kit. Next I bought a Canon IS 100mm macro on reputation. With my rheumy old eyes I really am not sure what sharpness is anymore but I can see a difference between the two lenses. Objects look solider, more real with the macro...more like ,well, Weston's 8 x 10 contacts.

0 upvotes
Hiram P Bumblesniger
By Hiram P Bumblesniger (Nov 29, 2011)

What are you measuring on these graphs? There are no labeled axes. Without telling us what the graphs are really showing, how can we assess anything? Please add x and y axis labels. The graphs are pretty meaningless without them. Thanks.

0 upvotes
RCicala
By RCicala (Nov 29, 2011)

Hiram,

Trying to keep graphs in the allowed image size made the labels too small so I identified the axis in the text: Both axis are MTF50 in Line Pairs / Image height. The horizontal axis is center point resolution, the vertical is average resolution of the entire lens.

0 upvotes
hansaus32
By hansaus32 (Nov 29, 2011)

How do I search on dpreview.com for cameras with the front/back focus (aka microfocus adjustment) feature?

0 upvotes
Richard Butler
By Richard Butler (Nov 30, 2011)

At present you can't, but I'll put in on the list of things to consider adding to the database.

0 upvotes
SteveHageman
By SteveHageman (Nov 29, 2011)

Well done article - I would love to have a blow up of that cut away lens hanging on my wall. How do they fit all that stuff in there? Can you imagine designing that in the days before computers?

0 upvotes
jimkahnw
By jimkahnw (Nov 28, 2011)

Roger confirms my intuition on equipment build quality--a "broad" range of tolerance. I went through a quest with two copies of a Nikkor 16-85 zoom. I told tech support that images were "soft" at short focal lengths and with distant subjects the lens never hit the infinity mark. They looked at my samples and said they were within tolerance. I disagreed and they serviced the lenses without improvement. I sent them back with the camera body, a D300, again failure. So, I took equipment to the Nikon repair station in Melville, NY and pleaded my case to the techie. I even shot a series of images of the Nikon headquarters to demonstrate the problem. When the equipment was returned, lo-and-behold, the lenses were sharp. I don't know what repair was performed. As I understand it, a lens calibration with the camera is only good for one focal length; useless on a zoom. Oh, and the lenses still do not hit the infinity mark on distant subjects and short focal lengths. So much for zone focus.

0 upvotes
jack1a
By jack1a (Dec 5, 2011)

I find the main article interesting however I feel quality should never be an issue it never was with film SLR's you knew what you were buying, you could also focus and set DOF and subject position within DOF from the lens barrell markings. I have never been happy with either of my crop sensor DSLR's (30D&40D&50D) with my L lens even the "kit" lens were "Soft" no pixel peeping required. I also was told by canon that there was nothing wrong twice with all bodies and 3 different L lens. So I experimented because I needed a work around and happened on a set up where i could consistantly repeat the annomallies and which helped me figure out the workaround. I foolishly believed it would improove with next model body but I believe it is a fundamental mismatch of 2.8 L series lenes to crop sensor bodies that canon can't admit.
No 60D for me I have enough paperwieghts I have resurected the F5 and some fuju 100 slide film

BTW Jim I can't get anything sharp in infinty either

Comment edited 2 minutes after posting
0 upvotes
AbrasiveReducer
By AbrasiveReducer (Nov 28, 2011)

This is good to know and not surprising. But I still think it's unintentionally misleading. I've never used a 100 macro of any brand or price that wasn't at least very good. However, I've had many dreadful wide angle zooms that no amount of variation/calibratrion could explain away. Since the consumer can't control the variation, the first question has to be "Are ALL copies of a certain lens, dogs?" It's helpful to know that certain lenses, by design are lousy--and this includes some very expensive ones. My mother would make a great lens reviewer because she would say "If you can't say anything nice, don't say anything at all".

0 upvotes
Marek Rucinski
By Marek Rucinski (Nov 28, 2011)

Fantastic article, thanks a lot!

I don't even dare to ask if you're willing to disclose all your lens testing data ;)

0 upvotes
DrugaRunda
By DrugaRunda (Nov 28, 2011)

the next stage is for someone like DPR to take 10 lens copies of each 70-200 f2.8 lenses

Canon L IS
Nikon VR
Sigma OS (Nikon + Canon version)
Tamron (Nikon + Canon version)

and test on 10 bodies Nikon 3Dx and 10 Canon 1Ds MKIII

So we have 30 lenses x10 bodies for Canon and Nikon = 600 tests

Only after that we would have it all settled, not.

The article should be named "There can be only one!".

2 upvotes
cbsva
By cbsva (Nov 29, 2011)

Not to forget Sony, Olympus, Pentax and all variations between FF and APS-C bodies... ;-)

0 upvotes
Antony John
By Antony John (Nov 29, 2011)

No actually it would seem that many camera users would like the camera manufacturers to come knocking on their door to check if the particular copy of a lens they have will suit the particular camera the photographer has before putting it on a shop shelf somewhere in the world.

0 upvotes
MalbikEndar
By MalbikEndar (Nov 28, 2011)

> something like 150 +/- 2 MHz

Are you sure you don't mean KHz? 150 MHz is very implausible for an ultrasonic motor.

0 upvotes
RCicala
By RCicala (Nov 28, 2011)

You are absolutely correct - I typoed that. Roger

0 upvotes
Musiclady
By Musiclady (Nov 28, 2011)

Interesting reading, but also filled with some hot air. Solder in a circuit that doesn't conduct as well as some other brand or type of solder? The amounts of solder used in these boards is so small as to have no significant or measurable effect on conductivity.

With lenses as with cameras, you usually do get what pay for, including tighter tolerances. Plus or minus 20% is probably way beyond the tolerance range for Leica glass and probably wider than Canon's L lenses. Also, the better lenses are made of materials that have a similar co-efficient of expansion and contraction so that the critical alignments are held in position regardless of whether it's cold or warm/hot weather.

For me, the bottom line is that the "soft" or inaccurate lens is the rarity compared to the lenses that perform just the way they're supposed to. I also think out-of-focus or front/rear focus issues have more to do with human error and/or a maladjusted camera AF system than they do with the lens itself.

1 upvote
mlackey
By mlackey (Nov 28, 2011)

It is HOW the solder is applied. Having 10 years experience In the automotive electronics world, a "small" quantity production line may produce around 200K modules per year, a "large" quantity line in the 1M range. Any manufacturing defect, no matter how small, is bound to surface.

Consider flow soldering: minor flaws show up sooner or later as manufacturing defects, no ifs, ands or buts about it. The defect can surface as intermittent, variable resistance, etc. It may not even surface unless the printed circuit board (PCB) is flexed (or mounted) in a certain way at a certain temperature. Solder itself is subject to manufacturing tolerance, and some brands may not perform (adhere) as well as a (chemically) different brands. PCBs have manufacturing tolerance, ICs have manufacturing tolerances... the effects of all this are multiplicative, and can add up to staggering odds AGAINST producing a perfect product each and every time.

Hot air... I disagree.

Mike

4 upvotes
veroman
By veroman (Nov 29, 2011)

You're both right ... except, as always, the proof is in the pudding. The number of lenses or cameras that are out of spec due to variations in solder application or solder type or anything else related to solder, including solder along the PCBs, is so small as to make it a moot issue. In fact, the amount or error that can be attributed to solder application is so small as to be immeasurable ... at least, on a lens or camera PCB. There was far, far, far more error when solder and components were applied by hand and there is far, far, far more error (or tolerance swing) with integrated circuits than with solder application. I believe the mention of the solder is part of the "hot air" of the article.

0 upvotes
mlackey
By mlackey (Nov 30, 2011)

"...so small as to be immeasurable ..."

Depends on scale. My guess is you've never seen an automobile dashboard that's been melted by a "thermal event".

If I use this same logic, atoms do not exist because they are "...so small as to be immeasurable ..."

Hot air... I disagree.

Mike

0 upvotes
HeezDeadJim
By HeezDeadJim (Dec 6, 2011)

It's not questioning if errors exist or not, it's about which errors are going to factor considerably the end result. A sniper shooting a soda can from 300ft is probably going to notice scope error first over trigger wobble. Trigger wobble error exists and will incur shake, but not noticeable at 300ft. And atoms do exist. Without Atom and Eve, we wouldn't be here...

0 upvotes
mlackey
By mlackey (Dec 7, 2011)

"It's not questioning if errors exist or not, it's about which errors are going to factor considerably the end result."

So you are saying that if the probability of an error is small it can be ignored? Sorry, but I can't accept that answer. Any error, no matter how small, is too large if you are the one that happens to experience it.

Mike

0 upvotes
mike16
By mike16 (Nov 28, 2011)

Could someone comment on 'random' softness that sometimes occurs with zoom lens. Most pictures with lens fine but perhaps once in about 1-200 times an area towards middle righthand side is noticably soft, no pixel peeping required. Has happened with a couple of mid range zoom lenses I've had, once in era of film and one digital. I take thousands of shots per year so this is a significant amount of shots lost and it seems unpredictable as to when it will happen, just at that particular zoom/focus combination which you can't find again to reproduce the effect. Do those computers they use to simulate how the lens will perform cover every possibility of focus/zoom and if they find a particular very small combination of focus/zoom that happens to be soft does it still get through into production?

0 upvotes
tbcass
By tbcass (Nov 28, 2011)

Leonard Shepherd; screw diameter tolerances can certainly effect IQ variances because it allows variations in lens placement from lens to lens. For example one lens element may be slightly to the right while the next may be slightly to the left.

0 upvotes
tbcass
By tbcass (Nov 28, 2011)

TORN, that is the reality pf economics. If you expect manufacturers to hold lenses to the tolerances you want be prepared to remortgage your house to pay for just one. The biggest problem I have with these gear forums is the ease with which people can zoom in on photographs to find and complain about minute IQ problems that have nothing to do with the quality of the photograph.

1 upvote
TORN
By TORN (Nov 28, 2011)

First I follow and appreciate Mr. Cicalas articles for quite some time now.

But with this evaluation of a 20% difference between best and worst acceptable copy being just fine the manufacturers should be very happy.

Follow the argument with the quality rating being barely within the 5-limit then what about enlargements? I prefer to often cut my shots sometimes giving away half of the pixels and thereby increasing the error by a large amount.

In my book a sample variation of 20% is not acceptable to me. At least I want to reach the middle ground and this can usually (!) be archieved by testing 2-3 samples (well knowing that by really bad luck I test three mediocre samples, but hopefully my experience helps me out in this case). Since I am not looking for the top of the best my goal can be reached with a very limited extra effort per lens buy. Ok, the risk of having a really off camera is still there but can be mitigrated by testing my other cams as well.

0 upvotes
julieng
By julieng (Nov 28, 2011)

20% difference may not be as big or significant as the number suggest.

0 upvotes
Leonard Shepherd
By Leonard Shepherd (Nov 28, 2011)

It is an interesting article - but how reliable is it?
Roger uses an example of a screw diameter - screw diameter does not affect assembly tolerance, though it might cause tightening issues.
In "whats it all about" point 5 there seems to be a misunderstanding of how AF works. If starting AF front infinity and then minimum focus gives different results the size of the parallel part of the target is not big enough relative to the length of the AF detection line.
What is happening is the AF is detecting detail further away coming from infinity and closer from minimum focus that would not occur with a bigger target. This is not "tolerance variation".
I agree lens and body variations likely to show in a 20 inch wide print are rare. If you have fast broadband this link I got from dpreview indicates the various electronic test stages in assembling a Nikon 24-70 - and includes a Nikon AF test target.
http://lens-club.ru/public/files/pdfs/4b4fb97547c5d54819491c07c715f2c2.pdf

0 upvotes
RCicala
By RCicala (Nov 28, 2011)

Leonard, for the example I use the same result occurs focusing on a star chart on a flat wall: there's nothing in front of or behind the target for the autofocus system to detect.

Roger

0 upvotes
paqman
By paqman (Nov 28, 2011)

EXCLLENT article - thank you! have suspected major take aways for years but you've given all the backup.

1 upvote
singveld
By singveld (Nov 28, 2011)

what about lapping? Just like people who lap the heatsink so that there is better contact with processor. Therefore, if there is a way to accurately measure the tolerance. The mount can be lap by sand paper.

0 upvotes
ulfie
By ulfie (Nov 28, 2011)

There is also thermal expansion and contraction of parts to consider as well as centering and the optical effects of multiple lens coatings. As Suzuki-Roshi said, "Seek perfection in imperfection."

0 upvotes
harold1968
By harold1968 (Nov 28, 2011)

quite correct

however "manufacturer tolerance" is key

I presume Canon L lenses have lower tolerance then normal ones ?

Now I am also using Leica. Leica's lenses on my M9 are bank on razor sharp from the factory.
Lower "tolerances" ? probably .......

1 upvote
Tord S Eriksson
By Tord S Eriksson (Nov 27, 2011)

A few months ago I bought a very old Pentax-M 400/5.6, which didn't work too well with my Pentax K-x, nor the K-5, but excellently with my new Sony NEX-5N, and its K Mount adapter!

Somehow that mix turned up far better, but you have explained it excellently ;-)!

Even with my wife's E-PL1 things seemed better with the 400 (Novoflex K Mount adapter).

Yours,

Tord

1 upvote
kimvette
By kimvette (Nov 27, 2011)

Somehow, I think even with the stats and analysis posted here in this article, we will still have daily "my lens is soft" and "I exchanged five copies of my foomatic f/2.8 lens before I got a sharp one" threads.

My used 18-55mm was sharp out of the box but the others were a bit soft - but sharp in liveview with contrast AF. A few minutes with a printed target and microadjust cleared it right up - no complaining and exchanging lenses nonsense because obviously given microns' worth of tolerance, even a "perfect" lens would no longer be perfect after being handled by UPS.

I'd love to see an article by Cicala covering high-end DSLR image "softness" compared to P&S and even entry-level DSLRs, along with an analysis of the underlying causes and reasons for it. And yet, that still would not stop the nonsense.

1 upvote
mick232
By mick232 (Nov 27, 2011)

The "nonsense" is there because many manufacturers don't allow AF adjustment except in high-end bodies.

3 upvotes
kimvette
By kimvette (Nov 28, 2011)

But even high-end body owners post that they keep exchanging lenses until they get a good one. It is a topic that comes up from time to time and explaining what the impact of even tighter tolerances when it's already in the micron range would have on the price of glass, they still insist that they should not have to adjust for front/back focus (aka microfocus adjustment).

The tolerance on these things is so tight that a lens that is "perfect" on your camera may not be perfect after being subjected to UPS's mishandling of packages, and one that is not "perfect" from the factory very well may be "perfect" for your body when it arrives.

IMHO If it is within the +/- range MFA allows for, it's a keeper (and it's very rare it won't be within that range).

1 upvote
photo nuts
By photo nuts (Nov 28, 2011)

Mick232:
a) MA does NOT correct misaligned elements in lenses
b) In their current implementation, Nikon and Canon MA do NOT work well for zoom lenses.

0 upvotes
mick232
By mick232 (Nov 28, 2011)

Yes, but we are not talking about misalignment here (which would be a defect) but about lenses that are within specifications.

I agree that it doesn't work so well for zoom lenses. But current MFA is far from perfect, isn't it? At least I'd want to have different correction values for the long and short end instead of just one.

0 upvotes
mbrobich
By mbrobich (Nov 27, 2011)

Excellent article. Just went thru all of my lenses and tuned them to my liking !!

0 upvotes
sherwoodpete
By sherwoodpete (Nov 27, 2011)

Why not use the correct word "sample" rather than "copy" which implies plagiarism or worse?

0 upvotes
CaseyComo
By CaseyComo (Nov 27, 2011)

Pixel-peeping, meet semantic quibbbling. You'll get along great.

11 upvotes
SteveHageman
By SteveHageman (Nov 29, 2011)

Dictionary.com lists as a definition for "Copy" as,

"one of the various examples or specimens of the same book, engraving, or the like."

So the authors use of the word "Copy" is within the definition of the word.

3 upvotes
em_dee_aitch
By em_dee_aitch (Nov 27, 2011)

.... Continued from last post: So when you say that your technicians can easily detect a "bad lens" but not distinguish among the acceptable "variations" in your chart, I think you are conveniently omitting cases in which the lens may be within the acceptable variation at the test point within the frame (or even on average) and yet be yielding images which are to some degree easily detectable as of asymmetrical quality to the naked eye. I bring up this omission because it is the primary mode of failure that I believe reasonable people can easily spot without going so far as Imatest or even charts.

Comment edited 1 minute after posting
0 upvotes
RCicala
By RCicala (Nov 27, 2011)

Actually I completely agree (and a great example of the same definition meaning different things to different people). But I consider theses one of the easily detectable "bad lenses". Optically it's readily apparent as a portion of the image not nearly as sharp as the rest. Imatest shows it as acceptable peak sharpness with bad weighted average sharpness.

But I think you've given me a nice lead for a new article: I can definitely do some examples of this.

Best,

Roger

0 upvotes
em_dee_aitch
By em_dee_aitch (Nov 27, 2011)

Dear Roger - Thanks for this new piece in your series. I enjoyed the new information contained here. However, I have one bit of constructive criticism. You go briefly into the aspect of visible decentering (i.e. you mention the phenomenon of some lenses being off target), but you did not go into the obvious effect this has on focus. I am concerned that you downplay this a bit in that this is one of the focus effects that can make a huge visible difference to a working photographer, and it happens more often than any of us would like in lenses like the Nikon 70-200 VR2. The issue is that when an element is notably off center you will get extremely sharp focus in one part of the intended focal zone and dull focus in another part of intended focal zone. Another way of putting this is that the focus zone is skewed rather than perpendicular. Often this reproduces with the lens on multiple camera bodies, i.e. is a true lens issue and can't simply be blamed on lens/body "combination." Cont...

0 upvotes
dopravopat
By dopravopat (Nov 27, 2011)

Maybe lenses should have MFA themselves. That would be great.

1 upvote
RCicala
By RCicala (Nov 27, 2011)

The interesting thing is lens MFA wouldn't be hard: each lens contains electronic hook-ups inside that the manufacturer uses with proprietary software to adjust focus. It wouldn't fix a decentered or tilted element, but it would make autofocus more accurate if we had the option of accessing those electronics.

0 upvotes
WIMorrison
By WIMorrison (Nov 27, 2011)

An excellent article which is entirely objective, and proves fairly succinctly that the subjective comments about lenses posted in the forums are essentially bunkum.

6 upvotes
kombizz0
By kombizz0 (Nov 27, 2011)

No wonder time to time I found soft images.
Not sure what did you mean by microfocus adjustment. How it works and how can I do it.
Thanks

0 upvotes
MarcusGR
By MarcusGR (Nov 27, 2011)

In real-life situations there is at least another element which - in theory - might affect lens performance: temperature. When using a camera outdoors the temperatures' range may easily exceed 40 degrees (Celsius), and both plastics and metal will be heavily affected ...
Then, of course, there will be slight accidents: most of us will occasionally end up shocking our lenses and cameras both inside or outside our bags ...
The only solution I can foresee is regular calibration : a truly professional camera should be equipped with several calibration options, both a the camera-body level and at the lens level. And the whole system should be regularly re-calibrated, after exposure to shocks or temperature extremes expecially.
Alternatively, I think software might be developed to "virtually" re-calibrate a system by adjusting the electronic manipulation of the collected image ?

1 upvote
Desert Cruiser
By Desert Cruiser (Nov 27, 2011)

Best article yet! I didn't realize all that was involved and now I know. Thanks, very informative article. All the years we've used D-SLR's we've never found one that was unacceptable, this explains why, we were lucky with bodies and lenses. And like you said people are too critical in many cases. With all that said, we've only found one lense (on a micro four thirds) that has an anomaly that we decided was unacceptable, to us anyway. We could never get anyone to tell us if this was the case or if we were maybe expecting too much? Please take a look at this video and tell me if this something that you'd consider as acceptable? I would love to know......
http://vimeo.com/25066664
We've had other cameras with video capabilities but none that had this problem, we believe it to be in the lens, as it clicks when mildly shook off the camera like something is loose in the lens. I'd really appreciate an expert's opinion.
Thanks in advance,
Don....

Comment edited 2 minutes after posting
0 upvotes
RCicala
By RCicala (Nov 27, 2011)

Don, that clicking is definitely not normal. Something is loose in that lens. Every time we've opened up a lens like that we've found a loose screw, or even a loose element.

0 upvotes
Desert Cruiser
By Desert Cruiser (Nov 27, 2011)

Thank you, Panasonic said it was normal, and so did a couple of people here on the forum. I'm still going to use the camera but the lens is a cheap one. I really think they should have fixed it, but they didn't, so we'll have to look for another better quality lens. Thanks now I know I was correct in my assumption, thanks again for a great article.

0 upvotes
jj74e
By jj74e (Nov 27, 2011)

At the end, what does it mean by microfocus adjust it? How exactly does one do that?

0 upvotes
SteveHageman
By SteveHageman (Nov 29, 2011)

Some middle to upper range changeable lens cameras now allow you to "tweak" the nominal focus point for an attached lens. Plus or minus some amount. The camera then remembers this setting with this specific lens (usually by lens model, not by lens serial number). The Canon 7D has this adjustment while the Canon 60D does not.

0 upvotes
Antony John
By Antony John (Nov 27, 2011)

For those who think that perfection can be reached by throwing money and QA at something think again.
The most obvious optical example can be summed up in one word - Hubble.

1 upvote
tagomilonga
By tagomilonga (Nov 27, 2011)

You are wrong!
Hubble main lens was made by Perkins Elmer.
NASA awarded contract to PE because their bid was $1m less than Kodak's. PE cheaper bid did not include testing resulting in $1m saving. NASA was fully aware of this but took the risk and paid for it $1.5b in subsequent repairs in space.

3 upvotes
SteveHageman
By SteveHageman (Nov 29, 2011)

NASA also didn't spend the $$$ to roll the Hubble outside and focus it on anything. This was recommended by the engineers, but alas... Pay me now or pay me later. At least it gave some Astronuts something to do.

0 upvotes
Antony John
By Antony John (Nov 29, 2011)

Tagomilonga you have justified my argument, not nullified it. No matter how much testing is done there are still possibilities that something could go wrong. Even if Kodak had manufactured the lens there is no certainty it would have performed any better - even if 'properly tested' - I cannot imagine PE would have done zero testing.
Ultimately even for a meg$ project there were restrictions on cost so how do you expect a commodity lens manufacturer to do 100% check on a 'low cost' lens?
No matter how much one spends on QA there will always be a possibility of an error. There is no such thing as 'zero tolerance'.

0 upvotes
jack1a
By jack1a (Dec 5, 2011)

Quality will never improve if we dont have "zero tolerance" for crapy equipment

0 upvotes
Total comments: 231
123