www.dpreview.com
www.dpreview.com

Variation Facts and Fallacies

RCicala | Technology | Published Nov 25, 2011

Lens sharpness is widely discussed on the internet, so it shouldn’t come as any surprise that lens softness is also a major topic. However, my experience both as a photographer and as the owner of a lens rental business has shown me that much of this discussion is based on a fallacy. So just what is it reasonable to expect from a lens and how doggedly should you pursue the ‘best copy?’

Fallacy:

A bit of background

When I started in photography most of the forums I learned from had at least one thread a day about someone’s 'soft lens.' They knew the lens was soft because their camera worked 'fine' with all their other lenses. After a few years of running a rental business, though, I found myself in the following situation three or four times a week:

'The lens you sent me front-focuses, it's not good.'
'OK, we’ll overnight you a replacement.'

Only to find, when the first lens came back, that all our tests suggested it was perfectly fine. And the customer is very happy with the replacement lens - apparently, this one is 'fine.'

So what causes this problem? It's rather simple, actually. The fallacy here is the definition of 'fine.' Most people assume that 'fine' means, 'perfectly calibrated.' It became apparent to me that cameras and lenses are not perfectly calibrated, but rather, they all have some variation. That realisation shouldn’t have been shocking; every manufactured product has variation, why should cameras and lenses be any different?

For the next several years I continued to investigate this issue, writing articles about it as I went. Reading all of them now is an overly-long exercise and I’ve learned a lot since I began writing on this subject. So it seems time to put a summary article together.

Manufacturing variation affects all lenses from all brands – the only difference is the tolerance levels considered acceptable.

Some Semantics

In most online discussions, things derail when different people understand the same terminology differently. Discussions of lens quality are particularly prone to that, so let me define the terms as I intend to use them.

A soft or bad copy is a copy of lens X that clearly does not perform as well as other copies of lens X.  There are bad lenses out there. They happen.  

A design choice affects all the copies of Lens X. Every lens has some design choices made that we may not like.

Copy/sample variation is a slight difference that can be detected between different copies of lens X, or different copies of Camera B. Copy variation is detectable, but usually not significant (compared to a bad copy which is always significant).

The Search for the Holy Lens Grail

Before we go further, there are some people that should probably stop reading here. If you ever think 'for $1,500 I demand perfection,' this is not the article for you, it will just get you upset. The laws of physics are not suspended, nor are techniques of manufacturing altered, just because you demand it be so. If you enjoy The Quest for the Perfect Lens and other fantasy games, just move along and save yourself the aggravation of being administered this particular dose of reality.

Why Variation is Inevitable

First, I should mention that the problem has received more attention in recent years, oddly enough, because our equipment has improved so much. A 6MP crop-sensor camera didn’t expose the flaws in a lens that now, a 24MP full-frame camera makes painfully obvious. Better lenses contribute too: when a lens had four really soft corners it was hard to tell if one side was worse than the other. On a newer lens with sharp corners the difference may be instantly obvious.

It’s probably not a coincidence that photographer and blogger Lloyd Chambers brought attention to the fact that camera mounts aren’t always perfectly parallel to the sensor when he was testing Zeiss ZF 21mm lenses on a 24MP Nikon D3x. There's every chance it would not have been apparent if he had been shooting a lesser lens on a lesser camera.

The key thing to realize is that the problem is not limited to one brand, one type of lens, or even just SLR gear. Landscape photographer Joseph Holmes found significant variation in medium format lenses and focusing. Testing website SLR Gear found a batch of 50mm lenses that were all softer on the right side than the left. At Lensrentals we found a group of 300mm f/4 lenses from the same serial number run almost all suffered electrical failures. Why is this so?

Manufacturing Tolerances

Manufacturing tolerances are just that: a range of acceptable values, not an exact point. In other words, what is specified as a 1/4 inch diameter screw may be anything between 0.247 inches and 0.253 inches in diameter (see this chart from an internationally-renowned industrial fastener manufacturer). The machines that make them can’t be more accurate than that at reasonable cost.

Did you know that every time a glass manufacturer makes a run of a given optical glass, the refraction index and dispersion vary a tiny bit? The glass manufacturer furnishes a melt sheet to the lens manufacturer so they can make tiny adjustments in thickness or curvature of that element to compensate for the differences.

The glass is probably a tiny source of variation compared to the other components that make up a lens or camera. In addition to the multiple glass elements, there are clips, shims, and grips that hold them in place within the lens. There are helicoids, barrels, gears, and rings move them around to focus. Electrical motors and circuit boards tell them what to do, and, as Chambers has shown us, even the lens mount that connects the lens to the camera is a source of variation. 

Cross-section of a Zeiss 21mm lens. Note all of the clips, shims, and barrels holding the elements in position.

Many, if not most, of those components are outsourced to other companies or factories. A change in subcontractor could result in a slightly different part being supplied. Something as simple as a set of ribbon cables more likely to crack or a solder that is slightly less conductive being used on a circuit board could result in significant changes to a camera or lens.

If you delve into the manufacturer’s parts lists (when you can get them) you’ll find they take these variations into account and plan for them within a given tolerance range. For example, if you need to replace the front element of your Nikon 14-24mm lens (part # 1K104-xxx) you need to replace the adjustment washer behind it with one of five specific thicknesses - each copy of the lens requires a slightly different thickness for proper spacing of the front element to focus sharply.

There are numerous other variable-sized spacers and parts for every lens (most lenses have variable thickness spacers in at least three different places). In many lenses even the lens mount comes in several thicknesses or with shims so that the image focuses properly on the sensor. As someone who has to shim lenses fairly frequently, I can assure you it can be done close to perfection, but not perfectly. When a 0.06mm shim is indicated, we often have the choice of 0.05mm or 0.07mm shims - close, but not perfect.

Similarly, the lens must be electronically calibrated. The circuit boards inside each lens with its own AF motor contain adjustment screws to calibrate the frequency and current of the electronic pulses sent to the motor used to move the lens during focusing. Manufacturers don’t build in this adjustment because every lens is exactly the same; they build it in because every lens is slightly different and adjustments are necessary. And by the way: the factory manual gives an acceptable range for the adjustment, something like 150 +/- 2 KHz.

Frequency and voltage adjustment variable resistors on the printed circuit board inside a Canon EF lens.

It’s not my purpose to list every source of variation in a given copy of a lens or camera body. They are far too numerous. But just to give an idea of some major ones:

For everything on the list there is a slight variation within an allowable tolerance. You can demonstrate this for yourself if you have some friends (or camera club members) and can get several copies of the same lens. Mount a camera to a tripod and focus on a target that is very visible in live view at 10X magnification. Then, just change from one lens to another and watch the target be a bit off center in different directions with each lens. Just a bit for most lenses, but you might find one that's off target by half of the screen or so. 

Camera bodies are no different than lenses with regard to variation, and in fact could be even more problematic. Is the sensor perfectly parallel to the lens mount? Is the AF sensor properly calibrated to the imaging sensor? Is the AF mirror exactly aligned and angled in relation to the AF sensor? The list goes on and on. And in every case the camera is made nearly perfect, but not exactly perfect.

Lens-Camera Matching

During, and at the end of, the assembly process, the lenses and cameras are tested to make sure they are within the manufacturer’s specifications. If they are out-of-spec and get by quality control, then someone, somewhere gets a truly bad camera or lens. It doesn’t happen frequently, but it happens. When it does, it isn't a subtle call; it's very obvious the lens is bad. In mass-produced elements, quality control is likely to be conducted on a 'sample' basis - only one in every ten or every hundred units will be checked.

But even the cameras and lenses that meet specifications are still going to vary slightly. Many people think, 'I’ll try eight lenses and take the best one.' The reality though is that the only sensible definition of 'best' is 'best with the camera body you are using,'. This is because there is plenty of room for variance in the behavior of bodies, too.

Let’s consider just the lens mount as a theoretical example. First, we say the lens mount of a camera must be parallel to the sensor with a range of +/ - 0.05 degrees (I have no idea what an acceptable range is, I only know that they cannot consistently be made perfectly parallel). Then, let’s say camera A's lens mount has a tilt of 0.04 degrees to the right. Lens #32 has a tilt of 0.04 degrees to the left. The two tilts would cancel each other out, all would be magical, and the owner would write sonnets on the various forums praising his lens.

But let's say he sells his lens to someone whose camera's lens mount has a tilt of 0.03 degrees to the left. The lens and camera now both tilt to the left and the new owner may say, 'the lens you sold me is a bad copy, it’s horribly soft on the sides.'

And this is just tilt. The mounts may also vary in thickness. There is variation, too, in autofocus systems. Certainly more than there is in lens mounts. Through-the-viewfinder manual focus will also vary (you might be surprised to know that viewfinder focusing screens are shimmed by hand). Even exposure metering varies slightly from camera to camera, and requires recalibration if it’s out of specification.

The bottom line is every lens varies slightly, in several respects. And every camera varies slightly too, in a number of different ways. A given lens on four different cameras will behave slightly differently on each of them, and four different copies of the same lens will each perform slightly differently on a given camera body. How differently? Well the short version is you probably will notice the differences if you’re a 'pixel-peeper'. If you use cameras and lenses to take pictures, though, it would be very unlikely you’d notice normal variation, even with large prints. The one exception might be high quality wide aperture lenses because the narrow depth of field may make subtle differences apparent.

So how much of a difference does this make? Click here to read page 2

What is normal variation?

When we at Lensrentals started using computerized analysis (we use the Imatest package) to assess the MTF of large numbers of lenses, it became obvious that there is sharpness variation among copies. The example below shows the groupings for several different 100mm lenses: six samples of the Carl Zeiss Makro-Planar T* 2/100, eight samples of the Canon EF 100mm f/2.8 Macro USM, and twenty-four samples of the Canon EF 100mm f/2.8L Macro IS USM.

This graph plots center sharpness against a sharpness figure for the whole lens (measured in line pairs per picture height for unsharpened RAW images)

Two things should be apparent from this chart. First, there clearly is variation between the different copies of each lens. On average the Canon 100mm f2.8 IS L lenses are the sharpest, but some copies are sharper than others. And some copies of the other two lenses are sharper again. You can see how this might lead two different reviewers to hold slightly different opinions on which 100mm macro lens is the best.

Second, a true 'bad lens' is truly an outlier, and you can see a bad one way down on the lower left. The difference between a soft or bad copy and the main group is very large. The copy-to-copy variation that occurs between the other lenses is really minor. If you want to know how bad that bad copy is: our techs could identify it looking at JPEGs at 50% on a computer screen, but they'd be unlikely to spot it by looking at a web-sized image.

A second point about copy-to-copy variation must be made: the data above were for all of those lenses on one single camera body. As I've already explained, when you change the body the results change - the overall pattern will look the same, but each lens result will be slightly different. The sharpest lens on test camera A may well not be the sharpest on test camera B.

The example below shows two things. The green triangles represent multiple different samples of a lens shot on the same camera (notice that the range on both axis has been changed to illustrate the difference more clearly). We then took one of those lenses (circled) and shot it on 11 other camera bodies - the performance is represented here by the blue and red squares. As you can see, there is significant variation in the results from the different cameras.

A single lens (circled) was re-tested on eleven camera bodies (red boxes and blue diamonds)
from two different serial number series. All were brand new from box.

As suggested by our name, Lensrentals is a lens rental house, and as such I'd like to clarify one point: the variations that Ive been talking about don't occur because we're looking at used lenses. When we first started testing lenses, we made sure to carefully compare brand new lenses with our stock lenses to make sure that our quality assurance was keeping the stock lenses in good shape.

We buy new lenses in significant quantities; so we usually start testing a given lens by examining a dozen new copies right out of the box, then we start testing our rental copies. The graph below compares a group of new-out-of-the-box lenses with a group from rental stock to give an example. But in the graph immediately above, all of the camera bodies were new stock. 

We’ve only found one lens so far that did show a difference with age, but results from that aren’t used in any of these illustrations (I have discussed this in detail in this article).

 Focusing Variation

To give some idea of how small the difference between various lenses really is, we can examine the variation of a single camera's autofocus with a single lens. It's reasonable to assume that you shoot dozens of images with your camera and depend on autofocus to be accurate. You never notice, unless you look very carefully, that if you automatically focus on the same shot several times the camera focuses slightly differently each time. That’s why for testing purposes the data points I’ve shown were obtained using the best possible live-view manual focus for each lens.

But let’s look at the graph below, which consists of multiple autofocus shots made with one lens on one camera. First, I let the camera autofocus the lens once, turned off autofocus, and took 4 consecutive shots, represented by the blue diamonds in the graph. In theory they should be identical, since nothing was changed. In reality, they are just a little different, which demonstrates the amount of shot-to-shot variation in the testing method - very little, in short.

Without changing anything, I then moved the focus ring to one extreme or the other, let the camera autofocus again, took a single shot, and repeated that process six times. Those six shots are the red boxes. You can see the camera autofocuses the shot a bit differently each time (and this is with center focus point on a star chart: a near ideal situation). Finally, I focused the lens using live-view magnified focusing four consecutive times (represented here by the green triangles).

A single lens shot repeatedly with no changes (blue), manually focused 4 separate
times (green), and autofocused 6 times (red).

The scale in this series is changed again to illustrate the point clearly. It should be obvious that the variation of the camera’s autofocus system on repeated shots with a single lens is about 1/3 as great as the variation between different lenses. Autofocus microadjustment to the camera would improve the average AF performance to be nearly as good as the live-view focusing but it wouldn’t change the shot-to-shot variation (it would shift the red cluster towards the green results but it would remain just as spread out).

So How Big is This Variation in Real Life?

That’s really the point of all this. The variation I’ve described is easy to detect using modern computerized optical analysis. And we’ve seen that the variation between a truly bad lens and the group of acceptable lenses is very large. But how big is the variation between the different acceptable copies?

The best answer I can give is probably that it might be detectable by pixel-peepers, but not working photographers, at least hardly ever. Lensrentals has six full-time pixel-peepers on our staff (the inspection technicians) who are armed with a large array of well-lit charts to analyze sharpness, aberration, back and front focus, you name it. They can pick out a bad lens fairly easily, but can’t tell the difference between the best and worst images in the groups above.

But there’s a more scientific way to look at things than 'our techs said so.' The Subjective Quality Factor (SQF) is a measurement developed by Ed Granger and K.N. Cupery in the 1970s for Kodak and used by Popular Photographer for their lens reviews. Basically, SQF uses a mathematical formula, taking the MTF data from the lens (which we get from these tests) to predict with good accuracy how sharp a print would be perceived at various sizes and distances.

I’m not going into detail about SQF (for a more thorough discussion, see Bob Atkins' excellent article or the references below). The important part is that several experts have shown an SQF difference of less than 5 for a reasonably sized print is basically not detectable by human vision.

It’s a simple matter to have the computer calculate the SQF from the data we’ve already obtained in our testing. We arbitrarily calculate it for 8 X 10 inch print size, but the SQF difference would be the same for any reasonably sized print.

The graph below is for the same lens that I used for the autofocus example above, plus I added several other copies of that lens tested on the same camera. Then I had the computer calculate the SQF data for the best and worst copies. 

SQF for 8 X 10 inch print taken from selected MTF points.

As you can see the variation in SQF is less than 5, meaning theoretically, you wouldn’t be able to detect the difference in a print. The difference is real and can be detected by Imatest, or by a careful pixel-peeper armed with a few test charts, a large monitor, and too much time on their hands. But it wouldn't be significant enough to make an obvious difference in a print.

So What’s the Point of All This?

The main points are fairly straightforward:

1) Every lens and every camera exhibits slight variations relative to its twins that are detectable, but rarely significant.

2) Variations that wouldn’t make the slightest difference in a print may seem quite different when the numbers are presented in a lens review. And, just because one copy of lens X is sharper than one copy of lens Y, doesn’t mean they all are, or that they all will be in your camera.

3) Occasionally, an acceptable lens mounted to an acceptable camera combine their variations in a way that makes them unacceptable together. The lens may be fine with a different camera, and the camera fine with a different copy of the lens. 

4) Really bad, soft, out-of-acceptable range lenses do occur. They are fairly rare though and easy to detect.

5) Camera autofocus is more variable and less accurate than you think.* 

* Before you go all Major League Fanboy about the superiority of your camera’s autofocus system: autofocus variation exists in every camera from every brand we’ve tested. Want to prove it? Put a wide aperture prime on your camera, mount it to a tripod, and focus on something in the middle distance. Now move the focus ring to infinity, let the camera autofocus and note exactly where it ends up on the distance scale. Now turn the focus ring to near focus, let the camera autofocus on the subject at middle distance, and note the number on the distance scale. They will be slightly different.

When you buy a lens, and assuming your camera allows you to, you should microfocus adjust it. If you do it properly, using a sensible focus distance, it really does make a difference. Then do some very basic tests to make sure it functions properly, and go take some pictures. If you like the pictures it makes, then keep it.

Oddly enough, the conclusion I’ve reached from several years of dedicated pixel-peeping and lens analysis is this: Trying to find exactly the sharpest copy of the sharpest lens is a fool’s errand: you’ll be looking for something that doesn’t exist.

Roger Cicala, Lensrentals.com 


For more of Roger's thoughts on lens and camera variation - and much, much more, visit his blog