Lens Reviews Help
Andy Westlake | Help | Published Aug 31, 2011
NOTE - This article relates to lens reviews produced using our 'version 1' widget from 2008 to 2010. Click here for a similar article relating to our more recent lens reviews, produced using our 'version 2' widget in collaboration with DxOMark
Our lens test data is presented through our unique interactive 'widget', which is designed to allow you to explore the optical characteristics of a lens and freely compare it to any other. Three display modes are available, which show Sharpness and Chromatic Aberration, Distortion, and Vignetting; test data is presented for all marked focal lengths on the lens.
Unlike some other sites, we don't concentrate purely on sharpness and present other lens qualities almost as an afterthought; we consider chromatic aberration, distortion and vignetting at an equal level. The reason for this is simple, as these other attributes will often have a more destructive effect on perceived image quality than any softness of the lens. Indeed sharpness may be considered the lens equivalent of megapixels, superficially easy to understand and therefore the property most people think of first, but when taken on its own, far from being the whole story.
The main panel of the widget is arranged in three sections; the menus are along the top, the main data display is in the middle, and the user controls and data readout are at the bottom. The desired focal length and aperture can be selected using either the mouse or the cursor keys (up/down to 'zoom', left/right for aperture). Using the cursor keys becomes especially useful when comparing two lenses.
Below the main window are three further buttons, for full screen mode, accessing this 'help' section, and permalink generation.
Lens selection menu
This selects the primary lens to be viewed; you can filter by system (which we define by the manufacturer and the sensor’s crop factor), and the camera body used for the test. In a review page this is fixed to the specific lens under test, however it becomes available in ‘full screen’ mode.
Display mode menu
This allows you to select between the 'Sharpness & Chromatic Aberration', 'Distortion', and 'Falloff' display modes.
Similar to the lens selection menu, however this allows you to view the results for two lenses side-by-side for direct comparison. Again you can filter your search by system or camera test body.
Sharpness and Chromatic Aberration Display
This is the opening view in a lens review, and looks the most complex. It is designed to display a how the sharpness of a lens varies across the frame at a comprehensive range of focal length and aperture combinations.
The underlying black image is a simplified representation of our sharpness test chart, showing the locations of the slanted-edge test patterns from which we measure our data, and a series of checkerboard patterns which give visual representations of how this looks in practice. This is overlaid by a colour gradient, indicating the measured sharpness across the frame, and ranging from blue for best to magenta for worst. On the right of the chart display are two graphs, ‘Sharpness (MTF-50)’ and ‘Chromatic Aberration’.
Along the lower panel are the user controls, which allow you to select any tested combination of focal length and aperture. We measure at all of the focal lengths marked on the lens, using whole stop aperture increments for zooms, and third stop increments for primes. You can operate these controls using either the mouse or the cursor keys (up/down to change focal length, and left/right for aperture).
This displays a value known as MTF50, which is considered to correlate well with perceived sharpness. The y-axis is in line widths per picture height, and the x-axis represents the distance from the image centre along the frame diagonal. This choice of scale has been specifically chosen to allow direct comparison between camera systems with different sensor sizes and aspect ratios.
To make it easier to visualize what point in the frame any specific measurement corresponds to, a thin red circular line follows your mouse as you move it over the display to show the radius being viewed, with a corresponding vertical red line on the sharpness graph.
Chromatic Aberration graph
Here we display the lens's chromatic aberration characteristics, defined as the amount by which the red- and blue-channel components of the test patterns are displaced from their 'correct' position (using the green channel as the reference). The y-axis shows the width of colour fringing across the frame, and the higher the value, the more visible fringing will be. The chart also predicts the colour of fringing which will be produced; the red line indicates red/cyan fringing and the blue line blue/yellow fringing, with a combination of the two resulting in green/magenta fringing. The shapes of the chromatic aberration profiles are also important; the closer they are to linear, the easier correction is likely to be in post-processing.
Just as important as the graphs themselves are the checkerboard patterns, which provide visual indications of what any specific combination of sharpness and chromatic aberration actually looks like. Hover your mouse over one of the checkerboard positions on the chart, and up will pop a 100% crop from the actual test image which was used to generate the data. We want to encourage you to explore this feature extensively, to see for yourself exactly what any measured point in the frame really looks like. However we would also point out that these patterns, with their high contrast and sharp white/black transitions, have been chosen specifically to represent a worst-case scenario for the lens’s imaging performance; real-life images will rarely look quite like this.
Show MTF-50 & CA graphs - hiding the graphs allows you to see checkerboard patterns across all four diagonals. On the right of the screen, a 'belt buckle' display illustrates the range of sharpness across the frame:
This shows how a rectangular grid is projected by the lens, and therefore how lines will deviate from being rendered as perfectly straight. We also calculate the degree of distortion along both axes of the frame. Again you can select any of the marked focal lengths on the lens, but as distortion is essentially independent of aperture, we only show data recorded at F8.
The display is a direct representation of the grid pattern captured by the lens (some tests only show you a mathematical curve fit using over-simplified equations). Our data therefore allows you to see any complexity in the distortion, and you can even use screen shots to help find appropriate correction parameters in your preferred image manipulation software.
Along with the grid representation, we present three results in the data panel:
Short edge: defined as the percentage difference in length between the central vertical grid line and the left/right ‘short edge’. Describes the degree of bowing of the upper and lower horizontal lines, which are normally the most distorted.
Long edge: defined as the percentage difference in length between the central horizontal grid line and the top/bottom ‘long edge’. Describes the degree of bowing of the outermost left and right vertical lines.
- Distortion type: Barrel or pincushion
This shows how the lens’s illumination falls off towards the edge of the frame. For easier visualization the display is posterized in third-stop steps, and you can choose whether or not to display the values of these rings. The shades of grey used are derived from the actual image file analyzed, and so represent accurately the magnitude of the falloff; because the initial third-stop is essentially imperceptible to the human eye, we have removed it to simplify the graphic. The central 'dpreview.com' logo is displayed at the actual brightness level of the extreme corners, giving an additional indication of overall falloff.
As usual you have focal length and aperture dials which can be controlled using either the mouse or the cursor keys. Data is measured at each marked focal length on the lens, and at all apertures in third-stop increments down to F11 (at which point falloff has all but disappeared from most lenses).
Falloff – shows the maximum falloff value at the far corners of the image
Show values – allows you to show/hide the falloff values on the main display
Full screen mode
This allows you to display the widget in a new full screen window for maximum legibility. Its is also used to show the results from two lenses side-by-side below in 'Compare' mode: both lens selection menus become available, so you can freely switch between any pair of lenses for comparison. Click on the image below to see 'Compare' mode in action.
Sharpness comparisons between lenses tested on different cameras must be treated with a fair degree of caution; whilst we've tried to keep the playing field as level as possible, differences in imaging qualities (and most notably the strength of the text cameras' anti-aliasing filters) will affect the results. This not an issue in the regions where the lens isn't very sharp anyway, but can limit the measured MTFs at higher frequencies.
Click here for a comparison using the same lens (the excellent Olympus Zuiko Digital 50mm F2 Macro) tested on two cameras with identical sensor size and pixel count, but very different anti-aliasing filters (the Olympus E-3 and the Panasonic L10). The E-3's strong anti-aliasing filter results in rather lower MTF-50 numbers, most evident at apertures around F4 where the lens is at its best; the L10 shows a much lighter touch in comparison. Note however that this is a fairly extreme case; most of our test cameras have rather lighter anti-aliasing filters than the E-3.
Note that all of the other lens properties tested (chromatic aberration, distortion and vignetting) are not expected to be dependent upon the camera body used.
This button allows you to generate a web link to a specific state of the widget, which can then (for example) be embedded into forum posts to illustrate points being made in a discussion. Click on the permalink button, double click to highlight the text, then copy the link and paste it where necessary. Prepare for war!
How the Test Data is Shot
Test images for are shot in RAW at the camera's base ISO, and processed using a common converter (Adobe Camera Raw) with all sharpening disabled – this eliminates any differences from in-camera processing. Using RAW is crucial, as many current cameras can apply lens aberration correction to JPEGs - for example many Nikon DSLRS feature automatic correction of lateral chromatic aberration, and recent Canon EOS models feature automatic peripheral illumination (falloff) correction. Using JPEGs from these cameras for testing would therefore not provide a true description of the lens itself.
Sharpness is calculated from the slanted-edge patterns arranged across the frame; the test chart has four ‘arms’ extending from the centre to each corner of the frame, and the sharpness data we display is obtained by averaging the results from each, to give the most accurate representation of the lens’s performance.
For wideangle to normal focal lengths (up to 50mm equivalent), we use a test chart 3m x 2m in size, which gives a far more realistic working distance than the A0 size charts more generally used (for example, using a 24mm equivalent lens the subject distance is 2m as opposed to 0.72m). Due to space constraints, we revert to a smaller chart (approximately 1.05m x 0.7m) for longer focal lengths. This gives a shooting distance of about 30x the 35mm-equivalent focal length of the lens.
Correct focusing is critical, so where available we use magnified manual focus in live view, which is the most reliable and consistent method available. On cameras that don't feature live view, correct focus is established by determining the maximum sharpness obtainable from the lens using our in-house analysis software. In all cases, at least two replicate data sets are shot to confirm reproducibility of the results.
The test chart is a simple grid pattern, and aligned extremely precisely with the camera. Images shot in JPEG are translated directly to the data displayed in the widget. In addition, for systems which are based on the use of automatic distortion correction (most notably Micro Four Thirds), we also convert the RAW versions of the test images using a program that does not correct distortion, and present these results later in the review. As distortion is essentially independent of aperture, we only shoot at F8. The chart is approximately 1.05m x 0.7m in size.
Results are obtained by shooting an evenly-illuminated white wall through a highly diffusing ExpoDisc filter. We shoot in JPEG with any in-camera vignetting correction turned off; the displayed results use greyscale values derived directly from the original image files. The falloff value in stops is calculated using the measured tone curve of the camera body used for testing.
Frequently Asked Questions
Q. How many copies of a lens do you test for a review?
Results are normally derived from a single lens. It's not practical for us to test multiple samples of a lens as a matter of rule - it would simply take far too long.
Q. So how do you know if you have a good copy?
Our analysis software provides us with more information than reaches the reader of the review. We measure the sharpness across the four diagonals of the frame, before averaging the results for the final data. This shows us whether there is any significant asymmetry, which is a good predictor for whether we have a 'good' or 'bad' copy. When we have a clearly problematic lens, we reject it and request a second copy for test.
Q. Can I compare results obtained using different test cameras?
Strictly speaking, sharpness results should not be directly compared between tests carried out using different DSLR bodies, as the camera itself influences the results. Most notably, results get unreliable at higher spatial frequencies due to the differing effects of the test cameras' anti-aliasing filters.
This shouldn't make a huge difference for cameras with similar pixel counts, and as most of our test bodies are in the 12-14Mp range, the results should be broadly comparable. But it will have a more significant effect the further apart two test cameras are in resolution; for example, if you're comparing data between 12Mp and 24Mp cameras, then any data in the 'blue region' of the sharpness display should be considered to be not directly comparable.
However, all of the other results (chromatic aberration, distortion, and falloff) should be directly comparable regardless of the camera body used, and we’d stress that these are equally as important as sharpness in evaluating the overall performance of a lens.
Q. Will I really see the difference between lenses if I don’t print large or pixel peep?
If you normally view photos full screen on your computer (as opposed to pixel-peeping at 100%), or only print relatively small (6” x 4” or 7” x 5”) you won’t see a huge effect from differences in sharpness, or much colour fringing from chromatic aberration. However, you will still see the effects of distortion and falloff no matter how you view your pictures.
Q. Are your MTF50 numbers in line widths / picture height or line pairs/picture height?
Line widths per picture height.
When we first developed our lens data widget, we made a mistake. We calculated the MTF50 data in line widths per picture height, but labeled the y-axis as line pairs/picture height, and included a dotted line marked 'Nyquist Frequency' at the appropriate point for each camera on that scale.
For a number of reasons, we came to realise that the numbers were mislabeled in the presentation widget, and on going back though our calculations discovered the source of the error. This obviously meant we had to relabel the graphs, and as a direct consequence the 'Nyquist' line has gone too (it would now be off-scale for all of the test cameras we've used). It's important to understand that this change is not about the measured values, and only affects presentation of those values (the scale of the graph) - it doesn't affect the actual MTF50 calculations.
The change to correct the error in the presentation widget doesn't affect the lens review conclusions themselves as these are based on extensive testing, comparison and analysis in and out of the studio, and not on the absolute values presented by the graphing widget.
Apologies to our readers for any confusion caused.
Q. Why doesn't the MTF50 sharpness data in lens reviews match the resolution measurement in camera reviews?
Quite simply, these are different methods measuring different things. The sharpness data in lens reviews is presented as MTF50 (i.e. the spatial frequency at which image contrast is reduced to 50%); this is determined by calculation from a slanted-edge target, using camera raw files subjected to standardized processing with no additional sharpening. In contrast, in camera reviews we're looking at 'resolution', which is a visual assessment of the point where individual lines in the test target can no longer be distinguished. These two tests are therefore fully expected to give different results.
Q. I can't see the lens review widget in my web browser, but get a blank space instead. What's the problem?
The widget requires Adobe Flash Player to operate, so you may need to update to the latest version or uninstall/reinstall if you're having problems. If you're using Flash blocking software (and given the number of bad flash-based sites in the web we can forgive you for this), you'll need to add www.dpreview.com to your list of allowed sites.
Q. I'm looking at a new lens review and the widget doesn't load. Instead it's just showing 'loading CSS...'
Your browser has failed to load the latest version of a configuration file needed to view the test data. You'll need to close the page, clear your browser's cache and try again. Here's how to do it in some of the more popular browsers:
- Internet Explorer: Tools > Internet Options. 'General' tab, under 'Browsing History', click the 'Delete...' button and check 'Temporary Internet Files'
- Firefox: Tools > Clear Recent History. Check 'Cache' and click 'Clear Now'.
- Safari: Click on Cog icon at top right corner, 'Reset Safari'. Check 'Empty the Cache', click 'Reset'.