Lens Reviews Explained
Andy Westlake | Help | Published Oct 30, 2012
We're incredibly pleased to announce the return of lens reviews to dpreview.com thanks to a joint venture with DxO Labs, involving the establishment of a dedicated DxO lens and camera testing facility in Seattle, and the incorporation of DxO test data into dpreview.com’s class-leading lens reviews.
We launched lens reviews back in 2008 and they were an immediate hit, gaining praise for the unique user-friendly presentation of complex data via a patented test data widget, supported by numerous real-world sample images and expert commentary. Logistical issues put the lens reviews on an extended hiatus in late 2010, and the new venture with DxO Labs not only ensures the return of lens reviews to dpreview.com, but allows them to be produced more quickly.
Our popular lens review data widget has been entirely re-written, to allow dpreview.com’s visitors to visualize the test results from the new lab and compare lenses just as they could before. As part of the joint venture agreement the test results obtained will also be made available on the DxOMark website (www.dxomark.com).
'We are very happy to provide dpreview with our measurement technology for testing cameras and lenses' said DxO Labs CEO Jerome Meniere. 'Dpreview’s articulate and creative writing style makes difficult photography concepts accessible to even the most novices of photographers – they are a perfect complement to DxOMark’s scientific measures.'
'Producing the data for lens reviews is an incredibly long-winded process requiring a large, dedicated studio and hundreds, sometimes thousands of high precision exposures and measurements', according to dpreview.com Editor-in-chief Simon Joinson. 'The establishment of a dedicated DxO Labs lens and camera testing facility on our doorstep allows us to entrust the measurement and studio testing of lenses to an established world leader in image quality analysis, and to work with its team to offer our readers the perfect combination of accurate, consistent measurements with real-world shooting experience and expert analysis.'
Introduction to the lens test widget
Alongside our partnership with DxOMark, we've completely re-written our unique lens test widget, that provides a simple and easily-interpretable display of lens properties. This allows us to include additional data that DxOMark measures for each lens. The new widget is also designed to work on mobile devices such as the iPad, as it's based on HTML5 rather than Flash. Note that the test data in the old widget is not directly comparable to that in the new one, due to differences in the testing methods and calculations used.
Our new widget is designed to look and behave similarly to the old one. Again it is designed to allow you to explore the optical characteristics of a lens and freely compare it to others. Three display modes are available, which show Sharpness and Chromatic Aberration, Distortion, and Vignetting.
Unlike some other sites, we don't concentrate purely on sharpness and present other lens qualities almost as an afterthought; we consider chromatic aberration, distortion and vignetting at an equal level. The reason for this is simple, as these other attributes can easily have a more destructive effect on perceived image quality than any softness of the lens. Indeed 'sharpness' may be considered the lens equivalent of megapixels, superficially easy to understand and therefore the property most people think of first, but when taken on its own, far from being the whole story.
The main panel of the widget is arranged in three sections; the display mode tabs are along the top, the main data display is in the middle, and the user controls and data readout are at the bottom. Simply select the display mode you want by clicking on the relevent tab, and explore the focal length and aperture settings by dragging the controls.
Below the display are three links; the one on the left will take you to DxOMark's own page for the lens with its complete set of data. Those on the right take you to fullscreen and compare modes, which open in new windows or tabs. In these, the focal length and aperture controls can be changed using your keyboard's cursor keys (up/down to 'zoom', left/right for aperture). This becomes especially useful when comparing two lenses.
Sharpness and Chromatic Aberration Display
This is the opening view in a lens review, and looks the most complex. It is designed to display a how the sharpness of a lens varies across the frame at a wide range of focal length and aperture combinations.
The underlying black image is a highly simplified representation of the sharpness test chart. This is overlaid by a colour gradient, indicating the measured sharpness across the frame, and ranging from blue for best to magenta for worst. On the right of the chart display are two graphs, ‘Sharpness (MTF-50)’ and ‘Chromatic Aberration’.
Along the lower panel are the user controls, which allow you to select any tested combination of focal length and aperture. You can operate these controls using the mouse or, in fullscreen and compare modes, the cursor keys (up/down to change focal length, and left/right for aperture).
This displays a value known as MTF50, which is considered to correlate well with perceived sharpness. The y-axis is in line widths per picture height, and the x-axis represents the distance from the image centre along the frame diagonal. This choice of scale has been specifically chosen to allow direct comparison between camera systems with different sensor sizes and aspect ratios.
Chromatic Aberration graph
Here we display the lens's chromatic aberration characteristics, defined as the amount by which the red- and blue-channel components of the test patterns are displaced from their 'correct' position (using the green channel as the reference). The y-axis shows the width of colour fringing across the frame, and the higher the value, the more visible fringing will be; the x-axis is shared with the sharpness graph. The chart also predicts the colour of fringing which will be produced; the red line indicates red/cyan fringing and the blue line blue/yellow fringing, with a combination of the two resulting in green/magenta fringing. The shapes of the chromatic aberration profiles are also important; the closer they are to linear, the easier correction is likely to be in post-processing.
Show graphs - Unchecking this box allows you to hide the graphs. On the right of the screen, a 'belt buckle' display illustrates the range of sharpness across the frame:
Measured focal length and T-stop display
At the bottom right of the graph is a new feature: a display of the lens's measured focal length range and maximum Transmission. For zoom lenses, you can see the T-stop at intermediate focal lengths, not just the extremes of the range.
This shows how a rectangular grid is projected by the lens, and therefore how lines will deviate from being rendered as perfectly straight. We also calculate the degree of distortion along both axes of the frame. Again you can select from a range of focal lengths on the lens, but as distortion is essentially independent of aperture, this control is hidden.
The display shows a detailed representation of the lens's distortion pattern (some tests only show you a mathematical curve fit using over-simplified equations). Our data therefore allows you to see any complexity in the distortion, and you can even use screen shots to help find appropriate correction parameters in your preferred image manipulation software.
An additional feature enabled by our collaboration with DxOMark is the distortion graph. This gives some extra information about the degree and complexity of the distortion. For example the graph above shows a complex curve with re-correction towards the corners. This means the distortion will be more difficult to correct in software, and probably require software that uses specific lens profiles to fix it completely.
Along with the grid representation, we present three results in the data panel:
Short edge: defined as the percentage difference in length between the central vertical grid line and the left/right ‘short edge’. Describes the degree of bowing of the upper and lower horizontal lines, which are normally the most distorted.
Long edge: defined as the percentage difference in length between the central horizontal grid line and the top/bottom ‘long edge’. Describes the degree of bowing of the outermost left and right vertical lines.
- Distortion type: Barrel or pincushion
Unchecking the 'Show graphs' box allows you to hide the graph display, and see the distortion grid across the entire frame.
This shows how the lens’s illumination falls off towards the edge of the frame. For easier visualization the display is posterized in third-stop steps. The shades of grey used represent accurately the magnitude of the falloff; because the initial third-stop is essentially imperceptible to the human eye, we have removed it to simplify the graphic.
An additional feature, again thanks to our collaboration with DxOMark, is the Falloff graph. This provides some extra information about how severe the vignetting will appear on your images. If the graph falls dramatically towards the corner of the frame, the vignetting will likely look more objectionable compared to a lens with similar corner illumination but a more gradual drop-off in brightness across the frame.
Falloff – shows the maximum falloff value at the far corners of the image
As with the other displays, you can choose to hide the graph, and see a representation of the light falloff pattern across the entire frame.
Unchecking the 'Posterize' box allows you to see the falloff pattern as a gradient, rather than posterized. In essence this is what you'd see if you shot an evenly-illuminated plain white wall at the chosen settings. Hairlines are included, showing the same 1/3 stop increments as in the posterized display.
Full Screen Mode
Clicking on the 'fullscreen' link under the widget will take you to a new window or tab containing just the widget.
This mode has two key new features:
Lens selection menus
At the top of the widget, two new drop-down menus appear. These allow you to choose a lens to view from all of the lenses for which we have data, and select from the cameras they've been tested on.
One useful feature that's not-so-obvious is keyboard control in fullscreen mode. The left and right keys change the aperture, and the up and down keys select the focal length. This makes exploring the lens's characteristics at various settings much easier. (These controls are disabled in lens reviews themselves, because the cursor keys also make the page scroll up and down.)
Clicking the 'Compare' link allows you to compare data for two lenses side-by-side. You can choose freely from all the lenses for which we have published data. Note though that MTF data is somewhat affected by the resolution of the test camera so isn't directly comparable between different bodies, and especially those with widely differing resolutions.
DxOMark's lens image quality testing protocols
DxO Labs – the testing facility behind DxOMark’s wealth of camera and lens data – measures and analyzes lenses and camera bodies for the following RAW-based image quality factors: resolution, transmission, distortion, vignetting, and chromatic aberration.
To ensure that these measurements are accurate, repeatable, and cover the desired range of image quality parameters, DxO Labs developed objective protocols and constructed state-of-the-art testing facilities to ensure accuracy and reliability.
DxO engineers execute these protocols using the following key elements: a dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc; a set of precisely-described and bias-free test protocols for each measurement category which strictly accounts for all physical parameters that influence measurements and ensures repeatability of the measurements; and software – DxO Analyzer - that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.
To eliminate setup bias, settings, instrument calibration, light levels, cleanliness (including that of the camera and lens), and all other parameters are checked and rechecked before each test to ensure that all cameras and lenses are tested under exactly the same conditions. Finally, each measurement is validated by at least two technicians who take their measurements on different days to minimize the risk of manipulation errors.
Testing protocol for MTF (resolution)
With the growing pixel count found in today’s camera sensors, it is increasingly important for lenses to be of the highest quality – their defects can actually lower the resolution of a digital camera.
DxO Labs measures Modulation Transfer Function (MTF) – the combined resolution of a camera body and lens – using a black and white checkered target, and according to the ISO 12233 standard SFR method (see MTF measurement definition). The checkered target is designed by DxO Labs and is produced using a high-resolution printer to achieve sharp transitions between black and white areas without aliasing.
Before shooting, DxO technicians make sure the target is tilted at a 5° angle and that the target fills the field of the camera. They then set the parallelism between the sensor and the target planes using a mirror flush against the target; perfect alignment is achieved when the reflected image of the lens appears in the center of the camera viewfinder.
DxO technicians then proceed by selecting the lowest actual ISO speed of the camera to acquire images with a minimum level of noise. The exposure is set so that the white squares of the target are just below sensor saturation in RAW format, to ensure that the entire dynamic of the sensor is used. Of course, DxO deactivates all sharpening options and stabilizing systems of the camera or lens.
The target is then illuminated using the open-flash method: the one-second exposure begins in complete darkness to ensure the image is not affected by reverberations; and the exposure ends with a brief flash using daylight color temperature set to 5500K intensity. To guarantee further stability and prevent any motion blur, the camera and lens are fastened to a geared head fixed to a heavy-duty studio stand. A graduated rail on ball bearings permits very accurate adjustment of the distance between the camera and the target. To also ensure that camera and lens vibrations do not affect the measurements, DxO staff use the reflex mirror lockup function (when available), and release the shutter with a remote control or self-timer.
For each focal length and aperture of the lens, a photo is taken at 40 different focusing positions around the focusing point set by the camera's autofocus system. The sharpest image is used to measure a camera and lens’ MTF.
Read more about resolution testing protocol on DxOMark website.
Testing protocols for distortion, lateral chromatic aberration, and vignetting
Lateral Chromatic Aberration and distortion are measured on a DxO Labs dot chart, which is a pattern of regularly distributed black dots on a glass support. Glass was chosen because it provides the flatness and shape stability necessary for these measurements. The dots printed on the chart are circular and perfectly aligned, forming a grid.
Measuring vignetting (the darkening of an image near its corners) requires using the white background of the same dot chart. Before shooting, DxO techs align the camera sensor on the target plane and carefully check the uniformity of the lighting to ensure that it remains within the +/-4% range. To best enhance the accuracy of the vignetting measurement, DxO characterizes the actual illumination uniformity of the chart using a calibrated camera-lens couple. The color temperature is also set to 5500K (corresponding to daylight).
A photo is then taken for each focal length and aperture, with distance between the target and the camera and lens decreasing or increasing depending on the respective lens’ focal length. Technicians then readjust the lens to frame the chart area so it appears exactly the same for every focal length and aperture.
Finally, DxO technicians take two additional pictures for each focal length at two different distances, with the lens focused at infinity, in order to calculate the Effective Focal length (EFL). Adjust the distance to the target to ensure that you are framing the same portion, the same area of the chart for each focal length. So when you increase the focal, you step back to frame the same area.
Read more about distortion, lateral chromatic aberration, and vignetting testing protocols on DxOMark website.
Testing protocol for light transmission
The photometric aperture, also known as “T-stop” (T = transmission), reveals the true aperture of a lens, as the measurement takes into account its transmission (or light) loss (see light transmission definition). This gives a photographer a better understanding of how their lens’ aperture behaves. For example, a lens used at aperture of f/2.8 may perform more closely to a T-stop of T=3.1.
The measurement consists of taking a picture of a uniformly illuminated (+/-1%) opalescent transmission target. The light source, chosen for its stability, is a halogen lamp filtered to achieve a daylight color temperature of 5500K. Before testing, this luminance is always measured on the diffusing surface (about 140 cd/m²) with a certified luminance-meter.
Knowing the entrance light flux, the sensor response, and the shutter speed, DxO Labs can then calculate the T-stop of the lens for a given focusing distance: DxO technicians place the camera at a distance equal to 40 times the focal length of the lens (2 meters for a 50mm lens). They proceed to take five pictures – which are averaged together – for every aperture assigned to a focal length (using full-stop increments). The transmission metric score is obtained by average the whole range of focal lengths.
Read more about light transmission testing protocol on DxOMark website.
DxOMark is the trusted industry standard for camera and lens independent image quality measurements and ratings. For years, DxOMark has established its reputation for the best:
- Rigorous hardware testing
- Industry-grade laboratory tools
- Database of thousands of camera-lens test results
DxOMark consists of a comprehensive RAW-based image quality measurement database and a set of scores used to evaluate and compare digital cameras and lenses.
A unique Measurement Database
All measurements are performed on straight-from-camera RAW files, the only reliable way to evaluate intrinsic hardware performance. All lenses are tested on every camera on which they can be mounted as camera sensors impact the optical performance of lenses.
Lenses and cameras are tested for their full range of possible settings such as focal length, aperture, exposure time, ISO speed, white balance, etc. To ensure highest precision and reliability, measurements are performed at DxO Labs’ dedicated testing laboratories, where conditions are controlled as in standard metrology labs.
To ensure that measurements for all lenses and cameras can be reliably compared, DxOMark has developed detailed protocols that are systematically repeated for all equipment which are tested. These protocols are detailed on the DxOMark’s website, so that anyone interested in performing measurements can do so.
DxOMark test commercial products: in other words, DxOMark buy or rent lenses and cameras from the very same retailers that consumers use. When DxOMark do test pre-production models (when commercial products are not yet available), DxOMark clearly indicate this on its site, and retest those models when they become commercially available. Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that DxOMark is completely independent from them.
DxOMark Scores for ranking and comparison of cameras and lenses
The DxOMark Measurement Database is so large and exhaustive that comparing or selecting camera or lens for specific needs could prove difficult without appropriate tools. Accordingly, DxOMark have defined a set of fifteen DxOMark Scores to facilitate easy ranking and sorting.
The DxOMark Scores are designed based on the following photographic use cases: Portrait, Landscape, and Sport.
To design DxOMark Scores, DxOMark have made choices about photographic use cases and their associated image quality requirements (such as resolution, distortion, noise, dynamic range, etc.).
DxOMark means objective, independent, comprehensive image quality data
DxOMark measurements are performed on unprocessed RAW files, prior to any digital image processing (except processing that may happen on the sensor chip itself). RAW format is the format by which objective performance for digital cameras is most effectively evaluated, as it does not depend on the efficiency of the RAW conversion applied. RAW is also intrinsically more relevant for measuring lens performance, given that no further processing was used that might have smoothed or hidden any intrinsic optical limitations or lens defects.
DxOMark lens measurements are unique in that they are performed on RAW and are specific to all camera bodies for which each lens is designed. The test results published on dxomark.com conclusively show that the same lens performs differently when mounted on different camera bodies.
Many sensor characteristics, such as physical size, pixel pitch, and microlenses, influence the optical performance of lens/camera combinations.
DxOMark perform measurements for each lens-camera combination using a wide range of shooting parameters (aperture, ISO speed, object distance, focal length, etc). Up to a thousand shots of various targets under different exposure conditions may be required for each combination; generating hundreds of thousands of measurement data reflecting lens performance.
DxOMark measurement methods and protocols are open and compliant with international and industry standards
All measurements follow the methods and protocols described in ISO (International Standard Organization) standard documentation or developed by industry players and considered as industry standards. DxO Labs image scientists and engineers have studied, developed, and implemented such standard methods and protocols and then packaged these tools and know-how in a commercial solution, DxO Analyzer.
DxO Analyzer is now used in the digital camera and component industry worldwide by such manufacturers as Nikon, Olympus, Panasonic and Samsung, to name just a few. DxO Analyzer is also used to perform certain tests by many web and print photo magazines, including Popular Photography, Nippon Camera, Chasseur d’Images, DIWA Labs, Imaging Resource, Réponse Photo, Let’s Go Digital, Digital Camera Magazine Japan, digitalkamera.de, and more than 30 other press partners.
Further, DxO Labs has put in place a very thorough data validation process. Once data are produced, they are analyzed in detail and compared with those in existing image quality measurements database (thousands of digital cameras, lenses and imaging systems have been tested and analyzed over time). This process has led DxOMark to develop specific protocols or tools to deal with difficult measurement issues such as best focus optical performance and sensor spectral sensitivity.
DxO Labs is an active participant in international imaging standardization activities. For example, it has been elected as Technical Editor of the International Imaging Industry Association (I3A) CPIQ standard for defining new methods to assess photographic quality from the consumer point of view, and DxO Labs image scientists regularly present their results at such renowned digital imaging conferences as the Electronic Imaging Conference, and Image Sensors (Europe and US).
To learn more, visit the official DxO Labs website.