DxOMark's lens image quality testing protocols
DxO Labs – the testing facility behind DxOMark’s wealth of camera and lens data – measures and analyzes lenses and camera bodies for the following RAW-based image quality factors: resolution, transmission, distortion, vignetting, and chromatic aberration.
To ensure that these measurements are accurate, repeatable, and cover the desired range of image quality parameters, DxO Labs developed objective protocols and constructed state-of-the-art testing facilities to ensure accuracy and reliability.
DxO engineers execute these protocols using the following key elements: a dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc; a set of precisely-described and bias-free test protocols for each measurement category which strictly accounts for all physical parameters that influence measurements and ensures repeatability of the measurements; and software – DxO Analyzer - that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.
To eliminate setup bias, settings, instrument calibration, light levels, cleanliness (including that of the camera and lens), and all other parameters are checked and rechecked before each test to ensure that all cameras and lenses are tested under exactly the same conditions. Finally, each measurement is validated by at least two technicians who take their measurements on different days to minimize the risk of manipulation errors.
Testing protocol for MTF (resolution)
With the growing pixel count found in today’s camera sensors, it is increasingly important for lenses to be of the highest quality – their defects can actually lower the resolution of a digital camera.
DxO Labs measures Modulation Transfer Function (MTF) – the combined resolution of a camera body and lens – using a black and white checkered target, and according to the ISO 12233 standard SFR method (see MTF measurement definition). The checkered target is designed by DxO Labs and is produced using a high-resolution printer to achieve sharp transitions between black and white areas without aliasing.
Before shooting, DxO technicians make sure the target is tilted at a 5° angle and that the target fills the field of the camera. They then set the parallelism between the sensor and the target planes using a mirror flush against the target; perfect alignment is achieved when the reflected image of the lens appears in the center of the camera viewfinder.
DxO technicians then proceed by selecting the lowest actual ISO speed of the camera to acquire images with a minimum level of noise. The exposure is set so that the white squares of the target are just below sensor saturation in RAW format, to ensure that the entire dynamic of the sensor is used. Of course, DxO deactivates all sharpening options and stabilizing systems of the camera or lens.
The target is then illuminated using the open-flash method: the one-second exposure begins in complete darkness to ensure the image is not affected by reverberations; and the exposure ends with a brief flash using daylight color temperature set to 5500K intensity. To guarantee further stability and prevent any motion blur, the camera and lens are fastened to a geared head fixed to a heavy-duty studio stand. A graduated rail on ball bearings permits very accurate adjustment of the distance between the camera and the target. To also ensure that camera and lens vibrations do not affect the measurements, DxO staff use the reflex mirror lockup function (when available), and release the shutter with a remote control or self-timer.
For each focal length and aperture of the lens, a photo is taken at 40 different focusing positions around the focusing point set by the camera's autofocus system. The sharpest image is used to measure a camera and lens’ MTF.
Read more about resolution testing protocol on DxOMark website.
Testing protocols for distortion, lateral chromatic aberration, and vignetting
Lateral Chromatic Aberration and distortion are measured on a DxO Labs dot chart, which is a pattern of regularly distributed black dots on a glass support. Glass was chosen because it provides the flatness and shape stability necessary for these measurements. The dots printed on the chart are circular and perfectly aligned, forming a grid.
Measuring vignetting (the darkening of an image near its corners) requires using the white background of the same dot chart. Before shooting, DxO techs align the camera sensor on the target plane and carefully check the uniformity of the lighting to ensure that it remains within the +/-4% range. To best enhance the accuracy of the vignetting measurement, DxO characterizes the actual illumination uniformity of the chart using a calibrated camera-lens couple. The color temperature is also set to 5500K (corresponding to daylight).
A photo is then taken for each focal length and aperture, with distance between the target and the camera and lens decreasing or increasing depending on the respective lens’ focal length. Technicians then readjust the lens to frame the chart area so it appears exactly the same for every focal length and aperture.
Finally, DxO technicians take two additional pictures for each focal length at two different distances, with the lens focused at infinity, in order to calculate the Effective Focal length (EFL). Adjust the distance to the target to ensure that you are framing the same portion, the same area of the chart for each focal length. So when you increase the focal, you step back to frame the same area.
Read more about distortion, lateral chromatic aberration, and vignetting testing protocols on DxOMark website.
Testing protocol for light transmission
The photometric aperture, also known as “T-stop” (T = transmission), reveals the true aperture of a lens, as the measurement takes into account its transmission (or light) loss (see light transmission definition). This gives a photographer a better understanding of how their lens’ aperture behaves. For example, a lens used at aperture of f/2.8 may perform more closely to a T-stop of T=3.1.
The measurement consists of taking a picture of a uniformly illuminated (+/-1%) opalescent transmission target. The light source, chosen for its stability, is a halogen lamp filtered to achieve a daylight color temperature of 5500K. Before testing, this luminance is always measured on the diffusing surface (about 140 cd/m²) with a certified luminance-meter.
Knowing the entrance light flux, the sensor response, and the shutter speed, DxO Labs can then calculate the T-stop of the lens for a given focusing distance: DxO technicians place the camera at a distance equal to 40 times the focal length of the lens (2 meters for a 50mm lens). They proceed to take five pictures – which are averaged together – for every aperture assigned to a focal length (using full-stop increments). The transmission metric score is obtained by average the whole range of focal lengths.
Read more about light transmission testing protocol on DxOMark website.