Is lens testing at chart distance really an accurate measure?

Started 2 months ago | Discussions
techjedi
techjedi Veteran Member • Posts: 3,961
Is lens testing at chart distance really an accurate measure?
3

Whenever we read tests on zoom lenses, there is logically different performance tested/observed over the gradient of focal lengths covered.

However, little is said about performance across the focus distance range from minimum focus distance to infinity.

I am not an expert in testing methodology for companies like DXO, but if you were to fit a MTF chart into frame for a given prime lens, that would imply you were only testing one subject distance within the focusing range for that lens.

This seems reasonable for some focal lengths where most photos would be at or near "chart distance", but I would imagine its a poor test for lenses whose focal length is used generally at focus distances beyond the size of any lab test room.

For example, to properly test a wide angle landscape lens at its primary useful focus distance (infinity), it seems you would need a massive chart on the horizon rather than focusing within a lab controlled room to a piece of paper.

So, maybe my understanding of these tests is flawed and there is some means of testing at all focus distances and not just whatever distance fits a chart into that field of view?

Have there been thoughts on this in past discussed here? Are there maybe any wide angle lenses whose testing scores are invalid because they were tested at an irrelevant focus distance?

 techjedi's gear list:techjedi's gear list
Sony a7R III Sony FE 70-200 F4 Sony FE 90mm F2.8 macro Sony RX100 Sony a6500 +18 more
Jack Hogan Veteran Member • Posts: 7,528
Re: Is lens testing at chart distance really an accurate measure?
1

You are correct, it would be best to measure every lens at a few representative distances. However there is a law of diminishing returns so it is typically not necessary to have a chart at infinity: it is said that for most lenses for normal photography a distance of 30X focal length or more is a decent proxy for infinity performance. You are also allowed to move a smaller chart around in the field of view, as some do.

Jack

Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
Re: Is lens testing at chart distance really an accurate measure?
2

Hi,

Lenses are generally designed for infinity, except 'macro lenses' that are designed for close up work. Many modern lenses use 'floating elements' to keep sharpness within the whole focusing range.

MTF tests on the optical bench are normally done at infinity.

When testing with charts it is usually regarded to be satisfactory to test at around 50 times the focal length distance. That needs a huge test chart, like 1.2x1.8 m for 24x36 mm. Also the test target needs to be near absolutely flat and the camera needs to be very well aligned with the chart.

What probably varies most focusing distance is probably field curvature and astigmatism. Just as an example, Hasselblad used to have three 'Planar' lenses for then 'V' series:

  • The 120/4 Macro Planar that was designed for close up work.
  • The 100/3 Planar that was designed for demanding distant work, like aerial photography.
  • The 80/2.8 Planar that may have been a compromise.

The 120/4 macro has a lot of field curvature at distance. That is something I have actually observed.

The 100/3.5 Planar has quite horrible astigmatism at close range, which I also have verified.

Zeiss designed a new 120/4 Apo Macro Planar for the Contax 645. That lens had a more complex design with a floating group.

One way to test with a small target is to have it at distance and move the camera so it is positioned in different places in the image, but that needs the camera being refocused, so that test ignores field curvature.

In essence:

  • Test charts need to be large.
  • Small test charts are OK, if you plan to shoot objects of similar size.
  • The smaller the distance, the higher the need for precision.

Hasselblad has an interesting article on evolution of lenses: https://static.hasselblad.com/2015/02/the_evolution_of_lenses.pdf

That article posts a comparison of MTF for the Planar 100/3.5 and the Hasselblad HC 100/2.2 at infinity:

Both these lenses are good, but probably not excellent by today's standard.

Looking at the same lenses at 1.2 m, a different picture arises:

Here the Zeiss CF 3.5/100 has astigmatism while the 2.2/100 HC maintains sharpness.

What a modern lens looks like?

This is Hasselblad's data for their new 90 mm lens. Almost certainly at infinity. These data are calculated, taking diffraction into account. The data is not measured from a len sample.

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

Bernard Delley Senior Member • Posts: 1,491
Re: exhaustive testing becomes -- exhaustive
4

of course MTF depends, a little, on imaging ratio, or object distance. One needs a different chart for each imaging ratio, if the full chart is to be used.

I guess for a 'normal' indoors test station, it gets hard to go beyond a chart 2.4 x 3.6 m sized, giving a 1:100 imaging ratio onto the 24x36mm format. A DIN A0 size chart gets imaging ratio 1:33 for the same format. To fill the FX frame with a 800mm tele-lens you need a test hall exceeding 28m in length from the A0 chart ~ possible.

A test stand for 1:77 imaging with wide and normal lenses.

There are also Huge outdoors charts ! for aerial or satellite optics testing.

Maybe, somebody will do MTF testing, using buildings some day. There may be suitable buildings.

For closeup MTF testing, one may use razor blades.

Optical bench apparatus have a collimated test target. Infinity testing is the standard. For finite distance testing one needs a different collimator for each distance.

techjedi
OP techjedi Veteran Member • Posts: 3,961
Re: exhaustive testing becomes -- exhaustive

Bernard Delley wrote:

of course MTF depends, a little, on imaging ratio, or object distance. One needs a different chart for each imaging ratio, if the full chart is to be used.

I guess for a 'normal' indoors test station, it gets hard to go beyond a chart 2.4 x 3.6 m sized, giving a 1:100 imaging ratio onto the 24x36mm format. A DIN A0 size chart gets imaging ratio 1:33 for the same format. To fill the FX frame with a 800mm tele-lens you need a test hall exceeding 28m in length from the A0 chart ~ possible.

A test stand for 1:77 imaging with wide and normal lenses.

There are also Huge outdoors charts ! for aerial or satellite optics testing.

Maybe, somebody will do MTF testing, using buildings some day. There may be suitable buildings.

For closeup MTF testing, one may use razor blades.

Optical bench apparatus have a collimated test target. Infinity testing is the standard. For finite distance testing one needs a different collimator for each distance.

Interesting information. I was not aware of those huge charts for satellite optics!

 techjedi's gear list:techjedi's gear list
Sony a7R III Sony FE 70-200 F4 Sony FE 90mm F2.8 macro Sony RX100 Sony a6500 +18 more
techjedi
OP techjedi Veteran Member • Posts: 3,961
Re: Is lens testing at chart distance really an accurate measure?

Erik Kaffehr wrote:

Hi,

Lenses are generally designed for infinity, except 'macro lenses' that are designed for close up work. Many modern lenses use 'floating elements' to keep sharpness within the whole focusing range.

This makes sense to me in terms of design intent, but when I think of floating elements, I go back to the idea that zoom lenses are tested across their zoom range. I feel like if lens elements are moving, there are going to be spots that are sweeter than others. Perhaps its more reliable than I think and modern lenses really can be tested at one distance and extrapolate that performance to all other focus distances.

MTF tests on the optical bench are normally done at infinity.

So if you have a 12mm prime, how do you place a MTF chart at infinity?

When testing with charts it is usually regarded to be satisfactory to test at around 50 times the focal length distance. That needs a huge test chart, like 1.2x1.8 m for 24x36 mm. Also the test target needs to be near absolutely flat and the camera needs to be very well aligned with the chart.

Okay, so are you saying that a valid "infinity" focus test for a 12mm lens is having the chart 600mm away? That is less than the length of my arm and not what I would consider infinity focus? Forgive me if I misinterpreted the math there of "50 times focal length distance".

What probably varies most focusing distance is probably field curvature and astigmatism. Just as an example, Hasselblad used to have three 'Planar' lenses for then 'V' series:

  • The 120/4 Macro Planar that was designed for close up work.
  • The 100/3 Planar that was designed for demanding distant work, like aerial photography.
  • The 80/2.8 Planar that may have been a compromise.

The 120/4 macro has a lot of field curvature at distance. That is something I have actually observed.

The 100/3.5 Planar has quite horrible astigmatism at close range, which I also have verified.

Zeiss designed a new 120/4 Apo Macro Planar for the Contax 645. That lens had a more complex design with a floating group.

One way to test with a small target is to have it at distance and move the camera so it is positioned in different places in the image, but that needs the camera being refocused, so that test ignores field curvature.

This makes sense. I was wondering if smaller charts were used like this, so thank you for confirming that.

In essence:

  • Test charts need to be large.
  • Small test charts are OK, if you plan to shoot objects of similar size.
  • The smaller the distance, the higher the need for precision.

Hasselblad has an interesting article on evolution of lenses: https://static.hasselblad.com/2015/02/the_evolution_of_lenses.pdf

That article posts a comparison of MTF for the Planar 100/3.5 and the Hasselblad HC 100/2.2 at infinity:

Both these lenses are good, but probably not excellent by today's standard.

Looking at the same lenses at 1.2 m, a different picture arises:

Here the Zeiss CF 3.5/100 has astigmatism while the 2.2/100 HC maintains sharpness.

I don't normally see MTF for two distances, are these from MFR or a testing site?

What a modern lens looks like?

This is Hasselblad's data for their new 90 mm lens. Almost certainly at infinity. These data are calculated, taking diffraction into account. The data is not measured from a len sample.

Best regards

Erik

Is a calculated MTF trustworthy?

 techjedi's gear list:techjedi's gear list
Sony a7R III Sony FE 70-200 F4 Sony FE 90mm F2.8 macro Sony RX100 Sony a6500 +18 more
AiryDiscus Senior Member • Posts: 2,160
Re: Is lens testing at chart distance really an accurate measure?
1

Erik Kaffehr wrote:

Looking at the same lenses at 1.2 m, a different picture arises:

Here the Zeiss CF 3.5/100 has astigmatism while the 2.2/100 HC maintains sharpness.

That's definitely third order coma.  It's field linear and astigmatism is quadratic.

AiryDiscus Senior Member • Posts: 2,160
Re: exhaustive testing becomes -- exhaustive
1

Bernard Delley wrote:

Optical bench apparatus have a collimated test target. Infinity testing is the standard. For finite distance testing one needs a different collimator for each distance.

Not really.  It wouldn't be a "collimator," either.  Trioptics MTF benches physically place a crossed pair of slits at the object distance.  Optikos uses a projection with variable focus.  As long as the projector has sufficiently low wavefront error over the range of distances it can project, it is a nonissue.

AiryDiscus Senior Member • Posts: 2,160
Re: Is lens testing at chart distance really an accurate measure?
3

The typical chart size is between 1 and 2 meters wide. If you actually see someone test a 14mm lens on a full-frame camera that way, you might find it comical because the lens is only a few feet (maybe 2) from the chart.

The 30 or 50x focal length rules given in this thread for "infinity" are old and outdated. Modern lenses are very tightly engineered around their applications, a lot of wide angles are terrible close up like the fuji 16/1.4, but just fine focused afar.

It is a "correct" view that data for an object distance of a few ones of feet is only useful to your application at infinity if you drench it in assumptions that may or may not be invalid.

In general, with zoom lenses the more assumptions you make that are favorable to the lens, the further from reality you get.

Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
Re: Is lens testing at chart distance really an accurate measure?
1

AiryDiscus wrote:

The typical chart size is between 1 and 2 meters wide. If you actually see someone test a 14mm lens on a full-frame camera that way, you might find it comical because the lens is only a few feet (maybe 2) from the chart.

The 30 or 50x focal length rules given in this thread for "infinity" are old and outdated. Modern lenses are very tightly engineered around their applications, a lot of wide angles are terrible close up like the fuji 16/1.4, but just fine focused afar.

It is a "correct" view that data for an object distance of a few ones of feet is only useful to your application at infinity if you drench it in assumptions that may or may not be invalid.

In general, with zoom lenses the more assumptions you make that are favorable to the lens, the further from reality you get.

I see your point, but what criteria can we use to wisely spend our money on lenses?

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

Bernard Delley Senior Member • Posts: 1,491
accuracy, distance, continuity
1

techjedi wrote:

Erik Kaffehr wrote:

Hi,

Lenses are generally designed for infinity, except 'macro lenses' that are designed for close up work. Many modern lenses use 'floating elements' to keep sharpness within the whole focusing range.

This makes sense to me in terms of design intent, but when I think of floating elements, I go back to the idea that zoom lenses are tested across their zoom range. I feel like if lens elements are moving, there are going to be spots that are sweeter than others. Perhaps its more reliable than I think and modern lenses really can be tested at one distance and extrapolate that performance to all other focus distances.

MTF tests on the optical bench are normally done at infinity.

So if you have a 12mm prime, how do you place a MTF chart at infinity?

When testing with charts it is usually regarded to be satisfactory to test at around 50 times the focal length distance. That needs a huge test chart, like 1.2x1.8 m for 24x36 mm. Also the test target needs to be near absolutely flat and the camera needs to be very well aligned with the chart.

Okay, so are you saying that a valid "infinity" focus test for a 12mm lens is having the chart 600mm away? That is less than the length of my arm and not what I would consider infinity focus? Forgive me if I misinterpreted the math there of "50 times focal length distance".

If you think of the point spread function and the MTF as a function of distance, it is clear that these functions are also functions of distance, which stirred up the OP question. These functions of distance must be continuous functions. It makes engineering sense to optimize lens performance such that performance stays reasonably high in the range of intended focus distances. The peak performance would probably happen at a finite distance.

The performance variations versus distance may be seen in comparison with performance variations across the image field. Such variations across the image field may easily amount to +- 0.1 [MTF] , as was shown in a previous thread, range of MTF variation . The image field MTF for a real lens is only within a range of what the radial MTF plots suggest.

One way to test with a small target is to have it at distance and move the camera so it is positioned in different places in the image, but that needs the camera being refocused, so that test ignores field curvature.

This makes sense. I was wondering if smaller charts were used like this, so thank you for confirming that.

In essence:

  • Test charts need to be large.
  • Small test charts are OK, if you plan to shoot objects of similar size.
  • The smaller the distance, the higher the need for precision.

I don't normally see MTF for two distances, are these from MFR or a testing site?

What a modern lens looks like?

Is a calculated MTF trustworthy?

Calculated MTF represent a design expectation. How the real lens with its manufacturing tolerances turns out, may be on a different page. It is certainly illuminating to measure, if you are interested in photographic science and technology.

Bernard Delley Senior Member • Posts: 1,491
measurement as reality check
2

Erik Kaffehr wrote:

I see your point, but what criteria can we use to wisely spend our money on lenses?

I returned a promising lens the same day, after I had measured its image field MTF .

Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
Re: measurement as reality check

Bernard Delley wrote:

Erik Kaffehr wrote:

I see your point, but what criteria can we use to wisely spend our money on lenses?

I returned a promising lens the same day, after I had measured its image field MTF .

Hi Bernard,

Thanks for sharing! Would be interesting to read more about it!

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
Re: Is lens testing at chart distance really an accurate measure?
1

AiryDiscus wrote:

Erik Kaffehr wrote:

Looking at the same lenses at 1.2 m, a different picture arises:

Here the Zeiss CF 3.5/100 has astigmatism while the 2.2/100 HC maintains sharpness.

That's definitely third order coma. It's field linear and astigmatism is quadratic.

Thanks for the correction! Learning something new each day. Lots of things learn so I hope for a long life.

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

TheGrammarFairy Regular Member • Posts: 360
Are There Any Rules of Thumb?

If a photographer wants to perform a test on a lens to find  the optimal subject-lens distance are there any shortcuts? Or is it just a matter of starting at 30cm from your test chart and moving backward in 30cm increments until you're at infinity.

Or, God help us, do the increments need to be shorter than that?

wrote:

The typical chart size is between 1 and 2 meters wide. If you actually see someone test a 14mm lens on a full-frame camera that way, you might find it comical because the lens is only a few feet (maybe 2) from the chart.

The 30 or 50x focal length rules given in this thread for "infinity" are old and outdated. Modern lenses are very tightly engineered around their applications, a lot of wide angles are terrible close up like the fuji 16/1.4, but just fine focused afar.

It is a "correct" view that data for an object distance of a few ones of feet is only useful to your application at infinity if you drench it in assumptions that may or may not be invalid.

In general, with zoom lenses the more assumptions you make that are favorable to the lens, the further from reality you get.

AiryDiscus Senior Member • Posts: 2,160
Re: Are There Any Rules of Thumb?

nothing wrong with curve fitting.  It will be a slowly varying smooth function away from the minimum focus.

burning1rr Contributing Member • Posts: 583
Re: Is lens testing at chart distance really an accurate measure?

techjedi wrote:

So if you have a 12mm prime, how do you place a MTF chart at infinity?

It's a good point, but I think that the original argument is valid; MTF tests are more useful when they reflect real world focus distances.

If you want to test the MTF of a 14mm lens, it's better to use a wall sized target at 2m than a poster sized target at half a meter.

 burning1rr's gear list:burning1rr's gear list
Sony RX100 VA Nikon D200 Sony a9 Sony a7 III Sony 135mm F2.8 (T4.5) STF +16 more
Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
It may depend on your needs...
1

I don't think charts are useful for testing. To begin with, they need to be flat and well aligned to camera sensor.

If you want to check out a lens, I would recommend Jim Kasson's Lens screening test:

https://blog.kasson.com/lens-screening-testing/

If you want more quantitative testing, it may make some sense to replace Jim's 'Siemens star' with slanted wedges for MTF testing.

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

Bernard Delley Senior Member • Posts: 1,491
Re: It may depend on your needs...
1

Erik Kaffehr wrote:

I don't think charts are useful for testing. To begin with, they need to be flat and well aligned to camera sensor.

If you want to check out a lens, I would recommend Jim Kasson's Lens screening test:

https://blog.kasson.com/lens-screening-testing/

If you want more quantitative testing, it may make some sense to replace Jim's 'Siemens star' with slanted wedges for MTF testing.

I like the quantitative results from chart MTF testing. I think it is worth the extra effort. It is mostly an initial effort learning to set up and analyze properly.

In any case, the real challenge is how to return an obviously sub-par lens without incurring excessive costs. Did you ever return a brand new lens ? Returning the lens, which I mentioned before, cost me 16% of the lens price. Probably a fair cost. However, I did not wish to try more lens samples of this model. -- I wonder what the shop does with the returned lens, which looks perfect by eyeball inspection.

Erik Kaffehr
Erik Kaffehr Veteran Member • Posts: 4,740
Re: It may depend on your needs...

Bernard Delley wrote:

Erik Kaffehr wrote:

I don't think charts are useful for testing. To begin with, they need to be flat and well aligned to camera sensor.

If you want to check out a lens, I would recommend Jim Kasson's Lens screening test:

https://blog.kasson.com/lens-screening-testing/

If you want more quantitative testing, it may make some sense to replace Jim's 'Siemens star' with slanted wedges for MTF testing.

I like the quantitative results from chart MTF testing. I think it is worth the extra effort. It is mostly an initial effort learning to set up and analyze properly.

In any case, the real challenge is how to return an obviously sub-par lens without incurring excessive costs. Did you ever return a brand new lens ? Returning the lens, which I mentioned before, cost me 16% of the lens price. Probably a fair cost. However, I did not wish to try more lens samples of this model. -- I wonder what the shop does with the returned lens, which looks perfect by eyeball inspection.

Hi,

I did it once, with a Tamron that was obviously broken,

I also claimed warranty on a Konica Minolta lens, but was told it was OK.

Best regards

Erik

-- hide signature --

Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles

Keyboard shortcuts:
FForum MMy threads