Comparing various measures of lens sharpness

[Malcolm Richardson]

My dilemma

I understand that all sharpness tests actually test a system that not only includes the lens but also sensor, +/- AA filter, de-mosaic and raw conversion algorithms, sharpening etc but the main variable day to day with one’s own setup will be the lenses used.

When trying to compare lens sharpness tests on different sites, I get confused that they often report their tests differently eg lw/ph, lp/mm, a numeric Imatest score, a descriptive score (good, excellent, outstanding etc) and even manufacturer MTF charts.

Manufacturer MTF charts are useful but in my case (Nikon) the charts are calculated, not measured, and they only give values for the widest aperture and longest and shortest focal lengths.

It is fairly easy to understand lw/ph and lp/mm and one can be derived from the other. Bar charts using lw/ph look very similar to those from Imatest scores but they are not the same; the latter simply gives a score without any units that I can see. I realise that Imatest use a number of measurements and calculations to arrive at a single score. Whilst not being the same thing their values will be related ie a lens with a high Imatest score will likely also score a high lw/ph value.

My Question

I guess the simple rule is that the higher any score is the better but is there a simple conversion factor or equation that allows one to calculate lw/ph from an Imatest score? Is it even meaningful to try? Are there absolute Imatest scores that correspond to the descriptive values such as fair, good, excellent, outstanding etc or do the values for the descriptive scores vary with each lens/sensor system?

I’d be grateful for other’s thoughts,

Regards,


Malcolm R

 

[ahaslett]

My dilemma

I understand that all sharpness tests actually test a system that not only includes the lens but also sensor, +/- AA filter, de-mosaic and raw conversion algorithms, sharpening etc but the main variable day to day with one’s own setup will be the lenses used.

LensRentals tests using OLAF test the lens only

When trying to compare lens sharpness tests on different sites, I get confused that they often report their tests differently eg lw/ph, lp/mm, a numeric Imatest score, a descriptive score (good, excellent, outstanding etc) and even manufacturer MTF charts.

Manufacturer MTF charts are useful but in my case (Nikon) the charts are calculated, not measured, and they only give values for the widest aperture and longest and shortest focal lengths.

They reflect lens only tests. Apart from lenses tested by LensRentals, you can’t compare them with lens tested using a camera body. They also ideal tolerances and assembly.

It is fairly easy to understand lw/ph and lp/mm and one can be derived from the other. Bar charts using lw/ph look very similar to those from Imatest scores but they are not the same; the latter simply gives a score without any units that I can see. I realise that Imatest use a number of measurements and calculations to arrive at a single score. Whilst not being the same thing their values will be related ie a lens with a high Imatest score will likely also score a high lw/ph value.

My Question

I guess the simple rule is that the higher any score is the better but is there a simple conversion factor or equation that allows one to calculate lw/ph from an Imatest score? Is it even meaningful to try? Are there absolute Imatest scores that correspond to the descriptive values such as fair, good, excellent, outstanding etc or do the values for the descriptive scores vary with each lens/sensor system?

This is the Imatest article https://www.imatest.com/docs/lens_testing/

You seem to understand test results at MTF50 that show Centre, Edge and Corner scores in either lppph or lppmm. You just need to convert from the sensor height to whatever the standard picture height is.

An MFT chart that shows MTF vs lppmm like the one in the Imatest article, can be converted to MFT50 by interpolating between the line frequencies for the point where MFT50 crosses them, ie 0.5 on the chart. I guess MTF50 means 50%.

MTF of the system is the MTF of each step in the process multiplied together. Sensor resolution is pretty important to high frequency MTF.

While the MTF50 charts are interesting, MTF vs line frequency is more interesting still. Low frequency MTF roughly equates in my mind to resolution. High frequency MTF to micro-contrast. It is not inevitable that a lens with good resolution also has good micro-contrast.

Go and have a look at Roger Cicala’s blog for OLAF tests on any lenses you own. Try the lenses on bodies with different sensor resolutions if you can. Use subjects like distant hedges with tonality and colour gradients on leaves and sudden transitions at edges.

See what that tells you about the appearance of different test results.

Roger has other interesting things to say about astigmatism, field curvature, copy variation…

I’d be grateful for other’s thoughts,

Regards,

Malcolm R

What is good or excellent seems to change with time.

A

--
Infinite are the arguments of mages. Truth is a jewel with many facets. Ursula K LeGuin
Please feel free to edit any images that I post

 

[Ephemeris]

My dilemma

I understand that all sharpness tests actually test a system that not only includes the lens but also sensor, +/- AA filter, de-mosaic and raw conversion algorithms, sharpening etc but the main variable day to day with one’s own setup will be the lenses used.

LensRentals tests using OLAF test the lens only

https://www.hopkinscenter.rochester.edu/assets/pdf/research/mtffullfielddisplays_dube.pdf

Olaf is a company which I have a feeling doesn't exist now. Anyhow, the method Lensrental uses I assume is what is described in the above paper. What is the sensor element of the system? I.e. what is doing the measuring?

When trying to compare lens sharpness tests on different sites, I get confused that they often report their tests differently eg lw/ph, lp/mm, a numeric Imatest score, a descriptive score (good, excellent, outstanding etc) and even manufacturer MTF charts.

Manufacturer MTF charts are useful but in my case (Nikon) the charts are calculated, not measured, and they only give values for the widest aperture and longest and shortest focal lengths.

They reflect lens only tests. Apart from lenses tested by LensRentals, you can’t compare them with lens tested using a camera body. They also ideal tolerances and assembly.

It is fairly easy to understand lw/ph and lp/mm and one can be derived from the other. Bar charts using lw/ph look very similar to those from Imatest scores but they are not the same; the latter simply gives a score without any units that I can see. I realise that Imatest use a number of measurements and calculations to arrive at a single score. Whilst not being the same thing their values will be related ie a lens with a high Imatest score will likely also score a high lw/ph value.

My Question

I guess the simple rule is that the higher any score is the better but is there a simple conversion factor or equation that allows one to calculate lw/ph from an Imatest score? Is it even meaningful to try? Are there absolute Imatest scores that correspond to the descriptive values such as fair, good, excellent, outstanding etc or do the values for the descriptive scores vary with each lens/sensor system?

This is the Imatest article https://www.imatest.com/docs/lens_testing/

You seem to understand test results at MTF50 that show Centre, Edge and Corner scores in either lppph or lppmm. You just need to convert from the sensor height to whatever the standard picture height is.

An MFT chart that shows MTF vs lppmm like the one in the Imatest article, can be converted to MFT50 by interpolating between the line frequencies for the point where MFT50 crosses them, ie 0.5 on the chart. I guess MTF50 means 50%.

MTF of the system is the MTF of each step in the process multiplied together. Sensor resolution is pretty important to high frequency MTF.

While the MTF50 charts are interesting, MTF vs line frequency is more interesting still. Low frequency MTF roughly equates in my mind to resolution. High frequency MTF to micro-contrast. It is not inevitable that a lens with good resolution also has good micro-contrast.

Go and have a look at Roger Cicala’s blog for OLAF tests on any lenses you own. Try the lenses on bodies with different sensor resolutions if you can. Use subjects like distant hedges with tonality and colour gradients on leaves and sudden transitions at edges.

See what that tells you about the appearance of different test results.

Roger has other interesting things to say about astigmatism, field curvature, copy variation…

I’d be grateful for other’s thoughts,

Regards,

Malcolm R

What is good or excellent seems to change with time.

A

--
Infinite are the arguments of mages. Truth is a jewel with many facets. Ursula K LeGuin
Please feel free to edit any images that I post

 

[ahaslett]

OLAF is what LensRentals call their Trioptics Imagemaster MTF bench. Trioptics make more than one optical bench, so it's not clear which one is used for OLAF.

How they allow for sensor characteristics and data processing is a mystery. Maybe Trioptics publish technical papers.

A

 

[Ephemeris]

OLAF is what LensRentals call their Trioptics Imagemaster MTF bench. Trioptics make more than one optical bench, so it's not clear which one is used for OLAF.

I understood it was Olaf Optical Testing LLC (which is linked to the paper I posted).

The paper doesn't describe the system that obtains the data which is to be processed. Some of Roger Cs testing uses a camera so I had wondered what was used, if not a camera in these tests.

How they allow for sensor characteristics and data processing is a mystery. Maybe Trioptics publish technical papers.

A

--
Infinite are the arguments of mages. Truth is a jewel with many facets. Ursula K LeGuin
Please feel free to edit any images that I post

 

[Ephemeris]

OLAF is what LensRentals call their Trioptics Imagemaster MTF bench. Trioptics make more than one optical bench, so it's not clear which one is used for OLAF.

I understood it was Olaf Optical Testing LLC (which is linked to the paper I posted).

The paper doesn't describe the system that obtains the data which is to be processed. Some of Roger Cs testing uses a camera so I had wondered what was used, if not a camera in these tests.

How they allow for sensor characteristics and data processing is a mystery. Maybe Trioptics publish technical papers.

A

 

[ahaslett]

OLAF is what LensRentals call their Trioptics Imagemaster MTF bench. Trioptics make more than one optical bench, so it's not clear which one is used for OLAF.

I understood it was Olaf Optical Testing LLC (which is linked to the paper I posted).

The paper doesn't describe the system that obtains the data which is to be processed. Some of Roger Cs testing uses a camera so I had wondered what was used, if not a camera in these tests.

How they allow for sensor characteristics and data processing is a mystery. Maybe Trioptics publish technical papers.

A

I can see why I was a little confused. So they called the Imagemaster OLAF? Which must link to the company at the time OLAF?

https://www.lensrentals.com/blog/2014/05/introducing-the-optical-bench/

I've seen machines that test automotive lenses across wide temperature ranges from the same company but I'm not sure how you use the data once it's bolted to a camera. Or do they (lensrentals) use it to check for faults / damage of lenses?

Lensrentals primary use was checking the adjustments of lenses returned from repair. Roger also did contract work on QC recommendations for lens manufacturers. When there was spare time and it interested him, he would do various tests on lenses in the LensRentals collection.

He produced about the only useful published work on copy variation. Some interesting stuff in his blogs on field curvature, tilt, astigmatism etc.

Here is an example blog with various topics in it https://www.lensrentals.com/blog/2018/01/finally-some-m43-mtf-testing-25mm-prime-lens-comparison/

A

 

[Ephemeris]

OLAF is what LensRentals call their Trioptics Imagemaster MTF bench. Trioptics make more than one optical bench, so it's not clear which one is used for OLAF.

I understood it was Olaf Optical Testing LLC (which is linked to the paper I posted).

The paper doesn't describe the system that obtains the data which is to be processed. Some of Roger Cs testing uses a camera so I had wondered what was used, if not a camera in these tests.

How they allow for sensor characteristics and data processing is a mystery. Maybe Trioptics publish technical papers.

A

I can see why I was a little confused. So they called the Imagemaster OLAF? Which must link to the company at the time OLAF?

https://www.lensrentals.com/blog/2014/05/introducing-the-optical-bench/

I've seen machines that test automotive lenses across wide temperature ranges from the same company but I'm not sure how you use the data once it's bolted to a camera. Or do they (lensrentals) use it to check for faults / damage of lenses?

Lensrentals primary use was checking the adjustments of lenses returned from repair. Roger also did contract work on QC recommendations for lens manufacturers. When there was spare time and it interested him, he would do various tests on lenses in the LensRentals collection.

He produced about the only useful published work on copy variation. Some interesting stuff in his blogs on field curvature, tilt, astigmatism etc.

Here is an example blog with various topics in it https://www.lensrentals.com/blog/2018/01/finally-some-m43-mtf-testing-25mm-prime-lens-comparison/

A

 

[Malcolm Richardson]

Many thanks for your reply ahaslett. You have given me plenty of homework reading but I think it is all beginning to make sense.

This is the Imatest article https://www.imatest.com/docs/lens_testing/

I had found my way to this site but not to this page. For me the most helpful section was on how to interpret the results and the link to the page which gives instructions and an explanation of Batchview. The original author (Norman Koren I believe) explains that each individual curve is a plot of MTF against LW/PH and that the Batchview is simply a composite display of multiple individual MTF50 measurements at various focal lengths and apertures and then displayed in bar graph form.

It seems that some sites (eg ePHOTOzine) include the units whilst others (eg Photography Life) don't when publishing their own tests.

You seem to understand test results at MTF50 that show Centre, Edge and Corner scores in either lppph or lppmm. You just need to convert from the sensor height to whatever the standard picture height is.

An MFT chart that shows MTF vs lppmm like the one in the Imatest article, can be converted to MFT50 by interpolating between the line frequencies for the point where MFT50 crosses them, ie 0.5 on the chart. I guess MTF50 means 50%.

MTF of the system is the MTF of each step in the process multiplied together. Sensor resolution is pretty important to high frequency MTF.

While the MTF50 charts are interesting, MTF vs line frequency is more interesting still. Low frequency MTF roughly equates in my mind to resolution. High frequency MTF to micro-contrast. It is not inevitable that a lens with good resolution also has good micro-contrast.

Understood and agreed.

Go and have a look at Roger Cicala’s blog for OLAF tests on any lenses you own. Try the lenses on bodies with different sensor resolutions if you can. Use subjects like distant hedges with tonality and colour gradients on leaves and sudden transitions at edges.

See what that tells you about the appearance of different test results.

Roger has other interesting things to say about astigmatism, field curvature, copy variation…

I have found my way to Roger's blog at LensRentals and while I can access some of his articles, I don't yet seem able to access the links to individual lens tests. My main motive has been to get the best out of the gear that I have and, as you suggest, trying it out in multiple different situations and settings is my practical answer.

Once again, many thanks

Malcolm R

 

[Malcolm Richardson]

Many thanks Ephemeris

https://www.hopkinscenter.rochester.edu/assets/pdf/research/mtffullfielddisplays_dube.pdf

I have downloaded the paper and will work through it.

Regards, M

 

[Bill Ferris]

Have you tried searching YouTube for reviews of the lens(es) in which you're interested? Depending on the genre of photography you're engaged in and the brand/model optic(s) of interest, you may find a YouTuber who performs side-by-side comparisons of lenses on test charts, staged scenes, and in real-world shooting environments. They can be quite illuminating.

 

[Malcolm Richardson]

Thanks Bill yes I have done this and have watched many YouTube videos. The link that ahaslett (Andrew?) posted also took me to the page on the Imatest site that answered my question. Regards,

 

[J A C S]

As much as I like measurements, I have not looked at any in the last few years. I prefer to look at photos. The-digital-picture has crops of chart shots for a huge collection of Canon lenses. Bryan tests some Nikon and Sony lenses, too. Then the galleries this site posts.

 

[Bernard Delley]

is lensrentals surname for their Trioptics bench.

You can fins specs at Trioptics for bench measurement performance.

The collimators are high precision optical systems projecting the target structures to virtual infinity.

The sensors are pixel sensors of course. They need well known corrections.

Pixel corrections and calibrations are described for camera pixel sensors in my article on full filed MTF with charts.

 

[Bernard Delley]

I have some measurements for you at dpreview:

MTF @ f/5.6 long lens comparison

full field through focus and stopped down measurementsin example from older thread here

Z 24-120mm f/4 sharpness : full field MTF

Z100-400mm sharpness: city view + full field MTF

and an open access link to my MTF methodologies and results publication in Applied Optics

 

[Malcolm Richardson]

Very helpful information Bernard. Thanks for providing such detailed testing.

 

[JimKasson]

I guess the simple rule is that the higher any score is the better but is there a simple conversion factor or equation that allows one to calculate lw/ph from an Imatest score? Is it even meaningful to try? Are there absolute Imatest scores that correspond to the descriptive values such as fair, good, excellent, outstanding etc or do the values for the descriptive scores vary with each lens/sensor system?

Imatest will give MTF curves with vertical axes of cycles per picture height and cycles per pixel. They also compute MTF50 and MTF30 for those dimensions.

 

[Malcolm Richardson]

Thanks Jim, I think that one of the things that was confusing me most was that some review sites don’t include any units when they publish MTF graphs.

 

[JimKasson]

Thanks Jim, I think that one of the things that was confusing me most was that some review sites don’t include any units when they publish MTF graphs.

Makes them not very useful, huh? I've noticed that some manufacturers don't provide all the units when they publish MTF curves, too.



Here's a more or less typical Imatest plot.

--
https://blog.kasson.com

 

[Ephemeris]

Thanks Jim, I think that one of the things that was confusing me most was that some review sites don’t include any units when they publish MTF graphs.

Makes them not very useful, huh? I've noticed that some manufacturers don't provide all the units when they publish MTF curves, too.

Here's a more or less typical Imatest plot.

--
https://blog.kasson.com

Hi Jim

Without wishing to colour the answer from my likely lack of knowledge what if any are the potential issues when reading these charts?

They do appear to be very much used in the photographic world rather than other industries that use similar systems. I don't know why but would appreciate your knowledge.