Reading resolution charts comparing MFT lenses to FX lenses

Started Mar 29, 2013 | Questions
Prairie Pal
Senior MemberPosts: 2,208Gear list
Like?
Reading resolution charts comparing MFT lenses to FX lenses
Mar 29, 2013

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

ANSWER:
This question has not been answered yet.
Chatokun
Regular MemberPosts: 447Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Prairie Pal, Mar 29, 2013

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

I remember reading an explanation on this site than the lw/ph range can't exceed the number of pixels. A perfect lens (read impossible) would translate the same lw/ph as the number of pixels to the sensor, and I think the sensor also has to have little too no filters in front. IIRC, there are color, Anti Aliasing, and possibly other filters, plus lenses are not perfect, having multiple elements etc, so the effective max lw/ph is less, but still relative to pixels. This is why the lenses tested on the GX1 score better than the ones tested on GF1, even if they in real world have similar sharpness.

If I'm remembering right (big if) then the FFs are probably on larger MP senors. Just glanced at the canon 85mm, it's tested on the 5,616 × 3,744 pixel Canon 5D Mark II. It had marks up to 3600ish at peak. The 50mm I glanced at was a little less. Let me see if they have some higher MP ones...

The Nikon tests seem to be done on the D3X (6,048 × 4,032). This one has lenses (such as a similar 85mm 1.4) that raises into the 4016 range. As expected with higher MP.

Sony has A900 (6048 x 4032) but the lenses tested don't go above 3700 it seems.

 Chatokun's gear list:Chatokun's gear list
Panasonic Lumix DMC-LX7 Olympus PEN E-PL1 Panasonic Lumix DMC-GF3 Olympus OM-D E-M5 Panasonic Lumix G 14mm F2.5 ASPH +10 more
Reply   Reply with quote   Complain
Great Bustard
Forum ProPosts: 24,127
Like?
This is a tough one...
In reply to Prairie Pal, Mar 29, 2013

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

...and Anders will hopefully chime in so you can watch us argue about it. 

First of all, what PZ measures is the MTF-50, which is low many line widths per picture height (lw/ph) are resolved at 50% contrast.

Now, Klaus clearly states:

Please note that the tests results are not comparable across the different systems!

but he does not go into detail about why, so let's discuss that point for the rest of this post.

First of all, the resolution in the final photo, which is what PZ measures, is a function of many different factors:

  • Lens sharpness
  • Pixel count
  • Sensor size
  • AA filter
  • RAW conversion
  • Additional processing

In other words, you aren't looking at lens tests, you are looking at system tests (this is true for all other testing sites as well).

I discuss the relationship between lens sharpness and sensor size here, and the relationship between pixel count and resolution here, so I'll move on to other factors in this post.

As we all know, a weaker AA filter will result in what appears to be more detailed files, but much of this additional "detail" may well be false detail.  Depending on the processing, this false detail can look very pleasing and also result in higher MTF-50 resolution scores.  Indeed, it's not unlike the idea behind upsampling a photo for more "resolution".  That is, you cannot get more resolution than the initial capture, but with good enough software, you can make enough "lucky guesses" that the distinction between real and made up is inconsequential.  Of course, there are limits, however.

In addition, we all know that different demosaicking algorithms can have a marked impact on the look of a photo, and that includes resolution.  Thus, using different RAW converters for different files, or even the same RAW converter that treats the files from different systems differently, and not necessarily differently in an optimum way for each system, can cause further problems still.

Lastly, and this is where Anders and I bump heads, is on the matter of sharpening.  We normally don't think of noise at base ISO in good light to be an issue.  However, the noise differential exists even if it's not visible.

You might ask, if it's not visible, then what does it matter?  Well, the reason it matters is that the lower noise photo can take more sharpening.  Thus, by comparing photos from, say, a D800 and EM5, with no sharpening applied, well, that artificially handicaps the D800 file which has lower noise and can take more sharpening than the EM5 file (for a given exposure).

Of course, then we have equivalent photos which are photos at the same DOF and shutter speed.  For these types of photos, no format is favored over the other -- the noise differential is due to differences in sensor efficiency, and that difference can vary throughout the ISO range.  For example, the D800 may be a stop more efficient than the EM5 sensor at base ISO, but a stop less efficient at ISO 3200 (just throwing numbers out there -- I haven't actually worked that particular comparison out).

So, in the end, what we have to ask is how much variation all these factors cause.  The answer is:  I don't have a freakin' clue.  But, to compound the issue further still, there's the issue of copy variation among lenses, for example, this article, or this article. or this article.

Lastly, we have to ask how the MTF-50 lw/ph measurements correspond to the visual properties of the final photo.  Can you notice a 50% difference?  One would think so, but at what display size would you not notice?  How about 40%?  30%?  20%?  10%?

Fact of the matter is that anyone telling you f/11 is a perfectly good aperture to shoot at is already losing massive amounts of resolution due to diffraction softening (the sharper the lens, or the greater the pixel count, the more resolution they are losing, although they'll always be left with more resolution than if they started off with a softer lens or a sensor with fewer pixels), so MTF-50 scores, and thus resolution, probably aren't much of a concern to them at all, as they have more than enough, anyway.

So, does that at least help in answering your question, or just confuse the hell out of you? 

Reply   Reply with quote   Complain
Prairie Pal
Senior MemberPosts: 2,208Gear list
Like?
Re: This is a tough one...
In reply to Great Bustard, Mar 29, 2013

So, does that at least help in answering your question, or just confuse the hell out of you? 

I will re-read your post and links a couple of times before I can commit an answer to your question.  .  That's a lot of information, and some of it touches on other related questions I had (such as how much effect different RAW converters have and if the manufacturers converter is always the best one to use, "bar_none" so to speak).  But I can leave that for another post.

But anyways, who are you and why do you know so much about this topic?  Does it apply to your profession?  And how did it come to pass that you are so much smarter than this Anders guy   Sorry, just being spitefull, as I don't really remember seeing your guys (or girls?) bump heads.

Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: This is a tough one...
In reply to Great Bustard, Mar 29, 2013

Great Bustard wrote:

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

...and Anders will hopefully chime in so you can watch us argue about it. 

First of all, what PZ measures is the MTF-50, which is low many line widths per picture height (lw/ph) are resolved at 50% contrast.

Now, Klaus clearly states:

Please note that the tests results are not comparable across the different systems!

but he does not go into detail about why, so let's discuss that point for the rest of this post.

First of all, the resolution in the final photo, which is what PZ measures, is a function of many different factors:

  • Lens sharpness
  • Pixel count
  • Sensor size
  • AA filter
  • RAW conversion
  • Additional processing

In other words, you aren't looking at lens tests, you are looking at system tests (this is true for all other testing sites as well).

I discuss the relationship between lens sharpness and sensor size here, and the relationship between pixel count and resolution here, so I'll move on to other factors in this post.

As we all know, a weaker AA filter will result in what appears to be more detailed files, but much of this additional "detail" may well be false detail.  Depending on the processing, this false detail can look very pleasing and also result in higher MTF-50 resolution scores.  Indeed, it's not unlike the idea behind upsampling a photo for more "resolution".  That is, you cannot get more resolution than the initial capture, but with good enough software, you can make enough "lucky guesses" that the distinction between real and made up is inconsequential.  Of course, there are limits, however.

In addition, we all know that different demosaicking algorithms can have a marked impact on the look of a photo, and that includes resolution.  Thus, using different RAW converters for different files, or even the same RAW converter that treats the files from different systems differently, and not necessarily differently in an optimum way for each system, can cause further problems still.

Lastly, and this is where Anders and I bump heads, is on the matter of sharpening.  We normally don't think of noise at base ISO in good light to be an issue.  However, the noise differential exists even if it's not visible.

I think we actually agree on most of the facts involved but differ in the conclusions we draw from them. I think that as far as tests are concerned, it is useful to keep MTF performance apart from noise performance as much as possible. There are multiple reasons for this.

First, the way Photozone does it means that we mix the two in unknown ways. There is no way for me as a potential user of Photozone data to know the mix they have chosen for a particular camera/sensor and how that mix compares to others. This means that the results lack transparency.

Second, as I am sure you agree, no mix is appropriate for all conditions. As you point out below, there are cases where a D800 beats an E-M5 for noise but also situations where it is the other way around.

Third, it makes a difference to me whether a certain MTF result has been achieved with or without sharpening. Sharpening in PP is to a considerable extent a way to fool the eye and based on the visual evidence I have looked at, I'll take the unsharpened output from a good lens over the sharpened output from an inferior one any day, even though they might both reach the same MTF values at the same noise level.

My preferred solution to these problems is to present the results of MTF tests in such a way that an informed reader can draw his/her own conclusions with all relevant information on the table. At the present time, the two sources of information that I find preferable in this regard are Lenstip and LensRentals. Both present results based on RAWs converted by the same known algoritms without any additional sharpening. The information from LensRentals is presented in a form that simplifies cross-format comparisons since it is reported as line pairs per image height (lp/ih). The results from Lenstip are reported as line pairs per mm (lp/mm) and therefore have to be converted before they can be compared. But this is not difficult to do. To compare between FF and MFT, for example, you merely have to multiply the FF results by two or, alternatively, divide the MFT results by two.

You might ask, if it's not visible, then what does it matter?  Well, the reason it matters is that the lower noise photo can take more sharpening.  Thus, by comparing photos from, say, a D800 and EM5, with no sharpening applied, well, that artificially handicaps the D800 file which has lower noise and can take more sharpening than the EM5 file (for a given exposure).

Of course, then we have equivalent photos which are photos at the same DOF and shutter speed.  For these types of photos, no format is favored over the other -- the noise differential is due to differences in sensor efficiency, and that difference can vary throughout the ISO range.  For example, the D800 may be a stop more efficient than the EM5 sensor at base ISO, but a stop less efficient at ISO 3200 (just throwing numbers out there -- I haven't actually worked that particular comparison out).

So, in the end, what we have to ask is how much variation all these factors cause.  The answer is:  I don't have a freakin' clue.  But, to compound the issue further still, there's the issue of copy variation among lenses, for example, this article, or this article. or this article.

Lastly, we have to ask how the MTF-50 lw/ph measurements correspond to the visual properties of the final photo.  Can you notice a 50% difference?  One would think so, but at what display size would you not notice?  How about 40%?  30%?  20%?  10%?

Fact of the matter is that anyone telling you f/11 is a perfectly good aperture to shoot at is already losing massive amounts of resolution due to diffraction softening (the sharper the lens, or the greater the pixel count, the more resolution they are losing, although they'll always be left with more resolution than if they started off with a softer lens or a sensor with fewer pixels), so MTF-50 scores, and thus resolution, probably aren't much of a concern to them at all, as they have more than enough, anyway.

So, does that at least help in answering your question, or just confuse the hell out of you? 

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Prairie Pal, Mar 29, 2013

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?

Not sure what you mean by relative but I certainly wouldn't call the difference real.

Surely even the best MFT glass isn't inferior to FX.

No it certainly isn't.

I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

The results of Photozone are meant for comparisons within a system, or more specifically, a certain sensor. If you want to compare across systems/sensors, there are, for reasons explained in my reply to Great Bustard above, better sources available, and these, not unexpectedly, yield a very different outcome when comparing FF and MFT lenses than do those of Photozone. Note that comparisons across systems/sensors always involve not only the lens but also the sensor. But as long as we keep that in mind, there is no problem.

To take but one example, here is how the best MFT normal lens (in terms of MTF values only, and according to LensRentals) mounted on an E-M5 compares to the best FF normal lens (by the same token) mounted on a Leica M9. The figures are lp/ih (line pairs per image height) for center/average.

Panasonic 20/1.7 on an E-M5 (16 MP)

1.7 870/735

2.8 1050/875

4.0 1075/880

Leica Summilux 50/1.4 on an M9 (18 MP, no AA filter)

1.4 600/530

2.0 950/740

2.8 1025/860

4.0 1110/980

Sources:

http://www.lensrentals.com/blog/2012/05/wide-angle-micro-43-imatest-results

http://www.lensrentals.com/blog/2012/01/the-great-50mm-shootout

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
richarddd
Senior MemberPosts: 2,854Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Anders W, Mar 29, 2013

Anders W wrote:

To take but one example, here is how the best MFT normal lens (in terms of MTF values only, and according to LensRentals) mounted on an E-M5 compares to the best FF normal lens (by the same token) mounted on a Leica M9. The figures are lp/ih (line pairs per image height) for center/average.

Panasonic 20/1.7 on an E-M5 (16 MP)

But everyone knows the Panasonic 25/1.4 is a much better lens. Just read the posts on this forum

1.7 870/735

2.8 1050/875

4.0 1075/880

Leica Summilux 50/1.4 on an M9 (18 MP, no AA filter)

1.4 600/530

2.0 950/740

2.8 1025/860

4.0 1110/980

Sources:

http://www.lensrentals.com/blog/2012/05/wide-angle-micro-43-imatest-results

http://www.lensrentals.com/blog/2012/01/the-great-50mm-shootout

 richarddd's gear list:richarddd's gear list
Olympus OM-D E-M5 Panasonic Lumix G Vario 7-14mm F4 ASPH Olympus M.Zuiko Digital ED 40-150mm 1:4-5.6 R Olympus M.Zuiko Digital 45mm 1:1.8 Samyang 7.5mm F3.5 UMC Fisheye MFT +3 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to richarddd, Mar 29, 2013

richarddd wrote:

Anders W wrote:

To take but one example, here is how the best MFT normal lens (in terms of MTF values only, and according to LensRentals) mounted on an E-M5 compares to the best FF normal lens (by the same token) mounted on a Leica M9. The figures are lp/ih (line pairs per image height) for center/average.

Panasonic 20/1.7 on an E-M5 (16 MP)

But everyone knows the Panasonic 25/1.4 is a much better lens. Just read the posts on this forum

Amazing what that Leica badge on the 25/1.4 can do, isn't it?

1.7 870/735

2.8 1050/875

4.0 1075/880

Leica Summilux 50/1.4 on an M9 (18 MP, no AA filter)

1.4 600/530

2.0 950/740

2.8 1025/860

4.0 1110/980

Sources:

http://www.lensrentals.com/blog/2012/05/wide-angle-micro-43-imatest-results

http://www.lensrentals.com/blog/2012/01/the-great-50mm-shootout

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Great Bustard
Forum ProPosts: 24,127
Like?
Re: This is a tough one...
In reply to Prairie Pal, Mar 29, 2013

Prairie Pal wrote:

So, does that at least help in answering your question, or just confuse the hell out of you?

I will re-read your post and links a couple of times before I can commit an answer to your question.  .

You mean it was not obvious on the first read? 

That's a lot of information, and some of it touches on other related questions I had (such as how much effect different RAW converters have and if the manufacturers converter is always the best one to use, "bar_none" so to speak).  But I can leave that for another post.

Sure.

But anyways, who are you and why do you know so much about this topic?

I'm Batman. 

Does it apply to your profession?

The other way around.

And how did it come to pass that you are so much smarter than this Anders guy   Sorry, just being spitefull, as I don't really remember seeing your guys (or girls?) bump heads.

I must have worded something poorly, 'cause I have nothing but the highest respect for Anders, and we get along rather well, although we will disagree from time to time.

Reply   Reply with quote   Complain
Great Bustard
Forum ProPosts: 24,127
Like?
Ideally...
In reply to Anders W, Mar 29, 2013

Anders W wrote:

Great Bustard wrote:

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

...and Anders will hopefully chime in so you can watch us argue about it. 

First of all, what PZ measures is the MTF-50, which is low many line widths per picture height (lw/ph) are resolved at 50% contrast.

Now, Klaus clearly states:

Please note that the tests results are not comparable across the different systems!

but he does not go into detail about why, so let's discuss that point for the rest of this post.

First of all, the resolution in the final photo, which is what PZ measures, is a function of many different factors:

  • Lens sharpness
  • Pixel count
  • Sensor size
  • AA filter
  • RAW conversion
  • Additional processing

In other words, you aren't looking at lens tests, you are looking at system tests (this is true for all other testing sites as well).

I discuss the relationship between lens sharpness and sensor size here, and the relationship between pixel count and resolution here, so I'll move on to other factors in this post.

As we all know, a weaker AA filter will result in what appears to be more detailed files, but much of this additional "detail" may well be false detail.  Depending on the processing, this false detail can look very pleasing and also result in higher MTF-50 resolution scores.  Indeed, it's not unlike the idea behind upsampling a photo for more "resolution".  That is, you cannot get more resolution than the initial capture, but with good enough software, you can make enough "lucky guesses" that the distinction between real and made up is inconsequential.  Of course, there are limits, however.

In addition, we all know that different demosaicking algorithms can have a marked impact on the look of a photo, and that includes resolution.  Thus, using different RAW converters for different files, or even the same RAW converter that treats the files from different systems differently, and not necessarily differently in an optimum way for each system, can cause further problems still.

Lastly, and this is where Anders and I bump heads, is on the matter of sharpening.  We normally don't think of noise at base ISO in good light to be an issue.  However, the noise differential exists even if it's not visible.

I think we actually agree on most of the facts involved but differ in the conclusions we draw from them. I think that as far as tests are concerned, it is useful to keep MTF performance apart from noise performance as much as possible. There are multiple reasons for this.

First, the way Photozone does it means that we mix the two in unknown ways. There is no way for me as a potential user of Photozone data to know the mix they have chosen for a particular camera/sensor and how that mix compares to others. This means that the results lack transparency.

Second, as I am sure you agree, no mix is appropriate for all conditions. As you point out below, there are cases where a D800 beats an E-M5 for noise but also situations where it is the other way around.

Third, it makes a difference to me whether a certain MTF result has been achieved with or without sharpening. Sharpening in PP is to a considerable extent a way to fool the eye and based on the visual evidence I have looked at, I'll take the unsharpened output from a good lens over the sharpened output from an inferior one any day, even though they might both reach the same MTF values at the same noise level.

My preferred solution to these problems is to present the results of MTF tests in such a way that an informed reader can draw his/her own conclusions with all relevant information on the table. At the present time, the two sources of information that I find preferable in this regard are Lenstip and LensRentals. Both present results based on RAWs converted by the same known algoritms without any additional sharpening. The information from LensRentals is presented in a form that simplifies cross-format comparisons since it is reported as line pairs per image height (lp/ih). The results from Lenstip are reported as line pairs per mm (lp/mm) and therefore have to be converted before they can be compared. But this is not difficult to do. To compare between FF and MFT, for example, you merely have to multiply the FF results by two or, alternatively, divide the MFT results by two.

You might ask, if it's not visible, then what does it matter?  Well, the reason it matters is that the lower noise photo can take more sharpening.  Thus, by comparing photos from, say, a D800 and EM5, with no sharpening applied, well, that artificially handicaps the D800 file which has lower noise and can take more sharpening than the EM5 file (for a given exposure).

Of course, then we have equivalent photos which are photos at the same DOF and shutter speed.  For these types of photos, no format is favored over the other -- the noise differential is due to differences in sensor efficiency, and that difference can vary throughout the ISO range.  For example, the D800 may be a stop more efficient than the EM5 sensor at base ISO, but a stop less efficient at ISO 3200 (just throwing numbers out there -- I haven't actually worked that particular comparison out).

So, in the end, what we have to ask is how much variation all these factors cause.  The answer is:  I don't have a freakin' clue.  But, to compound the issue further still, there's the issue of copy variation among lenses, for example, this article, or this article. or this article.

Lastly, we have to ask how the MTF-50 lw/ph measurements correspond to the visual properties of the final photo.  Can you notice a 50% difference?  One would think so, but at what display size would you not notice?  How about 40%?  30%?  20%?  10%?

Fact of the matter is that anyone telling you f/11 is a perfectly good aperture to shoot at is already losing massive amounts of resolution due to diffraction softening (the sharper the lens, or the greater the pixel count, the more resolution they are losing, although they'll always be left with more resolution than if they started off with a softer lens or a sensor with fewer pixels), so MTF-50 scores, and thus resolution, probably aren't much of a concern to them at all, as they have more than enough, anyway.

So, does that at least help in answering your question, or just confuse the hell out of you? 

...what we want is not numbers at all, but photos of various scenes with various systems (body + lens), converted with various RAW converters as well as OOC jpgs, and taken at various light levels.  Then, as photographers, we could just look at the resulting photos, and decide from there.

However, we all know this is impractical to do times a thousand.  So, what to do?

Well, it would be nice if we knew what effect various RAW converters had on the MTF-50 scores.  If all RAW converters resulted in MTF-50s within 10%, I'd say it's not a big deal, and just go with the converter than best matches all systems as a whole.  If not, then use the RAW converter that gives the "best" result for each system, but now we have to define "best", as "best" doesn't only mean detail.

As for sharpening and noise, well, I don't think we're ever going to convince each other.  As you know, my emphasis is always the visual properties of the final photo, and hence, sharpening and noise always go hand-in-hand.

However, here's what would be awesome:  take the make a bench that mounts the lens in front of the highest pixel density sensor out there.  Take some photos from various portions of the image circle, and use the same RAW converter with the exact same settings each time.  Now you're really testing the lens, and not the system.

Of course, this now opens a huge can o' worms, since the lenses for smaller sensor systems will, on average, resolve better than lenses for larger sensor systems, and we'll have the ignorant masses chanting "lw/ph = lw/ph" to go along with "f/2 = f/2", and any attempts to educate will be met with "entertainment".

But, now that I think about it, I'm good with that. 

Reply   Reply with quote   Complain
Dr_Jon
Senior MemberPosts: 1,494Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Prairie Pal, Mar 29, 2013

As I said previously:

(1) I think the Photozone results are comparable provided you want to compare the system performance and it includes a camera that is generally equivalent to the one he's testing with. My at-home results certainly agree with that. Note he does say more about it in his FAQ, which I guess I'm the only one that read?

(2) I don't think the Lens Rentals results are any use in this context as I believe they are usually just comparing the lenses and not the whole system (it would be silly otherwise, as everything on a D800 would be artifically inflated for non-D800 users). Exceptions are when he is doing a system comparison (e.g. the 24-70 lenses on D800 vs 5DmkIII).

I find my Canon lenses are better than my m43 ones for two reasons:

(i) The latest Canon 24-70 and 70-200 f2.8s (I only have the latter) are great wide-open. The Panasonic equivalents (err... let's not go anywhere near equivalence, note I have both) aren't great wide-open and f4 is a better bet (not that they're bad at f2.8, just down quite a bit). Note you pay a chunk more for the Canons. See Lenstip for Panny reviews which back me up here (f2.8 vs. f4 that is):

http://www.lenstip.com/367.4-Lens_review-Panasonic_G_X_VARIO_12-35_mm_f_2.8_ASPH._P.O.I.S_Image_resolution.html

http://www.lenstip.com/368.4-Lens_review-Panasonic_G_X_VARIO_35-100_mm_f_2.8_P.O.I.S._Image_resolution.html

(ii) When I was testing my 40/2.8 Canon lens (the first one I got was scarily decentred) I'd just received my 12-35 and I shot that too and was so shocked by how much better the Canon sharpness was I went back to the shop and compared my 12-35 with another one to check it wasn't a dud (it behaved the same). I feel this supports my conclusion about the Lens Rentals results (as Anders pointed out to me at the time they have the 12-35 as sharper and trust me it really isn't).

But do note there is more to lenses than sharpness. The cheapo lenses from Canon and Nikon are quite sharp these days. But you want more from a lens. For example I really like the out-of-focus rendering of the 35-100 a lot (plus the weight... of course, my Canon is 1490g vs. 360g).

John

(Edit) P.S. these are opinions, with some facts to back them up. The shakiest bit is the Lens Rentals conclusions but I think I have enough circumstantial evidence to say it seems that way to me.

 Dr_Jon's gear list:Dr_Jon's gear list
Fujifilm FinePix Real 3D W3 Sony RX100 Canon EOS 5D Mark II Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GH3 +24 more
Reply   Reply with quote   Complain
KenBalbari
Regular MemberPosts: 275Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Prairie Pal, Mar 29, 2013

First, LW/PH is "line widths per picture height".  This is a measure of the resolution of the final image.  Full frame will almost always have higher possible resolution here because it is a larger sensor.  A 24mm high sensor will typically resolve more line widths than a 13mm high sensor.

If you want to compare glass only, and ignore the sensor or the final image resolution, then you can use LP/MM, which is "line pairs per millimeter".  One catch here, if you want to convert between the two, you have to realize a line pair equals two line widths.  So a lens that averages 60 lp/mm resolution would be 120 lw/mm,  and could in theory produce 1560 lw/ph on m4/3 or 2880 lw/ph on full frame.  In practice though, this is really going to depend also on the characteristics of the actual sensor, including pixel density, noise characteristics, and the effect of the anti aliasing filter.

Keep in mind, how you measure this also depends on what level of resolution you deem acceptable for an individual line pair.  This can also differ between different test sources and methods.  MTF50 is one commonly used standard which tries to define this, but something like MTF10 might be closer to measuring what most people actually think of as "sharpness".

 KenBalbari's gear list:KenBalbari's gear list
Fujifilm FinePix F30 Zoom Olympus E-330 Olympus PEN E-PL2
Reply   Reply with quote   Complain
Dr_Jon
Senior MemberPosts: 1,494Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to KenBalbari, Mar 29, 2013

I'd disagree, lp/mm is not useful as a small sensor will win over a big one (it has smaller pixels to resolve detail and a smaller lens where it's easier to get more resolution per mm, essential even).

What you see in the final picture is the whole image. Let's say I have a phone which manages more lp/mm than a Canon 1Dx. Let's shoot a pic of someone's face which is 1/2 the picture height in both cases (same framing). Let's hypothesize that the phone gets 80 lp/mm over its 3mm high sensor and the 1Dx with a £2000 L series lens 50 lp/mm over its 24mm one. On your argument the phone is better (80 vs. 50).

However that means the Canon has 600 lp of resolution over the face's height and the phone 120 lp. I'm thinking the Canon will look a lot sharper.

However if using lp/ph you will see the Canon (1200) is sharper than the phone (240) and that will agree with what you see in the pictures, in direct proportion.

 Dr_Jon's gear list:Dr_Jon's gear list
Fujifilm FinePix Real 3D W3 Sony RX100 Canon EOS 5D Mark II Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GH3 +24 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Ideally...
In reply to Great Bustard, Mar 29, 2013

Great Bustard wrote:

Anders W wrote:

Great Bustard wrote:

Prairie Pal wrote:

I'm at photozone review website browsing through lenses for both FX and MFT.  I see that the highest resolving FX lenses can be near the 4000 lw/ph range.  The better MFT lenses rarely even get to 2300.  Is the difference between the 2 types of lenses relative?  Surely even the best MFT glass isn't inferior to FX.  I admit I can't technically explain what lw/ph is measureing, but to a certain extent I look at those graphs as a means of measuring one lens against another ie, when comparing the Sigma 35 1.4 to Nikon 35, or the various MFT primes between each other.

...and Anders will hopefully chime in so you can watch us argue about it. 

First of all, what PZ measures is the MTF-50, which is low many line widths per picture height (lw/ph) are resolved at 50% contrast.

Now, Klaus clearly states:

Please note that the tests results are not comparable across the different systems!

but he does not go into detail about why, so let's discuss that point for the rest of this post.

First of all, the resolution in the final photo, which is what PZ measures, is a function of many different factors:

  • Lens sharpness
  • Pixel count
  • Sensor size
  • AA filter
  • RAW conversion
  • Additional processing

In other words, you aren't looking at lens tests, you are looking at system tests (this is true for all other testing sites as well).

I discuss the relationship between lens sharpness and sensor size here, and the relationship between pixel count and resolution here, so I'll move on to other factors in this post.

As we all know, a weaker AA filter will result in what appears to be more detailed files, but much of this additional "detail" may well be false detail.  Depending on the processing, this false detail can look very pleasing and also result in higher MTF-50 resolution scores.  Indeed, it's not unlike the idea behind upsampling a photo for more "resolution".  That is, you cannot get more resolution than the initial capture, but with good enough software, you can make enough "lucky guesses" that the distinction between real and made up is inconsequential.  Of course, there are limits, however.

In addition, we all know that different demosaicking algorithms can have a marked impact on the look of a photo, and that includes resolution.  Thus, using different RAW converters for different files, or even the same RAW converter that treats the files from different systems differently, and not necessarily differently in an optimum way for each system, can cause further problems still.

Lastly, and this is where Anders and I bump heads, is on the matter of sharpening.  We normally don't think of noise at base ISO in good light to be an issue.  However, the noise differential exists even if it's not visible.

I think we actually agree on most of the facts involved but differ in the conclusions we draw from them. I think that as far as tests are concerned, it is useful to keep MTF performance apart from noise performance as much as possible. There are multiple reasons for this.

First, the way Photozone does it means that we mix the two in unknown ways. There is no way for me as a potential user of Photozone data to know the mix they have chosen for a particular camera/sensor and how that mix compares to others. This means that the results lack transparency.

Second, as I am sure you agree, no mix is appropriate for all conditions. As you point out below, there are cases where a D800 beats an E-M5 for noise but also situations where it is the other way around.

Third, it makes a difference to me whether a certain MTF result has been achieved with or without sharpening. Sharpening in PP is to a considerable extent a way to fool the eye and based on the visual evidence I have looked at, I'll take the unsharpened output from a good lens over the sharpened output from an inferior one any day, even though they might both reach the same MTF values at the same noise level.

My preferred solution to these problems is to present the results of MTF tests in such a way that an informed reader can draw his/her own conclusions with all relevant information on the table. At the present time, the two sources of information that I find preferable in this regard are Lenstip and LensRentals. Both present results based on RAWs converted by the same known algoritms without any additional sharpening. The information from LensRentals is presented in a form that simplifies cross-format comparisons since it is reported as line pairs per image height (lp/ih). The results from Lenstip are reported as line pairs per mm (lp/mm) and therefore have to be converted before they can be compared. But this is not difficult to do. To compare between FF and MFT, for example, you merely have to multiply the FF results by two or, alternatively, divide the MFT results by two.

You might ask, if it's not visible, then what does it matter?  Well, the reason it matters is that the lower noise photo can take more sharpening.  Thus, by comparing photos from, say, a D800 and EM5, with no sharpening applied, well, that artificially handicaps the D800 file which has lower noise and can take more sharpening than the EM5 file (for a given exposure).

Of course, then we have equivalent photos which are photos at the same DOF and shutter speed.  For these types of photos, no format is favored over the other -- the noise differential is due to differences in sensor efficiency, and that difference can vary throughout the ISO range.  For example, the D800 may be a stop more efficient than the EM5 sensor at base ISO, but a stop less efficient at ISO 3200 (just throwing numbers out there -- I haven't actually worked that particular comparison out).

So, in the end, what we have to ask is how much variation all these factors cause.  The answer is:  I don't have a freakin' clue.  But, to compound the issue further still, there's the issue of copy variation among lenses, for example, this article, or this article. or this article.

Lastly, we have to ask how the MTF-50 lw/ph measurements correspond to the visual properties of the final photo.  Can you notice a 50% difference?  One would think so, but at what display size would you not notice?  How about 40%?  30%?  20%?  10%?

Fact of the matter is that anyone telling you f/11 is a perfectly good aperture to shoot at is already losing massive amounts of resolution due to diffraction softening (the sharper the lens, or the greater the pixel count, the more resolution they are losing, although they'll always be left with more resolution than if they started off with a softer lens or a sensor with fewer pixels), so MTF-50 scores, and thus resolution, probably aren't much of a concern to them at all, as they have more than enough, anyway.

So, does that at least help in answering your question, or just confuse the hell out of you? 

...what we want is not numbers at all, but photos of various scenes with various systems (body + lens), converted with various RAW converters as well as OOC jpgs, and taken at various light levels.  Then, as photographers, we could just look at the resulting photos, and decide from there.

However, we all know this is impractical to do times a thousand.  So, what to do?

I certainly would like the above. But I'd like the numbers too for the simple reason than I can digest a set of numbers far more quickly than I can digest a set of comparisons between images with regard to IQ of the kind at issue here. The figures also help in settling various disputes about subjective impressions, within the obvious restriction that the numbers always imply a particular point of view.

Well, it would be nice if we knew what effect various RAW converters had on the MTF-50 scores.  If all RAW converters resulted in MTF-50s within 10%, I'd say it's not a big deal, and just go with the converter than best matches all systems as a whole.  If not, then use the RAW converter that gives the "best" result for each system, but now we have to define "best", as "best" doesn't only mean detail.

Yes, that would be nice but doesn't seem to be the case. Of course, one might also bypass conversion entirely and just go directly to the raw data. Can't see why it wouldn't be possible to measure MTF on the basis of, say, the green channel alone, without demosaicing.

As for sharpening and noise, well, I don't think we're ever going to convince each other.  As you know, my emphasis is always the visual properties of the final photo, and hence, sharpening and noise always go hand-in-hand.

I don't think we disagree about the visual properties of the final photo being what we are ultimately interested in. However, we might possibly still disagree about how we can, with the imperfect data at our disposal, form as good an impression of that as possible. I prefer data that I know are in a specific sense comparable and would then add and detract whatever it takes to get a good idea of the final verdict. I don't like data where someone else, by adding a black box, has tried to take us closer to that goal (but probably failed in a number of ways that are very difficult to sort out).

However, here's what would be awesome:  take the make a bench that mounts the lens in front of the highest pixel density sensor out there.  Take some photos from various portions of the image circle, and use the same RAW converter with the exact same settings each time.  Now you're really testing the lens, and not the system.

Yes. I have thought of it. If all lenses were manual, it would probably have been done already. But you can't even focus or stop down an MFT AF lens without the right body electronics. And if it's an OIS lens, the OIS lens group could be anywhere without power.

Of course, this now opens a huge can o' worms, since the lenses for smaller sensor systems will, on average, resolve better than lenses for larger sensor systems, and we'll have the ignorant masses chanting "lw/ph = lw/ph" to go along with "f/2 = f/2", and any attempts to educate will be met with "entertainment".

But, now that I think about it, I'm good with that. 

Not with you and me around to do the education.

As you know, I am open to the idea that it is fair to compare an MFT X mm lens at f/2 with an FF 2X mm lens at f/4, since that's the point where they are equivalent in terms of DoF, diffraction and light accumulation (and thus photon noise so that there would no longer be any difference with regard to the possibility to sharpen in PP). And I much prefer such comparisons to those we can make on the basis of Photozone results.

Still, I had better add that such comparisons are no panacea either. One reason is that we ignore the complicated read-noise issue when we just go by the general two-stop difference rule. Another is that it is not reasonable to expect an MFT X mm f/2 lens to do just as well wide open as an FF 2X mm f/4 lens or to expect the two to have similar price tags.

There are physical reasons why the MFT lens faces a tougher job in such a comparison, and we should take that fact into account when we look at how well they accomplish. Naturally, we should also take it into account when choosing between systems. If you chose to go with MFT without realizing that the MFT X mm f/2 lens would probably cost more and perform worse wide open than the corresponding FF 2X mm f/4 lens (if it were available ), it simply means you haven't done your homework properly.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Dr_Jon, Mar 29, 2013

Dr_Jon wrote:

As I said previously:

(1) I think the Photozone results are comparable provided you want to compare the system performance and it includes a camera that is generally equivalent to the one he's testing with. My at-home results certainly agree with that. Note he does say more about it in his FAQ, which I guess I'm the only one that read?

It follows from what is said in the FAQ that they are not comparable because the methods are not held constant across systems.

(2) I don't think the Lens Rentals results are any use in this context as I believe they are usually just comparing the lenses and not the whole system (it would be silly otherwise, as everything on a D800 would be artifically inflated for non-D800 users). Exceptions are when he is doing a system comparison (e.g. the 24-70 lenses on D800 vs 5DmkIII).

LensRentals certainly compare the entire systems (lens + sensor), the difference being that they use the same method irrespective of the system.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to KenBalbari, Mar 29, 2013

KenBalbari wrote:

First, LW/PH is "line widths per picture height".  This is a measure of the resolution of the final image.  Full frame will almost always have higher possible resolution here because it is a larger sensor.  A 24mm high sensor will typically resolve more line widths than a 13mm high sensor.

If the pixel count of the sensors is the same, and the impact of the AA filter comparable, they have the same resolution irrespective of sensor size.

If you want to compare glass only, and ignore the sensor or the final image resolution, then you can use LP/MM, which is "line pairs per millimeter".  One catch here, if you want to convert between the two, you have to realize a line pair equals two line widths.  So a lens that averages 60 lp/mm resolution would be 120 lw/mm,  and could in theory produce 1560 lw/ph on m4/3 or 2880 lw/ph on full frame.  In practice though, this is really going to depend also on the characteristics of the actual sensor, including pixel density, noise characteristics, and the effect of the anti aliasing filter.

Keep in mind, how you measure this also depends on what level of resolution you deem acceptable for an individual line pair.  This can also differ between different test sources and methods.  MTF50 is one commonly used standard which tries to define this, but something like MTF10 might be closer to measuring what most people actually think of as "sharpness".

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Dr_Jon
Senior MemberPosts: 1,494Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Anders W, Mar 29, 2013

I think we may have to agree to disagree, although as my testing supports my conclusions by a big margin I'm happy with what I'm saying. BTW Roger (of Lens Rentals, who has an excellent blog) has changed from a m43 to a FF Canon (the 6D, beat out all the Nikons) recently.

 Dr_Jon's gear list:Dr_Jon's gear list
Fujifilm FinePix Real 3D W3 Sony RX100 Canon EOS 5D Mark II Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GH3 +24 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 18,348Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Dr_Jon, Mar 29, 2013

Dr_Jon wrote:

I think we may have to agree to disagree, although as my testing supports my conclusions by a big margin I'm happy with what I'm saying.

Honestly, I don't care at all about your subjective impressions about this or that. I prefer to discuss on the basis of publicly available evidence.

BTW Roger (of Lens Rentals, who has an excellent blog) has changed from a m43 to a FF Canon (the 6D, beat out all the Nikons) recently.

So what?

I think his numbers are MTF50 x 1000, so if you compare them you artificially inflate the m43 ones (smaller lower-resolution sensor).

Could you please spell out what you mean by "MTF50 x 1000" and what the source of your thinking so is?

The figures reported by LensRentals are line pairs per image height (lp/ih) at a contrast criterion of 50 percent (MTF-50). It follows that the results of smaller sensors are not inflated relative to bigger ones. It's when the results are reported per mm rather than sensor height that you have to take sensor size into account if you want to compare across systems.

You could have a phone camera with a lens that had a MTF50 of 1 from centre to edge of its 5MP 1/3.2" sensor and it wouldn't look sharper than any m43, APS or FF camera with a good lens but would score 1000 on Lens Rentals and beat them all.

You are obviously wrong about that.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
KenBalbari
Regular MemberPosts: 275Gear list
Like?
Re: Reading resolution charts comparing MFT lenses to FX lenses
In reply to Dr_Jon, Mar 29, 2013

Dr_Jon wrote:

I'd disagree, lp/mm is not useful as a small sensor will win over a big one (it has smaller pixels to resolve detail and a smaller lens where it's easier to get more resolution per mm, essential even).

Which is why I said it was used to compare glass only, not sensors.

However that means the Canon has 600 lp of resolution over the face's height and the phone 120 lp. I'm thinking the Canon will look a lot sharper.

Sure, the Canon sensor is sharper.  But how do you propose to get that sensor into the phone then?  Otherwise, I'm not sure it's relevant.

As far as I know, no one currently makes an interchangeable sensor camera.  Though I'd like to see someone try it.  For now though, they do make interchangeable lens cameras, which is why lp/mm, comparing only the glass itself,  is a useful and practical measure.

 KenBalbari's gear list:KenBalbari's gear list
Fujifilm FinePix F30 Zoom Olympus E-330 Olympus PEN E-PL2
Reply   Reply with quote   Complain
Great Bustard
Forum ProPosts: 24,127
Like?
Re: Ideally...
In reply to Anders W, Mar 29, 2013

Anders W wrote:

Great Bustard wrote:

However, here's what would be awesome:  take the make a bench that mounts the lens in front of the highest pixel density sensor out there.  Take some photos from various portions of the image circle, and use the same RAW converter with the exact same settings each time.  Now you're really testing the lens, and not the system.

Yes. I have thought of it. If all lenses were manual, it would probably have been done already. But you can't even focus or stop down an MFT AF lens without the right body electronics. And if it's an OIS lens, the OIS lens group could be anywhere without power.

I did not know that!

Of course, this now opens a huge can o' worms, since the lenses for smaller sensor systems will, on average, resolve better than lenses for larger sensor systems, and we'll have the ignorant masses chanting "lw/ph = lw/ph" to go along with "f/2 = f/2", and any attempts to educate will be met with "entertainment".

But, now that I think about it, I'm good with that. 

Not with you and me around to do the education.

Maybe not with you, but you know that "entertainment" always finds me and I always find "entertainment".

As you know, I am open to the idea that it is fair to compare an MFT X mm lens at f/2 with an FF 2X mm lens at f/4, since that's the point where they are equivalent in terms of DoF, diffraction and light accumulation (and thus photon noise so that there would no longer be any difference with regard to the possibility to sharpen in PP). And I much prefer such comparisons to those we can make on the basis of Photozone results.

Except that often the larger sensor system is simply using a longer shutter speed for the same DOF.

Still, I had better add that such comparisons are no panacea either. One reason is that we ignore the complicated read-noise issue when we just go by the general two-stop difference rule. Another is that it is not reasonable to expect an MFT X mm f/2 lens to do just as well wide open as an FF 2X mm f/4 lens or to expect the two to have similar price tags.

Indeed.

There are physical reasons why the MFT lens faces a tougher job in such a comparison, and we should take that fact into account when we look at how well they accomplish. Naturally, we should also take it into account when choosing between systems. If you chose to go with MFT without realizing that the MFT X mm f/2 lens would probably cost more and perform worse wide open than the corresponding FF 2X mm f/4 lens (if it were available ), it simply means you haven't done your homework properly.

To be perfectly honest, I think we are well, well, well past the point where differences in resolution matter for the vast majority.  That is, if you took a pic of a scene with the kit lens and the 12-35 / 2.8, printed them even at 16x24 inches, put them both for sale, they'd sell for the same price and in the same amount.  Well, maybe you'll sell 53 of the pics shot with the 12-35 and 47 pics shot with the kit lens, but you get what I'm saying.

I'm just coming to the opinion that any IQ differential between lenses and systems is simply unimportant for the vast majority in the vast majority of situations, and it's only for the extremes that it matters.  And by "extremes", I mean the very extremes.

Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads