DXOMARK.com, did you notice

Started 11 months ago | Discussions
Joel Stern
Forum ProPosts: 10,530Gear list
Like?
DXOMARK.com, did you notice
11 months ago

Not one m43 lens gets a good rating, no matter what body it was tested with.  What's with that?

-- hide signature --

If i am typing on my iPad, please excuse any typos.

MAubrey
Senior MemberPosts: 1,536Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Joel Stern, 11 months ago

If you read their methodology, you'd already know.

They upsize their test images from μ43 sensors to the size of a FF sensor. That essentially cuts all the scores in half, by definition.

-- hide signature --

--Mike

 MAubrey's gear list:MAubrey's gear list
Olympus OM-D E-M5 Sony Alpha 7 Olympus M.Zuiko Digital ED 9-18mm 1:4.0-5.6 Voigtlander 58mm F1.4 Nokton SL II Voigtlander 35mm F1.2 Nokton +7 more
Reply   Reply with quote   Complain
Joel Stern
Forum ProPosts: 10,530Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to MAubrey, 11 months ago

MAubrey wrote:

If you read their methodology, you'd already know.

They upsize their test images from μ43 sensors to the size of a FF sensor. That essentially cuts all the scores in half, by definition.

-- hide signature --

--Mike

Thanks Mike, I really did not see that and it makes much more sense.

I went back and still could not find any wording to indicate what you say but I do not doubt you, here is what I found that got my attention:

  • DxOMark Score shows:
    • The amount of information captured by the lens for a given camera.
    • How well the lens and camera perform together.
  • DxOMark Score does not show:
    • The intrinsic quality of the camera sensor.
    • The camera sensor's performance under high-light condition

So if they are showing how well a lens works with a given camera the scores still look terribly low to me. I know I am missing something here and your explanation makes it better but I cannot find it on their site.. They make it kind of hard to find but I will look again...

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Reply   Reply with quote   Complain
Joel Stern
Forum ProPosts: 10,530Gear list
Like?
When I read this from DXO I would think m43 lenses would have a high score
In reply to Joel Stern, 11 months ago

From DxOMark:

Since DxOMark measurements are performed on RAW images, an imaging sensor is required to perform measurements.

While this statement is obvious, when talking about optical measurements, it is very common to refer to lens performance alone.

However, sensors significantly influence optical performance, so it makes a lot of sense to evaluate and compare lens-camera combinations, rather than just lenses alone.

Lenses can often be used on different sizes of sensors—for instance, most APS-C or APS-H cameras can mount lenses originally designed for 24x36mm cameras. In this case, the generated picture is a “crop” of the picture that a camera with a larger sensor would generate.

Obviously, if optical performance decreases from center to corners on a 24x36mm camera, the optical performance on a smaller sensor should be better, and this is true for distortion, chromatic aberration, vignetting and resolution.

In the case of resolution, since results are normalized (stretched in this instance) to a 24x36mm output, the resolution at the center of the field is lower, but is more homogeneous overall.

A digital imaging sensor is composed of many photosites, or pixels. The smaller the pixel size, the greater the sensor resolution. Resolution is highly influenced by both pixel size and sensor dimension.

Obviously both the sensor and the optics have an influence on resolution, and either one of them can be a limiting factor.

Each camera sensor model is designed carefully to maximize performance. The sensitive area of a pixel is an important factor that is often increased by using microlenses to focus more light on the sensitive area. This also has some influence on resolution.

Microlenses focus light on sensitive areas.

Since microlenses are maximally effective only for certain lens apertures, microlens performance decreases as the difference from the ideal aperture increases, meaning that the full T-stop range of the camera is far from optimal.

All of these sensor-related factors have an impact on lens performance and cannot be ignored. The very same lens can show differences in image quality performance when coupled with different camera bodies, which is why DxOMark puts in the time and effort to test and publish the results for as many lens-camera combinations as possible.

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Reply   Reply with quote   Complain
Joseph T Lewis III
Senior MemberPosts: 1,696Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Joel Stern, 11 months ago

Joel Stern wrote:

Not one m43 lens gets a good rating, no matter what body it was tested with. What's with that?

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Are you saying that by comparison, all lenses other than m4/3 score much higher? (Honest question, not trying to be a smarta--.) Reason I ask is that I just played around with DxOMark and it didn't look like m4/3 lenses scored poorly at all.  See below:

Random DxOMark Lens Comparisons

Another, with Carl Zeiss Lens Added

I admit that I did not go out and look at every single lens in their entire database, but the ones I spot checked didn't seem to score significantly higher than m4/3rds.  In fact, several were lower (some significantly so).

-- hide signature --

Tom

 Joseph T Lewis III's gear list:Joseph T Lewis III's gear list
Olympus XZ-1 Panasonic Lumix DMC-LX7 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Olympus E-M1 Olympus M.Zuiko Digital ED 9-18mm 1:4.0-5.6 +6 more
Reply   Reply with quote   Complain
Joel Stern
Forum ProPosts: 10,530Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Joseph T Lewis III, 11 months ago

Joseph T Lewis III wrote:

Joel Stern wrote:

Not one m43 lens gets a good rating, no matter what body it was tested with. What's with that?

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Are you saying that by comparison, all lenses other than m4/3 score much higher? (Honest question, not trying to be a smarta--.) Reason I ask is that I just played around with DxOMark and it didn't look like m4/3 lenses scored poorly at all. See below:

Random DxOMark Lens Comparisons

Another, with Carl Zeiss Lens Added

I admit that I did not go out and look at every single lens in their entire database, but the ones I spot checked didn't seem to score significantly higher than m4/3rds. In fact, several were lower (some significantly so).

-- hide signature --

Tom

Tom, I looked at a lot of the lens scores, Nikon and the top Canon lenses had really high scores and all others pretty low. If we look at a scale and a lens only gets half way or 1/4 way from the bottom to me that is a poor score. I would hope a lens and camera combination would score at least at the 66% and above level, but that was not what I saw. I am just scratching my head, and of course they do not rate any Fuji X sensor cameras so no lenses, that would have been interesting. Some of the lenses for Sony Nex cameras did well....that was a surprise but I do not keep up with Sony.

Here I selected a Panasonic body and 3 very popular m43 primes, take a look at the scores.

http://www.dxomark.com/Lenses/Compare/Side-by-side/Panasonic-LUMIX-G-20mm-F17-II-ASPH-versus-Sigma-60mm-F28-DN-A-Mount-43-versus-Olympus-MZuiko-Digital-ED-45mm-F18___1195_0_1103_0_532_0

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Reply   Reply with quote   Complain
Joseph T Lewis III
Senior MemberPosts: 1,696Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Joel Stern, 11 months ago

Here's a comprehensive list of lenses / ratings I found, that may be of interest.

DxOMark Lens Ratings

-- hide signature --

Tom

 Joseph T Lewis III's gear list:Joseph T Lewis III's gear list
Olympus XZ-1 Panasonic Lumix DMC-LX7 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Olympus E-M1 Olympus M.Zuiko Digital ED 9-18mm 1:4.0-5.6 +6 more
Reply   Reply with quote   Complain
Joel Stern
Forum ProPosts: 10,530Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Joseph T Lewis III, 11 months ago

Joseph T Lewis III wrote:

Here's a comprehensive list of lenses / ratings I found, that may be of interest.

DxOMark Lens Ratings

-- hide signature --

Tom

I saw that Tom, did you notice that for every lens they say the ideal shooting is wide open. I think maybe I am missing something here, and yes, I did see the listing for all of those lenses, they never get more than 1/2 way up the scale and some not even close to that. I hope the explanation above given by Mike is the right one but somehow it does not make sense to me and I could not find it on the site.

-- hide signature --

If i am typing on my iPad, please excuse any typos.

Reply   Reply with quote   Complain
DonSC
Contributing MemberPosts: 987
Like?
DXO scores are like MTF charts
In reply to Joel Stern, 11 months ago

in that they're mostly useful for comparing similar things. For MTF charts you can't use the charts across companies. For DXO scores you have to use the same size sensor.

For example, with sensors are of different sizes, where do you stand? To give equivalent FOV you need to stand further away with the smaller sensor. But that's a disadvantage. If you stand at the same spot with different FOV then the larger sensor camera is at a disadvantage. There simply isn't any perfect way to make the comparison.

I think they try their best. It's just not that easy.

Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 19,062Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to MAubrey, 11 months ago

MAubrey wrote:

If you read their methodology, you'd already know.

They upsize their test images from μ43 sensors to the size of a FF sensor. That essentially cuts all the scores in half, by definition.

But that's exactly what they should do, in my view. Keeping MFT images at one quarter the area of FF images implies that we expect to view MFT images at only half the magnification of FF images (whether in print or on screen). I certainly don't expect that. I expect to view my MFT images at the same display size as those from an FF camera.

Furthermore, MFT lenses don't have much trouble meeting this requirement, i.e., that of performing about twice as well as an FF lens in terms of lp/mm on the sensor so that the image can be enlarged about twice as much and still show about the same resolution.

Instead, I'd say that the primary explanations is that DxO's overall score factors in the capability of the lens to gather light and an f/2.8 FF lens gathers four times more light than an f/2.8 MFT lens (since the area of its image circle is four times larger). In a sense, DxO double-counts this aspect since their sensor scores are based on the same idea. So they penalize MFT two times around, first in their sensor scores and then in their lens scores. However, you can bypass that complication by just ignoring their scoring and look at the measurements, as I do.

Another factor affecting the comparison between MFT lenses and lenses for larger sensors is that MFT sensors currently do not have more than 16 MP whereas APS-C goes to 24 and FF to 36. What DxO measures is image resolution, and this depends not only on lens resolution but also on sensor resolution.

I am not saying that these are the only explanations but they are some of the more important ones.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
agentul
Contributing MemberPosts: 606Gear list
Like?
Re: DXO scores are like MTF charts
In reply to DonSC, 11 months ago

DonSC wrote:

There simply isn't any perfect way to make the comparison.

I think they try their best. It's just not that easy.

the best way is to look at some actual photos (real life and test scenes). in the end, that is what you are buying the camera to do. DxO and MTF charts are good, but none tell the whole story. nor do test (real life or studio) shots on review sites, for that matter.

 agentul's gear list:agentul's gear list
Panasonic Lumix DMC-GH2 Panasonic Lumix G Vario 14-42mm F3.5-5.6 ASPH OIS Panasonic Leica Summilux DG 25mm F1.4 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS Panasonic Lumix DMC-LZ2 +1 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 19,062Gear list
Like?
Re: DXO scores are like MTF charts
In reply to DonSC, 11 months ago

DonSC wrote:

in that they're mostly useful for comparing similar things. For MTF charts you can't use the charts across companies. For DXO scores you have to use the same size sensor.

The DxO scores are meant (by DxO) to be comparable across sensor sizes. Whether they actually accomplish that is another matter.

For example, with sensors are of different sizes, where do you stand? To give equivalent FOV you need to stand further away with the smaller sensor. But that's a disadvantage. If you stand at the same spot with different FOV then the larger sensor camera is at a disadvantage. There simply isn't any perfect way to make the comparison.

There is no problem with that. You should simply compare lenses with the same AoV, e.g., 12 mm for MFT versus 24 mm for FF.

I think they try their best. It's just not that easy.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
tt321
Senior MemberPosts: 4,125Gear list
Like?
Re: DXO scores are like MTF charts
In reply to agentul, 11 months ago

agentul wrote:

DonSC wrote:

There simply isn't any perfect way to make the comparison.

I think they try their best. It's just not that easy.

the best way is to look at some actual photos (real life and test scenes). in the end, that is what you are buying the camera to do.

In order for this to work, ultimately one needs to do one's own testing, on scenes that one is interested in and explore issues one cares most about, and this is not realistic for most people, especially in the current commercial environment where population centres of substantial size (even over 1M inhabitants) sometimes don't have even one comprehensive camera shop or camera+lens rental facility and most people have to or prefer to rely on web retailers.

Standard numerical measures, such as MTF scores, do have well known and established meanings. Review sites using standard testing scenes not only allow a common discussion to be held on forums like this, but also allow a careful consumer to try and correlate their real-life experiences with these stable and unchanging - or at least slow changing - tests. Example test scene shots may not mean a lot to the person first coming to one of them, who may not easily see differences or know how to interpret any differences, but for someone who has observed them for a longer time and done more homework, the comparisons could mean substantially more.

Reply   Reply with quote   Complain
Dr_Jon
Senior MemberPosts: 1,613Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Anders W, 11 months ago

I generally agree with your post, especially the first part, although I wouldn't say m43 lenses are twice as good as FF lenses as until someone comes out with a 64MP FF sensor we will have no idea how good some of the FF lenses are. (Although the ones with spectacular MFT curves will presumably do pretty well if that ever happens.)

Plus in any case you can only buy what is available, and other than good telephotos and some specialised lenses m43 does pretty well.

 Dr_Jon's gear list:Dr_Jon's gear list
Fujifilm FinePix Real 3D W3 Sony RX100 Canon EOS 5D Mark II Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GH3 +25 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 19,062Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Dr_Jon, 11 months ago

Dr_Jon wrote:

I generally agree with your post, especially the first part,

Always a good beginning.

although I wouldn't say m43 lenses are twice as good as FF lenses

I was only saying they are about twice as good in terms of lp/mm on the sensor. On a per-image basis, or at the same display size, they are about as good when the sensor resolution (again on a per-image basis) is held constant.

as until someone comes out with a 64MP FF sensor we will have no idea how good some of the FF lenses are. (Although the ones with spectacular MFT curves will presumably do pretty well if that ever happens.)

I don't think a 64 MP FF sensor versus a 16 MP MFT sensor is a very interesting comparison. Why would it be?

A more interesting comparison in my view is that of MFT lenses on a 16 MP MFT sensor versus FF lenses on a 16 MP FF sensor, as is possible by means of the DxO results for the Nikon D4 and Df (preferably relying on acutance rather than P-MPix).

Plus in any case you can only buy what is available, and other than good telephotos and some specialised lenses m43 does pretty well.

Agreed.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
agentul
Contributing MemberPosts: 606Gear list
Like?
Re: DXO scores are like MTF charts
In reply to tt321, 11 months ago

tt321 wrote:

agentul wrote:

DonSC wrote:

There simply isn't any perfect way to make the comparison.

I think they try their best. It's just not that easy.

the best way is to look at some actual photos (real life and test scenes). in the end, that is what you are buying the camera to do.

In order for this to work, ultimately one needs to do one's own testing, on scenes that one is interested in and explore issues one cares most about, and this is not realistic for most people, especially in the current commercial environment where population centres of substantial size (even over 1M inhabitants) sometimes don't have even one comprehensive camera shop or camera+lens rental facility and most people have to or prefer to rely on web retailers.

the internet has many review sites and blogs. it does take a while to look at them, but the advantage is that you can spread out the process across a month or two, so that you can get a better idea. doing one's own testing probably works for people who are professional photographers and can set aside dedicated time for this. as i said, test photos don't tell the whole story, nor do synthetic benchmarks of any kind. i prefer to combine everything along with the little technical understanding of cameras that i have in order to arrive to a conclusion. at least that's what i did when i was shopping for a camera.

i see that there are people for which DxO is the final word in camera performance. allow me to share a similar story: there are a few synthetic computer benchmarking tools out there, designed initially to simulate certain workloads such as archiving and gaming performance. one thing that has been noticed over the years in reviews is that some of these programs simulate conditions that are not really often encountered in real life. one such example is 3D Mark, which has grown to be more demanding than many high end modern games. this is one reason why CPU and video card reviews use real life games and programs - most people want to know how fast their game or video encoding will be, even with mid-level hardware. having 20% less points in a synthetic benchmark doesn't necessarily mean that you're getting sub-par hardware. it may mean that you're getting enough performance for your needs at a lower price.the other reason computer hardware reviews have moved away from synthetic benchmark programs is that some manufacturers have been caught cheating - optimizing drivers for the synthetic benchmarks used in reviews, while real life performance was lagging.

also, i recall some embarrassing articles back in 2008 when the X58 chipset was launched, and motherboard manufacturers were rushing to put out models that would be faster than the competition in terms of RAM performance. a closer look by AnandTech revealed that the BIOS did not match what was written on paper: even though a model was supposed to support more than 20GB of RAM, in real life it wasn't working properly with anything above the 3 or 6 GB used in reviews at the time.

another example would be how certain car manufacturers have been caught cheating at fuel efficiency tests by optimizing the ECUs to detect the test cycle and reduce fuel consumption to levels below what would be encountered in real life scenarios.

bottom line: synthetic benchmarks are useful, but the numbers should always be taken with a grain of salt.

 agentul's gear list:agentul's gear list
Panasonic Lumix DMC-GH2 Panasonic Lumix G Vario 14-42mm F3.5-5.6 ASPH OIS Panasonic Leica Summilux DG 25mm F1.4 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS Panasonic Lumix DMC-LZ2 +1 more
Reply   Reply with quote   Complain
Dr_Jon
Senior MemberPosts: 1,613Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Anders W, 11 months ago

I was just saying you don't know the lens resolution so it's no lp/mm on the sensor but lp/mm the sensor can resolve with that lens. A Canon 500 f4 might resolve 120 lp/mm on a sensor, but no-one has a sensor to see that (so it isn't relevant yet).

 Dr_Jon's gear list:Dr_Jon's gear list
Fujifilm FinePix Real 3D W3 Sony RX100 Canon EOS 5D Mark II Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GH3 +25 more
Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 19,062Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Dr_Jon, 11 months ago

Dr_Jon wrote:

I was just saying you don't know the lens resolution so it's no lp/mm on the sensor but lp/mm the sensor can resolve with that lens. A Canon 500 f4 might resolve 120 lp/mm on a sensor, but no-one has a sensor to see that (so it isn't relevant yet).

Of course I agree that the resolution depends on the sensor resolution as long as it is measured as DxO does it (by means of images shot by a certain body). What I tried to say is that given this restriction, the best we can do when comparing MFT lenses and FF lenses is to compare them on sensors having the same pixel count (and ideally the same strength AA filter).

Ideally, lens performance should also be measured on an optical bench so that we can eliminate (if I understand things correctly) the restriction imposed by having to rely on a certain sensor. However, I don't know if and how that works out with lenses that need to be mounted on a body in order to set focus and aperture (like all but the purely manual MFT lenses).

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Steen Bay
Veteran MemberPosts: 6,416
Like?
Re: DXOMARK.com, did you notice
In reply to Dr_Jon, 11 months ago

Dr_Jon wrote:

I was just saying you don't know the lens resolution so it's no lp/mm on the sensor but lp/mm the sensor can resolve with that lens. A Canon 500 f4 might resolve 120 lp/mm on a sensor, but no-one has a sensor to see that (so it isn't relevant yet).

Cameras like the E-M1 and D7100 have almost the same pixel size/density, and both are without AA-filter. Could be used to compare the resolution of mFT and Nikon FF/APS-C lenses measured in lp/mm (think that the D7100 should have app. 26mp in order to make it a totally fair comparison).

Reply   Reply with quote   Complain
Anders W
Forum ProPosts: 19,062Gear list
Like?
Re: DXOMARK.com, did you notice
In reply to Steen Bay, 11 months ago

Steen Bay wrote:

Dr_Jon wrote:

I was just saying you don't know the lens resolution so it's no lp/mm on the sensor but lp/mm the sensor can resolve with that lens. A Canon 500 f4 might resolve 120 lp/mm on a sensor, but no-one has a sensor to see that (so it isn't relevant yet).

Cameras like the E-M1 and D7100 have almost the same pixel size/density, and both are without AA-filter. Could be used to compare the resolution of mFT and Nikon FF/APS-C lenses measured in lp/mm (think that the D7100 should have app. 26mp in order to make it a totally fair comparison).

Sure you can do that but in what sense would such a comparison be "fair"? In my view, it would be just as unfair (in the opposite direction) as one where we'd compare resolution in lp/mm without taking the size of the image circle into account. To me, a fair comparison isĀ one where we measure lens resolution on a per-image basis using sensors with the same per-image resolution.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +21 more
Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads