DXOMARK score for A7S series: Why has low light score worsened?

[pyloricantrum]

I was wondering why the low light score for the A7S series has gotten worse since the release of the original A7S?

 

[lattesweden]

A7S-A7SII and also A7-A7II both mkII having the same sensor as the mkI got lower high ISO results due to the IBIS unit. Well, not the unit in itself, but that it was harder to get a heat sink onto the sensor then, and a hotter sensor produces more noise.

--
Best regards
/Anders
----------------------------------------------------
Mirrorless, mirrorless on the wall, say which is the best camera of them all?
Why don't I get a manual in Manual mode?
Come to the Sony side, we have lenses...
Some images:
https://www.dpreview.com/forums/post/65325637
https://www.dpreview.com/forums/post/64169208
https://www.dpreview.com/forums/post/64221482
https://www.dpreview.com/forums/post/65120847
https://www.dpreview.com/forums/post/65121520
https://www.dpreview.com/forums/post/65130731
Article about how to transfer, edit and share RAW-images wireless out in the field: https://www.dpreview.com/forums/thread/4609147

 

[whumber]

A7S-A7SII and also A7-A7II both mkII having the same sensor as the mkI got lower high ISO results due to the IBIS unit. Well, not the unit in itself, but that it was harder to get a heat sink onto the sensor then, and a hotter sensor produces more noise.

Sensor temperature has almost no effect on this type of measurement as it affects dark current noise, this really only affects long exposures unless the sensor temperature gets to some extreme level.

 

[QuietOC]

I wonder if DXOMark kept using the same lights and didn't account for them getting dimmer as they aged. Somewhat similar to how DPReview didn't notice their test scene fading. There seem to be some systemic non-random errors in DXO's procedures.

I've noticed that the bulbs I am currently using color shift as they are left on. I am sure they will fade over time too. Pretty much have to do back-to-back testing and repeat the first sample at the end of the cycle to at least get some idea of the changing conditions.

It is still much better than testing with natural light. I came to a realization yesterday how I've fooled myself with some comparisons I've done outside.

 

[lattesweden]

A7S-A7SII and also A7-A7II both mkII having the same sensor as the mkI got lower high ISO results due to the IBIS unit. Well, not the unit in itself, but that it was harder to get a heat sink onto the sensor then, and a hotter sensor produces more noise.

Sensor temperature has almost no effect on this type of measurement as it affects dark current noise, this really only affects long exposures unless the sensor temperature gets to some extreme level.

I'm no expert in this field, I relayed what others said at the time the second generation bodys came and DXOmark measurement them. It sounded logically to me. I should have pointed that out in my first answer to the OP, sorry for that.

If you are better knowledgeable about sensors then maybe you can explain why the same sensor got worse high ISO performance in the second generation bodys?

--
Best regards
/Anders
----------------------------------------------------
Mirrorless, mirrorless on the wall, say which is the best camera of them all?
Why don't I get a manual in Manual mode?
Come to the Sony side, we have lenses...
Some images:
https://www.dpreview.com/forums/post/65325637
https://www.dpreview.com/forums/post/64169208
https://www.dpreview.com/forums/post/64221482
https://www.dpreview.com/forums/post/65120847
https://www.dpreview.com/forums/post/65121520
https://www.dpreview.com/forums/post/65130731
Article about how to transfer, edit and share RAW-images wireless out in the field: https://www.dpreview.com/forums/thread/4609147

 

[whumber]

A7S-A7SII and also A7-A7II both mkII having the same sensor as the mkI got lower high ISO results due to the IBIS unit. Well, not the unit in itself, but that it was harder to get a heat sink onto the sensor then, and a hotter sensor produces more noise.

Sensor temperature has almost no effect on this type of measurement as it affects dark current noise, this really only affects long exposures unless the sensor temperature gets to some extreme level.

I'm no expert in this field, I relayed what others said at the time the second generation bodys came and DXOmark measurement them. It sounded logically to me. I should have pointed that out in my first answer to the OP, sorry for that.

If you are better knowledgeable about sensors then maybe you can explain why the same sensor got worse high ISO performance in the second generation bodys?

All kinds of reasons it could happen;

• Measurement error
• Measurement uncertainty
• Tweaking dual gain changeover point
• CFA changes
• RAW processing changes

For reference, here are Bill Claff's measurements for the same two cameras.





The other aspect you have to remember is that DxO's definition of high ISO score is a terrible way of characterizing high ISO performance. It's based on taking the (interpolated) ISO value that satisfies a minimum of three fairly arbitrarily selected criteria values. It's well known for producing nonsense results.

 

[pollup]

I was wondering why the low light score for the A7S series has gotten worse since the release of the original A7S?

The problem with DXOMark is that they define arbitrary scores that don't make sense, and the overall score is just an average of 3 scores (portrait, landscape, sports), which again doesn't make sense. You can't just take the average of 3 different things. If one of them is near 0, the overall score might still be high because of the average, but in fact it should be near 0.

Their 3 scores:

- Portrait = simply color depth ... doesn't really make sense

- Landscape = maximum DR

- Sports / low-light = "We have therefore defined low-light ISO as the highest ISO setting for a camera that allows it to achieve a SNR of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits."

First, their low-light score is based on the ISO, which camera manufacturers can easily shift left or right, so it's already not very scientific, and second, they get a value like 3116, which is not an ISO value that you can set in camera, so what is this?? And third, it still doesn't tell you if the camera is good in low light or not, it just says that you can get a high ISO with decent IQ, but it doesn't say that you can get a great IQ with this camera in low light.

For ex, maybe camera A can get fantastic low light IQ at low iso and also iso 1600 but not so good at iso 3200, and camera B gets average IQ from iso 100 to iso 3200, guess what? Camera B will score higher even though it's never good in low light any any iso.

And also, they don't even measure readout speed. That value could have been used for "Sports".

I mean, at this point, people should stop using DXOMark.