SMI (Sensitivity Metemerism Index) -- is it important?

[Great Bustard]

http://www.dxomark.com/About/In-depth-measurements/Measurements/Color-sensitivity

The sensitivity metamerism index (SMI) is defined in the ISO standard 17321 and describes the ability of a camera to reproduce accurate colors. Digital processing permits changing color rendering at will, but whether the camera can or cannot exactly and accurately reproduce the scene colors is intrinsic to the sensor response and independent of the raw converter.

The underlying physics is that a sensor can distinguish exactly the same colors as the average human eye, if and only if the spectral responses of the sensor can be obtained by a linear combination of the eye cone responses. These conditions are called Luther-Ives conditions, and in practice, these never occur. There are objects that a sensor sees as having certain colors, while the eye sees the same objects differently, and the reverse is also true.

SMI is an index quantifying this property, and is represented by a number lower than 100 (negative values are possible). A value equal to 100 is perfect color accuracy, and is only attained when Luther-Ives conditions hold (which, as previously stated, never happens in practice). A value of 50 is the difference in color between a daylight illuminant and an illuminant generated by fluorescent tubes, which is considered a moderate error.

More precisely, SMI is defined as

,

where

is the average CIELAB error observed on a set of various colors. In our experiments, we used the 18 colored patches of a Gretag McBeth color checker, as ISO 17321 recommends. The SMI varies depending on the illuminant.

In practice, the SMI for DSLRs ranges between 75 and 85, and is not very discriminating. It is different for low-end cameras (such as camera phones), which typically have a SMI of about 40. For this reason, we give this measurement as an indication but do not integrate it in DxO Mark.

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

 

[Leonard Migliore]

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

I was unable to get accurate colors for art copying with a Nikon D300 despite months of trying. I have everything calibrated and use a Passport to generate a Lightroom preset. Using this technique, I can get some colors correct but never all of them; there's always a wild tone somewhere.

I switched to a Canon 50D and immediately got quite acceptable results, easily corrected to good visual color accuracy.

So in my experience, the Canon has far less metamerism when looking at oil pigments than does the Nikon. How does that correlate with DxO's measurements?

--
Leonard Migliore

 

[wfektar]

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

Metamerism is not something you can generally correct in post. You lost the required data prior to capture. You can, of course, do wholesale replacement of specific color values, if that counts as "correction", though that seems to me even harder to get right in general than other data fabrication techniques like "Content Aware Fill" or "Moire Correction". Whether that's important or not ... depends

 

[Jack Hogan]

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

I was unable to get accurate colors for art copying with a Nikon D300 despite months of trying. I have everything calibrated and use a Passport to generate a Lightroom preset. Using this technique, I can get some colors correct but never all of them; there's always a wild tone somewhere.

I switched to a Canon 50D and immediately got quite acceptable results, easily corrected to good visual color accuracy.

So in my experience, the Canon has far less metamerism when looking at oil pigments than does the Nikon. How does that correlate with DxO's measurements?

 

[Mark S Abeln]

I switched to a Canon 50D and immediately got quite acceptable results, easily corrected to good visual color accuracy.

According to DxoMark , the 50D has an SMI of 87 for daylight and 80 for incandescent, while the D300 has numbers of 84 and 82.

But these numbers tell us nothing of where the cameras are accurate or not.

 

[Mark S Abeln]

In our experiments, we used the 18 colored patches of a Gretag McBeth color checker, as ISO 17321 recommends.

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

Cameras valued for good color typically have high SMI values, while those known for poor color usually have low numbers. But not always. Anecdotal evidence suggests that some photographers feel that their older cameras deliver better color than their newer ones — and the numbers have declined for Canon, Pentax, and Nikon — but not Sony.

I see a few problems with SMI:

• As a statistical measure, it only gives us an average and doesn’t tell us about the distribution of errors.
• If they only use the ColorChecker 18 color chart, then this is an amazingly poor sample. Really, there should be a better methodology using far more colors. What I find surprising is that manufacturers can’t even get these 18 colors right.
• There could be a potential problem of them ‘gaming the system’ where manufacturers only work on getting those 18 colors right, ignoring the others, giving a deceivingly high SMI number. This is a big problem with artificial illuminants such as fluorescent and LED lamps, which are often designed to delver a high Color Rendering Index even though they still have poor spectra — and the CRI test is even worse than SMI, using only 8 sample colors. Lamp manufacturers lately rejected the use of the full ColorChecker chart, which is rather distressing.
• I’m not to sure how good CIELab is as a color distance metric, although it is far better than Euclidian RGB distance.
• This does not take ease of retouchability of colors into account, which is related to color depth. High color depth does not mean that the colors are correct, but it does make them more correctable in post processing.

The single most important factor in getting both accurate and pleasing color is the camera’s automatic white balance, along with the white balance presets. If the white balance is off, then all the colors will be off and this will strongly bias subjective impressions of color quality.

Certainly this is an area that could benefit from a lot more research.

 

[Great Bustard]

In our experiments, we used the 18 colored patches of a Gretag McBeth color checker, as ISO 17321 recommends.

So, is the SMI an important, but overlooked or ignored, measure of IQ of a captured photo, or is it something that is easily taken care of in processing, either pre or post?

Cameras valued for good color typically have high SMI values, while those known for poor color usually have low numbers. But not always. Anecdotal evidence suggests that some photographers feel that their older cameras deliver better color than their newer ones — and the numbers have declined for Canon, Pentax, and Nikon — but not Sony.

I see a few problems with SMI:

• As a statistical measure, it only gives us an average and doesn’t tell us about the distribution of errors.
• If they only use the ColorChecker 18 color chart, then this is an amazingly poor sample. Really, there should be a better methodology using far more colors. What I find surprising is that manufacturers can’t even get these 18 colors right.

In other words, it's not the average, so much, but the standard deviation and the sample space.

• There could be a potential problem of them ‘gaming the system’ where manufacturers only work on getting those 18 colors right, ignoring the others, giving a deceivingly high SMI number. This is a big problem with artificial illuminants such as fluorescent and LED lamps, which are often designed to delver a high Color Rendering Index even though they still have poor spectra — and the CRI test is even worse than SMI, using only 8 sample colors. Lamp manufacturers lately rejected the use of the full ColorChecker chart, which is rather distressing.

This would be akin to weakening the dyes in the CFA to get a higher QE.

• I’m not to sure how good CIELab is as a color distance metric, although it is far better than Euclidian RGB distance.
• This does not take ease of retouchability of colors into account, which is related to color depth. High color depth does not mean that the colors are correct, but it does make them more correctable in post processing.

The single most important factor in getting both accurate and pleasing color is the camera’s automatic white balance, along with the white balance presets. If the white balance is off, then all the colors will be off and this will strongly bias subjective impressions of color quality.

Is AWB a factor for shooting RAW?

Certainly this is an area that could benefit from a lot more research.

Thank you very much! On a personal note, I just want to express my appreciation for all your posts -- they are always very informative and clearly expressed!

 

[wfektar]

I see a few problems with SMI:

• As a statistical measure, it only gives us an average and doesn’t tell us about the distribution of errors.
• If they only use the ColorChecker 18 color chart, then this is an amazingly poor sample. Really, there should be a better methodology using far more colors. What I find surprising is that manufacturers can’t even get these 18 colors right.

Good points.

In other words, it's not the average, so much, but the standard deviation and the sample space.

What do you mean by "standard deviation and sample space"? Because to me it seems to be useful to treat the sample space as a vector space, in which case you need the mean errors of the norms. A single number seems to be useless here. Especially as the source spectrum matters. How many numbers are we talking here to be 1) useful for color accuracy, and 2) usable?

This would be akin to weakening the dyes in the CFA to get a higher QE.

I would be shocked, <i>shocked!</i> to learn that anyone would do that!

 

[hjulenissen]

This would be akin to weakening the dyes in the CFA to get a higher QE.

I would be shocked, <i>shocked!</i> to learn that anyone would do that!

Why wouldn't they? If it makes for overall more pleasing images, then I am all for it.

-h

 

[hjulenissen]

Cameras valued for good color typically have high SMI values, while those known for poor color usually have low numbers. But not always. Anecdotal evidence suggests that some photographers feel that their older cameras deliver better color than their newer ones — and the numbers have declined for Canon, Pentax, and Nikon — but not Sony.

I see a few problems with SMI:

• As a statistical measure, it only gives us an average and doesn’t tell us about the distribution of errors.
• If they only use the ColorChecker 18 color chart, then this is an amazingly poor sample. Really, there should be a better methodology using far more colors. What I find surprising is that manufacturers can’t even get these 18 colors right.
• There could be a potential problem of them ‘gaming the system’ where manufacturers only work on getting those 18 colors right, ignoring the others, giving a deceivingly high SMI number. This is a big problem with artificial illuminants such as fluorescent and LED lamps, which are often designed to delver a high Color Rendering Index even though they still have poor spectra — and the CRI test is even worse than SMI, using only 8 sample colors. Lamp manufacturers lately rejected the use of the full ColorChecker chart, which is rather distressing.
• I’m not to sure how good CIELab is as a color distance metric, although it is far better than Euclidian RGB distance.
• This does not take ease of retouchability of colors into account, which is related to color depth. High color depth does not mean that the colors are correct, but it does make them more correctable in post processing.

Color science makes my head hurt...

Is there, or is there not some reference spectral sensitivity function that can be claimed to be "the right one"? If there is, why mess around with color science, why not measure the spectral response of each channel in e.g. 5nm steps, report the full error vectors and some simplified metric (i.e. rms error)? I don't have the equipment or know-how to do this, but I would expect that DXO and similar does?

Is it "fair" to compare the "color accuracy" of a 20MP camera to that of a 5MP camera? Won't the resolution buy you some spatial resolution headroom that can be spent doing noise reduction on a heavily color corrected signal? Does it even make sense to do full color correction of deep shadows (as color errors down there seems to bother us less, while noisy appearance is a problem)?

-h

 

[Jack Hogan]

Color science makes my head hurt...

Is there, or is there not some reference spectral sensitivity function that can be claimed to be "the right one"? If there is, why mess around with color science, why not measure the spectral response of each channel in e.g. 5nm steps, report the full error vectors and some simplified metric (i.e. rms error)? I don't have the equipment or know-how to do this, but I would expect that DXO and similar does?

Answer to your first question: Apparently not, as I understand it, mainly because of the unstable location/number of the metamers - when referred to a potential 'reference spectral sensitivity function' - as a result of even tiny changes in the spectral power distribution of the illuminant.

This issue was taken from a slightly different angle in this recent thread . Same less than satisying conclusion: it's perceptual, subjective and unstable .

Jack

 

[Mark S Abeln]

This would be akin to weakening the dyes in the CFA to get a higher QE.

I would be shocked, <i>shocked!</i> to learn that anyone would do that!

Why wouldn't they? If it makes for overall more pleasing images, then I am all for it.

The images may be pleasing because they have better high ISO performance — but may not have as pleasing color. Canon and Nikon were willing to make that trade-off, but not Sony.

 

[Mark S Abeln]

Color science makes my head hurt...

Is there, or is there not some reference spectral sensitivity function that can be claimed to be "the right one"? If there is, why mess around with color science, why not measure the spectral response of each channel in e.g. 5nm steps, report the full error vectors and some simplified metric (i.e. rms error)? I don't have the equipment or know-how to do this, but I would expect that DXO and similar does?

Answer to your first question: Apparently not, as I understand it, mainly because of the unstable location/number of the metamers - when referred to a potential 'reference spectral sensitivity function' - as a result of even tiny changes in the spectral power distribution of the illuminant.

This issue was taken from a slightly different angle in this recent thread . Same less than satisying conclusion: it's perceptual, subjective and unstable .

“Subjective” doesn’t mean “whatever”, nor does it mean you can’t measure it or do science on it. It simply means that the measure of some things are to be found within the human person, and that includes color. There is remarkable agreement among humans on color, and even good characterization of those cases where there is disagreement.

Color science isn’t epistemologically impossible, its just very difficult, but that is hardly a reason not to do it; rather, it is a reason for putting forth some extra effort to understand it.

However, this means that a purely physical understanding of color — like Newton’s prism — is bound to fail, and spectral studies will only tell a part of the story.

 

[Great Bustard]

This would be akin to weakening the dyes in the CFA to get a higher QE.

I would be shocked, <i>shocked!</i> to learn that anyone would do that!

Why wouldn't they? If it makes for overall more pleasing images, then I am all for it.

The images may be pleasing because they have better high ISO performance — but may not have as pleasing color. Canon and Nikon were willing to make that trade-off, but not Sony.

But, even noise considerations aside, the trade-off is not necessarily a bad thing, is it? As you said, higher SMI scores may not be representative of actual performance, because of the colors measured, the standard deviations, and subjective preferences.

For example, let's say Canon colors are more off, on average, than Sony colors, but for particular colors, such as skin tones, they are more on the mark.

 

[Mark S Abeln]

The images may be pleasing because they have better high ISO performance — but may not have as pleasing color. Canon and Nikon were willing to make that trade-off, but not Sony.

But, even noise considerations aside, the trade-off is not necessarily a bad thing, is it? As you said, higher SMI scores may not be representative of actual performance, because of the colors measured, the standard deviations, and subjective preferences.

For example, let's say Canon colors are more off, on average, than Sony colors, but for particular colors, such as skin tones, they are more on the mark.

That is a possibility, and is difficult to tell from just one number. But an SMI of 87 is probably better than one with an SMI of 75, although the details are so important.

You might find more information here:

http://www.imaging-resource.com

--

The Refracted Light

A website about the Art and Science of Photography

therefractedlight.blogspot.com

 

[Great Bustard]

The images may be pleasing because they have better high ISO performance — but may not have as pleasing color. Canon and Nikon were willing to make that trade-off, but not Sony.

But, even noise considerations aside, the trade-off is not necessarily a bad thing, is it? As you said, higher SMI scores may not be representative of actual performance, because of the colors measured, the standard deviations, and subjective preferences.

For example, let's say Canon colors are more off, on average, than Sony colors, but for particular colors, such as skin tones, they are more on the mark.

That is a possibility, and is difficult to tell from just one number. But an SMI of 87 is probably better than one with an SMI of 75, although the details are so important.

I'd also like to ask how significant such a difference may be. For example, if one sensor had a QE of 75% and another a QE of 87%, well, that's pretty much the same thing, really (0.2 stops apart). Not saying it's the same for the SMI, but just putting that analogy out there as a possibility based on my ignorance of the measure.

You might find more information here:

http://www.imaging-resource.com

Could I trouble you for a more specific link? Thanks!

 

[Ember42]

Cameras valued for good color typically have high SMI values, while those known for poor color usually have low numbers. But not always. Anecdotal evidence suggests that some photographers feel that their older cameras deliver better color than their newer ones — and the numbers have declined for Canon, Pentax, and Nikon — but not Sony.

I see a few problems with SMI:

• As a statistical measure, it only gives us an average and doesn’t tell us about the distribution of errors.
• If they only use the ColorChecker 18 color chart, then this is an amazingly poor sample. Really, there should be a better methodology using far more colors. What I find surprising is that manufacturers can’t even get these 18 colors right.
• There could be a potential problem of them ‘gaming the system’ where manufacturers only work on getting those 18 colors right, ignoring the others, giving a deceivingly high SMI number. This is a big problem with artificial illuminants such as fluorescent and LED lamps, which are often designed to delver a high Color Rendering Index even though they still have poor spectra — and the CRI test is even worse than SMI, using only 8 sample colors. Lamp manufacturers lately rejected the use of the full ColorChecker chart, which is rather distressing.
• I’m not to sure how good CIELab is as a color distance metric, although it is far better than Euclidian RGB distance.
• This does not take ease of retouchability of colors into account, which is related to color depth. High color depth does not mean that the colors are correct, but it does make them more correctable in post processing.

Color science makes my head hurt...

Is there, or is there not some reference spectral sensitivity function that can be claimed to be "the right one"? If there is, why mess around with color science, why not measure the spectral response of each channel in e.g. 5nm steps, report the full error vectors and some simplified metric (i.e. rms error)? I don't have the equipment or know-how to do this, but I would expect that DXO and similar does?

Is it "fair" to compare the "color accuracy" of a 20MP camera to that of a 5MP camera? Won't the resolution buy you some spatial resolution headroom that can be spent doing noise reduction on a heavily color corrected signal? Does it even make sense to do full color correction of deep shadows (as color errors down there seems to bother us less, while noisy appearance is a problem)?

-h

The problem is that you are mapping a continuous spectrum to three discrete values. If you had a prism + sensor array at each pixel, you would get perfect colour accuracy, but that is not how a digital camera sensor works.

The 18 colour swatches have known reflective spectrums so that under known incident light spectrums (i.e, standard daylingt, incandescent, etc) they have a known spectrum. These spectrums have known mappings to RGB values. These are compared to what a DSLR actually records, and the difference is the errors. If the DSLR bayer pigments had the same response as the standard eye cone responses, then the colour out of the DSLR would be 'perfect'. Unfortunately, creating this filter is not trivial, and may absorb much more light than is desirable for high ISO performance purposes.

Resolution does not matter for colour accuracy, as they are comparing to large uniform patches that are much larger than a pixel. Noise reduction does not help you if your average RGB values don't match the 'correct' ones. A metamarism error is not recoverable in post processing because the full spectrum data has been discarded in the mapping.

This is the colour problem for a foveon style sensor - the multi layer abosrbance spectrums are much more different from the human eye cone sensitivity spectrum than the bayer filter pass spectrum is.

 

[hjulenissen]

Resolution does not matter for colour accuracy, as they are comparing to large uniform patches that are much larger than a pixel. Noise reduction does not help you if your average RGB values don't match the 'correct' ones. A metamarism error is not recoverable in post processing because the full spectrum data has been discarded in the mapping.

Here is my take:

You have (essentially) 3 channels of e.g. 14 bits each. The essential amount of information is somewhat less, say (for simplicity more than accuracy) that it is 8 bits per channel. That is, each full color sample (post demosaic) contains 3x8=24 bits of information about color. Now, this representation might not be ideal or conform to any standard representation of color. But through a full 24-bit table look-up we can re-map it. So what is the problem with this method (besides memory requirements, processing time and the issues of estimating the table)?

1. Two different colors might register as the same bit-pattern (I believe that this is the metamerism problem). There is no apparent fix for this, one would have to go for a compromise (e.g. render them as some intermediate color, or setup a rule that makes assumptions about which is most likely to appear in a scene etc).

2. Even if different colors generally result in different bit-patterns, the 3-d mapping could be highly "non-smooth". Two colors that are "close" in the camera representation might be "less close" in the corrected output representation. Or the other way around. This is not so much of a (simplified) theoretical problem, but it might be a large practical problem. With real-world problems such as noise and quantization, stretching the signal representation could make readily visible issues that would otherwise be invisible.

My gut-feeling is that 2) is more of a problem than 1), given the number of bits, level of noise and accuracy of CFAs?

-h

 

[xpatUSA]

This is the colour problem for a foveon style sensor - the multi-layer absorbance spectrums are much more different from the human eye cone sensitivity spectrum than the Bayer filter pass spectrum is.

And yet the early Foveon sensors apparently did quite well:

"Table 1. The Metamerism Index: a Kodak frame-transfer ccds, an Agilent CMOS sensor, a Sony interline ccd, and the Foveon X3 sensor.

Lower values correlate with the camera having better ability to colorimetrically capture a scene.

Camera, Sensor, Metamerism index
Kodak DCS-460, Kodak, 0.2974
Concord EyeQ, Agilent, 0.2873
Sigma SD9, Foveon X3 – F7, 0.1999
HP 618, Sony ICX-284, 0.1802"

Transcribed from: http://kronometric.org/phot/sensor/fov/CIC10_Lyon_Hubel_FINAL.pdf

 

[Bernard Delley]

and can be off by a lot. There must be a lot of constraints on the colours of the CFA concerning micro fabrication, precision, stability and transmission. Obvious problem areas are in the differentiation of the deep reds, and specially narrow pigments at the red-green transition, the yellows, and even more at the green-blue transition near 480nm. The eye has a dip in sensitivity there, seeing relatively dark blue-green. The CFA separation gets these colours much brighter and easily errs to one or the other side as compared to the eye.

I recently posted a quite striking example, (the dark green jacket)

http://www.dpreview.com/forums/post/53032750