CMY + RGB Color Filter Array

Photato

Senior Member
Messages
3,156
Solutions
1
Reaction score
1,263
Location
US
Wondering about the possible merits of a hexachromatic CFA, one that sample more points along the visible light spectrum rather than the typical 3 RGB.
Is there any reason why such CFA scheme has not been tried?


The benefits to that approach would be.
  • Higher Color Accuracy, specially skin tones. RMY
  • Higher Dynamic Range
  • Higher Sensitivity due to CMY
  • Less noticeable moire artifacts given the wider spread and more complex pattern.
  • Probably more resilient to IR contamination, IOW better Reds with the help of MY
The only disadvantage I see is more complex processing algorithms but given today processing power that shouldn't be an issue.
 
Wondering about the possible merits of a hexachromatic CFA, one that sample more points along the visible light spectrum rather than the typical 3 RGB.
Is there any reason why such CFA scheme has not been tried?

The benefits to that approach would be.
  • Higher Color Accuracy, specially skin tones. RMY
  • Higher Dynamic Range
  • Higher Sensitivity due to CMY
  • Less noticeable moire artifacts given the wider spread and more complex pattern.
  • Probably more resilient to IR contamination, IOW better Reds with the help of MY
The only disadvantage I see is more complex processing algorithms but given today processing power that shouldn't be an issue.
If you're going to have 6 filters, I don't think that RGB/CMY is the best choice, especially if C = 1-R, M = 1- G, and Y = 1-B. You want to have a set of basis functions that work well to get linearly to the Standard Observer with the lighting and subject reflectances that you think important.

And this is not without penalty in the form of chromaticity resolution, which suffers as the number of filter types increases, for a fixed pixel count.

There have been studies that indicated that adding a fourth filter can help reduce capture metameric error a lot, but as I remember they were done with filter wheels, and the resolution loss was not addressed. I also seem to remember a study that said that, once you got to 10 or 12 optimized filters, there wasn't much point in going any further. Of course, this would depend on how spiky the lighting and reflectance spectra were.

Jim
 
Last edited:
Wondering about the possible merits of a hexachromatic CFA, one that sample more points along the visible light spectrum rather than the typical 3 RGB.
Is there any reason why such CFA scheme has not been tried?
Some early digital cameras were CMY; of course, most color printers always include cyan, magenta, and yellow inks as a minimum.

However, CMY are the subtractive primaries, useful when you start with white paper as your brightest tones possible, and the inks subtract from this white. Any other primaries will lead to dark prints. The brightest, most saturated blue pigment is rather dark, and compromises have to be made.

RGB are the additive primaries, when we are directly dealing with light.
The benefits to that approach would be.
  • Higher Color Accuracy, specially skin tones. RMY
As C=G+B; M=R+B; and Y=R+G, I see no additional benefit from having three more filters, as they will duplicate what's already captured. Furthermore, the Bayer array of 2x2 color filters already lead to color error, which would be made worse by having more colors, and sensor resolution would decrease.

The human eye only uses three color receptors, and so that would seem to be sufficient.
  • Higher Dynamic Range
Probably lower dynamic range, as those mixed colors would involve additional subtractions of one color channel from another to get an sRGB value out of the raw data. A while back I calculated that a pure CMY sensor would have about an 18% lower signal-to-noise ratio compared to an RGB sensor, and if you include the read noise, the results would be even worse. A mixed RGB-CMY sensor wouldn't be quite as bad, but still not as good as RGB.
  • Higher Sensitivity due to CMY
That's true. You'd get a higher native ISO rating.

However, it seems that chroma noise is much more objectionable than luma noise, and this will increase the chroma noise.
  • Less noticeable moire artifacts given the wider spread and more complex pattern.
The color defects would become more complex and intractable, as we see on the Fuji X-Trans array.
  • Probably more resilient to IR contamination, IOW better Reds with the help of MY
It depends almost entirely on the IR cut filter, which can be used with any filter array.

In CMY, R = (Y + M - C) / 2, which would have more relative noise than just an R by itself.

A bigger problem with any color filtering is that we don't have a good match with human vision. Modern filters are close, but aren't quite there yet.
The only disadvantage I see is more complex processing algorithms but given today processing power that shouldn't be an issue.
That wouldn't be much additional processing, so you are right, it wouldn't be much of an issue.
 
And such a camera would have a base ISO of 10, or 1 :-)
 
Wondering about the possible merits of a hexachromatic CFA, one that sample more points along the visible light spectrum rather than the typical 3 RGB.
Is there any reason why such CFA scheme has not been tried?

The benefits to that approach would be.
  • Higher Color Accuracy, specially skin tones. RMY
  • Higher Dynamic Range
  • Higher Sensitivity due to CMY
  • Less noticeable moire artifacts given the wider spread and more complex pattern.
  • Probably more resilient to IR contamination, IOW better Reds with the help of MY
The only disadvantage I see is more complex processing algorithms but given today processing power that shouldn't be an issue.
If you're going to have 6 filters, I don't think that RGB/CMY is the best choice, especially if C = 1-R, M = 1- G, and Y = 1-B. You want to have a set of basis functions that work well to get linearly to the Standard Observer with the lighting and subject reflectances that you think important.
Dr. Bayer stated that the CFA could be either RGB or CMY, hence the sensors that came were in fact in either of those schemes. I'm assuming that is so because they are evenly spaced across the visible spectrum and of complementary nature. Those two color groups are also called Additive and Substantive and they are diametrically opposed when layed out in a hue wheel .
And this is not without penalty in the form of chromaticity resolution, which suffers as the number of filter types increases, for a fixed pixel count.
A Bayer CFA is always a compromise, but this shouldn't be less chromatic resolution than just a RGB CFA.
There have been studies that indicated that adding a fourth filter can help reduce capture metameric error a lot, but as I remember they were done with filter wheels, and the resolution loss was not addressed. I also seem to remember a study that said that, once you got to 10 or 12 optimized filters, there wasn't much point in going any further. Of course, this would depend on how spiky the lighting and reflectance spectra were.

Jim
That is exactly the point with this approach, more samples across the light spectrum, more accuracy in rendering colors.
In the early days of Digital photography maybe this approach of hybrid CFA was discarded given the low sensor pixel count and lower chip processing power.
I'd argue that with current sensor pixel densities this hybrid CFA scheme could be revised and bring to market.

That experiment using a panchromatic sensor looks interesting, done with these 6 filters should give us an answer in regards to color accuracy.

It could be that the gains in the end are not as significant to justify the extra effort, but I am convinced that such camera would have this type of superior synergistic color quality, the kind that is hard to demonstrate with numbers, courtesy of the human visual perception.
 
And such a camera would have a base ISO of 10, or 1 :-)

--
http://therefractedlight.blogspot.com
Why? In theory the base ISO of this hybrid CFA should be the average between an RGB and a CMY sensor.
IOW higher base ISO than RGB, lower than CMY.
I was responding to Jim's suggestion that 10 or 12 filters would give the best results (assuming of course that perfect RGB filters aren't available. And they aren't. :-) )

--
http://therefractedlight.blogspot.com
 
Last edited:
Wondering about the possible merits of a hexachromatic CFA, one that sample more points along the visible light spectrum rather than the typical 3 RGB.
Is there any reason why such CFA scheme has not been tried?

The benefits to that approach would be.
  • Higher Color Accuracy, specially skin tones. RMY
  • Higher Dynamic Range
  • Higher Sensitivity due to CMY
  • Less noticeable moire artifacts given the wider spread and more complex pattern.
  • Probably more resilient to IR contamination, IOW better Reds with the help of MY
The only disadvantage I see is more complex processing algorithms but given today processing power that shouldn't be an issue.
If you're going to have 6 filters, I don't think that RGB/CMY is the best choice, especially if C = 1-R, M = 1- G, and Y = 1-B. You want to have a set of basis functions that work well to get linearly to the Standard Observer with the lighting and subject reflectances that you think important.
Dr. Bayer stated that the CFA could be either RGB or CMY, hence the sensors that came were in fact in either of those schemes. I'm assuming that is so because they are evenly spaced across the visible spectrum and of complementary nature. Those two color groups are also called Additive and Substantive and they are diametrically opposed when layed out in a hue wheel .
The basis functions are not sufficiently different to make any progress towards a Luther-Ives sensor. Given 3 RGB values for a constant patch, I could tell you the 3 CMY values, and vice versa.
And this is not without penalty in the form of chromaticity resolution, which suffers as the number of filter types increases, for a fixed pixel count.
A Bayer CFA is always a compromise, but this shouldn't be less chromatic resolution than just a RGB CFA.
The chroma would be sampled less frequently, if the filters were non linear combinations of each other.
There have been studies that indicated that adding a fourth filter can help reduce capture metameric error a lot, but as I remember they were done with filter wheels, and the resolution loss was not addressed. I also seem to remember a study that said that, once you got to 10 or 12 optimized filters, there wasn't much point in going any further. Of course, this would depend on how spiky the lighting and reflectance spectra were.

Jim
That is exactly the point with this approach, more samples across the light spectrum, more accuracy in rendering colors.
In the early days of Digital photography maybe this approach of hybrid CFA was discarded given the low sensor pixel count and lower chip processing power.
I'd argue that with current sensor pixel densities this hybrid CFA scheme could be revised and bring to market.

That experiment using a panchromatic sensor looks interesting, done with these 6 filters should give us an answer in regards to color accuracy.

It could be that the gains in the end are not as significant to justify the extra effort, but I am convinced that such camera would have this type of superior synergistic color quality, the kind that is hard to demonstrate with numbers, courtesy of the human visual perception.
Actually, this kind of thing can be demonstrated with numbers. Take the filter response spectra and the procedure for converting to, say, XYZ. Multiply those by the camera response spectra, and the spectra of a reflectance sample set (say, the Macbeth CC), and the desired test lighting (say, D50). Then compute differences from XYZ obtained from the samples. This can all be simulated; you don't need to actually make the filters.

Give it a try.

Jim
 
If we ever reach the point where adding more pixels does not improve spatial resolution (lense limited), it might make sense to use the «surplus pixels» for increased chromatic resolution. Ie multi-spectral color filters where one cluster of colors are about as large as the spatial extent of the PSF.

For well lit scenes, those filters could be narrow (wasting many photons). For poorly lit scenes, we might want filters that waste fewer photons. We could still have many different center frequencies (rgbcma) or some complex PCA/KLT-derived «hairy» set of responses that happen to combine into a well-behaved description of the scenes we tend to photograph.

-h
 
If we ever reach the point where adding more pixels does not improve spatial resolution (lense limited), it might make sense to use the «surplus pixels» for increased chromatic resolution.
More filters will not increase chromatic resolution, but may increase chromatic accuracy.
Ie multi-spectral color filters where one cluster of colors are about as large as the spatial extent of the PSF.

For well lit scenes, those filters could be narrow (wasting many photons). For poorly lit scenes, we might want filters that waste fewer photons. We could still have many different center frequencies (rgbcma) or some complex PCA/KLT-derived «hairy» set of responses that happen to combine into a well-behaved description of the scenes we tend to photograph.

-h
 
If we ever reach the point where adding more pixels does not improve spatial resolution (lense limited), it might make sense to use the «surplus pixels» for increased chromatic resolution. Ie multi-spectral color filters where one cluster of colors are about as large as the spatial extent of the PSF.

For well lit scenes, those filters could be narrow (wasting many photons). For poorly lit scenes, we might want filters that waste fewer photons. We could still have many different center frequencies (rgbcma) or some complex PCA/KLT-derived «hairy» set of responses that happen to combine into a well-behaved description of the scenes we tend to photograph.

-h
I probably sound like a broken record, but as long as you are adding the smaller signals from the smaller pixels together, all is good from an SNR point of view. If you start needing to do any subtraction (negative CCM coefficients) then you need to worry about SNR. You also need to worry about color cross-talk between pixels. (e.g. photon comes in at an angle thru one filter and gets absorbed by silicon under another filter, and gets counted as the latter color).
 
Eric Fossum
I probably sound like a broken record, but as long as you are adding the smaller signals from the smaller pixels together, all is good from an SNR point of view. If you start needing to do any subtraction (negative CCM coefficients) then you need to worry about SNR. You also need to worry about color cross-talk between pixels. (e.g. photon comes in at an angle thru one filter and gets absorbed by silicon under another filter, and gets counted as the latter color).
If a scene is dark enough that a 2018 camera sensor struggles with noise, then surely increasing sensel count by 10x you will still have a noise issue.

For a scenario like that, going from 3 wide/overlapping rgb sensels to 10 narrow multispectral filters, you are converting lots of information-bearing photons into heat.

That would be problematic for image quality, right?

-h
 
More filters will not increase chromatic resolution, but may increase chromatic accuracy.
I may not have used the correct terminology.

My point was that if we view the spectrum at a single point in space as a continous waveform, then sampling it with denser and narrower filters will increase the «spectral resolution».

-h
 
Eric Fossum

I probably sound like a broken record, but as long as you are adding the smaller signals from the smaller pixels together, all is good from an SNR point of view. If you start needing to do any subtraction (negative CCM coefficients) then you need to worry about SNR. You also need to worry about color cross-talk between pixels. (e.g. photon comes in at an angle thru one filter and gets absorbed by silicon under another filter, and gets counted as the latter color).
If a scene is dark enough that a 2018 camera sensor struggles with noise, then surely increasing sensel count by 10x you will still have a noise issue.

For a scenario like that, going from 3 wide/overlapping rgb sensels to 10 narrow multispectral filters, you are converting lots of information-bearing photons into heat.

That would be problematic for image quality, right?
There is an assumption that is creeping in here that multispectral sensors will use narrower filters. I'm not sure that is the best way to do things, either for more accurate color or for lower noise when converted to a Standard Observer based color space. In fact, I think it is unlikely that it is even approximately the best way.

Jim

--
http://blog.kasson.com
 
Last edited:
More filters will not increase chromatic resolution, but may increase chromatic accuracy.
I may not have used the correct terminology.

My point was that if we view the spectrum at a single point in space as a continous waveform, then sampling it with denser and narrower filters will increase the «spectral resolution».
Gotcha. But the objective is not increased spectral sampling density, it's more accurate color n'est pas?

Jim
 
Assuming that:

* Binning by summing sensel charge has inherent advantages over post-dac summing in that only a single read noise is added (for the sum) rather than per sensel

* Filtering digitally has advantages in that more complex algorithms can be used (large kernel, nonlinear, signal-adaptive) and that smoothing is possible for cfa filtered sensors

Might one design a cfa explicitly for charge binning?

Alt 1: horizontally/vertically binnable green channel, 2:12:2 r:g:b ratio

r g r g

g g g g

b g b g

g g g g

Alt 2: Supersampled Bayer. Easily binnable to a Bayer equivalent format, but can its full

resolution sensels be used for higher resolution?

r r g g

r r g g

g g b b

g g b b
 
Gotcha. But the objective is not increased spectral sampling density, it's more accurate color n'est pas?

Jim
Surely, sampling the spectrum with a Nyquist-compliant 5nm sampling density would be sufficient for

»accurate color» (or as accurate as we can do without doing measurements outside of the main sensor and/or manual adjustements).

Not saying that it is an economic approach, nor that it is posible while keeping a reasonable SNR or spatial resolution.

-h
 

Keyboard shortcuts

Back
Top