Regarding The Creation Of Real Color Infrared Using Digital Processing

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter; simply take identical photos with the same perspective and framing, one with each filter on the camera, of a non-moving scene preferably with non-moving subjects; then stack each photo using the appropriate method. There can be some alternative methods to use for such a technique though, and this is where the question is relevant. Instead of using the Infrared channel for the Red, and visible into Blue & Green, what happens when the Infrared into Red and Green channel with Visible image becoming associative with the Blue channel? Alternatively, you might be able to get some interesting result by assigning the Infrared image, along with the Red channel from the (Hot Mirror) image, both into the Red channel, while having the Blue & Green from the Hot Mirror image be blue. and so on, there are so many different alternatives to creating false colors without actually playing around with the various color sliders, such as in photoshop (especially color mixing adjustments), but instead, associating various channels into other channels, etc. I've found this fact as really crazy myself (not necessarily in a negative meaning though, but rather considering how many options there are). I've done some such methods before, but have not gotten too many chances to try more combinations now, so was wondering if anyone has ever done so?

If you've run out of experiments yet (just kidding) you can also convert your image to CMYK space and split & mix those channels. And then try Lab space. So in the end you can have 10 channels to play around with: RGBCMYKLab. I personally haven't found much use for that, but I think it's a good way of understanding what each of the modes and channels do. Lee Varis has produced some very interesting videos under the rubric "10 Channel Workflow" that shows a number of ways to use the information in the channels to enhance images.

Some people will tell you that if you convert an image between modes too many times, it will degrade. That may be technically true, but if you're working in (at least) 16 bits I think it's a very minor concern.

Best wishes,
Sterling
--
Lens Grit

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter

Multispectral imaging is a well-developed field, mostly targeting things like identifying tanks hiding in a forest or measuring the health of crops in fields. There is no reason to restrict imaging to just 3 or 4 wavelength bands and there is tons of literature about choice of color mappings, with several well-accepted standards for different multispectral applications. However, you're talking about doing this for art's sake, so anything goes. 

It also can be feasible to extract more bands by solving for band contributions using wider-spectrum filters with sufficiently "bumpy" curves -- we used a genetic algorithm solver in our 2018 work: Multispectral, high dynamic range, time domain continuous imaging . In particular, look at Figure 4: that's really easy to do with the right choice of camera...

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter; simply take identical photos with the same perspective and framing, one with each filter on the camera, of a non-moving scene preferably with non-moving subjects;

My filter wheel moves pretty quickly.

then stack each photo using the appropriate method. There can be some alternative methods to use for such a technique though, and this is where the question is relevant. Instead of using the Infrared channel for the Red, and visible into Blue & Green, what happens when the Infrared into Red and Green channel with Visible image becoming associative with the Blue channel? Alternatively, you might be able to get some interesting result by assigning the Infrared image, along with the Red channel from the (Hot Mirror) image, both into the Red channel, while having the Blue & Green from the Hot Mirror image be blue.

One of my favorite combinations is emulating Kodak "Aerochrome" or EIR color infrared slide film. EIR was a three-layer slide film, with the conventional dye couplers, but unconventional chemistry. The layer that printed as red was sensitive to infrared, the layer that printed as green was sensitive to visible red, and the layer that printed as blue was sensitive to green. Nothing was sensitive to blue. It's a simple map [IRG] -> [RBG]

I have a nice set of IR band-pass filters: I can’t remember the exact wavelengths but they centered around 900, 800, and 700nm, so I could also map [900,800,700] -> [RGB].

I also have a UV filter and sometimes do the opposite of the Aerochrome map: [GBU]->[RGB] or do a variety of band-compression maps, like [IVU] -> [RGB].

I also enjoy playing "brain games" with color. You've got two relatively independent processing channels in the brain, each of which processes rod LMS (long, medium, short) signals into RGB. You can feed different signals into each eye and instead of seeing a couple million colors, you can now see over a trillion. You can use entirely visible maps, like various combinations of ROYGBV (or even ROYGBIV) or throw in one or more I or U channels.

and so on, there are so many different alternatives to creating false colors without actually playing around with the various color sliders, such as in photoshop (especially color mixing adjustments), but instead, associating various channels into other channels, etc. I've found this fact as really crazy myself (not necessarily in a negative meaning though, but rather considering how many options there are). I've done some such methods before, but have not gotten too many chances to try more combinations now, so was wondering if anyone has ever done so?

More than you could imagine, and often in real-time. I did a lot of work based loosely on Dan Angel or Berns and Taplin.

ProfHankD wrote:

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter

Multispectral imaging is a well-developed field, mostly targeting things like identifying tanks hiding in a forest or measuring the health of crops in fields. There is no reason to restrict imaging to just 3 or 4 wavelength bands and there is tons of literature about choice of color mappings, with several well-accepted standards for different multispectral applications. However, you're talking about doing this for art's sake, so anything goes.

It also can be feasible to extract more bands by solving for band contributions using wider-spectrum filters with sufficiently "bumpy" curves -- we used a genetic algorithm solver in our 2018 work: Multispectral, high dynamic range, time domain continuous imaging . In particular, look at Figure 4: that's really easy to do with the right choice of camera...

Fascinating.

Glad someone is still dabbling with this stuff. Many years ago, I was inspired by the work of Roy Berns and Larry Taplin over at RIT, starting with this paper: https://scholarworks.rit.edu/article/885/

Dan Angel was a research fellow at Ford, working in optoelectronics, basically the yang to my speech-recognition research yin. One of his pet projects was running three monochrome cameras off beam splitters, and building extremely esoteric filter packs to try to achieve an accurate colorimetric response. Errors in observer metamerism being the bugaboo multiple-domain color matching, an M-class (stakes in the millions of dollars) problem in the auto industry.

I did several iterations of dual RGB cameras with a simpler beam splitter, which really increased the range of lenses one could use. The very first of my RGBR'G'B' cameras beat Angel's best three-filter camera.

I've also done multi spectral stuff with one of my Trinoc (three cameras in an equilateral triangle) playthings, wrote some algorithms for fusing the spatially diverse information with color diverse, exposure diverse, and focal-plane diverse info.

ProfHankD wrote:

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter

I have an issue with calling these spectral bands 'true color' or 'real color'. Color has no meaning for infra red as our eye's/brains can't see it. Hank's reply below avoids this trap.

Multispectral imaging is a well-developed field, mostly targeting things like identifying tanks hiding in a forest or measuring the health of crops in fields. There is no reason to restrict imaging to just 3 or 4 wavelength bands and there is tons of literature about choice of color mappings, with several well-accepted standards for different multispectral applications. However, you're talking about doing this for art's sake, so anything goes.

It also can be feasible to extract more bands by solving for band contributions using wider-spectrum filters with sufficiently "bumpy" curves -- we used a genetic algorithm solver in our 2018 work: Multispectral, high dynamic range, time domain continuous imaging . In particular, look at Figure 4: that's really easy to do with the right choice of camera...

I have a filter that only transmits about 2nm either side of it's peak wavelength. With a complete set of similar narrowband NIR filters you could end up with over 150 wavelength bands, not very practical & extremely expensive. I suspect a pass range of +/- 50nm filters would be more workable, but even these work out far too expensive for me to play with.

For scientific imaging the wavelengths chosen will be selected to specifically show a feature of interest. This will often be done by controlling the lighting.

Joseph S Wisniewski wrote:

ProfHankD wrote:

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter

Multispectral imaging is a well-developed field, mostly targeting things like identifying tanks hiding in a forest or measuring the health of crops in fields. There is no reason to restrict imaging to just 3 or 4 wavelength bands and there is tons of literature about choice of color mappings, with several well-accepted standards for different multispectral applications. However, you're talking about doing this for art's sake, so anything goes.

It also can be feasible to extract more bands by solving for band contributions using wider-spectrum filters with sufficiently "bumpy" curves -- we used a genetic algorithm solver in our 2018 work: Multispectral, high dynamic range, time domain continuous imaging . In particular, look at Figure 4: that's really easy to do with the right choice of camera...

Fascinating.

Glad someone is still dabbling with this stuff. Many years ago, I was inspired by the work of Roy Berns and Larry Taplin over at RIT, starting with this paper: https://scholarworks.rit.edu/article/885/

Interesting. I don't recall seeing this before, but their results seem consistent with what I've seen experimentally.

Dan Angel was a research fellow at Ford, working in optoelectronics, basically the yang to my speech-recognition research yin. One of his pet projects was running three monochrome cameras off beam splitters, and building extremely esoteric filter packs to try to achieve an accurate colorimetric response. Errors in observer metamerism being the bugaboo multiple-domain color matching, an M-class (stakes in the millions of dollars) problem in the auto industry.

I'm familiar with the paint-matching-plastic pains of the auto industry. Toyota has a major facility in KY and a close relationship with the University of Kentucky, and there's this now-outdated thing: Painting Technology Consortium. Prof. Saito has continued to do a lot of "watching paint dry" for Toyota. 

I personally have tried to avoid "color science," using multispectral mainly for removing unwanted artifacts (e.g., PF) and working with Ag Eng folks on precision agriculture. I have a lot of respect for anyone who can get the color science to work well enough to make automotive folks happy; in my experience, only the digital cinematography folks are pickier...

I did several iterations of dual RGB cameras with a simpler beam splitter, which really increased the range of lenses one could use. The very first of my RGBR'G'B' cameras beat Angel's best three-filter camera.

I've also done multi spectral stuff with one of my Trinoc (three cameras in an equilateral triangle) playthings, wrote some algorithms for fusing the spatially diverse information with color diverse, exposure diverse, and focal-plane diverse info.

Very nice.

It sounds like we're in agreement that CFA cameras really can be pretty effective tools for this sort of thing, although the usual industry wisdom tends to favor monochrome sensors and filter wheels. BTW, I have a set of very expensive notch filters from a former colleague here who played with that approach, but I've never found them to be particularly worthwhile compared to things as simple as theatrical gel filters.

petrochemist wrote:

I have a filter that only transmits about 2nm either side of it's peak wavelength. With a complete set of similar narrowband NIR filters you could end up with over 150 wavelength bands, not very practical & extremely expensive. I suspect a pass range of +/- 50nm filters would be more workable, but even these work out far too expensive for me to play with.

I inherited a set of very expensive 10nm notch filters from a colleague... I find them mostly annoying. They also happen to be tiny, intended for C-mount industrial cameras.

On the other hand, I love LEDs! It's quite cheap to buy LEDs that have nearly all their energy output in a 10nm band, with lots of wavelength choices. If you want to measure reflectance wavelengths really cheaply, using LED lighting works well... AND it also sort-of works in reverse: using a LED as a point sensor gives a very narrow sensitivity too. The wavelength range LEDs detect is slightly shifted from what they emit, but it's similar. Incidentally, my brother, Paul Dietz, wrote THE paper on use of LEDs as detectors:

Very Low-Cost Sensing and Communication Using Bidirectional LEDs

ProfHankD wrote:

petrochemist wrote:

I have a filter that only transmits about 2nm either side of it's peak wavelength. With a complete set of similar narrowband NIR filters you could end up with over 150 wavelength bands, not very practical & extremely expensive. I suspect a pass range of +/- 50nm filters would be more workable, but even these work out far too expensive for me to play with.

I inherited a set of very expensive 10nm notch filters from a colleague... I find them mostly annoying. They also happen to be tiny, intended for C-mount industrial cameras.

On the other hand, I love LEDs! It's quite cheap to buy LEDs that have nearly all their energy output in a 10nm band, with lots of wavelength choices. If you want to measure reflectance wavelengths really cheaply, using LED lighting works well... AND it also sort-of works in reverse: using a LED as a point sensor gives a very narrow sensitivity too. The wavelength range LEDs detect is slightly shifted from what they emit, but it's similar. Incidentally, my brother, Paul Dietz, wrote THE paper on use of LEDs as detectors:

Very Low-Cost Sensing and Communication Using Bidirectional LEDs

I don't understand everything that you and others are saying, but it does seem intriguing. I'm aware though that a narrowband filter can be used to improve images in some sense. For example, in astronomy, narrowband filters can be used to show things of interest more prominently by "selecting" specific wavelength ranges. Also, in astrophotography, merging images made by color filters can be used to make a better-focused image, possibly (or at least for a monochrome camera to make for a color image after post processing). But as has been mentioned, this kind of "playing around" can easily become expensive, which is a main reason why I stopped investing in some ideas that I had. My favorite type of images using Infrared are when the normal image (or, alternatively, the red channel alone) is merged appropriately with a pure IR image, using the luminosity layer style in Photoshop, to be able to preserve luminosity from the IR data. There are still methods that I've yet to try using my full-spectrum camera, such as UV-Reflectance. Some of the methods of merging IR with visible, UV with IR, and others are methods that I don't see people doing, maybe mainly because it might require two or more images to stack. I might've already mentioned this, but I've found Infrared Fluorescence as intriguing to myself. It might be interesting to use narrowband filters for the imaging of fluorescence, whether it be UVIVF or IR fluorescence (of course, induced by a shorter wavelength). I've visited Ultravioletphotography.com to see what people were saying; I've learned from there that combining images from narrowband filters can be interesting, but more success can be obtained (more differences can become visible) if each narrow band being photographed is relatively far from the other in the spectral graph.

Joseph S Wisniewski wrote:

MacM545 wrote:

As you might not know, it's possible to create a true color-infrared image by using a full spectrum camera, along with an external hot mirror and Pure-Infrared filter; simply take identical photos with the same perspective and framing, one with each filter on the camera, of a non-moving scene preferably with non-moving subjects;

My filter wheel moves pretty quickly.

then stack each photo using the appropriate method. There can be some alternative methods to use for such a technique though, and this is where the question is relevant. Instead of using the Infrared channel for the Red, and visible into Blue & Green, what happens when the Infrared into Red and Green channel with Visible image becoming associative with the Blue channel? Alternatively, you might be able to get some interesting result by assigning the Infrared image, along with the Red channel from the (Hot Mirror) image, both into the Red channel, while having the Blue & Green from the Hot Mirror image be blue.

One of my favorite combinations is emulating Kodak "Aerochrome" or EIR color infrared slide film. EIR was a three-layer slide film, with the conventional dye couplers, but unconventional chemistry. The layer that printed as red was sensitive to infrared, the layer that printed as green was sensitive to visible red, and the layer that printed as blue was sensitive to green. Nothing was sensitive to blue. It's a simple map [IRG] -> [RBG]

I have a nice set of IR band-pass filters: I can’t remember the exact wavelengths but they centered around 900, 800, and 700nm, so I could also map [900,800,700] -> [RGB].

Are they overlapping in the sense of the spectrum which they pass? If or if not, does it make for any images that are of interest artistically and/or scientifically?

I also have a UV filter and sometimes do the opposite of the Aerochrome map: [GBU]->[RGB] or do a variety of band-compression maps, like [IVU] -> [RGB].

Seems like it could make for interesting images. If you want, I'd like to have an idea on what they look like. Are you into photographing landscapes with such technique, or something else?

I also enjoy playing "brain games" with color. You've got two relatively independent processing channels in the brain, each of which processes rod LMS (long, medium, short) signals into RGB. You can feed different signals into each eye and instead of seeing a couple million colors, you can now see over a trillion. You can use entirely visible maps, like various combinations of ROYGBV (or even ROYGBIV) or throw in one or more I or U channels.

Yes, I know at least some of this and it is neat stuff for sure.

and so on, there are so many different alternatives to creating false colors without actually playing around with the various color sliders, such as in photoshop (especially color mixing adjustments), but instead, associating various channels into other channels, etc. I've found this fact as really crazy myself (not necessarily in a negative meaning though, but rather considering how many options there are). I've done some such methods before, but have not gotten too many chances to try more combinations now, so was wondering if anyone has ever done so?

More than you could imagine, and often in real-time. I did a lot of work based loosely on Dan Angel or Berns and Taplin.

Joseph S Wisniewski wrote:

EIR was a three-layer slide film, with the conventional dye couplers, but unconventional chemistry. The layer that printed as red was sensitive to infrared, the layer that printed as green was sensitive to visible red, and the layer that printed as blue was sensitive to green. Nothing was sensitive to blue.

Not quite my understanding; everything was sensitive to blue in Kodak Infrared Ektachrome and Aerochrome, and you needed a Wratten #12 (yellow) filter to block blue light.

It's a simple map [IRG] -> [RBG]

But not really, unless you account for blue light polluting all three channels.

The most promising technique I've seen is to block the blue with the proper filter. Then, in theory, the camera's blue channel contains only IR. This channel then becomes a term in modifying the other two channels to remove their IR sensitivity, as re-discovered by JD Wong on Flickr :

Assuming the blue channel contains only IR, you can subtract that channel from both the green and red channels, then re-map IR -> R, R -> G, and G -> B.

Of all the so-called KIE emulations I've seen, Wong's appear to be the closest. I'm waiting on my conversion to get back so I can try it.

I've wondered if it might be possible to make a filter wheel for the Sony RX100 2, or at least a filter slide (rectangular instead of wheel). But unfortunately, I might need to invest into an interchangeable camera mod because the Sony has rather severe of a hot spot issue unless the given aperture is larger than about F/2.8, at which point, loss of sharpness is a problem, especially for macro/closeup.

Jan Steinman wrote:

Assuming the blue channel contains only IR, you can subtract that channel from both the green and red channels, then re-map IR -> R, R -> G, and G -> B.

I downloaded a full spectrum image, filtered with a Tiffen #12 yellow filter, for experimenting with this.

I used ImageMagick, an open-source box of command-line image processing tools.

Based on advice I got on photo.stackexchange.com, I tried to do the transform I sketched out above. It only partially works, so I must be doing something wrong.

This is what I tried. It resulted in classic magenta foliage, but a red car and red bricks did not get transformed to green, as I had expected. And the sky was still cyan, which should have been blocked. So there may be a problem with the sample image I downloaded. It will have to wait for my converted camera to get here.

# magick YellowFilter.webp -color-matrix '0 0 1 0 0 0\

1 0 -1 0 0 0\

0 1 -1 0 0 0 ' YellowFilter-conv.jpg

Note that the command must be on three separate lines. The first three columns are the original RGB channels; the three rows represent the output RGB channels. Each cell can have a number between -1 and 1, which will multiply that channel, sum it with any other non-zero channels, and then put it into the corresponding output channel.

Columns four and five are unused. Column six is an "offset" that can be used to keep output from going negative. It did not seem to have much effect. I also found that the factors shown in my original diagram didn't have very much effect. This is the matrix I used, using the numbers from the original diagram (formatted picture for clarity):

You can download and install ImageMagick for Mac, Linux, and Windows.

Any help from ImageMagick gurus would be appreciated!

I'm not surprised your transition didn't simulate aerochrome completely. On a bayer based full spectrum camera infrared is recorded in all three channels, above about 850nm these generally show fairly similar intensities. There are noticeable variations around 700nm, where the red channel sees significantly more, and a relatively small band where blue is increased - with the right WB a 720nm filter can give a blue tinge for some IR typically seen with died hair!

Aerochomes handling of infra red is very different, which is IMO better mimicked by Foveon sensors, that have been modified to capture IR. Actually a very easy procedure with many of the Sigma Foveon cameras where the hot mirror is on the user removable dust filter rather than on the sensor itself. - These will typically show IR strongly in the red channel, but I think they'll still differ from aerochrome in their handling of visual red colours.

This example https://flic.kr/p/2fe3NhU shows IR as red, but red is also seen as red not switched to green...

petrochemist wrote:

I'm not surprised your transition didn't simulate aerochrome completely. On a bayer based full spectrum camera infrared is recorded in all three channels…

The technique I think I'm using accounts for that.

It uses a "blue blocking" yellow filter, typically Tiffen/Wratten #12. Thus, the blue channel should only contain IR.

That is then subtracted from the red and green channels, leaving them with only red and green.

Then, the channels are swapped: blue —> red, red —> green, green —> blue.

I tested on a #12-filtered image that I downloaded from a flickr discussion on the technique, so I'm not sure about its quality. Here's the original, followed by my run through ImageMagick:

Foliage came out nice and magenta, but red in the car and houses did not come out green. So I'm unsure the technique is doing what I think it should be doing.

… above about 850nm these generally show fairly similar intensities. There are noticeable variations around 700nm, where the red channel sees significantly more…

I think this is fairly typical sensitivities of sensors, Bayer filters, and hot mirrors:

So, it would seem that the IR in the filtered blue channel (where the blue curve crosses the gray curve on the right) is only some 40% or so, not too different from the IR in the green channel. But the IR in the red channel is more like twice as much as the other two channel, so perhaps tuning the parameters is all that is needed.

But I'm going to wait for my converted camera to come back, and take my own controlled test photos to work with.

Aerochomes handling of infra red is very different, which is IMO better mimicked by Foveon sensors…

This example https://flic.kr/p/2fe3NhU shows IR as red, but red is also seen as red not switched to green...

Yea, the tail lights are a give-away! I have seen Aerochrome emulations that show green tail lights. So, the Sigma Foveon is not doing such a good emulation job in this case.