Computational cost of Bayer interpolation

[Ron Parr]

A few months ago there were some wild arguments about the computational cost of Bayer interpolation. The following document from Texas Instruments describes how to use a TI DSP for the digital image processing in a camera with a Bayer pattern sensor:

http://www-s.ti.com/sc/psheets/spra651/spra651.pdf

In table 1, they give a breakdown of the processing cycles required for each of the tasks. 27% of the total cycles required to process an image are used for interpolation. An additional 18% is used for, "Edge Enhancement and False Color Suppression." The text indicates that these are required as a consequence of the Bayer interpolation, so the total amount of processing time devoted to Bayer interpolation is 45%.

(Note that table 1 lumps together Bayer interpolation and interpolation to hide faulty pixels. The latter should not be a significant contribution to the total computation cost.)

Of course, this does not necessarily imply that removing Bayer interpolation from the image processing workflow will reduce cost by allowing us of a cheaper processor or improve camera speed by keeping the same processor. Depending upon the camera design, the cost savings may be an insignificant part of the total camera cost and other bottlenecks may limit camera speed.

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/

 

[Shay Stephens]

So for example, the Foveon may have processing steps that mosaic sensors do not, and the total time used to process an image may be close to equal?

I would hope otherwise since the Foveon info states:

"Foveon X3 image sensors will streamline the electronic designs of digital cameras. Today's digital cameras use mosaic color filters and require complex mathematical algorithms to estimate the red, green and blue value of pixel, since only one color per pixel is being truly measured. To accomplish this interpolation, dedicated processing hardware and software are required inside the camera. Color interpolation adds cost and complexity to a camera while increasing delays between clicking the shutter button and capturing a picture, which can result in the difference between capturing a great picture or missing an important moment. By capturing all three colors at every pixel, Foveon X3-based image sensors eliminate the need for costly, complex and error-prone color interpolation. Foveon X3 enables improved image quality, simplified camera designs and increased click-to-capture performance."

I know that is marketing stuff, but it would still lead me to believe there is a savings in the computational cost of the Foveon design. Even if only a fractional one.

Oh and thank you for the info by the way, good stuff

--
Shay

My Sony F707 Gallery: http://www.shaystephens.com/portfolio.asp

 

[Ron Parr]

So for example, the Foveon may have processing steps that mosaic
sensors do not, and the total time used to process an image may be
close to equal?

I didn't mean to imply that. It's just that for a digital SLR, the cost of the DSP may not be a significant part of the cost of the device. Also, file write times to compact flash may be the bottleneck for speed, not image processing.

I know that is marketing stuff, but it would still lead me to
believe there is a savings in the computational cost of the Foveon
design. Even if only a fractional one.

Well, I think it's about 45%, judging from those DSP cycle counts. The only question is whether saving 45% there really helps you much in practice. For expensive cameras, my guess is that it doesn't. For a low end design, it might let them use cheaper or fewer parts which is always a plus in that market segment.

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/

 

[Erik Magnuson]

So for example, the Foveon may have processing steps that mosaic
sensors do not, and the total time used to process an image may be
close to equal?

It may or may not have more processing steps. However, as the Foveon advocates are fond of pointing out, it has over 10 million photosites (or whatever) as opposed to 6 million for a roughly equiv bayer. That means that it has 66% more samples to process. If we assume that the other 55% of the processing is identical for both sensors, and that the amount of processing is linear with the number of samples, that brings us back up to 92% of the bayer processing.

Of course not all processing is created equal and there are too many other assumptions and gaps in our knowledge to make any definitive statements. Heck, we don't even know the actual frame rate of the SD-9. The Sigma specs say "up to 3fps" but don't say at what resolution. The F7 sensor specs say max 2fps at full resolution. I guess we'll know for sure in 2-3 weeks.

--
Erik

 

[Shay Stephens]

Ok, sorry, I am on the same page now makes sense.

So for example, the Foveon may have processing steps that mosaic
sensors do not, and the total time used to process an image may be
close to equal?

I didn't mean to imply that. It's just that for a digital SLR, the
cost of the DSP may not be a significant part of the cost of the
device. Also, file write times to compact flash may be the
bottleneck for speed, not image processing.

I know that is marketing stuff, but it would still lead me to
believe there is a savings in the computational cost of the Foveon
design. Even if only a fractional one.

Well, I think it's about 45%, judging from those DSP cycle counts.
The only question is whether saving 45% there really helps you much
in practice. For expensive cameras, my guess is that it doesn't.
For a low end design, it might let them use cheaper or fewer parts
which is always a plus in that market segment.

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/

--
Shay

My Sony F707 Gallery: http://www.shaystephens.com/portfolio.asp

 

[Jimmy Chen]

Also, file write times to compact flash may be the
bottleneck for speed, not image processing.

That is generally the case with computer devices. Speed degrades as you move from CPU buffer to internal RAM then to storage devices.

--
jc
Sony F707
http://www.reefkeepers.org/gallery/f707
http://www.reeftec.com/gallery

 

[Ron Parr]

It may or may not have more processing steps. However, as the
Foveon advocates are fond of pointing out, it has over 10 million
photosites (or whatever) as opposed to 6 million for a roughly
equiv bayer. That means that it has 66% more samples to process. If
we assume that the other 55% of the processing is identical for
both sensors, and that the amount of processing is linear with the
number of samples, that brings us back up to 92% of the bayer
processing.

The only thing that would hurt you is the preprocessing, since the rest of the pipeline is applied to the full RGB image. Assuming you need to do 3X as much preprocessing is an extremely pessimistic assumption b/c with the Bayer interpolation step removed, white balance can be folded in to the color conversion step. (If you postmultiply your change of basis matrices by the white balance matrix, you get your white balance conversion for free.) Also note that there is no lens distortion compensation with cameras other than webcams.

Of course not all processing is created equal and there are too
many other assumptions and gaps in our knowledge to make any
definitive statements. Heck, we don't even know the actual frame
rate of the SD-9. The Sigma specs say "up to 3fps" but don't say at
what resolution. The F7 sensor specs say max 2fps at full
resolution. I guess we'll know for sure in 2-3 weeks.

The Chasseur D'Images guys said 2FPS for their preproduction unit.

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/

 

[Shay Stephens]

For the record, there are:
"10.29Million (2268 Columns x 1512 Rows x 3 Layers) Color Photo Detectors"
According to the sigma web page.

Now if we assume that of the 152 cycles per pixel for mosaic sensor can be simplified by removing the interpolation step and the 4:1:1 decimation & jpeg encoding for the Foveon we come up with a total of 59 cycles per pixel assuming again that the preprocessing, color space conversion, and edge enhancement and false color suppression remain the same. That is 39% of the mosaic load not 55%. So with 10.29 million pixels for the Foveon and 6 million pixels for the mosaic, the Foveon still would come in at 67% the number of cycles of the mosaic (607,110,000 foveon cycles per image : 912,000,000 mosic cycles per image).

I know we are dueling with assumptions here hehehe so nothing can be defended at this point but it does raise some interesting possibilities.

--
Shay

My Sony F707 Gallery: http://www.shaystephens.com/portfolio.asp

 

[Erik Magnuson]

Assuming
you need to do 3X as much preprocessing is an extremely pessimistic
assumption b/c with the Bayer interpolation step removed,

Yes, it was pessimistic. Sometimes "back of the envelope" calculations are only good for computing postage. However, my point has been that like in most systems, the performance bottleneck is more likely to be more reading/writing the data where the X3 approach is at a slight disadvantage. The issue all along has been Foveon's pointing to a minor benefit ("look ma, no interpolation HW needed") while ignoring the man behind the curtain busily try to shovel all that data out. Marketing slides rarely list the disadvantages or limitations of the technology they are trying to sell.

It would be interesting to know exactly what's going on inside X3 processing though.

--
Erik

 

[Karl Guttag]

While the table may give you some idea of the relative number of computations, it would be a mistake to read too much into an AP note. I know at least on of the people who wrote the article and he is in TI's R&D labs and not a consumer product group. This looks like an AP note that is fishing for a market as opposed something that is about to be put into a real product.

I seriously doubt anyone is using a fully programmable DSP like the 54xxxx for doing Bayer interpolation. It would be cheaper to use an asic directly aimed at Bayer conversion.

I don't expect to see programmable DSP's in Cameras until they get wireless connect. Eventually I expect that we will have cameras that automatically transfer picture wirelessly through the internet.

Karl

A few months ago there were some wild arguments about the
computational cost of Bayer interpolation. The following document
from Texas Instruments describes how to use a TI DSP for the
digital image processing in a camera with a Bayer pattern sensor:

http://www-s.ti.com/sc/psheets/spra651/spra651.pdf

In table 1, they give a breakdown of the processing cycles required
for each of the tasks. 27% of the total cycles required to process
an image are used for interpolation. An additional 18% is used
for, "Edge Enhancement and False Color Suppression." The text
indicates that these are required as a consequence of the Bayer
interpolation, so the total amount of processing time devoted to
Bayer interpolation is 45%.

(Note that table 1 lumps together Bayer interpolation and
interpolation to hide faulty pixels. The latter should not be a
significant contribution to the total computation cost.)

Of course, this does not necessarily imply that removing Bayer
interpolation from the image processing workflow will reduce cost
by allowing us of a cheaper processor or improve camera speed by
keeping the same processor. Depending upon the camera design, the
cost savings may be an insignificant part of the total camera cost
and other bottlenecks may limit camera speed.

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/

--
Karl

 

[Erik Magnuson]

Now if we assume that of the 152 cycles per pixel for mosaic sensor
can be simplified by removing the interpolation step and the 4:1:1
decimation & jpeg encoding for the Foveon

Of course the problem with that is that you can eliminate the same steps for the Bayer system (on board) and just record the RAW data too. And you still have 33% less data to read/write ;-) It's tough to claim that no onboard JPEG is a feature, but that doesn't stop Foveon/Sigma from trying!

--
Erik

 

[Ron Parr]

While the table may give you some idea of the relative number of
computations, it would be a mistake to read too much into an AP
note. I know at least on of the people who wrote the article and
he is in TI's R&D labs and not a consumer product group. This
looks like an AP note that is fishing for a market as opposed
something that is about to be put into a real product.

I seriously doubt anyone is using a fully programmable DSP like the
54xxxx for doing Bayer interpolation. It would be cheaper to use
an asic directly aimed at Bayer conversion.

I don't expect to see programmable DSP's in Cameras until they get
wireless connect. Eventually I expect that we will have cameras
that automatically transfer picture wirelessly through the internet.

TI is a major supplier of fully programmable DSPs for digital cameras:

http://www.apogeephoto.com/sept2001/digicams092001.shtml

http://www.ti.com/sc/docs/news/2000/00153.htm

http://www.dpreview.com/news/0007/00071703ti_dsp.asp

--
Ron Parr
FAQ: http://www.cs.duke.edu/~parr/photography/faq.html
Gallery: http://www.pbase.com/parr/