Resolution, aliasing and light loss - why we love Bryce Bayer's baby anyway
It's unlikely Kodak's Bryce Bayer had any idea that, 40 years after patenting a 'Color Imaging Array,' his design would underpin nearly all contemporary photography and live in the pockets of countless millions of people around the world.
|It seems so obvious, once someone else has thought of it, but capturing red, green and blue information as an interspersed, mosaic-style array was breakthrough.
Image: based on original by Colin M.L Burnett
The Bayer Color Filter Array is a genuinely brilliant piece of design: it's a highly effective way of capturing color information from silicon sensors that can't inherently distinguish color. Most importantly, it does a good job of achieving this color capture while still capturing a good level of spatial resolution.
However, it isn't entirely without its drawbacks: It doesn't capture nearly as much color resolution as a camera's pixel count seems to imply, it's especially prone to sampling artifacts and it throws away a lot of light. So how bad are these problems and why don't they stop us using it?
There's a limit to how much resolution you can capture with any pixel-based sensor. Sampling theory dictates that a system can only perfectly reproduce signals at half the sampling frequency (a limit known as the Nyquist Frequency). If you think about trying to represent a single pixel-width black line, you need at least two pixels to be sure of representing it properly: one to capture the line and another to capture the not-line.
Just to make things more tricky, this assumes your pixels are aligned perfectly with the line. If they're slightly misaligned, you may get two grey pixels instead. This is taking into consideration by the Kell factor, which says that you'll actually only reliably capture resolution around 0.7x your Nyquist frequency.
But, of course, a Bayer sensor doesn't sample all the way to its maximum frequency because you're only sampling single colors at each pixel, then deriving the other color values from neighboring pixels. This lowers resolution (effectively slightly blurring the image).
So, with these two factors (the limitations of sampling and Bayer's lower sampling rate) in mind, how much resolution should you expect from a Bayer sensor? Since human vision is most sensitive to green information, it's the green part of a Bayer sensor that's used to provide most of the spatial resolution. Let's have a look at how it compares to sampling luminance information at every pixel.
|Counter-intuitive though it may sound, the green channel captures just as much horizontal and vertical detail as the sensor capturing data at every pixel. Where it loses out is on the diagonals, which sample at 1/2 the frequency.|
Looking at just the green component, you should see that a Bayer sensor can still capture the same horizontal and vertical green (and luminance) information as a sensor sampling every pixel. You lose something on the diagonals, but you still get a good level of detail capture. This is a key aspect of what makes Bayer so effective.*
It's a less good story when we look at the red and blue channels. Their sampling resolution is much lower than the luminance detail captured by the green channel. It's worth bearing in mind that human vision is much more sensitive to luminance resolution than it is to color information, so viewers are likely to be more tolerant of this shortcoming.
So what happens to everything above the Nyquist frequency? Well, unless you do something to stop it, your camera will try to capture this information, then present it in a way it can represent. A process called aliasing.
Think about photographing a diagonal black stripe with a low resolution camera. Even with a black and white camera, you risk the diagonal being represented as a series of stair steps: a low-frequency pattern that acts as an 'alias' for the real pattern.
The same thing happens with fine repeating patterns that are a higher frequency than your sensor can cope with: they appear as spurious aliases of the real pattern. These spurious patterns are known as moiré. This isn't unique to Bayer, though, it's a side-effect of trying to capture higher frequencies than your sampling can cope with. It will occur on all sensors that use a repeating pattern of pixels to capture a scene.
Sensors that use the Bayer pattern are especially prone to aliasing though, because the red and blue channels are being sampled at much lower frequencies than the full pixel count. This means there are two Nyquist frequencies (a green/luminance limit and a red/blue limit) and two types of aliasing you'll tend to encounter: errors in detail too fine for the sensor to correctly capture the pattern of and errors in (much less fine) detail that the camera can't correctly assess the color of.
To reduce this first kind of error most cameras have, historically, included Optical Low Pass Filters, also known as Anti-Aliasing filters. These are filters mounted in front of the sensor that intentionally blur light across nearby pixels, so that the sensor doesn't ever 'see' the very high frequencies that it can't correctly render, and doesn't then misrepresent them as aliasing.**
|The point at the center of the Siemens star is too fine for this monochrome camera to represent, so it's produced a spurious diamond-shaped 'alias' at the center instead.||This image second was shot with a very high resolution camera, blurred to remove high frequencies, then downsized to the same resolution as the first shot. It still can't accurately represent the star, but doesn't alias when failing.|
These aren't so strong as to completely prevent all types of aliasing (very few people would be happy with a filter that blurred the resolution down to 1/4 of the pixel height: the Nyquist frequency of red and blue capture), instead they blur the light just enough to avoid harsh stair-stepping and reduce the severity of the false color on high-contrast edges.
|With a Bayer filter, you get a fun color component to this aliasing. Not only has the camera tried to capture finer detail than its sensor can manage, you get to see the side-effect of the different resolutions the camera captures each color with.||Again, if you compare this with a significantly over-sampled image, blurred then downsized, you don't see this problem. However, look closely you can still see traces of the false color that occurred at the much higher frequency this camera was shooting at.|
This means that, a camera with an anti-aliasing filter, you shouldn't see as much false color in the high-contrast mono targets within our test scene, but it'll do nothing to prevent spurious (aliased) patterns in the color resolution targets.
At the silicon level, modern sensors are pretty amazing. Most of them operate at an efficiency (the proportion of light energy converted into electrons) around 50-80%. This means there's less than 1EV of performance improvement to be had in that respect, because you can't double the performance of something that's already over 50% effective. However, before the light can get to the sensor, the Bayer design throws away around 1EV of light, because each pixel has a filter in front of it, blocking out the colors it's not meant to be measuring.
around 1EV of light'
This is why Leica's 'Monochrom' models, which don't include a color filter array, are around one stop more sensitive than their color-aware sister models. (And, since they can't produce false color at high-contrast edges, they don't include anti aliasing filters, either).
It's this light loss component that may eventually spell the end of the Bayer pattern as we know it. For all its advantages, Bayer's long term dominance is probably most at risk if it gets in the way of improved low-light performance. This is why several manufacturers are looking for alternatives to the Bayer pattern that allow more light through to the sensor. It's telling, though, that most of these attempts are essentially variations on the Bayer theme, rather than total reinventions.
These variations aren't the only alternatives to the Bayer design, of course.
Sigma's Foveon technology uses a fundamentally different approach to measure multiple colors at the same location, so promises higher color resolution, no light loss to a color filter array and less aliasing. But, while these sensors are capable of producing very high pixel-level sharpness, this currently comes at an even greater noise cost (which limits both dynamic range and low light performance), as well as struggling to compete with the color reproduction accuracy that can be achieved using well-tuned colored filters. More recent versions reduce the color resolution of two of their channels, sacrificing some of their color resolution advantage for improved noise performance.
Meanwhile, Fujifilm has struck out on its own, with the X-Trans color filter pattern. This still uses red, green and blue filters but features a larger repeat unit: a pattern that repeats less frequently, to reduce the risk of it clashing with the frequency it's trying to capture. However, while the demosaicing of X-Trans by third-party software is improving, and the processing power needed to produce good-looking video looks like it's being resolved, there are still drawbacks to the design.
Ironically, devoting so much of the sensor to green/luminance capture appears to have the side-effect of reducing its ability to capture and represent foliage (perhaps because it lacks the red and blue information required to render the subtle tint of different greens).
Which leaves Bayer in a situation akin to Winston Churchill's take on Democracy as: 'the worst form of Government except all those other forms that have been tried from time to time.'
40 not out
As we've seen before, the sheer amount of effort being put into development and improvement of Bayer sensors and their demosaicing is helping them overcome the inherent disadvantages. Higher pixel counts keep pushing the level of color detail that can be resolved, despite the 1/2 green, 1/4 red, 1/4 blue capture ratio.
And, because the frequencies that risk aliasing relate to the sampling frequency, higher pixel count sensors are showing increasingly little aliasing. The likelihood of you encountering frequencies high enough to cause aliasing falls as your pixel count helps you resolve more and more detail.
Add to this the fact that lenses can't perfectly transmit all the detail that hits them, and you start to reach the point that the lens will effectively filter-out the very high frequencies that would otherwise induce aliasing. In recent years we've seen filter-less sensors starting at 36MP on full frame, 24MP on APS-C and Four Thirds sensors from 16MP (all of which are sampling their lenses at over 200 pixels per mm), and seen that these only produce significant moiré when paired with very sharp lenses shot wide-enough open that diffraction doesn't end up playing the anti-aliasing role.
So, despite the cost of light and of color resolution, and the risk of error, Bryce Bayer's design remains firmly at the heart of digital photography, more than 40 years after it was first patented.
|Intrepid View-072500 by vbuhay|
|Jazz Hands_ by Imagemi|
from Musical instruments
|Fire Urchin by sgitlin|
from Ricoh Challenge
Montana judge Dana L. Christensen has ruled the Republican National Committee did not infringe upon the copyright of photographer Erika Peterman after they took a photo from a Democratic candidate's Facebook page without permission and altered it to use in a derogatory promotional mailer.
Nikon has launched updates for three of its programs to address various bugs and glitches that could cause crashes and unwanted results.
LEE Filters has launched the LEE100, its next-generation filter holder that improves the design and looks in all the right places.
With the arrival of some much-needed sunshine and final production firmware for the Panasonic S1, we've been able to get outside and really start putting the camera through its paces.
Importing, culling and tagging photos is about to get a whole lot faster and look a whole lot better with the impending arrival of Photo Mechanic 6.
On its own, the FTZ adapter retails for $250 and when bundled it dropped the cost to just $150. Now, Nikon is offering it for free with all Z6, Z7 purchases in the United States.
Profoto said it spoke with Godox back at Photokina 2018 and continues to contact Godox in an effort to stop it from marketing its V1 light.
Product renders in Italian publication Notebook Italia show an unusual design that conceals all cameras with the help of a slider mechanism.
Canon says its new EF 400mm F2.8L IS III and EF 600mm F4L IS III lenses can suffer from an intermittent flickering when shooting video in M or Av modes with certain cameras.
Leica recently announced the Q2, a digital rangefinder with a fixed 28mm F1.7 lens. It's a heck of a lot of fun to shoot with, but is it right for you? Based on our time with the camera, and its specifications, we've examined how well-suited it is for common photography use-cases.
Now that our Panasonic Lumix S1R has final firmware, we couldn't wait to get out shooting with it - and we also tried the high-res mode, which combines files to get 187 megapixel images. Because sometimes, 47 megapixels just isn't enough.
In this article, travel and landscape photographer Mitch Green encourages us to spend more time in the the field.
the lens lacks any electronics whatsoever and is constructed entirely of glass and metal. Of course, that comes at the expense of weight — this thing weighs in at 1.1kg / 2.43lbs.
Drones can be useful tools in urban areas, where they're utilized for everything from news reporting to building inspections, but flying in these areas requires careful preparation. Here's what you need to know to do so safely.
Hasselblad has released a new cable release and USB double battery charger for its X1D medium format camera .
After a report published by NBC News, Flickr has taken heat for allegedly letting IBM 'scrape' photos for use in its facial recognition datasets. But the problem isn't what it seems on the surface.
Samyang has announced the impending arrival of the AF 85mm F1.4 FE lens for full-frame Sony cameras.
Some Photoshop shortcuts are simple and obvious. Others, not so much. Here are 15 shortcuts that are actually useful.
Twitter has redesigned its in-app camera for easier access from the timeline screen.
Independent cinema lens manufacturer SLR Magic has announced it will offer all of its existing MicroPrime range in the Fujifilm X mount and has even created a Fuji-specific 12mm lens.
We've updated our buying guides with three more cameras: the Canon EOS RP, Nikon Z6 and Olympus E-M1X.
CFexpress 2.0 cards will come in three different form factors, each of which will offer different maximum speeds.
Lensbaby has added a third tilt lens to its Optic Swap system, this time a 35mm lens, adding to the existing 50mm and 80mm options.
Sigma has released firmware updates for a number of its lenses as well as its EF-E adapter to address various errors and features with Canon, Nikon and Sony cameras.
We've added the Fujifilm X-T30 and Sony a6400 to our 'Best Cameras under $1000' buying guide. These two mirrorless models pack in a lot of features for just $900 body only.
Instagram, Facebook and other Facebook-owned services are down for users around the world.
Think Tank Photo has unveiled its new Vision series of shoulder bags, including the Vision 10, Vision 13 and Vision 15.
The OPPO Reno series will be launched on April 10 but some details have already been spotted on the web.
Insta360 has unveiled its latest camera, as well as a new VR headset app and a specialized smartphone cover that makes it possible to view 3D video on standard smartphones.
A fresh crop of ready-for-anything compacts has been added to our buying guide – just in time for Spring Break.