'Perfect' sensors may be possible, but might not come to cameras
|Fossum's team has created a prototype chip with a variety of pixel designs and readout methods. This included combinations with sufficiently low read noise to allow individual photons to be counted.|
The future could include sensors that perfectly describe the light in the scene, that offer new computational possibilities and give film-like latitude in the highlights. And yet we may not ever see them in cameras, says father of the CMOS sensor, Professor Eric Fossum.
We spoke to Fossum shortly after he received, alongside three other pioneers of digital photography, the Queen Elizabeth Prize for Engineering for his work on CMOS sensors. But the topic of our conversation is the future, rather than his past achievements. He now leads a group at the Thayer School of Engineering at Dartmouth, New Hampshire, working on what he calls Quanta Image Sensors (QIS). The team has recently published a paper announcing a breakthrough using the same fabrication process used to make CMOS image sensors.
The perfect sensor?
The principle is to use nanoscale, specialized pixels, called 'Jots' to capture light at the level of individual photons. They work in a binary fashion: they've either received a photon or they haven't (as opposed to conventional sensors which accumulate the charge generated by lots of photons during exposure). These jots are read repeatedly to see whether another photon has arrived since they were last checked.
While Fossum is keen to stress that other teams are having some success in the same field (using a slightly different approach), his own team's work is looking very promising. The paper in the journal Optica shows the team's technology has been refined such that a 1MJot chip can be read 1000 times per second while still exhibiting sufficiently low read noise that it can distinguish between individual photons.
We can count every photon: you can't do any better than that
"The Holy Grail is no read noise," says Fossum: "so that the read signal is proportional to the signal as it arrived." And the team's latest paper says they've got very close to this, with noise levels so low that the sensor can distinguish between individual photons without getting confused by read noise. This opens up the possibility of cameras that could perfectly describe the light in the scene, even in near total darkness.
Eliminating read noise from the sensor wouldn't mean totally noiseless photos, since the randomness of the light being captured is a key source of noise, but it's the best any sensor can possibly achieve. "We can count every photon: you can't do any better than that," he says.
The paper, perhaps conservatively, says the technology could be suited to scientific, space, security and low-light imaging applications, but Fossum has clearly also been thinking about conventional photography.
A classic response
"Because it's binary in nature, its response is comparable to old photographic film," he says. "In film, when the silver halide was hit by a photon, it's reduced to a silver atom that isn't washed away [during processing]. If it's hit by two photons, it doesn't make any additional difference."
This ends up meaning that in bright regions of the image there are ever fewer unexposed silver ions as the exposure goes on. This, in turn makes it less likely that the last few ions will be hit by a photon, so it becomes increasingly difficult to fully saturate the system. The same is true for the tiny, binary Jots: as more of them become saturated, it becomes increasingly difficult to saturate the last few.
"The response is linear at moderate exposure but it trails off to give significant overexposure latitude. It's a pattern first observed by Hurter and Driffield in 1890," says Fossum: "they showed the same curve that we measure, experimentally, in our QIS devices."
"That has obvious interest both for still photographers who're used to shooting film and for cinematographers who're looking for that kind of response."
The use of such tiny pixels has other benefits, too: "Jots are below diffraction limits in size. This means the resolution of the system is always higher than the resolution of the lens, which means we never have to worry about aliasing." While the group's prototype sensors feature one million Jots, Fossum says their target is one billion.
Beyond conventional photography
Fossum isn't just thinking about photographic history, though. The tiny size and the approach of repeatedly reading out the sensor challenges the existing concept of single exposures. "At the moment we make motion pictures by shooting a series of snapshots. With QIS it's more like the reverse process," he says: constructing still images from precisely captured movement.
|Professor Fossum has already been responsible for one revolution in photography: the invention of the CMOS sensor. In December 2017 he was awarded the Queen Elizabeth Prize for Engineering for his work.|
Essentially, taking lots of short, sub-frames during an exposure gives you an extra dimension to your images: time. "If you take a single frame, you get a bunch of ones and zeros. If you take another, you quickly build up a cube of ones and zeros," Fossum says: "For example, if you shoot 100 frames at 1000 frames per second, you get a cube that's x pixels wide by y pixels tall, but also 100 frames deep."
This presents some interesting questions, he says: "What do you do with that data? How do you create an image from that very faithful map of where photons arrived?"
"You could choose a number of pixels in x and y but also in the time axis. If you wanted a very sensitive pixel in low light you could combine 10 x 10 Jots in x and y and then maybe combine the data from 100 frames: it's essentially like increasing the grain size in a more sensitive film."
Of course you can achieve something comparable to this in conventional digital photography by downscaling an image, but Jots allow greater flexibility, Fossum says: "your pixel size could vary between different parts of the image, so in some places you'd have bigger but more sensitive grains."
What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was?
The time component also opens up additional possibilities, he says: "if an object moves during these hundred frames, instead of adding all the values from the same location, you could add them at an angle that corresponds to the movement," so that all the pixels relating to the same object are combined. "We could take out motion blur or remove the scanning effect of a computer screen in video."
The idea of combining multiple frames in interesting ways is, of course, already becoming a core part of mobile photography, and Fossum says finding all the things that are possible is a challenge he is leaving for others: "From my point of view, we're building a platform for computational imaging, it's for others to develop all the ways to use it. A camera would have to take account of the new sensor capabilities."
But it'll ask interesting questions, he believes: "What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was? Some of the things we associate with photography are artifacts of the way we capture them."
Not the only future
With all this going for it, it might seem odd that Fossum isn't promising to deliver a second revolution in digital imaging. But, having devoted a career to developing technologies and teaching about the challenges, he's realistic both about the work left to do and the competition any product would face.
"What we've already achieved is wonderful. The next challenge is adding color [awareness], but I don't think that's going to be particularly problematic. Then there's power: we've shown we can produce a large chip that doesn't consume or disperse a prohibitively large amount of power. We're currently at around 27mW but scale it up by 1000 [to get to one billion Jots] and that's 27W, so we need to cut that by about a factor of ten."
His concern is more about the current state of the rival technologies: "In order to bring a new technology to replace the existing one, it has to be compellingly better in a number of ways," he says. "For a few niches, [our technology] is already compelling." But for photography, the bar is already set very high.
I don't want our startup to be another esoteric imaging product that fails to find a market
"CMOS technology is pretty awesome right now," he says, before almost embarrassedly stressing that he's not claiming the credit for this: "where it is today is the result of the input from thousands of engineers from different companies who've contributed towards where we are now."
|Professor Eric Fossum pictured with Dr Jiaju Ma, one of the co-authors of the Optica paper and a co-founder of the spin-off company, Gigajot Technology.|
But, for all his cautious words, Fossum is convinced enough by the technology's potential to have created a company, Gigajot Technology, with his co-researchers. "Finding a sweet spot in the market is a really important part of challenge. It comes back to the things I teach: 'who is your customer?' 'what is your market?' 'how are we going to get there?'"
"I don't want our startup to be another esoteric imaging product that fails to find a market," he says.
While it's by no means certain that QIS sensors will make their way into mainstream cameras, it already looks like the technology has tremendous potential for niches such as scientific measurement. This alone shows just how far the technology has come from Fossum's original idea. As he readily admits: "When we first started this project I wasn't even sure it could be made to work."
Nikon looks to be positioning its D850 as a serious video rig with today's announcement of its D850 Filmmaker's Kit. The kit includes the body, 20/35/85mm F1.8G lenses, an Atomos Ninja Flame external recorder, two microphones and an extra battery.
Photographers shopping around for Lightroom alternatives have likely encountered Alien Skin's Exposure X3. Here's an overview of its organization and editing controls, and how they differ from the competition.
Alien Skin has released a significant update for its Exposure X3 image editor, adding greater precision to adjustment tools and more printing capabilities, among other improvements.
The FAA has ordered helicopter pilots and operators to halt certain doors-off flights in the wake of a tragedy that killed five passengers.
Analysts TechInsights have torn down a Samsung Galaxy S9 Plus to have a closer look at the device's internal components and their cost.
Oppo's new high-end phones bear an uncanny resemblance to the iPhone X, with features like face unlock to a portrait lighting mode.
Recently we visited the 2018 CP+ show in Yokohama, Japan and as usual, we booked interviews with senior executives from several major manufacturers, including Sigma.
At this year's CP+ show in Yokohama, we sat down with senior executives from several major manufacturers, including Canon. Topics of conversation included Canon's ambitions for high-end mirrorless cameras, and the importance of responding to the demands of the smartphone generation.
We were recently able to follow local frame builder Max Kullaway as he created one of his AirLandSea bikes. Here are our picks of the photos we got, as the project progressed from bare tubes all the way to rideable bicycle.
On paper, the Sony a7 III is a tempting option for photographers who've been considering a switch to full-frame mirrorless. But how does its image quality stack up? We compare it to the Mark II and a few of its other peers.
Erez Marom shares the details behind this beautiful aurora photograph, captured on Haukland Beach in the Lofoten Islands, Arctic Norway, on a moonless evening.
Google Lens uses artificial intelligence and 'computer vision' to identify and provide information about businesses, landmarks and other objects using your phone's camera. And now it's available for iPhone users, too.
The company posted a record quarterly revenue of $2.08 billion for the first quarter of the 2018 fiscal year. That represents incredibly healthy year-over-year growth of 24 percent.
In the job posting, the Times' describes this role as "one of the most important and high-profile jobs in visual journalism." If you're looking for a high profile job in photojournalism, you could do a lot worse than being Photo Director at The Gray Lady.
According to a recent report out of South Korea, Samsung is increasing production of its ISOCELL image sensors in a bid towards market leadership for image sensors. To reach this goal, Samsung will have to dethrone current market leader Sony... no small task.
In this video, large format photographer Ben Horne shows off the incredible resolving power of 8x10 slide film by pixel peeping a massive 709.6-megapixel drum scan of one of his landscape shots. And you thought 100MP medium format was big...
Photographer Wendy Teal tells the heart-breaking story of a wedding she shot at a hospital on just 24-hours notice. The mother of the bride had been given one week to live, and Wendy responded to the couple's desperate social media plea for someone to capture their special day.
This tiny little plug-and-play VR/AR camera for Android phones uses a pair of greater-than-180° FOV fisheye lenses to offer both 360° video/photo capture and 360° livestreaming at 1440p resolution.
Syrp has announced the Magic Carpet Pro: a slider that offers filmmakers an 'infinitely extendable' range thanks to built-in track levers that let you connect lengths of track without the use of tools.
At CP+ we sat down with executives from several major manufacturers. Among them was Kenji Tanaka, of Sony, who talked to us about the a7 III as well as its plans to attract more pro shooters – without ignoring APS-C and entry-level customers.
How do you shoot macro photography on an 18x24cm large format wet plate camera? You 'connect' two large format cameras together! That's how wet plate photographer Markus Hofstaetter did it, and you can read about the whole process in this article.
The Fujifilm X-H1 is a top-of-the-range 24MP mirrorless camera with in-body stabilization and the company's most advanced array of video capabilities. We've tested the X-T2's big brother extensively to see how it performs.
Motorsports photojournalist Jamey Price recently flew to Canada with Lamborghini for the car company's Winter Accademia 2018, where clients get to drive the latest Lamborghini supercars on snow and ice. Yes... it is exactly as awesome as it sounds.
For the Pixel 2 smartphone's Motion Photos feature, Google built on its existing Motion Stills technology by adding advanced stabilization that combines software and hardware capabilities to optimize trimming and stabilization.
This "high-capacity advanced spider tripod" system can handle a maximum load of 65kg / 143lbs thanks to its reinforced design and 8-layered carbon fiber legs.
Photographer William Briscoe captured the beautiful two-for-one timelapse just outside Fairbanks, Alaska on January 31st, braving -31°F (-35°C) temperatures to get the shot.
"After his camera was stolen from his room in the orphanage, he switched to an iPhone for his photography, reasoning that the image quality of a big, heavy camera was less important than the freedom of a cell phone. 'Quality? Screw it, I’d sketch things with a pencil if I could draw,' he wrote in a blog post."
Chinese manufacturer Vivo has announced some AI-powered Super HDR tech to compete with Google's HDR+ system. Both systems combine multiple images to create a final shot with more dynamic range and less noise, but Super HDR claims to do so more intelligently.
The YouTube channel JerryRigEverything recently tore down (or rather, tore apart...) the new Samsung Galaxy S9, giving us the closest look at yet at the new smartphone's camera hardware.
The Leica l Model A, dating from between 1926 and 1927, comes with a card signed by Earhart herself. Unfortunately, this is the only 'proof' that the camera really did belong to her.