'Perfect' sensors may be possible, but might not come to cameras
|Fossum's team has created a prototype chip with a variety of pixel designs and readout methods. This included combinations with sufficiently low read noise to allow individual photons to be counted.|
The future could include sensors that perfectly describe the light in the scene, that offer new computational possibilities and give film-like latitude in the highlights. And yet we may not ever see them in cameras, says father of the CMOS sensor, Professor Eric Fossum.
We spoke to Fossum shortly after he received, alongside three other pioneers of digital photography, the Queen Elizabeth Prize for Engineering for his work on CMOS sensors. But the topic of our conversation is the future, rather than his past achievements. He now leads a group at the Thayer School of Engineering at Dartmouth, New Hampshire, working on what he calls Quanta Image Sensors (QIS). The team has recently published a paper announcing a breakthrough using the same fabrication process used to make CMOS image sensors.
The perfect sensor?
The principle is to use nanoscale, specialized pixels, called 'Jots' to capture light at the level of individual photons. They work in a binary fashion: they've either received a photon or they haven't (as opposed to conventional sensors which accumulate the charge generated by lots of photons during exposure). These jots are read repeatedly to see whether another photon has arrived since they were last checked.
While Fossum is keen to stress that other teams are having some success in the same field (using a slightly different approach), his own team's work is looking very promising. The paper in the journal Optica shows the team's technology has been refined such that a 1MJot chip can be read 1000 times per second while still exhibiting sufficiently low read noise that it can distinguish between individual photons.
We can count every photon: you can't do any better than that
"The Holy Grail is no read noise," says Fossum: "so that the read signal is proportional to the signal as it arrived." And the team's latest paper says they've got very close to this, with noise levels so low that the sensor can distinguish between individual photons without getting confused by read noise. This opens up the possibility of cameras that could perfectly describe the light in the scene, even in near total darkness.
Eliminating read noise from the sensor wouldn't mean totally noiseless photos, since the randomness of the light being captured is a key source of noise, but it's the best any sensor can possibly achieve. "We can count every photon: you can't do any better than that," he says.
The paper, perhaps conservatively, says the technology could be suited to scientific, space, security and low-light imaging applications, but Fossum has clearly also been thinking about conventional photography.
A classic response
"Because it's binary in nature, its response is comparable to old photographic film," he says. "In film, when the silver halide was hit by a photon, it's reduced to a silver atom that isn't washed away [during processing]. If it's hit by two photons, it doesn't make any additional difference."
This ends up meaning that in bright regions of the image there are ever fewer unexposed silver ions as the exposure goes on. This, in turn makes it less likely that the last few ions will be hit by a photon, so it becomes increasingly difficult to fully saturate the system. The same is true for the tiny, binary Jots: as more of them become saturated, it becomes increasingly difficult to saturate the last few.
"The response is linear at moderate exposure but it trails off to give significant overexposure latitude. It's a pattern first observed by Hurter and Driffield in 1890," says Fossum: "they showed the same curve that we measure, experimentally, in our QIS devices."
"That has obvious interest both for still photographers who're used to shooting film and for cinematographers who're looking for that kind of response."
The use of such tiny pixels has other benefits, too: "Jots are below diffraction limits in size. This means the resolution of the system is always higher than the resolution of the lens, which means we never have to worry about aliasing." While the group's prototype sensors feature one million Jots, Fossum says their target is one billion.
Beyond conventional photography
Fossum isn't just thinking about photographic history, though. The tiny size and the approach of repeatedly reading out the sensor challenges the existing concept of single exposures. "At the moment we make motion pictures by shooting a series of snapshots. With QIS it's more like the reverse process," he says: constructing still images from precisely captured movement.
|Professor Fossum has already been responsible for one revolution in photography: the invention of the CMOS sensor. In December 2017 he was awarded the Queen Elizabeth Prize for Engineering for his work.|
Essentially, taking lots of short, sub-frames during an exposure gives you an extra dimension to your images: time. "If you take a single frame, you get a bunch of ones and zeros. If you take another, you quickly build up a cube of ones and zeros," Fossum says: "For example, if you shoot 100 frames at 1000 frames per second, you get a cube that's x pixels wide by y pixels tall, but also 100 frames deep."
This presents some interesting questions, he says: "What do you do with that data? How do you create an image from that very faithful map of where photons arrived?"
"You could choose a number of pixels in x and y but also in the time axis. If you wanted a very sensitive pixel in low light you could combine 10 x 10 Jots in x and y and then maybe combine the data from 100 frames: it's essentially like increasing the grain size in a more sensitive film."
Of course you can achieve something comparable to this in conventional digital photography by downscaling an image, but Jots allow greater flexibility, Fossum says: "your pixel size could vary between different parts of the image, so in some places you'd have bigger but more sensitive grains."
What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was?
The time component also opens up additional possibilities, he says: "if an object moves during these hundred frames, instead of adding all the values from the same location, you could add them at an angle that corresponds to the movement," so that all the pixels relating to the same object are combined. "We could take out motion blur or remove the scanning effect of a computer screen in video."
The idea of combining multiple frames in interesting ways is, of course, already becoming a core part of mobile photography, and Fossum says finding all the things that are possible is a challenge he is leaving for others: "From my point of view, we're building a platform for computational imaging, it's for others to develop all the ways to use it. A camera would have to take account of the new sensor capabilities."
But it'll ask interesting questions, he believes: "What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was? Some of the things we associate with photography are artifacts of the way we capture them."
Not the only future
With all this going for it, it might seem odd that Fossum isn't promising to deliver a second revolution in digital imaging. But, having devoted a career to developing technologies and teaching about the challenges, he's realistic both about the work left to do and the competition any product would face.
"What we've already achieved is wonderful. The next challenge is adding color [awareness], but I don't think that's going to be particularly problematic. Then there's power: we've shown we can produce a large chip that doesn't consume or disperse a prohibitively large amount of power. We're currently at around 27mW but scale it up by 1000 [to get to one billion Jots] and that's 27W, so we need to cut that by about a factor of ten."
His concern is more about the current state of the rival technologies: "In order to bring a new technology to replace the existing one, it has to be compellingly better in a number of ways," he says. "For a few niches, [our technology] is already compelling." But for photography, the bar is already set very high.
I don't want our startup to be another esoteric imaging product that fails to find a market
"CMOS technology is pretty awesome right now," he says, before almost embarrassedly stressing that he's not claiming the credit for this: "where it is today is the result of the input from thousands of engineers from different companies who've contributed towards where we are now."
|Professor Eric Fossum pictured with Dr Jiaju Ma, one of the co-authors of the Optica paper and a co-founder of the spin-off company, Gigajot Technology.|
But, for all his cautious words, Fossum is convinced enough by the technology's potential to have created a company, Gigajot Technology, with his co-researchers. "Finding a sweet spot in the market is a really important part of challenge. It comes back to the things I teach: 'who is your customer?' 'what is your market?' 'how are we going to get there?'"
"I don't want our startup to be another esoteric imaging product that fails to find a market," he says.
While it's by no means certain that QIS sensors will make their way into mainstream cameras, it already looks like the technology has tremendous potential for niches such as scientific measurement. This alone shows just how far the technology has come from Fossum's original idea. As he readily admits: "When we first started this project I wasn't even sure it could be made to work."
|Dubai by Nilesh Trivedi|
|Hummingbird Tight by Dennis Bayer|
from -Vivid Purple- (in Full Colours Only)
After shaking up the Lightroom ecosystem with Lightroom CC last year, Adobe has released version 2.0 of the cloud-centric photo organizer and editor. We look at new features like People View, how far Lightroom CC has come in its first year, and where Lightroom is headed.
Today, at Adobe MAX 2018, Adobe previewed Photoshop CC on iPad, a full-featured, desktop-class version of Photoshop for iOS.
The weather and has most definitely taken a turn toward fall here, and our shooting opportunities have followed suit. We brought the Canon RF 35mm F1.8 along to a harvest festival of sorts and a few of our usual haunts.
Pennsylvania Governor Tom Wolf has signed House Bill 1346 into effect, which imposes a fine upwards of $300 to drone operators who invade the privacy or harm the physical wellbeing of citizens.
Sigma is a company in flux, but CEO Kazuto Yamaki is undaunted by the upcoming prospect of developing lenses for eight lens mounts. The challenge will be keeping the company's identity along the way.
If you've been meaning to convert all of your old photos, video, and audio to digital formats, but simply lack the time or willpower to get through it all, a new service from Kodak will help you get the job done.
Almost all new cameras include impressive video features, but for the best results you'll often need an off-camera recorder. Chris and Jordan take a look at the brand new Ninja V from Atomos, and explain why it might just be one of the most useful tools you can add to your camera.
Collect allows you to transform 360-degree into a more easily digestible format by transforming it into directed traditional videos.
Sick of using your plain ol' keyboard to edit your photos in Lightroom and Photoshop? TourBox is hoping to expedite your post-production workflow using a clever combination of dials, buttons, and knobs.
Bag and accessory manufacturer Hex has launched two bags as part of its latest collection: the Clamshell Backpack and DSLR Sling.
Crank out instant photos with Holga Digital's new analog printer, currently being funded on Kickstarter.
We got some hands-on time with Leica's new S3 medium format camera, which boasts a new higher-res sensor as well as other improvements.
Luna Display started its life as a crowdfunding project on Kickstarter. Now, it's available to purchase directly online.
We sat down with the Google Pixel camera team to learn about key new camera features on the Pixel 3 and Pixel 3 XL, and an explanation of the sophisticated software advancements that power them.
A lawsuit filed on Tuesday claims the cameras in Apple's iPhone 7 Plus and newer dual-camera models infringe on a patent that was granted in 2003.
Nikon's Coolpix P1000 has moved the zoom needle from 'absurd' to 'ludicrous,' with an equivalent focal length of 24-3000mm. So far, it's a fun camera to shoot with – if a bit over the top.
Like the LG V40 ThinQ the A9 combines a super-wide-angle, regular wide-angle and tele camera, but adds a depth-sensor to the mix as well.
The FAA has issued a warning to drone pilots in anticipation of disaster response following Hurricane Michael, noting that fines for interfering with emergency operations can exceed $20,000.
According to a report from Fortune, Apple acquired Danish masking technology startup Spektral in December 2017 for "more than $30 million."
Insta360's latest model comes with a range of features that allow for the creation of unique action cam footage.
The Photogrip can be used as a camera grip, mini tripod or phone stand and comes with a detachable remote.
At a time when manufacturers are adding triple and even quad-cameras to their flagship smartphones, Google is sticking with one main camera. But given the sophistication of the company's computational efforts, we think it's the right approach for now.
DPReview is hiring! We're seeking three Software Development Engineers at a range of experience levels to join our Seattle-based team.
The University of Dayton Research Institute created a video detailing what damage is caused when a drone strikes the wing of an airplane.
Lenovo's upcoming high-end smartphone will be the first model to feature four cameras on the back.
The Pixel 3 and Pixel 3 XL offer a second front-facing camera and a host of improved computational features such as digital zoom based on super-resolution capture, better depth mapping and a fill-light effect for low light portraits.
Canon has ported a large chunk of its Digital Photo Professional (DPP) Raw processing software's feature set to iOS and launched the DPP Express app.
The Panasonic LX100 II offers a higher-resolution sensor over its predecessor, but it's the addition of a touchscreen that makes the Mark II so gosh-darn enjoyable to shoot with. We've got some fresh samples from Panasonic's new premium compact camera.
Sony has announced a new "Alpha Female" program, a creator-in-residence opportunity that will award six-month grants to five female filmmakers and photographers.
The new 490, 492 and 492LCD are targeted at amateur photographers and come with a 4kg/8.82lbs payload.