'Perfect' sensors may be possible, but might not come to cameras
|Fossum's team has created a prototype chip with a variety of pixel designs and readout methods. This included combinations with sufficiently low read noise to allow individual photons to be counted.|
The future could include sensors that perfectly describe the light in the scene, that offer new computational possibilities and give film-like latitude in the highlights. And yet we may not ever see them in cameras, says father of the CMOS sensor, Professor Eric Fossum.
We spoke to Fossum shortly after he received, alongside three other pioneers of digital photography, the Queen Elizabeth Prize for Engineering for his work on CMOS sensors. But the topic of our conversation is the future, rather than his past achievements. He now leads a group at the Thayer School of Engineering at Dartmouth, New Hampshire, working on what he calls Quanta Image Sensors (QIS). The team has recently published a paper announcing a breakthrough using the same fabrication process used to make CMOS image sensors.
The perfect sensor?
The principle is to use nanoscale, specialized pixels, called 'Jots' to capture light at the level of individual photons. They work in a binary fashion: they've either received a photon or they haven't (as opposed to conventional sensors which accumulate the charge generated by lots of photons during exposure). These jots are read repeatedly to see whether another photon has arrived since they were last checked.
While Fossum is keen to stress that other teams are having some success in the same field (using a slightly different approach), his own team's work is looking very promising. The paper in the journal Optica shows the team's technology has been refined such that a 1MJot chip can be read 1000 times per second while still exhibiting sufficiently low read noise that it can distinguish between individual photons.
We can count every photon: you can't do any better than that
"The Holy Grail is no read noise," says Fossum: "so that the read signal is proportional to the signal as it arrived." And the team's latest paper says they've got very close to this, with noise levels so low that the sensor can distinguish between individual photons without getting confused by read noise. This opens up the possibility of cameras that could perfectly describe the light in the scene, even in near total darkness.
Eliminating read noise from the sensor wouldn't mean totally noiseless photos, since the randomness of the light being captured is a key source of noise, but it's the best any sensor can possibly achieve. "We can count every photon: you can't do any better than that," he says.
The paper, perhaps conservatively, says the technology could be suited to scientific, space, security and low-light imaging applications, but Fossum has clearly also been thinking about conventional photography.
A classic response
"Because it's binary in nature, its response is comparable to old photographic film," he says. "In film, when the silver halide was hit by a photon, it's reduced to a silver atom that isn't washed away [during processing]. If it's hit by two photons, it doesn't make any additional difference."
This ends up meaning that in bright regions of the image there are ever fewer unexposed silver ions as the exposure goes on. This, in turn makes it less likely that the last few ions will be hit by a photon, so it becomes increasingly difficult to fully saturate the system. The same is true for the tiny, binary Jots: as more of them become saturated, it becomes increasingly difficult to saturate the last few.
"The response is linear at moderate exposure but it trails off to give significant overexposure latitude. It's a pattern first observed by Hurter and Driffield in 1890," says Fossum: "they showed the same curve that we measure, experimentally, in our QIS devices."
"That has obvious interest both for still photographers who're used to shooting film and for cinematographers who're looking for that kind of response."
The use of such tiny pixels has other benefits, too: "Jots are below diffraction limits in size. This means the resolution of the system is always higher than the resolution of the lens, which means we never have to worry about aliasing." While the group's prototype sensors feature one million Jots, Fossum says their target is one billion.
Beyond conventional photography
Fossum isn't just thinking about photographic history, though. The tiny size and the approach of repeatedly reading out the sensor challenges the existing concept of single exposures. "At the moment we make motion pictures by shooting a series of snapshots. With QIS it's more like the reverse process," he says: constructing still images from precisely captured movement.
|Professor Fossum has already been responsible for one revolution in photography: the invention of the CMOS sensor. In December 2017 he was awarded the Queen Elizabeth Prize for Engineering for his work.|
Essentially, taking lots of short, sub-frames during an exposure gives you an extra dimension to your images: time. "If you take a single frame, you get a bunch of ones and zeros. If you take another, you quickly build up a cube of ones and zeros," Fossum says: "For example, if you shoot 100 frames at 1000 frames per second, you get a cube that's x pixels wide by y pixels tall, but also 100 frames deep."
This presents some interesting questions, he says: "What do you do with that data? How do you create an image from that very faithful map of where photons arrived?"
"You could choose a number of pixels in x and y but also in the time axis. If you wanted a very sensitive pixel in low light you could combine 10 x 10 Jots in x and y and then maybe combine the data from 100 frames: it's essentially like increasing the grain size in a more sensitive film."
Of course you can achieve something comparable to this in conventional digital photography by downscaling an image, but Jots allow greater flexibility, Fossum says: "your pixel size could vary between different parts of the image, so in some places you'd have bigger but more sensitive grains."
What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was?
The time component also opens up additional possibilities, he says: "if an object moves during these hundred frames, instead of adding all the values from the same location, you could add them at an angle that corresponds to the movement," so that all the pixels relating to the same object are combined. "We could take out motion blur or remove the scanning effect of a computer screen in video."
The idea of combining multiple frames in interesting ways is, of course, already becoming a core part of mobile photography, and Fossum says finding all the things that are possible is a challenge he is leaving for others: "From my point of view, we're building a platform for computational imaging, it's for others to develop all the ways to use it. A camera would have to take account of the new sensor capabilities."
But it'll ask interesting questions, he believes: "What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was? Some of the things we associate with photography are artifacts of the way we capture them."
Not the only future
With all this going for it, it might seem odd that Fossum isn't promising to deliver a second revolution in digital imaging. But, having devoted a career to developing technologies and teaching about the challenges, he's realistic both about the work left to do and the competition any product would face.
"What we've already achieved is wonderful. The next challenge is adding color [awareness], but I don't think that's going to be particularly problematic. Then there's power: we've shown we can produce a large chip that doesn't consume or disperse a prohibitively large amount of power. We're currently at around 27mW but scale it up by 1000 [to get to one billion Jots] and that's 27W, so we need to cut that by about a factor of ten."
His concern is more about the current state of the rival technologies: "In order to bring a new technology to replace the existing one, it has to be compellingly better in a number of ways," he says. "For a few niches, [our technology] is already compelling." But for photography, the bar is already set very high.
I don't want our startup to be another esoteric imaging product that fails to find a market
"CMOS technology is pretty awesome right now," he says, before almost embarrassedly stressing that he's not claiming the credit for this: "where it is today is the result of the input from thousands of engineers from different companies who've contributed towards where we are now."
|Professor Eric Fossum pictured with Dr Jiaju Ma, one of the co-authors of the Optica paper and a co-founder of the spin-off company, Gigajot Technology.|
But, for all his cautious words, Fossum is convinced enough by the technology's potential to have created a company, Gigajot Technology, with his co-researchers. "Finding a sweet spot in the market is a really important part of challenge. It comes back to the things I teach: 'who is your customer?' 'what is your market?' 'how are we going to get there?'"
"I don't want our startup to be another esoteric imaging product that fails to find a market," he says.
While it's by no means certain that QIS sensors will make their way into mainstream cameras, it already looks like the technology has tremendous potential for niches such as scientific measurement. This alone shows just how far the technology has come from Fossum's original idea. As he readily admits: "When we first started this project I wasn't even sure it could be made to work."
|First, Let me check its expiry date. by rajeev22675|
from Best Photo of the Week
|Dairy Way by BodkinsBest|
from Best Astrophotography Landscape #4
Residents of a Paris street plagued by Instagrammers, selfie takers and music video crews are asking the city government for a weekend and evening ban to give them some peace.
The adapter plugs into the Osmo Pocket's USB Type-C port and features a 3.5mm TRS jack to plug in various external microphones.
Checkout allows Instagram users to select products for purchase and make payments directly in the app.
GauGAN as it's known, can create photorealistic images from basic drawings using the power of artificial intelligence.
The EOS RP is Canon's latest full-frame mirrorless camera, with diminutive dimensions and a diminutive price. Find out how it stacks up and get our thoughts in our early review.
Montana judge Dana L. Christensen has ruled the Republican National Committee did not infringe upon the copyright of photographer Erika Peterman after they took a photo from a Democratic candidate's Facebook page without permission and altered it to use in a derogatory promotional mailer.
Nikon has launched updates for three of its programs to address various bugs and glitches that could cause crashes and unwanted results.
LEE Filters has launched the LEE100, its next-generation filter holder that improves the design and looks in all the right places.
With the arrival of some much-needed sunshine and final production firmware for the Panasonic S1, we've been able to get outside and really start putting the camera through its paces.
Importing, culling and tagging photos is about to get a whole lot faster and look a whole lot better with the impending arrival of Photo Mechanic 6.
On its own, the FTZ adapter retails for $250 and when bundled it dropped the cost to just $150. Now, Nikon is offering it for free with all Z6, Z7 purchases in the United States.
Profoto said it spoke with Godox back at Photokina 2018 and continues to contact Godox in an effort to stop it from marketing its V1 light.
Product renders in Italian publication Notebook Italia show an unusual design that conceals all cameras with the help of a slider mechanism.
Canon says its new EF 400mm F2.8L IS III and EF 600mm F4L IS III lenses can suffer from an intermittent flickering when shooting video in M or Av modes with certain cameras.
Leica recently announced the Q2, a digital rangefinder with a fixed 28mm F1.7 lens. It's a heck of a lot of fun to shoot with, but is it right for you? Based on our time with the camera, and its specifications, we've examined how well-suited it is for common photography use-cases.
Now that our Panasonic Lumix S1R has final firmware, we couldn't wait to get out shooting with it - and we also tried the high-res mode, which combines files to get 187 megapixel images. Because sometimes, 47 megapixels just isn't enough.
In this article, travel and landscape photographer Mitch Green encourages us to spend more time in the the field.
the lens lacks any electronics whatsoever and is constructed entirely of glass and metal. Of course, that comes at the expense of weight — this thing weighs in at 1.1kg / 2.43lbs.
Drones can be useful tools in urban areas, where they're utilized for everything from news reporting to building inspections, but flying in these areas requires careful preparation. Here's what you need to know to do so safely.
Hasselblad has released a new cable release and USB double battery charger for its X1D medium format camera .
After a report published by NBC News, Flickr has taken heat for allegedly letting IBM 'scrape' photos for use in its facial recognition datasets. But the problem isn't what it seems on the surface.
Samyang has announced the impending arrival of the AF 85mm F1.4 FE lens for full-frame Sony cameras.
Some Photoshop shortcuts are simple and obvious. Others, not so much. Here are 15 shortcuts that are actually useful.
Twitter has redesigned its in-app camera for easier access from the timeline screen.
Independent cinema lens manufacturer SLR Magic has announced it will offer all of its existing MicroPrime range in the Fujifilm X mount and has even created a Fuji-specific 12mm lens.
We've updated our buying guides with three more cameras: the Canon EOS RP, Nikon Z6 and Olympus E-M1X.
CFexpress 2.0 cards will come in three different form factors, each of which will offer different maximum speeds.
Lensbaby has added a third tilt lens to its Optic Swap system, this time a 35mm lens, adding to the existing 50mm and 80mm options.
Sigma has released firmware updates for a number of its lenses as well as its EF-E adapter to address various errors and features with Canon, Nikon and Sony cameras.
We've added the Fujifilm X-T30 and Sony a6400 to our 'Best Cameras under $1000' buying guide. These two mirrorless models pack in a lot of features for just $900 body only.