How Google developed the Pixel 3's Super Res Zoom technology
In a blog post on its Google AI Blog, Google Software Engineer Bartlomiej Wronski and Computational Imaging Lead Scientist Peyman Milanfar have laid out how they created the new Super Res Zoom technology inside the Pixel 3 and Pixel 3 XL.
Over the past year or so, several smartphone manufacturers have added multiple cameras to their phones with 2x or even 3x optical zoom lenses. Google, however, has taken a different path, deciding instead to stick with a single main camera in its new Pixel 3 models and implementing a new feature it is calling Super Res Zoom.
Unlike conventional digital zoom, Super Res Zoom technology isn't simply upscaling a crop from a single image. Instead, the technology merges many slightly offset frames to create a higher resolution image. Google claims the end results are roughly on par with 2x optical zoom lenses on other smartphones.
The Google engineers are using the photographer's hand motion - and the resulting movement between individual frames of a burst - to their advantage. After optical stabilization removes macro movements (5-20 pixels), the remaining high frequency movement due to hand tremor naturally shifts the image on the sensor by just a few pixels. Since any shift is unlikely to be exactly (a multiple of) a single pixel, scene detail can be localized with sub-pixel precision, provided you interpolate between pixels when synthesizing the super resolution image.
When the device is mounted on a tripod or otherwise stabilized natural hand motion is simulated by slightly moving the camera's OIS module between shots.
The pictures from a burst - of up to 15 frames on the Pixel 3 - are aligned on a base grid of higher resolution than that of each individual frame. First a reference frame is chosen, and then all other frames are aligned relative to it with sub-pixel precision. This leads to increased detail - albeit ultimately limited by the lens' resolving power - and cleaner images, since frame averaging reduces noise. When there are objects that have moved relative to the reference frame, the software only merges information from other frames if it has confidently found the correct corresponding feature, thus avoiding ghosting.
|Left: crop of a 7x zoomed image on Pixel 2 (digital zoom). Right: crop of Super Res Zoom image on Pixel 3. Note not only the increase in detail, but the decrease in noise due to frame averaging and not having to demosaic.|
As a bonus there's no more need to demosaic, resulting in even more image detail and less noise. With enough frames in a burst any scene element will have fallen on a red, green, and blue pixel on the image sensor. After alignment R, G, and B information is then available for any scene element, removing the need for demosaicing.
Furthermore, Google's merge algorithm takes into account edges in the image and adapts accordingly, merging pixels along the direction of edges as opposed to across them. This approach provides a suitable trade-off between increased resolution and noise suppression, and avoids the artifacts less sophisticated techniques introduce (see dots and increased perception of noise in the 'Dynamic Pixel Shift' crop on right, here).
One might initially think 'wouldn't it be easier to just put an optical 2x zoom in the phone', but perhaps that isn't the question to ask. Super resolution can increase the resolution of even the standard camera without zoom*, and all zooms in between a wide and a tele module. And any technique that makes a single camera better will make multiple camera approaches that much better. Imagine a smartphone with 3 or 4 lens modules that allows you to smoothly zoom between all focal lengths, with super resolution ensuring that focal lengths in between those of each lens module remain detailed.
*For now, the Pixel 3 does not use super resolution for standard 1x shots. Super Res only kicks in at 1.2x and above, currently for performance reasons.
Canon Rumors has rounded up a collection of specifications from multiple, unrelated sources that could point to Canon's next R-series camera being both a stills and video behemoth.
Moment has packed a bunch of features into its affordable new variable ND filter product line.
First published in 1980, Photo District News has been sharing the latest photography news and content for decades.
Kodak Alaris is warning photographers to ask for manual inspection at airport screening checkpoints that use CT scanners when traveling with undeveloped film.
The app is one of the first to take advantage of the iPhone 11's multi-cam recording capabilities.
The Jollylook Mini Auto is a second-generation camera constructed of a laminated fiberboard that now includes a built-in flash and automatic exposure.
Phase One's latest Capture One 20 software package is a powerful tool, but we do have a reservation or two - find out more in our review.
The Chinese manufacturer has published a list of improvements it wants to make after a videography-focused feedback session with its users.
Autel has confirmed to DPReview that its EVO II series drones won't hit shelves until March 2020 after discovering a bug that ‘could limit flight performance under normal operation.’
Chinese company Xiaomi has so far (as far as we know) worked without the help of a big name in the camera business, but it appears this could be about to change.
DNP Photo is back with another dye sublimation photo printer. The QW410 is relatively compact and includes wireless support for mobile users.
Photographer Pye Jirsa is teaching photographers how to pose couples as part of a new educational video series from Adorama.
Award-winning French photographer Florian Ledoux has captured photos of the arctic from both a drone's perspective and the ground.
Photographer and filmmaker Mark Holtze shares his top five reasons why vintage lenses are still relevant in 2020.
What if astrophotography was as simple as using an app? The Stellina smart telescope promises to make this a reality, allowing anyone to channel their own inner Galileo while capturing images of the night sky.
The Fujifilm XC 35mm F2 prime lens has the same optical formula as the company's XF 35mm F2 WR prime but costs half as much. How can this be? Chris and Jordan explain the differences.
In the latest episode of DPReview TV, Chris and Jordan compared the new, inexpensive Fujifilm XC 35mm F2 with the more durable XF version that's twice the price. See some of the photos they took side-by-side in this sample gallery.
The 2019 winners and finalists of the Travel Photographer of the Year Awards have been revealed.
PBS recently gave viewers a look at MIT Museum's new The Polaroid Project instant photography exhibition.
Researchers with IBM and MIT have published a new demonstration tool that allows anyone to 'paint' new elements into existing digital images.
Venus Optics says the lens is the first Laowa macro lens specifically designed for APS-C mirrorless camera systems.
The updates mostly address only a single issue for each lens, but it continues to show Sigma's continued dedication to addressing various issues, even small ones, across its lens lineup.
Spider Holster has launched four new products, including two updated versions of existing gear, that make life easier for photographers.
While nearly all of Tokina's F-mount lenses will work without issue on Nikon Z-series mirrorless cameras, there are a select few that can't use autofocus.
The X-T200 is arguably the company's most compelling entry-level option yet. Take a closer look in our detailed hands-on article.
Last year, DJI announced they will be adding ADS-B sensors to their commercial-grade drones weighing over 250 grams. Now, it is rumored that 3 new drone models will be released this year.
The Fujifilm X-T200 is what we wish the X-T100 had been, offering a solid AF system, proper 4K video and a more responsive interface. Click through to see what else is new and improved.
Fujifilm just announced its newest camera, the X-T200. In this hands-on preview, Chris and Jordan explain how it fits into Fujifilm's lineup and why they like it.
Fujifilm will add the weather-sealed GF 45-100mm F4 to its medium format lens lineup in February. The approximately 35-80mm equivalent zoom will cost around $2300.
Two new primes are hitting the roadmap for Fujifilm's GF system: the 24mm equiv. 30mm F3.5 R WR and an ultra-fast 80mm F1.7.