How Google developed the Pixel 3's Super Res Zoom technology
In a blog post on its Google AI Blog, Google Software Engineer Bartlomiej Wronski and Computational Imaging Lead Scientist Peyman Milanfar have laid out how they created the new Super Res Zoom technology inside the Pixel 3 and Pixel 3 XL.
Over the past year or so, several smartphone manufacturers have added multiple cameras to their phones with 2x or even 3x optical zoom lenses. Google, however, has taken a different path, deciding instead to stick with a single main camera in its new Pixel 3 models and implementing a new feature it is calling Super Res Zoom.
Unlike conventional digital zoom, Super Res Zoom technology isn't simply upscaling a crop from a single image. Instead, the technology merges many slightly offset frames to create a higher resolution image. Google claims the end results are roughly on par with 2x optical zoom lenses on other smartphones.
The Google engineers are using the photographer's hand motion - and the resulting movement between individual frames of a burst - to their advantage. After optical stabilization removes macro movements (5-20 pixels), the remaining high frequency movement due to hand tremor naturally shifts the image on the sensor by just a few pixels. Since any shift is unlikely to be exactly (a multiple of) a single pixel, scene detail can be localized with sub-pixel precision, provided you interpolate between pixels when synthesizing the super resolution image.
When the device is mounted on a tripod or otherwise stabilized natural hand motion is simulated by slightly moving the camera's OIS module between shots.
The pictures from a burst - of up to 15 frames on the Pixel 3 - are aligned on a base grid of higher resolution than that of each individual frame. First a reference frame is chosen, and then all other frames are aligned relative to it with sub-pixel precision. This leads to increased detail - albeit ultimately limited by the lens' resolving power - and cleaner images, since frame averaging reduces noise. When there are objects that have moved relative to the reference frame, the software only merges information from other frames if it has confidently found the correct corresponding feature, thus avoiding ghosting.
|Left: crop of a 7x zoomed image on Pixel 2 (digital zoom). Right: crop of Super Res Zoom image on Pixel 3. Note not only the increase in detail, but the decrease in noise due to frame averaging and not having to demosaic.|
As a bonus there's no more need to demosaic, resulting in even more image detail and less noise. With enough frames in a burst any scene element will have fallen on a red, green, and blue pixel on the image sensor. After alignment R, G, and B information is then available for any scene element, removing the need for demosaicing.
Furthermore, Google's merge algorithm takes into account edges in the image and adapts accordingly, merging pixels along the direction of edges as opposed to across them. This approach provides a suitable trade-off between increased resolution and noise suppression, and avoids the artifacts less sophisticated techniques introduce (see dots and increased perception of noise in the 'Dynamic Pixel Shift' crop on right, here).
One might initially think 'wouldn't it be easier to just put an optical 2x zoom in the phone', but perhaps that isn't the question to ask. Super resolution can increase the resolution of even the standard camera without zoom*, and all zooms in between a wide and a tele module. And any technique that makes a single camera better will make multiple camera approaches that much better. Imagine a smartphone with 3 or 4 lens modules that allows you to smoothly zoom between all focal lengths, with super resolution ensuring that focal lengths in between those of each lens module remain detailed.
*For now, the Pixel 3 does not use super resolution for standard 1x shots. Super Res only kicks in at 1.2x and above, currently for performance reasons.
Since the launch of the EOS R line in 2018, Canon has mostly concentrated its attentions on high-end RF lenses. The RF 85mm F1.2L IS USM is among the very best.
Following a number of reports in the DPReview forums and on Fuji X Forums, Fujifilm has confirmed that a 'very small percentage' of X-Pro 3 units are experiencing an issue with the EVF display being too bright.
The new mount options use the same overall design as the original Micro Four Thirds version released back in 2018.
The Fragment 8 from Loft Factory is a digital camera that aims to replicate the look, feel and image quality of 1960s and 70s Super 8 cameras using modern technologies.
According to a report from FujiAddict, lens maker Venus Optics is working on a new 'Argus' lens under its Laowa brand that will feature an ultra-fast F0.95 aperture.
Previously available only for Sony E-mount cameras, the Nokton 21mm F1.4 is now available for Leica M-mount camera systems. No pricing outside of the Japan region has been given, nor has a release date.
Professional and hobbyist photographers, alike, have this challenge in common: organizing their images so they're findable. Wordroom was created to help.
Camera launches often prompt comments about how overpriced the new model is, and how it should cost something nearer to the outgoing one, or a well-established rival. Here's why it doesn't work like that.
The Canadian Internet Registration Authority (CIRA), the organization in charge of Canada's '.ca' top-level domain has published a series of free-to-use stock photos that poke fun at Canadien stereotypes.
Considered in isolation, the D780 is a tremendously well-rounded camera. But that's also true of the D750 and the Z6, which may make it a hard sell for many photographers.
Pentax week continues! In this video, Chris takes us for a ride on the wayback machine to look at the most important Pentax cameras in history.
Canon USA takes down two eBay sellers accused of selling counterfeit batteries using the company's trademarks.
Fotodiox has launched another Canon EF to Fuji GFX lens mount adapter, this one with a built-in neutral density filter and support for automated functions.
The incremental firmware updates address a few minor issues and improve overall stability, but don't add any new or revolutionary features.
It's tempting to scoff at a black-and-white-only digital rangefinder, but there are a few reasons why going without a color filter is more than a gimmick. Here's why we're excited to go shooting with the M10 Monochrom.
Leica has introduced its latest dedicated black and white camera, the M10 Monochrom. Based largely on the M10-P, it features a newly designed 40MP sensor, touchscreen LCD and lower base ISO than previous mono models.
The 85mm F1.8 FE lens is the first in Tokina’s new atx-m mirrorless lens series, which will feature lenses for multiple mounts and sensor formats.
Australian researchers have pioneered the development of a lithium-sulfur battery technology that may revolutionize consumer gadgets, cameras, electric cars and more.
The TZC-01 will work with over 60 EF-mount lenses from Canon, Sigma, Tamron, Tokina and Zeiss.
A new power adapter from Kessler makes it possible to charge cameras and other gear using ordinary DeWalt power tool batteries.
The team behind the Platypod is back with a new accessory called Platyball, an entirely new type of tripod head with electronic stabilization.
Heathrow Airport, the busiest in the United Kingdom, has installed an anti-drone system that detects and blocks drones entering its airspace.
A recent look into the latest macOS beta release revealed evidence of an upcoming feature called 'Pro Mode' that boosts performance for professionals.
We've just gotten our hands on a full production Nikon D780 – the long-awaited update to the D750, which we'd called one of the most well-rounded cameras we'd tested. Take a deep dive into all that the new model offers right here.
DJI, the world's leading drone manufacturer, posted an article outlining the need for Remote ID. However, the company does not agree with what's proposed in the FAA's NPRM.
Sharp's 8K Video Camera has once again been showcased at CES, this time with more details on what the camera will offer and when it will be available.
Tamron announced that its 20mm F2.8 Di III OSD M1:2 lens for full-frame Sony E-mount bodies will be available on January 30th for $350.
Boasting only 2 propellers, instead of the typical 4 on a consumer-grade drone, Falcon's bi-copter can fly up to 50 minutes – about 20 minutes longer than leading competitors.
Luminar 4.1 brings a number of new tools and features to improve your post-processing workflow.
Acer joined HP and MSI in unveiling new laptops at CES 2020 that are made specifically for photographers and other graphics professionals.