As part of our regular appearances on the TWiT Network (named after its flagship show, This Week in Tech) show 'The New Screen Savers', our Science Editor Rishi Sanyal joined host Leo Laporte and co-host Megan Morrone to talk about how smartphone cameras are revolutionizing photography. Watch the segment above, then catch the full episode here.
Rishi has also expounded upon some of the topics covered in the segment below, with detailed examples that clarify some of the points covered. Have a read after the fold once you've watched the segment.
So who wins? iPhone X or Pixel 2?
Not so fast.
Each has its strengths, which we talk about in our video segment above and in our examples below. Google and Apple take different approaches, and each has its pros and cons, but there are common overlapping practices and themes as well. And that's before we begin discussing video, where the iPhone's 4K/60p HEVC video borders on professional quality while Google's stabilization may make you chuck your gimbal.
Smartphones have to deal with the fact that their cameras, and therefore sensors, are tiny. And since we all (now) know that, generally speaking, it's the amount of light you capture that determines image quality, smartphones have a serious disadvantage to deal with: they don't capture enough light. But that's where computational photography comes in. By combining machine learning, computer vision, and computer graphics with traditional optical processes, computational photography aims to enhance what is achievable with traditional methods.
Intelligent exposure and processing? Press. Here.
One of the defining characteristics of smartphone photography is the idea that you can get a great image with one button press, and nothing more. No exposure decision, no tapping on the screen to set your exposure, no exposure compensation, and no post-processing. Just take a look at what the Google Pixel 2 XL did with this huge dynamic range sunrise at Banff National Park in Canada:
Sunrise at Banff, with Mt. Rundle in the background. Shot on Pixel 2 with one button press. I also shot this with my Sony a7R II full-frame camera, but that required a 4-stop reverse graduated neutral density ('Daryl Benson') filter, and a dynamic range compensation mode (DRO Lv5) to get a usable image. While the resulting image from the Sony was head-and-shoulders above this one at 100%, I got this image from a device in my pocket by just pointing and shooting.
Apple's iPhones try to achieve similar results by combining multiple exposures if the scene has enough contrast to warrant it. But iPhones can't achieve these results (yet) since they don't average as many 'samples' as the Google Pixel 2. Sometimes Apple's longer exposures can blur subjects, and iPhones tend to overexpose and blow highlights for the sake of exposing the subject properly. Apple is also still pretty reticent to enable HDR in 'Auto HDR'.
The Pixel 2 was able to achieve the image above by first determining the correct focal plane exposure required to not blow large bright (non-specular) areas (an approach known as ETTR or 'expose-to-the-right'). When you press the shutter button, the Pixel 2 goes back in time 9 frames, aligning and averaging them to give you a final image with quality similar to what you might expect from a sensor with 9x as much surface area.
How does it do that? It's constantly keeping the last 9 frames it shot in memory, so when you press the shutter it can grab them, break each into many square 'tiles', align them all, and then average them. Breaking each image into small tiles allows for alignment despite photographer or subject movement by ignoring moving elements, discarding blurred elements in some shots, or re-aligning subjects that have moved from frame to frame. Averaging simulates the effects of shooting with a larger sensor by 'evening out' noise.
That's what allows the Pixel 2 to capture such a wide dynamic range scene: expose for the bright regions, while reducing noise in static elements of the scene by image averaging, while not blurring moving (water) elements of the scene by making intelligent decisions about what to do with elements that shift from frame to frame. Sure, moving elements have more noise to them (since they couldn't have as many of the 9 frames dedicated to them for averaging), but overall, do you see anything but a pleasing image?
Who focuses better? Google Pixel 2, hands down. Its dual pixel AF uses nearly the entire sensor for autofocus (binning the high-resolution sensor into a low-resolution mode to decrease noise), while also using HDR+ and its 9-frame image averaging to further decrease noise and have a usable signal to make AF calculations from.
Google Pixel 2 can focus lightning fast even in indoor artificial light, which allowed me to snap this candid before it was over in a split second. The iPhone X captured a far less interesting moment seconds later when it finally achieved focus, missing the candid moment.
And despite the left and right perspectives the split pixels in the Pixel 2 sensor 'see' having less than 1mm stereo disparity, an impressive depth map can be built, rendering an optically accurate lens blur. This isn't just a matter of masking the foreground and blurring the background, it's an actual progressive blur based on depth.
That's what allowed me to nail this candid image the instant after my wife and child whirled around to face the camera. Nearly all my iPhone X images of this scene were either out-of-focus or captured a less interesting, non-candid moment because of the shutter lag required to focus. The iPhone X only uses approximately 3% of its pixels for its 'Dual PDAF' autofocus, as opposed to the Pixel 2's use of its entire sensor combined with multi-frame noise reduction, not just for image capture but also for focus.
While we've been praising the Pixel phones, Apple is leading smartphone photography in a number of ways. First and foremost: color accuracy. Apple displays are all calibrated and profiled to display accurate colors, so no matter what Apple or color-managed device (or print) you're viewing, colors look the same. Android devices are still the Wild West in this regard, but Google is trying to solve this via a proper color management system (CMS) under-the-hood. It'll be some time before all devices catch up, and even Google itself is struggling with its current display and CMS implementation.
But let's talk about Portrait Lighting. Look at the iPhone X 'Contour Lighting' shot below, left, vs. what the natural lighting looked like at the right (shot on a Google Pixel 2 with no special lighting features). While the Pixel 2 image is more natural, the iPhone X image is far more interesting, as if I'd lit my subject with a light on the spot.
|Apple iPhone X, 'Contour Lighting'||Google Pixel 2|
Apple builds a 3D map of a face using trained algorithms, then allows you to re-light your subject using modes such as 'natural', 'studio' and 'contour' lighting. The latter highlights points of the face like the nose, cheeks and chin that would've caught the light from an external light source aimed at the subject. This gives the image a dimensionality you could normally only achieve using external lighting solutions or a lot of post-processing.
Currently, the Pixel 2 has no such feature, so we get the flat lighting the scene actually had on the right. But, as you can imagine, it won't be long before we see other phones and software packages taking advantage of—and even improving on—these computational approaches.
HDR and wide-gamut photography
And then we have HDR. Not the HDR you're used to thinking about, that creates flat images from large dynamic range scenes. No, we're talking about the ability of HDR displays—like bright contrasty OLEDs—to display the wide range of tones and colors cameras can capture these days, rather than sacrificing global contrast just to increase and preserve local contrast, as traditional camera JPEGs do.
iPhone X is the first device ever to support the HDR display of HDR photos. That is: it can capture a wide dynamic range and color gamut but then also display them without clipping tones and colors on its class-leading OLED display, all in an effort to get closer to reproducing the range of tones and colors we see in the real world.
iPhone X is the first device ever to support HDR display of HDR photos
Have a look below at a Portrait Mode image I shot of my daughter that utilizes colors and luminances in the P3 color space. P3 is the color space Hollywood is now using for most of its movies (it's similar, though shifted, to Adobe RGB). You'll only see the extra colors if you have a P3-capable display and a color-managed OS/browser (macOS + Google Chrome, or the newest iPads and iPhones). On a P3 display, switch between 'P3' and 'sRGB' to see the colors you're missing with sRGB-only capture.
Or, on any display, hover over 'Colors in P3 out-of-gamut of sRGB' to see (in grey) what you're missing with a sRGB-only capture/display workflow.
|iPhone X Portrait Mode, image in P3 color space||iPhone X Portrait mode, image in sRGB color space||Colors in P3 out-of-gamut of sRGB highlighted in grey|
Apple is not only taking advantage of the extra colors of the P3 color space, it's also encoding its images in the 'High Efficiency Image Format' (HEIF), which is an advanced format aimed to replace JPEG that is more efficient and also allows for 10-bit color encoding (to avoid banding while allowing for more colors) and HDR encoding to allow the display of a larger range of tones on HDR displays.
But will smartphones replace traditional cameras?
For many, yes, absolutely. You've seen the autofocus speeds of the Pixel 2, assisted by not only dual pixel AF but also laser AF. You've seen the results of HDR+ image stacking, which will only get better with time. We've seen dual lens units that give you the focal lengths of a camera body and two primes, and we've seen the ability to selectively blur backgrounds and isolate subjects like the pros do.
Below is a shot from the Pixel 2 vs. a shot from a $4,000 full-frame body and 55mm F1.8 lens combo—which is which?
|Full Frame or Pixel 2?||Pixel 2 or Full Frame?|
Yes, the trained—myself included—can pick out which is the smartphone image. But when is the smartphone image good enough?
Smartphone cameras are not only catching up with traditional cameras, they're actually exceeding them in many ways. Take for example...
The image below exemplifies an interesting use of computational blur. The camera has chosen to keep much of the subject—like the front speaker cone, which has significant depth to it—in focus, while blurring the rest of the scene significantly. In fact, if you look at the upper right front of the speaker cabinet, you'll see a good portion of it in focus. After a certain point, the cabinet suddenly-yet-gradually blurs significantly.
The camera and software has chosen to keep a significant depth-of-focus around the focus plane before blurring objects far enough away from the focus plane significantly. That's the beauty of computational approaches: while F1.2 lenses can usually only keep one eye in focus—much less the nose or the ear—computational approaches allow you to choose how much you wish to keep in focus even if you wish to blur the rest of the scene to a degree where traditional optics wouldn't allow for much of your subject to remain in focus.
Surprise and delight...
Digital assistants. Love them or hate them, they will be a part of your future, and they're another way in which smartphone photography augments and exceeds traditional photography approaches. My smartphone is always on me, and when I have my full-frame Sony a7R III with me, I often transfer JPEGs from it to my smartphone. Those images (and 720p video proxies) automatically upload to my Google Photos account. From there any image or video that has my or my daughter's face in it automatically gets shared with my wife without my so much as lifting a finger.
Better yet? Often I get a notification that Google Assistant has pulled a cute animated GIF from my movie it thinks is interesting. And more often than not, the animations are adorable:
|Splash splash! in Xcaret, Quintana Roo, Mexico. Animated GIF auto-generated from a movie shot on the Pixel 2.|
Machine learning allowed Google Assistant to automatically guess that this clip from a much longer video was an interesting moment I might wish to revisit and preserve. And it was right. Just as it was right in picking the moment below, where my daughter is clapping in response to her cousin clapping at successfully feeding her... after which my wife claps as well.
Claps all around!
|Google Assistant is impressive in its ability to pick out meaningful moments from photos and videos. Apple takes a similar approach in compiling 'Memories'.|
But animated GIFs aren't the only way Google Assistant helps me curate and find the important moments in my life. It also auto-curates videos that pull together photos and clips from my videos—be it from my smartphone or media I've imported from my camera—into emotionally moving 'Auto Awesome' compilations:
At any time I can hand-select the photos and videos, down to the portions of each video, I want in a compilation—using an editing interface far simpler than Final Cut Pro or Adobe Premiere. I can even edit the auto-compilations Google Assistant generates, choosing my favorite photos, clips and music. And did you notice that the video clips and photos are cut down to the beat in the music?
This is a perfect example of where smartphone photography exceeds traditional cameras, especially for us time-starved souls that hardly have the time to download our assets to a hard drive (not to mention back up said assets). And it's a reminder that traditional cameras that don't play well with such automated services like Google and Apple Photos will only be left behind simpler services that surprise and delight a majority of us.
The future is bright
This is just the beginning. The computational approaches Apple, Google, Samsung and many others are taking are revolutionizing what we can expect from devices we have in our pockets, devices we always have on us.
Are they going to defy physics and replace traditional cameras tomorrow? Not necessarily, not yet, but for many purposes and people, they will offer pros that are well-worth the cons. In some cases they offer more than we've come to expect of traditional cameras, which will have to continue to innovate—perhaps taking advantage of the very computational techniques smartphones and other innovative computational devices are leveraging—to stay ahead of the curve.
But as techniques like HDR+ and Portrait Mode and Portrait Lighting have shown us, we can't just look at past technologies to predict what's to come. Computational photography will make things you've never imagined a reality. And that's incredibly exciting.
Appendix: Studio Scene
We've added the Google Pixel 2 and Apple iPhone X to our studio scene widget. You can compare the Daylight and Low Light scenes below, keeping in mind that we shot the smartphones in their default camera apps without controlling exposure to see how they would perform in these light levels (10 and 3 EV, respectively, for Daylight and Low Light).
Note that we introduced some motion into the Low Light scene to simulate what the iPhone does when there's movement in the scene. Hence, the ISO 640, 1/30s iPhone X image is more reflective of low light image quality for scenes that can't be shot at the 1/4s shutter speed (ISO 125) the iPhone X will tend to drop to for completely static (tripod-based) low light scenes.
The Pixel 2 rarely drops to shutter speeds slower than 1/30s in low light, yet impressively almost matches the performance of a 1"-type sensor at these shutter speeds in low light (though the 'i' tab shows the RX100 shot at 1/6s F4, you'd get an equivalent exposure at 1/30s were you to shoot the Sony at F1.8 like the Pixel 2).
|Tomatillos and Red Jalepeno Peppers by Kukla|
from Herbs, spices, sauces, or condiments
|Little Kickers by Gfancher|
from sports T&I including banners
|ABD_9819 Bald Eagle by kasjun|
from A big year - birds 2018
|the eyes by aarif|
from -Portrait Challenge 2018-
At this year's CP+ show in Yokohama, we sat down with senior executives from several major manufacturers, including Canon. Topics of conversation included Canon's ambitions for high-end mirrorless cameras, and the importance of responding to the demands of the smartphone generation.
We were recently able to follow local frame builder Max Kullaway as he created one of his AirLandSea bikes. Here are our picks of the photos we got, as the project progressed from bare tubes all the way to rideable bicycle.
On paper, the Sony a7 III is a tempting option for photographers who've been considering a switch to full-frame mirrorless. But how does its image quality stack up? We compare it to the Mark II and a few of its other peers.
Erez Marom shares the details behind this beautiful aurora photograph, captured on Haukland Beach in the Lofoten Islands, Arctic Norway, on a moonless evening.
Google Lens uses artificial intelligence and 'computer vision' to identify and provide information about businesses, landmarks and other objects using your phone's camera. And now it's available for iPhone users, too.
The company posted a record quarterly revenue of $2.08 billion for the first quarter of the 2018 fiscal year. That represents incredibly healthy year-over-year growth of 24 percent.
In the job posting, the Times' describes this role as "one of the most important and high-profile jobs in visual journalism." If you're looking for a high profile job in photojournalism, you could do a lot worse than being Photo Director at The Gray Lady.
According to a recent report out of South Korea, Samsung is increasing production of its ISOCELL image sensors in a bid towards market leadership for image sensors. To reach this goal, Samsung will have to dethrone current market leader Sony... no small task.
In this video, large format photographer Ben Horne shows off the incredible resolving power of 8x10 slide film by pixel peeping a massive 709.6-megapixel drum scan of one of his landscape shots. And you thought 100MP medium format was big...
Photographer Wendy Teal tells the heart-breaking story of a wedding she shot at a hospital on just 24-hours notice. The mother of the bride had been given one week to live, and Wendy responded to the couple's desperate social media plea for someone to capture their special day.
This tiny little plug-and-play VR/AR camera for Android phones uses a pair of greater-than-180° FOV fisheye lenses to offer both 360° video/photo capture and 360° livestreaming at 1440p resolution.
Syrp has announced the Magic Carpet Pro: a slider that offers filmmakers an 'infinitely extendable' range thanks to built-in track levers that let you connect lengths of track without the use of tools.
At CP+ we sat down with executives from several major manufacturers. Among them was Kenji Tanaka, of Sony, who talked to us about the a7 III as well as its plans to attract more pro shooters – without ignoring APS-C and entry-level customers.
How do you shoot macro photography on an 18x24cm large format wet plate camera? You 'connect' two large format cameras together! That's how wet plate photographer Markus Hofstaetter did it, and you can read about the whole process in this article.
The Fujifilm X-H1 is a top-of-the-range 24MP mirrorless camera with in-body stabilization and the company's most advanced array of video capabilities. We've tested the X-T2's big brother extensively to see how it performs.
Motorsports photojournalist Jamey Price recently flew to Canada with Lamborghini for the car company's Winter Accademia 2018, where clients get to drive the latest Lamborghini supercars on snow and ice. Yes... it is exactly as awesome as it sounds.
For the Pixel 2 smartphone's Motion Photos feature, Google built on its existing Motion Stills technology by adding advanced stabilization that combines software and hardware capabilities to optimize trimming and stabilization.
This "high-capacity advanced spider tripod" system can handle a maximum load of 65kg / 143lbs thanks to its reinforced design and 8-layered carbon fiber legs.
Photographer William Briscoe captured the beautiful two-for-one timelapse just outside Fairbanks, Alaska on January 31st, braving -31°F (-35°C) temperatures to get the shot.
"After his camera was stolen from his room in the orphanage, he switched to an iPhone for his photography, reasoning that the image quality of a big, heavy camera was less important than the freedom of a cell phone. 'Quality? Screw it, I’d sketch things with a pencil if I could draw,' he wrote in a blog post."
Chinese manufacturer Vivo has announced some AI-powered Super HDR tech to compete with Google's HDR+ system. Both systems combine multiple images to create a final shot with more dynamic range and less noise, but Super HDR claims to do so more intelligently.
The YouTube channel JerryRigEverything recently tore down (or rather, tore apart...) the new Samsung Galaxy S9, giving us the closest look at yet at the new smartphone's camera hardware.
The Leica l Model A, dating from between 1926 and 1927, comes with a card signed by Earhart herself. Unfortunately, this is the only 'proof' that the camera really did belong to her.
The Rokinon AF 35mm F2.8 FE is a budget-friendly option for users of Sony's a7-series that are looking to get into the 35mm focal length.
The 'semantic image segmentation model' categorizes every pixel in an image and assigns it a label, such as “road”, “sky”, “person” or “dog.” And now, Google has released its latest version as open source, making it available to any developers whose apps could benefit from the tech.
Huawai is teasing the upcoming P20 smartphone's low-light and zoom capabilities in a couple of tongue-in-cheek teaser videos on YouTube.
Fuji's latest firmware update for the GFX 50S adds two new features: a focus stacking mode, and a 35mm format mode that takes 30.5MP photos using the center portion of the camera's medium format sensor.
The crash has raised serious questions about 'startling safety gaps' in the doors-off photo tour industry. After a brief safety video, passengers are strapped in with heavy-duty harnesses and given only a knife to cut themselves loose in case of emergency.
For the first time in five years, Adobe is raising the price of some Creative Cloud subscription packages. The good news for photographers: The $10/month CC Photography plan that includes Photoshop CC, Lightroom CC, and Lightroom Classic CC will stay the same.
In a statement, Canon CEO Fujio Mitarai said the company will "go on the offensive" in mirrorless cameras, aiming to clinch 50% of the entire interchangeable-lens camera market.