How a 1" sensor can beat FF (with computational photography)

DPReview says that the sensor used in the Google Pixel 2 smartphone is 1/2.55", which "can often behave like one nine times its size (approaching Micro 4/3)."
https://www.dpreview.com/reviews/google-pixel-2/3

My own experience with a Pixel 2 (described elsewhere on DPReview forums) agrees with this assessment. Are there any technical (as opposed to business or IP related) reasons why the technology used in the Pixel 2 can't be used in cameras with larger sensors?

If there isn't any reason, then a camera with a 1" sensor using Pixel 2 technology would behave like it had an area of 1,044mm2, which is larger than the area of a conventional FF sensor (864mm2).

A Canon APS-C sensor (329mm2) using Pixel 2 technology would behave like a 2,961mm2 sensor, which is larger than the Phase One P65+ MF sensor (2,177mm2).

A FF camera (864mm2) using Pixel 2 technology would behave like a 7,776mm2 sensor, which is larger than 6x7 MF film (3,920mm2).

4x5 film (11,737mm2) and 8x10 film (51,562mm2) will be safe for a while.

Below is a chart showing more of these calculations for some common sensor (and film) sizes:



Sensor size data from https://en.wikipedia.org/wiki/Image_sensor_format (I added the fourth "Area x9 mm2..." column by multiplying the area mm2 data by nine.)

"Pixel 2 technology" is shorthand for Google's implementation of computational photography in the Pixel 2 (and Pixel 2 XL). There are other benefits for current implementations of computational photography beyond sensor area multiplication, but I'll leave this for another post.

Wayne

Frame stacking works well as long as you don't have significant subject motion, although this technique can handle camera shake fairly well. I use that technique occasionally with my FF Nikon: it is a good alternative to using a dark neutral destiny filter, and it is also useful when I'm trying to get a decent image at night and I don't want to use my tripod.

Basically, the signal-to-noise ratio roughly increases according to the square root of number of images in the stack: four images gives you double the SNR, nine images gives you triple, etc.

One trick found in many smartphones is a low base ISO, which can reduce noise, and Pixel 2 has a base ISO of 50. Other smartphones might have a base ISO of 35 or 25.

Wayne Larmon said:

DPReview says that the sensor used in the Google Pixel 2 smartphone is 1/2.55", which "can often behave like one nine times its size (approaching Micro 4/3)."
https://www.dpreview.com/reviews/google-pixel-2/3

Click to expand...

This is basically a variant of in-camera HDR, and several larger format cameras today offer it - though only in the creation of JPGs. I believe one or two may generate RAWs, which is what you would want.

With certain forms of HDR you are selectively combining areas of the scene and then blending; with others you are averaging the whole frame with removal of clipped or other undesireable portions, then tone-mapping. A benefit of this, particularly as the number of bracketed exposures goes up, is a reduction in observed noise.

I would love if camera makers got fully onboard with this stuff, which is basically nothing more than median stacking the exposures while intelligently avoiding most artifacts.

Maybe an interim solution is for PC software to automate the process using existing raw files captured in high-speed bursts, rather than force yet another computationally intensive task onto camera hardware where compromises would have to be made.

All this would take is for the software to recognize bursts of photos of the same subject, and then silently perform the alignment and stacking (and artifact avoidance), putting the result into the DAM library with a flag indicating that it's not a "straight exposure".

mosswings said:

Wayne Larmon said:

DPReview says that the sensor used in the Google Pixel 2 smartphone is 1/2.55", which "can often behave like one nine times its size (approaching Micro 4/3)."
https://www.dpreview.com/reviews/google-pixel-2/3

Click to expand...

This is basically a variant of in-camera HDR, and several larger format cameras today offer it - though only in the creation of JPGs. I believe one or two may generate RAWs, which is what you would want.

With certain forms of HDR you are selectively combining areas of the scene and then blending; with others you are averaging the whole frame with removal of clipped or other undesireable portions, then tone-mapping. A benefit of this, particularly as the number of bracketed exposures goes up, is a reduction in observed noise.

Click to expand...

The Pixels don't work like other cameras do

1. "Frame stacking" is much more sophisticated. It has a rolling buffer of (currently) nine frames. When the shutter is pressed it goes with the most recent frames. A reference frame is chosen (the sharpest). It then breaks each frame down into thousands of tiles and aligns each tile with the corresponding tile in the reference frame. If a tile can't be aligned then it is discarded. This process reduces or eliminates motion blur. The tiles for each frame are combined into a single raw image that has reduced noise (because of stacking.)
2. Google implements HDR by underexposing all nine frames by several stops and then tone mapping the single combined frame. Because the noise has been reduced by several stops, lifted shadows are more usable. Also, underexposing allows for faster shutter speeds.
3. They do not bracket exposures at all. Each frame is metered the same.

The pixel "sensor can often behave like one nine times its size" when there is minimal motion blur. However, in actual practice it works surprisingly well and most images I've taken with my Pixel 2 look like they were taken with a camera that had a larger sensor. I've been taking images with my Pixel 2 and my Canon M6. Most of the time (processed raw) M6 images don't look that much better when examined at the pixel level. Plus or minus accounting for the resolution difference. My results correspond to what DPReview has reported.

The Pixel 2 processing pipeline would be less advantageous for people that shoot multiple frames per second. But having that pipeline wouldn't hurt. If you weren't shooting multiple frames per second you'd get the all the benefits of the process. If you shot faster than the process could keep up with, then less frames would be buffered and you might end up with a single frame.

The worst case (single frame buffered) is that the camera's sensor has no area multiplication. The best case is effective sensor area multiplied by however many frames could be buffered and combined. I see using a variant of the Pixel 2 processing process as no loss and a potential massive gain.

My explanation, above, is a gross simplification of Google Research's 12 page white paper:

Burst photography for high dynamic range and low-light imaging on mobile cameras (PDF)

Wayne

Wayne Larmon said:

mosswings said:

Wayne Larmon said:

DPReview says that the sensor used in the Google Pixel 2 smartphone is 1/2.55", which "can often behave like one nine times its size (approaching Micro 4/3)."
https://www.dpreview.com/reviews/google-pixel-2/3

Click to expand...

This is basically a variant of in-camera HDR, and several larger format cameras today offer it - though only in the creation of JPGs. I believe one or two may generate RAWs, which is what you would want.

With certain forms of HDR you are selectively combining areas of the scene and then blending; with others you are averaging the whole frame with removal of clipped or other undesireable portions, then tone-mapping. A benefit of this, particularly as the number of bracketed exposures goes up, is a reduction in observed noise.

Click to expand...

The Pixels don't work like other cameras do

1. "Frame stacking" is much more sophisticated. It has a rolling buffer of (currently) nine frames. When the shutter is pressed it goes with the most recent frames. A reference frame is chosen (the sharpest). It then breaks each frame down into thousands of tiles and aligns each tile with the corresponding tile in the reference frame. If a tile can't be aligned then it is discarded. This process reduces or eliminates motion blur. The tiles for each frame are combined into a single raw image that has reduced noise (because of stacking.)
2. Google implements HDR by underexposing all nine frames by several stops and then tone mapping the single combined frame. Because the noise has been reduced by several stops, lifted shadows are more usable. Also, underexposing allows for faster shutter speeds.
3. They do not bracket exposures at all. Each frame is metered the same.

The pixel "sensor can often behave like one nine times its size" when there is minimal motion blur. However, in actual practice it works surprisingly well and most images I've taken with my Pixel 2 look like they were taken with a camera that had a larger sensor. I've been taking images with my Pixel 2 and my Canon M6. Most of the time (processed raw) M6 images don't look that much better when examined at the pixel level. Plus or minus accounting for the resolution difference. My results correspond to what DPReview has reported.

The Pixel 2 processing pipeline would be less advantageous for people that shoot multiple frames per second. But having that pipeline wouldn't hurt. If you weren't shooting multiple frames per second you'd get the all the benefits of the process. If you shot faster than the process could keep up with, then less frames would be buffered and you might end up with a single frame.

The worst case (single frame buffered) is that the camera's sensor has no area multiplication. The best case is effective sensor area multiplied by however many frames could be buffered and combined. I see using a variant of the Pixel 2 processing process as no loss and a potential massive gain.

My explanation, above, is a gross simplification of Google Research's 12 page white paper:

Burst photography for high dynamic range and low-light imaging on mobile cameras (PDF)

Wayne

Click to expand...

I'll read the white paper. That is a significant difference, and it also points out why a lot of research is being directed towards high frame rate conversion.

desertsp said:

I would love if camera makers got fully onboard with this stuff, which is basically nothing more than median stacking the exposures while intelligently avoiding most artifacts.

Maybe an interim solution is for PC software to automate the process using existing raw files captured in high-speed bursts, rather than force yet another computationally intensive task onto camera hardware where compromises would have to be made.

All this would take is for the software to recognize bursts of photos of the same subject, and then silently perform the alignment and stacking (and artifact avoidance), putting the result into the DAM library with a flag indicating that it's not a "straight exposure".

Click to expand...

I'd rather they use dedicated hardware that already exists


Google Pixel Visual Core Image Processing Unit (IPU)

to do processing in camera. Google already does most of the Pixel 2 processing on multiple raw files in-camera and only combines the multiple raw files near the end of the processing pipeline. They could save the combined raw file so that photographers could process it in PP. In addition to saving the fully processed JPEG.

However

A Google employee discusses using HDR+ for DSLRs
HDR+ Pipeline Implementation
...same high level pipeline to raw images off a Canon 5D III DSLR, with modified algorithms.
http://timothybrooks.com/tech/hdr-plus/

has already been done. He processed multiple stock 5D III raw files to make a single image.

I'd just as soon not have to deal with multiple raw files for each image. The Pixel 2 in camera processing is already good enough that I don't miss having raw files. Most of the time. I would like to have a single raw file for the rare times where I disagree with the in-camera processing.

Wayne

Finding matching blocks is already done for video coding. Reverse those motion vectors to align that video frame to the reference frame and average (only do that if the residuals for the block are in line with the selected ISO setting). And if there are very low freqency components in th eresidual, those may be caused by lighting changes - these could also be applied in reverse.

So, parts of a standard MPEG encoder could be worked into something that at least aligns subparts of each image.

Note that one can improve all frames individually by selecting the desired reference frame..

Wayne Larmon said:

Burst photography for high dynamic range and low-light imaging on mobile cameras (PDF)

Click to expand...

The paper mentions using raw-level dehazing (section 6 "Finishing", operation #8) to reduce the impact of veiling glare, which you presume would be a significant issue in a mobile phone lens. I wonder if future image processing in DSLRs will also apply raw-level dehazing in high-contrast situations?

Dan.

Wow that's really cool!

Way beyond my own abilities...someone needs to make a Lightroom plugin or something lol

desertsp said:

I would love if camera makers got fully onboard with this stuff, which is basically nothing more than median stacking the exposures while intelligently avoiding most artifacts.

Maybe an interim solution is for PC software to automate the process using existing raw files captured in high-speed bursts, rather than force yet another computationally intensive task onto camera hardware where compromises would have to be made.

All this would take is for the software to recognize bursts of photos of the same subject, and then silently perform the alignment and stacking (and artifact avoidance), putting the result into the DAM library with a flag indicating that it's not a "straight exposure".

Click to expand...

This is implemented today on Sony bodies using the Smooth Reflections app - unfortunately it doesn't support artifact removal. More unfortunate still is that Sony appears to be moving away from their app platform.

According to Arsenal's founder, it will have stacking capabilities. It's unclear where exactly they'll implement it. If it's in the smartphone then it will be limited by the WiFi transfer speed. If they instead manage to implement it on the device side that connects to the camera via USB then it would be a usable solution.

https://witharsenal.com/

Horshack said:

According to Arsenal's founder, it will have stacking capabilities. It's unclear where exactly they'll implement it. If it's in the smartphone then it will be limited by the WiFi transfer speed. If they instead manage to implement it on the device side that connects to the camera via USB then it would be a usable solution.

https://witharsenal.com/

Click to expand...

I've studied the Pixel 2 because there is a lot of material to study. DPReview has covered it with at least two feature articles. Google Research has lots of background design information.

Does Arsenal have similar design info available? The web site seems kind of hand wavey.

I'm not asking from the perspective of a Pixel 2 fanboy. I'm really psyched about computational photography after experiencing it with my Pixel 2 and seeing how many OTC Pixel 2 JPEGs are close to the best I can pull out of M6 raw files.

The point of my OP was to find out more about what is possible with computational photography using hardware/software that either exists now, or will likely exist in the foreseeable future. The Pixel 2 has drawn a bright line in the sand because it exists now and performs very well.

The more computational photography, the better, no matter which companies make it.

Wayne

Wayne Larmon said:

Horshack said:

According to Arsenal's founder, it will have stacking capabilities. It's unclear where exactly they'll implement it. If it's in the smartphone then it will be limited by the WiFi transfer speed. If they instead manage to implement it on the device side that connects to the camera via USB then it would be a usable solution.

https://witharsenal.com/

Click to expand...

I've studied the Pixel 2 because there is a lot of material to study. DPReview has covered it with at least two feature articles. Google Research has lots of background design information.

Does Arsenal have similar design info available? The web site seems kind of hand wavey.

Click to expand...

The only information I've seen related to stacking are the tidbits I mentioned. I agree, the details are scarce, which is why I chose wait until others try and review it rather than pre-ordering myself.

Wayne Larmon said:

I'm not asking from the perspective of a Pixel 2 fanboy. I'm really psyched about computational photography after experiencing it with my Pixel 2 and seeing how many OTC Pixel 2 JPEGs are close to the best I can pull out of M6 raw files.

The point of my OP was to find out more about what is possible with computational photography using hardware/software that either exists now, or will likely exist in the foreseeable future. The Pixel 2 has drawn a bright line in the sand because it exists now and performs very well.

The more computational photography, the better, no matter which companies make it.

Click to expand...

Btw, another option is qDslrDashboard, which has remote control (USB and Wifi) and stacking capability. However, the app can be really finicky.

I'm a big fan of the promise of computational photography as well. Smartphones are well suited to the task because they have very powerful processors (relative to the embedded processors in cameras) and are really pushing the envelope due to hyper-competition in that industry. I'm less enthusiastic about this trickling up into standalone-cameras - their business models mete out improvements to keep upgrade cycles in check, so quantum leaps from computational methods might throw a monkey wrench into their plans.

Horshack said:

Wayne Larmon said:

Horshack said:

According to Arsenal's founder, it will have stacking capabilities. It's unclear where exactly they'll implement it. If it's in the smartphone then it will be limited by the WiFi transfer speed. If they instead manage to implement it on the device side that connects to the camera via USB then it would be a usable solution.

https://witharsenal.com/

Click to expand...

I've studied the Pixel 2 because there is a lot of material to study. DPReview has covered it with at least two feature articles. Google Research has lots of background design information.

Does Arsenal have similar design info available? The web site seems kind of hand wavey.

Click to expand...

The only information I've seen related to stacking are the tidbits I mentioned. I agree, the details are scarce, which is why I chose wait until others try and review it rather than pre-ordering myself.

Horshack said:

I'm not asking from the perspective of a Pixel 2 fanboy. I'm really psyched about computational photography after experiencing it with my Pixel 2 and seeing how many OTC Pixel 2 JPEGs are close to the best I can pull out of M6 raw files.

The point of my OP was to find out more about what is possible with computational photography using hardware/software that either exists now, or will likely exist in the foreseeable future. The Pixel 2 has drawn a bright line in the sand because it exists now and performs very well.

The more computational photography, the better, no matter which companies make it.

Click to expand...

Btw, another option is qDslrDashboard, which has remote control (USB and Wifi) and stacking capability. However, the app can be really finicky.

I'm a big fan of the promise of computational photography as well. Smartphones are well suited to the task because they have very powerful processors (relative to the embedded processors in cameras) and are really pushing the envelope due to hyper-competition in that industry. I'm less enthusiastic about this trickling up into standalone-cameras - their business models mete out improvements to keep upgrade cycles in check, so quantum leaps from computational methods might throw a monkey wrench into their plans.

Click to expand...

I also think that traditional cameras are not as well suited for this sort of stuff - certainly DSLRs. When you're talking hyperrapid burst shooting and combining, mirrorless products are a more natural fit. Also when you consider that even though contemporary cameras are intensely electronic, the traditional manufacturers still seem to think of themselves as lens manufacturers.

That’s a good point about the lenses. Do we know how much profit is made from lenses vs bodies?

And I wonder if the main roadblock is a lack of talent (it’s all siphoned up by the phone makers?), lack of demand (so many people here seem very afraid of or disinterested in computational photography), or the laws of diminishing returns (it doesn’t help a big sensor as much as a small one, and a big sensor is still better than a small one “boosted” by computational techniques).

Or maybe a combination of those plus other reasons.

mosswings said:

Horshack said:

Btw, another option is qDslrDashboard, which has remote control (USB and Wifi) and stacking capability. However, the app can be really finicky.

Click to expand...

Click to expand...

I didn't know about this project. But it doesn't look like they are doing any kind of computational photography. But it is interesting nevertheless.

mosswings said:

mosswings said:

I'm a big fan of the promise of computational photography as well. Smartphones are well suited to the task because they have very powerful processors (relative to the embedded processors in cameras) and are really pushing the envelope due to hyper-competition in that industry. I'm less enthusiastic about this trickling up into standalone-cameras - their business models mete out improvements to keep upgrade cycles in check, so quantum leaps from computational methods might throw a monkey wrench into their plans.

Click to expand...

I also think that traditional cameras are not as well suited for this sort of stuff - certainly DSLRs. When you're talking hyperrapid burst shooting and combining, mirrorless products are a more natural fit. Also when you consider that even though contemporary cameras are intensely electronic, the traditional manufacturers still seem to think of themselves as lens manufacturers.

Click to expand...

We can't get away from business models when talking about what is possible with computational photography. The interview with Shigemi Sugimoto, Head of Olympus's imaging business unit that DPReview just put up doesn't give much hope. He only talks about traditional, well, lens considerations as the kind of innovations that Olympus is planning (4/3's 2X crop gives more reach; they will be releasing improved IS, etc.)

Which is a pity because the chart I displayed in my OP shows that a 4/3 sensor would have an effective area of 2,025mm2, if the camera used the same technology that the Pixel 2 does. 4/3 is all we need.

Maybe. Mosswings, have you read the Google Research paper I linked to earlier? If so, have you found any reasons why this technology can't be used with larger sensors? (Other than sluggish business models.)

Wayne

Wayne Larmon said:

mosswings said:

Horshack said:

Btw, another option is qDslrDashboard, which has remote control (USB and Wifi) and stacking capability. However, the app can be really finicky.

Click to expand...

Click to expand...

I didn't know about this project. But it doesn't look like they are doing any kind of computational photography. But it is interesting nevertheless.

Wayne Larmon said:

Wayne Larmon said:

I'm a big fan of the promise of computational photography as well. Smartphones are well suited to the task because they have very powerful processors (relative to the embedded processors in cameras) and are really pushing the envelope due to hyper-competition in that industry. I'm less enthusiastic about this trickling up into standalone-cameras - their business models mete out improvements to keep upgrade cycles in check, so quantum leaps from computational methods might throw a monkey wrench into their plans.

Click to expand...

I also think that traditional cameras are not as well suited for this sort of stuff - certainly DSLRs. When you're talking hyperrapid burst shooting and combining, mirrorless products are a more natural fit. Also when you consider that even though contemporary cameras are intensely electronic, the traditional manufacturers still seem to think of themselves as lens manufacturers.

Click to expand...

We can't get away from business models when talking about what is possible with computational photography. The interview with Shigemi Sugimoto, Head of Olympus's imaging business unit that DPReview just put up doesn't give much hope. He only talks about traditional, well, lens considerations as the kind of innovations that Olympus is planning (4/3's 2X crop gives more reach; they will be releasing improved IS, etc.)

Which is a pity because the chart I displayed in my OP shows that a 4/3 sensor would have an effective area of 2,025mm2, if the camera used the same technology that the Pixel 2 does. 4/3 is all we need.

Maybe. Mosswings, have you read the Google Research paper I linked to earlier? If so, have you found any reasons why this technology can't be used with larger sensors? (Other than sluggish business models.)

Wayne

Click to expand...

I did read it, and my note above about mirrorless cameras being more amenable to this than DSLRs follows from my reading.

There's nothing preventing larger format cameras from using this, particularly if they're able to rapidly capture frames. A DSLR is not inherently suited for continuous burst mode operation, and would be horribly intrusive if it did so. Hence this technology is optimal for quiet-shutter products, especially those with global shutter. Since a smartphone incorporates a mirrorless camera, this type of camera would be the most likely to incorporate it.

HDR+ replaces the entire RAW imaging chain of a camera, including distortion correction. This gets into very sensitive areas of a manufacturer's IP, as the author noted. This is a matter of licensing, but it's an open question as to whether an insular company would want to pay the royalties for this.

HDR+ isn't perfect across the frame, which is what traditional camera companies strive for with superior lenses and such. This doesn't mean that HDR+ wouldn't be of benefit, just that a large-format HDR+ would be of a higher grade than what is in the PIxel in order to deliver expected quality images at the expected rate. 4 seconds for a picture is a long time to wait when you're expecting 8 or higher frames per second to be pumped to the memory card.

My point? HDR+ serves the needs of the smartphone shooter quite well. The smartphone shooter isn't pushing into those areas that traditional cameras do well - fast burst mode shooting of rapidly moving subjects with extremely high image quality, made possible by large areas and fast lenses. It's great for the typical smartphone single shot. I'm being a bit glib here, but I align smartphones functionally with cameras like an EM10 III or something similar - great for casual shooting of slow-moving subjects. This is not to say that HDR+ is incapable of satisfying results for the larger format shooter. I can see that it would be. Just that, like any computational photography application, it takes time and processor power to execute. Traditional cameras promise extremely rapid response.

I note also that HDR+ varies only shutter speed, if I read correctly. The lens aperture is fixed, as is the case for smartphones. This is a great boon for frame alignment. Imagine having to align images taken at a myriad of apertures. The pyramids would get quite fuzzy. There's a great opportunity for research here.

Another question I would have is - how does HDR+ fit in to the growing trend for fast-frame-rate video capture? This is being contemplated for 8K applications, but also for frame stacking and optimal-frame-selection features in still photos; effectively, the camera is always taking a video and selecting the shot after the fact. HDR+ does operate something like this, but not continuously. Probably this is an easy adaptation.

Finally - the "underexposed" protocol used is reminiscent of QIS' extremely high frame rate capture. In both, the goal is the same: avoid clipping. But noise reduction proceeds as the square root of the number of frames combined, and HDR+ starts off at a deficit.

HDR+ is a very clever technique that reduces some very expensive computational methods to practical use. For that the creators should be applauded.

mosswings said:

I note also that HDR+ varies only shutter speed, if I read correctly. The lens aperture is fixed, as is the case for smartphones. This is a great boon for frame alignment. Imagine having to align images taken at a myriad of apertures. The pyramids would get quite fuzzy. There's a great opportunity for research here.

Click to expand...

Actually, aperture stacking is a known technique.

Basically, it smooths your out-of-focus blur, similar to what you get with an apodization lens such as the Minolta STF 135mm f/2.8 [T4.5], now sold by Sony, the Laowa 105mm f/2 Smooth Trans Focus (STF), and the FUJINON XF56mmF1.2 R APD, and there was a film camera in the 1990s that would do aperture stacking as well. This technique reduces or eliminates harsh bokeh found with many lenses otherwise well-corrected for spherical aberration. You get a T-Stop which is rather higher than the f/stop, and less depth of field that then the f/stop would indicate, but the smoothness of the bokeh is so good that some even think that it looks fake.

The Greek-derived word 'apodization' mean 'cutting off the foot', and what's being cut off here is the foot of diffraction blur or the "point spread function", which resembles a bullseye target. Basically, it makes diffraction weaker and more tractable, and so sharpening becomes more effective on diffracted images.

I inadvertently discovered the technique when I was doing shutter speed stacking of waterfalls: I wanted water that looked smooth but also still had a residual crispness to it, so I'd take several photos varying the shutter speed and stack them together.

--
http://therefractedlight.blogspot.com

Mark Scott Abeln said:

mosswings said:

I note also that HDR+ varies only shutter speed, if I read correctly. The lens aperture is fixed, as is the case for smartphones. This is a great boon for frame alignment. Imagine having to align images taken at a myriad of apertures. The pyramids would get quite fuzzy. There's a great opportunity for research here.

Click to expand...

Actually, aperture stacking is a known technique.

Basically, it smooths your out-of-focus blur, similar to what you get with an apodization lens such as the Minolta STF 135mm f/2.8 [T4.5], now sold by Sony, the Laowa 105mm f/2 Smooth Trans Focus (STF), and the FUJINON XF56mmF1.2 R APD, and there was a film camera in the 1990s that would do aperture stacking as well. This technique reduces or eliminates harsh bokeh found with many lenses otherwise well-corrected for spherical aberration. You get a T-Stop which is rather higher than the f/stop, and less depth of field that then the f/stop would indicate, but the smoothness of the bokeh is so good that some even think that it looks fake.

Click to expand...

One of the problems with all the knowledge in the world is that it is impossible to know all the knowledge in the world. At least in one small part of it, Mark, you seem to have

Thanks! The world is filled with such exciting things to learn.

mosswings said:

HDR+ is a very clever technique that reduces some very expensive computational methods to practical use. For that the creators should be applauded.

Click to expand...

Let's say, for the sake of argument, that Google continues open sourcing their IP and maintains their

We hope that publicly sharing our system with the community will make it easier for other groups in academia and industry to reproduce and further improve upon state-of-art systems, train models on new datasets, and envision new applications for this technology.

stance. And makes their Pixel Visual Core IPU (Image Processing Unit) chip available at a reasonable cost. Remember that Google's business model involves giving stuff away--they only want clicks in exchange.

In which case the IP and processing hardware would be freely available. If any conventional camera design incorporated this technology, the worst case (shooting multiple frames a second) would mean that nothing changes: you get the same performance that you get now processing a single frame at a time.

But if you can shoot at a slower pace, then you get the full benefits. Landscape photographers, for example, would get medium format sensor performance at 35mm prices. People capturing memories generally don't need frames-per-second.

This aside, I want to discuss another benefit of computational photography: using depth maps to emulate shallow DOF (bokeh) The Pixel 2 does this (Portrait Mode) but doesn't let you adjust anything. The iPhone X is better in that it can save the depth map and you can use it to adjust DOF and lens type in PP. DPReview's Rishi Sanyal covered this in an Interview he did with Leo Laporte. Leo demonstrated the Focos app and was quite blown away with the lens emulations. At one point he referred to dialing in an Otus.

Unfortunately you need to dig into the video to see this. Open the video here and scroll to about 29:00. (The full interview with Rishi starts around 11:00.) Rishi did a text summary of the interview but didn't cover bokeh emulation extensively in the text. (He also discusses lens emulation earlier in the interview. It helps to watch the whole interview.)

Note that we aren't talking about smearing a simple Gaussian blur around. We are talking about emulating the lens characteristics of different real world lenses (Otus...) I don't know how successful this is because (right now) this isn't available on the Pixel 2. But the Pixel 2's lens emulation is no slouch; Rishi compares Pixel 2 emulated shallow DOF to a FF (Sony) body with fast glass shallow DOF. He shows this comparison in both the interview and in the text summary, with a "can you tell the difference?" comparison.

For lens emulation this is a case where the smaller the sensor, the better. We want high DOF to start with. We don't want any fast lenses contaminating the deep DOF.

Assuming that lens emulation is real, then doing shallow DOF with depth map techniques is better than doing it with fast glass, because you can paint the part that you want in focus, after the fact, in PP.

Well, speaking as a lens emulation purist. In practical terms, I'm sure that real world 35mmish lenses would work fine. Note that I linked to Google open sourcing this "Portrait Mode" technology above.

Also note that Google derives the depth maps from a dual pixel sensor, similar to the sensors used in Canon's DPAF sensors. So adapting this processing to conventional cameras shouldn't require heroic measures.

Wayne