Can pixel shift increase resolution?

If you mean pixel shift as capturing all three color channels at all pixel locations - it depends, for instance on scene content, but in practice it makes very little difference.
It makes a substantial difference. First there is the benefit of a longer synthetic exposure created by stacking 4 images. It reduces the impact of shot noise in the final resulting merged image.

Red and blue are sampled at twice the frequency and green at 1.4x the frequency. So seams between different colors are captured more distinctly.

As if that were not enough, It all but eliminates color aliasing / moire.

For any static subject where you can use a tripod or camera stand, it is always worth the effort.

As for slanted edge tests, and so on and so forth, and general sophistry in the subject, merely look at the studio scene comparison in DPR with / without it enabled. Don't accept our pontifications on the subject, myself included, DPT gives you abundant evidence as to the practical effect and value of this feature.

As an example, a 400% crop of a micrograph of a flatbed scanner linear CCD sensor. You see the blue and green color filters. Left is standard capture, right is pixel shift ( this is from a Pentax K-1 ) - it is very visibly effective. Its not a gimmick, I use is wherever it is practical.
Hi Bob, I mostly agree with you per my posts in this thread, though I take a more nuanced view of the practical benefits for your average photographer. For context see:

https://www.strollswithmydog.com/bayer-cfa-effect-on-sharpness/

As a fun counterexample, which of these two unsharpened images from raw files captured with the same camera, lens and setup shows higher resolution and less aliasing? One is pixel shift, the other not.

469f11616e9b4f3391c8c3e510f92d45.jpg.png

For those of you who figured it out, how hard did you have to look and what gave it away?

Jack
Hi Jack,

I guess this is one of your monochrome conversion tricks? I see little difference.

In the end, I would think that the image on the right is pixel shift, but I don't see the usual 'give aways'.

Best regards

Erik

--
Erik Kaffehr
Website: http://echophoto.dnsalias.net
Magic uses to disappear in controlled experiments…
Gallery: http://echophoto.smugmug.com
Articles: http://echophoto.dnsalias.net/ekr/index.php/photoarticles
 
Sure JACS, 'educated guesses' would be better wording than 'knowledge'. Though as you say all demosaicers have to make such guesses by definition. Surprisingly (?), in general photographers appear to be pretty happy about the guesses that they make: excluding AI, algorithms in commercial raw converters seem to have remained relatively stable over the last decade, a question of diminishing returns I suspect.
This moves the discussion to a different corner. I am satisfied with today's RAW converters as well. This is not what we were discussing, however. You suggested a new algorithm which does not exist commercially or even on a research level: to look for monochromatic or almost so areas somehow (by making some guesses) and using this to diminish the increased resolution of pixel shift to almost nothing, at least in those aeras of the image.

Here are three crops. The first one is just the jpeg converted from RAW, posted by Dpreview (A7IVR). The second one is converted by your method, the same crop I posted before. I uses the a priori knowledge that this part of the scene must be monochromatic (and the rest of the image, not shown, is a complete mess). The third one is the pixel shift image, as posted by Dpreview as a jpeg. The first two are upsized to 200% to match the third one. I did not match the contast well.

standard JPEG converted from RAW

standard JPEG converted from RAW

converted with the a priori knowledge that this crop must be monochrome

converted with the a priori knowledge that this crop must be monochrome

pixel shift

pixel shift

And here is the pixel shift compared to PO 150mp.

Just use the arrow keys in your keyboard in full size view to flip between them.

It is not even funny how wrong the first two conversions are. On the other hand, people happily buy and consume fake stuff...
 
Good show JACS, though the 200% images are one step removed from what my comments have been about throughout this thread: 4 capture pixel shift vs single capture, which remove effective pixel pitch as a variable and thus can be indicative of the difference in performance of Bayer CFA vs monochrome sensors. The difference can be hard to spot in the conditions shown earlier.

On the other hand, doubling the number of linear captures, pixel pitch and hence image size as in your example just above does indeed increase rendered resolution as well as reduce aliasing further, as mentioned upthread. The effect is quite obvious.

One more note on B&W honesty before leaving this interesting subject.
J A C S Wrote :

Patterns on the cloths and on the faces look good, actually, better than the demosaiced versions I have seen. So, this would be the honest representation of a B&W image if we knew it is B&W.
It is 'a' B&W representation. And this begs the question of what would be an honest one.

One way to answer it is to assume that honest B&W is what a monochromat would see, let's say in daylight conditions. Let's further assume that the monochromat only has the cone type that is sensitive to Medium wavelengths. That means that such a person effectively sees something proportional to Luminance, because that cone type has spectral sensitivity practically equivalent to the relative luminosity function (V).

On the other hand a typical Bayer CFA sensor has three types of color filters with peaks in what we normally refer as the R, G and B wavelengths - none of which are even close to the luminosity function. But let's say that we take the G channel as being equivalent to V, thus honestly (?) taking care of about half of the pixels in our B&W rendering. After all, the G plane is usually where cameras set their focus on.

The other half of pixels, R&B, have sensitivity functions that are even further removed from V. But let's give them the benefit of the doubt: after white balancing, if the illuminant is uniform and the scene is neutral, we could get intensity somewhat proportional to the others, right?

The R and B pixels are however on different color planes. That means that they are most likely out of focus and more aberrated (e.g. lateral chromatic) compared to G. So what we get is truly a Frankenstein of a B&W rendering, which its proud parents try to pass on as the real thing. Honest :-)

Jack

PS By these standards a pixel shifted render is not very honest either. In fact no render with commercial sensors is. But of course they are good enough for most purposes and we are all happy with the results.
 
Last edited:
Good show JACS, though the 200% images are one step removed from what my comments have been about throughout this thread: 4 capture pixel shift vs single capture, which remove effective pixel pitch as a variable and thus can be indicative of the difference in performance of Bayer CFA vs monochrome sensors. The difference can be hard to spot in the conditions shown earlier.
The OP mentioned the A7RIV. I do not know if the K1 does things differently.
On the other hand, doubling the number of linear captures, pixel pitch and hence image size as in your example just above does indeed increase rendered resolution as well as reduce aliasing further, as mentioned upthread. The effect is quite obvious.

One more note on B&W honesty before leaving this interesting subject.
J A C S Wrote :

Patterns on the cloths and on the faces look good, actually, better than the demosaiced versions I have seen. So, this would be the honest representation of a B&W image if we knew it is B&W.
It is 'a' B&W representation. And this begs the question of what would be an honest one.

One way to answer it is to assume that honest B&W is what a monochromat would see, let's say in daylight conditions. Let's further assume that the monochromat only has the cone type that is sensitive to Medium wavelengths. That means that such a person effectively sees something proportional to Luminance, because that cone type has spectral sensitivity practically equivalent to the relative luminosity function (V).

On the other hand a typical Bayer CFA sensor has three types of color filters with peaks in what we normally refer as the R, G and B wavelengths - none of which are even close to the luminosity function. But let's say that we take the G channel as being equivalent to V, thus honestly (?) taking care of about half of the pixels in our B&W rendering. After all, the G plane is usually where cameras set their focus on.

The other half of pixels, R&B, have sensitivity functions that are even further removed from V. But let's give them the benefit of the doubt: after white balancing, if the illuminant is uniform and the scene is neutral, we could get intensity somewhat proportional to the others, right?

The R and B pixels are however on different color planes. That means that they are most likely out of focus and more aberrated (e.g. lateral chromatic) compared to G. So what we get is truly a Frankenstein of a B&W rendering, which its proud parents try to pass on as the real thing. Honest :-)
It would be an honest representation of what is in the data.

Now, B&W renderings do not need to reproduce what color blind people see but this is a different conversation.
 
Last edited:
Today most cameras do not resolve the Nyquist limit of the sensor anyway due to limitations in the lens
I can't tell precisely what you mean by that sentence.

If you mean that real lenses can't deliver an MTF of nearly unity at the Nyquist frequency of most sensors, then I agree, but don't see the relevance.
Yes that is what I meant
If you mean that real lenses don't have sufficient contrast at the Nyquist frequency of most sensors to cause visible aliasing with high-frequency subjects, then I disagree strongly.
Of course there is aliasing you can see it so it is there
To talk about the system I'm most familiar with, all of the Fujifilm GF lenses are capable of aliasing at some f-stops on axis with the GFX 100x.

One way to think of lens and sensor resolution is to consider the balance between the two:

https://blog.kasson.com/the-last-word/whats-your-q/
Pixel shift cures this problem
Sometimes. Sometimes there is still aliasing. More often there are artifacts.
so it deliver a benefit in that respect but also others
This fact that pixel shift delivers a benefit sometimes is not in dispute. The nature of the benefit is.
So at the end what matters if resolution was increased or not which is the op question? The important part is that overall image quality is improved even if the so called high resolution shot does not deliver more resolution (but makes useable the resolution you can resolve without issues)
I'll bet you're in the camp that calls all lenses that have aberrations that are not raidally symmetric "decentered". Or as Roger Cicala once put it: "Calling everything that’s wrong with a lens decentering is like calling every reason your car won’t start ‘out of gas.’"

The point of the distinction between aliasing and MTF is to allow photographers to understand what they will -- and won't -- get from pixel shift.
Pixel shift will give me better image quality. That improves or does not improve resolution measured with a set of monochrome targets is not important to a photographer only to a scientist or an engineer as we don't shoot test patterns, well at least I do not!
Actually you do, because any sharp boundary at a suitable angle, such as the edge of a building, can be used to derive an MTF curve.

Don
 
Good show JACS, though the 200% images are one step removed from what my comments have been about throughout this thread: 4 capture pixel shift vs single capture, which remove effective pixel pitch as a variable and thus can be indicative of the difference in performance of Bayer CFA vs monochrome sensors. The difference can be hard to spot in the conditions shown earlier.

On the other hand, doubling the number of linear captures, pixel pitch and hence image size as in your example just above does indeed increase rendered resolution as well as reduce aliasing further, as mentioned upthread. The effect is quite obvious.

One more note on B&W honesty before leaving this interesting subject.
J A C S Wrote :

Patterns on the cloths and on the faces look good, actually, better than the demosaiced versions I have seen. So, this would be the honest representation of a B&W image if we knew it is B&W.
It is 'a' B&W representation. And this begs the question of what would be an honest one.

One way to answer it is to assume that honest B&W is what a monochromat would see, let's say in daylight conditions. Let's further assume that the monochromat only has the cone type that is sensitive to Medium wavelengths. That means that such a person effectively sees something proportional to Luminance, because that cone type has spectral sensitivity practically equivalent to the relative luminosity function (V).

On the other hand a typical Bayer CFA sensor has three types of color filters with peaks in what we normally refer as the R, G and B wavelengths - none of which are even close to the luminosity function. But let's say that we take the G channel as being equivalent to V, thus honestly (?) taking care of about half of the pixels in our B&W rendering. After all, the G plane is usually where cameras set their focus on.

The other half of pixels, R&B, have sensitivity functions that are even further removed from V. But let's give them the benefit of the doubt: after white balancing, if the illuminant is uniform and the scene is neutral, we could get intensity somewhat proportional to the others, right?

The R and B pixels are however on different color planes. That means that they are most likely out of focus and more aberrated (e.g. lateral chromatic) compared to G. So what we get is truly a Frankenstein of a B&W rendering, which its proud parents try to pass on as the real thing. Honest :-)

Jack

PS By these standards a pixel shifted render is not very honest either. In fact no render with commercial sensors is. But of course they are good enough for most purposes and we are all happy with the results.
I would like to see a high resolution scan of that engraving.

Some of the examples posted here show various tones of grey, but and engraving is a 1-bit image with only black and the near-white background paper. Here's a photo of a wood engraving in a 19C book that I have. The original is about A4 size.

That's what the engraving should look like. But maybe what DPR have in their test scene is not a real engraving but a photo of an engraving, like the photos of faces that some people think tell you something about how a camera records skin colours.

68e66cadbbb0444ea1c720cc5d40d27b.jpg.gif

Don Cox
 
If you take the "interpretative" idea to its limit, one could think of a smart machine learning algorithm that could guess the full 50MP scene from a handful of pixels. Problem solved.

If you take the conservative approach to its limit, you would produce a totally unpredictable scene (uncorrelated spectro-spatial noise at more than 3 color bands and many pixels per sensel). The system will then (at best) perform as estimated by Shannon-Nyquist, diffraction etc.

The practical performance should lie between to two extremes, and having a stable, predictable characterization of its performance seems challenging?

What about a camera or an algorithm that produce subjectively (or objectively) "slightly better" results 90% of the time, but significantly worse results 10% of the time, requiring manual intervention in order to save those 10%? (Thinking of OLPF-less here).

If you take pixel shift to the limit (infinite amount of exposures), it is like having an "infinite" pixel count sensor, prefiltered by a sensel-area sized continuous spatial pre-filter. The CFA sparsity also disappears when you get to sample the color channels at an infinite number of positions. Ignore the consequence of signal to noise, scene movement and such practicalities. Say that the continuous prefilter is like a square (100% fill-rate). Can that be deconvolved? Seems like the 2d version of a sinc, should contain infinite zeros that cannot be inversed?

However, for a camera featuring very small active photo sites (the original Canon 5D?), in the absence of an explicit OLPF, the effective optical prefilter from sensel extent when doing pixel shift would be narrower, approaching a delta function in the limit (but then the intensity capture would also tend towards zero). Thus having a lot to gain from pixel shift.

-h
 
Today most cameras do not resolve the Nyquist limit of the sensor anyway due to limitations in the lens
I can't tell precisely what you mean by that sentence.

If you mean that real lenses can't deliver an MTF of nearly unity at the Nyquist frequency of most sensors, then I agree, but don't see the relevance.
Yes that is what I meant
If you mean that real lenses don't have sufficient contrast at the Nyquist frequency of most sensors to cause visible aliasing with high-frequency subjects, then I disagree strongly.
Of course there is aliasing you can see it so it is there
To talk about the system I'm most familiar with, all of the Fujifilm GF lenses are capable of aliasing at some f-stops on axis with the GFX 100x.

One way to think of lens and sensor resolution is to consider the balance between the two:

https://blog.kasson.com/the-last-word/whats-your-q/
Pixel shift cures this problem
Sometimes. Sometimes there is still aliasing. More often there are artifacts.
so it deliver a benefit in that respect but also others
This fact that pixel shift delivers a benefit sometimes is not in dispute. The nature of the benefit is.
So at the end what matters if resolution was increased or not which is the op question? The important part is that overall image quality is improved even if the so called high resolution shot does not deliver more resolution (but makes useable the resolution you can resolve without issues)
I'll bet you're in the camp that calls all lenses that have aberrations that are not raidally symmetric "decentered". Or as Roger Cicala once put it: "Calling everything that’s wrong with a lens decentering is like calling every reason your car won’t start ‘out of gas.’"

The point of the distinction between aliasing and MTF is to allow photographers to understand what they will -- and won't -- get from pixel shift.
Pixel shift will give me better image quality. That improves or does not improve resolution measured with a set of monochrome targets is not important to a photographer only to a scientist or an engineer as we don't shoot test patterns, well at least I do not!
Actually you do, because any sharp boundary at a suitable angle, such as the edge of a building, can be used to derive an MTF curve.
But it will only tell me about the sharpness of that particular edge... ;-)

By this time, it's a bit late because I can already tell if it is sharp or not.

So how does a lab measurement predict whether it will be sharp or not?
 
Today most cameras do not resolve the Nyquist limit of the sensor anyway due to limitations in the lens
I can't tell precisely what you mean by that sentence.

If you mean that real lenses can't deliver an MTF of nearly unity at the Nyquist frequency of most sensors, then I agree, but don't see the relevance.
Yes that is what I meant
If you mean that real lenses don't have sufficient contrast at the Nyquist frequency of most sensors to cause visible aliasing with high-frequency subjects, then I disagree strongly.
Of course there is aliasing you can see it so it is there
To talk about the system I'm most familiar with, all of the Fujifilm GF lenses are capable of aliasing at some f-stops on axis with the GFX 100x.

One way to think of lens and sensor resolution is to consider the balance between the two:

https://blog.kasson.com/the-last-word/whats-your-q/
Pixel shift cures this problem
Sometimes. Sometimes there is still aliasing. More often there are artifacts.
so it deliver a benefit in that respect but also others
This fact that pixel shift delivers a benefit sometimes is not in dispute. The nature of the benefit is.
So at the end what matters if resolution was increased or not which is the op question? The important part is that overall image quality is improved even if the so called high resolution shot does not deliver more resolution (but makes useable the resolution you can resolve without issues)
I'll bet you're in the camp that calls all lenses that have aberrations that are not raidally symmetric "decentered". Or as Roger Cicala once put it: "Calling everything that’s wrong with a lens decentering is like calling every reason your car won’t start ‘out of gas.’"

The point of the distinction between aliasing and MTF is to allow photographers to understand what they will -- and won't -- get from pixel shift.
Pixel shift will give me better image quality. That improves or does not improve resolution measured with a set of monochrome targets is not important to a photographer only to a scientist or an engineer as we don't shoot test patterns, well at least I do not!
Actually you do, because any sharp boundary at a suitable angle, such as the edge of a building, can be used to derive an MTF curve.
But it will only tell me about the sharpness of that particular edge... ;-)

By this time, it's a bit late because I can already tell if it is sharp or not.

So how does a lab measurement predict whether it will be sharp or not?
An MTF test will put an upper bound on how sharp the edge will be. That is a kind of prediction. In actual usage, with more uncontrolled variables, you may get far worse sharpness.
 
Today most cameras do not resolve the Nyquist limit of the sensor anyway due to limitations in the lens
I can't tell precisely what you mean by that sentence.

If you mean that real lenses can't deliver an MTF of nearly unity at the Nyquist frequency of most sensors, then I agree, but don't see the relevance.
Yes that is what I meant
If you mean that real lenses don't have sufficient contrast at the Nyquist frequency of most sensors to cause visible aliasing with high-frequency subjects, then I disagree strongly.
Of course there is aliasing you can see it so it is there
To talk about the system I'm most familiar with, all of the Fujifilm GF lenses are capable of aliasing at some f-stops on axis with the GFX 100x.

One way to think of lens and sensor resolution is to consider the balance between the two:

https://blog.kasson.com/the-last-word/whats-your-q/
Pixel shift cures this problem
Sometimes. Sometimes there is still aliasing. More often there are artifacts.
so it deliver a benefit in that respect but also others
This fact that pixel shift delivers a benefit sometimes is not in dispute. The nature of the benefit is.
So at the end what matters if resolution was increased or not which is the op question? The important part is that overall image quality is improved even if the so called high resolution shot does not deliver more resolution (but makes useable the resolution you can resolve without issues)
I'll bet you're in the camp that calls all lenses that have aberrations that are not raidally symmetric "decentered". Or as Roger Cicala once put it: "Calling everything that’s wrong with a lens decentering is like calling every reason your car won’t start ‘out of gas.’"

The point of the distinction between aliasing and MTF is to allow photographers to understand what they will -- and won't -- get from pixel shift.
Pixel shift will give me better image quality. That improves or does not improve resolution measured with a set of monochrome targets is not important to a photographer only to a scientist or an engineer as we don't shoot test patterns, well at least I do not!
Actually you do, because any sharp boundary at a suitable angle, such as the edge of a building, can be used to derive an MTF curve.
But it will only tell me about the sharpness of that particular edge... ;-)

By this time, it's a bit late because I can already tell if it is sharp or not.

So how does a lab measurement predict whether it will be sharp or not?
Lab data will tell you what you need to do to achieve optimal sharpness.



A concrete example:

I have a Sony 200-600/5.6-6.3 G lens. Shooting in the lab, I can see that it can perhaps reach around 30 MP of resolution, but typical may be more like 6MP. A large part of that is dismal focusing accuracy of my Sony A7rIV.

Shooting in the lab teaches me what the lens can do under ideal conditions, but it may also give some insight on how to achieve best focus.

Best regards

Erik
 
Today most cameras do not resolve the Nyquist limit of the sensor anyway due to limitations in the lens
I can't tell precisely what you mean by that sentence.

If you mean that real lenses can't deliver an MTF of nearly unity at the Nyquist frequency of most sensors, then I agree, but don't see the relevance.
Yes that is what I meant
If you mean that real lenses don't have sufficient contrast at the Nyquist frequency of most sensors to cause visible aliasing with high-frequency subjects, then I disagree strongly.
Of course there is aliasing you can see it so it is there
To talk about the system I'm most familiar with, all of the Fujifilm GF lenses are capable of aliasing at some f-stops on axis with the GFX 100x.

One way to think of lens and sensor resolution is to consider the balance between the two:

https://blog.kasson.com/the-last-word/whats-your-q/
Pixel shift cures this problem
Sometimes. Sometimes there is still aliasing. More often there are artifacts.
so it deliver a benefit in that respect but also others
This fact that pixel shift delivers a benefit sometimes is not in dispute. The nature of the benefit is.
So at the end what matters if resolution was increased or not which is the op question? The important part is that overall image quality is improved even if the so called high resolution shot does not deliver more resolution (but makes useable the resolution you can resolve without issues)
I'll bet you're in the camp that calls all lenses that have aberrations that are not raidally symmetric "decentered". Or as Roger Cicala once put it: "Calling everything that’s wrong with a lens decentering is like calling every reason your car won’t start ‘out of gas.’"

The point of the distinction between aliasing and MTF is to allow photographers to understand what they will -- and won't -- get from pixel shift.
Pixel shift will give me better image quality. That improves or does not improve resolution measured with a set of monochrome targets is not important to a photographer only to a scientist or an engineer as we don't shoot test patterns, well at least I do not!
Actually you do, because any sharp boundary at a suitable angle, such as the edge of a building, can be used to derive an MTF curve.
But it will only tell me about the sharpness of that particular edge... ;-)

By this time, it's a bit late because I can already tell if it is sharp or not.

So how does a lab measurement predict whether it will be sharp or not?
An MTF test will put an upper bound on how sharp the edge will be. That is a kind of prediction. In actual usage, with more uncontrolled variables, you may get far worse sharpness.
I agree.

I just think it would interesting to focus on some of those uncontrolled variables and their effects, given that at the end of the day, all we care about is what it ends up looking like.

And whereas we can't change a sensor or lens we already own, many decisions we make will change their real-world performance - even the medium we view an image on.

The OP asked if it would be possible to see an improvement, and the consensus was that there should be an improvement in colour resolution and less false colour.

But how can we measure colour MTF in a useful way (ie something akin to Delta-E) and how visible is this in comparison to luminance MTF?
 
But how can we measure colour MTF in a useful way (ie something akin to Delta-E) and how visible is this in comparison to luminance MTF?
I measure MTF on each of the raw channels, and for the past year or so I've been doing that on both sides of and through the focal plane. That gives lots of information about LoCA, LaCA, and various aberrations. However, I don't see much value in merging those together and converting to a CIE space.
 
But how can we measure colour MTF in a useful way (ie something akin to Delta-E) and how visible is this in comparison to luminance MTF?
I measure MTF on each of the raw channels, and for the past year or so I've been doing that on both sides of and through the focal plane. That gives lots of information about LoCA, LaCA, and various aberrations. However, I don't see much value in merging those together and converting to a CIE space.
Perhaps 57 is wondering about the perceptual side of things. I have often heard it said that the HVS takes sharpness cues from its monochromatic channel, and that it is much less sensitive to it in the color difference channels. That makes good intuitive sense and seems to match personal experience, though it would be nice to see some data to support the feeling.
 
Last edited:
But how can we measure colour MTF in a useful way (ie something akin to Delta-E) and how visible is this in comparison to luminance MTF?
I measure MTF on each of the raw channels, and for the past year or so I've been doing that on both sides of and through the focal plane. That gives lots of information about LoCA, LaCA, and various aberrations. However, I don't see much value in merging those together and converting to a CIE space.
Wasn't suggesting you did, per se. Only asking how we can judge colour as opposed to luminance resolution. ie how does colour discrimination improve in the frequency domain with pixel shift, and how does it compare to human sensitivity?
 
The OP asked if it would be possible to see an improvement, and the consensus was that there should be an improvement in colour resolution and less false colour.

But how can we measure colour MTF in a useful way (ie something akin to Delta-E) and how visible is this in comparison to luminance MTF?
Why do you want to measure “color MTF” in the first place? MTF makes no sense for a system mapping a continuous image to a discrete one in the first place.


And again, it is not like we resolve the luma up to Nyquist of the sensor (ignoring the Bayer filter), and we are missing the chroma. We are missing it all.
 
Last edited:

Keyboard shortcuts

Back
Top