Pixel Shift vs Stitching

Hi everyone,

I was hoping the brilliant minds I've come to highly respect over the years I've lurked about on DPR without posting much would help me shed some light on this, and confirm or disprove my conclusions if I am wrong.

I'm familiar with Jim Kasson's (controversial) assertion that pixel shift (16-shots like on Fuji GFX, Sony) does not increase resolution, specially spatial resolution. I don't have a problem with that. The question I had that needed answering was how does pixel shift compare to stitching?

For a given camera, say the Fujifilm GFX 100S that I own:
- Pixel shift 16-shot (4x the native resolution, so 404 megapixels)
vs
- Multi-row 3x3 stitch with 50% overlap, resulting stitch would also end up at 404 megapixels.

Questions:
1) If the subject of interest occupies the same pixel dimensions in both cases, (to achieve this, you would double the magnification to capture the for-stitching frames), all else being equal, which would have higher (spatial) resolution?

2) Subtly different and more complex question: Which would look better perceptually/subjectively, given that the stitch is made of single frames that will still potentially have aliasing, while the pixel shift would be virtually free of it (if done properly), yet both are identical in pixel dimensions/resolution.

3) Does pixel shift affect the diffraction limited aperture? I may not have phrased this question quite as precisely as I want to mean it, bear with me. For a certain set of criteria to determine the diffraction limited aperture for a given sensor, does pixel shift 16-shots (1/2 pixel sampling offsets) mean you need half the aperture value (e.g. f/4 vs f/8)? Or is the diffraction limited aperture still the same as the single frame because the sensor is the same, the pixel aperture/pitch remains the same even when pixel shifting?

I've published an article about this on my site with many example image crop comparisons of real world images (I do not know how to generate artificial idealised simulations), and came to some conclusions, which may be right or wrong. I've not seen these types of comparisons anywhere else before despite searching quite a bit for it, so if you're interested, here it is.

Thanks for reading this and much appreciation to anyone who has some insight to share.


Setup details for this comparison are in the link above.


Setup details for this comparison are in the link above.

Comparing pixel-shift and stitching, can the differences in image quality be quantified or measured ? (If the difference is modest, would that influence our choice of method ?)

Under what range of conditions can the difference be readily seen ? In other words, are these differences only obvious at 100% ? 50 % ?

Are pixel-shift and stitching mutually exclusive ? How much improvement occurs when we combine both methods ?

Would a test be more coherent if the same lens were used throughout, changing the subject distance instead of the lens ?

As a first stab, Pixel Shift as described means doubling the linear sampling rate compared to Single Shot, which would double the foldback spatial frequency and reduce aliasing correspondingly. However the lens and pixel aperture are unchanged, which means that in theory the MTF curve would be exactly the same. How much aliasing is there likely to be and how much does it impact the end result?

Leaving stitching aside and ignoring corner aberrations for a moment, let's say that we compared a central crop of the pixel-shifted image above to a single shot at twice the magnification, both with the same number of pixels. Twice the magnification would impact the effective f-number of the lens all else equal, hence its performance. How much better or worse is the lens at 0.5x than at 1x? It depends on the lens, and the two scenarios become harder to compare without specific lens performance data.

Also, assuming the same subject, the minimum size of detail in the image is doubled at the 1x magnification, effectively halving aliasing compared to 0.5x and potentially bringing it in line with pixel shift. The relative size of pixel aperture is smaller compared to such detail, potentially resulting in less blurring than in the pixel-shifted version if the lenses used in the two cases perform more or less equivalently (bottom right image).

At first glance, those are the variables at play. In theory only, alas.

Jack

PS I haven't had a chance to read your blog post.

Kenneth Morris Lee said:

Would a test be more coherent if the same lens were used throughout, changing the subject distance instead of the lens ?

Click to expand...

I've already tried that experiment, doubling the focal length and changing the subject distance to maintain the same pixel size rather than the magnification. You can see the results in my link above.

Kenneth Morris Lee said:

Comparing pixel-shift and stitching, can the differences in image quality be quantified or measured ? (If the difference is modest, would that influence our choice of method ?)

Click to expand...

For sure it can. The difference isn't modest to me.

Kenneth Morris Lee said:

Under what range of conditions can the difference be readily seen ? In other words, are these differences only obvious at 100% ? 50 % ?

Click to expand...

I'm sure how noticeable it is is subjective, because our end goals are different. You're not going to notice it in an Instagram post. I've met tons of people who said they couldn't notice or aren't bothered by the shutter shake of the original a7R. I'm not one of those. It was really bad for me, so I'm "that kind of noticeable" person. I'm also the kind who doesn't tolerate a focus tilt of over 20 microns for astrophotography ...

Kenneth Morris Lee said:

Are pixel-shift and stitching mutually exclusive ? How much improvement occurs when we combine both methods ?

Click to expand...

No, and it depends on a lot of real-world factors. You can also simulate idealised scenarios in theory (perfect diffraction-limited lenses for one) to understand the limits and realise that real world results would always be worse. But it's a huge data load (~2.3GB per 16-shot sequence for the GFX100S, ~3.6GB for a7RIV/V) for one, and pixel shift is very hard to use outdoors reliably. Jim Kasson said he has never been successful at all, I've never even bothered trying outside a controlled indoors environment.

Even in a controlled environment where I was pretty sure I had a very stable setup (Fujifilm GFX100S, Gitzo 3 series carbon tripod, Arca Swiss Cube C1 head, GF 35-70mm, timer release, concrete tiled floor, doors and windows shut, no fans, no machines vibrating etc.), the pixel shift process was brittle and a very astute photographer pointed out to me there was some combining artifacts. Even in the crops I've shared in the OP, you can see some artifacts which look like clusters of hot pixels that are randomly scattered about, more subtle than those Horshack posted about here.

Here's a crop from a relatively smooth region:


100% zoom, no contrast enhancement, no sharpening, no noise reduction, 16-shot pixel shift


Above but with contast increase as shown


Above result at 400% zoom

First, the point is made nonetheless: pixel shift process is very brittle and it's not hard to cause artifacts, however small and hard they are to see, which are not present in single shot images. If you have any subject movement or camera vibration or lighting changes, it will be much worse than just the above, which seems to be some kind of subtle misalignment issue. It was a crop taken from the same image that I shared in my article. While I've done stitching many times in many difficult situations, even moving subjects, successfully. And you are free of the aspect ratio of the sensor with stitching, not so with pixel shift, so you are more free compositionally.

And secondly, yes, I too couldn't perceive these subtle artifacts at 100% zoom with no contrast increase. But I don't like the fact that it's there, and in less controlled situations it would likely be only worse than this.

Thirdly, I'll bet that fussy photographers who like to shoot 8x10 film and drum scan them at high res (like the folks on the large format photo forum) would be unhappy to see ugly film grain with weird patterns like this.

Hi Jack! I was hoping you would respond, so thank you!

Jack Hogan said:

Leaving stitching aside and ignoring corner aberrations for a moment, let's say that we compared a central crop of the pixel-shifted image above to a single shot at twice the magnification, both with the same number of pixels. Twice the magnification would impact the effective f-number of the lens all else equal, hence its performance.

Click to expand...

I was hoping you or Jim could simulate or theorise an idealised thought experiment. Assuming a perfect diffraction-limited lens, and say it's set to f/1.4 or something where diffraction isn't affecting the results no matter what, so the system is not constrained by lens resolution or diffraction. What are the theoretical limits of resolution of each method?

I asked ChatGPT and this was what it said, I'm not sure if it is right or wrong:

Revisiting Your 0.5x vs 1x Case — With This Corrected Understanding

• At 1x magnification, the 3.76µm pixels give you a Nyquist of ~133 lp/mm.
• At 0.5x magnification, the subject is demagnified 2x, so each 3.76µm pixel covers 7.52µm of the subject. The subject-side Nyquist is now ~66.5 lp/mm.
• Pixel shift at 0.5x doesn’t change this Nyquist limit.
  It removes Bayer penalties, so you’ll have cleaner color and slightly better contrast near the Nyquist limit (66.5 lp/mm), but you won’t get to 133 lp/mm on the subject.
• At 1x magnification (single shot), you have full 133 lp/mm Nyquist on the subject.
• Therefore, at the same effective aperture and lens quality, 1x single-shot will always have the potential for higher spatial resolution than 0.5x pixel shift.

Key Correction Summary

• 0.5x pixel shift does not "increase resolution" to match 1x.
• What it does do is make better use of the sensor’s existing resolution (within the lower 66.5 lp/mm subject-side limit).
• It will have cleaner color, better microcontrast at lower spatial frequencies, and fewer artifacts compared to single-shot at 0.5x.

Jack Hogan said:

How much better or worse is the lens at 0.5x than at 1x? It depends on the lens, and the two scenarios become harder to compare without specific lens performance data.

Click to expand...

The details are in my article, but just to respond directly to you, the Mejiro Genossen FL0530 lens was used wide open (F4) for the 0.5x and 1.0x samples in my OP, crops taken on-axis, and the MTF for that region is about 50% contrast at 100lp/mm for 0.5x, and about 40% contrast at 100lp/mm at 1.0x. The performance at 1.0x is so good it's a flat line out to 30mm, it's like this (I know it's a different lens from the same company, but I am not allowed to share the MTFs from my taking lens, and I can tell you it is very, very close):



Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

Jack Hogan said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

SamuelChia said:

I asked ChatGPT and this was what it said, I'm not sure if it is right or wrong:

Click to expand...

Mostly wrong.

SamuelChia said:

Revisiting Your 0.5x vs 1x Case — With This Corrected Understanding

• At 1x magnification, the 3.76µm pixels give you a Nyquist of ~133 lp/mm.
• At 0.5x magnification, the subject is demagnified 2x, so each 3.76µm pixel covers 7.52µm of the subject. The subject-side Nyquist is now ~66.5 lp/mm.
• Pixel shift at 0.5x doesn’t change this Nyquist limit.

Click to expand...

It does, by definition. You are sampling denser.

SamuelChia said:

• It removes Bayer penalties, so you’ll have cleaner color and slightly better contrast near the Nyquist limit (66.5 lp/mm), but you won’t get to 133 lp/mm on the subject.

Click to expand...

You will but with the same blur kernel, which would be the limiting factor. On the other hand, removing Bayer penalties is increasing resolution.

SamuelChia said:

• At 1x magnification (single shot), you have full 133 lp/mm Nyquist on the subject.
• Therefore, at the same effective aperture and lens quality, 1x single-shot will always have the potential for higher spatial resolution than 0.5x pixel shift.

Click to expand...

Mostly the right conclusion but for the wrong reasons. The improvement would be due to the weaker blur filter, on the subject, caused by the averaging over the active area of the pixel.

It is a bit more complicated than that, because it depends on how the pixel shift is done. The main point remains: pixel shift does increase the sampling rate, and the Nyquist limit. It doesn’t change the blur by a pixel though.

SamuelChia said:

Hi Jack! I was hoping you would respond, so thank you!

Click to expand...

Hi Samuel, my pleasure.

SamuelChia said:

Jack Hogan said:

Leaving stitching aside and ignoring corner aberrations for a moment, let's say that we compared a central crop of the pixel-shifted image above to a single shot at twice the magnification, both with the same number of pixels. Twice the magnification would impact the effective f-number of the lens all else equal, hence its performance.

Click to expand...

What are the theoretical limits of resolution of each method?

Click to expand...

The variables in the two cases (0.5x pixel-shift vs 1x single shot aebe) are effective sampling rate, effective pixel aperture, and lens performance.

As JACS says, sampling rate affects the amount of aliasing because it determines the foldback frequency (Nyquist). In theory it can be considered somewhat equivalent in both cases because a sinusoid that would appear at c lp/mm on the sensor at 0.5x would appear at c/2 lp/mm on the same sensor at 1x.

Pixel aperture is a major determinant of 'sharpness' at this scale because it acts like a low-pass filter. The 'radius' of the filter (what JACS calls the kernel size) depends on the pixel configuration, the presence and/or shape of microlenses, etc. It is physically the same in both cases, but in the 1x case it acts on an image twice the 0.5x size so its effect is correspondingly smaller, meaning that it penalizes the 0.5x pixel-shift capture more.

SamuelChia said:

SamuelChia said:

How much better or worse is the lens at 0.5x than at 1x? It depends on the lens, and the two scenarios become harder to compare without specific lens performance data.

Click to expand...

The details are in my article, but just to respond directly to you, the Mejiro Genossen FL0530 lens was used wide open (F4) for the 0.5x and 1.0x samples in my OP, crops taken on-axis, and the MTF for that region is about 50% contrast at 100lp/mm for 0.5x, and about 40% contrast at 100lp/mm at 1.0x.

Click to expand...

The lens has different performance in the two cases, as you mention above. Also, 100 lp/mm becomes 50 lp/mm at 1x, so it would be good to know its performance at that spatial frequency. It would be safe to guess that it should be better than 100 lp/mm at 0.5x.

Effective pixel aperture and lens performance spatial frequency responses interact. They will change with different equipment and setups. With the setup shown in this thread they apparently favor the 1x single shot, as seen.

All of this could be quantified a bit better but I don't have time to do it at the moment.

SamuelChia said:

Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

SamuelChia said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Click to expand...

Click to expand...

I should clarify that another simplification I am making is that the target or the sensor is monochrome for now. The difference can be subtle, or not. In this case, in the limit 100MP to 400MP is twice the linear magnification (horizontally and vertically)

SamuelChia said:

SamuelChia said:

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

Click to expand...

This requires further dissection because terms like 'resolution' and 'diffraction limited' are not clear in this context.

Jack

Kenneth Morris Lee said:

Comparing pixel-shift and stitching, can the differences in image quality be quantified or measured ? (If the difference is modest, would that influence our choice of method ?)

Under what range of conditions can the difference be readily seen ? In other words, are these differences only obvious at 100% ? 50 % ?

Click to expand...

Stitching allows for greater underlying analog resolution, as each image can do another full set of line pairs at a given level of contrast. Stitching is like using a larger sensor, with multiple copies of the same lens projecting onto fractions of the sensor. Pixel shift, on the other hand, only simulates a higher pixel density; not multiple lenses on a larger sensor. Pixel shift is less prone to spatial artifacts, though, because lens distortion is irrelevant to pixel shifts of small numbers or fractions of pixels; all the images distort together, regardless of whether you use lens corrections, or not.

Kenneth Morris Lee said:

Are pixel-shift and stitching mutually exclusive ? How much improvement occurs when we combine both methods ?

Click to expand...

When stitching without pixel-shift, you may find a registration difference between two images to be 1/2 pixel, so the software has to either blur or leave one side of the stitch 1/2 pixel off, unless stitching first upsamples the sub-images to 4x or 9x the original pixels to give more registration precision, something that pixel-shift does not need because it is already precisely aligned.

Kenneth Morris Lee said:

Would a test be more coherent if the same lens were used throughout, changing the subject distance instead of the lens ?

Click to expand...

If that's what you would do in the real world, then your test should simulate it. People will sometimes use longer lenses for stitching, though, from the same distance.

SamuelChia said:

SamuelChia posted:

Above result at 400% zoom

Click to expand...

This is certainly not a normal part of pixel-shifting. It is either because the two green channels or different sub-images have different gains or spectral sensitivities, which were not corrected in-camera, and/or the converter did some sloppy math.

My guess is that your camera *always* has mis-calibrated greens, but they are suppressed by both demosaicing and a higher frequency pattern at the pixel level that goes into visual extinction, whereas a shift pattern that thickens the influence of the green channel sensitivities to half the frequency, preventing visual extinction.

The two green channels in each sub-image and between the sub-images must either have the same sensitivity, or be calibrated before demosaicing, to avoid such patterns. If you bin 4x4 or downsample 25%, the pattern disappears, and you can filter the shifted image to get rid of it too, but that softens the image. If you made a flat-field image like people do in some astronomical photography, you can apply it without losing the contrast of detail.



--
Beware of correct answers to wrong questions.
John

Jack Hogan said:

Jack Hogan said:

Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

Jack Hogan said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Click to expand...

Click to expand...

I should clarify that another simplification I am making is that the target or the sensor is monochrome for now. The difference can be subtle, or not. In this case, in the limit 100MP to 400MP is twice the linear magnification (horizontally and vertically)

Jack Hogan said:

Jack Hogan said:

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

Click to expand...

This requires further dissection because terms like 'resolution' and 'diffraction limited' are not clear in this context.

Jack

Click to expand...

Let's assume that we have two images of the same subject:

Image 1: 100MP shot at f8 (effective aperture) at 1x

F8.0 would still work fine for a 100MP sensor, you'd still get some false colors on a Bayern sensor but the details will look nicely resolved and crisp.

Image 2: 400MP shot at f6 (effective aperture) at 0.5x

The effective aperture alone would prevent the 400MP image from actually being able to capture that amount of image-based-information. An effective aperture of 6.0 would allow for around 150-160MP of information, anything beyond is "empty resolution", meaning it won't reveal much more and might as well be upscaled.

If we now take a 100MP crop from that 400MP image it will not be as well resolved, it'll be slightly blurred - but at least there won't be any false colors (or there shouldn't be). And that's precisely what we can observe in the comparison above.

So even if we disregard pixel shift a 100MP crop from the 400MP image would not carry as much information as the 100MP image at 1x because it's already in diffraction territory on a pixel level. In order to get 400MP one would have to use at least approx. f3.5 effective aperture.

All my tests and the tests done by my fellow photographers show that pixel shift can increase the resolution/details considerably as long as the effective aperture allows for that to happen.

EDIT: whether pixel shift or the pixel size of the pixelshift-sensor presents a further limitation regarding the resolution/details - I can't say. I've not yet had cause or reason to make a comparison of a 0.5x PixelShift vs 1x SingleShot. But if such a comparison would be done the apertures should be set according to the desired resolution, otherwise the pixelshift image is artificially held back and any comparison will have a bias towards the singleshot.

Crispy_Bee said:

Jack Hogan said:

Crispy_Bee said:

Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

Crispy_Bee said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Click to expand...

Click to expand...

I should clarify that another simplification I am making is that the target or the sensor is monochrome for now. The difference can be subtle, or not. In this case, in the limit 100MP to 400MP is twice the linear magnification (horizontally and vertically)

Crispy_Bee said:

Crispy_Bee said:

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

Click to expand...

This requires further dissection because terms like 'resolution' and 'diffraction limited' are not clear in this context.

Jack

Click to expand...

Let's assume that we have two images of the same subject:

Image 1: 100MP shot at f8 (effective aperture) at 1x

F8.0 would still work fine for a 100MP sensor, you'd still get some false colors on a Bayern sensor but the details will look nicely resolved and crisp.

Click to expand...

That's part of it. The other part is that, say, sinusoidal detail at the scene projected on the sensor at 100 lp/mm in the pixel-shift case, is instead projected at 50 lp/mm here because magnification is doubled. This is an example of the spatial frequency response of the interaction between effective pixel aperture and the lens:



1 c/p in the plot corresponds to about 150 lp/mm. The yellow curve is the lens only, so reading off it, say, about 40% MTF at 100 lp/mm (0.5x) and 60% at 50 lp/mm (1x).

But at 0.5x the lens performs a little better than would be suggested by the yellow curve, as Samuel says: "50% contrast at 100lp/mm for 0.5x [effective f-number f/6], and about 40% contrast at 100lp/mm at 1.0x [effective f-number f/8]".

My educated guess is that the difference would be less at 50 lp/mm, so I would guess that the earlier advantage in MTF enjoyed by the 1x lens would be somewhat reduced but still there, say 55% MTF at 50 lp/mm.

55% is quite a bit better than 40% so with this setup 1x wins the lens component competition. What about the effective pixel (aperture) component?

Crispy_Bee said:

Image 2: 400MP shot at f6 (effective aperture) at 0.5x

The effective aperture alone would prevent the 400MP image from actually being able to capture that amount of image-based-information. An effective aperture of 6.0 would allow for around 150-160MP of information, anything beyond is "empty resolution", meaning it won't reveal much more and might as well be upscaled.

Click to expand...

The lens is a little better at f/6 than at f/8 as we have seen above. Pixel shift is taking multiple single shot images, so each capture's 'limiting resolution' (whatever that means) is unchanged. The fact that these single shot images are shifted a bit and combined, allows the combined image to be supersampled with the aliasing advantages mentioned earlier.

However, pixel aperture stays the same in all cases. I realize that this is a different sensor with 6.75um pitch and I am ignoring Bayer implications - but indulge me in reading off that plot as an example of how to do this (lens = yellow curve, pixel = purple, system = yellow times purple).

Image 2, 400MP central crop, f/6, 0.5x, at 100 lp/mm: Lens MTF = 40%, Pixel Aperture MTF = 55%, Combined MTF = 0.22%

Crispy_Bee said:

If we now take a 100MP crop from that 400MP image it will not be as well resolved, it'll be slightly blurred - but at least there won't be any false colors (or there shouldn't be). And that's precisely what we can observe in the comparison above.

Click to expand...

Image 1, 100MP, f/8, 1x, at 50 lp/mm: Lens MTF = 55%, pixel aperture MTF = 85%, Combined MTF = 0.47%

Crispy_Bee said:

So even if we disregard pixel shift a 100MP crop from the 400MP image would not carry as much information as the 100MP image at 1x because it's already in diffraction territory on a pixel level. In order to get 400MP one would have to use at least approx. f3.5 effective aperture.

Click to expand...

If we had MTF plots for the OP's sensor and lens at various effective f-numbers we could figure out at which f-number the MTFs would be the same in both cases.

Jack

Jack Hogan said:

Crispy_Bee said:

Jack Hogan said:

Jack Hogan said:

Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

Jack Hogan said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Click to expand...

Click to expand...

I should clarify that another simplification I am making is that the target or the sensor is monochrome for now. The difference can be subtle, or not. In this case, in the limit 100MP to 400MP is twice the linear magnification (horizontally and vertically)

Jack Hogan said:

Jack Hogan said:

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

Click to expand...

This requires further dissection because terms like 'resolution' and 'diffraction limited' are not clear in this context.

Jack

Click to expand...

Let's assume that we have two images of the same subject:

Image 1: 100MP shot at f8 (effective aperture) at 1x

F8.0 would still work fine for a 100MP sensor, you'd still get some false colors on a Bayern sensor but the details will look nicely resolved and crisp.

Click to expand...

That's part of it. The other part is that, say, sinusoidal detail at the scene projected on the sensor at 100 lp/mm in the pixel-shift case, is instead projected at 50 lp/mm here because magnification is doubled. This is an example of the spatial frequency response of the interaction between effective pixel aperture and the lens:



1 c/p in the plot corresponds to about 150 lp/mm. The yellow curve is the lens only, so reading off it, say, about 40% MTF at 100 lp/mm (0.5x) and 60% at 50 lp/mm (1x).

But at 0.5x the lens performs a little better than would be suggested by the yellow curve, as Samuel says: "50% contrast at 100lp/mm for 0.5x [effective f-number f/6], and about 40% contrast at 100lp/mm at 1.0x [effective f-number f/8]".

My educated guess is that the difference would be less at 50 lp/mm, so I would guess that the earlier advantage in MTF enjoyed by the 1x lens would be somewhat reduced but still there, say 55% MTF at 50 lp/mm.

55% is quite a bit better than 40% so with this setup 1x wins the lens component competition. What about the effective pixel (aperture) component?

Jack Hogan said:

Image 2: 400MP shot at f6 (effective aperture) at 0.5x

The effective aperture alone would prevent the 400MP image from actually being able to capture that amount of image-based-information. An effective aperture of 6.0 would allow for around 150-160MP of information, anything beyond is "empty resolution", meaning it won't reveal much more and might as well be upscaled.

Click to expand...

The lens is a little better at f/6 than at f/8 as we have seen above. Pixel shift is taking multiple single shot images, so each capture's 'limiting resolution' (whatever that means) is unchanged. The fact that these single shot images are shifted a bit and combined, allows the combined image to be supersampled with the aliasing advantages mentioned earlier.

However, pixel aperture stays the same in all cases. I realize that this is a different sensor with 6.75um pitch and I am ignoring Bayer implications - but indulge me in reading off that plot as an example of how to do this (lens = yellow curve, pixel = purple, system = yellow times purple).

Image 2, 400MP central crop, f/6, 0.5x, at 100 lp/mm: Lens MTF = 40%, Pixel Aperture MTF = 55%, Combined MTF = 0.22%

Jack Hogan said:

If we now take a 100MP crop from that 400MP image it will not be as well resolved, it'll be slightly blurred - but at least there won't be any false colors (or there shouldn't be). And that's precisely what we can observe in the comparison above.

Click to expand...

Image 1, 100MP, f/8, 1x, at 50 lp/mm: Lens MTF = 55%, pixel aperture MTF = 85%, Combined MTF = 0.47%

Jack Hogan said:

So even if we disregard pixel shift a 100MP crop from the 400MP image would not carry as much information as the 100MP image at 1x because it's already in diffraction territory on a pixel level. In order to get 400MP one would have to use at least approx. f3.5 effective aperture.

Click to expand...

If we had MTF plots for the OP's sensor and lens at various effective f-numbers we could figure out at which f-number the MTFs would be the same in both cases.

Jack

Click to expand...

I'm just saying that 0.5x at f6.0 on a 44x33mm sensor isn't going to allow for 400MP of information to be available, regardless of how well the lens does or what size the pixel aperture is. That alone is an issue that would make a comparison on a pixel-level pretty much useless.

Just from looking at the 0.5x 400MP crop you can see it resolves more than the 0.5x 100MP but not more than the 1x 100MP crop - and that alone might be due to the aperture limitation of f6.0.

I know it's a crude table but roughly speaking
an effective aperture of 3.5 provides information for roughly 390-420MP.
an effective aperture of 6.0 provides information for roughly 150-160MP.
an effective aperture of 8.0 provides information for roughly 95-100MP.

Again: a crude, rough approximation but it's still pretty reliable and usable.
Even if we disregard all the potential issues and pitfalls inherit in the pixel shift process, the pixel size, pixel aperture,... the available potential resolution that could be collected (regardless of the sensor resolution itself) is limited.
So when you compare a 100% crop from the 100MP at f8.0 to a 100% crop from the 400MP at f6.0, the 100% crop of the 400MP image will not be able to compete in terms of fine detail/effective resolution no matter what.

I happen to own the same lens (actually 2 of them) and it's a stellar performer but it can't cheat physics when it comes to diffraction.

All I'm saying is that we should at least give the pixel shift 0,5x image a chance to potentially match the single shot 1x image before making a final judgement to say why it can't.

Crispy_Bee said:

I'm just saying that 0.5x at f6.0 on a 44x33mm sensor isn't going to allow for 400MP of information to be available, regardless of how well the lens does or what size the pixel aperture is. That alone is an issue that would make a comparison on a pixel-level pretty much useless.

Click to expand...

I'll leave the practical evaluations to you Crispy_Bee, since I don't own the kit.

With regards to 'allowing 400MP of information' I don't know what that means: if the sensor had delta-like pixel aperture you could pixel-shift to your heart's content, in theory creating meaningful images with unlimited MPs (well...).

Since the GFX100s does have a finite pixel aperture, you can't - and there are diminishing returns. At least you now have a theoretical framework to understand what happens to the resulting image 'sharpness'.

Jack

Jack Hogan said:

Crispy_Bee said:

I'm just saying that 0.5x at f6.0 on a 44x33mm sensor isn't going to allow for 400MP of information to be available, regardless of how well the lens does or what size the pixel aperture is. That alone is an issue that would make a comparison on a pixel-level pretty much useless.

Click to expand...

I'll leave the practical evaluations to you Crispy_Bee, since I don't own the kit.

With regards to 'allowing 400MP of information' I don't know what that means: if the sensor had delta-like pixel aperture you could pixel-shift to your heart's content, in theory creating meaningful images with unlimited MPs (well...).

Click to expand...

Not really though. That's not how image projection works.

And that's what it's about: the projected image has to provide enough information to allow for such a high megapixel count to make any sense.

You could have a 5 gigapixel image from a pixel-dimension but if you shot it on a 44x33mm sensor using an aperture of f8 it won't show the amount of detail a 5 gigapixel sensor would be capable of, simply due to the constriction of the aperture.

Sure, you'd have some nice images of huge airy disk blobs but not detail.

And that's the whole point, if you don't let enough information reach the sensor it won't be able to turn that into a highly resolved image - not matter how many pixels you inflate it with.

Jack Hogan said:

Since the GFX100s does have a finite pixel aperture, you can't - and there are diminishing returns. At least you now have a theoretical framework to understand what happens to the resulting image 'sharpness'.

Jack

Click to expand...

John Sheehy said:

[…]

Pixel shift is less prone to spatial artifacts, though, because lens distortion is irrelevant to pixel shifts of small numbers or fractions of pixels; all the images distort together, regardless of whether you use lens corrections, or not.

Click to expand...

On the other hand, stitching gives you data to perform lens correction – you can optimise lens correction parameters as part of the stitching.

Crispy_Bee said:

I'm just saying that 0.5x at f6.0 on a 44x33mm sensor isn't going to allow for 400MP of information to be available, regardless of how well the lens does or what size the pixel aperture is. That alone is an issue that would make a comparison on a pixel-level pretty much useless.

Just from looking at the 0.5x 400MP crop you can see it resolves more than the 0.5x 100MP but not more than the 1x 100MP crop - and that alone might be due to the aperture limitation of f6.0.

I know it's a crude table but roughly speaking
an effective aperture of 3.5 provides information for roughly 390-420MP.
an effective aperture of 6.0 provides information for roughly 150-160MP.
an effective aperture of 8.0 provides information for roughly 95-100MP.

Again: a crude, rough approximation but it's still pretty reliable and usable.
Even if we disregard all the potential issues and pitfalls inherit in the pixel shift process, the pixel size, pixel aperture,... the available potential resolution that could be collected (regardless of the sensor resolution itself) is limited.
So when you compare a 100% crop from the 100MP at f8.0 to a 100% crop from the 400MP at f6.0, the 100% crop of the 400MP image will not be able to compete in terms of fine detail/effective resolution no matter what.

I happen to own the same lens (actually 2 of them) and it's a stellar performer but it can't cheat physics when it comes to diffraction.

All I'm saying is that we should at least give the pixel shift 0,5x image a chance to potentially match the single shot 1x image before making a final judgement to say why it can't.

Click to expand...

I've been hard at work running a new series of experiments, now I have something to show for it. This is to address what Crispy_Bee was telling me/us that I had set up an unfair stage for the pixel shift 16-shots, because the taking effective aperture is far too small, causing diffraction blur to dominate, limiting resolution. I've deliberately chosen not to continue dabbling in the insurmountable challenge of the 0.5x-1x magnification range. The impact of higher magnifications on the effective aperture is too severe, and the availability of macro lenses with f/2 physical apertures for the GFX is pretty much non-existent, I don't want to be put in a spot where I would not be able to conduct further tests.

However, I realise the theory can be tested at any magnification scale, it does not have to be 0.5x vs 1x, and if we go with much lower magnification, we have more good choices of lenses! Enter the GF 110mm F2, far and away the highest resolving lens Fujifilm makes for their GFX system.

The magnification (computed from the tape measure placed in the capture frames) and apertures I shot at are as follows--

Pixel Shift 16-shots:
0.0496x
Nominal f/2, measured by PhotonstoPhotos Optical bench at f/2.06, is f/2.1622 effective
Nominal f/2.8, measured by PhotonstoPhotos Optical bench at f/2.87, is f/3.0124 effective

Stitching-equivalent:
0.0939x (it should be 0.0992x, thus giving pixel shift the magnification advantage very slightly)
Nominal f/2, measured by PhotonstoPhotos Optical bench at f/2.06, is f/2.2644 effective
Nominal f/2.8, measured by PhotonstoPhotos Optical bench at f/2.87, is f/3.1547 effective

The effective apertures are well within the values listed by Crispy-Bee. I have done nothing to bias the stitching-equivalent shot, rather I have actually given a slight advantage to pixel shift, if anything.

I made two test setups, one was of a 537 ppi smartphone display and the other was a section of a banknote. The raw files were all processed in Capture One this time (I downloaded a free trial version), so there is no raw processor conflict. The chosen lens, the GF110, is made by Fujifilm and designed for the sensor stack of their own cameras, so there is no likelihood of optical incompatibility or wrong float ring adjustment etc. I've also tested this copy of the GF110 and it exhibits high radial symmetry, very high resolution as is to be expected, and there is no meaningful focus tilt.

The GF110 has lower resolution at f/2 than at f/2.8, where it is near optimal. The crops from the test shots bear this out. Just because a wider aperture is available does not necessarily mean the system will automatically be capable of higher resolution, as we all know, which is why I wanted specifically to test using the 110mm, because of its extraordinary performance attained at just f/2.8, more than wide enough to meet Crispy_Bee's criteria for a fair test.

Kindly note whether you're looking at the nominally f/2 or f/2.8 crop samples, as indicated in the image captions.

Jack Hogan said:

Pixel aperture is a major determinant of 'sharpness' at this scale because it acts like a low-pass filter. The 'radius' of the filter (what JACS calls the kernel size) depends on the pixel configuration, the presence and/or shape of microlenses, etc. It is physically the same in both cases, but in the 1x case it acts on an image twice the 0.5x size so its effect is correspondingly smaller, meaning that it penalizes the 0.5x pixel-shift capture more.

Click to expand...

Jack, my latest experiments show you were right (not surprised!) on this. The pixel aperture is indeed a major determinant of 'sharpness'. The pixel shifted 16-shots never came close in acutance, but they are certainly very clean and mostly free of aliasing, albeit with a sprinkling of artifacts, despite the care I took to ensure a stable and controlled setup.




F/2.8 Banknote




F/2.8 Smartphone Display




F/2 Banknote




F/2 Smartphone Display

Jack Hogan said:

SamuelChia said:

Hi Jack! I was hoping you would respond, so thank you!

Click to expand...

Hi Samuel, my pleasure.

Jack Hogan said:

Jack Hogan said:

Leaving stitching aside and ignoring corner aberrations for a moment, let's say that we compared a central crop of the pixel-shifted image above to a single shot at twice the magnification, both with the same number of pixels. Twice the magnification would impact the effective f-number of the lens all else equal, hence its performance.

Click to expand...

What are the theoretical limits of resolution of each method?

Click to expand...

The variables in the two cases (0.5x pixel-shift vs 1x single shot aebe) are effective sampling rate, effective pixel aperture, and lens performance.

As JACS says, sampling rate affects the amount of aliasing because it determines the foldback frequency (Nyquist). In theory it can be considered somewhat equivalent in both cases because a sinusoid that would appear at c lp/mm on the sensor at 0.5x would appear at c/2 lp/mm on the same sensor at 1x.

Pixel aperture is a major determinant of 'sharpness' at this scale because it acts like a low-pass filter. The 'radius' of the filter (what JACS calls the kernel size) depends on the pixel configuration, the presence and/or shape of microlenses, etc. It is physically the same in both cases, but in the 1x case it acts on an image twice the 0.5x size so its effect is correspondingly smaller, meaning that it penalizes the 0.5x pixel-shift capture more.

Jack Hogan said:

Jack Hogan said:

How much better or worse is the lens at 0.5x than at 1x? It depends on the lens, and the two scenarios become harder to compare without specific lens performance data.

Click to expand...

The details are in my article, but just to respond directly to you, the Mejiro Genossen FL0530 lens was used wide open (F4) for the 0.5x and 1.0x samples in my OP, crops taken on-axis, and the MTF for that region is about 50% contrast at 100lp/mm for 0.5x, and about 40% contrast at 100lp/mm at 1.0x.

Click to expand...

The lens has different performance in the two cases, as you mention above. Also, 100 lp/mm becomes 50 lp/mm at 1x, so it would be good to know its performance at that spatial frequency. It would be safe to guess that it should be better than 100 lp/mm at 0.5x.

Effective pixel aperture and lens performance spatial frequency responses interact. They will change with different equipment and setups. With the setup shown in this thread they apparently favor the 1x single shot, as seen.

All of this could be quantified a bit better but I don't have time to do it at the moment.

Jack Hogan said:

Regarding diffraction affecting the comparison I made, could you tell me if this is correct:

Jack Hogan said:

The problem lies in the effective aperture vs resolution. Using pixel shift you're quadrupling the resolution (100MP to 400MP), not doubling the resolution, even though you're "only" trying to match double the magnification.

Click to expand...

Click to expand...

I should clarify that another simplification I am making is that the target or the sensor is monochrome for now. The difference can be subtle, or not. In this case, in the limit 100MP to 400MP is twice the linear magnification (horizontally and vertically)

Jack Hogan said:

Jack Hogan said:

Or to put it in numbers:A 100MP sensor hat a (theoretical) diffraction limit of around f7.1A 400MP sensor has a (theoretical) diffraction limit of around f3.5 (!)
At f6.0 the 100MP sensor will not be diffraction limited but the 400MP sensor will be diffraction limited (more than 1 full stop!). So if you want a 100% crop of the 400MP image to match the 100MP image at 100% you'd have to use a lens with an effective aperture of f3.5 or wider.

In our case at 0.5x that would require a lens with f2.4(!).

Click to expand...

Click to expand...

This requires further dissection because terms like 'resolution' and 'diffraction limited' are not clear in this context.

Jack

Click to expand...

My apologies for the delayed reply Jack, and thanks for writing with such high clarity about the details, it can be a bit hard to grasp the nuances sometimes. Also thanks to JACS for providing their perspective that is consistent with yours, and I guess I can say Jim Kasson indirectly as well.

My latest testing with the GF110mm at f/2 and 2.8 basically prove both of you right, that the pixel aperture of the 100S is the limiting factor, not the chosen lens aperture. The great thing about the 110 is that it reaches its peak resolution at about f/2.8 in many tests Jim has done, and this peak resolution is so high it is much higher than any other GF lens can achieve so far. So if I had chosen the 80 F1.7 instead, while a wider aperture can be had, higher resolution it is not.

It is interesting that the conclusions I have from the three categories of lenses are all the same, so pixel aperture effects are dominating the final result, even when there's a mild dose of diffraction in the mix (the GF35-70 F8 tests). My real-world test shots are demosaiced from bayer raw files, so the simulations are not as constrained as well as a theoretical experiment you could simulate mathematically. Still, it's nice to see reality match up with theory.

Jack Hogan said:

The lens has different performance in the two cases, as you mention above. Also, 100 lp/mm becomes 50 lp/mm at 1x, so it would be good to know its performance at that spatial frequency. It would be safe to guess that it should be better than 100 lp/mm at 0.5x.

Click to expand...

For the sake of completeness, I can report that 1x 50 lp/mm for this lens achieves 65% contrast on-axis.

John Sheehy said:

John Sheehy said:

SamuelChia posted:


Above result at 400% zoom

Click to expand...

This is certainly not a normal part of pixel-shifting. It is either because the two green channels or different sub-images have different gains or spectral sensitivities, which were not corrected in-camera, and/or the converter did some sloppy math.

My guess is that your camera *always* has mis-calibrated greens, but they are suppressed by both demosaicing and a higher frequency pattern at the pixel level that goes into visual extinction, whereas a shift pattern that thickens the influence of the green channel sensitivities to half the frequency, preventing visual extinction.

The two green channels in each sub-image and between the sub-images must either have the same sensitivity, or be calibrated before demosaicing, to avoid such patterns. If you bin 4x4 or downsample 25%, the pattern disappears, and you can filter the shifted image to get rid of it too, but that softens the image. If you made a flat-field image like people do in some astronomical photography, you can apply it without losing the contrast of detail.

Click to expand...

Hi John, thanks for sharing your insights. I did more looking and I suspect this was due to subtly changing ambient light (natural light) falling on the scene. In my later tests, I've reduced the proportion of natural light in the scene to irrelevant levels, I don't face this issue anymore.

Good work Samuel, nice to see that practice matches theory.

What is that pixelation in the 'stitched' Siemens Star images, incorrect white balance?

Pixelshift and Stiching , an extreme combination



115 , 1Gb PIXELSHIFT (TIFF) (panasonic s1R). (Downsampled to 700 MB each,)

Lense: Sigma 50 mm/1,2@6,3

Focus matrice: ( hight 15 *, breadth 8, (appoximatetely)

Parallallaxx error :not taken care of at all

Put into PS: Alignment and focus stacking, heavy cropping .

Remained:

3,4 GB= 225* 195 cm ( 300PPI) (sharpening : Capture sharpening and sharpening of the collage)

some comments/questions after having read your posts:



a) Better results at aperture less than 6,3?

b) Skip capture sharpening altogether?

c) A bigger collage possible if parallallaxx error is eliminated (then PTGUI is usable, not here)

d) perceived sharpness is a function of original resolution, downsamling procedure (I used PS algoritm, and sharpening.

The original resolution in turn depends on "Pixel aperture", quality of the stichning procedure (among other things)












Bjoern G