State of the art in upsizing algorithms?

Started Jun 9, 2014 | Discussions
Mark Scott Abeln
Mark Scott Abeln Forum Pro • Posts: 15,915
State of the art in upsizing algorithms?

I’ve lately benefited from the great information in this article:

http://www.imagemagick.org/Usage/filter/nicolas/

which is on how best to use ImageMagick for resampling images — this is really good stuff especially if you get disappointing results from Photoshop’s clunky old algorithms.

Now I have to make a large image from a somewhat tight crop of an image, and it is one that will have a good amount of perspective correction. And since this will be a prominently displayed image, I’d like to really raise the bar on quality.

Is there any research on going directly from a mosaiced Bayer raw file to an upsampled image? Lately I’ve gotten comfortable with doing my own processing in Octave with good results. Clearly there will be at least some data loss when doing the demosaicing and upsizing steps independently.

Then there is the problem with perspective correction, which I’d like to be a data-preserving as possible.

The final image will be rather low-resolution, and so it needs all the help that can be given it.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
The_Suede Contributing Member • Posts: 652
Re: State of the art in upsizing algorithms?
1

The best you find at the moment are a priori trained superresolution engines. That's basically a neural network that tries to identify objects and shapes that it "knows about" since earlier, and then fills in the voids.

It's used a lot in aerial surveillance, it's also used to enhance the staples of ground surveillance - i.e faces and numberplates.

That has to be seen as a known unknown though, since your result will never be better than the training set of the neural.

Using classical interpolation schemes for Bayer images quite quickly falls apart due to noise effects. It's not like in weather forecasting, where a change of a hundredth of a percent in one single variable can make the difference between calm and a storm - but actually the metaphor isn't that far off... :/

hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?

Doing the processing "joint" as opposed to stage-wise might be expected to always be at least as good or better. One might also expect it to always be harder. In the end, each output pixel will probably mainly depend on a (small-ish) set of corresponding input pixels, plus some global/regional processing. If you can define this (nonlinear, signal-adaptive) function, then you can apply it in Octave or whatever.

The baseline for image scaling is working in some desirable domain (gamma vs linear, colour space, ...) and choosing the appropriate linear filter (some time vs frequency trade-off, for instance lanczos2/3). For some applications you might want to do post sharpening tightly integrated with the scaling operation. The ImageMagick link that you found is an excellent resource.

The "better than linear filtering" commercial applications that are available seems to often rely on some kind of edge-adaptive algorithm, meaning that sharp edges (e.g. text) can be blown up sharp and smooth.

I think that there are some applications of "non-local" methods for denoising/upscaling that might benefit certain kinds of images (i.e. exploiting image self-similarity).

If you put 100 monkeys in front of photoshop, they will (eventually) produce genuine works of art. Which is to say that if an image is really important to you, you can spend any amount of time in front of Photoshop (using both manual and automatic tools) to improve the image beyond what automatic tools are able to themselves as long as the quality metric is your own eyes.

For my own humble needs (18MP APS-C, A2 printer), I have found that the sensel grid is seldom a significant limitation (but lens quality/focus can be).

-h

(unknown member) Senior Member • Posts: 1,321
Re: State of the art in upsizing algorithms?

Just let the printer firmware do it. The engineers who designed the printing algorithms know are the best people to design the upsizing algorithms.

hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?
1

wchutt wrote:

Just let the printer firmware do it.

A number of experienced people seems to disagree with you:

http://www.luminous-landscape.com/forum/index.php?topic=90225.0

"Well, you image data does get transformed to 360ppi but it's not by the print driver, it's the OS level print pipeline that does it...ad it does it with a relatively poor bilinear algorithm. So, if you want the optimal results, you should consider upsampling the image data using a better algorithm in your image app or output app."

I believe that the most agreed-upon way to print for "optimal quality" is to resize images <=300/360 dpi (Canon/Epson) to 300/360 dpi, and those >300/360 dpi to 600/720dpi, then apply final sharpening.

The engineers who designed the printing algorithms know are the best people to design the upsizing algorithms.

The engineers at Epson etc ought to have a deep knowledge about the appearance of the dot pattern and halftoning algorithm of their printers. Those people have zero knowledge about the actual image that I may be processing at any given time, or my intents with it.

Even if they are excellent at their line of work, practical considerations (execution time, memory usage, backwards compability, OS APIs...) may mean that the released product/code does not reflect their skill or the state-of-the-art.

-h

Mark Scott Abeln
OP Mark Scott Abeln Forum Pro • Posts: 15,915
Re: State of the art in upsizing algorithms?
1

hjulenissen wrote:

The baseline for image scaling is working in some desirable domain (gamma vs linear, colour space, ...) and choosing the appropriate linear filter (some time vs frequency trade-off, for instance lanczos2/3). For some applications you might want to do post sharpening tightly integrated with the scaling operation. The ImageMagick link that you found is an excellent resource.

Dr. Robidoux, the author of the paper, gave me the link himself.

One striking finding in the link is that while downsampling ought to be done in a linear space, up sampling instead should be done in a gamma space or another space that is gamma-like such as Lab.

Now I’m not quite sure how this finding ought to be applied — that demosiacing ought to be done after gamma correction, if I am ultimately upsizing an image?

From my reading and experimentation, it seems that demosaicing ought to be done rather late in the image processing pipeline. One manual for dcraw makes the case that it ought to be done after white balance, and for the same reasons it seems prudent to do it after color correction. But after gamma also?

One technique that I found very useful, back in 2008 with Photoshop CS3, was using ACR to produce a  larger image — I used about 11 MP — from my 6 MP camera images I got from my old D40. I found that I could do a lot of geometric image transformation, including perspective distortion, barrel distortion removal, defringing, etc. without losing too much detail or having too much additional softening, and the perception of sharpness could be restored with good downsampling algorithms and sharpening. Note that this was a terribly slow ordeal on my old iMac with 1 GB memory but well worth it.

The "better than linear filtering" commercial applications that are available seems to often rely on some kind of edge-adaptive algorithm, meaning that sharp edges (e.g. text) can be blown up sharp and smooth.

I think that there are some applications of "non-local" methods for denoising/upscaling that might benefit certain kinds of images (i.e. exploiting image self-similarity).

My photograph is of a large, historic, top-quality mosaic, and so has lots of brightly colored triangles and trapezoids separated by thin lines of mortar. This is a subject that could really benefit from an algorithm that detects and preserves edges.

There is that commercial package (Perfect Resize 8) which upsizes using fractal decomposition. I’ll see if this might work well also.

Another direction I’ve been thinking of is using software to create a Scalable Vector Graphics or SVG representation of my image — IF it can detect the edges well enough and crisply enough, then I can blend the perfectly resized vector graphics with the upsampled image. Seems plausible, but I don’t know if anyone’s tried this or what software might work best.

For my own humble needs (18MP APS-C, A2 printer), I have found that the sensel grid is seldom a significant limitation (but lens quality/focus can be).

If I had known that my photograph would be purchased, I’d have done a more heroic effort in my photography.  But as it happens, I did use good technique (heavy tripod, manual live view focus, mirror lockup and remote shutter release) and a good lens, because I did want a quality final image for my own use. 
My concern is that the final image will be printed at 100 DPI or maybe less (they haven’t yet told me the final dimensions). Certainly I’ve had good success with even lower resolution large prints, albeit heavily processed — and most civilians aren’t pixel peepers — but I do want even better results this time.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
Detail Man
Detail Man Forum Pro • Posts: 17,172
Re: State of the art in upsizing algorithms ?

Mark Scott Abeln wrote:

hjulenissen wrote:

The baseline for image scaling is working in some desirable domain (gamma vs linear, colour space, ...) and choosing the appropriate linear filter (some time vs frequency trade-off, for instance lanczos2/3). For some applications you might want to do post sharpening tightly integrated with the scaling operation. The ImageMagick link that you found is an excellent resource.

Dr. Robidoux, the author of the paper, gave me the link himself.

One striking finding in the link is that while downsampling ought to be done in a linear space, up sampling instead should be done in a gamma space or another space that is gamma-like such as Lab.

Now I’m not quite sure how this finding ought to be applied — that demosiacing ought to be done after gamma correction, if I am ultimately upsizing an image?

From my reading and experimentation, it seems that demosaicing ought to be done rather late in the image processing pipeline. One manual for dcraw makes the case that it ought to be done after white balance, and for the same reasons it seems prudent to do it after color correction. But after gamma also?

One technique that I found very useful, back in 2008 with Photoshop CS3, was using ACR to produce a larger image — I used about 11 MP — from my 6 MP camera images I got from my old D40. I found that I could do a lot of geometric image transformation, including perspective distortion, barrel distortion removal, defringing, etc. without losing too much detail or having too much additional softening, and the perception of sharpness could be restored with good downsampling algorithms and sharpening. Note that this was a terribly slow ordeal on my old iMac with 1 GB memory but well worth it.

When I used to shoot JPG only with earlier cameras, and used (the 8-bit processing) PaintShop Pro 9.01 as an editor to post-process those JPGs, I would use PSP (Bicubic, then with fixed parameters) up-sampling to create BMPs (not needing the alpha-channels that TIFF image-files include) of an integer multiple of the original JPG's pixel-size, and perform editing on that item.

Found that a great deal of nasty looking (too often spikey and prominent) artifacts that tweaking with various PSP controls often caused were significantly smoothed, and thus attenuated, due to low-pass filtering occuring upon down-sampling prior to usually applying some post-downsampling USM, then followed by using PSP's export to JPG function - which, similar to Adobe's various similar offerings, it turns out does not allow for a JPG Quality Factor=100%, though it is possible to specify the Chroma Sub-sampling mode in versions of PSP.

(Whether or not in-processing cropping was performed), I would (at times) select whole-number fractional up-sampling (followed by down-sampling) ratios to use in that processing. Down-sampling (decimating only) by an odd integer such as 3 seems to collapse 3x3 arrays of pixels into the image-data of a single pixel - although this is impractical to achieve when cropping in editing.

Later, using (the 16-bit processing) PSP X4.3 (which I generally only use for it's 16-bit USM in my RAW>JPG workflow), similar positive effects regarding RGB histogram smoothness can be achieved.

If a (TIF or JPG) source-image is of smaller pixel-size than my desired size, I still up-sample on the TIF level prior to editing with DxO Optics Pro and/or PSP X4.3, then down-sample using a 16-bit application with a GUI using some particular Lanczos-3 algorithm (that appears identical to the result of performing Lanczos-3 down-sampling on a TIF using RT 4.x in some eyeball tests).

I don't think that the application that I use performs (a linearized) re-sampling. Don't know about RT 4.x's re-sampling processes. That would be good to know. Considering up-sampling only, it appears (in that paticular case) that (gamma-corrected process) results may be better ?

B-Spline, Gaussian, etc. (at least in the 8-bit XnView - which has an excellent export to JPG that allows straightforward user-control of Quality Factor as well as Chroma Sub-sampling) yield very soft results. As a result, I use (described unspecified flavor of 16-bit) Lanczos-3 for up/down-sampling.

PSP (X4, anyway) uses a Bicubic (with a "sharpness/softness" adjustment slider that does not appear to have very much effect), and (I think) the Adobe apps still utilize their own "in-house" Bicubic item.

The "better than linear filtering" commercial applications that are available seems to often rely on some kind of edge-adaptive algorithm, meaning that sharp edges (e.g. text) can be blown up sharp and smooth.

I think that there are some applications of "non-local" methods for denoising/upscaling that might benefit certain kinds of images (i.e. exploiting image self-similarity).

My photograph is of a large, historic, top-quality mosaic, and so has lots of brightly colored triangles and trapezoids separated by thin lines of mortar. This is a subject that could really benefit from an algorithm that detects and preserves edges.

There is that commercial package (Perfect Resize 8) which upsizes using fractal decomposition. I’ll see if this might work well also.

Another direction I’ve been thinking of is using software to create a Scalable Vector Graphics or SVG representation of my image — IF it can detect the edges well enough and crisply enough, then I can blend the perfectly resized vector graphics with the upsampled image. Seems plausible, but I don’t know if anyone’s tried this or what software might work best.

For my own humble needs (18MP APS-C, A2 printer), I have found that the sensel grid is seldom a significant limitation (but lens quality/focus can be).

If I had known that my photograph would be purchased, I’d have done a more heroic effort in my photography. But as it happens, I did use good technique (heavy tripod, manual live view focus, mirror lockup and remote shutter release) and a good lens, because I did want a quality final image for my own use.

My concern is that the final image will be printed at 100 DPI or maybe less (they haven’t yet told me the final dimensions). Certainly I’ve had good success with even lower resolution large prints, albeit heavily processed — and most civilians aren’t pixel peepers — but I do want even better results this time.

It sounds like the (composite) "down-sampling" (including presentation effects) could dominate ?

DM

The_Suede Contributing Member • Posts: 652
Re: State of the art in upsizing algorithms?
1

Demosaic in gamma corrected space does induce color fringes along sharp diagonal edges. I haven't seen any type of algorithm that would avoid this, it would have to be specially written for the application.

(unknown member) Senior Member • Posts: 1,321
Re: State of the art in upsizing algorithms?

The lab who makes my prints know best. They have to compete in the market place. They upscale a lot of work for corporate clients. I will depend on people whose livelihood requires the best possible results day in- day out year after year. You can rely on arm-chair experts at LuLA if you prefer.

JimKasson
JimKasson Forum Pro • Posts: 30,191
Re: State of the art in upsizing algorithms?

wchutt wrote:

The lab who makes my prints know best. They have to compete in the market place. They upscale a lot of work for corporate clients. I will depend on people whose livelihood requires the best possible results day in- day out year after year. You can rely on arm-chair experts at LuLA if you prefer.

Let me get this straight. You're saying you've got this great lab, and when they upsize, they let the printer firmware (probably the driver, actually) do the resampling?

Jim

-- hide signature --
 JimKasson's gear list:JimKasson's gear list
Nikon Z7 Fujifilm GFX 100 Sony a7R IV Sony a9 II +1 more
JimKasson
JimKasson Forum Pro • Posts: 30,191
Re: State of the art in upsizing algorithms?

Mark Scott Abeln wrote:

Is there any research on going directly from a mosaiced Bayer raw file to an upsampled image? Lately I’ve gotten comfortable with doing my own processing in Octave with good results. Clearly there will be at least some data loss when doing the demosaicing and upsizing steps independently.

I've long thought that was the way to go, but I haven't seen any evidence of commercial tools working that way. Demosaicing involves building a model of the scene based on sparse samples, and that model should be valuable in producing an upsampled result; it seems a shame to throw it away.

Take a look at some of the frequency-domain methods in this survey paper:

http://www4.comp.polyu.edu.hk/~cslzhang/paper/conf/demosaicing_survey.pdf

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

If you stay in the space domain, there's a possible win in not going to 16-bit integer precision between demosaicing and resampling. That win should be more significant as the scaling methods and sharpening filters get more complex.

Jim

-- hide signature --
 JimKasson's gear list:JimKasson's gear list
Nikon Z7 Fujifilm GFX 100 Sony a7R IV Sony a9 II +1 more
Mark Scott Abeln
OP Mark Scott Abeln Forum Pro • Posts: 15,915
Frequency domain
1

JimKasson wrote:

Mark Scott Abeln wrote:

Is there any research on going directly from a mosaiced Bayer raw file to an upsampled image? Lately I’ve gotten comfortable with doing my own processing in Octave with good results. Clearly there will be at least some data loss when doing the demosaicing and upsizing steps independently.

I've long thought that was the way to go, but I haven't seen any evidence of commercial tools working that way. Demosaicing involves building a model of the scene based on sparse samples, and that model should be valuable in producing an upsampled result; it seems a shame to throw it away.

Yes it does seem to be a shame. But every time I think about how some aspect of image processing could be improved, the solution seems to be turning the image processing pipeline into a mathematical construct of Byzantine complexity. Not that it is necessarily a bad thing, just that it is difficult for me to conceive.

Take a look at some of the frequency-domain methods in this survey paper:

http://www4.comp.polyu.edu.hk/~cslzhang/paper/conf/demosaicing_survey.pdf

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

If you stay in the space domain, there's a possible win in not going to 16-bit integer precision between demosaicing and resampling. That win should be more significant as the scaling methods and sharpening filters get more complex.

My experience with frequency domain processing was disappointing, but I admit that I really don’t remember most of the math, except vaguely. Certainly, loss of precision is important, and I’m seeing a number of places where lack of precision in general harms the final image. Perhaps this is a good opportunity to experiment with extended precision arithmetic in Mathematica or whatever else is out there — since I’m concerned with a one-off image, long processing times is hardly an issue.  However, I do need speed for the typical large project that I do a couple times a year.

My current favorite raw processing software is Raw Therapee, which still has some problems, sadly, particularly the white balance when using CIECAM02. It can use 64 bit processing, and I’m generally pleased by the fine gradation of tones that it can create when doing shadow brightening.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?

wchutt wrote:

The lab who makes my prints know best.

In my experience, this is not generally true. I have no knowledge of your specific lab, but I assume that you are offering this as a general advice.

They have to compete in the market place.

One image quality alone? Hardly. The labs where I live seems to compete primarily on cost.

They upscale a lot of work for corporate clients.

And do these corporate clients care? I know musicians who play for a lot of corporate clients without this being an indication of quality.

I will depend on people whose livelihood requires the best possible results day in- day out year after year.

Day in day out often means finding the compromise that is the least fuzz. Just like a Toyota is a robust, economic car, but might not fit everyones opionon of a "great" car.

You can rely on arm-chair experts at LuLA if you prefer.

I am sorry, but I think that your arm-chair advice is a poor one. I'd rather rely on someone who can present coherent arguments and results.

-h

hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?
1

JimKasson wrote:

I've long thought that was the way to go, but I haven't seen any evidence of commercial tools working that way. Demosaicing involves building a model of the scene based on sparse samples, and that model should be valuable in producing an upsampled result; it seems a shame to throw it away.

If this model (or something equally good) can be built from the demosaiced image, then one could equally well do the processing in stages. The question is if there is some unique information that needs to be propagated between the traditional blocks.

I think that demosaicing would benefit from insight into the noise, but also from knowing how the denoised samples would look like. I think that deconvolution might benefit from something similar. So perhaps some iterative process could reduce the problem that every stage would ideally like to work on the improved output (or not) of every other stage?

If you had access to a practical camera (say, a D800) and an impractical camera (say, a D800 using 20x20 stitched images at base ISO and using some multi-spectral pre-filter), then you could generate two images of the same scene. One test-image, and one reference. The question then is what nonlinear, signal-adaptive transformation (i.e. raw development) would make the test image as similar as possible to the reference image (in some numeric or perceptual fashion).

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

I see "the frequency domain" as just a linear transform. So why should we expect something like a 2d DFT to matter much for image quality (processing speed is a different matter). Why not some time/space transform tradeoff (wavelet, filterbank,...)?

I believe that straight image FFT processing often has issues with padding. Specifically, the inherent assumption that the input is infinitely periodically repeated gives less than ideal results.

Perhaps those more into statistics than myself could offer some ideas. If each image sensel is a filtered, noisy sample of some process of specific stochastic parameters (unknown, but perhaps guessable, hopefully locally stationary), then one might be able to "guess" the image (possibly at a denser grid) while minimizing some expected error criterion. Again, I am sceptical (even if you could do the mathematics) that minimizing a simple error will end up looking good.

Or is this all moot, current cameras provide "good enough" accuracy under good conditions, and we all end up doing unrealistic colors, over-sharpening etc because that is what we want in the first place?

-h

The_Suede Contributing Member • Posts: 652
Re: State of the art in upsizing algorithms?
1

JimKasson wrote:

I've long thought that was the way to go, but I haven't seen any evidence of commercial tools working that way. Demosaicing involves building a model of the scene based on sparse samples, and that model should be valuable in producing an upsampled result; it seems a shame to throw it away.

Take a look at some of the frequency-domain methods in this survey paper:

http://www4.comp.polyu.edu.hk/~cslzhang/paper/conf/demosaicing_survey.pdf

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

If you stay in the space domain, there's a possible win in not going to 16-bit integer precision between demosaicing and resampling. That win should be more significant as the scaling methods and sharpening filters get more complex.

Jim

The data accuracy is never greater than 14-bits worth of precision, meaning that only resonant algorithms are affected by using floating point. But since you have to handle maths functions, float isn't really that big of a performance concern in modern processors.

............

One of the things about the sparse sampling is that it is discrete in nature, set in a fixed pattern of sparsity (the Bayer pattern).

The Bayer pattern is the most densely populated sparsity root possible in the chain, so it (or rather; the mathematically perfect re-modelling of the underlying material in it) defines the maximum possible resolution you can hope to achieve.

You don't actually GAIN anything by trying to merge the re-population of the sparse samples and the scaling, since your limitation is in the original sparse pattern. What you could gain is maybe some computational efficiency - IF you managed to write a sufficiently streamlined code for it. But I actually doubt that is possible, seeing the problems I have already now actually filling the data pipeline in modern processors.

What I guess you COULD do with great success is to minimize the amount of scaling aliasing - meaning a slightly blurry output image, but without nasty jaggies - but you won't "gain" any real resolution.

............

Well written code spends about 30% of the time waiting for data to process. As long as you keep your working dataset in L2-L3, it doesn't (much) access speed difference what thread you're in if the code is optimized.

hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?

The_Suede wrote:

The Bayer pattern is the most densely populated sparsity root possible in the chain, so it (or rather; the mathematically perfect re-modelling of the underlying material in it) defines the maximum possible resolution you can hope to achieve.

If you (successfully) model the input scene, and for some definitions of "resolution", I guess that there are few limits to the output resolution that can be had.

You don't actually GAIN anything by trying to merge the re-population of the sparse samples and the scaling, since your limitation is in the original sparse pattern.

If scaling and deconvolution performs worse than expected due to the nature of a demosaiced signal, then there might be potential gains to be had from making them aware of this process (or integrating the whole thing)?

What you could gain is maybe some computational efficiency

It seems that computational efficiency is not what the OP had in mind. I am guessing that the commercial parties have optimized their 16-bit fixed-point (?) SIMD implementations quite well for x86 and a given, moderately good, quality level?

-h

Mark Scott Abeln
OP Mark Scott Abeln Forum Pro • Posts: 15,915
Re: State of the art in upsizing algorithms?

hjulenissen wrote:

If this model (or something equally good) can be built from the demosaiced image, then one could equally well do the processing in stages. The question is if there is some unique information that needs to be propagated between the traditional blocks.

The unique information would be the actual values read from the mosaic. There are common algorithms that assume correlations between color channels which is a rather perilous assumption, I’d think, in many cases, particularly where there is fine color differences that need to be preserved.

I think that demosaicing would benefit from insight into the noise, but also from knowing how the denoised samples would look like. I think that deconvolution might benefit from something similar. So perhaps some iterative process could reduce the problem that every stage would ideally like to work on the improved output (or not) of every other stage?

Yes, that is a good idea. Getting a good idea of what the overall image looks like can in turn help determine what demosaicing algorithm would work well (even locally), instead of simply treating each pixel as an individual in isolation from the whole. This might even help with automatic white balance and color transformation.

If you had access to a practical camera (say, a D800) and an impractical camera (say, a D800 using 20x20 stitched images at base ISO and using some multi-spectral pre-filter), then you could generate two images of the same scene. One test-image, and one reference. The question then is what nonlinear, signal-adaptive transformation (i.e. raw development) would make the test image as similar as possible to the reference image (in some numeric or perceptual fashion).

Well this is a big data problem, and so there is opportunity for improvement. Looking over the scholarly literature, so often I see judgements made on small data sets, along with an over-reliance on a purely inductive approach. This of course leads to rather narrow conclusions, limited to the very limited original data set.

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

I see "the frequency domain" as just a linear transform. So why should we expect something like a 2d DFT to matter much for image quality (processing speed is a different matter). Why not some time/space transform tradeoff (wavelet, filterbank,...)?

I believe that straight image FFT processing often has issues with padding. Specifically, the inherent assumption that the input is infinitely periodically repeated gives less than ideal results.

Perhaps those more into statistics than myself could offer some ideas. If each image sensel is a filtered, noisy sample of some process of specific stochastic parameters (unknown, but perhaps guessable, hopefully locally stationary), then one might be able to "guess" the image (possibly at a denser grid) while minimizing some expected error criterion. Again, I am sceptical (even if you could do the mathematics) that minimizing a simple error will end up looking good.

I wonder if changing the transform basis vector to square waves might help? That might eliminate some artifacts. But I’m not a mathematician.

Or is this all moot, current cameras provide "good enough" accuracy under good conditions, and we all end up doing unrealistic colors, over-sharpening etc because that is what we want in the first place?

Hmmm.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
hjulenissen Senior Member • Posts: 2,279
Re: State of the art in upsizing algorithms?
1

Mark Scott Abeln wrote:

hjulenissen wrote:

If this model (or something equally good) can be built from the demosaiced image, then one could equally well do the processing in stages. The question is if there is some unique information that needs to be propagated between the traditional blocks.

The unique information would be the actual values read from the mosaic.

Let me try to phrase what I was trying to say differently:

The intuitive approach to raw development pipeline is something ala what is done today; a stepwise "reversal" of the limitations made by the camera and associated capture mechanisms. For a radically different (more complex) algorithm, where the blocks are made "aware" of each other, or merged to have some merit, there must be some information _besides_ the demosaiced pixels that can benefit subsequent stages.

There are common algorithms that assume correlations between color channels which is a rather perilous assumption, I’d think, in many cases, particularly where there is fine color differences that need to be preserved.

There is ample evidence that the color-difference-channels _tends_ to be spatially smooth for natural scenes. Well-known exceptions includes birds feathers.

http://en.wikipedia.org/wiki/YCbCr

I wonder if changing the transform basis vector to square waves might help? That might eliminate some artifacts. But I’m not a mathematician.

I think of a finite-length DFT as a linear weighting applied to the input sequence. The DFT has some nice properties (it is orthogonal and ortonormal, and it can be computed cheaply). Further, it can be interpreted as "frequencies", something that is a lot more intuitive than "seemingly random correlator XYZ" that one might get from running some kind of PCA or ICA decomposition.

I am not so convinced that applying some linear transform is magically going to make the problem simple. Then, I could be wrong.

I would rather think that a problem description/solution that explicitly modelled 1)The signals that we expect to encounter (possibly adapted to the actual image), and 2)The analysis carried out in the human visual system, would have more promise. Now, understanding the HVS is not something that I pretend to do, and last I heard science struggles with that as well.

-h

JimKasson
JimKasson Forum Pro • Posts: 30,191
Re: State of the art in upsizing algorithms?

Mark Scott Abeln wrote:

I wonder if changing the transform basis vector to square waves might help? That might eliminate some artifacts.

You mean like a Hadamard transform?

-- hide signature --
 JimKasson's gear list:JimKasson's gear list
Nikon Z7 Fujifilm GFX 100 Sony a7R IV Sony a9 II +1 more
JimKasson
JimKasson Forum Pro • Posts: 30,191
Re: State of the art in upsizing algorithms?
1

Mark Scott Abeln wrote:

The unique information would be the actual values read from the mosaic. There are common algorithms that assume correlations between color channels which is a rather perilous assumption, I’d think, in many cases, particularly where there is fine color differences that need to be preserved.

Yes. After demosacing, all the pixels are equal in the eyes of further processing. However, with a nod to George Orwell, some of the pixels, the ones that haven't been interpolated or otherwise guessed at, should be "more equal" than the others, and carry more weight in subsequent resampling.

Or maybe not. I'm just speculating here.

Jim

-- hide signature --
 JimKasson's gear list:JimKasson's gear list
Nikon Z7 Fujifilm GFX 100 Sony a7R IV Sony a9 II +1 more
Keyboard shortcuts:
FForum MMy threads