State of the art in upsizing algorithms?

Started Jun 9, 2014 | Discussions thread
Mark Scott Abeln
OP Mark Scott Abeln Forum Pro • Posts: 15,716
Re: State of the art in upsizing algorithms?

hjulenissen wrote:

If this model (or something equally good) can be built from the demosaiced image, then one could equally well do the processing in stages. The question is if there is some unique information that needs to be propagated between the traditional blocks.

The unique information would be the actual values read from the mosaic. There are common algorithms that assume correlations between color channels which is a rather perilous assumption, I’d think, in many cases, particularly where there is fine color differences that need to be preserved.

I think that demosaicing would benefit from insight into the noise, but also from knowing how the denoised samples would look like. I think that deconvolution might benefit from something similar. So perhaps some iterative process could reduce the problem that every stage would ideally like to work on the improved output (or not) of every other stage?

Yes, that is a good idea. Getting a good idea of what the overall image looks like can in turn help determine what demosaicing algorithm would work well (even locally), instead of simply treating each pixel as an individual in isolation from the whole. This might even help with automatic white balance and color transformation.

If you had access to a practical camera (say, a D800) and an impractical camera (say, a D800 using 20x20 stitched images at base ISO and using some multi-spectral pre-filter), then you could generate two images of the same scene. One test-image, and one reference. The question then is what nonlinear, signal-adaptive transformation (i.e. raw development) would make the test image as similar as possible to the reference image (in some numeric or perceptual fashion).

Well this is a big data problem, and so there is opportunity for improvement. Looking over the scholarly literature, so often I see judgements made on small data sets, along with an over-reliance on a purely inductive approach. This of course leads to rather narrow conclusions, limited to the very limited original data set.

If you're adventurous, you could use one of the frequency domain methods and effectively do your resampling when converting back to the xy domain. However, my experience with image frequency domain processing is quite limited, and I have seen minor divergence from expected results, even with 64-bit floating point precision.

I see "the frequency domain" as just a linear transform. So why should we expect something like a 2d DFT to matter much for image quality (processing speed is a different matter). Why not some time/space transform tradeoff (wavelet, filterbank,...)?

I believe that straight image FFT processing often has issues with padding. Specifically, the inherent assumption that the input is infinitely periodically repeated gives less than ideal results.

Perhaps those more into statistics than myself could offer some ideas. If each image sensel is a filtered, noisy sample of some process of specific stochastic parameters (unknown, but perhaps guessable, hopefully locally stationary), then one might be able to "guess" the image (possibly at a denser grid) while minimizing some expected error criterion. Again, I am sceptical (even if you could do the mathematics) that minimizing a simple error will end up looking good.

I wonder if changing the transform basis vector to square waves might help? That might eliminate some artifacts. But I’m not a mathematician.

Or is this all moot, current cameras provide "good enough" accuracy under good conditions, and we all end up doing unrealistic colors, over-sharpening etc because that is what we want in the first place?


 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
Post (hide subjects) Posted by
(unknown member)
(unknown member)
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow