A smooth and sharp image interpolation you probably haven't heard of

https://wordsandbuttons.online/a_smooth_and_sharp_image_interpolation.html

by okaleniuk • 2 years ago

102 50 2 years ago

Why aren't the results of different interpolation methods shown in the website?

mtekman• 2 years ago

probably because the visual differences are so slight that there is no benefit in showing them to someone who isn't deeply entrenched in the topic

ImHereToVote3• 2 years ago

I think that would be me. Or anyone who would be interested in viewing this website. Would anyone not interested in this topic not want to ever see the differences?

okaleniuk3• 2 years ago

Good point. Why not? I'll add them.

WithinReason• 2 years ago

Don't forget Smoothstep:

https://en.wikipedia.org/wiki/Smoothstep

ImHereToVote3• 2 years ago

Thank you very much.

roger_• 2 years ago

Sounds like this: https://en.m.wikipedia.org/wiki/Inverse_distance_weighting

ur-whale2• 2 years ago

I note that the "example 1 on one dimension" diagram in the link you provided produces a very strange curve, typically not the one you'd draw intuitively by hand.

Asooka• 2 years ago

I am a bit confused how the formula is different from a simple lerp.

    k(x) = 1/x
    
        y_i * k(x - x_i) + y_i+1 * k(x_i+1 - x)
    F = --------------------------------------- =
             k(x - x_i) + k(x_i+1 - x)
    
      y_i / (x - x_i) + y_i+1 / (x_i+1 - x)
    = --------------------------------------- =
           1 / (x - x_i) + 1 / (x_i+1 - x)
    
      y_i * (x_i+1 - x) / A + y_i+1 * (x - x_i) / A
    = ---------------------------------------------
           (x_i+1 - x) / A + (x - x_i) / A
    
    where A = (x - x_i) * (x_i+1 - x)
    scaling to x_i+1 - x_i = 1, and using a = (x - x_i)
    
    F = (1 - a) * y_i + a * y_i+1

The only time this wouldn't hold is when A = 0, but then F is not defined anyway. However, the graph of F shown is decidedly not that of the lerp function.

okaleniuk3• 2 years ago

Oh, it seems that a square fell off. It should have been 1/x^2. 1/x works fine for bitmaps, but in 1-dimensional case 1/x indeed turns the whole thing into linear interpolation, you're right.

okaleniuk3• 2 years ago

Yep, there is a square in the source:

    // interpolation/extrapolation with no basis functions
    var weight2d1 = 2;
    const wf2d1 = function(x) {return 1./Math.pow(x, weight2d1);}

GrantMoyer2• 2 years ago

With k = 1/x, the inverse weight interpolation is equivalent to bilinear. Perhaps it's better to show the result for k = 1/x^2.

Neat aside, k = 1/x^∞ seems to be equivalent to nearest neighbor. (please excuse the abuse of notation)

mark-r• 2 years ago

Bilinear isn't 1/x, it's 1-x.

GrantMoyer2• 2 years ago

Right, but the more complicated expression ends up simplifying to the bilinear expression.

MereInterest• 2 years ago

The abuse of notation is a useful one, as it also captures the sense that as the exponent approaches infinity, the interpolation approaches nearest neighbor interpolation. This seems similar to the “max norm”, described in the wiki link below, which also uses infinity to describe the relationship with other distance formulas.

https://en.wikipedia.org/wiki/Uniform_norm

lloeki• 2 years ago

Could be nice if there were a log-scale slider to interactively vary the exponent from 1 to ∞-ish and see how that affects the frame-cropped preview.

deadbabe4• 2 years ago

AI based interpolation will soon make these mathematical interpolation methods obsolete right?

krapp4• 2 years ago

No, why would they? What would you expect an "AI" to do besides somehow implement an interpolation algorithm? The math is unavoidable.

deadbabe4• 2 years ago

With AI based interpolation, I should be able to zoom in forever without seeing pixelation at any level of zoom.

wmf• 2 years ago

AI algorithms encode domain knowledge in their weights that isn't present in simple interpolation functions. This can lead to better perceptual quality.

egorfine• 2 years ago

For practical reasons mostly yes.

For scientifically correct scaling (think forensics) you will of course need to interpolate the source data instead of generating synthetic data which is done by AI upscaling.

ur-whale2• 2 years ago

Not entirely sure the AI scaling is more or less "correct" than the "interpolate the source data" method.

Both use priors, the former a large and complex one, the latter a simplistic one, but both "invent data based on assumptions".

Dwedit3• 2 years ago

AI-based image upscaling happened many years ago. Waifu2x first released back in 2015, and ESRGAN first released in 2018.

waffletower2• 2 years ago

SwinIR hails from 2021: https://arxiv.org/abs/2108.10257

NayamAmarshe• 2 years ago

Yes, with Upscayl, we’re already doing that (https://upscayl.org).

It’s not as fast yet but it’s extremely good. Good enough that you could print a low resolution image after upscaling.

deadbabe4• 2 years ago

Thank you for everything you do!!

amelius4• 2 years ago

If you prefer hands with six fingers ...

edflsafoiewq• 2 years ago

In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. You can't use them for stuff like scaling an <img> in a browser.

Incidentally, the way I always thought it would work is the huge and slow AI would be used to discover a tiny algorithm that just so happens to be an amazing image rescaler. Instead we just ship the whole AI.

Anybody know of any work on the former strategy?

eru• 2 years ago

In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow.

Aren't games using these for real time upscaling many dozen times a second already?

Dwedit3• 2 years ago

My go-to for any kind of very fast image upscaling is the SuperXBR algorithm. It's fast, a pixel shader implementation runs at over 60FPS on Skylake IGP.

SuperXBR is in the family of Edge-Directed interpolation algorithms, so diagonal or curved lines will properly maintain their shape when upscaled, and not produce "staircase" artifacts that you see with other classical interpolation. It does not improve the sharpness though, images will still look blurry after being upscaled, just like when you perform bilinear upscaling.

But if you are doing any kind of non-realtime image upscaling today, you'd want to use the modern AI algorithms, such as Waifu2x or ESRGAN.

waffletower2• 2 years ago

I find the transformer based SwinIR model to be both relatively computationally efficient (when compared to ESRGAN) and provides excellent quality upscaling for a very general set of input images.

amelius4• 2 years ago

Ok. How can I apt-get install it?

PS: is it me or has it become more difficult to find cool stuff in your typical Linux distro?

Dwedit3• 2 years ago

It's not a standalone program, it's usually a set of pixel shaders that can be integrated into another program.

I once made an image viewer that would apply the effect, but it's not publicly released.

There is an extremely roundabout way to see the effect in action on a Linux system using a package you can apt-get. It involves installing RetroArch, starting the "Image Viewer" core, loading an image, loading the shader preset named "super-xbr-2p.glslp", changing the number of shaders from 3 to 2 to exclude the "custom Jinc resharper" step and selecting Apply, then scaling the window to be exactly 2x the size of the original image. You probably don't want to do this.

But if you really want a standalone program, I could build a Windows .NET program that will apply the shader to images you paste in from the clipboard.

jasonjmcghee• 2 years ago

I found the original "pixelated" image to look higher resolution / crisper / more detailed.

Really at any size I tried, I found the "upscale" to look blurry / low quality.

Am I an outlier here?

eddd-ddde• 2 years ago

You are right. It looks more pixelated, but better quality at the same time somehow?

chefandy• 2 years ago

There's no universal best option, here. Even two different pictures of the same thing at slightly different angles might look best with different upscaling algorithms. There are also variants on the smoother one to favor certain kinds of transitions. Really, though, all of those options suck. AI-powered options in Photoshop (much more controllable and reliable than SD or the like) actually invent details, so if provenance or accuracy is not important, (as it would be in a historic artifact, legal document, piece of visual art, or unique artisinal product,) that's always a possibility, but most projects have too many images to make tweaking each one individually viable.

Except in extremely rare situations, I flat-out refuse to work with low-quality source images in my design jobs. They can either give me a proper source image, pay me to rebuild it in a vector program if it's a logo or similar, or go down to the local sign shop and get someone to "design" something for them using a template. Most people end up having a higher quality copy around and just didn't know the difference between that and the screenshot they took of it, or the thumbnail they downloaded. At 300DPI with 4 colors, those flaws are glaringly obvious compared to screen resolution and sRGB color representation.

dheera• 2 years ago

I think at this point diffusion model super-resolution out-beats everything else.

Dylan168072• 2 years ago

When I think of "smooth and sharp" my mind goes to, uh, I guess "sharp bilinear" is the most common term?

It's like nearest neighbor interpolation but with less artifacting when the zoom is not an integer. The middle of each source pixel becomes a solid color block, and then bilinear filtering is applied to the single-pixel-wide seams between each block.

It's equivalent to doing a nearest neighbor upscale to floor(scale), then finishing with bilinear.

esafak• 2 years ago

There was a shareware site popular with the photo enthusiasts twenty years ago that implemented numerous resizing algorithms and allowed you to compare them visually. Unfortunately I can not recall its name.

A smooth and sharp image interpolation you probably haven't heard of

Related Stories