Jpeg2png: Silky smooth JPEG decoding – no more artifacts (2016)

by todsacerdoti • 6 years ago

187 83 6 years ago

webmaven• 6 years ago

"jpeg2png gives best results for pictures that should never be saved as JPEG. Examples are charts, logo's, and cartoon-style digital drawings."

Interesting value proposition.

There may be equivalents for other misused file types/formats. The obvious one is the inverse case of photos saved as GIFs, but I imagine audio and text (eg. PDF malapropisms?) are possible too.

dmitriid3• 6 years ago

And then someone uploads this image to twitter/facebook, and it gets triply re-encoded and resized with the worst algorithms known to man :(

BlackLotus89• 6 years ago

https://mobile.twitter.com/NolanOBrien/status/12045574989900... twitter now doesn't cripple uploaded images, but tries to preserve all information except exiv data.

jandrese5• 6 years ago

I just tried it with a text heavy screenshot, a common scenario where the JPEG encoding produces a poor result with lots of speckle around each letter.

The result is very good. The speckling is effectively eliminated without smearing the letters.

JPEG Encoded, 25% quality: http://jubei.ceyah.org/overcompressed/crop-jpg.png

JPEG2PNG: http://jubei.ceyah.org/overcompressed/crop-converted.png

indigo945• 6 years ago

I find the text on the reference decoded image easier to read, despite the colorful speckles surrounding the words. JPEG2PNG doesn't smear the letters a lot, but does appear to slightly deform them. Consider the leading "a" in "add license, readme", whose upper half seems to be closer together after the JPEG2PNG run.

However, now I'm wondering whether my impression that the reference decoded text is easier to read is a form of overfitting. I have spent the past 20 years reading low-quality JPEG screenshots on the internet, maybe the perceptual centers of my brain have developed an unblocking algorithm?

fireattack2• 6 years ago

Why the whole picture shifted? This can't be good

makapuf• 6 years ago

I d say the issue is the prpgram here, a compressor program might have heuristic to detect png-favored images vs photos. I cannot imagine we couldn't detect the easy cases reliably (because I'd say often images are more in a category or another). Crop the image to 64x64, try png and if the result is sufficiently small continue, else convert to jpg ?

0-_-0• 6 years ago

Now we just need an adversarially trained neural network to fill in detail [1] and we have a 21st century JPEG decoder!

[1] Implemented for a custom compressor: https://hific.github.io/

leeoniya• 6 years ago

here's one that upscales extremely well:

https://github.com/TianZerL/Anime4KCPP

lazyjones• 6 years ago

And one day, a law enforcement officer will use it on a bad quality JPEG image and one of the innocent persons whose photos were used to train the NN will be arrested.

kevin_thibedeau• 6 years ago

Nexus-6 just needs to get better at hiding from the law.

IgorPartola2• 6 years ago

This looks like it would be super useful in making phone photography look better. I can’t seem to find, is there a way to run this yourself?

etaioinshrdlu• 6 years ago

The interesting thing is that advanced encoders might create JPEGs that produce odd results with this decoder.

a_e_k• 6 years ago

Yes, this is definitely an interesting question. Earlier this year I implemented a version of trellis quantization for some compression experiments that I've been tinkering with in my spare time. My code considers more possibilities than just rounding to the nearest quantization step when it deems that the bit rate savings in after entropy encoding may be worth the additional image loss (e.g., it might even completely zero a coefficient). That would violate this decoder's assumption that the original DCT coefficient must have been within a limited range around the quantization step.

I know that mozjpeg [1] features trellis quantization for JPEG encoding. I wonder how this decoder would do with that?

[1] https://github.com/mozilla/mozjpeg

ache73• 6 years ago

It goes fine with mozjpeg. I tested a little, didn't find any problems.

wtallis4• 6 years ago

This strikes me as conceptually similar to the problem of printing floating point numbers. A naive algorithm will convert the binary floating point to decimal and print that out, with potentially many trailing digits that are more or less garbage. The more human-friendly algorithms figure out the shortest/simplest decimal that would be rounded to the same binary floating point quantity, and prints that instead.

jandrese5• 6 years ago

Yes, I've long wanted a floating point print function that understands that 3.999999999999 should be printed as "4".

colejohnson66• 6 years ago

My $.02: I’d prefer it say 3.9999999... than 4 so students learn of the inaccuracy of floating point. If one wants a rounded output, one should round before printing.

Kednicma• 6 years ago

You and the rest of this thread's participants may be interested to know that the problem was solved a while ago; you want the "Dragon4" algorithm from "How to Print Floating-Point Numbers Accurately" [0]. Indeed, the problem is difficult, and having implemented the algorithm, I can confirm that it has earned its reputation. Numeric algorithms are hard and the only reward is accuracy.

[0] https://lists.nongnu.org/archive/html/gcl-devel/2012-10/pdfk... or use your search engine

nextaccountic• 6 years ago

Nowadays I think you want Ryu

https://www.reddit.com/r/ProgrammingLanguages/comments/930pc...

ris• 6 years ago

Interesting project, but its name sells it incredibly short.

locofocos• 6 years ago

"For pictures that have been reencoded to JPEG multiple times I recommend shred." (https://www.gnu.org/software/coreutils/manual/html_node/shre...)

That's the funniest thing I've read in a while.

Lanrei• 6 years ago

That reminds me of a quote about chlorine trifluoride:

"...It can be kept in some of the ordinary structural metals—steel, copper, aluminum, etc.—because of the formation of a thin film of insoluble metal fluoride that protects the bulk of the metal ... If, however, this coat is melted or scrubbed off, and has no chance to reform, the operator is confronted with the problem of coping with a metal-fluorine fire. For dealing with this situation, I have always recommended a good pair of running shoes."

derefr2• 6 years ago

I've always wondered if there was something you really could do to neutralize a reaction that "really wants to happen" like that. Presuming you're in a well-equipped chem lab and have access to all sorts of non-mundane substances... maybe you could pour liquid helium on the fire? (It'd splatter from the Leidenfrost effect, but the flying Helium droplets wouldn't harm anything in any chemical sense, "merely" a physical "being very cold" sense. So, suit up against frostburn, and move away any glassware to ensure it isn't shattered by the opposite of unequal heating.)

dtparr• 6 years ago

Ah, Ignition! quotes are always great. Including test engineers in the list of things ClF3 is hypergolic with gets me every time

pents90• 6 years ago

It's long past time to retire Lena as the canonical test image in CG.

pkulak2• 6 years ago

Lena and Big Buck Bunny for life!

pkulak2• 6 years ago

Welp... I had no idea this was a centerfold photo and never gave it a second thought until now. I appreciate the heads up, op.

enriquto4• 6 years ago

Why? It's like "lorem ipsum" at this point.

NoSorryCannot2• 6 years ago

If lorem ipsum were softcore erotica...

KarlKemp2• 6 years ago

Fish have no concept of water in very much the same way as you not seeing the problem here.

crazygringo3• 6 years ago

Seriously. People wonder why there aren't enough women in CS.

Things like the canonical test image being a sexy "come hither" look, coming from a Playboy centerfold, definitely don't help.

And it's not enough to say "well the rest of the centerfold isn't shown". Come on. We all know the source of the image.

And "tradition" is far less important than CS being a welcoming environment for all people.

Computer science is a profession. Let's use images that are appropriate for a professional setting.

nisuni4• 6 years ago

As a woman in CS I couldn’t care less about Lena.

Let’s focus on real things, not on this phony issues!

anchpop• 6 years ago

I don't have any hard stats, but I'm going to try to talk about my experiences. I took a programming class in highschool - it was almost all boys at the time. This resembles my programming classes in college, and the professional environments I've worked in. Crucially, none of us back in highschool had ever even heard of Lena when we signed up for the class, and I doubt very many of the girls who decided not to take it had heard of Lena either.

It feels to me like there's some deeper reason for the gender disparity in tech than Lena. I don't have any issue with changing the picture whatsoever - I'm sure we can find some other picture to use as a baseline (big buck bunny?). But I would be willing to bet that it will do precisely nothing to change the fact that there aren't very many women in CS.

I can think of many other factors that seem more plausible to me. Me and my friends were into minecraft - at the time, you installed minecraft mods by overwriting files inside the minecraft.jar. If you wanted to set up a minecraft sever, you were given some command-line program to run and you had to set up port-forwarding in your router. Just doing this stuff makes you more comfortable with computers, and makes the jump to "I want to start programming" seem much smaller than someone who has never stepped outside Chrome and MS Office. And PC gaming is much more popular among young men than young women, so this avenue of becoming comfortable with the computer is going to be much more accessible to men. Not to mention, if you like games, eventually you'll want to make one - I think every PC gamer has at least thought about installing Unreal and trying to make their dreams into reality. I think if we actually want to increase how many women are in CS (and nothing would make me happier), this is the kind of stuff we should be thinking about, not whether image processing programmers use a picture of an attractive woman with bare shoulders too often.

rotskoff• 6 years ago

Came here to make the exact same comment, agree completely!

karmakaze• 6 years ago

For the first image I much prefer the jpeg decoded image to the de-noised one. The de-noised one just looks like it was sent through a low-pass/median filter and loses contrast, detail, and texture. If you flip between the full-sized images the jpeg2png looks like vaseline on the lens.

The naming is also odd. I don't imagine it would do well with text (which are the sorts of images I associate with png) that was jpeg encoded then decoded with it.

KarlKemp2• 6 years ago

It says it is specifically targeted at images that should have been png in the first place.

Scaevolus2• 6 years ago

Knusperli is similar: https://github.com/google/knusperli

Instead of trying to smooth the entire image, it reduces blocking artifacts by finding plausible coefficients that reduce discontinuities at block boundaries (jpegs are encoded as a series of 8x8 blocks). "jpeg2png gives best results for pictures that should never be saved as JPEG", but knusperli works well on normal photographic content, since it only tries to remove blocking artifacts.

A JPEG encoder quantizes DCT coefficients by rounding coefficients to the nearest multiple of the elements of the quantization matrix. For every coefficient, there is an interval of values that would round to the same multiple. A traditional decoder uses the center of this interval to reconstruct the image. Knusperli instead chooses the value in the interval that reduces discontinuities at block boundaries. The coefficients that Knusperli uses, would have rounded to the same values that are stored in the JPEG image.

enriquto4• 6 years ago

Instead of trying to smooth the entire image, it reduces blocking artifacts by finding plausible coefficients that reduce discontinuities at block boundaries

This is essentially the same thing, but using slightly different regularizers. In both cases, an image is found that whose jpeg compression is identical to the given one, by selecting the "most regular" image among the large family of such images. Jpeg2png minimizes the total variation over the whole image, and knusperly only along the boundaries of the blocks. The effect is quite similar at the end.

Scaevolus2• 6 years ago

The effects are quite different, since reducing total variation also reduces texture, which is important to perceived quality. Here's a direct comparison-- note how knusperli retains slightly more detail instead of oversmoothing: https://mod.ifies.com/f/200710_lena_decode_comparison.png

bscphil• 6 years ago

I was going to mention Knusperli as well. It's worth mentioning that the "no more artifacts" claim in the title depends heavily on what you understand to be an artifact. With modern, good quality JPEG encoders (e.g. mozjpeg), blocking only appears when the file is heavily compressed, or after a lot of generation loss. A decoder like this can't recover information that isn't there, obviously. So at more moderate levels of compression, where there's no noticeable blocking but the information inside the blocks is distorted or missing, it's not likely to help you a lot.

(As a citation, I can only offer the fact that I've frequently tried to clean up JPEG images I've found online, and Knusperli is usually my first step. It's never a sufficient step, though.)

ajb• 6 years ago

This is a poor-mans version of the deblocking filter which you find in recent video standards, but jpeg was to early to get it.

The advantage of those over this is that the encoder knows that the decoder is going to run the filter, and it can use that in its perceptual model to choose an encoding which looks most like the original after deblocking. Video codecs also use the deblocked version as reference frames. So you see these artifacts in mpeg-2 video and jpeg, but not more recent ones.

pornel• 6 years ago

I'd call newer formats to have poor-man's deblocking, because they literally apply blur to blocky images, instead of reconstructing an optimal gradient.

In newer codecs you can notice that sometimes blocks of 8 or 16 pixels get blurred with a 4px-wide blur, which is barely enough to cover up the edges, but not enough to mask square-ish shapes underneath.

crazygringo3• 6 years ago

This is really cool, but in other comments I'm reading that good JPEG encoders and decoders depend on knowing the other.

That you can design a better JPEG encoder for a known reference decoder (which decoded version best matches the original), and you can likewise design a better JPEG decoder for a known reference encoder.

But now that there are both encoders and decoders trying to be more efficient and non-blocky respectively... it feels like a mess. Even more so because a decoder never knows which encoder was used, and an encoder never knows which decoder.

Is there any solution to this mess? Is there a best practice here? Since most JPEG's are probably viewed on the web, do all browsers share the same decoder, and so effort is best spent on the encoder? Or even while encoder improvements are made specifically for the algorithms browsers use, can improvements to decoders still be made that remain improvements, instead of e.g. "overfitting" and actually making things worse?

This just feels like a bit of an "improvements race" on both sides which leaves me a little dizzy in trying to understand it...

IgorPartola2• 6 years ago

The obvious improvement is to use a better format than JPEG. Maybe something that can include information about the encoding process so the decoder can use that instead of guessing. Or maybe even a non-lossy format so we can focus more on data compression and less on visual compression.

ur-whale• 6 years ago

Surprised no one has tried to build a GANN to do this (remove JPEG artifacts).

Denoiser in ray tracing are quite a success, and this strikes me as quite similar.

In fact, now that I think of it, you probably don't even need a GANN, since you can create an infinite training set for quasi-free.

ache73• 6 years ago

I've seen these projects on github. I don't remember the names, only that waifu2x upscaler has an NN filter for JPEG artifacts.

Jpeg2png: Silky smooth JPEG decoding – no more artifacts (2016)

Related Stories