JPEG Committee releases a call for evidence for image compression based on AI

https://jpeg.org/items/20200217_press.html

by jonsneyers • 6 years ago

113 70 6 years ago

jpeg.org

jonsneyers4• 6 years ago

Highlights:

- Call for Evidence launched for AI-based image codecs

- Call for Evidence launched for point cloud codecs

- Next-gen image codec JPEG XL reaches Draft International Standard stage

- Light-weight JPEG XS codec is looking at compression of camera raws

bloak• 6 years ago

Apparently they're "responsible for the popular JPEG, JPEG 2000, JPEG XR, JPSearch, JPEG XT and more recently, the JPEG XS, JPEG Systems, JPEG Pleno and JPEG XL families of imaging standards".

Some of those may be more popular than others:

https://xkcd.com/2254/

jonsneyers4• 6 years ago

JPEG 2000 is actually quite popular, for example any movie you see in a cinema is encoded using JPEG 2000 – they don't use video codecs for digital cinema, but encode each frame separately, to avoid video compression artifacts. Also in medical imaging, nearly everything is using JPEG 2000. Apple products have supported it for a long time.

JPEG XR is also known as "Windows Media Photo" and "HD Photo" and has been supported in the Microsoft ecosystem since Windows Vista and Internet Explorer 9.

JPEG XS, XL and Pleno are of course much newer and will still have to be battle-tested.

But yes, none of the newer JPEG standards have reached the popularity of the first JPEG codec, which is by far the most popular image format ever.

todotask2• 6 years ago

JPEG XL repo seems to have adopt Google Brunsli.

PaulHoule2• 6 years ago

The FBI uses JPEG 2000 for fingerprints.

Ajedi322• 6 years ago

Personally I'm really excited about the potential of JPEG XS for wireless VR. Currently most of the high quality wireless setups require expensive, high-bandwidth solutions like WiGig in order to avoid the latency penalty of compressing the video stream. If JPEG XS can get those streams small enough to fit into a regular 5 GHz Wi-Fi connection, that would be a _huge_ deal for the affordability of wireless VR setups.

clouddrover• 6 years ago

An advantage JPEG XL has over previous efforts is that it’s royalty free and that JPEGs can be losslessly converted to JPEG XL (and back again) and still get about a 20% file size reduction. Being able to convert to JPEG XL with no quality loss will help adoption.

userbinator2• 6 years ago

for example any movie you see in a cinema is encoded using JPEG 2000 – they don't use video codecs for digital cinema, but encode each frame separately, to avoid video compression artifacts.

They also use very expensive dedicated hardware to decode Motion JPEG2000 at movie framerate, because it's far more computationally intensive than regular JPEG or even the newer video codecs like H264/H265.

rasz• 6 years ago

Red cameras "Redcode" is just obfuscated JPEG 2000 pretending to be some novel thing.

The_rationalist2• 6 years ago

I wonder if this is a reaction to the issue I opened :)

https://github.com/google/brunsli/issues/60

PaulHoule2• 6 years ago

I wouldn't be surprised if you could make an AI that knows something about what scenes look like, and something about what JPEG artifacts look like that and could clean up existing JPEG images.

The_rationalist2• 6 years ago

Isn't that task called image denoising? : https://paperswithcode.com/task/image-denoising

ska• 6 years ago

Artifact removal and denoising are only sort-of-kind-of the same thing.

At a high enough level, you are using a model of some sort to estimate (or guesstimate) changes in the image but the models for "X type of noise from the sensor" and "artifact due to block size choice" or "artifact due to aliasing in the RF domain" or whatever are quite different.

derefr2• 6 years ago

The JPEG artifact “look” comes from the fact that JPEG data is essentially just a matrix of point-samples (like any other raster format) which happen to carry a gradient and angle, rather than a solid color. So, at a low sampling resolution, you see weird discontinuities where one sample gradient (gradel?) sharply transitions to another sample gradient, with a discontinuity in either color or angle.

An AI that “fixes” JPEG artifacts would just be a regular upscaling/supersampling AI—just one trained on JPEG sample-matrices instead of BMP sample-matrices, so that it can take advantage of the extra information each JPEG sample-point carries.

skunkworker2• 6 years ago

My understanding of lossy compression like JPEG may be limited, but wouldn't this ML network be guessing the eigenvectors to put back into a matrix that were removed because they weren't dominant enough?

rahidz• 6 years ago

Get ESRGAN and a model that's trained on JPG compression and you got it https://upscale.wiki/wiki/Model_Database has a nice list, and I use IEU to make things even easier: https://github.com/ptrsuder/IEU.Winforms

Ajedi322• 6 years ago

I find the idea of AI-based image compression rather fascinating. In theory it seems like a great fit for AI; "perceived quality" is exactly the sort of fuzzy metric AI tends to be good at optimizing for, and a compression algorithm possessing an understanding of high-level visual features like shapes, lighting, etc seems like it would have a lot of potential.

Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data; which could lead to some really bizarre looking compression artifacts when the system breaks down.

derefr2• 6 years ago

Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data

Or, you could have a compression system created specifically for photographic images (as opposed to a “general compressor” like LZ77), where these high-level features are part of the (de)compressor itself.

Now, lossy domain-specific compressors are pretty common; one good example is audio “voice codecs” for telecom, in which every snippet of audio is encoded as a phoneme-like symbol, such that noise comes out the other end as a gobbledygook that sounds a lot like words.

But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) In such a compressor, the input would be “rebased” against the model, such that the output looks like an instruction stream ala “start with [this synthesized template with these parameters]; and then add point-modifications {A, B, C, ...} to get from there back to the original input.” So inputs that were more like something it “understood” would compress well (i.e. just be described by a template) while inputs that were less-well-understood wouldn’t get much from the procedural-generation phase, and would end up being specified entirely as point-corrections, and so not really compress at all.

imurray• 6 years ago

A relevant NeurIPS 2019 paper for lossless compression (for e.g., medical imaging):

https://papers.nips.cc/paper/9383-integer-discrete-flows-and...

jfkebwjsbx• 6 years ago

FLAC for audio is one

sdenton4• 6 years ago

It's pretty easy to build a lossless learned encoder. Encode the image with your model, then apply a lossless encoding to the error. You can also include a measure of the entropy of the error in your model's loss function.

The model should make the errors small, and including the entropy measure of the error in the loss function means that the model produces "cheaper to correct" errors.

A nice keyword on this front is "bits back argument".

woeirua• 6 years ago

What you're basically describing is compressive sensing. The basic idea is that if you can find a domain in which your data is maximally sparse, then you can transmit the same information with many fewer bits. Unless you're extremely lucky with your basis functions, it is not strictly speaking a lossless compression algorithm.

leijurv• 6 years ago

Yes, exactly! One of the winning entries at http://prize.hutter1.net/ (challenge to compress English Wikipedia) is/was a LSTM that predicted the next character then used arithmetic coding on the final bitstream (correct bit / incorrect bit).

Specifically, see https://arxiv.org/abs/1609.07959

0-_-04• 6 years ago

But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!)

A bit academic, but if you check out this paper from 2015 https://arxiv.org/pdf/1502.04623.pdf on a neural network trained on binarised MNIST (28x28 pixel images), Table 2 quotes entropy figures of 116.815 after converting to bits, which would be 28*28/116.815=6.71x compression (if used as a compressor)

dabei• 6 years ago

The tricky part is how to measure the perceived quality. With video there is VMAF but seems there has been no development on the image quality assessment.

andbberger• 6 years ago

In theory it is a great fit for ML. But not for the reasons you're suggesting.

The success of a compression scheme ultimately comes down to its ability to estimate p(x), for some sample x. And estimating p(x) is precisely the problem the field of ML addresses.

taneq• 6 years ago

Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data

I think that's the major risk for straightforward "AI" compression. You'll end up with some high level description of the scene and the decompressed image will then be an "artist's impression" which contains details that may not have been in the original image. You'll take a photo of a cheque, compress it, decompress it, and you'll get a picture of a cheque, but it might not be the same cheque.

bllguo• 6 years ago

the idea of learning as compression is actually an old one. Take a look at https://en.wikipedia.org/wiki/Minimum_description_length

leeoniya• 6 years ago

sure, what can go wrong?

https://www.zdnet.com/article/xerox-scanners-alter-numbers-i...

basilgohar• 6 years ago

That still baffles me to this day.

--Edit-- I think there are differences between the case of using this for compression and how Xerox implemented (which they claim was due to compression).

It can be argued that most advanced codecs of today already handle compression via very complex and advanced heuristics already comparable to what a lot of people would confuse for AI and/or ML today, so I think it's not as spooky as it may sound at first.

The Xerox implementation actually changed values on the page from one to another innocuously. I would hope that the AI being talked about is not "This 6 looks more like it should be an 8, so I'm going to put one there upon decode" and more like, "this block or unit of the image has these characteristics, and theses methods work very well to reproduce these characteristics when next to these other blocks..." etc.

But that's the best way I can phrase it.

fowl2• 6 years ago

thought experiment: if this was our only way of compressing we could lose the ability to represent things "too far" from whatever the AI understands.

m463• 6 years ago

How funny if it happened with video compression.

Movies could accidentally "transpose characters" too - meaning Mark Walhberg instead of Matt Damon

ksec3• 6 years ago

JPEG XL

The JPEG XL Image Coding System (ISO/IEC 18181) has produced an open source reference implementation available on the JPEG XL Gitlab repository. The software is available under Apache 2, which includes a royalty-free patent grant. Speed tests indicate the multithreaded encoder and decoder outperforms libjpeg-turbo. Independent subjective and objective evaluation experiments have indicated competitive performance with commonly used image coding solutions while offering new functionalities such as lossless transcoding from legacy JPEG format to JPEG XL. The standardization process has reached the Draft International Standard stage.

That to me is the biggest news. I remember reading a presentation where it had even better quality at medium bitrate than AVIF. Since JPEG does not belong to Google or Apple, hopefully it would be a format easier to swallow from both parties.

wmf• 6 years ago

Half the technology in JPEG XL is from Google but hopefully people won't get petty about that.

A bigger problem is that various companies just did a lot of work to adopt HEIF so they're not going to be eager to churn onto JPEG XL. Generally people wait 5-10 years before they're open to upgrading.

eln1• 6 years ago

Also got this ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems

ksec3• 6 years ago

Half the technology in JPEG XL is from Google

Yes but it is submitted as an open standard to an Organisation that doesn't run or belong to Google.

Is HEIF used anywhere other than on Apple devices?

jonsneyers4• 6 years ago

HEIF has a few problems though that make it not very suitable to replace JPEG as the default image format:

- Patent situation. If you're a camera or phone maker, it is quite a risk to use HEIF. Even if you pay all the major patent pools, you only cover part of the patent claims. You'll need dedicated hardware, so once you've produced the devices, the patent trolls have quite some power since you can't just switch to another codec with a software update.

- The codec is not optimized for high-quality still images – it's a video codec, and has its sweet spot in the lower bitrate regions. Photographers care about subtle texture, high bit depth, etc.

- There's no progressive decoding – again, it's a video codec, where progressive decode of a single frame is not useful, but for still images on the web, progressive can be a desirable option.

theandrewbailey• 6 years ago

Repo in question: https://gitlab.com/wg1/jpeg-xl

FWIW, AVIF is a project of the Alliance for Open Media, which all the major browser vendors are founding members.[0] AVIF should get browser support soon.

[0] https://aomedia.org/membership/members/

RcouF1uZ4gsC• 6 years ago

I think we already have the tools to do this, but there might be issues with fidelity.

First: Use AI to generate the text description of an image. http://homepages.inf.ed.ac.uk/keller/publications/jair16.pdf https://www.captionbot.ai/

Second: Use AI to generate an image from the text description. https://news.developer.nvidia.com/ai-model-can-generate-imag...

mehh• 6 years ago

What do you mean by us AI? Do you mean a Neural Network or something else, I find the use of the broad term AI a bit weird, there is lots of stuff that can fit under the AI category.

carapace• 6 years ago

rcarmo• 6 years ago

I was instantly reminded of this “pineapple”:

https://twitter.com/cassmarketos/status/1229473344480673792?...

Of course, an actual working compression algorithm is quite likely. I recall that recently someone was upscaling and colorizing ancient movie reels to modern standards, and there is plenty of low-hanging fruit in fine-tuning parameters to suit specific images, so I actually look forward to this.

I just hope they don’t name it JPEG2020, JPEG2000 was a bit of a dud.

BubRoss• 6 years ago

Jpeg2000 was a dud because of licensing and patents, not having a year in the name.

userbinator2• 6 years ago

...and performance (or lack thereof) --- AFAIK the baseline J2K was always implementable freely (similar to the situation with JPEG; arithmetic compression was specified but largely unimplemented due to patents, while baseline JPEG was supposed to be free), but the abysmal performance relative to JPEG for a minor increase in compression efficiency, along with the surprising amount of complexity in the spec, made it a poor choice.

There are lots of scanned ebooks on archive.org that use J2K compression, and the sluggishness when turning pages is extremely noticeable compared to others which use normal JPEG.

sgdread• 6 years ago

DeOldify [1] by Jason Antic got really good results with old photos and movies colorization.

[1] https://github.com/jantic/DeOldify

hnews_account_1• 6 years ago

I don't know how recolorization algos work, but a month or so ago, I saw this old 1900s image of a new York street that was colorized on reddit. Everyone on the thread was commenting about how there was so much violet hue in every single person's dress in the photo. Iirc that recolorization was done by a neural net too.

Then I realised what must've happened. There were tons of open air carts on the street selling eggplant. My guess is that the NN was either shown that or recognized that part of the image and colored almost everything else it saw using that as its base point. Not the best technique.

keenmaster• 6 years ago

Does anyone have an estimate of how small an AI-compressed image will be compared to an image compressed using traditional compression algorithms? On average of course, and controlling for image quality.

0-_-04• 6 years ago

As an example from 29 Feb 2016, for lossless compression you can go from 8 to 3 bits per color according to Table 5 here: https://arxiv.org/abs/1601.06759

rhacker• 6 years ago

I know this is more relevant for movies, but it would be rather cool to have a movie compressed for a year of computation, and deliverable in about 100MB, then let it decompress for a few hours on your computer back to its original 9GB size.

0-_-04• 6 years ago

How big is the market for that vs streaming 9 GB on demand?

SrslyJosh• 6 years ago

Fake news, copyright violations, media forensics, privacy and security are emerging challenges in digital media. JPEG has determined that blockchain and distributed ledger technologies (DLT) have great potential as a technology component to address these challenges in transparent and trustable media transactions.

You've gotta be kidding me.

jiggawatts2• 6 years ago

I've been in the C#, Java, and Rust world for so long that every time I come across C/C++ library source it cracks me up that an image compressor has to have functions to redefine basic concepts like memory allocation, assertions, and threading.

Reminds me of a "lightweight" zip compression library that I recently used that was literally 98% overhead and 2% actual compression code. It had its own definitions of everything: integers, integer properties, memory allocation, streams (as in sequences of memory blocks, not even real I/O!), error handling, progress notifications, etc... Pages and pages of conditional macros and compiler-specific hacks. Just to call a single a function that took an array and returned an array. Ridiculous.

https://gitlab.com/wg1/jpeg-xl/-/blob/master/include/jpegxl/...

https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/compil...

https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe...

https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/arch_s...

That last one is trying to work out the NUMA topology in a portable way... good luck with that! That's not something I expected from a JPEG library. Like memory management, they'd actually be better off requesting thread-management function pointers from the hosting application. Otherwise, this would not be a "good citizen" in constrained environments such as embedded in a web server or database engine.

Random examples of silliness from just this one file:

They ignore the first 2 cores on 3+ core systems: https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe...

They'll be limited max 64 threads (Half the threads on the larger single-socket AMD EPYC 2 CPUs!): https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe... because they're not using the Processor Group Aware API: https://docs.microsoft.com/en-us/windows/win32/procthread/pr...

Sure, 64-core monsters are still somewhat rare today, but 16-core/32-thread desktops CPUs are now affordable and the TSMC 5nm process will likely result in 96-128 core monstrosities this year. That's 384-512 threads for a dual-socket server or workstation! If you're making an image codec library "for the next decade", a little future-proofing might go a long way. Look at it this way: Image decoding throughput is one of the main performance issues I have right now with Lightroom and I work with "only" 36 Mpx images. There are 80 Mpx cameras just around the corner...

For example, there was a blog article a couple of years ago basically saying that the major Amdahl's law limitation to decoding formats like JPEG is the Huffman bitstream decoding, which is Step #1 in the process. The author pointed out that simply restarting the Huffman bitstream a few times for a typical PNG or JPEG file would increase the size less than 1%, but would allow truly parallel decoding for every step.

For all the apparent effort to make JPEG-XL parallelised by default, it seems odd that there doesn't appear to be any mention of parallelising the ANS bitstream decoding...

janwas3• 6 years ago

JXL dev here. Thanks for raising these points. Indeed regrettable we have to have a bunch of boilerplate.

FYI we actually do support hooks for malloc (JpegxlMemoryManagerStruct) and threading (JpegxlParallelRunner) precisely in order to be a good citizen :)

The topology is more about #cores/sockets, used by the benchmark_xl app to decide how many threads to spawn.

I agree adding Processor Groups makes sense, but Windows hasn't been our focus yet.

FYI it is indeed critical to parallelize ANS streams. Each 256x256 "group" can be decoded in parallel.

jiggawatts2• 6 years ago

Indeed regrettable we have to have a bunch of boilerplate.

Just to clarify: I'm not criticising the JPEG XL code, it seems to be quite clean at first glance! I was merely making a general observation that would apply to practically all portable C-interface libraries, such as ZStandard, libpng, etc...

PS: I wonder if it would be possible to make a "portable library header" that all such similar libraries could share to avoid the rework on both ends (library authors and library consumers)...

Windows hasn't been our focus yet.

That's a real worry. Not just because of the word "Windows", but what else it implies.

To provide some constructive feedback: JPEG XL will be yet another failure thrown on the trash heap of similarly well-intentioned formats. It'll die along with JPEG 2000, JPEG EXR, and all of the others. Don't be so arrogant to think that somehow your format will be special and adopted by the masses. The others were just as technically advanced for their time and they all failed.

Please, please, please learn this lesson, because this history of failure saddens me. The lack of a capable image formats compatible with all client platforms is what stops me as a photographer from sharing my images in full quality. In 2020, still, it is effectively impossible to share images in any other format than 8-bit sRGB JPEG across platforms. You can't email a HDR, 10-bit, or wide gamut picture to anyone and expect them to be able to open it and have it look even vaguely right. You can't post such an image on any website you don't control yourself. Even if you control the website, it will look wrong for 99.9% of users OR they'll just see the 8-bit sRGB version anyway, even if they have 10-bit HDR display.

It is literally only the walled garden of the Apple ecosystem that gets this right. Microsoft and Google most certainly doesn't. As an iPhone user, I can share a wide-gamut HEIF picture and 100% of the time it will look correct for other iPhone users. However, Apple "cheated" a little bit. When you send one of their HEIFs to any non-Apple-controlled app, it's silently converted to... 8-bit sRGB JPEG. It's just sad.

What everyone gets wrong with new image formats is this: They think that the problem that needs solving is the encoder/decoder. That's step one of many problems that users need solved. Image format developers assume their format will be "adopted", but there's no network effect. Why would anyone use a format they can't email, can't open on Windows 7, and can't edit in Photoshop?

If you stop with the JPEG XL library, with Windows compilation an afterthought, I absolutely guarantee that you will fail to gain traction and the entire effort will be essentially wasted.

To get actual adoption, you would need to:

* Most importantly, develop an image decoder plugin for Windows.

* Help Firefox, Safari, and other less common browsers merge support.

* Ensure that Chromium gets support, but I assume that's a given.

* Send pull requests to add JPEG XL support for all major open source libraries that perform image processing, such as WordPress plugins and server-side image processing utilities for websites.

* Develop a plugin for all major image editing tools such as Photoshop.

* Work with Adobe on Lightroom export support.

* Contact Sony, Nikon, and Canon and help them with in-body encoding support.

But even that's not enough. If you want 10-bit and the wide gamut (let alone HDR), you have to take steps to force support, or it won't magically materialise. You'll end up with two JPEG XL formats: The 8-bit SDR sRGB variety with 99.9999% adoption and the "fancy" JPEG XL that nobody uses and looks wrong if they try.

IMHO the only way to do this is to forcibly break applications that aren't processing the ICC profiles properly. Much like Google's Chrome experiment with randomising some TLS header fields to occasionally break middle boxes written by idiots that think version numbers are checked with equality, not inequality operators.

You'd have to do something drastic, like leaving the image planes unnamed and randomised so that literally the only way to decode the image correctly is via a compliant colour space transformation.

It sounds drastic, but other people have tried the easy way and failed. As I said: It's 2020!!! Email me a non-sRGB image that I can view in my mail client...

t0rakka• 6 years ago

Guilty as charged.. the 1% was a napkin estimate.

The True Cost in this specific case is (0..7 bits to fill a byte) + 2 bytes for the RST marker for each interval. 320x240 image and RST interval for each 240/8 block means 30 RST's so something close to 75 bytes "wasted", or, other way to look at it is ~3 bits per row.

The reason is that for the JPEG the Huffman tree is fixed and is transmitted with the image data so restarting is cheap; it does not start from scratch and build a new tree each time; each decoded symbol is also independent so you can stop and restart at any time you want. I don't have any ANS experience at all so won't say anything about that.. but it would be nice if some sort of way would be possible to isolate the ranges.

Good observations, btw. :)

JPEG Committee releases a call for evidence for image compression based on AI

Related Stories