AI Poised to Transform Video Compression Landscape

https://spectrum.ieee.org/ai-video-codecs-waveone

by mfiguiere • 3 years ago

51 46 3 years ago

spectrum.ieee.org

At a very (very!) abstract level AI is compression and compression is AI.

There's been rumblings about how Google's AI edge would lead to jumps in video and image coding but nothing big seems to have happened yet, though lots of micro-optimisations appear to be ML based.

Some suggested that the AV1 throw a bunch of tools at it spec design was built for AI to use.

Possibly VMAF is the biggest ML impact on codecs so far.

bick_nyers• 3 years ago

A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs.

Shot or scene based detection can be used to place your I, P, and B frames in better locations for example.

For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.

astrange• 3 years ago

I-frame placement isn't hard; it's very obvious when a scene changes or when an I-frame would be profitable. You generally need to have them far more often than you'd like anyway, as otherwise you can't seek in videos.

The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.

echelon2• 3 years ago

At a very (very!) abstract level AI is compression and compression is AI.

I think it's not so abstract at all. The relationship feels fundamental.

Encoder, decoder, codec. Compression, entropy, lossiness, ...

CuriouslyC2• 3 years ago

They share find low bit count representations of information. The difference is everything else.

retrac• 3 years ago

nothing big seems to have happened yet

There's been some cool stuff in the speech area: https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-...

Also from Microsoft: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...

lrog• 3 years ago

At a very (very!) abstract level AI is compression and compression is AI.

There's actually a pretty nice read from one of the key figures in data compression field - Matt Mahoney: "Text Compression as a Test for Artificial Intelligence" [1]. Some interesting reads are also on his personal website, though a bit outdated now: https://mattmahoney.net/dc/rationale.html

[1] https://static.aminer.org/pdf/PDF/000/014/009/text_compressi...

3rd3• 3 years ago

AI = compression + goal. Though strictly speaking I think compression is only required in partially-observable and/or stochastic worlds where the information content of the world (far) exceeds the actor's model capacity.

esafak• 3 years ago

It's so amusing how AI became AGI and ML became AI. It used to be a marketing thing and now it's on the IEEE Web site.

TapWaterBandit• 3 years ago

Are LLMs ML except called "AI"?

xipix3• 3 years ago

The thing about video compression is that it's already very compute intensive. Most devices have dedicated hardware cores for encoding and decoding in order to save power. They work exclusively with integer values, chiefly unsigned 8 and 10 bit values with sequential and block memory transactions.

Humans not only design for compression quality, we also really focus on cost. And we design direct to what is cheap in hardware: integer adds, shifts, binary logic.

I do believe a different paradigm of ML is necessary for this particular application. The neural primitives used need to be more closely related to transistors than to matrix multiplications.

echelon2• 3 years ago

That's said as if matrix math isn't going to become the most important silicon-enabled process in the world ever.

The gravitas will shift to enabling matrix math rather than algorithms being designed for special hardware.

Matrices are the new paradigm.

xipix3• 3 years ago

Matrix math units are a smart use of silicon for many ML applications, true. But Von Neumann compute isn't going away. Neither are fixed function modules for shading, ray tracing and massively parallel GPGPU type applications. Also not video coding.

My point was that if training could emit an efficient video codec as a network of logic gates, rather than as a dense array of neural network weights, it might just produce something practical for video playback on mobile devices and video processing at hyperscale.

bradleyjg2• 3 years ago

Most videos are only going to be viewed a tiny number of times, often just on the devices they were taken on. It’s probably not worth spending too much effort on compressing these.

A small number of videos are going to be viewed millions of times. For these few videos it’s worth the compute to do the best possible compression.

There’s no reason the same video formats and decoding hardware can’t be used for both. The high compute version can use an ML model to figure out how to optimize the bandwidth budget across every scene while the cheap version uses some easy heuristics.

2h• 3 years ago

best possible compression

This is a common incorrect assumption. You have to balance ratio with speed, otherwise you end up with something that spends more time decompressing than actually downloading, as is literally the case today with XZ. this is why alternatives like Zstandard are getting popular. 9 times faster decompressing for 10% worse ratio.

kristopolous• 3 years ago

I just realized there's inevitably going to be dedicated hardware to help speed things up with ai related tasks. The GPU helps significantly but I'd assume something even more suitable could be made.

louismerlin• 3 years ago

There has been an Apple Neural Engine (ANE) on iPhone chips since 2017.

https://machinelearning.apple.com/research/neural-engine-tra...

xipix3• 3 years ago

yup and, as ever, Apple definitely weren't the first with this.

ThrowawayTestr• 3 years ago

https://www.nvidia.com/en-us/data-center/tensor-cores/

bob1029• 3 years ago

Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in normal conditions. Reviews from third parties show that when benchmarked with conventional distortion metrics as well as human opinion scores, conventional video encoders still outperform neural-network compression, especially when conventional encoders are enhanced with AI tools.

I don't know how much better AI can do in practice.

If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality.

Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point...

Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size.

Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.

All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.

andyjohnson0• 3 years ago

DCT = Discrete Cosine Transform

https://en.wikipedia.org/wiki/Discrete_cosine_transform

ZeroGravitas2• 3 years ago

Kind of tangential but: JPEG has always had arithmetic encoding in the spec. No one implemented it due to patents (even after the patents expired about a decade ago).

Roughly 10% improvement I believe.

pixelchai• 3 years ago

How could an AI system improve on techniques that optimize for the Human Visual System

One way it may believed that AI may improve upon the traditional methods you mention is the assumption that AI can learn a tighter optimisation for the human system (than humans can handcraft)

CorrectHorseBat2• 3 years ago

Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.

Exactly that. AI could recognize what is important in a frame and use less compression there, while using pattern generation for high entropy parts which are not important.

whitemary• 3 years ago

We truly don't need AI to tell us what is important or not. We already know this, objectively. It's mathematically defined as the visible spectrum of the a available information, adjusted with relative high/low thresholds for compression level.

dylan604• 3 years ago

I don't know how much better AI can do in practice.

Coming soon, you show AI the script and the poster of the movie you want to compress, and then it just makes shit up on the fly when you hit play. There is no compression! The movie will be unique to each viewer as the algo will take a seed from each playback device and create the music, images, dialog all on the fly.

A 2 hour movie in <1MB of storage

MontyCarloHall• 3 years ago

I believe Zoom already utilizes deep learning-based compression to some degree. I have seen some compression artifacts on video calls that look exactly like CNN hallucinations (e.g. https://thenewstack.io/deep-learning-neural-networks-google-...)

speed_spread• 3 years ago

At some point it'll become like the "Be kind, rewind" movie where an entire movie can be compressed to a text description of each scene then and entirely regenerated on demand by AI.

criddell2• 3 years ago

I remember reading about fractal-based image compression in the 1990’s. IIRC, there was some product out there that used it (I vaguely remember product ads with a paint can in them) but I don’t think it ever took off.

I wonder if coupling an AI to help find self-similar portions of the image could revive this idea?

tudorw2• 3 years ago

Fractal and Wavelet Image Compression Techniques Book by Stephen T. Welstead

not sure about actual implementations

mistercow• 3 years ago

Are you thinking of Genuine Fractals?

I got super into the concept of fractal compression when I was young, and then became pretty disillusioned at how primitive the techniques turned out to be. IIRC it wasn’t much more than “divide the image into, say 16x16 blocks, then search for 8x8 blocks which are similar to a 50% scaled version of a large block, and store the delta”. Young me was hoping for some cool math, but it was more like brute force motion compensation within a single image.

criddell2• 3 years ago

Somehow I got a copy of James Gleick's book Chaos and it had a big impact on me. I was kind of obsessed with chaos and fractals for a while. Right around that time a whole bunch of similar pop sci books came out and I read a bunch of them. Steven Levy's Artificial Life, a Mondo 2000 collection, John Koza's Genetic Programming, and Cliff Stoll's The Cuckoo's Egg all stand out in my memory.

partitioned• 3 years ago

I’m working on research for a journal article on this right now. Basically, get a bunch of videos you want compressed, use an auto encoder and just store the very small latent space variables instead of the video. Keep the decoder sitting around that was trained on those videos, and badaboom you can take a 512x512 movie and store it in 16x16x3 or another similarly sized latent space. However, this only works for fairly consistent video. Right now it works best for constantly monitoring video systems of physical phenomena. I.e. if you have multiple cameras pointed at the ocean to watch waves, you’re going to end up training a latent space that describes the wave physics. These small latent spaces are also the key to removing the black box. You can grab the latent spaces and connect the fields to physical quantities. For example, the wave reconstruction latent space has examples of velocity or wave height (variables needed to mathematically construct a wave field). Cool stuff!

kevmo314• 3 years ago

A nice extension of this is it provides a natural way to do generative video as well by finding an alternate encoder for the embeddings.

andai• 3 years ago

How do you find that?

tudorw2• 3 years ago

nice, if I understand, it would be great at ignoring the waves and spotting something less usual, a boat, or whale :)

IshKebab• 3 years ago

The trouble is at some point it isn't really video compression, it's video synthesis from wave velocity data.

I mean, it's fine for stuff that doesn't really matter like video conferencing or TV/films, but I think you'd be mad to use this for CCTV or science or whatever.

fnordpiglet• 3 years ago

It feels like if you’re willing to make computation >> storage, which is likely the bad optimization right now, you can use generative AI trained on the original video and key frames chosen to optimize size : fidelity and let the AI fill in the frames dynamically.

initramfs2• 3 years ago

I'm going to draw a line with AI at what I call the "eyes on forehead" problem. With Ai Image rendering, there is the emergent trend of AI images being like a horror film, with faceless and extra-features on faces.

When AI is using a LLM dataset to fill in the gaps, it becomes a problem, because it's guessing what should be there when the "natural" noise would be better depicting nothing. Predictive vectors such as SVG use smooth images. i think the rasterization of a dataset turns single pixels into quasi-intelligible datapoints, but when treated indendently by a dumb AI, ends up rendering eye balls on a forehead. This is your problem, not mine. Please fix it.

shpx• 3 years ago

The other case I would worry about is over-compressing text changing the letters, but it's an old and known problem https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...

initramfs2• 3 years ago

That or you're in a lucid dream. I once read the way to test if you're in a lucid dream, is to look around for an alarm clock. If it tells the wrong time, it's probably a lucid dream. This conversation can't be a lucid dream unless it's the script of Waking Life. ;)

That Xerox bug is interesting- I once used a Parallel port cable to print to an Apple Laser Printer from the 90s, until Windows 7 no longer supported the driver. I don't remember how many megabytes were transferred for large print jobs, but it seemed to take a while. I thought- perhaps it would be better if it sent the data in low-byte, text only code, so that the printer-side can reconstruct text using hard-coded translation the as opposed rendering all the bitmap- and while some data was xferred as a JPEG, perhaps it was compressed, and thus altered.

ksec• 3 years ago

I have yet to see a true AI ( or ML ) based Video Codec, or some potential Generative Video Codec like what Google did with Audio For Voice. And if it cant be done for Audio I doubt it will work on Video given the complexity. The current working prototype of H.267 ECM 8.0 aims to squeeze another 40% to 50% reduction in bitrate from H.266 / VVC Codec. There will likely be some ML usage on the Encoder side, but in terms of codec the exploration path hasn't provided much evidence how it will fundamentally change the codec efficiency.

AI Poised to Transform Video Compression Landscape

Related Stories