Brotli-G: A GPU compression/decompression standard for digital assets

https://gpuopen.com/brotli-g-sdk-announce/

by josephscott • 4 years ago

88 22 4 years ago

There's an interesting side point here: There exist (and there will be more of these in the future) data compression algorithms which are in general, too slow for specific software use-cases. (related: https://superuser.com/questions/263335/compression-software-...).

The thing is -- they typically run too slow for their intended applications on current, consumer-grade CPU's...

But, could some of them be optimized to take advantage of GPUs (as Brotli is here) -- and would that then increase their performance to a level such that applications which previously could not use them because the algorithm previously took so long -- can now make use of them IF the software end-user has the proper GPU?

I think there's a huge amount of possibilities here...

Especially when you get to compression algorithms that include somewhat esoteric stuff, like Fourier Transforms, Wavelet Transforms -- and other weird and esoteric math algorithms both known and yet-to-be-discovered...

In other words, we've went far beyond Huffman and Lempel-Ziv for compression when we're in this territory...

(In fact, there should be a field of study... the confluence/intersection of all GPUs and all known compression algorithms... yes, I know... something like that probably already exists(!)... but I'm just thinking aloud here! <g>)

In conclusion, I think there's a huge amount of interesting future possibilities in this area...

eklitzke• 4 years ago

All the things you're talking about are widely used in audio and video codecs, which typically do have hardware acceleration support.

peter_d_sherman5• 4 years ago

True!

But, there's a huge difference in say, the hardware available to a well-funded Silicon Valley company in the richest place in the richest country in the world, and hardware that a poor person in Sub-Saharan Africa or Rural India might be able to scrap together (if they can even acquire or afford electricity and an Internet connection...)

Around the world, globally -- some people might not have access to hardware which is accelerated enough (much less a GPU!) -- to run the latest and greatest algorithms or the applications that depend on those algorithms...

Conversely, in places like Silicon Valley -- you might have all sorts of new-fangled software running on (and requiring!) super high-end hardware -- that just isn't available or affordable for even the middle class in America!

The democratization of information (on the one hand!) requires that computers, no matter how old, no matter how incapable or unaccelerated (i.e., no GPU) -- run the 'best in class' algorithm -- for whatever their hardware is capable of.

But progressivism, at least in the technological sphere, that is, advancing technology, making progress, "pushing the envelope", "working at the cutting edge", progressivism -- requires the opposite! It requires that we run the absolute newest, latest, most experimental, most "bleeding edge" algorithms on the newest, latest, most experimental hardware!

What emerges from these two countervailing goals -- is a matrix.

A matrix of "hardware capability" X and "best in class algorithm for" Y.

For any given algorithm/application.

If we were to go through the process (er, pain, er process! <g>) to create such a matrix, such a table... then we'd see (much like with incomplete historical snapshots of the Periodic Table of elements) that there are missing / "not well-defined" entries!

We'd also see that there are yet additional combinations, er, elements, er, combinations <g> -- to be discovered in the future!

So, there you have it... (technological!) progressivism vs. (technological) democratization of information (AKA, "equity" as applied to information).

If both countervailing goals are to be pursued (and I claim both are equally virtuous!) -- then what we need is matrices of what is currently "best in class" for any given hardware/algorithmic combination!

And... we might have a whole lot of great hardware and algorithms -- but I strongly believe there is still a lot to be discovered (and improved upon!)...

But yes, in general, I agree that many audio and video codecs do have hardware acceleration support!

the-alchemist2• 4 years ago

Actually, there is a huge amount of work in this area. Take a look at https://encode.su/forums/2-Data-Compression

MuffinFlavored• 4 years ago

How do you get around the performance hit of having to buffer from disk/network -> CPU -> RAM -> GPU and back or whatever?

peter_d_sherman5• 4 years ago

I didn't know about those forums! Awesome link!

miohtama2• 4 years ago

See also Blosc, with their tag line ”faster than memcpy”

https://www.blosc.org/pages/blosc-in-depth/

Blosch it optimised to have the work buffers that fit into L1 cache, so it can outperform memcpy for certain workflows, e.g. numeric arrays. Because the bottleneck is not CPU, but RAM and slower caches.

peter_d_sherman5• 4 years ago

Excellent link!

I had not heard of blosc before -- but that looks pretty cool!

darrinm• 4 years ago

Benchmarks?

corysama• 4 years ago

There is a new interest in GPU-side general-purpose decompression due to Microsoft pushing DirectStorage.

https://devblogs.microsoft.com/directx/directstorage-1-1-com...

nevi-me• 4 years ago

To digress and talk about DirectStorage, reading their documentation and announcements still leaves me with an unanswered question. Maybe someone knows the answer.

Does DirectStorage only work for games, or can one use it for compute workloads?

Context: I've been learning some basic GPU programming (via rust-gpu though, not CUDA), and one of the things that sound easy to implement are offloading compute kernels to the GPU (e.g. arrow-rs).

Being able to load datasets via DirectStorage could be great, but as I'm still really learning the basics, I can't figure out whether I could leverage this for my work/learning.

mmozeiko• 4 years ago

You can use it for any workload that can use D3D12 buffers or textures as input. All the API does for you is transfer data from disk to ID3D12Resource object. After that it is up to you to do whatever you want - use it for fragment shader or compute shader input, etc.. If you use other api's like cuda or vulkan, then you'll need to use interop to create their resources using D3D12 backed resource (or do a copy, whatever is possible there).

pstuart2• 4 years ago

It requires the source asset to be compressed using brotli-g as well, so it's dependent on the hosting servers to be of any value.

the-alchemist2• 4 years ago

I don't think so, if I read the article correctly.

Existing optimized Brotli decompression functions (CPU implementations) should be able to decompress the Brotli-G bitstream, while more optimal data-parallel implementations on hosts or accelerators can further improve performance.

pstuart2• 4 years ago

Pre-coffee non-clarity on my part. I was referring to this part:

One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations

efortis• 4 years ago

Static content can be uploaded already compressed. For example, using `brotli_static` in Nginx you can upload `foo.js` and `foo.js.br`

telendram• 4 years ago

Interesting, nVidia selected to go with zstd instead: https://developer.nvidia.com/nvcomp

ttoinou• 4 years ago

Anyone knows if there is a way to tune this algorithm to be lossy ?

miohtama2• 4 years ago

Brotli is based on Huffman and LZ99 generic data compression algorithms.

Any lossy compression step needs to come before Brotli.

Brotli alone makes little sense for image compression, but is ideal for binary files.

rektide• 4 years ago

Im interested to see what image format specific compressors go gpu. JpegXL, AVIF, WebP... who wants to show up & throw down? Or even just fastpng?

Meanwhile we dont really hear or regard many of the gpu-oriented compression techs. TIL Basis/KTX2 is itself zstd compressed (formerly LZ apparently?). https://github.com/BinomialLLC/basis_universal

Brotli-G: A GPU compression/decompression standard for digital assets

Related Stories