A Python Library to 6-7x the inference speed of your HF models

by MDK8888 • 2 years ago

75 15 2 years ago

MDK88883• 2 years ago

A few months back, the PyTorch team released GPT, Fast (https://news.ycombinator.com/item?id=38477197), a collection of techniques to 10x the inference speed of Llama-2-7b.

This library generalizes those techniques (quantization, torch.compile, speculative decoding) to all Huggingface models, achieving an inference speed speedup of 6-7x.

As more optimization algorithms are discovered, I will post updates to the repo to HN.

theboat• 2 years ago

does this work for diffusion models, e.g. stable diffusion xl?

jurgenaut23• 2 years ago

A quick look at the codebase seems to indicate that it doesn't. This seems very targeted at LLMs.

ipsum2• 2 years ago

Pretty confusing that both the pytorch repo and yours is named gpt fast.

mkesper• 2 years ago

Please add (something like) that to the Readme.

codethief2• 2 years ago

+ setup instructions please :)

MDK88883• 2 years ago

This has been added to the Readme under the 'Background' section.

bradneuberg• 2 years ago

Has anyone tried this on OpenCLIP based models?

behnamoh• 2 years ago

does anyone use transformers in production? last I checked people either use exllama2, vLLM, or llama.cpp. AFAIK llama.cpp already does speculative decoding (even batched grammar based spec. decoding!).

daemonologist• 2 years ago

We use transformers in production for T5 models, which aren't supported by the usual suspects. I did a bit of poking around and might try out TensorRT or CTranslate2 at some point; for now the single GPU we have dedicated to the project has been able to keep up with incoming data, but I'll probably have to address it in the next couple of months.

If anyone has suggestions for other T5 inference options I would welcome them.

jph00• 2 years ago

vLLM uses transformers.

iAkashPaul• 2 years ago

Indirectly via TGI

dmezzetti• 2 years ago

It's important to note that Transformers can be used for a lot more than just LLMs. There are plenty of models that can be fine-tuned to perform a single task better than LLM prompting.

A Python Library to 6-7x the inference speed of your HF models

Related Stories