Show HN: VLLM with JSON Guided Generation

https://outlines-dev.github.io/outlines/reference/vllm/
by btwillard • 3 years ago
11 3 3 years ago

Our project, Outlines, now offers guided/constrained generation (e.g. according to a JSON schema) via the VLLM library.

My colleague, Rémi, created some patches that allow one to pass vLLM a JSON schema along with the prompt, which dramatically simplifies deployment of JSON-guided generation. He also added a new `serve` interface that puts it all together and makes serving such models a 2-3 line process.

Check it out and tell us what you think!

Related Stories

Loading related stories...

Source preview

outlines-dev.github.io