HNGlance

Top Best New Show

Bitfrost – LLM gateway 90x faster than Litellm at p99

https://github.com/maximhq/bifrost

by havercosine • 12 months ago

8 1 12 months ago

GI

Related Stories

Loading related stories...

Comments

1

havercosine• 12 months ago

Bifrost is the fastest LLM gateway on the market. Built in Go with careful garbage collection, it adds just about 11 microseconds of overhead at 5,000 requests per second (with 4,100 RPS throughput) on a t3.xlarge instance.

The benchmarks are here: https://github.com/maximhq/bifrost/blob/main/docs/benchmarks...

Some features: • Built-in governance and routing rules • Supports over 1,000 models from different providers • MCP gateway included (HTTP, SSE, and console transport) • Out-of-the-box observability and OTel-compatible metrics

Source preview

github.com

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms