detente18

1. I think rust is the better long term solution over go. Especially for a gateway. You can see other proxy's (e.g. cloudflare's pingora - https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...) doing the same thing.

Rust/Python hybrids are also quite well established, so it allows us to work off of a stable base - while ensuring we can deliver - low memory utilization - low request latency overhead

which is the primary goal here (be fast, lightweight and cheap to deploy).

2. LiteLLM has the most mature, and broadest range of unified api's x providers. This means you do not need to give developers raw LLM API keys, ever.

We see devs using/building agents that consume a lot of different API's - responses api, realtime, chat completions, messages - and no matter what they use we want them to be able to switch across providers without if/else statements in their code.

I can't comment on others, but that's our goal and what we work on doing everyday. So I would trust that we do it well.

Beyond that, we're also growing to become the single point of access for all AI resources. This makes it a lot easier when building agents, because you can give an agent 1 key, and it will have access to LLM's + MCP's (and in the future other resources like skills, api credentials, sandbox api's, etc.).

this is pretty interesting. nice work!

Update 2 (03/25/2026):

- We will be holding a townhall on Friday to review the incident and share next steps (https://lnkd.in/gsbTdCe7)

- We can confirm a bad version of Trivy security scanner ran in our CI/CD pipeline, which would have led to the supply chain attack

- We have paused new releases until we've completed securing our codebase and release pipeline to ensure safe releases for users

- We've added additional github/gitlab ci scripts for checking if you're impacted: https://lnkd.in/gGicMkby

We hope to share a full RCA in the coming days. Until then, if there's anything we can do to help your team - please let me know. You can email me (krrish@berri.ai), or join the discussion on github (https://lnkd.in/g9TuuQ2H).

I deleted it, to be safe.

It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy.

We have deleted all our pypi publishing tokens.

Our accounts had 2fa, so it's a bad token here.

We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).

Update:

- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted

We are still scanning our project to see if there's any more gaps.

If you're a security expert and want to help, email me - krrish@berri.ai

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far:

1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09

2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt

3. The package is in quarantine on pypi - this blocks all downloads.

We are investigating the issue, and seeing how we can harden things. I'm sorry for this.

- Krrish

Should we expect the current API (https://docs.goodfire.ai/introduction) to be good for prod or is it for testing only?

I see that the sampling API is OpenAI-compatible (nice!). Considering if we can add a native integration for this to LiteLLM with a provider specific route - `goodfire/`. Would let people test this in projects like aider and dspy.

```python

from litellm import completion

import os

os.environ["GOODFIRE_API_KEY"] = "your-api-key"

response = completion( model="goodfire/meta-llama/Llama-3.3-70B-Instruct", messages=[{ "content": "Hello, how are you?","role": "user"}] )

```

I wish you could filter by startup name. Would be curious to know where we (litellm) ended up

Saw you just support claude for now. If your backend code is in Python, happy to add support for other models (OpenAI/Gemini/etc.) via a consistent API interface like LiteLLM - https://docs.litellm.ai/docs/#litellm-python-sdk

Was looking at adding this to litellm - https://docs.litellm.ai/docs/providers

but the website crashed on me while reviewing the docs. Is this stable yet?

Your changelog is neat - is this custom built or via some embeddable tool?

Interesting - how do you use LiteLLM + OpenRouter here?

I would assume they're alternatives

Just saw `from litellm import completion` code

Thank you for using litellm + all your issues on our repo! @Ey7NFZ3P0nzAe

- litellm maintainer

It's interesting to see how short-lived some of these were (e.g. HumanEval).

Interesting — does your backend server use Python? I couldn't find much about it on your site.

It would be great to see this tested with more commercial LLMs (O1 / Amazon Nova, / Llama 3.2 / etc.). If you're open to it, I’d be happy to contribute support for these models via LiteLLM - https://docs.litellm.ai/docs/providers

+1 on the github issues. It's very useful to have this on the litellm docs

nice! thanks for using litellm! - maintainer :)

Hey, are you just looking for litellm - https://github.com/BerriAI/litellm

context - i'm the repo maintainer

LiteLLM proxy (100+ LLMs in OpenAI format) is exactly compatible with the OpenAI endpoint. Here's how to call it with the openai sdk:

``` import openai

client = openai.OpenAI( api_key="anything", # proxy key - if set base_url="http://0.0.0.0:8000" # proxy url )

# request sent to model set on litellm proxy,

response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ])

print(response)

``` Docs - https://docs.litellm.ai/docs/proxy/quick_start

Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd...

Anything I'm doing incorrectly?

nice - is this using LiteLLM for making the completion calls?

*Update*: for those asking how to run self-hosted llama2, we just added the ollama integration.

Here's the tutorial - https://github.com/BerriAI/litellm/blob/main/cookbook/liteLL...

Any reason you're doing that vs. using Lambda Labs / Replicate / together.ai / Banana.dev, etc.

There's a lot of good model deployment platforms that would make it easy to call your model behind a hosted endpoint

-- If you do want to self-host - there's some great libraries like https://github.com/lm-sys/FastChat and https://github.com/ggerganov/llama.cpp that might be helpful

If none of these really solve your issue - feel free to email me and I'm happy to help you figure something out - krrish@berri.ai

You're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response,

why would you cache the openai call instead of the endpoint that's receiving the user call?

If you'd like to add any of these ideas as notebooks to our cookbook - https://github.com/BerriAI/litellm/tree/main/cookbook, we'd love the contribution!

Those are really cool use-cases. I wrote a bit about our initial motivation for LiteLLM here - https://hackernoon.com/litellm-call-every-llm-api-like-its-o....

tldr; reliable model switching involved multiple 100-line if/else statements, which made our code messy, and debugging in prod pretty hard.

Hey bredren - our supported list (if that's helpful) is here https://litellm.readthedocs.io/en/latest/supported/

We're adding new integrations every day, so if there's any specific one you'd like to add feel free to let us know (discord/ticket/email/etc.) - here's my email: krrish@berri.ai

Hi @d4rkp4ttern - yes it's standardized to the openai format - https://litellm.readthedocs.io/en/latest/stream/

How do you plan on deploying llama 2? Is that via ollama/fastchat/etc.? We're actively building out integrations to different providers, so if you have any preferred tooling - let me know!