Show HN: I built an analytics tool to track cost and usage of any LLM provider

by lexokoh • 2 years ago

2 5 2 years ago

verdverm3• 2 years ago

One issue you will encounter is that many of the LLM calls are hidden and nested below abstractions in tools like LlamaIndex. You will need to integrate with these libraries or users will not be able to get the granularity needed.

See LangFuse as an example. It is what I am using

topicseed• 2 years ago

Quick question, what do you use such an observability platform for? Can't be the price as you get it straight from Openai's dev dashboard. So what else do you use Languse for?

verdverm3• 2 years ago

I don't use OpenAI because it is an unreliable service, they don't know how to run production.

LangFuse supports all providers, allows for user and session level metrics, and is self hostable. Custom spans are really helpful as is the ability to take feedback on responses and build data sets from them.

How do you include your RAG queries and non-ai ops into your overall chat observability?

I'm somewhat of a mind to switch to an Open Telemetry based system, since that is the standard now

Show HN: I built an analytics tool to track cost and usage of any LLM provider

Related Stories