vmihailenco

We’ve just released Uptrace v2.0, a major re-architecture of our open-source observability platform (traces, logs, metrics).

This release focuses on performance and scalability:

- 5–10x faster queries using ClickHouse’s new JSON column type (nested fields can be queried directly with dot-notation, no ETL or flattening required)

- Multi-project support – manage projects, users, and orgs via UI or YAML seed files

- Real-time data transformations (rename, filter, enrich spans/logs, sample dynamically) using an [Expr-based](https://github.com/expr-lang/expr) scripting system

- Redesigned dashboards and query builder for faster incident response and exploration

- Handles massive traces (10k–1M spans) with tunable resource limits

It’s still open-source and free to self-host. We’ve published migration guides, Ansible and Kubernetes deploy instructions, and full [release details](https://uptrace.dev/blog/uptrace-v20.html).

Would love feedback from the HN crowd — especially from folks scaling ClickHouse or building custom observability pipelines.

Links:

- Blog & details: https://uptrace.dev/blog/uptrace-v20.html

- Install guide: https://uptrace.dev/get/hosted/install

- GitHub: https://github.com/uptrace/uptrace

Uptrace is an open-source APM tool that supports distributed tracing, metrics, and logs. You can use it to monitor applications and set up alerts to receive notifications via email, Slack, Telegram, and more.

In v1.4 release Uptrace got built-in alerting capabilities that allow to monitor metrics, manage alerts, and receive notifications without using AlertManager. But AlertManager is still supported.

While OpenTelemetry remains to be the main source of telemetry data, Uptrace also provides integrations for Prometheus metrics, Vector logs, FluentBit, Sentry, and CloudWatch.

You can quickly try Uptrace by:

- Running a Docker example: https://github.com/uptrace/uptrace/tree/master/example/docke...

- Checking Uptrace Cloud demo: https://app.uptrace.dev/play

If you have any feedback or questions, let me know in the comments!

Echarts which we use supports exporting charts as images so it probably can be added relatively easy. Embedding is another possible option.

I use memory profiling with Go and it is indeed very useful. I think that whatever Go already provides is enough, but I guess Uptrace could try to automate some things and/or provide some fancy UI.

But I find CPU profiling a lot less useful, because production profiles tend to be too broad and it is hard to make sense from them, for example, Uptrace profile mostly consists of memory allocations and network calls.

So I would not say that CPU profiling is superior/better/can replace tracing.

Thanks for the feedback. Any projects that you could recommend that do it right?

When compared with ElasticSearch, ClickHouse can handle the same amount data using 10x less resources and that is not an exaggeration. It is even worse with MariaDB/MySQL/PostgreSQL.

I guess ElasticSearch is still relevant when it comes to searching text, but ClickHouse is much faster when it comes to filtering and analyzing the data.

Give ClickHouse a try and you won't be disappointed.

https://benchmark.clickhouse.com/

My understanding is that APM became or always was a marketing term which is used rather freely. For that reason I try to avoid it, but search engines love it and I don't know a better alternative.

Whether I can find out why that component is slow might not be as easy (not sure what granularity tracing happens inside a component).

It is true that you can't always guess what operation going to be slow and instrument it, but it is almost always a network or a database call. There is still no way to tell *why* it is slow, but the more data you have the more hints you get.

What's the usual overhead of a single tracing

Depending on what is your base comparison point the answer can be very different.

Usually, you trace or instrument network/filesystem/database calls and in those cases the overhead is negligible (few percents at most).

But does that mean taking a trace is very cheap?

What you've described is tail-based sampling and it only helps with reducing storage requirements. It does not reduce the sampling overhead. Check https://uptrace.dev/opentelemetry/sampling.html

But is taking a trace cheap? Definitely. Billions of traces? Not so.

request sampling decision something that's common or that's a totally unique capability some have.

It is a common practice to reduce cost of sampling when you have billions of traces, but it is an advanced feature because it requires backends to buffer incoming spans in memory so you can decide if the trace is anomalous or not.

Besides, you can't store only anomalous traces because you will lose a lot of useful details and you can't really detect anomaly without knowing what is the norm.

Hopefully that helps.

Uptrace uses the BSL license to forbid or rather not allow other companies creating a cloud service using Uptrace code, because that is how we are planning to monetize. But you can self-host Uptrace and use it as you want to monitor your (production) application. I think this is fair enough.

If there is another license that better reflects our intentions, let us know.

I am not a lawyer so I can't keep up the discussion on the necessary level so I will just clarify few things.

Uptrace uses the BSL license to forbid or rather not allow other companies creating a cloud service using Uptrace code, because that is how we are planning to monetize. But you can self-host Uptrace and use it as you want to monitor your (production) application. I think this is fair enough.

I am sure there are many complications with re-licensing, but it happens in practice, for example, Sentry now uses BSL license. And you can't do anything about it except to fork old Sentry. But then you will have to maintain it yourself.

And I am not arguing with anything you've said, but there are not that many financially thriving truly OSS projects. That's why people like me have to complicate their lives with BSL, AGPL, and others.

Thanks for the comment!

You can self-host Uptrace and use it in production environment. This is explicitly stated in the FAQ - https://github.com/uptrace/uptrace#faq . But you can't resell Uptrace to others.

I can only repeat that the same BSL license is used by MariaDB, Sentry, CockroachDB, Couchbase, and others. I am not a lawyer and thus not qualified to discuss details.

the possibility they will be restricted further

It is true for all licenses, for example, it is possible to keep old code available under old permissive license but release all new code under a more restricted license.

None is safe! :)

We already use JWT tokens that are passed in a HTTP cookie so perhaps we could document how it works and let users sign JWT tokens themselves. That way your app only needs to set a cookie and the user should be authorized.

Let's continue the discussion on GitHub https://github.com/uptrace/uptrace/issues/76

It is the same license used by MariaDB, Sentry, CockroachDB, Couchbase and many others. Technically it is not open source and instead is called source available, but you can enjoy pretty much the same benefits.

Out of curiosity, what makes you uncomfortable about the license?

Thanks! Don't hesitate to send any feedback you have so we have a chance to improve :)

DataDog has a high learning curve and can be rather expensive if you need to monitor a lot of hosts and microservices.

Uptrace tries hard to stay simple while providing almost the same set of features. It can also be self-hosted without paying anything which can save a lot of money.

Uptrace aims to be an open source alternative to DataDog, but realistically we are not there yet.

It is possible using Vector Logs which Uptrace supports out-of-the-box, for example:

- https://vector.dev/docs/reference/configuration/sources/dock...

- https://uptrace.dev/get/ingest/vector.html

If you are having troubles making it work, feel free to open an issue on Github and I will provide a complete example.

Thanks for the feedback!

- Are there any recommendations for scaling (e.g. benchmarks) on how many spans/s are supported on what hardware?

With Uptrace, I was able to achieve 10k spans / second on a single core by running the binary with GOMAXPROCS=1. That is 1-3 terrabytes of compressed data each month which is more than most users need.

Practically, you are limited by the $$$ you are willing to spend on ClickHouse servers, not by Uptrace ingestion speed.

So my recommendation is to scale Uptrace vertically by throwing more cores at it. That will allow you to go very very far.

- Is there support for SSO for self-hosted installation?

So far the only way to add news users is via the YAML config. We are considering to add a REST API or a CLI tool for the same purpose, but it is not clear how that would work with the YAML.

Regarding the SSO, it would be nice if you can provide an app that already does that so we can better estimate the complexity. But so far we don't have such plans.

- Can I easily disable certain features? (e.g. alerting)

Yes, most YAML sections just can be removed / commented out to disable the feature.

You can ingest data using OpenTelemetry Protocol (OTLP), Vector Logs, and Zipkin API. You can also use OpenTelemetry Collector to collect Prometheus metrics or receive data from Jaeger, X Ray, Apache, PostgreSQL, MySQL and many more.

The latest Uptrace release introduces support for OpenTelemetry Metrics which includes:

- User interface to build table-based and grid-based dashboards.

- Pre-built dashboard templates for Golang, Redis, PostgreSQL, MySQL, and host metrics.

- Metrics monitoring aka alerting rules inspired by Prometheus.

- Notifications via email/Slack/PagerDuty using AlertManager integration.

There are 2 quick ways to try Uptrace:

- Using the Docker container - https://github.com/uptrace/uptrace/tree/master/example/docke...

- Using the public demo - https://app.uptrace.dev/play

I will be happy to answer your questions in the comments.

So far my experience is that it is best to avoid trying to solve such problems with a query language and instead provide a much simpler UI to achieve the same. Solving such problems with SQL is tedious enough and learning another custom language is not fun / too much to ask from users.

Sometimes using a UI is not possible, for example, if you want to automate such checks. In that case, I would build a custom metric or two and would use that metric for monitoring purposes. That requires some programming / instrumentation, but it still looks like a better solution to me.

Thanks for the answer - that all makes sense... "but" :) there is also some sense in NOT having to support different storages / plugins API and instead supporting more features that work out of the box. At least I hope so.

I will try Grafana Tempo & Loki - thanks for reminding. It is just that all their products look the same like the "original" Grafana for metrics which TBH does not look especially ergonomic too... Somehow Jaeger still comes first when people talk about tracing.

Those are 2 separate projects and they don't work together. I still did not have a chance to try loki / tempo so can't say how well they work in practice...

"But" do you need another storage? :)

I understand that it is not ideal to have so many competing tools, but contributing to an existing mature project is a nightmare. It is by far easier to start a new one.

Jaeger UI is anyway not the main tool people tend to work with.

Which tools / features do you have in mind?

Uptrace OS competes with Jaeger / Zipkin / SigNoz / SkyWalking and I believe it already does a pretty good job.

Uptrace / Go process will buffer spans in memory for some short period of time (5-15 seconds). It does not work for long traces, but most traces are short.

There is some discussion at https://github.com/open-telemetry/opentelemetry-collector-co...

Maybe, are you already using that feature in production? Buffers are available for years and just work. Hard to say how async inserts perform in real applications.

You are right about the license - it allows everything Apache 2.0 allows except reselling monitoring as a service.

OpenTelemetry Java should work with Android, but I haven't really tried it. Java supports true auto-instrumentation so it should not be hard to figure out if you have an Android project.

I've provided a Docker example so users can quickly try Uptrace without downloading and configuring anything. I don't use Docker to run anything besides examples so can't give any recommendations.

Thanks for the clickhouse-operator though. I guess we will have to provide something for k8s sooner or later.

Well, I had a delusion that Uptrace will have clean and simple UI, but I guess for others the UI is just as confusing. :(

I think we've done pretty good job with filtering+grouping+aggregation and data exploration in general. That is something I am proud of.

Uptrace is significantly cheaper.

As for the rest, it is the same but different. DD is bigger and more complex. I guess that is not the problem when you get used to the UI.

By cluster support I mean:

- ability to use ReplicatedMergeTree in the table schema

- round-robin writing to multiple nodes

It is mostly a matter of providing configuration options. Thought I could skip it in the first release.

How are you handling persistent storage?

If you mean avoiding data loss by using ClickHouse cluster, then yes - we use CH cluster and replication :)

ClickHouse handles data corruption surprisingly good - even if there are broken parts CH continues to serve the rest of data.

Sure, I am very interested to know what features are people looking for.

I don't provide direct comparison, because it is hard to judge which features are relevant. For example, the few people I've asked told me that they don't use "service dependency analysis" and I don't find that functionality useful myself. But you will find that feature in most products so perhaps I've asked the wrong people :)