Improving Observability of Golang Services

https://flow.com/engineering-blogs/golang-services-improving-observability

by dmicher • 3 years ago

14 4 3 years ago

dmicher• 3 years ago

This blog post is aimed at GoLang developers who are looking to improve the observability of their services. It skips the basics and jumps straight to advanced topics, such as asynchronous structured logging, metrics with exemplars, tracing with TraceQL, aggregating pprof and continuous profiling, microbenchmarks and basic statistics with benchstat, blackbox performance tests, and basic PID controllers for determining a system's maximum load. We'll also briefly touch on current research in the observability space, including active casual profiling and passive critical section detection.

sheyda• 3 years ago

Great, well-researched write-up with practical examples. Well done - Alexey Ivanov!

veshij• 3 years ago

There are quite a few different options described in the paper. And I guess it was a long journey to find a solution which you are happy with. Which configuration you ended up with?

SaveTheRbtz• 3 years ago

We're mostly on the Grafana stack, so we tend to favor their open-source solutions for metric backends. For client libraries, we're currently using Prometheus for metrics and OpenTelemetry (OTEL) for tracing, but we're looking to migrate fully to OTEL once metrics are stable there. As for logging, it depends on the project, but generally we use either Zerolog or Zap, with either Loki Docker plugin or Promtail.

Improving Observability of Golang Services

Related Stories