valyala

Sentry stores logs in ClickHouse - https://blog.sentry.io/how-sentry-queries-unstructured-data-...

It would be great comparing NanoTDB to VictoriaMetrics. Both are optimized for running on Raspberry Pi. Both have built-in web UI for the exploration of the stored metrics. Both consist of a single executable written in Go, which stores the collected metrics to a single directory on a local filesystem. I don't know whether NanoDB is optimized for low-cost flash storage with small number of lifetime writes. VictoriaMetrics is optimized for such a storage, so it saves the flash drive life.

Agreed 100%. https://itnext.io/go-evolves-in-the-wrong-direction-7dfda8a1...

https://lwn.net is a goldmine of great contents about Linux kernel.

Let me explain why I post links to posts from https://internals-for-interns.com . I hate AI slop. The author of these posts uses AI for generating these posts - this is clearly visible via AI-generated images, emojis and em dashes. But the posts themselves do not contain misleading slop you could see in a typical AI-generated content. These posts are very clear and accurate. While they contain some inaccuracies and mistakes, the number of these mistakes is very low. These posts help learning and understanding complex technical topics such as internals of Go, filesystems, databases and linux kernel, by reading a clear easy to understand text.

It is important to differentiate between low-quality AI slop and high-quality contents generated with the help of AI. The posts at https://internals-for-interns.com belong to the second category.

While at it, I recommend reading the article from ClickHouse author on how to properly use AI - https://clickhouse.com/blog/agentic-coding .

This AI slop, on which I was able to quickly find four mistakes

Could you provide more details about these mistakes?

VictoriaMetrics CTO here.

We at VictoriaMetrics took another approach. We tried using ClickHouse as a database for metrics in 2017, but then decided implementing a specialized database for metrics. This database uses ClickHouse architecture ideas for achieving the best performance and the lowest resource usage. The main difference between ClickHouse and VictoriaMetrics is that VictoriaMetrics is optimized solely for typical observability tasks. It supports all the popular data ingestion protocols, it provides promql-compatible querying API, it provides Graphite-compatible querying API, it provides Prometheus-compatible service discovery and relabeling, it provides Prometheus-compatible alerting and recording rules. It provides built-in web UI for quick exploration and analysis of the ingested metrics, with the ability to investigate the source of high cardinality. It consists of a single small executable (~20MB) without external dependencies with minimum configs and minimum maintenance. See https://altinity.com/wp-content/uploads/2021/11/How-ClickHou... for more details.

We used the same approach for building VictoriaLogs - a specialized database for logs. It uses the most appropriate architecture ideas from ClickHouse for achieving high performance and low resource usage. It is schemaless and zero-config. It contains of a single small executable without external dependencies. It accepts logs via popular data ingestion protocols. It provides a specialized query language for typical queries over production logs - LogsQL. This language is much simpler to use than SQL for querying typical logs. It provides a built-in web UI for quick exploration of the ingested logs. It provides a Grafana plugin for building arbitrary complex dashboards from the stored logs. It provides the ability to build alerts and metrics from the stored logs. See https://docs.victoriametrics.com/victorialogs/faq/#what-is-t...

I'm core developer at VictoriaMetrics. This information is one click away - just click my name here.

Yes, there is a solution for masking the read latency at object storage - to run many readers in parallel. I tweeted about it some time ago - https://x.com/valyala/status/1965093140525715714

Interesting solution! According to the provided numbers at "query latency" chapter, the query over cold data, which selects samples for 497 time series over 6 hours time range takes 15 seconds if the queried data isn't available in the cache. This means that typical queries over historical data will take eternity to execute ;(

There is another approach for solving this issue - to use increase_pure() function from MetricsQL - https://docs.victoriametrics.com/metricsql/#increase_pure . Of course, you need to switch to VictoriaMetrics, since Mimir doesn't support this function.

It is interesting why Airbnb uses vmagent for streaming aggregation and didn't switch from Mimir to VictoriaMetrics. This could save them a lot of costs on infrastructure and operations, like in cases of Roblox, Spotify, Grammarly and others - https://docs.victoriametrics.com/victoriametrics/casestudies...

This is a good alternative for those who wants storing petabytes of historical logs, metrics or traces in VictoriaLogs, VictoriaMetrics and VictoriaTraces, and wants saving 2x-4x on the persistent storage pricing (compare EBS pricing to S3 pricing).

If the server cannot keep up with the given workload because of some bottleneck (CPU, network, disk IO), then it cannot guarantee any response times - incoming queries will be either rejected or queued in a long wait queue, which will lead to awfully big response times. This doesn't depend on the programming language or the framework the server written in.

If you want response time guarantees, make sure the server has enough free resources for processing the given workload.

If you are struggling with observability solutions which require object storage for production setups after such news (i.e. Thanos, Loki, Mimir, Tempo), then try alternatives without this requirement, such as VictoriaMetrics, VictoriaLogs and VictoriaTraces. They scale to petabytes of data on regular block storage, and they provide higher performance and availability than systems, which depend on manually managed object storage such as MinIO.

Why do you need non-trivial dependency on the object storage for the database for logs in the first place?

Object storage has advantages over regular block storage if it is managed by cloud, and if it has a proven record on durability, availability and "infinite" storage space at low costs, such as S3 at Amazon or GCS at Google.

Object storage has zero advantages over regular block storage if you run it on yourself:

- It doesn't provide "infinite" storage space - you need to regularly monitor and manually add new physical storage to the object storage.

- It doesn't provide high durability and availability. It has lower availability comparing to a regular locally attached block storage because of the complicated coordination of the object storage state between storage nodes over network. It usually has lower durability than the object storage provided by cloud hosting. If some data is corrupted or lost on the underlying hardware storage, there are low chances it is properly and automatically recovered by DIY object storage.

- It is more expensive because of higher overhead (and, probably, half-baked replication) comparing to locally attached block storage.

- It is slower than locally attached block storage because of much higher network latency compared to the latency when accessing local storage. The latency difference is 1000x - 100ms at object storage vs 0.1ms at local block storage.

- It is much harder to configure, operate and troubleshoot than block storage.

So I'd recommend taking a look at other databases for logs, which do not require object storage for large-scale production setups. For example, VictoriaLogs. It scales to hundreds of terabytes of logs on a single node, and it can scale to petabytes of logs in cluster mode. Both modes are open source and free to use.

Disclaimer: I'm the core developer of VictoriaLogs.

You can expose Unikernel application metrics in Prometheus text exposition format at `/metrics` http page and collect them with Prometheus or any other collector, which can scrape Prometheus-compatible targets. Alternatively, you can push metrics from the Unikernel to the centralized database for metrics for further investigation. Both pull-based and push-based metrics' collection is supported by popular client libraries for metrics such as https://github.com/VictoriaMetrics/metrics .

You can emit logs by the Unikernel app and send them to a centralized database for logs via syslog protocol (or any other protocol) for further analysis. See, for example, how to set up collect ing logs via syslog protocol at VictoriaLogs - https://docs.victoriametrics.com/victorialogs/data-ingestion...

You can expose various debug endpoints via http at the Unikernel application for debugging assistance. For example, if the application is written in Go, it is recommended exposing endpoints for collecting CPU, memory and goroutines profiles from the running application.

There is no need in the operating system to run Unikernels. Every Unikernel includes parts of operating system needed for interacting with the underlying hardware. So Unikernels can run on bare metal if they know how to interact with the underlying hardware (i.e. if they have drivers for that hardware). Usually Unikernels are targeted to run on virtual machines because virtual machines have unified virtualised hardware. This allows running the same Unikernel on virtual machines across multiple cloud providers, since they have similar virtual hardware.

It monitors all the applications in your network and automatically detects the most common issues with the applications. It also collects metrics, traces, logs and CPU profiles for the monitored applications, so you could quickly investigate the root cause of various issues if needed.

I like that Coroot works out of the box without the need in complicated configuration.

Coroot is the future of observability

The number of active users at StackOverflow started dropping in the middle of 2020, i.e. long time before ChatGPT release in the end of 2022.

https://data.stackexchange.com/stackoverflow/revision/192836...

Nice article about the usefulness of wide events! It's pity it doesn't name open-source solutions optimized for wide events such as VictoriaLogs.

Try open-source databases specially designed for traces, such as Grafana Tempo or VictoriaTraces. They can handle the data ingestion rate of hundreds of thousands trace spans per second on a regular laptop.

MinIO also allows losing your data, since it doesn't provide high durability guarantees unlike S3 and GCS.

But what's the point to use these DIY object storage systems, when they do not provide durability and other important guarantees provided by S3 and GCS?

Consider migrating to VictoriaMetrics and saving on storage costs and operations costs. You also won't need MinIO, since it stores data to local filesystem (aka to regular persistent volumes). See real-world reports from happy users who saved costs on a large-scale Prometheus-compatible monitoring - https://docs.victoriametrics.com/victoriametrics/casestudies...

What is the purpose of MinIO, Seaweedfs and similar object storage systems? They lack durability guarantees provided by S3 and GCS. They lack "infinite" storage promise contrary to S3 and GCS. They lack "infinite" bandwidth unlike S3 and GCS. They are more expensive than other storage options, unlike S3 and GCS.

we did, I think the entire journey was 7 years long, communicated many times, over at least 6 major releases. maintaining dashboards in two languages increased complexity, whilst reducing compatibility, and gave a very large security surface to be worried about. we communicated clearly, provided migration tools, put it in release notes, updated docs, repeated it at conferences and on community calls

If this migration appeared to be so painful, why did you decide to finish it (and make users unhappy) instead of cancelling the migration at early stages? What are benefits of this migration?

many are adopting to gain independence from vendors

This is the worst reason to migrate to OTEL format for metrics, since every vendor and every solution for metrics has its' own set of transformation rules for the ingested OTEL metrics before saving them into the internal storage (this is needed in order to align OTEL metrics to the internal data model unique per each vendor / service). These transformation rules are incompatible among vendors and services. Also, every vendor / service may have its own querying API. This means that users cannot easily migrate from one vendor / service to another one by just switching from the old format to OTEL format for metrics' transfer. Read more about this at https://x.com/valyala/status/1982079042355343400

I'm the core developer at VictoriaMetrics - you can find this information by clicking my account name here.

Now it's your turn to disclose your employer name.

Boring monitoring software is rare, since it is hard to refuse adding yet another shiny feature (and breaking the core functionality along the way) instead of focusing on improving the usability for the core functionality.

We at VictoriaMetrics try making boring monitoring solution which just works out of the box - https://docs.victoriametrics.com/victoriametrics/goals/