Mach: A Pluggable Metrics Storage Engine for the Age of Observability [pdf]

https://www.cidrdb.org/cidr2022/papers/p12-solleza.pdf

by greghn • 4 years ago

33 8 4 years ago

Terretta• 4 years ago

Paper from CIDR’22, January 9-12, 2022, Chaminade, USA

“This paper describes Mach, a pluggable storage engine we are building specifically to handle high-volume metrics data. Similar to many popular libraries (e.g., Berkeley DB, LevelDB, RocksDB, WiredTiger), Mach provides a simple API to store and retrieve data. Mach’s lean, loosely coordinated architecture aggressively leverages the characteristics of metrics data and observability work- loads, yielding an order-of-magnitude improvement over existing approaches—especially those marketed as ‘time series database systems’ (TSDBs). In fact, our preliminary results show that Mach can achieve nearly 10× higher write throughput and 3× higher read throughput compared to several widely used alternatives.”

candiddevmike• 4 years ago

I just want one binary to store metrics, traces, and logs

awestroke• 4 years ago

That would make way too much sense

greghn• 4 years ago

According to the paper, that's one of the goals of the project:

"In the future, we plan to investigate several natural extensions to Mach. Specifically, we believe that metrics workloads exhibit similar characteristics to other observability data types: logs, events, and traces [20, 21]. For example, integrating log data will require the investigation of fundamental design considerations (e.g., trading off compression speed with size, enabling search over compressed data), whereas supporting traces might necessitate the development of entirely new data models (e.g., graph-based approaches to facilitate root cause analysis). Our ultimate goal is to establish Mach as a unifying storage engine for observability workloads."

ignoramous• 4 years ago

We (ab)use our logging pipeline for metrics too (can afford to do so because of our rather meagre scale, which also means we don't really need tracing much).

Whenever metrics are needed, we pull (hive-compatible time-partitioned) logs (transformed to parquet or csv) from the blob store into DuckDB and on-demand query from there. No online dashboards, but pretty good for the cost.

https://duckdb.org/docs/extensions/httpfs

catiopatio• 4 years ago

I know that naming is hard, and almost every short name has already been taken, but “Mach” is really already taken:

https://en.m.wikipedia.org/wiki/Mach_(kernel)

Mach is still quite relevant on Apple’s platforms, very visible in public APIs, and you’d be ill-advised to try to claim symbols with a “mach_” prefix.

valyala• 4 years ago

VictoriaMetrics core developer here.

Intetesting paper! It mentions VictoriaMetrics, but doesn't compare its write and read performance and scalability to Mach. It would be great performing such a comparison, since VictoriaMetrics should scale better than both Prometheus and ImfluxDB (the systems used in performance comparisons) with the number of available CPU cores according to [1].

It looks like Mach maintains an in-memory append-only buffer per each active time series with 4KiB size. This means that its memory usage can be estimated as 4KiB * active_series. For example, it needs at least 4GiB of RAM for a million of active time series. This is 4x bigger than VictoriaMetrics' memory usage.

A 4KiB in-memory buffer may contain up to 4KiB/16=256 (timestamp,value) samples. If a time series receives every new sample every 10 seconds, then 256 samples may cover up to 42 minutes worth of data. If per-series in-memory buffers are persisted when they become full, then Mach may lose up to 42 minutes of data on unclean shutdown such as out of memory crash or power off. This looks like not very good tradeoff for higher data ingestion speed.

It is also unclear how efficiently does Mach compress data stored on disk. Production stats from VictoriaMetrics users show that the needed disk space is the most expensive part of their monitoring stack. That's why on-disk data compression efficiency is very important here. VictoriaMetrics provides better on-disk metrics compression than other monitoring solutions at the cost of slighlty higher CPU usage [2].

[1] https://valyala.medium.com/measuring-vertical-scalability-fo...

[2] https://faun.pub/victoriametrics-achieving-better-compressio...

codeduck• 4 years ago

Hi. As a long-time user of VM I just want to give you a shout out for your work on it as a system.

In over 3 years we've only had one moderate issue with it - a cpu usage profile change between versions of vmstorage that necessitated an emergency instance change.

It is a phenomenal piece of software and you should feel proud of what you have accomplished.

Mach: A Pluggable Metrics Storage Engine for the Age of Observability [pdf]

Related Stories