beorn7

The checkpointing is just a way to minimize sample losses from unfinished chunks upon a crash. If you don't care or if you don't crash, you can disable checkpointing. Normal operation will be the same.

Invalidation of old data is super easy with the chunked files as you can simply "truncate from the beginning", which is supported by various file systems (e.g. XFS). However, benchmarks so far showed that the relative amount of IO for purging old chunks is so small compared to overall IO that we didn't bother to implement it yet. Could be done if it turns out to be critical.

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

(Disclaimer: I'm a Prometheus author.)

Chunks are only written to that one file per time series once they are complete. Depending on their compression behavior, they will contain at least 64 samples, but usually a couple of hundreds. Even then, chunks are just queued for disk persistence. The storage layer operates completely from RAM, only has to swap in chunks if it was evicted from memory previously. Obviously, if you consistently create more sample data than your disk can write, the persist queue will back up at some point and Prometheus will throttle ingestion.