pahae

I set up a fairly large Prom-based architecture which I later on migrated to VictoriaMetrics (VM) so I think I can chime in here.

Both Prom and VM are exceptionally stable in my opinion, even on _very_ large scales. There were times when I had a single (Prom, later VM) and not-overly-large instances scrape 2Mio samples/s without any issues. In addition to fairly spiky query loads.

However, if something does go wrong, the single most impactful difference between VM and Prom is simply the difference in startup time. Prometheus with 2TB of metrics takes _forever_ to start up. We're talking up to 2 hours on SSD while VM just... starts.

Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.

The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.

I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.

I mentioned it in another reply, but https://perses.dev/ is probably the most promising alternative.

Besides that, if you're feeling masochistic you could use Prometheus' console templates or VictoriaMetrics' built-in dashboards.

Though these are all obviously nowhere near as feature rich and capable as Grafana and would only be able to display metrics for the single Prom/VM node they're running on. Might enough for some users.

I only know of https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.

I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.

I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.

Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.

I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously.

My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider.

As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links.

I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time.

Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?

Correct. This can be disabled [0] but you need to work around this then. Usually you can "just" use host-networking and manage iptable rules manually. Not pretty but in that case you at least know what's done to the system and iptables.

[0] https://docs.docker.com/engine/network/packet-filtering-fire...

Quadlets are pretty nice but require podman > 4.4 to function properly. Debian 12, for example, still only has podman ~4.3 in its repos.

I actually like their take on the vacat page [0] even more.

0: https://en.wikipedia.org/wiki/Intentionally_blank_page

We used Seafile in $old_job a couple of years back. Still had some rough edges back then (mostly with user management) but the rest was rock solid. Loved working with it, managing and sharing files was pretty easy.

I'm not a aware of any other full-fledged solution. There is some opinionated tooling written in Python [0] and Go but nothing coming close to defined.net. This is definitely a point where other overlay solutions look way better.

After having searched (and implemented) this myself for work, the only practical solutions I found were 1) smallstep [1] or 2) Terraform (with the nebula provider [2]) and a CM tool of your choice. The latter can be nicely combined with the ansible provider if that's your CM of choice.

0: nebula-cert-py 1: https://smallstep.com/docs/step-ca/integrations/#nebula 2: https://registry.terraform.io/providers/TelkomIndonesia/nebu...

Catering more to a German speaking audience but nonetheless a pretty good overview (updated quarterly?):

Performance: http://www.3dcenter.org/artikel/fullhd-ultrahd-performance-u...

Performance/Price w/ German market prices: http://www.3dcenter.org/artikel/grafikkarten-marktueberblick...

It is striking that such a backdoor in seemingly safe OSs like Linux and FreeBSD [...]

Not saying these are a 100% secure but you're plying this was _built_ into Linux (and other OSes). It was not. And the PDF makes no such claim either.

Nebula is fantastic, absolute love it. We use it in production. Cert management can be a bit of a pain on a large scale but there's an excellent Terraform provider [0] that can help. Coupled with the Terraform ansible provider and a little bit of scripting you can automate anything related to cert provisioning and renewal.

[0] https://registry.terraform.io/providers/TelkomIndonesia/nebu...

You should give nebula a try. I've recently switched my private VPN setup from wireguard to nebula and am looking into using it for work. It has some really nice features (for our use case), so ymmv. But so far it's been fantastic and very easy to use.

https://github.com/slackhq/nebula

Yeah, I think many "buy it for life" items are generally only sensible if you depend on them either for work, a somewhat regular hobby, or for everyday use.

Sometimes it almost hurts me to buy cheap junk. But then I also realize that this is perfectly adequate for my use cases and even $10 more for another product would not be worth it. (And who says that the other product would really be better?).

Apparently anything more than clearing the launch pad was a win. Musk's dead stare seems to disagree tho

After clicking through a few random lectures, this seems fantastic. I wish I'd found this during my physics B.Sc.

I'd rather have some physical knob that I can use to turn something off, even if I have to bust out the pliers. Good luck shutting off some touch control that's on the fritz.

The stove at my parents place has some touch control to activate another heating ring(?). Not only is that hard to activate with wet or greasy fingers, but after a couple of years of use it keeps turning itself off now. Thank god that thing fails "closed" and does not (yet?) spontaneously activate effectively doubling the heating power.

This is one of the reasons I bought a USB hub were I can switch connected devices off and on (disconnects their power lines). If you're lazy like me you can plug another USB hub into one of the ports so that can switch multiple devices (webcam and mic) with one button press instead of two :) Searching for "usb hub power switch" should get you a few hits.

I've been using something similar: https://github.com/vincentdoerig/latex-css Seems like it could easily be adapted to match the style.

Thank you for your reply and clarification. This is quite an interesting topic for me as I've tested and implemented similar setups.

Does image processing, runs our analytics, [...]

Fair enough, I was strictly going by the diagrams. From my experience with a somewhat similar setup (HA Loki, HA Prom + Thanos with a MinIO storage backend using Terraform + Ansible and docker) I have to say that the most complex and frustrating part was configuring Loki (this was way before they expanded their documentation, which still isn't great). I'd imagine this would be even more challenging under k8s at least if you stray from the vanilla deployment and/or charts. I agree with your statement regarding Ceph, we use it extensively in production (probably on a much bigger scale). However, I think Ceph, unlike MinIO, just adds unnecessary complexity to your setup.

Well yes and no, the number of metrics isn't relevant per se, but its cardinality is very relevant [...]

Cardinality is something you should avoid when using Prometheus - for exactly that reason. There are, in my opinion, very few good reasons for dynamic labels (ignoring the baked-in cardinality from a setup like k8s). On first impulse I'd say you're doing metrics wrong but then again, I do not know enough about your use case. Maxing out a single Instance of Prometheus is no easy feat however, especially if your infra isn't that complex and/or big. I've used Thanos for so long now, how does the Cortex compactor handle range queries? Does it also compact and create additional 5m & 1h resolution metrics? These might help with your larger range queries.

Just out of curiosity, have you had any look at alternatives like Victoriametrics?

7k logs per second [...] are nothing

My remark was just regarding the added complexity as this depends solely on the size of your log messages. If you don't need or use the (awesome!) capabilities of Loki + Grafana and just need a place for long-term storage of your logs, a 'simple' rsyslog server will do just fine.

we collect prometheus metrics for MySQL, PHP-FPM, Varnish [...]

Many (if not all) of these can be handled by Telegraf or Fluentd plus InfluxDB (not that I'd used that myself, I absolutely love Prometheus and its Eco-system). My tongue-in-cheek comment was mostly about the Prometheus instance you deploy on every server just to scrape metrics locally and remote-write them into Cortex. Why not the more usual setup of (one or more) Prometheus instances scraping their targets and writing to Cortex?

There are quite a few on ebay if you're desperate for one. Probably won't find the newer 4-NIC models, though.

Ebay or a licensed reseller (if there are any) should be easier anyways. I remember them requiring a VAT ID for EU customers. I ordered a couple for a small business so that was no problem back then but private individuals are pretty much out of luck.

What are you referring to exactly? I do remember having trouble with their official OS until I switched to Armbian on my aged BPI-M1+

Looking at their diagrams it seems that the k8s cluster exists solely to handle their monitoring and logging needs which would be extreme overkill, especially since 18k metrics/samples and 7k logs per second are nothing. Plus you now suddenly need a S3-compatible storage backend for all your logs and metrics. Good thing Ceph comes 'free' with Proxmox, I guess.

Deploying an instance of Prometheus with *every host is also unusual, to say the least and I don't quite understand their comment to that. If you don't like a pull-based architecture (which is a valid point) why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.