Fly’s Prometheus Metrics

by elithrar • 5 years ago

164 79 5 years ago

dexen• 5 years ago

Not to pick on Fly (seems nice), but on the trend for containers:

if you’ve got a Docker container, it can be running on Fly in single-digit minutes.

I used to laugh at the old Plan 9 fortune, "... Forking an allegro process requires only seconds... -V. Kelly". Guess I'm not laughing anymore?

FWIW, performance of components is the barrier to composition in system design and development. You can't compose modules that take seconds to act, and still have something that is usable real-time.

tptacek12• 5 years ago

You’re remarking on a development time number. Like, the amount of time it’ll take to create your account.

simonw• 5 years ago

I can see how someone would misunderstand that claim though: the idea of getting up and running on a new hosting provider that quickly seems so unlikely that thinking "well they must be talking about container launch times here instead" is an understandable mistake.

maxmcd• 5 years ago

Fly runs the contents of the container in a VM. Bring your own runtime in that VM, use kernel sandboxing features, do whatever you want (in linux).

I’m all for the “k8s is not erlang” position, but in fly’s case it seems like the right tool for the job. Fly is much faster than a new EC2 instance.

n3mes1s• 5 years ago

More info about this in the blogpost Docker without Docker[0]

[0] https://fly.io/blog/docker-without-docker/

chrisweekly2• 5 years ago

You missed the mark, mate. The "single-digit minutes" applies to the one-time onboarding process, not cold-start times for a deployed service!

hstaab• 5 years ago

Does anyone here have experience using Fly? I’ve seen a few of their posts and it seems quite nice.

mrkurt11• 5 years ago

You can get a reasonable idea of the problems people run into on our community forum. I use Fly for everything but that may not be the best signal for you: https://community.fly.io/

wferrell• 5 years ago

I recently setup my own http://logpaste.com to give them a try. Fly was quite easy to use.

mtlynch2• 5 years ago

I discovered fly a month or two ago, and I've been incredibly impressed. The dev experience is really nice, and the team is super responsive.

Aeolun• 5 years ago

It seems fairly expensive for what you get?

itsjloh• 5 years ago

I use fly to host a few apps and its fantastic. Extremely responsive and it handling TLS offload is great from a performance perspective.

I find their billing dashboard leaves a bit to be desired, I'm still very nervous I'm going to get a big unexpected bill.

mrkurt11• 5 years ago

If you get a big surprise bill, email me and we'll fix it. We're not interested in surprising people with bills.

jbarham• 5 years ago

I recently started using Fly to host the anycast DNS name servers for my DNS hosting service SlickDNS (https://www.slickdns.com/).

There were some initial teething issues as Fly only recently added UDP support, but they were very responsive to the various bugs I reported and fixed them. My name servers have been running problem free for several weeks now.

The Fly UX via the flyctl command-line app is excellent, very Heroku-like.

For apps that need anycast the only real alternative to Fly that I found is AWS Global Accelerator, but it limits you to two anycast IP addresses, it's much more expensive than Fly and you're fighting the AWS API the whole time.

thegreatpeter• 5 years ago

Love Fly. Been using it for about a year now? We host Draftbit in a bunch of regions. The community is growing and they know their shit.

ryanschneider• 5 years ago

How big is the fly eng team at this point? You all seem to be doing a ton, I’m always kind of surprised these posts don’t end with the usual “we’re hiring” blurb that’s become the norm on these sorts of tech posts.

mrkurt11• 5 years ago

We're small, only 7 people. That's on purpose (so far). The magic 8 ball says we'll grow in the next few months though.

chrisweekly2• 5 years ago

You guys really punch above your weight. Like Bruce Lee. Here's hoping you grow slow and stay that way!

yannoninator• 5 years ago

I'm not quite sure if fly is production ready for our usecase yet, but it does look awesome though.

mrkurt11• 5 years ago

It depends entirely on your stack + use case! If you feel like emailing I can give you an honest take. :)

krono• 5 years ago

First time I've heard of Fly, went to check out their docs and found this awesome little mention:

We use Lets Encrypt to issue certificates, and donate half of our SSL fees to them at the end of each calendar year.

tptacek12• 5 years ago

We wrote a similarly tedious piece about how our certificate system works, if you're interested: https://fly.io/blog/how-cdns-generate-certificates/

corobo• 5 years ago

Just for a bit of social recognition I love bumping into one of you folks' similarly tedious pieces!

Any chance of an RSS feed on that blog? [e: ah it's just missing the indicator meta tag thingy, https://fly.io/feed.xml]

Sytten• 5 years ago

They mentionned that they decided against thanos for the storage of metrics, but would be curious to hear if other TSDB were considered. It is a hot space, I know about M3BD, Clickhouse, Timescale, influx, QuestDB, opentsdb, etc.

mrkurt11• 5 years ago

We kind of naturally got from Prometheus -> VictoriaMetrics. Vicky is very simple to operate, which was a win. I'm a huge fan of Timescale, though, and we have big Postgres plans that I hope include Timescale. :)

sagichmal6• 5 years ago

VictoriaMetrics

Not actually Prometheus-compatible, sloppy code, spotty docs. I have no idea why this dumb product continues to attract users.

https://prometheus.io/blog/2021/05/04/prometheus-conformance...

Telegraf, which is to metrics sort of what Logstash is to logs: a swiss-army knife tool that adapts arbitrary inputs to arbitrary output formats. We run Telegraf agents on our nodes to scrape local Prometheus sources, and Vicky scrapes Telegraf. Telegraf simplifies the networking for our metrics; it means Vicky (and our iptables rules) only need to know about one Prometheus endpoint per node.

Normally you just use a regular Prometheus server to do this. Why add another, different technology to the stack?

We spent some time scaling it with Thanos, and Thanos was a lot, as far as ops hassle goes.

It really isn't -- assuming you're not trying to bend Prometheus into something it isn't. Prometheus works using a federated, pull-based architecture. It expects to be near the things it's monitoring, and expects you to build out a hierarchy of infrastructure, in layers, to handle larger scopes.

This is structurally different to what I'll call the "clustering" model of scale, where you have all your data sources pushing their data, aggregating maybe on the machine or datacenter level, but then shuttling everything to a single central place, which you scale vertically from the perspective of your users. This appears to be what you want to do, based on the prevalence of push-based tech in your stack.

Prometheus doesn't work this way. Some people really want it to work this way, and have even created entire product lines that make it look as if it works this way (Cortex, M3db) but it's fundamentally just not how it's designed to be used. If you try to make it work this way yourself, you'll certainly get frustrated.

mrkurt11• 5 years ago

Normally you just use a regular Prometheus server to do this. Why add another, different technology to the stack?

Our physical hosts have hundreds of services exporting metrics. And many of those exported metrics are from untrusted sources. So we can both rewrite labels and decrease the scrape endpoint discoverability problem by aggregating them in one place.

Not actually Prometheus-compatible, sloppy code, spotty docs. I have no idea why this dumb product continues to attract users.

Because it works incredibly well, it's easy to operate, and handles multi tenancy for us.

sagichmal6• 5 years ago

Our physical hosts have hundreds of services exporting metrics. And many of those exported metrics are from untrusted sources. So we can both rewrite labels and decrease the scrape endpoint discoverability problem by aggregating them in one place.

OK, but Prometheus can do all of this just fine?

Because it works incredibly well, it's easy to operate, and handles multi tenancy for us.

Again, Prometheus itself ticks all of these boxes, too, if you're not trying to force it to be something it's not.

vira28• 5 years ago

I really like those diagrams. What tools/software is used to generate?

mrkurt11• 5 years ago

excalidraw.com

It's pretty great, we're happy plus subscribers.

pm902• 5 years ago

Whose the target audience for fly?

I’m trying to understand the market they’re operating in. Big ol enterprises would probably want to run on AWS/GCP right? So would startups? What’s the long game? Genuine question.

ttul• 5 years ago

It all starts with a developer at BigCo who is frustrated with the complexity of LegacyCo’s stack... so she build it on NewCo’s coolness. And then suddenly BigCo wants enterprise features from NewCo, and so it goes...

tantalor• 5 years ago

If you’re an Advent of Code kind of person and you haven’t already written a parser for this format, you’re probably starting to feel a twitch in your left eyelid. Go ahead and write the parser; it’ll take you 15 minutes and the world can’t have too many implementations of this exposition format. There's a lesson about virality among programmers buried in here somewhere.

Huh? Who gets excited about writing a parser?

What was wrong with "${key} ${value}" on separate lines?

tptacek12• 5 years ago

Is "excited" the word? I don't know. I think "obsessively compelled" is more what I was going for. It's one of those formats where --- despite apparently being originally intended for human consumption --- you can immediately see the rule of construction for, like you're not just reading the data, but also the pseudocode for how it's formatted.

chris_st• 5 years ago

Having written a lot of parsers, I really loved how you put a newline in a "keyword"-type thing, just to keep it interesting :-)

mdoms• 5 years ago

Well I suppose what's wrong with it is that these metrics are more complicated than key=>value. The metric referred to in your quote contains a metric name, multiple attributes (status, app ID etc) and a value.

devoutsalsa• 5 years ago

I love writing one liner parsers in Elixir when I’m bored. Pipes and pattern matching for the win!!

jimmyed• 5 years ago

Thanos is a lot of ops, true. But did they try Cortex? Oh, and M3?

mrkurt11• 5 years ago

We looked at Cortex but running a Cassandra cluster was a little too much for us. I don't think we saw M3 until after we'd started using VictoriaMetrics.

bboreham2• 5 years ago

Cortex doesn’t need a NoSQL database now, since about a year ago. It does need something S3-like.

(I am a Cortex maintainer)

kasey_junk3• 5 years ago

Alas, we’ve found the first thing I _dont_ like about fly. They’ve sadly bought into the TSDB model that everything is a counter instead of the more modern model that everything is a histogram.

Google has long since abandoned the Borgmon data model for histograms with monarch. The closest non google implementation is probably circonus. Sadly neither is available as open source software.

I can’t really blame fly for not individually building an open source modern metric db. But it’s sort of sad that the infra team I’m most impressed with has to use metric systems from 15 years ago when the rest of their stack is so cutting edge.

tptacek12• 5 years ago

We do plenty of histograms! Vicky is good at them! Counters are just easier to write about.

kasey_junk3• 5 years ago

Vicky is about the best case scenario for a counter based metrics db. I am routinely impressed by the things it accomplishes but monarch & irondb from circonus use histograms as their base data structure which means they avoid all the hacks that counter based tsdb have to deal with. The obvious one in the Prometheus stack being having to pre-declare your histogram bounds.

polskibus• 5 years ago

What would you recommend today as a self-hosted solution today?

kasey_junk3• 5 years ago

Circonus’ Irondb is the only histogram db available outside of google that I know of.

They have a self hosted option but it’s not free or open source.

darkwater• 5 years ago

So, most shops (at least the ones that have been existing for a few years) are just starting their move to Prometheus (or in the middle of it) and we already have a newer, better implementation to migrate to? I love the tech world! (I'm half ironic and half serious with my last sentence)

Spivak• 5 years ago

Well yeah because Prometheus was inspired by Google's Borgmon and soon there will be another product inspired by Google's Monarch.

Google's SRE stack is lightyears ahead of anything else out there simply because they're at the scale where they can afford to hire dedicated developers to just write internal ops software where most other SREs are understaffed and overworked just on operational projects.

StreamBright2• 5 years ago

For those who are curious and not familiar with Monarch:

https://www.youtube.com/watch?v=2mw12B7W7RI

https://www.vldb.org/pvldb/vol13/p3181-adams.pdf

bboreham2• 5 years ago

Note there is work under way in Prometheus to make histograms more flexible:

https://static.sched.com/hosted_files/promcononline2021/33/2...

xfer• 5 years ago

Do you plan to join the bandwidth alliance at cloudflare? Data prices are quite costly. It's the same as GCP/AWS.

0xbadcafebee3• 5 years ago

Fly.io transforms container images into fleets of micro-VMs running around the world on our hardware.

Oh boy!

None of us have ever worked for Google, let alone as SREs. So we’re going out on a limb

Oh.... boy.

We spent some time scaling it with Thanos, and Thanos was a lot, as far as ops hassle goes.

You know, they have these companies now, that will collect your metrics for you, so that you don't have to deal with ops hassle.

In each Firecracker instance, we run our custom init,

... in Rust. Yes, the thing that is normally a shell script, is now a compiled program in a new language, that mostly just runs mkdir(), mount() and ethtool(). (https://github.com/superfly/init-snapshot/blob/public/src/bi...). In a few years, when that component is passed off to a dedicated Ops team, and they find it hard to hire a sysadmin who also knows Rust, there will be some poor intern who learned Rust over the summer whose job is to rewrite that thing back into a shell script.

tptacek12• 5 years ago

It sounds like you're saying that it would be better if our init were literally a bash script. Is that what you're saying?

0xbadcafebee3• 5 years ago

Not literally, no, as you need an init program that performs certain steps first which bash can't do. But once those few operations are done you can pass execution off to a script.

It looks like your init is running the gamut of typical linux init steps: handling signals, spawning processes, handling TTYs, running typical system start-up commands. Then it gathers host and network metrics, reports results, and operates as some kind of websocket server? All of that other stuff should live in a separate dedicated program that the init script runs. Unless I missed something and that's not possible?

StreamBright2• 5 years ago

Now I like Fly even more. I used to be an SRE for AMZN and I 100% on board with replacing shell scripts and init with types configuration files (like Dhall for example) and a system that is written in Rust for parsing and understanding those files. I think this is the best part of SystemD. There are other projects like S6 for example. I need to understand more about Fly's implementation of init, I am very curious.

jzelinskie• 5 years ago

Hadn't heard of promxy before. In the past to reduce cardinality/deduplicate metrics, I've just ran another instance of Prometheus entirely in-memory and used rewrite rules.

Exposing a metrics endpoint for customers is nice. How do you manage the cardinality? I haven't used Victoria before, is it just better at high cardinality time series?

uaas• 5 years ago

That’s its promise, e.g. https://valyala.medium.com/high-cardinality-tsdb-benchmarks-...

FWIW, we are in the process of considering it as a single monitoring plane for multiple clusters with high cardinality metrics

mrkurt11• 5 years ago

Vicky isolates tenants into their own "shards" so we mostly don't have to worry about it. It's also pretty good at high cardinality.

twic• 5 years ago

When it comes to automated monitoring, there are two big philosophies, “checks” and “metrics”.

There's a third, "events". Just push an event out whenever something interesting happens, and let the monitoring tool decide whether to count, aggregate, histogram, alert, etc.

Events require less code in the app (no storage, no aggregation, no web server), and allow more flexibility in processing. I have used events to great effect. I am baffled as to why monitoring people still only talk about metrics.

Fly’s Prometheus Metrics

Related Stories