Observe a Service; Not a Server

https://blog.last9.io/services-not-server-observability/

by kiyanwang • 5 years ago

17 6 5 years ago

blog.last9.io

pmlnr• 5 years ago

Right. Do this and you'll never be able to find that single machine that believes it's on a 100mbit line instead of 10G and keep wondering why every X user has a problem.

Monitor both, please.

imwillofficial• 5 years ago

A billion times this. The cattle still need love.

cotillion• 5 years ago

Yes monitor. You really want to find that server which is busy encrypting all that nice data your service needs.

derefr• 5 years ago

The article is talking about "monitoring" in the sense of alerting + stakeholder incident reporting. Things that reach out and tell ops staff / users that the service is having a problem.

You do want to be capturing metrics about your servers — observability into their configuration and behavior — but capturing that type of data is not precisely the same thing as "monitoring" the server. "Monitoring" usually implies having a defined acceptance threshold on the metric, and triggering some (probably human) response when the metric is in an unacceptable range.

If you don't have either a clear threshold for the metric, or a clear "thing that should happen" when the metric goes out-of-tolerance, then you aren't "monitoring" the metric per se; or at least, you shouldn't be.

As the article says:

Here’s an example: A disk consuming 100% at 3 AM may or may not be actionable depending on its impact on service KPI. For instance, if it’s a logout service and the KPI says 99.999% availability despite a disk failure, It can wait till the morning.

If you have set up alerting for server disks being full, where that alert often doesn't translate into any problem with your running services, then that alert is a useless/noise alert, that will only serve to make you treat notifications from your alerting infrastructure with less urgency. It's the "check-engine light" of SaaS.

It's fine to have a dashboard that holds all that stuff. But it shouldn't be reaching out and telling you/users about it. You can go look at that dashboard when investigating why the service KPIs you're alerting on are down.

In your example, the appropriate service KPI to alert on would be "[service] 99th percentile end-to-end response latency." Upon receiving an alert that that number is out-of-tolerance, you'd then start to investigate your individual — or, hopefully, aggregated! — server metrics, to see what-if-anything is acting weird.

nix232• 5 years ago

"Monitoring" usually implies having a defined acceptance threshold on the metric, and triggering some (probably human) response when the metric is in an unacceptable range.

Like the Smart data on your Hard-drives? The increasingly (over month) thrown away packets of a dying Network card? Or that one Fan that increasingly spins slower then all others?

Look Hard is not different from Software, both need to be Monitored having a close look on "trends" and want to be Maintained.

nix232• 5 years ago

Full hit! If one think in services, the Hardware is the first service one should observe, just thinking about the running Software ON IT as a service is ridiculous. Good luck "debugging" your performance problems if your Hardware is struggling to keep the Raid up, trowing packets away a hardware level...or hey, running at 600Mzh because the CPU is boiling, two fan's reported errors, hell even fans are services (by providing airflow)

Observe a Service; Not a Server

Related Stories