SLO alerting for mortals

https://ervinbarta.com/2021/10/19/slo-alerting-for-mortals/

by ervinb • 5 years ago

66 10 5 years ago

ervinbarta.com

denysvitali• 5 years ago

Very informative and clear article. Thanks for sharing!

phkx2• 5 years ago

Is there a common framework how to come up with the values (e.g. 99.9%) set as SLOs in the first place?

I currently want to do that for a service built on several cloud services with their respective SLAs and my approach is to go through the combined probabilities to get an effective error rate. That‘s a bottom-up approach. I‘d combine that we a top-down derivation of what SLOs are required from the business side. If the first number doesn’t fulfill the business requirements with some buffer, we‘ll need to redesign. How do others do it?

LeonB2• 5 years ago

The SLO should be tied to customer satisfaction as much as possible. I think you need to collect a bunch of metrics (potential SLIs) for a period of time and then say — in which windows were things bad enough that they caused real issues? And of the metrics we gathered which ones actually show the issue? Then use those metrics as your SLIs and set the SLOs such that they reflect that reality.

phkx2• 5 years ago

I like the practicality of analyzing where the customer had pain and adjusting SLOs accordingly. Our system is not open to customers yet, so we‘re lacking historic data with real load (besides system tests, but they currently don‘t include edge cases/chaos). Also, we‘ll probably need to have SLAs from the start, which would be derived from the SLOs, so I need something beforehand.

—-

SLA: service level agreement, values of KPIs promised to customers

SLO: service level objective, internal target values for those KPIs, typically slightly more demanding than the SLA

SLI: service level indicator, measured values of the KPIs to check against SLOs/SLAs

Raed667• 5 years ago

We have been experimenting with using Web-Vitals [0] as SLOs. The idea is that they represent the closest user satisfaction.

If anyone else has come up with better metrics I'd love to hear about your feedback.

[0] https://web.dev/vitals/

Jensson• 5 years ago

Step 1: Measure you current uptime. That is your baseline SLO, to not make things worse.

Step 2: Evaluate how much money you would earn by increasing uptime, and evaluate how much the processes required to do that would cost. Incrementally raise SLO levels as long as you benefit from it.

kwhitefoot• 5 years ago

SLO = Service level objective

bhelkey• 5 years ago

Time wise, a 0.1% error rate for a month means a 43 minute complete downtime.

This article assumes relatively constant usage. If a service is used heavily in a single time zone this number could be considerably lower.

The flip side of that is that it might not make sense to alert during night hours of that time zone. You wouldn't want to get paged by a single user spaming retry on an error page.

SLO alerting for mortals

Related Stories