Building an in-house on-call training program for your SRE team

https://fiberplane.com/blog/how-to-build-an-in-house-on-call-training-program

by LenkaBo • 3 years ago

60 24 3 years ago

An on-call training program is essential to building and maintaining a successful incident response practice and culture.

I think that employee training in general is essential to the overall efficiency and quality of your business. Companies always skimp on it, and you can see how it hurts productivity and the bottom line. But it's not necessarily going to impact your culture or day to day practices.

The thing that's going to maintain a successful incident response practice is follow-through. Training just tells them what they're supposed to do. But if they're not doing it, training doesn't matter.

What's the most common problem with on-call? It's not waking people up in the middle of the night, or a lack of runbooks, or lack of postmortems. It's the follow-through. When you have an incident, and you "deal with it", and even have a postmortem, what's the last thing on the page? "Action Items" or "Next Steps". And are those action items actually getting done? Is a regular review of your practices getting done? Is the same alert popping up again and again? That is the biggest problem with on-call, and training doesn't solve that.

To run a good incident response process you need people who are getting paid to do two things: 1) due diligence, 2) continuous improvement.

1) Give someone the primary responsibility of ensuring everyone is following the process. Do not let action items stagnate. Do not let weeks go by without writing the postmortem. Do not have alerts without resolutions, or runbooks for that matter. Somebody needs to get paid to do this, because it's boring as hell, not fun, but needs to be done.

2) Make sure the process is getting easier, and better. You don't want to have to check up on everything all the time, of course, and it doesn't scale well. So you want to improve your process so that it's more likely that the right things get done the first time.

You can train your folks what to do. And you can "get by" with an incident response process that nobody actually follows and is never improved. But as incidents mount, it will drain your people and org more and more. By all means, if you're currently facing a deluge of incidents, do training! But the due diligence and continuous improvement is what's actually gonna push incident numbers down.

ElevenLathe• 3 years ago

Can't agree more. I've worked in incident management for years and my superiors are always happy when we have an incident but one of our incident managers and a few SREs solve it relatively quickly. Yes, there is some talk of "how do we avoid something like this in the future?" but the answer (even when using "5 Whys") is always a shallow one like adding even more gates and change review or, best-case, adding active-active failover. The answer is never "This team ships once every other month. We need to give them a mandate to move that up to multiple times a day."

Still, it's nice to have SREs who understand the rhythms of an active incident instead of somebody whose last conversation with someone outside their immediate team was in 2016 and seemingly doesn't understand that they're on call or for what services.

blue0392• 3 years ago

So many words and none of them include “pay people overtime for working on call”.

dopylitty• 3 years ago

Exactly this. On call is an abusive practice. If you think your service should be up 24/7 then pay people to be online to take care of it 24/7.

Forcing people to sit by their phones/computers 24/7 unpaid waiting for a system to maybe have a problem should be a practice explicitly barred by the FLSA.

olvy0• 3 years ago

Completely agree. My team has on-call duty, but we don't have to do it 24/7, just during business hours. I hope we never get to the point where we're giving support 24/7.

Slightly off topic: stdout's On-Call.

https://www.youtube.com/watch?v=OiApm-OFfsY

makestuff• 3 years ago

Yeah IMO you either use a follow-the-sun rotation and have staff in multiple parts of the world, hire a dedicated nightshift, or pay OT.

If the SRE is unable to solve the issue and needs to page the escalation engineer they should get OT for that if it outside business hours. The whole "oh you got woken up at 3am just log in late" is BS because I still have the same amount of work to do.

dangerwill• 3 years ago

Yeah I went from a job where being oncall meant a LOT of pages but the oncall rota was a follow the sun model and so the oncall shifts just ended up being normal work days where you were just 100% interrupted constantly. I thought this was hell but now that I work at a company with 72 hour long oncall shifts, even though the after-hours page rate is very low (once a month or so), the "always on" feeling is even worse.

I am constantly thinking about how abusive this practice is

MacsHeadroom• 3 years ago

At a unicorn I was paid 1.5x for every on-call hour, even while sleeping. That makes 40 hours in a week and 128 on-call hours worth 1.5x for a total of: 232 paid hours in one week with on-call.

That's almost 6 weeks of pay in a single week. Yet we still had to pull teeth to get people to be on-call because of how much it sucks.

tayo42• 3 years ago

Yeah, There's diminishing returns on happiness derived from money. the extra few thousand when Im making 12k a month already doesn't offset it.

sambull• 3 years ago

Took my company being bought by a european one to get that to happen.

On-call for many companies (most I've worked at in the past 23 years) is a borderline abusive practice; and often times they already run the team on fumes.

LenkaBo• 3 years ago

You're right. And that's the most crucial point!

Ancalagon• 3 years ago

I think the position "Site Reliability Engineer" has become synonymous with people being perpetually on-call as their job, at least in the U.S. The job is basically just there to put out fires.

jedberg2• 3 years ago

I did on-call for work for 20 years. I just assumed my paycheck needed to cover that hassle. I usually got paid more than other people with similar experience because my job included on-call and theirs didn't.

Overtime isn't necessary if expectations are set up front and the salary is boosted accordingly.

But yeah, it makes people feel better if you pay them less overall and then give them overtime when they're on-call. It's also better for the company because then the raises are smaller since they're based on the base salary and not the overtime.

Honestly I'd rather just get paid a higher salary with expected on-call hours up-front.

blue0392• 3 years ago

It’s not about “making them feel better”. It’s about wage theft from workers.

If you pay me N extra hours, where N is less than the hours I worked on call then you stole from me. Plain and simple. This should be pursued by rabid labor lawyers and interest at the prevailing 30 year rate should accrue on those dollars hourly until they are paid back.

The problem is, of course, the law dictating “exempt workers” was written in a time where the only people that would be exempt were executives with actual million dollars golden parachutes. Now, white collar labor is almost universally exempt allowing the state sanctioned theft of millions of dollars a year by corporations who lobby to keep the laws the same, and promoted by “grind culture” immoral bottom-barrel blood-suckers like PagerDuty.

It’s bullshit. You’re suffering from Stockholm syndrome. You’ve somehow managed to say stealing from you is okay because they put some nominal amount of unrelated dollars in front of you.

bennyp101• 3 years ago

First step should be to ensure you have enough people that you can do an on-call schedule without burning out your handful of employees, second should be that you tailor your notifications so they don't get overwhelmed by "outages" that can actually be dealt with by other teams or left until the workday, thirdly make sure you pay people for doing so - so many places just assume that you will be about.

JTbane• 3 years ago

Knowing corporate bean counters, step 1 and 3 will be unattainable due to "budget constraints".

magicloop• 3 years ago

When I started out one of my first jobs came with an on-call pager. It was a puck shaped item that you carried on your belt. When sitting on the toilet it would slide out of its carrier and fall into the basin. Nearly happened twice to me (I caught it). Several others had it fall in. It had to be vigorously cleaned before hand over to the next engineer. That is what is called a "Clean Handover".

cody_coconuts• 3 years ago

So this is what they were talking about when they said "shitty team culture"

netik• 3 years ago

Advertisement disguised as a useful post. Like a lot of marketing, it ends with “use our product”

shahargl• 3 years ago

I'm trying to build OSS tool that will help fighting on-call fatigue so if you hate it let's talk and tell me how can I help

https://calendly.com/d/zkv-4f8-jgj/keep-intro

hummus_bae• 3 years ago

Good luck! That's an incredibly tough problem to solve.

squalo• 3 years ago

SRE and DevOps people are already struggling to automate build and run for products and infrastructure they don't know shit about. Now you also want them to work 24/7 to support it when it flops?

colonCapitalDee• 3 years ago

My team runs incident drills as part of our training. The person running the drill (the instructor) breaks something in our pre-production environment. This could be pushing a buggy PR, changing some configuration, taking a node offline, etc. The instructor and the learner then hop in a video call. The learner shares their screen, then works through the incident with help from the instructor. They acknowledge the incident in our management tool, use logs and metrics to track it down, take steps to mitigate it, then mark the incident as mitigated in our management tool.

This builds skills and confidence for the learner, and has the additional benefit of exposing pain points in our tooling.

Building an in-house on-call training program for your SRE team

Related Stories