Breaking Up with On-Call

https://reflector.dev/articles/breaking-up-with-on-call/

by 0xlosh • 1 year ago

152 197 1 year ago

caseyohara2• 1 year ago

Not to distract from the article, but I'm pretty sure that photo of the guard tower is from Manzanar, one of the concentration camps in California that interned Japanese Americans during WWII. Probably not in great taste to use that photo to represent the idea of "guard duty" in software.

https://en.wikipedia.org/wiki/Manzanar

idiotsecant• 1 year ago

Eh, intent matters. They probably just googled 'guard tower' and used the first one that looked close enough.

caseyohara2• 1 year ago

That’s my assumption and why I said something. I’m guessing they didn’t know.

strken3• 1 year ago

Apart from anything else, a fire tower is a better metaphor for on-call than a guard tower, too.

brigandish3• 1 year ago

Who, specifically, would that picture offend or change protect?

Xylakant3• 1 year ago

People that have been interned in this or one of the other camps, or their descendants. It’s one generation ago, people born and raised in camps are still alive. George Takeo for example was born in one of the segregation camps.

It’s a low stakes change.

Nobody assumes harmful intentions from the author - I would not have recognized the picture either. But now that it’s been pointed out that it’s from a site where people were held illegally against their will, the reaction is a tell-tale. Knowing this, and insisting on keeping the images is now willfully associating with harmful behavior.

Apart from that, I cannot associate with the picture either - as an on-call engineer for some widely used infrastructure, I am not a guard on duty keeping people in a camp. I am an emergency responder. I fix things when they go haywire or an accident happens. A firefighter, paramedics, civil emergency responder would IMO be a much better metaphor for what I do.

pokoleo• 1 year ago

From my experience working on SaaS, and improving ops at large organizations, I've seen that "on-call culture" often exists inversely proportional to incentive alignment.

When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and planning, you get perpetual firefighting.

The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated.

Big companies aren't missing the resources to fix this; they just don't have the aligned incentive structures that make fixing it rational for individuals involved.

The most rational thing to do as an individual on a bad rotation: quit or transfer.

tacticus• 1 year ago

IMO it's when the incident response and readiness practice imposes a direct backpressure on feature delivery that you get the issues actually fixed and a resilient system.

if it's just the engineer while product and management see no real cost then people burn out and leave.

The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated.

100%

hinkley• 1 year ago

The people who do the real work don’t get raises and promotions because the annual review system punishes them for doing the right thing.

rednafi3• 1 year ago

When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes.

Making engineers handle 3 AM issues caused by their code is one thing, but making them bear the financial consequences is another. That’s how you create a blame-game culture where everyone is afraid to deploy at the end of the day or touch anything they don’t fully understand.

cyberax• 1 year ago

"Financial consequences" probably mean "the success of the startup, so your options won't be worth less than the toilet paper", rather than "you'll pay for the downtime out of your salary".

DanHulton• 1 year ago

This assumes that the engineers in question get to choose how to allot their time, and are _allowed_ to spend time to add graceful failure modes. I cannot tell you how many stories I have heard of, and companies I have directly worked at, where this power is not granted to engineers, and they are instead directed to "stop working on technical debt, we'll make time to come back to that later". Of course, time is never found later, and the 3am pages continue because the people who DO choose how time is allocated are not the ones waking up at 3am to fix problems.

nijave4• 1 year ago

Definitely an issue but I think there's a little room for push back. Work done outside normal working hours is automatically the highest priority, by definition. It's helpful to remind people of that.

If it's important enough to deserve a page, it's top priority work. The reverse is also true (if a page isn't top priority, disable the paging alert and stick it on a dashboard or periodic checklist)

adrianN• 1 year ago

Minimizing 3am pages is good for engineers but it is not necessarily the best investment for the company. Beyond a certain scale it is probably not a good investment to try to get rid of all pages.

mook• 1 year ago

By that point wouldn't it start to make sense to have people across time zones so that it will be working hours somewhere?

northern-lights2• 1 year ago

This assumes that the engineers who wrote the code that caused the 3 AM pages will still be around to suffer the consequences of the 3 AM pages. This is a lot of times, not true, especially in an environment which fostered moving around internally every now and then. Happens in at least one of the FAANGs.

meeshmuesh• 1 year ago

Sounds like Amazon

mrkeen• 1 year ago

Engineers don't pick their work, management does.

A manager no longer needs to choose between system reliability and churning out new features with on-call:

The manager can get all the credit for pushing out new features during the day, and sleep well at night knowing that the engineers aren't.

procaryote3• 1 year ago

At a lot of companies engineers are involved in picking the work. It's silly to hire competent problem solvers and treat them as unskilled workers needing micro-management.

Besides, if you set the on-call system up so people get free time the following day to compensate for waking up at night, the manager can't pretend there's no cost.

Bad management will fail on both of these of course, but there's no saving that beyond finding a better company.

stackskipton2• 1 year ago

I have no idea what article was trying to convey since on call was poorly defined

I also couldn’t take it seriously when article opened with this.

Startups cannot afford engineers to baby sit software, big tech does.

Say what? As ops person, I’ve seen multiple startups where devs are drowning in operational issues because software was written enough for feature MVP, ship with poor testing then constantly poke at with sticks to keep it working in hopes they could get enough revenue to not flame out.

apeescape2• 1 year ago

Well, did those startups succeed? If not, I think it means the author was spot on in saying startups can't _afford_ that.

stackskipton2• 1 year ago

One did, two did not. None of due to engineering.

Engineering rarely matters unfortunately but ability to weave a story and sell that story.

jiveturkey• 1 year ago

I have no idea what article was trying to convey

An ad for their consultancy. It's a call to action, not a balanced analysis.

danpalmer• 1 year ago

As a current SRE, and having worked in a small startup, this doesn't echo my experience at all. What the author describes is possible what we would call "on duty" work, the grunt/maintenance work that comes with big software systems. It's not fun, and most companies/teams have friction getting this sort of work done. It's also however not how my SRE role is defined by any stretch. Our on-call work is much more about support during exceptional, somewhat rare circumstances.

gpi• 1 year ago

Agreed. Author's point of view seemed to have been influenced by their bad experience being part of the "on duty" work.

strken3• 1 year ago

Yeah, I was similarly confused. My experience has been that on-call is a roster of who will keep their laptop with them on the weekend. Incidents involve little to no customer comms, because if anything more than a sentence once an hour is necessary then there's someone else on call to handle it.

Support work is a necessary nuisance, but it's also not what on-call is meant to be.

coolgoose• 1 year ago

Sounds like normal support work not sure why that's affecting morale.

It's normal in any kind of system to also cover issues.

Yes, at some point you might want to have a customer support / customer success later to at least triage them, but that makes more sense as you get bigger not when you are small.

I actually like having discussions on support days with customer. Yes, sometimes they're more annoying but it's direct feedback of people trying to use your software.

cdnthrownawy392• 1 year ago

My on call experience required that I had to be able to respond within 10 ten minutes of the call, with 24/7/365 coverage. But if I couldn't get the issue resolved remotely it meant that I'd have to be in the office lab to recreate and reproduce the problem. It effectively restricted my movements personal movements to stay within commute distance of my office, and that includes all my vacation time as well.

That was the better part of a year in my life, continuous. Of constantly considering that every decision, every meal, every movement, every action at all times, and weighing it against the risk of impacting my ability to respond to a customer call. Maintaining an extended period of alertness for a threat that very rarely materializes is frustrating in many ways that I'd like very much to forget.

I didn't burn out from it, but it was a major factor in my decision to resign from that company. Obviously people out there that can handle this lifestyle, but I couldn't. And frankly I'm quite content to never try again.

dmoy2• 1 year ago

Was author's big tech experience with Amazon?

Because Amazon oncall is ... well it's something. But I'm not sure it's really indicative of the rest of big tech oncall.

northern-lights2• 1 year ago

The author's Linkedin indeed lists working at AWS for 3 years.

dmoy2• 1 year ago

Ok that explains a lot about this article

rednafi3• 1 year ago

A moderate on-call ritual is a necessary evil. I’ve worked at places that tried to get rid of it with all kinds of automation and playbooks, only to revert back to PagerDuty a few months later.

That said, my last workplace completely burned me out with a terrible on-call policy and an absurdly short recovery period. Not to mention, upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.

I get why it’s necessary, but I’m lucky enough to be in a position where I can flat-out decline any job that mandates unpaid on-call.

lolinder• 1 year ago

upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.

If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations. If you knew about that going in then you weighed it against the offered salary and decided it was worth it. It's not "unpaid on-call" in that case, it's paid on-call.

Obviously it can also be done poorly in ways that are harmful and dishonest. Not communicating it clearly during the job interview and salary negotiation, applying it inconsistently, or changing the frequency or the difficulty of the rotation after you have started are all real problems.

rednafi3• 1 year ago

I agree with your expansion. In this particular case, the on-call policy was enforced after the hiring.

AdieuToLogic7• 1 year ago

> upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.
If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations.

Where this logic breaks down is when on-call expectations are in addition to what a salaried position compensates - a 40-ish hour work week.

Expecting an employee to be paid industry rate for services rendered and then expecting periodic 24-168 hour near real-time availability without compensation is, by definition, "unpaid on-call".

genocidicbunny• 1 year ago

For me personally, there's no possible way that a company can correctly factor on-call costs into my salary, because the on-call costs are based on what I am doing at the time.

You'd have to pay me a biblically large multiplier on my base salary to get me to pick work over my kids or wife. My kids aren't going to remember that I made a few thousand bucks by ducking out halfway through an important-to-them event. My wife is sure as shit ain't going to forgive me _or you_ if you call me into work in the middle of some adult time. And the cost for you to make sure that I'm not participating in such activities and am available to work on very short notice is...well, it's the exact same number as if you had interrupted such activities.

If you really need the coverage in the middle of the night, hire more engineers whose normal 8-ish hour shifts are during the night? Can't afford to? Perhaps you're not so good at business then. Better yet, stop building a fucking house of cards that can topple your company if it wobbles a little bit. If your systems going down for a few hours can take down your company, outside of some exceptional circumstances, that's a failure on your part.

bigstrat2003• 1 year ago

Not to mention, upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.

Every position I've held (save my current one), that is most definitely the norm. If you have a busy night managers are cool with you coming in late the next day (or potentially not at all), but it's very unusual to be paid for on call in my experience.

spike021• 1 year ago

Fortunately every job I've held had the situation where the manager was fine with me/the team taking make-up time the following day if you were interrupted during offhours. Sure extra comp would be nice, but if on-call has to be done, this isn't the worst way.

tbrownaw• 1 year ago

tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.

It is part of normal work for roles where it's a thing, and I'd think it could be either baked in or on top as long as which one it was was known at salary negotiation time.

fzeroracer• 1 year ago

I don't think on-call is a necessary evil; I think it's the result of managers and leaders not caring about having multiple failsafes instead opting to foist that problem onto engineers via unpaid labor.

You can have a system with enough redundancy, ability to rollback and deployment scheduling where any sort of on-call incident is highly rare and low impact. But that requires spending time and money on solving those problems which is money that could be spent on developing more features faster. A buggy and broken product doesn't matter as long as you can shovel out More faster.

I've been on both types of teams and I'm at the point in my career that if I'm going to be on-call I'm expecting to be compensated appropriately with actual on-call hours worked. And part of the problem is that even on a team where you're informed of on-call duties and rotation, if the team gets cut or people leave then you're on the hook for working longer hours for less pay. It's inherently exploitative.

andrewaylett2• 1 year ago

We get time off in lieu for hours worked, and paid (at a fixed rate across the company) for hours where we need to be available to be paged. You don't want to only get paid if you're paged, especially if your services are normally quite reliable.

Fail-safes are one thing, but they don't always kick in when we need them. My team is only paged one or two times a month, and when we are it's probably for something we've not seen before because whatever it was that broke last time has been fixed.

chgs3• 1 year ago

My company has operated a 24/7 shift rota for 100 years. American “On call” is not normal.

Some companies might have exceptional on call - the C suite will get called back from holidays or whatever should things really hit the fan. That’s once every few years.

snapetom• 1 year ago

I'll add to this some companies (my current one) still has a hero culture. People that are forced to work all weekend to fix a problem are lauded as the ones who went the extra mile. An ego boost and kudos, for some reason are reward enough. Companies don't have substantially reward these employees nor does the company have to invest in paying technical debt, infrastructure, or automation/monitoring. It's incentivisation gone horribly wrong.

smcameron• 1 year ago

Been a few years since I worked at Google as an SRE, but I did not find "There’s no incentive in big tech to write software with no bugs" to be particularly true. Perhaps because I was an SRE for a few of the of the older, intrinsic, core products. A lot of the pages we got were for things outside our control (e.g. some fiber optic cables got broken (we found out later), so we've got to drain some cluster because nothing's getting through there, or something's getting overloaded because it's the superbowl today, so throw more machines/memory at it, or other weird external things.) I don't remember a lot of pages due to outright bugs in the code... though it's been enough years now that I might have forgotten.

nijave4• 1 year ago

Were the systems designed to scale based on load and handle transient failures?

It seems like lack of automated remediation would be a bug unless it's an "accepted business risk" i.e. cheaper to throw people at to manually fix than build a software solution.

Xylakant3• 1 year ago

You can’t plan for all failure modes. Weird shit happens and it needs human intervention to figure out what went wrong. Sometimes someone needs to assess what path forward is the lowest (financial) harm and weigh the options. No computer should make that call.

VirusNewbie2• 1 year ago

If a bug is causing multiple pages, SRE will absolutely either force the devs to fix it, or fix the bug themselves.

AdieuToLogic7• 1 year ago

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die.

First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case.

Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system. This is obviously not the case.

Third, and finally, a system failure addressed by someone "at 3am" assumes said person can rectify the problem in real time and without any coworker collaboration nor stakeholder input.

The last is most important:

  If a resolution requires a system change, what organization
  would allow one person at 3am to make it and push to prod?

hn_user82179• 1 year ago

great points. I think over the 8 years of my SRE experience, I've probably caused a few outages after being paged at 3-4am and/or prolonged them. I've fallen asleep during one responding time (fortunately the main outage had been fixed but we went way beyond our internal SLO for backups as I had fallen asleep before running them).

That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org

AdieuToLogic7• 1 year ago

That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well

I completely agree.

The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly.

Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.

whstl5• 1 year ago

Great points.

If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?

The only advantage of having a micromanaging manager: if they want to review all PRs and system changes, they also must be on call every single night of the year.

procaryote3• 1 year ago

If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call

If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.

I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

Pain from a problem needs to live where the problem can be fixed.

inetknght7• 1 year ago

If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.

If that's the case, then the person responding at 03:00 needs to have a serious conversation with the dev, or with whoever's assigning/prioritizing tickets to the dev.

AdieuToLogic7• 1 year ago

I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot.

Pain from a problem needs to live where the problem can be fixed.

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

markus_zhang• 1 year ago

Well during on-call I think usually there is a chain of wake-ups. The on-call guy tries to fix the issue but finds out he is not familiar with it so creates a channel and wakes up the next guy in the right team, and so on. The first guy usually just wakes up to "acknowledge" that someone is taking care of the issue, and writes the wrap-ups and closes the channel.

rob74• 1 year ago

If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?

Well yeah, if you have on call, I guess you have to trust the on-call person to make that decision. If they get called at 3am, the problem is obviously so bad that almost anything they do can only make it better, right? If it's some overlooked edge case that crashes the whole application and can be fixed with a one line change, it's probably Ok to do it a 3am. If it's more complicated, maybe wait until 7am and discuss it with others too. And, in any case, a thorough post mortem (during office hours) and checking if the 3am fix is really a proper fix for the issue is in order...

jjice2• 1 year ago

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years.

While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure).

So this may end up being a cultural difference between on-call situations.

AdieuToLogic7• 1 year ago

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe.

While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure).

My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support.

If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same.

Just my $0.02

asdf6969• 1 year ago

In my experience we patch the hole in the tire and then get directed to patch a hole in another tire. Manager says work on the highest priority tire so now there are 2 partially broken tires but the problem is technically fixed. Repeat for 10 years and now it’s time to rebuild it “correctly”, but actually we just have two cars to patch since nobody knows what awful hacks we need to port over for “expected” behaviour

jyounker3• 1 year ago

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations.

When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.

AdieuToLogic7• 1 year ago

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

As I mentioned in a peer response:

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

chilldsgn• 1 year ago

Filing for a new feature implementation would require a thorough documentation, rightfully so, followed by a political campaign to convince the political party of principal engineers and managers to accept the new feature. These stakeholders carry incentives and principals of their own - that do not necessarily always align with the true engineering spirit of solving the problem, nor with satisfying the customer.
The same friction applied to fixing a bug or a flawed process. I would reproduce the bug: spin up the entire environment, the appropriate binary artifact and the reproduced state of application, create the test cases and pin-point the exact problem for the stakeholders as well as present the possible solutions. I would get sent to a dozen of meetings, bouncing my ideas back and forth until receiving the dire verdict - rejection to fix this bug altogether.

This has been my experience for the entire duration of my tenure at the current place I am employed at. I've stopped doing this, because my backlog is filled with "best effort" features and when I attempt to slot some of these into a quiet sprint, management says no.

Many features I request is not from assumption, but data I gathered with analytics for the customer-facing website. One example is to change a page layout and add better search functionality for this page and its data, because I noticed a +70% no-results rate in the search analytics. I suggested a change but marketing shot it down saying it's low priority for them, while they frequently say that the website can perform better at generating leads. I might be wrong, but to me it feels like utter short-sightedness and goes against the strategic goal of the company.

I'm just an engineer, what do I know?

aorloff• 1 year ago

First rule of on-call : if your team has it, then you either take shifts during work hours or are being paid a metric shitload.

VirusNewbie2• 1 year ago

At google, SRE pays quite well and you don’t have more than 12 hour shifts.

EZ-E• 1 year ago

Or your job pays extra for on-call shifts outside work hours

brikym• 1 year ago

I work for a big tech and your solution has already been built. The on-call software uses an LLM to summarize the tickets, and a vector db to find troubleshooting guides and similar tickets.

jldugger• 1 year ago

No offense, but this is wildly underselling the goals of oncall SRE. LLMs are extremely crappy about causal analysis, or even just mitigation techniques for services that haven't been widely discussed on stackoverflow.com (i.e. your service).

Creating an on-call process to manually inspect errors in test suites is more valuable than improving the project to be more reliable, as you can directly measure the amount of tests that failed on a weekly basis. It is measurable and presentable to the upper management.

You can also measure requests that failed on a weekly basis, and I do. In fact, I added a dashboard panel to do exactly that today for a service (10 years old!!) on a new team I just reorg'd into. I did this because I was annoyed to discover the first (internal) customer outage report of the day could have been repaired by the east coast half of the team team hours before the west coast QA team logged in for the day, but they were unaware anything was wrong. This is a trivial promQL query to implement, and yet it wasn't until today.

The problem isn't visibility but risk -- what if you make reliability fixes but the data gets worse? This is not hypothetical, a Youtube engineer documented a similar tale[1]. You can also imagine all kinds of fixes that sound good on paper but can produce paradoxical outcomes (i.e. adding retries causes a metastable failure state[2]). And heck, what if you make no changes, and the numbers decline all on their own? Are you going to scuttle this quarters project work (and promotion fodder!) just to bring this KPI back to normal? Of course, all numbers, even the test suite pass rate, come with risks of missing targets, so the incentives are to commit to reporting as few of them as possible.

tools automate the mundane tasks of an on-call engineer: searching for issues related to a customer report, tracking related software (or hardware) crashes, verifying if the current issue that arose during an on-call is a regression or a known bug and so on.

I have a coworker trying to use LLMs for ticket triage, but there's a huge GIGO risk here. Very few people correctly fill in ticket metadata, and even among the more diligent set there will be disagreement. Try an experiment: pick 10 random tickets, and route copies to two of your most diligent ticket workers. Then see how closely their metadata agrees. Is it P1 or P3? Is the bug reported against the puppet repo or the LB repo? Is a config change feature work, bug fix, or testing? Do they dupe known issues, and if to, to the same ticket, or do they just close it as a NTBF known issue? If these two can't agree on basics, then your fine tuning is essentially just additional entropy. Worse, you can't even really measure quality without this messy dataset, and the correct answers should change over time as the software and network architecture evolves.

[1] https://blog.chriszacharias.com/page-weight-matters [2]: https://www.usenix.org/publications/loginonline/metastable-f...

rohgpt• 1 year ago

I used to be on-call too. I still have muscle memory of the buzz that’d make me leap off my couch like my house was on fire. Except it wasn’t. Just prod, doing prod things.

We all know it. You trade your peace of mind, sleep, and sometimes personal freedom (hello, “don’t go beyond cell signal” life). And I agree with many of you—it shouldn’t be this way.

We’ve seen teams turn their on-call rotations into something way more humane. Not perfect (nothing in ops is, let’s be real), but sustainable.

At Zenduty, we work with a lot of SREs, DevOps folks, and incident commanders who’ve said, “We can’t keep doing this.” So they started rethinking on-call. One of our partners cut their alert volume by 50%. Half! That’s not a magic trick. Just better processes, better tooling, and some good old-fashioned empathy for the humans behind the screens.

And if your team’s not giving you breathing room after an all-nighter? You deserve better. I’ll just say it.

I saw some of you mention how this stuff becomes political or gets deprioritized in favor of shiny new features. Ugh. I feel that in my bones. The best teams we’ve worked with treat incidents as opportunities to make things better. Every alert is basically your system saying “Help me, I’m tired.” And when you listen and fix it? Less 3 AM wake-ups. Win-win.

Anyway, I didn’t mean to ramble, but this topic is close to home. If you’re still stuck in on-call hell, or just want to vent about it. Always happy to swap war stories or chat about how folks are making on-call suck less.

Stay sane out there, Rohan DevRel @ Zenduty

chgs3• 1 year ago

Why don’t you have shift work?

dakiol• 1 year ago

For me the worst thing about being on-call is not the actual work outside business hours (it’s usually not much), but the potential work: if something happens I need to jump into my laptop within X minutes (changes from company to company, but it’s usually within 10 minutes). This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip). Basically, all I can do is stay at home and be available. It sucks, and the money is not worth it.

rwmj3• 1 year ago

This is why people used to be paid time-and-a-half or even double-time for being on call. Ask your union to demand that.

https://en.wikipedia.org/wiki/Time-and-a-half

chgs3• 1 year ago

Last time I was on call it was a one hour payment per day, a 4 hour payment for answering the phone.

That lasted about 3 months, just not acceptable.

liveoneggs• 1 year ago

For nurses on-call is a tiny amount - $3/hr and then you get 1.5x if you actually get called in

zild3d• 1 year ago

So for say a weekly on-call rotation you would be paid all 24 hours x 7 days at double rate? (- ~40)

Also most tech companies don't have unions...

smitelli2• 1 year ago

Fun fact: In the US the concept of time-and-a-half (also minimum wage, and not hiring child labor) was created by the Fair Labor Standards Act. Most tech employees are classified as "exempt" -- the FLSA and its protections doesn't apply to them.

gosub1005• 1 year ago

If I had a union it would demand a bunch of unqualified people join my team (and get paid the same as me) and it would forbid me from doing certain things because,say, moving the computer or plugging in a cable is IT's job, whereas I'm SE. No thanks

kazinator• 1 year ago

Are you confusing overtime and on-call?

Nobody's going to pay you 1.5 your hourly rate to just sit and wait until something happens. Is that really a thing?

Now if you are called and spring into action, there may be time-and-a-half, if it's outside of your regular hours.

ekimekim2• 1 year ago

When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation.

In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager).

The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that's what the secondary role is for - to catch you. This means it's perfectly acceptable to go for a run, go to a movie, etc.

You relax the responsibility on the individual and let a sensible amount of redundancy solve the problem instead. Everyone is less stressed, and sure you get the occasional 5min delay in response but I'm willing to bet that the overall MTTR is lower since people are well rested and happier to be on call to begin with.

WhyIsItAlwaysHN• 1 year ago

So it takes 10 min until you've gone to the drastic solution? With this time-frame it would be risky to go the bathroom, not go to a movie. Also even the backup sounds like a primary in this scenario.

smitelli2• 1 year ago

Having the primary/secondary rotation is arguably worse. In that model, from the perspective of any one participant, now they're on-call for two weeks each time around instead of one.

notnaut2• 1 year ago

I don’t see how this changes the problem where there is an expected guarantee of a rapid response except that now two people are expected to be available and would now need to directly coordinate in order to ensure one person’s going for a swim doesn’t interfere with the other’s WoW raid.

hylaride2• 1 year ago

This is pretty much how it should be done. If the business demands more, they should have a properly manned 24x7 NOC.

You also need *ownership*. There is nothing worse than having to support somebody else's work and not being allowed (either via time or other restrictions) to do things "right" so that you're not always paged for fixable problems. Everywhere I worked where the techs had ownership (which varied from OPS people being allowed to override the backlog to fix issues or developers being given enough free reign to fix technical debt) has usually meant that oncall is barely an issue. My current gig I often forget I'm even on call at all and the main issues that do crop up are usually external.

andrewaylett2• 1 year ago

We want to ack within five minutes, and be at a laptop within 30. So long as I'm within mobile signal when the page goes off, it doesn't really matter what I'm doing — an ack is a button press on a push notification. And I can stay within 30 minutes of my laptop and an Internet connection by carrying said laptop and my phone (with "unlimited" data).

If the primary (paid) on-call doesn't catch the notification, the secondary (unpaid) will be paged. And so on, down a couple more steps, to a senior manager. There's no expectation that anyone other than the primary would actually be available to ack the alert.

Anon1096• 1 year ago

We have a primary/backup setup and I would be pretty pissed if my primary just started going out for movies or a date night during their shift tbh. My job as a backup is to be there for unexpected events, ie they did not wake up or had an accident. Not be on call effectively 2 weeks in a row just because the primary doesn't take it seriously.

inetknght7• 1 year ago

The expectation is that most of the time you should be able to respond within 5 minutes

That's an unreasonable expectation unless it's clearly said in writing and is billable hours.

spiffytech• 1 year ago

Any particular reason you can't handle incidents while out and about?

I know it varies by situation. When I've been on call I've been able to mostly go about my life. I just had to keep my laptop close, stay in cell signal, and accept I would sometimes have interruptions (typically brief). We fought to keep them infrequent enough that they didn't ruin our lives.

cdnthrownawy392• 1 year ago

Not the GP, but I was in a similar situation. It was a requirement to be able to get to the office if the situation required the lab to diagnose the problem.

dkdbejwi3832• 1 year ago

I do long(ish) distance running as a hobby - it's not feasible to take a laptop out on a two hour run.

If I want to go meet a friend for a drink or food, I have to lug around a backpack, keep an eye on it to make sure it's not stolen. If I wanted to have a beer or wine, I can't because I may need to work at any point.

Favourite band is performing? I suppose you could take a backpack and the laptop to the venue, but again there's a chance it's pinched, and they'll make you check it at the cloakroom for the performance.

yodsanklai• 1 year ago

10 min doesn't seem tenable over 24/7. Most likely you need to run errands and so on. In my team, our alerts aren't that critical, I just acknowledge on the phone and make sure I can get to the computer with 30 mn. I take it with me if needed.

No, for me, the real pain with oncall is that there are a lot of systems in my team. I understand well maybe 30% of it. I'm clueless about 30%. In between for the rest. I can try to fix issues myself (take long time, issues can add up), triage (but means bothering someone else). There are also things than nobody understand, and if they break, it can mean extra days of work for you with extra-stress because you don't know how hard it can be to fix.

Then, some people in the team ship code without adequate testing, because of pressure to ship. Which often adds work to the oncall. So there's all this extra tensions with colleagues which can be hard to deal with for an introvert.

Overall, it "kind of" works for us, but I agree with the conclusion, it sucks. It's really the worst part of my job. I went into software engineering because I like coding. Not because I liked monitor unreliable systems. And I think unreliability is encouraged by management to some extent. That keeps people at work.

ZaoLahma3• 1 year ago

This is exactly why I gave up a position as a full stack / devops engineer in favor of going back to low level drivers - there were too many unknowns, and far too many unknown unknowns often paired with expectations of prompt (and cheap) solutions to complicated issues.

Technically it was interesting and challenging, but in terms of stress just not worth it. You could pay me twice my current salary and I still would not go back to it. Now I try to place myself as far away from paying customers as technically possible.

re-thc• 1 year ago

all I can do is stay at home and be available

If home is fine then usually all you need is Internet and laptop.

I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip)

Sounds more like setting expectations and explaining the situation than a "cannot" (maybe except movies).

You can explain to your family for example that you're on-call and may need to leave urgently. I mean e.g. police do that. It's not that uncommon.

You can take runs within a 10 minute distance back home. The route is up to you. You can start by acknowledging on the phone as someone else commented, which would grant you maybe another few minutes.

There are lots of options. It's on you to workaround it. On-call isn't perfect, sure.

dkdbejwi3832• 1 year ago

If you're on-call, you're working - it doesn't matter if there's an active incident or not. Unless you're a contractor (in which case, you're unlikely to be on-call) the company you work for pays for your time, not delivery of specific work-items. On-call pay should reflect this.

jobs_throwaway3• 1 year ago

99% of people on-call are salaried anyways

srhtftw• 1 year ago

I've voiced my opinion on this before¹.

The problem is on-call is an essential and critical part of a managerial role, but toxic to those in a developer role.

Managers must be on-call to ensure the appropriate people and resources are brought to bear on unexpected problems that threaten the business.

Developers must NOT be on-call to ensure appropriate attention is spent designing, developing and maintaining the code that makes the business possible.

The rise of software-as-a-service led to companies promoting "devops" engineering which conflates these roles and unfortunately helps unscrupulous executives unfairly squeeze more work from employees.

The core idea of devops, that managers/operators and developers should understand and be capable of performing each other's role, isn't a bad one. Those who understand how the business works at all levels can do more to make it successful. It goes hand-in-hand with continuous delivery.

The best engineers alternate between these roles in a predictable schedule. When in the managerial role they need to observe, react, delegate and escalate problems as appropriate. When in the development role they need to deliver features that create recurring value for the business.

But businesses should not expect engineers to play both roles at the same time!

This form of "on-call" is a toxic moral hazard. It's a sign of instabilty. It's a signal of executive grift looking for a quick pop. "on-call" robs developers of attention they need to develop the features and increases risks that schedules will slip.

It doesn't need to be this way. If a business needs software development it should hire or train engineers with that experience. Likewise if it needs managers or operators to deliver software as a service.

As an operator or manager I look forward to working a shift, but as a developer I will never again accept on-call rotation.

¹ https://news.ycombinator.com/item?id=42230215

h1fra• 1 year ago

If you allow yourself to sleep then it's not entirely true. I totally agree that it's lurking and you think twice before starting an activity, but what's the worst that can happen? Run 10 minutes away from your house, take your laptop to the movies, and to your friends and family. I do it all the time and yes sometimes I need to isolate or find a place to work, I still enjoy the rest of the day

inetknght7• 1 year ago

This means: I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip).

I live in rural Texas. The same things apply here, and more: I'm lucky to have good internet (which enables working remotely) but half my home doesn't get cell coverage so being responsive to a text message or phone call means not even going around my own home (for example, no cell signal in the kitchen means I can't cook while on-call); and with large tracts of land, I can't go out to do land maintenance (good luck hearing a phone ring or feeling it vibrate from a call when you're operating heavy machinery, assuming you even have cell signal there); all services are 15 minutes or more away: groceries, doctor, contractors, government, etc etc.

It's important to stress how much being on-call ruins my capability to use the time effectively for my own purposes (Texas Guidebook for Employers [0]; 29 CFR 785.16 [2] and 785.17 [3]). I tried telling this to a previous employer when they started wanting me to be on-call (3+ years after start of employment), and they indicated that those laws are only used for hourly employees but being salary + exempt means I do not qualify for additional pay and falls under "and other duties as assigned" in the employment contract. So the employer effectively started getting 60 hours of work for 40 hours of pay. Oof.

I also absolutely refuse to mix my personal devices with work; just at a minimum, I refuse to make my personal device available to legal discovery related to any legal issues with the employer. So if the employer wanted me to have cell phone availability, then I demanded that the employer provide that cell phone. That was a fun conversation that ended with some relaxed requirements (eg, I don't have to have cell phone availability if I'm responsive at my work desk already) which further reinforced the fact that I couldn't use the time for my own purposes.

Thankfully multiple years in this industry at (what was) fair compensation allows me to be picky for new employment contracts. And lesson learned: I'll be a lot more careful about contract language from now on, and specifically look for (or negotiate) carve-outs around being on-call and work/personal device separation. I recognize that having 10+ years of experience makes me able to handle that, but newcomers to the industry won't yet have that buffer and it sucks for them to not have that safety net for negotiation leverage.

A lot of this disagreement comes from businesses demanding rapid response while insisting on not taking on new hardware/payment obligations. To contrast: take the fireman who's waiting for an alarm (29 CFR 785.15 [1]): they are often often idle and can often go out for groceries but they're easily reachable. Ever seen a firetruck in front of a grocery store and the firemen are just inside shopping for groceries? Then see them come running out and turn on the lights & siren and drive off? I have. It's an interesting event, and it sucks for the grocery store that has to put those groceries (for ~15 people) back on the shelves and refrigerators. Nonetheless, those firemen are paid to do so and have special equipment (eg radios or cell phones) to be able to receive those messages, and the firement generally don't pay for that equipment themselves (the community does either through taxes or donations). I see analogies about on-call software engineers being called to put out (virtual) fires as very apt in this case.

[0] https://efte.twc.texas.gov/c_waiting_or_on_call_time.html

[1] https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

[2] https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

[3] https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...

wnc3141• 1 year ago

The airlines pay a certain rate when you go on call - since your'e still effectively on company time.

therealfiona• 1 year ago

I mean, I'm salary, so I get paid for on-call as part of that, but I don't get any extra pay for picking up my team's on-call rotation.

ianmcgowan• 1 year ago

In California, as a non-exempt employee (basically not a manager if you're at a big company), you'd have to be paid for that on-call time with those requirements. The key term is "restricted" and the 10 minute expectation is quite a severe restriction.

https://www.dir.ca.gov/dlse/callbackandstandbytime.pdf https://www.shrm.org/topics-tools/tools/policies/california-...

This doesn't apply to a lot of people reading, but just a PSA for those in CA where it might.

steveBK123• 1 year ago

I once had a job with a lot of 2-4am wakeup outage calls. The timing was perfect such that you can't fall back asleep generally.

An aggravatingly large percent of them could be resolved by voice over the phone by walking the offshore support team through the same 2-3 runbook items.

"Did you look at the log... I see, OK are you looking at it now? Does it say X? Did you do Y? Good now? Great, goodnight."

"Did you try restarting it.. ok then try that now. Is it good now that you restarted it? Great, I'm going back to sleep"

Ironically we'd have less of these outage calls when the offshore person went on holiday because they'd send one of the competent NY support staff over for 2 weeks. Slept like a log every time.

prmoustache• 1 year ago

All companies I have worked for wanted you to answer within minutes, but you had half an hour to actually connect and try fixing the issue, so you could totally go out and if you were in a larger than a 30min drive/ride/walk you would just keep a laptop in a vehicle or backpack.

I used to do 3hours rides with my bicycle and go to dinner or social events with a gpd pocket 2 in a small bag.

jjtang1• 1 year ago

Most of the software I wrote requires mostly no interference or fix-ups, unless of course the requirements have changed

Bug-free software is great, but changing requirements are precisely the reason on-call needs to exist.

Even if you have a whole team of engineers who write bug-free software like this guy, you'll still have failures. Because the world is constantly slipping out from under your assumptions.

Customers never stop changing their usage patterns. They add load at different rates, come up with unexpected requests of all shapes and sizes, and invent new use cases that fly in the face of the original project requirements.

Even if you have created a software system with no bugs that perfectly meets both the functional and non-functional requirements of the project, changes in the state of the world vis a vis customer behavior will come along and change what counts as a bug. If your system has a blanket 60-second database query timeout, and everything's working fine, then there's no bug. But as soon as a new API usage pattern causes certain queries to run on average 10 times longer than before, now you have connection starvation and an urgent bug to fix.

I'm not saying that "timely maintenance and improvement" and "a culture of perpetual ownership" won't have positive effects on reliability. But it's unrealistic that any amount of responsible, careful software development will fully eliminate the occurrence of sudden and unexpected failures. Human on-call, as uncomfortable as it is, will remain a requirement as long as reliability is taken seriously.

FWIW my perspective is that of someone that runs an on-call/incident management platform (Rootly).

bobbiechen• 1 year ago

I agree. The potential work is worse than the actual work. I used to think it wasn't so bad, but then I was so relieved when I left the on-call rotation that I must have been suppressing my feelings about it.

I wrote about my experience here: https://bobbiechen.com/blog/2022/7/20/being-on-call-sucks

wvh• 1 year ago

This. And alerts are often just a fluke anyway. Sure many of us can step up and pull an all-nighter if it saves some company and you make some minor sacrifice to do something heroic. Being woken up several times at night for no reason but some metric that is a bit off is pretty soul crushing, and then come the knock-on effects in professional and personal life that you're always tired and demotivated during the day.

kermatt• 1 year ago

A 10 minute response is "On the clock", not "on-call". A symptom of the "Do more with less" cancer in tech.

steelegbr• 1 year ago

I was on-call for over a decade, usually in roles where there was no compensation for working out of hours other than maybe TOIL. We're not talking FAANG gigs here - like £20-50k in the UK stuff. It's amazing how much having to carry an extra phone or making sure your laptop is in your car impacts your day-to-day life. Any social thing you're at could be interrupted at zero notice. Heck, I've taken calls in supermarkets and concert venues.

One place I worked had a 1 in 2 rotation. Every other week on call or weeks back to back if your colleague was on holiday. There was no front-line service screening calls which meant you could be woken several times in one night. All for £30 pcm towards broadband costs.

Most places are more sane than that example but suffer from the same core problem. Follow the sun support is incredibly expensive when compared to putting your existing staff to be on call. Here in the UK, so long as your equivalent hourly rate doesn't drop below national minimum wage and you're opted out of the working time directive (a lot of employers slip an opt-out form into your paperwork implying it's normal to sign it), then it's legal.

Unfortunately I'm yet to find anywhere that on-call operational teams have the clout to get code induced issues high up the priority list outside of cases where they've had to drag developers out of bed at 2am. In my experience that also plays out with getting anything infrastructure based into tech debt budgets. Why focus on fixing problems you don't directly suffer from when you can spend the time on a refactor, integrating a cool new library or spaffing out one more feature in the sprint?

xyst• 1 year ago

Question to tech folks in general: when you are "on-call" are you provided with differentials in pay regardless of whether an incident actually occurs?

I have seen it vary across companies:

- Some companies don’t pay any differential

- Some companies only pay out _per incident_

- One company I worked with paid out per shift of on-call rotation

- One company treated it as overtime pay

inetknght7• 1 year ago

Some companies don’t pay any differential

This is quite scummy unless your role's primary duties are support and nothing else. I would count Site Reliability Engineer as support being a primary duty, while I would not count Software Engineer as support being a primary duty.

Some companies only pay out _per incident_

That's a bad incentive. It incentivices more incidents to occur.

One company I worked with paid out per shift of on-call rotation

Sounds decent on paper but everyone has their breaking point!

One company treated it as overtime pay

Sounds decent on paper but everyone has their breaking point!

zombot• 1 year ago

I must have missed the "breaking up" part. Anyway, the article is just a really long ad for the author.

eweise• 1 year ago

I'm pretty confident the idea of on-call would go away if the company had to pay for your time. But being salaried, they assume you work for them 24/7.

inetknght7• 1 year ago

Yup, that was my experience as described in another comment:

https://news.ycombinator.com/item?id=43401441

AStonesThrow2• 1 year ago

I was at the tender age of 21, and a mere contractor at my first job. I recall being paid hourly, but never filling a time card.

I had just been promoted to a systems admin role, after demonstrating prowess with coding and system configuration, and also the actual qualified admin had quit. One of my projects involved a rewrite of server code for efficiency as our /etc/passwd balooned to thousands of users, and too many logging in simultaneously.

So on Thanksgiving Day, we just got in the door with Mom after shopping, and my supervisor rings the home phone; his first words are "There's a hacker on [production shell server]". And I told him a few commands to try, and we did not seem to suffer downtime or need our backups.

That may have been the most consequential job I had for a decade. There were thousands of customers depending on us giving "four nines" of available service, including the corporate customers, but we had no formal on-call agreement, not even a way for me to access systems from outside the office, and I was still a novice after a few college courses and a lot of unstructured hacking.

My [salaried] father, on the other hand, often brought a pager [which never seemed to beep] home, and one day we were all sitting around the living room when an explosion rattled our windows. We were still a bit rattled ourselves when the phone rang, and it was his cue to go down to the site and respond as a health & safety officer.

So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!"

inetknght7• 1 year ago

So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!"

So if you had family that needed your immediate help, they would not receive it?

That's indeed a perverse incentive.

AStonesThrow2• 1 year ago

I'm unsure how this got flipped turned upside-down, but this was Mom asserting boundaries to Dad's work-life balance.

The central controversy of on-call duty is that it leaves the household and family short-staffed, contributing to burnout and "workaholism". We were accustomed to Dad being at work M-F, 9-5, and being available and supporting us at all other times.

While a boiler explosion is certainly exceptional circumstances, it can be the "camel's nose in the tent" and eventually, Dad is potentially driving up there every Saturday night that a lab worker spilled 1 gram of mercury. Then he's not eating dinner with us, resting, mowing the lawn, helping Mom to shop or do errands.

I'm trying to recall any times Dad responded to a family emergency by leaving work, but it was nearly never necessary, since preparedness and response plans defined his career.

MisterBastahrd• 1 year ago

Here's my answer to on-call requests: no. And if you ask me to clarify my statement, I will answer: no. Because no is good enough, and it's what I negotiated when I took the job.

I don't work for employers who can't manage their staff properly. If you're on-call at a company with any sort of scale, you aren't being managed properly. One of the things I tell hiring managers is that I don't mind working later if we're reaching the end of a project and a late afternoon or two might get the product pushed out of the door sooner, or if I've gotten into the zone and am making reat progress. But these are my choices to make. There will never ever be a time when I will subject myself to being tied to a phone for the purposes of waking up in the middle of the night and messing with servers or software. My time is MY time. I only ever have reciprocal relationships with employers and refuse to upset that balance.

mertleee• 1 year ago

Just hire an SRE. If you want to piss off your team and incur more costs than the annual salary of one decent SRE... ask your team to do on-call rotations.

In this day and age if you're a properly capitalized startup you should have the budget for one decent SRE hire. They often times double as a technical writer and are great at preventing knowledge silos in teams / members of a small team since they have to sort of know everything to be an effective SRE.

Best process improvement I ever made for my teams.

SpicyLemonZest3• 1 year ago

I don't follow how a single SRE hire would be able to replace an entire team's oncall rotation.

0xbadcafebee• 1 year ago

Typical HN blogspam. A blog post that makes vague but incredibly certain claims, generalizing about an entire industry, with no evidence outside a single person's limited experience. Yet there he is at the end of his post, asking people to hire him for consulting. But it's perfect for HN, because it gives people in the comments something to complain about, driving up engagement.

I will admit, though, the SaaS tech industry as a whole has a problem. The cargo cult of tech has convinced itself that SaaS is special. That it's never done. That it must be never finished, never stable, always changing, always breaking. Like it's some immutable law of nature.

This of course is a convenient lie. Before every business was an online service, software worked like every other product in the world. You built it to do a thing, you tested it thoroughly to ensure it didn't have defects, and then you coordinated one large release. No constant changes or late Friday deploys. No schema changes destroying columns. There was no on-call, because there was no 24/7 service. There were floppy disks and CDROMs, and people running software on their own computers, and a very small number of 24/7 enterprise systems shared by lots of users (mostly run by ISPs, and one or two large tech companies and industry bodies).

But you can deliver online software like it's desktop software. You can ensure your software is bug-free before you burn your master disk. But then you don't get to skip all those time-consuming tests. Then you don't get to ship your features faster than your competitor. Then you don't get to do A/B testing and micro-tweaks and experiments and feature flags and blue/green deploys. We want to do all the things, at any time, with no consequences. So on-call is still a thing, because people want to have their cake and eat it too.

The actual infrastructure on the backend shared by millions of users? Not actually hard to maintain. It's the same shit as 20 years ago, but much, much easier. Make sure the disks don't fill up. Auto-scale the VMs and containers. Design your apps to not exceed network bandwidth. There's really not much else to break. The only thing that breaks a system is changing it, or bugs from not enough testing. So do the testing, and the bare minimum of autoscaling in the cloud, and nothing should ever break in production.

But constantly mutating SaaS, without due diligence, is too addictive. Nobody's going to abandon the freedom of doing whatever the fuck they want, just to keep from waking people up in the middle of the night. The business people sure as fuck aren't going to abandon their "competitive advantage". And devs who don't care if their software works or not don't want to abandon untested deploys from their laptops. So on-call is here to stay.

jyounker3• 1 year ago

The author's description of how on-call works at "big tech" seems to be a complaint about a specific tech company, and the author is hastily generalizing to the rest of the industry.

His experience has nothing in common with my on-call experience working at a different "big tech" company.

a_imho• 1 year ago

IANAL, but my understanding is that in Europe on-call can fall under working time, depending on the exact nature of the requirements. In fact, I'm very interested whether anyone has firsthand experience arguing developer on-call falls under such category. E.g. being 8 minutes away from a work machine and network access is not too dissimilar to a firefighter being 8 minutes away from the fire station.

https://www.eurofound.europa.eu/en/european-industrial-relat...

staplung• 1 year ago

It seems that companies often end up with On-Call after they've reached some critical size but decide to do it in the cheapest possible way: let's just make our engineers do it!

One big problem is that they tend not to make any attempt to ensure that the on-call rotations are equitable across teams. You might need to be on call one week out of 4 or one week out of 20. At review time, this matters. Since you're typically not getting paid for the on call work or having it factor into your performance review, you're actually incentivized to snooze alerts and kick the can down the road until after your rotation or to join another team with lower rates of on call incidents.

Ironically, if you're spending 25% or your time on call and actually being diligent about the role, you're more likely to be adept with the typically hodgepodge collection of tools that you need to use to diagnose and address problems and will be better able to handle incidents than if you only ever use those tools once every 5 months.

F-W-M• 1 year ago

I am on-call for a week every ~8 weeks. I get 500 € and two additional days of vacation for it. If I do get a call, the additional hours go to my time account.

Fair deal, I'd say. Only problem is that most of the applications are from vendors, so getting problems fixed is an ordeal.

vismit2000• 1 year ago

Getting 404, archived version: https://web.archive.org/web/20250318031523/https://reflector...

Breaking Up with On-Call

Related Stories