Ask HN: How often are you on call?

https://news.ycombinator.com/item?id=21881003

by buttman2077 • 7 years ago

29 41 7 years ago

news.ycombinator.com

Please share a few words on what size company you work for, and what size team you’re apart of. If you’d like to share, mention how often you’re paged or activated!

beginrescueend• 7 years ago

One week out of two, for a company size of 20.

cptn_brittish• 7 years ago

Company the size of about 10.

I am the only person on call 24/7.

markyang12• 7 years ago

once every 5 weeks, FAANG

acranox• 7 years ago

It depends on how many people we have on the team! Right now it’s every three weeks. A while ago it was every 6 weeks.

I used to get paged dozens of times a week. Now it’s either once a week, or a few dozen times a week (usually this is a blip that can just be ignored because something went wrong and then fixed itself). Typically there’s one small actionable item every week I’m on call.

The company is 500 people, my team varies in size a lot as people quit and we try and hire replacements. It’s been as small as 3 and as big as 12.

arboroia• 7 years ago

I’d also be interested if people could add if being on call is (a) voluntary, or (b) required on their team.

Also, it’d be good to know if on call comes with additional compensation?

mikelward• 7 years ago

Large cross-site team. We do 12x4/12x3 rotations (Mon-Thu, Fri-Sun) so everyone stays fresh.

I've heard others are trying one-day rotations to make scheduling holidays fairer and more flexible for people who have commitments on certain days.

caiobegotti• 7 years ago

Small startup, tech team of about 10 (2 SREs + 7 devs + 1 biz, total company size is 20, fully remote distributed across 3 continents), roughly 8 hours twice a week and only during my local timezone business hours, then the same 8 hours in one weekend day every two weeks, also roughly. We have lots of automated checks and alerts and only react to critical ones paging us via Slack (most of the alerts that page us are known, some are flaky some are false positives we can't ignore).

It was kind of forced on everybody (prematurely IMHO) but respecting people's timezones and business hours was key to make it acceptable and you can take the next day off no-questions-asked if you were on call on a weekend day even if you didn't have to act on anything. It's far from ideal considering how we still react to the critical alerts and what in them usually triggers actual on-call support but I've had worse in the past so it's alright... for now.

esotericn• 7 years ago

Currently contracting for a ~10 person company.

In the past I've mostly worked at small companies and have never officially been on call. I've fixed the odd out of hours issue here and there without there being a proper arrangement.

The systems I work on (trading / betting software) are such that by the time you're through testing it's very rare to hit a game breaking issue because there's a very large incentive to not like, lose the bank.

Dependencies in the critical path are kept to a minimum, etc.

Usually problems are more 'soft', like an API being turned off or changed or something.

Proziam• 7 years ago

I'm on call 24/7. It's fairly normal to wake up at 2 am because there's a pressing situation and a client is in need. That said, I actively chose this situation for myself and find it to be pretty rewarding. Of course, it requires some planning and coordination, so my team divvied up our responsibilities accordingly to ensure that there are always resources available for taking meetings on a normal schedule.

It's probably worth noting, we work with large brands and marketing agencies from all around the world and are often called upon to 'save the day' once someone's plans have fallen through. As a result, time is usually the one thing our clients don't have enough of and being able to solve their problems ASAP is what allows our clients to get back to normal operation. Given that, it's easy for me to pick up the phone at 2 am because I know that whoever is calling me is going to be having a worse time than I am.

probinso• 7 years ago

Company size: 2 programmers

only when the other guy is on vacation

6700417• 7 years ago

Large company. Team of 12. Do very specialized work.

On call: Never and never have been in 25+ years across multiple large tech companies.

Seriously considered looking into a job at Facebook where this would have been required and it was definitely one of the considerations that made that position very unappealing.

perl4ever• 7 years ago

I just started working for the government, and I got yelled at for sending an email on Saturday. The union contract says we're paid for overtime if we work more than 40 hours, but in turn, we never are supposed to work overtime unless maybe the world is ending.

I used to work for a company that had a departmental phone and we would rotate to a different person each week to be on call. However, waking people was essentially eliminated by adding people on third shift (including me) and in India.

In both cases, I was on a team of about 5-10 in an agency or division of a couple or three hundred, within a much larger organization.

dmitrygr3• 7 years ago

Never, and I don't consider any job that demands this. Learn what work-life separation means.

DarthGhandi• 7 years ago

I'm at the complete opposite of the spectrum there, on-call 24/7/365. Then again partially own the company and enjoy my colleagues. If I go dark for a few days answering non-urgent queries no one questions it because they know I wake up at 3am and put out digital fires within short timeframes. Life and balance are just words in the end, you can clock off at exactly 5pm to the nanosecond and still not find that holy grail ever. It is what you make it.

geofft2• 7 years ago

That's clearly not a sustainable answer for the world - professions ranging from doctors to priests to soldier to sailor to CEO (can't ignore a crisis if it happens on Saturday morning) to lawmaker all require working the occasional unscheduled hour. And that's not even counting jobs like pilot or convenience store clerk or paramedic that require at least some people to work on what the world considers holidays (even if it's scheduled as part of their 40 hours a week).

Is there a meaning of "work-life separation" that encompasses these professions too, or shall we say the world relies on some people not having it?

dmitrygr3• 7 years ago

or shall we say the world relies on some people not having [...]?

It does rely on that. Supply and demand works that way. Eventually the offered price is enough to change some minds. I'll change mine at about $3k/hour-on-call.

lostdog• 7 years ago

If someone is dying, go ahead and page me!

Otherwise, companies need to hire night shifts if they want someone covering while the devs are asleep. If you want full uptime, then hire for it! Your employees shouldn't give up their sleep and weekends for a measly 0.003% of your company.

Also, I would strongly support extra pay for low-wage workers working holidays, weekends, or night shifts.

Viliam1234• 7 years ago

Working N hours in unusual schedule is different from working N hours in usual schedule and then being on call.

With the former, you get some compensation, for example you may work on Saturday but have a free Monday. Depending on your priorities, it may actually be a good deal -- there are things you can do more conveniently when you have a free Monday.

With the latter, it's "when you're at work, you're at work, and when you're at home, there is still a tiny voice in your head remininding you that your boss could call you at literally any moment". This may limit the ways you spend your free time, for example you can't go to places where it would be impossible or very inconvenient to get quickly online when called. And generally, you just can't make promises even about how you spend your free time.

For a sufficiently large company, it should be possible to find a few people willing to take an unusual schedule, and then the company could be covered 24/7, with every employee able to turn off their work phone the moment they leave. It's just... more convenient for the managers to do it the other way round.

sushid2• 7 years ago

Seems unreasonable to say never. Like what a sibling commenter said, it seems like you’ll miss out on potential earnings in your career by being this inflexible.

dmitrygr3• 7 years ago

Just like I miss out on some earnings by not being a contract killer. Some things aren't worth it. That is a personal choice. And this is mine.

astura• 7 years ago

Never ever.

Current company has 400 employees my team is around seven but we'll have to grow very fast very shortly.

Karupan• 7 years ago

I feel I’m in the sweet spot. Work for a company based across 5 time zones. Our team of 12 has optional out of hours support and people have to volunteer. But it is only for Sev 1 issues where the entire app/service is down.

Since it is paid, we end up taking turns for the extra money. In the year I’ve worked there, haven’t been pinged even once.

whatthesmack• 7 years ago

tl;dr One week out of four.

I’m an SRE on a team of 4 (incl manager). The company is about 140 people. Each of my team members participates in on-call (incl manager) and is on for a week at a time. We’ve dialed things in such that we might get paged once or twice during that week and usually the pages wind up being minor/non-issues. It’s really not bad, in my opinion.

vanniv• 7 years ago

Presently, one week in five. Average shift has one overnight page and one other out of hours page (with high variance).

I am compensated for the oncall, but not a meaningful amount.

Sevii• 7 years ago

Work for FAANG, team size is 6-8~. I'm oncall 24/7 roughly one week out of 6. Usually paged 1-2 times per shift except during 'peak' periods but we usually subdivide the rotation for those.

My worst shift I was paged during off hours twice, but was only woken one of those times.

stephbu• 7 years ago

My tier 3 team is on 1-week point-Dev rotations. Roughly 1wk in 9 on primary, same again on secondary. They take about 2-3 calls per primary+secondary. I’m on Tier 4 in mgmt roster, and on call every other week. Take escalations and fall thrus every 3-4mths. Tier 4 also plays duty manager during events such as product launches.

geofft2• 7 years ago

Largeish company (engineering staff around 1000, I think?), and my team owns the OS platform (packaging, config management), hardware purchasing, and the installer, as well as our private cloud, which hosts a good chunk of services (notably including dev machines, notably excluding the most money-making parts of the business). A separate team is first-line sysadmin+SRE for most of the company, so we don't respond unless you can't install machines or if every machine is broken, or if the private cloud (as infrastructure, not a specific VM on it) is acting up.

Private cloud was originally a separate team, and we had about 5 people on a weekly rotation, where you'd do support and 24/7 on-call for a week. Now that we've merged, it's about a dozen people on the on-call rotation, though we still have a subteam of about 5 for the weekly rotation of who's paying attention to private cloud support tickets during working hours.

Thanks to some work that we did a bit after I joined, we got the off-hours paging to be much less frequent - in particular we no longer page if any hypervisor is down, only if either a whole availability zone is down, or a hypervisor hosting personal dev machines is down and it's business hours. (Otherwise, higher-level services ought to have failover between multiple VMs, and devs working odd hours can use backup machines.) Usually that second condition gets noticed by the support person (in Jira) before it escalates to the on-call person. So it's rare to have anyone actually paged, and it usually means something unexpected is on fire. So, any member of the team is as good as any other at figuring out what broke and whether it can wait until morning, which is why we were able to throw all of the combined team into the rotation without training them all in everything.

There are still improvements I want to do, but the expected number of pages rounds to zero (it looks like we've had 6 off-hours pages this year) so it's certainly manageable.

theonemind• 7 years ago

Technically 24/7, except for clustered stuff that hardly goes down. More than a year since the last time; I think. Thousands of employees, team of 7 or so

sidcool• 7 years ago

Quite a lot. 30% of the day on average.

pyr0hu• 7 years ago

Small bootstrappes company working on SaaS. Im the lead and has 2 programmers on the team, so Im the one usually on call, 24/7.

gvurrdon• 7 years ago

Large university, team of 8. All out-of-hours work is on a “best effort” basis only, i.e. I’ll do what I can (which may be nothing) if emailed.

codegeek• 7 years ago

I run a bootstrapped SAAS business with total team size of 12. I am technically always on call if shit hits the fan. Not in the traditional sense though. If server goes down (has happened like 2 times in past 5 years), I get a call no matter what time of the night it is.

antoinealb• 7 years ago

SRE at Google. We are oncall for 12h then the other site takes over for the other 12h, so no night shifts. You do that for 7 days in a row once every 5 week-ish. Some team have slight variations on that.

We get paged around once per 12h on average but our service is fairly young. More mature services tend to page less I think.

wikibob2• 7 years ago

What is the on-call compensation?

Last I knew it was 33.3% time credited for on call hours with 30 minute response requirement and 66.6% time credited for on call hours with 5 minute response time. And it could be taken as additional holiday or as cash.

twunde• 7 years ago

I work at a 100-person startup. We currently do 1 week secondary followed by 1 week primary. This comes up every 2.5 months. On-call is 8am-8pm with any overnight pages going to execs. I'll not that we originally had people doing daily shifts and it was a huge pain for those who were religiously observant. It was much harder to get someone to cover a Saturday or Sunday shift since nobody wants to cover those. Will send much better

xchaotic• 7 years ago

do you have any staff in other timezones to get better night coverage?

cm2012• 7 years ago

I own a marketing agency. I check my email often just on case but I rarely have to hop on my laptop after hours. I'll probably pen an after hours short email a few times a week though.

Ask HN: How often are you on call?

Related Stories