Goddamn AWS seems to be down again 4th time this month alone

https://downdetector.in/status/aws-amazon-web-services/

by freakynit • 5 years ago

261 125 5 years ago

downdetector.in

altdataseller• 5 years ago

It’s Christmas Eve, I aint answering any PagerDuty notification today or tomorrow.

gumby• 5 years ago

If pager duty uses aws you won’t have a problem

pmarreck• 5 years ago

DNSception!

tyingq5• 5 years ago

You could just borrow the AWS wording.

Say that you're answering the on-call pager, but with "elevated latency".

Krisjohn• 5 years ago

Yoink

Shorn2• 5 years ago

"My standard Service Level Agreement for on-call is an average incident response time of < 24 hours (including weekends). Leave requirements may see periods of elevated response times."

In other words, I'll start to work on your problem at 0800 local time the next work day.

mushufasa• 5 years ago

what the hell this is unacceptable. is this some sort of sabotage? AWS may have to live this down for years in RFPs

voidfunc• 5 years ago

Wonder how much of this is caused by engineers rushing to fix log4j issues at AWS so they can enjoy the holidays? Their stack is heavily Java-based isnt it?

joebiden2024• 5 years ago

Unclear on this one, but wasn’t the last one caused by power failure in an AZ?

wlonkly• 5 years ago

Log4j on the transfer switch, then!

gjsman-10002• 5 years ago

Amazon’s SLAs are quickly becoming only useful for kindling.

At least your fireplace will give some warmth for your relatives. Merry Christmas!

gowthamgts12• 5 years ago

looks like it's a connectivity outage across all services - https://www.thousandeyes.com/outages/

darkwater• 5 years ago

Nice link, didn't know it. Thanks!

theossuary• 5 years ago

Complete speculation; but I wonder if how Amazon works their employee's so hard is the root cause of this. Layer on COVID as an additional stressor and give it a few years and this could be the result. Curious to hear from those at Amazon, what's it been like during COVID?

valleyjo• 5 years ago

Attrition has been super high during Covid and they raised the pay a lot for new offers to try and attract talent. (source: I got an offer for S3 mid-Nov. this is what the hiring manager told me).

dv_dt• 5 years ago

The rule of thumb of efficiency is that it can make things worse when it breaks. How that applies to organizations is if you have an incredible org with a higher plates to plate spinner ratio than ever, then if you have a catastrophe, the org may not have enough people to actually get the plates spinning again as fast as a less efficient, larger crew with more reserve.

loeg2• 5 years ago

They're losing engineers left and right, especially in H2 2021. It's probably all related. Their recruiters' emails are increasingly desperate. If I was considering a job at Amazon now, I would be very interested in a good answer for how they plan to address their attrition problem (and paying new hires well isn't a viable answer alone).

ridaj2• 5 years ago

Seems like every company in tech is losing engineers left and right at this point

smolder2• 5 years ago

All they need to do to hire me is lay off the draconian IP assignment clauses or give me an explicit exception for side projects. That's what I tell the desperate recruiters, and then they go away until the next one comes along.

joshgachnang• 5 years ago

They have been getting desperate, much more so than my quarterly Facebook/Google email. I've also had multiple Amazon recruiters reaching out, whereas I usually have a single Facebook/Google one. But apparently they aren't desperate enough to allow remote work after Covid, so I keep reminding them I'm not interested in non-remote work.

efitz• 5 years ago

AWS was always stressful. Great people for the most part. Culture had strong pros and strong cons. Same for treatment of employees; mixed bag.

During COVID everyone who didn’t need to be in the office worked from home. That was sometimes more stressful as you had lots more time in video meetings and no hallway interactions to make quick decisions on easy stuff.

For the problem reports today I am in wait and see mode. It’s not clear to me yet that AWS is the problem; could be a six-pack-and-a-backhoe kind of problem with a network provider.

nickjj• 5 years ago

Complete speculation; but I wonder if...

If we're going into pure speculation mode I think we might also want to include that maybe their outages are staged and have factored in that the temporary bad press and paying out SLA credits is more profitable because it guides existing customers to start scaling to multiple AWS regions to offset these infrequent downtime events in 1 specific region at a time.

For example if you have hosting on US West but now to help reduce time down you duplicate your infrastructure in the EU region then I'm pretty sure the outcome here is if you were paying $6,000 a month before now you would be paying $12,000 a month (+/- any regional cost differences).

Realistically I don't believe this is the case but it's not impossible.

jen20• 5 years ago

Likely more because of internet-region data transit costs.

tomcooks• 5 years ago

Happens when you delegate, don't give them money.

forty• 5 years ago

I got worried when I saw this then checked https://status.aws.amazon.com/ and all seems good, I'm relieved ;)

gjsman-10002• 5 years ago

I hope you are kidding because Amazon has been very late with updating that page in the past despite problems being obvious.

darwinwhy• 5 years ago

That's the joke.

roozbeh18• 5 years ago

no fun if you have to explain it haha

mkl95• 5 years ago

Some poor soul is in charge of updating that page manually.

rosndo2• 5 years ago

For once the AWS status page is actually telling the truth, there is no outage.

tyingq5• 5 years ago

I would be curious to hear what any current or former AWS employees think the internal consequences become for this.

From the outside, it feels like they are going to have to do something different to get some customer confidence back. Some sort of "mea culpa" with an explanation of what they are going to change.

oonis• 5 years ago

It’ll be a lot of finger pointing and teams covering their own ass

tyingq5• 5 years ago

That seems to be the status quo. I'm curious if the current trend is enough to break that tradition.

sgt• 5 years ago

Or maybe breaking stuff is celebrated? For every type of downtime there will be lessons learned.

manojlds• 5 years ago

Jeff Bezos comes back /s

awsthrow8264• 5 years ago

Can't speak for other teams, but the dynamics in mine are pretty great. We don't overwork and leadership puts explicit emphasis on work/life quality.

From what I've heard in other teams (including some that own the big services you're familiar with), it can be a shitshow. Oncalls get paged 10*N times a week (typically 24/7 2-week rotation with two active oncalls, but depends), teams are leaking talent, desperately trying to hire good engineers while keeping the lights on.

gregjor6• 5 years ago

Apologies are not actionable. Service providers promise some number of 9s percent uptime, not zero problems. It should be obvious that perfection is not achievable, and that at scale small problems can get magnified.

Customer confidence is only an issue if there are alternative providers that don’t have problems. That isn’t the case with cloud hosting.

tyingq5• 5 years ago

I'm unclear on what you're responding to. I wasn't saying an apology was the right remedy, nor did I suggest they should have zero issues.

Customer confidence is an issue because they've had 4 notable issues in a very short timeframe.

Edit: I think you're reading something into my comments that's more than what I've said. I am curious what the AWS response to 4 major incidents in a month's time will be. That's it. At least in my circles, it is an issue for customers. I respect that it appears not to be in your circles.

Ahh. By "response", I did not mean the incident summary. I meant the overarching company response, if any. Like policies around change control, capacity additions, and so on.

inaprovaline• 5 years ago

It takes years to build trust with your customers, and very little to disrupt it. I think these recent events will have people rethink their cloud strategy. This could be a good opportunity for Google to take on Amazon.

moonchrome• 5 years ago

I'm not usually the person to complain about UI performance but Google cloud admin is the slowest trash I had to use in a while, even Azure is better. So here's me hoping it doesn't pick up steam.

monkeybutton• 5 years ago

The GCP admin console is painfully slow and yet is still somehow faster than the new Google Ads UI.

motoboi• 5 years ago

Azure CLI is absolutely insanely bad.

Google CLI is great.

Aws is trying to get better.

GUI on the three sites is awful.

remote-dev• 5 years ago

It has been awhile since I used the GCP admin, but I am intimately familiar with how awful the Azure UI is... is it really better than GCP? I have never liked how Azure seems to be designed as if you're in a virtual machine or something. The horizontal scrolling is a mess.

Alcha2• 5 years ago

The company I work for switched from Azure to GCP at the start of the year and personally I don't agree with this statement. Azure was a royal pain in the ass compared to GCP, which I find to be rather smooth and responsive.

Maybe you're going off of past experiences and it's since gotten better?

tata714• 5 years ago

Or for offline-friendly webapps to flourish?

spmurrayzzz• 5 years ago

All you have to do is go back to the outage/event timeline for GCP to know exactly how ill-informed of a decision that would be. (Not to mention all the other product offering differences)

noodlesUK• 5 years ago

This is getting to be a massive joke. Any reputation AWS has for being robust has been wiped out over the past month or so. I feel bad for the various engineers at AWS and other companies who are having to work on their day off.

yjftsjthsd-h• 5 years ago

Goddamn AWS seems to be down again 4th time this month alone

I feel obligated to point out that this is a very editorialized title, which is against site guidelines. Not... that I don't sympathize, just that it's perhaps a bit out of line even for providing context.

nefitty3• 5 years ago

You felt obligated to the benefit of who, exactly?

rootusrootus• 5 years ago

The entire HN community, presumably.

Ostrogodsky• 5 years ago

AWS

collegeburner• 5 years ago

True, but it's funny... perhaps next time a "Tell HN" with the link?

300bps• 5 years ago

I'm honestly sick of seeing these posts. Every single post I've seen on HN, I have had no down time in EC2, S3, Workspaces, FSx for Windows, Directory Service, Console, etc.

Over 200 services, 86 availability zones and 26 regions. We might as well post every 5 minutes if we post every time something on AWS goes down. And yes AWS is more dependent on us-east-1 but one of the outrages posts was for a minor outage in us-west-1.

ineedasername3• 5 years ago

Well, there really shouldn't be be a service-- even across all regions and products-- going down every five minutes. Especially because it would mean service that needed to be in multiple regions around the world would almost always be having some type of issue.

tyingq5• 5 years ago

one of the outrages posts was for a minor outage in us-west-1.

It was both us-west-1 and us-west-2, for 45 minutes or so. Was interesting for Auth0 customers, as they use us-west-2 as their primary, and us-west-1 is the failover location.

walrus013• 5 years ago

Every single post I've seen on HN, I have had no down time

and I'm averaging six nines uptime over the past year on a $5 KVM virtual machine hosted at a sketchy hosting company, but that doesn't mean something catastrophic can't or won't happen unexpectedly

ksec• 5 years ago

I think it would have been fine if it had Tell HN in front of it. Without it the title would be against the rules and guidelines. One could argue the word Goddamn seems to be provocative. Which could be appropriate if it was really down. But it is not. I think @dang need to edit the title.

SubiculumCode• 5 years ago

It certainly could be editorialized. Alternatively, AWS may be in fact damned by god.

walrus013• 5 years ago

It may be editorialized but it's also exactly the words I would expect to come out of the mouths of some of my most experienced, senior, and clueful colleagues when asked to describe the present reliability of AWS US-EAST-1

codeduck• 5 years ago

Good old chaos-east-1

t8e56vd4ih• 5 years ago

nice try, Jeff ...

k__• 5 years ago

That's because they're running on-premise.

dainiusse• 5 years ago

That's an old joke already

mr-karan• 5 years ago

AS6453 is having a mega outage according to https://www.thousandeyes.com/outages/. Lot of services for users in India were inaccessible for ~1h or so.

markus_zhang2• 5 years ago

Some cloudfare seem to be down too?

cowsandmilk• 5 years ago

So not AWS, but internet outage in general in India?

JCM9• 5 years ago

…or there are Internet issues that are causing issues for people. Do we actually know AWS is having issues or people are speculating?

Not seeing any issues here, but am seeing people reporting broader internet issues at the moment. Post title seems a bit quick on the trigger to point blame.

organic_popcorn• 5 years ago

I have been wondering if Charlie Bell leaving AWS has anything to do with these recent outages. His ops meetings were fucking brutal. I’m curious how they are since he left.

greatpostman• 5 years ago

A lot of senior AWS employees have left. What the root causes of these outages are is a well guarded secret. I work at Amazon and no one I know knows anything in detail about what went down and why

ferdowsi• 5 years ago

Why isn't there any investigative journalism looking into wtf is happening at AWS? What the point of all these tech journalists writing all those puff pieces for access if they're not going to use it to pierce the NDA shield at times like this?

hotpotamus2• 5 years ago

I wonder how many are hosted on AWS and if that has any influence on whether they'd be willing to investigate.

albert_e• 5 years ago

Being merely hosted on AWS should not be an influence at all.

johnday• 5 years ago

If there is sufficiently competent investigative journalism going on, you wouldn't know it either way until a report was released.

chrisseaton• 5 years ago

You can't figure these kind of outages out from the outside, and Amazon aren't going to explain it you except for their official explanations. Plus - 'access' means doing what the subject tells you, certainly not doing anything investigative.

gregjor6• 5 years ago

Amazon has been under the spotlight for labor practices, safety, wages. I’m sure the journalists can still order stuff from Amazon.

As a programmer I can probably guess how AWS experiences problems: programming and networking are hard problems, they get exponentially harder at scale, and there’s no known way to prevent every problem.

If AWS or some journalists gave us details we would just see 200 posts about how they could have done it better with Rust, nothing actually useful for anyone.

bruce343434• 5 years ago

So this is the 4th time an uninteresting update like this has come to the front page... Why? What can be gathered from it? Get the AWS post-mortem on here, for that I'll be piqued. HN is not a status page. I don't instinctively check HN when my service is down because why would I?

mplanchard• 5 years ago

As a counterpoint, HN is one of the first places I check if a big service seems to be down, because it usually shows up here before their status pages.

throwaway82931• 5 years ago

I've always heard about technical debt at AWS, with thousand line functions everyone is terrified to touch, demotivated employees doing whatever it takes to close tickets without dealing with the root causes, and so on.

I presume that this rash of downtime is just the rickety structure inevitably creaking and breaking. It's probably too late now to fix things — dealing with legacy code requires patience, discipline, and understanding which the management of Amazon doesn't have. They'll just yell at people louder, and hold people "accountable" by punishing anyone where anything breaks.

goldenkey2• 5 years ago

I worked there and can vouch for what you said. AWS is built like a house of cards. They just painted it orange and white to make it look cohesive. What's funny though is that the CSS isn't even sourced from one place. They tell front-end engineers to use a dropper tool and to just copy and paste style from other places. DRY is not in Amazon's vocabulary.

Gotta love their leadership principles:

Frugality (we'll give you a shitty laptop, shitty chair, and a shitty desk.)
Be right a lot (are you too stupid to forecast the future?!)"

dralley• 5 years ago

Bryan Cantrill did an interesting deconstruction on Amazon leadership principles.

https://m.youtube.com/watch?v=9QMGAtxUlAc&t=26m42s

sergiotapia• 5 years ago

I never ever want to hear people disparage DigitalOcean or Linode ever again. The "cloud" has now been objectively proven to be just as flakey.

rosndo2• 5 years ago

There is no AWS outage, just routing issues with shit asian ISPs affecting connectivity to many providers. Amazon has nothing whatsoever to do with this.

Imagine Comcast or Verizon fucking up their configs, you might not be able to access AWS stuff but that doesn’t mean that AWS is down.

But hey, the real story wouldn’t make the front page, so lets just stick to the fabricated narrative.

qbasic_forever• 5 years ago

Anyone deploying to production Christmas eve, or even the week of Christmas, should have to get direct approval from the CTO of the company to do so. Just a terrible, terrible idea.

oleg_antonyan• 5 years ago

perfect idea if you're small b2b company b/c the trafic is super low

kw-maller• 5 years ago

I don't know about "4th time this month alone" which seems a bit editorialised, but our company is off AWS for now, and most likely for good.

We're not in the cloud or tech business per se, and as such our customers are not really understanding of technical issues which unfortunately means they are blaming us, and our own reputation is on the line because of AWS' shortcomings.

We did consider Google but for now OVH is the only major provider which is both reliable and secure (w.r.t. court-and-gag letters from government and intelligence agencies) as far as we're concerned. We still use Hetzner and Scaleway for some older stuff and also because of Scaleway's low prices, but it's likely we're moving everything to OVH in the future.

blibble• 5 years ago

meanwhile my raspberry pi cluster at home on residential FTTP has 100% uptime over the last 2 years

(not entirely serious)

authed• 5 years ago

But you know it wouldn't if it had to rely on your home connection AND Amazon. It's like all those major website that rely on 10 domains to serve one webpage...

tillinghast• 5 years ago

This looks to be an ISP issue and not AWS.

sathyabhat• 5 years ago

10:41 AM PST Between 8:59 AM and 9:32 AM PST and between 9:40 AM and 10:16 AM PST we observed Internet connectivity issues with a network provider outside of our network in the AP-SOUTH-1 Region. This impacted Internet connectivity from some customer networks to the AP-SOUTH-1 Region. Connectivity between EC2 instances and other AWS services within the Region was not impacted by this event. The issue has been resolved and we continue to work with the external provider to ensure it does not reoccur.

https://status.aws.amazon.com/#AP_block

bgroat• 5 years ago

There's a video that's being going around HN lately about civilization collapsing and the loss of the ability to maintain robust systems.

Now that I'm looking for it I see it everywhere.

BackBlast• 5 years ago

I would like more information about this. This is the first that I have seen this video mentioned.

yosefjaved12• 5 years ago

According to ThousandEyes, it looks like it was AS16509 that was affected. The outage lasted for 28 min; however, it didn't seem to affect many applications for too long.

I think it's because there was a backup server that kicked in for AS16509; however, it also went down but only for 8 min.

https://www.thousandeyes.com/outages/

yosefjaved12• 5 years ago

According to ThousandEyes, the bigger outage though is TATA Communications server AS6453. That was down or at least having issues of some sort for 1 hr 11 min.

markus_zhang2• 5 years ago

212 cases doesn't seem to be a lot. Does anyone know if it's legit? If they are burning left and right the 9s are going to leave soon.

gregjor6• 5 years ago

These threads always remind me of Umberto Eco's essay Sports Chatter. He describes the degress of participation in sport:

1. Playing the game yourself.

2. Watching other people play.

3. Talking about people playing the game.

4. Listening to someone else talk about people playing the game (i.e. sports commentators).

5. Talking about what sports commentators have to say.

In sports and in threads like this we're mostly at 3, 4, or 5. Even people who work at AWS posting here may only be "watching," i.e. working with second-hand information. While it's entertaining to nerds to speculate about root causes and possible solutions, and the dire predicted fallout from problems (especially when they happen to companies they dislike), it's just chatter and doesn't actually address any real issue, or inform any decisions that mean anything.

ryrymcfly• 5 years ago

It's almost as though it were a bad idea to put all the eggs in one basket

skyzyx• 5 years ago

Pro-Tip: Get the hell out of US-East-1.

Goddamn AWS seems to be down again 4th time this month alone

Related Stories