AWS multiple services outage in us-east-1

https://health.aws.amazon.com/health/status?ts=20251020

by kondro • 9 months ago

2246 2033 9 months ago

health.aws.amazon.com

atymic3• 9 months ago

Looks like maybe a DNS issue? https://www.whatsmydns.net/#A/dynamodb.us-east-1.amazonaws.c...

Resolves to nothing.

immibis12• 9 months ago

It's plausible that Amazon removes unhealthy servers from all round-robins including DNS. If all servers are unhealthy, no DNS.

Alternatively, perhaps their DNS service stopped responding to queries or even removed itself from BGP. It's possible for us mere mortals to tell which of these is the case.

Nextgrid13• 9 months ago

Chances are there's some cyclical dependencies. These can creep up unnoticed without regular testing, which is not really possible at AWS scale unless they want to have regular planned outages to guard against that.

theshrike792• 9 months ago

It's always DNS.

Sparkyte4• 9 months ago

Maybe they forgot to pay the bills.

lexandstuff3• 9 months ago

Yes, we're seeing issues with Dynamo, and potentially other AWS services.

Appears to have happened within the last 10-15 minutes.

atymic3• 9 months ago

Yep, first alert for us fired @ 2025-10-20T06:55:16Z

jimrandomh• 9 months ago

The RDS proxy for our postgres DB went down.

comp_throw7• 9 months ago

We're seeing issues with RDS proxy. Wouldn't be surprised if a DNS issue was the cause, but who knows, will wait for the postmortem.

romanhotsiy• 9 months ago

We're also seeing issues with Lambda and RDS proxy endpoint.

richardwardza2• 9 months ago

We changed our db connection settings to go direct to the db and that's working. Try taking the proxy out the loop

qianli_cs• 9 months ago

We're seeing issues with multiple AWS services https://health.aws.amazon.com/health/status

atymic3• 9 months ago

https://news.ycombinator.com/item?id=45640754

thundergolfer• 9 months ago

This is widespread. ECR, EC2, Secrets Manager, Dynamo, IAM are what I've personally seen down.

mpcoder• 9 months ago

I can't even see my EKS clusters

mopatches2• 9 months ago

Similar: our EC2s are gone.

imstil3earning• 9 months ago

everything is gone :(

Aachen4• 9 months ago

Signal is down from several vantage points and accounts in Europe, I'd guess because of this dependence on Amazon overseas

We're having fun figuring out how to communicate amongst colleagues now! It's when it's gone when you realise your dependence

lexandstuff3• 9 months ago

Thankfully Slack is still holding up.

tubs• 9 months ago

It’s super broken for me. Random threads no longer appear.

LostMyLogin4• 9 months ago

It’s acting up for me but wondering if it’s unrelated. Imagines failing to post and threads acting strange.

esskay12• 9 months ago

Looks like its gone down now

BoredPositron3• 9 months ago

Our last resort fallbacks are channels on different IRC servers. They always hold.

endre• 9 months ago

same here, we never left IRC

Aachen4• 9 months ago

Self hosting is golden. Sadly we already feel like we have too many services for our company's size, and the sensitivity of vulnerabilities in customer systems precludes unencrypted comms. IRC+TLS could be used but we also regularly send screenshots and such in self-destructing messages (not that an attacker couldn't disable that, but to avoid there being a giant archive when we do have some sort of compromise), so we'd rather fall back to something with a similar featureset

As a degraded-state fallback, email is what we're using now (we have our clients configured to encrypt with PGP by default, we use it for any internal email and also when the customer has PGP so everyone knows how to use that)

tcumulus• 9 months ago

Well, at least I now know that my Belgian university's Blackboard environment is running on AWS :)

Traubenfuchs• 9 months ago

We're having fun figuring out how to communicate amongst colleagues now!

When Slack was down we used... google... google mail? chat. When you go to gmail there is actually a chat app on the left.

nik736• 9 months ago

Twilio seems to be affected as well

richardwardza2• 9 months ago

Their entire status page is red!

rickette• 9 months ago

Couple of years ago us-east was considered the least stable region here on HN due to its age. Is that still a thing?

sammy22552• 9 months ago

Yes

immibis12• 9 months ago

Couple of weeks or months ago the front page was saying how us-east-1 instability was a thing of the past due to <whatever chang of architecture>.

esskay12• 9 months ago

Yup, never add anything new to us-east-1. There is never a good reason to willingly use that region.

motiejus2• 9 months ago

When I was there at aws (left about a decade ago), us-east-1 was considered least stable, because it was the biggest.

I.e. some bottle-necks in new code appearing only _after_ you've deployed there, which is of course too late.

It didn't help that some services had their deploy trains (pipelines in amazon lingo) of ~3 weeks, with us-east-1 being the last one.

I bet the situation hasn't changed much since.

shawabawa38• 9 months ago

It didn't help that some services had their deploy trains (pipelines in amazon lingo) of ~3 weeks, with us-east-1 being the last one.

oof, so you're saying this outage could be cause by a change merged 3 weeks ago?

ZeWaka• 9 months ago

Alexa devices are also down.

nvarsj• 9 months ago

And ring! Don’t know why the chime needs an AWS connection. That was surprising.

roosgit• 9 months ago

Can confirm. I was trying to send the newsletter (with SES) and it didn't work. I was thinking my local boto3 was old, but I figured I should check HN just in case.

montek01singh• 9 months ago

I cannot create a support ticket with AWS as well.

starkindustries3• 9 months ago

Zoom is unable to send screenshots.

starkindustries3• 9 months ago

zoom unable to send messages now as well.

sammy22552• 9 months ago

Can't resolve any records for dynamodb.us-east-1.amazonaws.com

However, if you desperately need to access it you can force resolve it to 3.218.182.212. Seems to work for me. DNS through HN

curl -v --resolve "dynamodb.us-east-1.amazonaws.com:443:3.218.182.212" https://dynamodb.us-east-1.amazonaws.com/

rstupek3• 9 months ago

thank you for that info!!!!!

sam1r3• 9 months ago

Dude!! Life saver.

XCSme2• 9 months ago

It's always DNS

alex_suzuki3• 9 months ago

Confirmed.

Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1.

planckscnst3• 9 months ago

There's also dynamodb-fips.us-east-1.amazonaws.com if the main endpoint is having trouble. I'm not sure if this record was affected the same way during this event.

al_james• 9 months ago

Cant even login via the AWS access portal.

pageandrew• 9 months ago

Can't even get STS tokens. RDS Proxy is down, SQS, Managed Kafka.

bpye• 9 months ago

Amazon.ca is degraded, some product pages load but can't see prices. Amusing.

ares623• 9 months ago

Did someone vibe code a DNS change

renatovico• 9 months ago

docker hub or github cache internal maybe is affected:

Booting builder /usr/bin/docker buildx inspect --bootstrap --builder builder-1c223ad9-e21b-41c7-a28e-69eea59c8dac #1 [internal] booting buildkit #1 pulling image moby/buildkit:buildx-stable-1 #1 pulling image moby/buildkit:buildx-stable-1 9.6s done #1 ERROR: received unexpected HTTP status: 500 Internal Server Error ------ > [internal] booting buildkit: ------ ERROR: received unexpected HTTP status: 500 Internal Server Error

mopatches2• 9 months ago

DockerHub shows full outage: https://www.dockerstatus.com/

BartjeD2• 9 months ago

So much for the peeps claiming amazing Cloud uptime ;)

JackSlateur• 9 months ago

Could you give us that uptime, in number ?

BartjeD2• 9 months ago

I'm afraid you missed the emoticon at the end of the sentence.

A `;)` is normally understoond to mean the author isn't entirely serious, and is making light of something or other.

Perhaps you American downvoters were on call and woke up with a fright, and perhaps too much time to browse Hacker News. ;)

Ekaros2• 9 months ago

Wasn't the point why AWS is so much premium that you will always get at least 6 nines if not more in availability?

miohtama• 9 months ago

1. Competitors are not any better, or worse

2. Trusted brand

abujazar8• 9 months ago

Last time I checked the standard SLA is actually 99 % and the only compensation you get for downtime is a refund. Which is why I don't use AWS for anything mission critical.

systemvoltage2• 9 months ago

What do you do if not AWS?

rustc• 9 months ago

Does any host provide more compensation than refund for downtime?

torginus• 9 months ago

They guarantee the dashboard will be green 99.999% of the time

Ekaros2• 9 months ago

I take dashboard is not covered by SLA?

crbaker• 9 months ago

us-east-1 the worst region for availability

esskay12• 9 months ago

It's usually true if you arent in US-East-1 which is widely known to be the least reliable location. Theres no reason anyone should be deploying anything new to it these days.

Unroasted61542• 9 months ago

You are supposed to build multi regional services if you need higher resilience.

Nextgrid13• 9 months ago

Actual multi-region replication is hard and forces you to think about complicated things like the CAP theorem/etc. It's easier to pretend AWS magically solves that problem for you.

Which is actually totally fine for the vast majority of things, otherwise there would be actual commercial pressures to make sure systems are resilient to such outages.

that_guy_iain3• 9 months ago

You can. You just need to do the work to make it work. That's the bit where everyone but Netflix and Amazon fail.

shawabawa38• 9 months ago

the highest availability service i think is S3 at 4 nines

you might be thinking of durability for s3 which is 11 nines, and i've never heard of anyone losing an object yet

alex_suzuki3• 9 months ago

no, it's probably Route 53, touted as having "100% availability" (https://en.wikipedia.org/wiki/Amazon_Route_53)

Nextgrid13• 9 months ago

The point of AWS is to promise you the nines and make you feel good about it. Your typical "growth & engagement" startup CEO can feel good and make his own customers feel good about how his startup will survive a nuclear war.

Delivery of those nines is not a priority. Not for the cloud provider - because they can just lie their way out of it by not updating their status page - and even when they don't, they merely have to forego some of their insane profit margin for a couple hours in compensation. No provider will actually put their ass on the line and offer you anything beyond their own profit margin.

This is not an issue for most cloud clients either because they keep putting up with it (lying on the status page wouldn't be a thing if clients cared) - the unspoken truth is that nobody cares that your "growth & engagement" thing is down for an hour or so, so nobody makes anything more than a ceremonial stink about it (chances are, the thing goes down/misbehaves regularly anyway every time the new JS vibecoder or "AI employee" deploys something, regardless of cloud reliability).

Things where nines actually matter will generally invest in self-managed disaster recovery plans that are regularly tested. This also means it will generally be built differently and far away from your typical "cloud native" dumpster fire. Depending on how many nines you actually need (aka what's the cost of not meeting that target - which directly controls how much budget you have to ensure you always meet it), you might be building something closer to aircraft avionics with the same development practices, testing and rigor.

mightyham• 9 months ago

I can tell you from personal experience that improving/maintaining uptime (by doing root cause analysis, writing correction of error reports, going through application security reviews, writing/reviewing design docs for safely deploying changes, working on operational improvements to services) probably takes up a majority of most AWS engineers' time. I'm genuinely curious what you are basing the opinion "Delivery of those nines is not a priority" off of.

hexbin010• 9 months ago

Why after all these years is us-east-1 such a SPOF?

nikolay• 9 months ago

Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!

tokioyoyo3• 9 months ago

Doing pretty well up here in Tokyo region for now! Just can't log into console and some other stuff.

happymellon• 9 months ago

Check the URL, we had an issue a couple of years ago with the Workspaces. US East was down but all of our stuff was in EU.

Turns out the default URL was hardcoded to use the us east interface and just by going to workspaces and then editing your URL to be the local region got everyone working again.

Unless you mean nothing is working for you at the moment.

thdhhghgbhy• 9 months ago

Doesn't this mean you are not regionally isolated from us-east-1?

Sparkyte4• 9 months ago

I am down with that lets all build in US-East-1.

sam1r3• 9 months ago

Sometimes we all need a tech shutdown.

sunrunner• 9 months ago

As they say, every cloud outage has a silver lining.

* Give the computers a rest, they probably need it. Heck, maybe the Internet should just shut down in the evening so everyone can go to bed (ignoring those pesky timezone differences)

* Free chaos engineering at the cloud provider region scale, except you didn't opt in to this one and know about in advance, making it extra effective

* Quickly figure out a map which of the things you use have a dependency on a single AWS region without no capability to change or re-route

kshacker• 9 months ago

Disconnect day

rdhatt• 9 months ago

One unexpected upside moving from a DC to AWS is when a region is down, customers are far more understanding. Instead of being upset, they often shrug it off since nothing else they needed/wanted was up either.

scandox• 9 months ago

This is a remarkable and unfair truth. I have had this experience with Office365...when they're down a lot of customers don't care because all their customers are also down.

ta124310• 9 months ago

It took me so long to realise this is what's important in enterprise. Uptime isn't important, being able to blame someone else is what's important.

If you're down for 5 minutes a year because one of your employees broke something, that's your fault, and the blame passes down through the CTO.

If you're down for 5 hours a year but this affected other companies too, it's not your fault

From AWS to Crowdstrike - system resilience and uptime isn't the goal. Risk mitigation isn't the goal. Affordability isn't the goal.

When the CEO's buddies all suffer at the same time as he does, it's just an "act of god" and nothing can be done, it's such a complex outcome that even the amazing boffins at aws/google/microsoft/cloudflare/etc can't cope.

If the CEO is down at a different time than the CEO's buddies then it's that Dave/Charlie/Bertie/Alice can't cope and it's the CTO's fault for not outsourcing it.

As someone who likes to see things working, it pisses me off no end, but it's the way of the world, and likely has been whenever the owner and CTO are separate.

shermantanktop2• 9 months ago

A slightly less cynical view: execs have a hard filter for “things I can do something about” and “things I can’t influence at all.” The bad ones are constantly pushing problems into the second bucket, but there are legitimately gray area cases. When an exec smells the possibility that their team could have somehow avoided a problem, that’s category 1 and the hammer comes down hard.

After that process comes the BS and PR step, where reality is spun into a cotton candy that makes the leader look good no matter what.

balls187• 9 months ago

It took me so long to realise this is what's important in enterprise. Uptime isn't important, being able to blame someone else is what's important.

Yes.

What is important is having a Contractual SLA that is defensible. Acts of God are defensible. And now major cloud infrastructure outtages are too.

arvindh-manian• 9 months ago

“No one ever got fired for hiring IBM”

ej_campbell5• 9 months ago

And all your dependencies are co-located.

Gigachad2• 9 months ago

I was once told that our company went with Azure because when you tell the boomer client that our service is down because Microsoft had an outage, they go from being mad at you, to accepting that the outage was an act of god that couldn’t be avoided.

myhf• 9 months ago

Azure outages: happens all the time, understandable, no way to prevent this

AWS outages: almost never happens, you should have been more prepared for when it does

kelseydh3• 9 months ago

Is us-east-1 equally unstable to the other regions? My impression was that Amazon deployed changes to us-east-1 first so it's the most unstable region.

morshu90012• 9 months ago

Would think that Amazon safeguards their biggest region more, but no idea, I've never worked at AWS

hinkley20• 9 months ago

I've heard this so many times and not seen it contradicted so I started saying it myself. Even my last Ops team wanted to run some things in us-east-1 to get prior warning before they broke us-west-1.

But there are some people on Reddit who think we are all wrong but won't say anything more. So... whatever.

Nothing in the outage history really stands out as "this is the first time we tried this and oops" except for us-east-1.

It's always possible for things to succeed at a smaller scale and fail at full scale, but again none of them really stand out as that to me. Or at least, not any in the last ten years. I'm allowing that anything older than that is on the far side of substantial process changes and isn't representative anymore.

systemvoltage2• 9 months ago

workos is down too, timing is highly correlated with AWS outage: https://status.workos.com/

That means Cursor is down, can't login.

rdm_blackhole2• 9 months ago

Vercel functions are down as well.

gianpaj• 9 months ago

Yes https://www.vercel-status.com/

__coder__• 9 months ago

Perplexity also have outage.

https://status.perplexity.ai

abujazar8• 9 months ago

I find it interesting that AWS services appear to be so tightly integrated that when there's an issue in a region, it affects most or all services. Kind of defeats the purported resiliency of cloud services.

tokioyoyo3• 9 months ago

You know how people say X startup is ChatGPT wrapper? A significant chunk of AWS services are wrappers of main services (DynamoDB, EC2, S3 and etc).

abujazar8• 9 months ago

Yes, and that's exactly the problem. It's like choosing a microservice architecture for resiliency and building all the services on top of the same database or message queue without underlying redundancy.

bananapub3• 9 months ago

you can't possibly know that?

surely you mean:

I find it interesting that AWS services appear to be so tightly integrated that when there's an issue THAT BECOMES VISIBLE TO ME in a region, it affects most or all services.

AWS has stuff failing alllllllll the time, it's not very surprising that many of the outages that become visible to you involve multi-system failures - lots of other ones don't become visible!

abujazar8• 9 months ago

Sure, but none of those other issues are ever documented by AWS as their status page is usually just one big lie.

bradhe6• 9 months ago

Yeah I think there are a number of "hidden" dependencies on different regions, especially us-east-1. It's an artifact of it being AWS' largest region, etc.

hshdhdhehd4• 9 months ago

why dont they have us east 2, 3, 4 etc. Actually have different cities.

esskay12• 9 months ago

I think a lot of its probably technical debt. So much internally still relies on legacy systems in US-East-1, and every time this happens I'm sure theres a discussion internally about decoupling that reliance which then turns into a massive diagram that looks like a family tree dating back a thousand years of all the things that need to be changed to stop it happening.

abujazar8• 9 months ago

Sounds plausible. It's also a "fat and happy" symptom not to be able to fix deep underlying issues despite an ever growing pile of cash in the company.

Nextgrid13• 9 months ago

There's also the issue of sometimes needing actual strong consistency. Things like auth or billing for example where you absolutely can't tolerate eventual consistency or split-brain situations, in which case you need one region to serve as the ultimate source of truth.

yuvadam• 9 months ago

During the last us-east-1 apocalypse 14 years ago, I started awsdowntime.com - don't make me regsiter it again and revive the page.

Sparkyte4• 9 months ago

I remember the one where some contractor accidentally cut the trunk between AZes.

oasisbob2• 9 months ago

In us-east-1? That doesn't sound that impactful, have always heard that us-east-1's network is a ring.

Back before AWS provided transparency into AZ assignments, it was pretty common to use latency measurements to try and infer relative locality and mappings of AZs available to an account.

padjo5• 9 months ago

Friends don’t let friends use us-east-1

seviu2• 9 months ago

I can't log in to my AWS account, in Germany, on top of that it is not possible to order anything or change payment options from amazon.de.

No landing page explaining services are down, just scary error pages. I thought account was compromised. Thanks HN for, as always, being the first to clarify what's happening.

Scary to see that in order to order from Amazon Germany, us-east1 must be up. Everything else works flawlessly but payments are a no go.

that_guy_iain3• 9 months ago

I just ordered stuff from Amazon.de. And I highly any Amazon site can go down because of one region. Just like Netflix are rarely affected.

seviu2• 9 months ago

I can’t even login, I get the internal error treatment. This is on Amazon.de

serial_dev• 9 months ago

I wanted to log into my Audible account after a long time on my phone, I couldn't, started getting annoyed, maybe my password is not saved correctly, maybe my account was banned, ... Then checking desktop, still errors, checking my Amazon.de, no profile info... That's when I started suspecting that it's not me, it's you, Amazon! Anyway, I guess, I'll listen to my book in a couple of hours, hopefully.

Btw, most parts of the amazon.de is working fine, but I can't load profiles, and can't login.

echelon_musk• 9 months ago

You might be interested in Libation [0]. I use it to de-DRM my Audible library and generate a cue sheet with chapters in for offline listening.

[0] https://getlibation.com/

vladvasiliu2• 9 months ago

We use IAM Identity Center (née SSO) which is hosted in the eu-central-1 region, and I can log in just fine. Its admin pages are down, though. Ditto for IAM.

Other things seem to be working fine.

jumploops• 9 months ago

"Never choose us-east-1"

ta124310• 9 months ago

Never choose a single point of failure.

Or rather

Ensure your single point of failure risk is appropriate for your business. I don't have full resilience for my companies AS going down, but we do have limited DR capability. Same with the loss of a major city or two.

I'm not 100% confident in a Thames Barrier flood situation, as I suspect some of our providers don't have the resilience levels we do, but we'd still be able to provide some minimal capability.

tosh2• 9 months ago

seeing issues with SES in us-east-1 as well

grebc3• 9 months ago

Good thing hyperscalers provide 100% uptime.

mcbain• 9 months ago

When have they ever claimed that?

grebc3• 9 months ago

Plenty of people last week here claiming hyperscalers are necessary while ignoring more normal hosting options.

empressplay• 9 months ago

Can't check out on Amazon.com.au, gives error page

kondro9• 9 months ago

This link works fine from Australia for me.

sam1r3• 9 months ago

Chime has completely been down for almost 12 hours.

Impacting all banking series with red status error. Oddly enough, only their direct deposits are functioning without issues.

https://status.chime.com/

nodesocket2• 9 months ago

Affecting Coinbase[1] as well, which is ridiculous. Can't access the web UI at all. At their scale and importance they should be multi-region if not multi-cloud.

[1] https://status.coinbase.com

bradhe6• 9 months ago

Seems the underlying issue is with DynamoDB, according to the status page, which will have a big blast radius in other services. AWS' services form a really complicated graph and there's likely some dependency, potentially hidden, on us-east-1 in there.

Splizard• 9 months ago

The issue appears to be cascading internationally due to internal dependencies on us-east-1

port3000• 9 months ago

Even railway's status page is down (guess they use Vercel):

https://railway.instatus.com/

ctbellmar• 9 months ago

Various AI services (e.g. Perplexity) are down as well

rvz2• 9 months ago

Just tried Perplexity and it has no answer.

Damn, this is really bad.

Looking forward to the postmortem.

bartvk• 9 months ago

I don't like how they phrased it. From the Verge:

“Perplexity is down right now,” Perplexity CEO Aravind Srinivas said on X. “The root cause is an AWS issue. We’re working on resolving it.”

What he should have said, IMHO, is "The root cause is that Perplexity fully depends on AWS."

I wonder if they're actually working on resolving that, or that they're just waiting for AWS to come back up.

ta124310• 9 months ago

Meanwhile my pair of 12 year old raspberry pi's hangling my home services like DNS survive their 3rd AWS us-east-1 outage.

"But you can't do webscale uptime on your own"

Sure. I suspect even a single pi with auto-updates on has less downtime.

__alexs• 9 months ago

Is there any data on which AWS regions are most reliable? I feel like every time I hear about an AWS outage it's in us-east-1.

nodesocket2• 9 months ago

I don't recommend to my clients they use us-east-1. It's the oldest and most prone to outages. I usually always recommend us-east-2 (Ohio) unless they require West Coast.

dr-smooth2• 9 months ago

and if they need West Coast, it's us-west-2. I consider us-west-1 to be a failed region. They don't get some of the new instance types, you can't get three AZs for your VPCs, and they're more expensive than the other US regions.

bradhe6• 9 months ago

us-east-1 was, probably still is, AWS' most massive deployment. Huge percentage of traffic goes through that region. Also, lots of services backhaul to that region, especially S3 and CloudFront. So even if your compute is in a different region (at Tower.dev we use eu-central-1 mostly), outages in us-east-1 can have some halo effect.

This outage seems really to be DynamoDB related, so the blast radius in services affected is going to be big. Seems they're still triaging.

gnaman• 9 months ago

your website loads for a second and then suddenly goes blank. There is one fatal errors from Framer in the console

Thev00d00• 9 months ago

Anywhere other than us-east-1 in my experience is rock solid.

linsomniac• 9 months ago

Agreed, my company had been entirely on us-east-1 predating my joining ~12 years ago. ~7 years ago, after multiple us-east-1 outages, I moved us to us-west-2 and it has been a lot less bumpy since then.

theherk• 9 months ago

Trouble is one can't fully escape us-east-1. Many services are centralized there like: S3, Organizations, Route53, Cloudfront, etc. It is THE main region, hence suffering the most outages, and more importantly, the most troubling outages.

dangoodmanUT2• 9 months ago

This is not true. There are a FEW high-level ones that have their dashboards there, but there's a reason you log into region-specific dashboards

Dave3of5• 9 months ago

If you're using AWS then you are most likely using us-east-1 there is no escape. When big problems happen on us-east-1 it affect most of AWS services.

Macha2• 9 months ago

We're mostly deployed on eu-west-1 but still seeing weird STS and IAM failures, likely due to internal AWS dependencies.

Also we use Docker Hub, NPM and a bunch of other services that are hosted by their vendors on us-east-1 so even non AWS customers often can't avoid the blast radius of us-east-1 (though the NPM issue mostly affects devs updating/adding dependencies, our CI builds use our internal mirror)

donavanm• 9 months ago

FYI: 1. AWS IAM mutations all go through us-east-1 before being replicated to other public/commercial regions. Read/List operations should use local regional stacks. I expect you'll see a concept of "home region" give you flexibility on the write path in the future. 2. STS has both global and regional endpoints. Make sure you're setup to use regional endpoints in your clients https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...

philipp-gayret• 9 months ago

Our Alexa's stopped responding and my girl couldn't log in to myfitness pal anymore.. Let me check HN for a major outage and here we are :^)

At least when us-east is down, everything is down.

kryptn• 9 months ago

Wonder if this is related

https://www.dockerstatus.com/pages/533c6539221ae15e3f000031

Titan2189• 9 months ago

Yup

We have identified the underlying issue with one of our cloud service providers.

ngruhn• 9 months ago

Can't login to Jira/Confluence either.

Xenoamorphous2• 9 months ago

Seems to work fine for me. I'm in Europe so maybe connecting to some deployment over here.

danias• 9 months ago

You are already logged in. If you try to access your account settings, for example, you will be disappointed...

dude2507112• 9 months ago

They are amazing at LeetCode though.

tokioyoyo3• 9 months ago

Let's be nice. I'm sure devs and ops are on fire right now, trying to fix the problems. Given the audience of HN, most of us could have been (have already been?) in that position.

dude2507112• 9 months ago

They choose their hiring-retention practices and they choose to provide global infrastructure, when is the good time to criticise them?

Granted, they are not as drunk on LLM as Google and Microsoft. So, at least we can say this outage had not been vibe-coded (yet).

fragmede5• 9 months ago

hugops ftw

rirze2• 9 months ago

No we wouldn’t because there’s like a 50/50 chance of being a H1B/L1 at AWS. They should rethink their hiring and retention strategies.

rwky6• 9 months ago

To everyone that got paged (like me), grab a coffee and ride it out, the week can only get better!

esskay12• 9 months ago

To everyone who was supposed to get paged but didn't, do a postmortem, chances are your service is running via Twilio and needs migrating elsewhere.

rwky6• 9 months ago

Hah good point, pagerduty is working at least.

avian• 9 months ago

grab a coffee and ride it out

The way things are today I'm thankful the coffee machine still works without AWS.

rwky6• 9 months ago

Feel sorry for anyone with a "smart" coffee machine.

greenavocado• 9 months ago

With how long it may last, pour some cold water on the coffee and come back to drink the supernatant in a few hr

hshdhdhehd4• 9 months ago

Burn that SLO, next persons problem eh!

amadeoeoeo• 9 months ago

Oh no... may be LaLiga found out pirates hosting on AWS?

agos2• 9 months ago

this is how I discover that is not just Serie A doing this shenanigans. I'm not really surprised

sofixa6• 9 months ago

All the big leagues take "piracy" very seriously and constantly try to clamp down on it.

TV rights is one of their main revenue sources, and it's expected to always go up, so they see "piracy" as a fundamental threat. IMO, it's a fundamental misunderstanding on their side, because people "pirating" usually don't have a choice - either there is no option for them to pay for the content (e.g. UK's 3pm blackout), or it's too expensive and/or spread out. People in the UK have to pay 3-4 different subscriptions to access all local games.

The best solution, by far, is what France's Ligue 1 just did (out of necessity though, nobody was paying them what they wanted for the rights after the previous debacles). Ligue 1+ streaming service, owned and operated by them which you can get access through a variety of different ways (regular old TV paid channel, on Amazon Prime, on DAZN, via Bein Sport), whichever suits you the best. Same acceptable price for all games.

donmb• 9 months ago

Asana down Postman workspaces don't load Slack affected And the worst: heroku scheduler just refused to trigger our jobs

SeanAnderson• 9 months ago

Looks like it affected Vercel, too. https://www.vercel-status.com/

My website is down :(

(EDIT: website is back up, hooray)

maximefourny• 9 months ago

Have you done anything for it to be back up? Looks like mines are still down.

LostMyLogin4• 9 months ago

Looks as if they are rerouting to a different region.

hugh-avherald• 9 months ago

mines are generally down

l5870uoo9y• 9 months ago

Static content resolves correctly but data fetching is still not functional.

TiredOfLife• 9 months ago

Service that runs on aws is down when aws is down. Who knew.

jellyfishbeaver• 9 months ago

I had a chuckle on my way home yesterday. Standing on the train platform and seeing "Next departure in: (Vercel Connection Error)" on the screen. :P

TechDebtDevin• 9 months ago

Imagine using vercel, a company that literally contributes to the starvation of children and is proud of it. Also, literally just learn to use a Dockerfile and a vps, like why do these PaaS even exist, you're paying 3x for the same AWS services.

magnio• 9 months ago

npm and pnpm are badly affected as well. Many packages are returning 502 when fetched. Such a bad time...

samsepia• 9 months ago

Yup, was releasing something to prod and can't even build a react app. I wonder if there is some sort of archive that isn't affected?

JCharante3• 9 months ago

AWS CodeArtifact can act as a proxy and fetch new packages from npm when needed. A bit late for that though but sharing if you want to future proof against the yearly us-east-1 outage

JCharante3• 9 months ago

Oh damn that ruins all our builds for regions I thought would be unaffected

alex_suzuki3• 9 months ago

Paddle (payment provider) is down as well: https://paddlestatus.com/

mcintyre19942• 9 months ago

Presumably the root cause of the major Vercel outage too: https://www.vercel-status.com/

hyruo• 9 months ago

No wonder, when I opened Vercel it showed a 502 error.

colesantiago3• 9 months ago

It seems that all the sites that ask for distributed systems in their interview and has their website down wouldn't even pass their own interview.

This is why distributed systems is an extremely important discipline.

mangamadaiyan• 9 months ago

Maybe actually making the interviews less of a hazing ritual would help.

Hell, maybe making today's tech workplace more about getting work done instead of the series of ritualistic performances that the average tech workday has degenerated to might help too.

Ergo, your conclusion doesn't follow from your initial statements, because interviews and workplaces are both far more broken than most people, even people in the tech industry, would think.

colesantiago3• 9 months ago

Well it looks like if companies and startups did their job in hiring the proper distributed systems skills more rather than hazing for the wrong skills we wouldn't be in this outage mess.

Many companies on Vercel don't think to have a strategy to be resilient to these outages.

I rarely see Google, Ably and others serious about distributed systems being down.

dist-epoch2• 9 months ago

distributed systems != continuous uptime

colesantiago3• 9 months ago

100% wrong.

Serious engineering teams that care about distributed systems and multi region deployments don't think like this.

mumber_typhoon• 9 months ago

Oct 20 12:51 AM PDT We can confirm increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share.

Weird that case creation uses the same region as the case you'd like to create for.

easton• 9 months ago

The support apis only exist in us-east-1, iirc. It’s a “global” service like IAM, but that usually means modifications to things have to go through us-east-1 even if they let you pull the data out elsewhere.

transitivebs• 9 months ago

can't log into https://amazon.com either after logging out; so many downstream issues

ksajadi• 9 months ago

A lot of status pages hosted by Atlasian StatusPage are down! The irony…

atonse• 9 months ago

I can't believe this. When status page first created their product, they used to market how they were in multiple providers so that they'd never be affected by downtime.

Maybe all that got canned after the acquisition?

trusche• 9 months ago

Both Intercom and Twilio are affected, too.

- https://status.twilio.com/ - https://www.intercomstatus.com/us-hosting

I want the web ca. 2001 back, please.

Xenoamorphous2• 9 months ago

Slack now failing for me.

jcmeyrignac• 9 months ago

Impossible to connect to JIRA here (France).

baumschubser• 9 months ago

binsquare2• 9 months ago

Don't miss this

JCharante3• 9 months ago

Ring is affected. Why doesn’t Ring have failover to another region?

voxadam2• 9 months ago

That's understandably bad for anyone who depends on Ring for security but arguably a net positive for the rest of us.

Amazon’s Ring to partner with Flock: https://news.ycombinator.com/item?id=45614713

lsllc• 9 months ago

The Ring (Doorbell) App isn't working, nor is any the MBTA (Transit) Status pages/apps.

LostMyLogin4• 9 months ago

My apartment uses “SmartRent” for access controls and temps in our unit. It’s down…

arch-choot• 9 months ago

So there's no way to get back in if you step out for food?

mittermayr2• 9 months ago

Careful: NPM _says_ they're up (https://status.npmjs.org/) but I am seeing a lot of packages not updating and npm install taking forever or never finishing. So hold off deploying now if you're dependent on that.

drinchev• 9 months ago

Also npm audit times out.

gjvr• 9 months ago

Yep. It's the auditing part that is broken. As a (dangerous) workaround use --no-audit

olex• 9 months ago

They've acknowledged an issue now on the status page. For me at least, it's completely down, package installation straight up doesn't work. Thankfully current work project uses a pull-through mirror that allows us to continue working.

tonyhart7• 9 months ago

"Thankfully current work project uses a pull-through mirror that allows us to continue working."

so there is no free coffee time???? lmao

binsquare2• 9 months ago

The internal disruption reviews are going to be fun :)

Msurrow3• 9 months ago

The fun is really gonna start if the root cause of this somehow implicates an AI as a primary cause.

aurareturn7• 9 months ago

It's never an AI's fault since it's up to a human to implement the AI and put in a process that prevents this stuff from happening.

So blame humans even if an AI wrote some bad code.

karel-3d3• 9 months ago

I haven't seen the "90% of our code is AI" nonsense from Amazon.

portaouflop• 9 months ago

It’s gonna be DNS

blibble• 9 months ago

if that's the case it'll be buried

codebolt2• 9 months ago

Atlassian cloud is having problems as well.

seanieb• 9 months ago

Clearly this is all some sort of mass delusion event, the Amazon Ring status says everything is working.

https://status.ring.com/

(Useless service status pages are incredibly annoying)

storgaard• 9 months ago

Atlassian is down as well so they probably can't access their Atlassian Statuspage admin panel to update it.

netdevphoenix2• 9 months ago

When you know a service is down but the service says it's up: it's either your fault or the service is having a severe issue

ArcHound3• 9 months ago

Good luck to all on-callers today.

It might be an interesting exercise to map how many of our services depend on us-east-1 in one way or another. One can only hope that somebody would do something with the intel, even though it's not a feature that brings money in (at least from business perspective).

hipratham• 9 months ago

Strangely some of our services are scaling up on east-1, and there is downtick on downdetector.com so issue might be resolving.

jug• 9 months ago

Of course this happens when I take a day off from work lol

Came here after the Internet felt oddly "ill" and even got issues using Medium, and sure enough https://status.medium.com

qrush• 9 months ago

AWS's own management console sign-in isn't even working. This is a huge one. :(

cpfleming• 9 months ago

Seems to be upsetting Slack a fair bit, messages taking an age to send and OIDC login doesn't want to play.

AtNightWeCode4• 9 months ago

Considering the history of east-1 it is fascinating that it still causes so many single point of failure incidents for large enterprises.

askonomm• 9 months ago

Docker is also down.

1659447091• 9 months ago

Also:

Snapchat, Ring, Roblox, Fortnite and more go down in huge internet outage: Latest updates https://www.the-independent.com/tech/snapchat-roblox-duoling...

To see more (from the first link): https://downdetector.com

tedk-42• 9 months ago

Internet, out.

Very big day for an engineering team indeed. Can't vibe code your way out of this issue...

rvz2• 9 months ago

Can't vibe code your way out of this issue...

Exactly. This time, some LLM providers are also down and can't help vibe coders on this issue.

fragmede5• 9 months ago

Qwen3 on lm-studio running fine on my work Mac M3, what's wrong with yours?

LostMyLogin4• 9 months ago

Pour one out for everyone on-call right now.

xvector4• 9 months ago

After some thankless years preventing outages for a big tech company, I will never take an oncall position again in my life.

Most miserable working years I have had. It's wild how normalized working on weekends and evenings becomes in teams with oncall.

But it's not normal. Our users not being able to shitpost is simply not worth my weekend or evening.

And outside of Google you don't even get paid for oncall at most big tech companies! Company losing millions of dollars an hour, but somehow not willing to pay me a dime to jump in at 3AM? Looks like it's not my problem!

drevil-v2• 9 months ago

Easiest day for engineers on-call everywhere except AWS staff. There’s nothing you can do except wait for AWS to come back online.

Pour one out for the customer service teams of affected businesses instead

codeduck3• 9 months ago

and by one I trust you mean a bottle.

darkwater3• 9 months ago

Well, but tomorrow there will be CTOs asking for a contingency plan if AWS goes down, even if planning, preparing, executing and keeping it up to date as the infra evolves will cost more than the X hours of AWS outage.

There are certainly organizations for which that cost is lower than the overall damage of services being down due to AWS fault, but tomorrow we will hear CTOs from smaller orgs as well.

aswegs8• 9 months ago

Can confirm, pretty chill we can blame our current issues on AWS.

mvdtnz2• 9 months ago

No really true for large systems. We are doing things like deploying mitigations to avoid scale-in (eg services not receiving traffic incorrectly autoscaling down), preparing services for the inevitable storm, managing various circuit breakers, changing service configurations to ease the flow of traffic through the system, etc. We currently have 64 engineers in our on-call room managing this. There's plenty of work to do.

fisf3• 9 months ago

Well, some engineer somewhere made the recommendation to go with AWS, even tho it is more expensive than alternatives. That should raise some questions.

devjam• 9 months ago

Easiest day for engineers on-call everywhere

I have three words for you: cascading systems failure

DebtDeflation• 9 months ago

Can't vibe code your way out of this issue...

I feel bad for the people impacted by the outage. But at the same time there's a part of me that says we need a cataclysmic event to shake the C-Suite out of their current mindset of laying off all of their workers to replace them with AI, the cheapest people they can find in India, or in some cases with nothing at all, in order to maximize current quarter EPS.

moomoo11• 9 months ago

/ai why is AWS down? can you bring it back up

SilverSlash• 9 months ago

I expect it's their SREs who are dealing with this mess.

mslm• 9 months ago

Happened to be updating a bunch of NPM dependencies and then saw `npm i` freeze and I'm like... ugh what did I do. Then npm login wasn't working and started searching here for an outage, and wala.

skolsuper• 9 months ago

voila

kalleboo3• 9 months ago

It's fun watching their list of "Affected Services" grow literally in front of your eyes as they figure out how many things have this dependency.

It's still missing the one that earned me a phone call from a client.

zenexer• 9 months ago

It's seemingly everything. SES was the first one that I noticed, but from what I can tell, all services are impacted.

hvb210• 9 months ago

In AWS, if you take out one of dynamo db, S3 or lambda you're going to be in a world of pain. Any architecture will likely use those somewhere including all the other services on top.

If in your own datacenter your storage service goes down, how much remains running

goatking2• 9 months ago

Agreed, but you can put EC2 on that list as well

mlrtime7• 9 months ago

When these major issues come up, all they have is symptoms and not causes. Maybe not until the dynamo oncall comes on and says its down, then everyone knows at least the reason for their teams outage.

The scale here is so large they don't know the complete dependency tree until teams check-in on what is out or not, growing this list. Of course most of it is automated, but getting on 'Affected Services' is not.

hbn• 9 months ago

I know Postman has kinda gone to shit over the years but it's hilarious my local REST client that makes requests from my machine has AWS as a dependency .

I found that out about Plex during an outage too.

robertpohl• 9 months ago

Looks like we're back!

DataDaemon2• 9 months ago

But but this is a cloud, it should exist in the cloud.

stavros3• 9 months ago

AWS truly does stand for "All Web Sites".

circadian• 9 months ago

BGP (again)?

klon• 9 months ago

Statuspage.io seems to load (but is slow) but what is the point if you can't post an incident because Atlassian ID service is down.

edtech_dev• 9 months ago

Signal is also down for me.

ArcHound3• 9 months ago

My messages are not getting through, but status page seems ok.

ArcHound3• 9 months ago

Seems fixed now.

chaidhat• 9 months ago

is this why docker is down?

dolibasija• 9 months ago

yes hub.docker.com. 75 IN CNAME elb-default.us-east-1.aws.dckr.io.

sph3• 9 months ago

10:30 on a Monday morning and already slacking off. Life is good. Time to touch grass, everybody!

saejox• 9 months ago

AWS has been the backbone of the internet. It is single point of failure most websites.

Other hosting services like Vercel, package managers like npm, even the docker registeries are down because of it.

pmig• 9 months ago

Thanks god we built all our infra on top of EKS, so everything works smoothly =)

whatsupdog• 9 months ago

I can not login to my AWS account. And, the "my account" on regular amazon website is blank on Firefox, but opens on Chrome.

Edit: I can login into one of the AWS accounts (I have a few different ones for different companies), but my personal which has a ".edu" email is not logging in.

fujigawa• 9 months ago

Appears to have also disabled that bot on HN that would be frantically posting [dupe] in all the other AWS outage threads right about now.

altairprime2• 9 months ago

That’s done by human beings.

bootsmann• 9 months ago

Apparently hiring 1000s of software engineers every month was load bearing

mrcsharp• 9 months ago

Bitbucket seems affected too [1]. Not sure if this status page is regional though.

[1] https://bitbucket.status.atlassian.com/incidents/p20f40pt1rg...

gramakri22• 9 months ago

npm registry also down

thomas_witt2• 9 months ago

Seems to be really only in us-east-1, DynamoDB is performing fine in production on eu-central-1.

fairity• 9 months ago

As this incident unfolds, what’s the best way to estimate how many additional hours it’s likely to last? My intuition is that the expected remaining duration increases the longer the outage persists, but that would ultimately depend on the historical distribution of similar incidents. Is that kind of data available anywhere?

greybeard69• 9 months ago

To my understanding the main problem is DynamoDB being down, and DynamoDB is what a lot of AWS services use for their eventing systems behind the scenes. So there's probably like 500 billion unprocessed events that'll need to get processed even when they get everything back online. It's gonna be a long one.

jewba• 9 months ago

500 billions events. Always blows my mind how many people use aws

froobius10• 9 months ago

Yes, with no prior knowledge the mathematically correct estimate is:

time left = time so far

But as you note prior knowledge will enable a better guess.

matsemann2• 9 months ago

Yeah, the Copernican Principle.

I visited the Berlin Wall. People at the time wondered how long the Wall might last. Was it a temporary aberration, or a permanent fixture of modern Europe? Standing at the Wall in 1969, I made the following argument, using the Copernican principle. I said, Well, there’s nothing special about the timing of my visit. I’m just travelling—you know, Europe on five dollars a day—and I’m observing the Wall because it happens to be here. My visit is random in time. So if I divide the Wall’s total history, from the beginning to the end, into four quarters, and I’m located randomly somewhere in there, there’s a fifty-percent chance that I’m in the middle two quarters—that means, not in the first quarter and not in the fourth quarter.
Let’s suppose that I’m at the beginning of that middle fifty percent. In that case, one-quarter of the Wall’s ultimate history has passed, and there are three-quarters left in the future. In that case, the future’s three times as long as the past. On the other hand, if I’m at the other end, then three-quarters have happened already, and there’s one-quarter left in the future. In that case, the future is one-third as long as the past.

https://www.newyorker.com/magazine/1999/07/12/how-to-predict...

tsimionescu10• 9 months ago

Note that this is equivalent to saying "there's no way to know". This guess doesn't give any insight, it's just the function that happens to minimize the total expected error for an unknowable duration.

Edit: I should add that, more specifically, this is a property of the uniform distribution, it applies to any event for which EndsAfter(t) is uniformly distributed over all t > 0.

seydor• 9 months ago

1440 min

rwky6• 9 months ago

Generally expect issues for the rest of the day, AWS will recover slowly, then anyone that relies on AWS will recovery slowly. All the background jobs which are stuck will need processing.

jameshart4• 9 months ago

Rule of thumb is that the estimated remaining duration of an outage is equal to the current elapsed duration of the outage.

movpasd2• 9 months ago

I used Claude to get the outage start and ends from the post-event summaries for major historical AWS outages: https://aws.amazon.com/premiumsupport/technology/pes/

The cumulative distribution actually ends up pretty exponential which (I think) means that if you estimate the amount of time left in the outage as the mean of all outages that are longer than the current outage, you end up with a flat value that's around 8 hours, if I've done my maths right.

Not a statistician so I'm sure I've committed some statistical crimes there!

Unfortunately I can't find an easy way to upload images of the charts I've made right now, but you can tinker with my data:

    cause,outage_start,outage_duration,incident_duration
    Cell management system bug,2024-07-30T21:45:00.000000+0000,0.2861111111111111,1.4951388888888888
    Latent software defect,2023-06-13T18:49:00.000000+0000,0.08055555555555555,0.15833333333333333
    Automated scaling activity,2021-12-07T15:30:00.000000+0000,0.2861111111111111,0.3736111111111111
    Network device operating system bug,2021-09-01T22:30:00.000000+0000,0.2583333333333333,0.2583333333333333
    Thread count exceeded limit,2020-11-25T13:15:00.000000+0000,0.7138888888888889,0.7194444444444444
    Datacenter cooling system failure,2019-08-23T03:36:00.000000+0000,0.24583333333333332,0.24583333333333332
    Configuration error removed setting,2018-11-21T23:19:00.000000+0000,0.058333333333333334,0.058333333333333334
    Command input error,2017-02-28T17:37:00.000000+0000,0.17847222222222223,0.17847222222222223
    Utility power failure,2016-06-05T05:25:00.000000+0000,0.3993055555555555,0.3993055555555555
    Network disruption triggering bug,2015-09-20T09:19:00.000000+0000,0.20208333333333334,0.20208333333333334
    Transformer failure,2014-08-07T17:41:00.000000+0000,0.13055555555555556,3.4055555555555554
    Power loss to servers,2014-06-14T04:16:00.000000+0000,0.08333333333333333,0.17638888888888887
    Utility power loss,2013-12-18T06:05:00.000000+0000,0.07013888888888889,0.11388888888888889
    Maintenance process error,2012-12-24T20:24:00.000000+0000,0.8270833333333333,0.9868055555555555
    Memory leak in agent,2012-10-22T17:00:00.000000+0000,0.26041666666666663,0.4930555555555555
    Electrical storm causing failures,2012-06-30T02:24:00.000000+0000,0.20902777777777776,0.25416666666666665
    Network configuration change error,2011-04-21T07:47:00.000000+0000,1.4881944444444444,3.592361111111111

thomas_witt2• 9 months ago

DynamoDB is performing fine in production in eu-central-1.

Seems to be really limited to us-east-1 (https://health.aws.amazon.com/health/status). I think they host a lot of console and backend stuff there.

okr2• 9 months ago

Yet. Everything goes down the ... Bach ;)

t0lo2• 9 months ago

Can't log into tidal for my music

antihero5• 9 months ago

Navidrome seems fine

kedihacker• 9 months ago

Only us east 1 gets new services immediately others might do but not a guarantee. Which regions are a good alternative

frays• 9 months ago

Robinhood's completely down. Even their main website: https://robinhood.com/

mittermayr2• 9 months ago

Amazing, I wonder what their interview process is like, probably whiteboarding a next-gen LLM in WASM, meanwhile, their entire website goes down with us-east-1... I mean.

1-62• 9 months ago

Friends tell their friends about more mature brokerages once the account goes over $100k.

zwnow6• 9 months ago

I love this to be honest. Validates my anti cloud stance.

flanked-evergl8• 9 months ago

No service that does not run on cloud has ever had outages.

Nextgrid13• 9 months ago

But at least a service that doesn't run on cloud doesn't pay the 1000% premium for its supposed "uptime".

zwnow6• 9 months ago

At least its in my control :)

emrodre• 9 months ago

Their status page (https://health.aws.amazon.com/health/status) says the only disrupted service is DynamoDB, but it's impacting 37 other services. It is amazing to see how big a blast radius a single service can have.

thmpp• 9 months ago

AWS engineers are trained to use their internal services for each new system. They seem to like using DynamoDB. Dependencies like this should be made transparent.

Nextgrid13• 9 months ago

Not sure why this is downvoted - this is absolutely correct.

A lot of AWS services under the hood depend on others, and especially us-east-1 is often used for things that require strong consistency like AWS console logins/etc (where you absolutely don't want a changed password or revoked session to remain valid in other regions because of eventual consistency).

bsjaux628• 9 months ago

Not "like using", they are mandated from the top to use DynamoDB for any storage. At my org in the retail page, you needed director approval if you wanted to use a relational DB for a production service.

myroon5• 9 months ago

Dependencies like this should be made transparent

even internally, Amazon's dependency graph became visually+logically incomprehensible a long time ago

stevepotter• 9 months ago

Ex employee here who built an aws service. Dynamo is basically mandated. You need like VP approval to use a relational database because of some scaling stuff they ran into historically. That sucks because we really needed a relational database and had to bend over backwards to use dynamo and all the nonsense associated with not having sql. It was super low traffic too

littlecranky674• 9 months ago

The same page now says 58 services - just 23 minutes after your post. Seems this is becoming a larger issue.

kalleboo3• 9 months ago

When I first visited the page it said like 23 services, now it says 65

nevada_scout• 9 months ago

It's now listing 58 impacted services, so the blast radius is growing it seems

jamesbelchamber5• 9 months ago

It's not surprising that it's impacting other services in the region because DynamoDB is one of those things that lots of other services build on top of. It is a little bit surprising that the blast radius seems to extend beyond us-east-1, mind.

In the coming hours/days we'll find out if AWS still have significant single points of failure in that region, or if _so many companies_ are just not bothering to build in redundancy to mitigate regional outages.

I'm looking forward to the RCA!

XorNot3• 9 months ago

I'm real curious how much of AWS GovCloud has continued through this actually. But even if it's fine, from a strategic perspective how much damage did we just discover you could do with a targeted disruption at the right time?

tonymet10• 9 months ago

i'm surprised / bothered that the history log shows the issues starting AM 10/20 -- when they seemed to have started around midnight 10/19

glemmaPaul2• 9 months ago

LOL making one db service a central point of failure, charge gold for small compute instances. Rage about needing Multi AZ, make the costs come onto the developer/organization. But, now fail on a region level, so are we going to now have multi-country setup for simple small applications?

philipallstar7• 9 months ago

Just don't buy it if you don't want it. No one is forced to buy this stuff.

benterix3• 9 months ago

No one is forced to buy this stuff.

Actually, many companies are de facto forced to do that, for various reasons.

DrScientist• 9 months ago

According to their status page the fault was in DNS lookup of the Dynamo services.

Everything depends on DNS....

mlrtime7• 9 months ago

Dynamo had a outage last year if I recall correctly.

xtracto• 9 months ago

Lol ... of course it's DNS fault again.

Hamuko• 9 months ago

I thought it was a pretty well-known issue that the rest of AWS depends on us-east-1 working. Basically any other AWS region can get hit by a meteor without bringing down everything else – except us-east-1.

yellow_lead• 9 months ago

But it seems like only us-east-1 is down today, is that right?

KettleLaugh• 9 months ago

We maybe distributed, but we die united...

glemmaPaul2• 9 months ago

AWS Communist Cloud

hangsi• 9 months ago

Divided we stand,

United we fall.

Tuna-Fish• 9 months ago

It's not DNS

There's no way it's DNS

It was DNS

codegladiator• 9 months ago

They haven't listed SES there yet in the affected services on their status page

t0lo2• 9 months ago

It's weird that we're living in a time where this could be a taste of a prolonged future global internet blackout by adversarial nations. Get used to this feeling I guess :)

munchlax2• 9 months ago

Nowadays when this happens it's always something. "Something went wrong."

Even the error message itself is wrong whenever that one appears.

urbandw311er2• 9 months ago

Displaying and propagating accurate error messages is an entire science unto itself... ...I can see why it's sometimes sensible to invest resource elsewhere and fall back to 'something'.

munchlax2• 9 months ago

IMHO if error handling is rocket science, the error is you

foobar19622• 9 months ago

I use the term “unexpected error” because if the code got to this alert it wasn’t caught by any traps I’d made for the “expected” errors.

zigzag312• 9 months ago

Reddit shows:

"Too many requests. Your request has been rate limited, please take a break for a couple minutes and try again."

world2vec• 9 months ago

Slack and Zoom working intermittently for me

werdl• 9 months ago

Looks like a DNS issue - dynamodb.us-east-1.amazonaws.com is failing to resolve.

lgats• 9 months ago

"Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1."

it seems they found your comment

socalgal2• 9 months ago

Amazon itself apperas to be out for some products. I get a "Sorry, We couldn't find that page" when clicking on products

andreygubarev3• 9 months ago

https://status.tailscale.com/ clients' auth down :( what a day

kondro9• 9 months ago

That just says the homepage and knowledge base are down and that admin access specifically isn’t effected.

andreygubarev3• 9 months ago

yep, admin panel works, but in practice my devices are logged out and there is no way to re-authorize them.

thecopy• 9 months ago

I did get 500 error from their public ECR too

goodegg2• 9 months ago

Happy Monday People

XCSme2• 9 months ago

Yeah, noticed from Zoom: https://www.zoomstatus.com/incidents/yy70hmbp61r9

goodegg2• 9 months ago

Terraform Cloud is having problem too

rirze2• 9 months ago

We just had a power outage in Ashburn starting at 10 pm Sunday night. It restored at 3:40am ish, and I know datacenters have redundant power sources but the timing is very suspicious. The AWS outage supposedly started at midnight

OliverGuy• 9 months ago

Their latest update on the status page says it's a Dynamodb DNS issue

shawabawa38• 9 months ago

but the cause of that could be anything, including some kind of config getting wiped due to a temporary power outage

Hilift• 9 months ago

Even with redundancy, the response time between NYC and Amazon East in Ashburn is something like 10 ms. The impedance mismatch and dropped packets and increased latency would doom most organizations craplications.

franktankbank2• 9 months ago

craplications

LOL

tdiff• 9 months ago

That strange feeling of the world getting cleaner for a while without all these dependant services.

andrewinardeer• 9 months ago

Signal not working here for me in AU

moribvndvs• 9 months ago

So, uh, over the weekend I decided to use the fact that my company needs a status checker/page to try out Elixir + Phoenix LiveView, and just now I found out my region is down while tinkering with it and watching Final Destination. That’s a little too on the nose for my comfort.

lbreakjai• 9 months ago

Well at least you don't have to figure out how to test your setup locally.

hobo_mark• 9 months ago

When did Snapchat move out of GCP?

dijit12• 9 months ago

They might have an implicit dependency on AWS, even if they're not primarily hosted there.

freeqaz2• 9 months ago

Since I'm 5+ years out from my NDA around this stuff, I'll give some high level details here.

Snapchat heavily used Google AppEngine to scale. This was basically a magical Java runtime that would 'hot path split' the monolithic service into lambda-like worker pools. Pretty crazy, but it worked well.

Snapchat leaned very heavily on this though and basically let Google build the tech that allowed them to scale up instead of dealing with that problem internally. At one point, Snap was >70% of all GCP usage. And this was almost all concentrated on ONE Java service. Nuts stuff.

Anyway, eventually Google was no longer happy with supporting this and the corporate way of breaking up is "hey we're gonna charge you 10x what did last year for this, kay?" (I don't know if it was actually 10x. It was just a LOT more)

So began the migration towards Kubernetes and AWS EKS. Snap was one of the pilot customers for EKS before it was generally available, iirc. (I helped work on this migration in 2018/2019)

Now, 6+ years later, I don't think Snap heavily uses GCP for traffic unless they migrated back. And this outage basically confirms that :P

garbthetill• 9 months ago

Thats so interesting to me, I always assume companies like google who have "unlimited" dollars will always be happy to eat the cost to keep customers, especially given gcp usage outside googles internal services is way smaller compared to azure and aws. Also interesting to see snapchat had a hacky solution with AppEngine

igleria• 9 months ago

funny that even if we have our app running fine in AWS europe, we are affected as developers because of npm/docker/etc being down. oh well.

dijit12• 9 months ago

AWS has made the internet into a single-point-of failure.

What's the point of all the auto-healing node-graph systems that were designed in the 70s and refined over decades: if we're just going to do mainframe development anyway?

voidUpdate2• 9 months ago

To be fair, there is another point of failure, Cloudflare. It seems like half the internet goes down when Cloudflare has one of their moments

antihero5• 9 months ago

My website on the cupboard laptop is fine.

goinggetthem• 9 months ago

This is from Amazon's latest earnings call when Andy Jessy was asked why they aren't growing as much as there competitors

"I think if you look at what matters to customers, what they care they care a lot about what the operational performance is, you know, what the availability is, what the durability is, what the latency and throughput is of of the various services. And I think we have a pretty significant advantage in that area." also "And, yeah, you could just you just look at what's happened the last couple months. You can just see kind of adventures at some of these players almost every month. And so very big difference, I think, in security."

JCM917• 9 months ago

Well that aged well

spprashant• 9 months ago

That was a bit of ramble. Not something I d expect from a CEO of AWS who probably handles press all the time.

It reminds me of that viral clip from a beauty pageant where the contestant went on a geographical ramble while the question was about US education.

ssehpriest• 9 months ago

Airtable is down as-well.

A lot of businesses have all their workflows depending on their data on airtable.

polaris64• 9 months ago

It looks like DNS has been restored: dynamodb.us-east-1.amazonaws.com. 5 IN A 3.218.182.189

miyuru• 9 months ago

I wonder if the new endpoint was affected as well.

dynamodb.us-east-1.api.aws

rafa___• 9 months ago

"Oct 20 2:01 AM PDT We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1..."

It's always DNS...

gadders• 9 months ago

Substack seems to by lying about their status: https://substack.statuspage.io/

littlecranky674• 9 months ago

Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony.

[0] https://news.ycombinator.com/item?id=45614922

bigblind2• 9 months ago

I don't have an opinion either way, but for now, this is just anecdotal evidence.

brazukadev4• 9 months ago

Looks fine for pointing an irony

DataDaemon2• 9 months ago

Finally IT managers will start understanding that cloud is no difference than Hetzner.

aembleton3• 9 months ago

When things go wrong, you can point at a news article and say its not just us that have been affected.

benterix3• 9 months ago

Well, we have a naming issue (Hetzner also has Hetzner Cloud, it looks people still equal cloud with the three biggest public cloud providers).

In any case, in order for this to happen, someone would have to collect reliable data (not all big cloud providers like to publish precise data, usually they downlplay the outages and use weasel words like "some customers... in some regions... might have experienced" just not to admit they had an outage) and present stats comparing the availability of Heztner Cloud vs the big three.

jpalomaki• 9 months ago

When AWS is down, everybody knows it. People don’t really question your hosting choice. It’s the IBM of cloud era.

sisve• 9 months ago

That is 100% true. You cant be fired for picking AWS... But I doubt its the best choice for most people. Sad but true

stefan_• 9 months ago

And yet they still all activate their on call people (wait why do we have them if we are on the cloud?) to do .. nothing at all.

petesergeant2• 9 months ago

100%. When AWS was down, we'd say "AWS is down!", and our customers would get it. Saying "Hetzner is down!" raises all sorts of questions your customers aren't interested in.

jamesbelchamber5• 9 months ago

To back up this point, currently BBC News have it as their most significant story, with "live" reporting: https://www.bbc.co.uk/news/live/c5y8k7k6v1rt

This is alongside "live" reporting on the Israel/Gaza conflict as well as news about Epstein and the Louvre heist.

This is mainstream news.

abujazar8• 9 months ago

That depends on the service. Far from everyone is on their PC or smartphone all day, and even fewer care about these kinds of news.

patshead• 9 months ago

On the other side of that coin, I am excited to be up and running while everyone else is down!

throw-10-133• 9 months ago

most people dont even know aws exists

JCM917• 9 months ago

Yes, but those days are numbered. For many years AWS was in a league of its own. Now they’ve fallen badly behind in a growing number of areas and are struggling to catch up.

There’s a ton of momentum associated with the prior dominance, but between the big misses on AI, a general slow pace of innovation on core services, and a steady stream of top leadership and engineers moving elsewhere they’re looking quite vulnerable.

antihero5• 9 months ago

Amazon is up, what are they doing?

citrin_ru• 9 months ago

On one hand it's allows to shift the blame but on other hand is shows a disadvantage of hyper centralization - if AWS is down too many important services are down at the same time which makes it worse. E. g. when AWS is down it's important to have communication/monitoring services UP so engineers can discuss / co-ordinate workarounds and have good visibility but Atlassian was (is) significantly degraded today too.

nutjob2• 9 months ago

Which eventually leads to the headline "AWS down indefinitely, society collapses".

neverminder• 9 months ago

You can argue about Hetzner's uptime, but you can 't argue about Hetzner's pricing which is hands down the best there is. I'd rather go with Hetzner and cobble up together some failover than pay AWS extortion.

Lio• 9 months ago

For the price of AWS you could run Hetzner, a second provider for resiliancy and still make a large saving.

Your margin is my opportunity indeed.

motorest10• 9 months ago

I'd rather go with Hetzner and cobble up together some failover than pay AWS extortion.

Comments like this are so exaggerated that they risk moving the goodwill needle back to where it was before. Hetzner offers no service that is similar to DynamoDB, IAM or Lambda. If you are going to praise Hetzner as a valid alternative during a DynamoDB outage caused by DNS configuration, you would need to a) argue that Hetzner is a better option regarding DNS outages, b) Hetzner is a preferable option for those who use serverless offers.

I say this as a long-time Hetzner user. Herzner is indeed cheaper, but don't pretend that Herzner let's you click your way into a highly-availale nosql data store. You need non-trivial levels of you're ow work to develop, deploy, and maintain such a service.

sreekanth8503• 9 months ago

TBH, in my last 3 years with Hetzner, i never saw a downtime to my servers other than myself doing some routin maitenance for os updates. Location Falkenstein.

briHass• 9 months ago

What is the Hetzner equivalent for those in Windows Server land? I looked around for some VPS/DS providers that specialize in Windows, and they all seem somewhat shady with websites that look like early 2000s e-commerce.

grebc3• 9 months ago

I got a downvote already for pointing this out :’)

brazukadev4• 9 months ago

Unfortunately, HN is full of company people, you can't talk anything against Google, Meta, Amazon, Microsoft without being downvoted to death.

bert-ye• 9 months ago

I work at a small / medium company with about ~20 dedicated servers and ~30 cloud servers at Hetzner. Outages have happened, but we were lucky that the few times it did happen, it was never a problem / actual downtime.

One thing to note is that there are some scheduled maintenances were we needed to react.

yard2010• 9 months ago

I'm not affiliated and won't be compensated in any way for saying this: Hetzner are the best business partners ever. Their service is rock solid, their pricing is fair, their support is kind and helpful.

Going forward I expect American companies to follow this European vibe, it's like the opposite of enshitification.

otikik• 9 months ago

the opposite of enshitification.

Why do you expect American companies to follow it then? >:)

koliber2• 9 months ago

My recommendation is to use AWS, but not the US-EAST-1 region. That way you get benefits of AWS without the instability.

dijit12• 9 months ago

AWS has internal dependencies on US-EAST-1.

Admittedly they're getting fewer and fewer, but they exist.

The same is also true in GCP, so as much as I prefer GCP from a technical standpoint: the truth is, if you can't see it, it doesn't mean it goes away.

maccard• 9 months ago

We have nothing deployed in us east 1, yet all of our CI was failing due to IAM errors this morning.

krsdcbl• 9 months ago

We've been running our services on Hetzner for 10 years, never experienced any significant outages.

That might be datacenter dependant of course, since our root servers and cloud services are all hosted in Europe, but I really never understood why Hetzner is said to be less reliable

YetAnotherNick• 9 months ago

Stop making things up. As someone who commented on the thread in favour of AWS, there is almost no mention of better uptime in any comment I could find.

I could find one or two downvoted or heavily critisized comments, but I can find more people mentioning the opposite.

jwr4• 9 months ago

As a data point, I've been running stuff at Hetzner for 10 years now, in two datacenters (physical servers). There were brief network outages when they replaced networking equipment, and exactly ONE outage for hardware replacement, scheduled weeks in advance, with a 4-hour window and around 1-2h duration.

It's just a single data point, but for me that's a pretty good record.

It's not because Hetzner is miraculously better at infrastructure, it's because physical servers are way simpler than the extremely complex software and networking systems that AWS provides.

jamesbelchamber5• 9 months ago

physical servers are way simpler than the extremely complex software and networking systems that AWS provides.

Or, rather, it's your fault when the complex software and networking systems you deployed on top of those physical servers go wrong (:

sapiogram• 9 months ago

Do you monitor your product closely enough to know that there weren't other brief outages? E.g. something on the scale of unscheduled server restarts, and minute-long network outages?

yread• 9 months ago

7 years, 20 servers, same here.

elktown• 9 months ago

Cloud marketing and career incentives seems to have instilled in the average dev that MTBF for hardware is in days rather than years.

saynay• 9 months ago

On the other hand, I had the misfortune of having a hardware failure on one of my Hetzner servers. They got a replacement harddrive in fairly quickly, but still complete data loss on that server, so I had to rebuild it from scratch.

This was extra painful, because I wasn't using one of the OS that is blessed by Hetzner, so it requires a remote install. Remote installs require a system that can run their Java web plugin, and that have a stable and fast enough connection to not time out. The only way I have reliably gotten them to work is by having an ancient Linux VM that was also running in Hetzner, and had the oldest Firefox version I could find that still supported Java in the browser.

My fault for trying to use what they provide in a way that is outside their intended use, and props to them for letting me do it anyway.

bradhe6• 9 months ago

Hetzner users are like the Linux users for cloud.

jijijijij2• 9 months ago

Love, after your laptop's wifi antenna, it's Linux users all the way down.

cdelsolar3• 9 months ago

Btw I use Hertzner

me551ah3• 9 months ago

I’m more curious to understand how we ended up creating a single point of failure across the whole internet.

ffsm8• 9 months ago

Haha, yeah that's a nugget

99.99% uptime infra significantly cheaper than the cloud.

I guess that's another person that has never actually worked in the domain (SRE/admin) but still wants to talk with confidence on the topic.

Why do I say that? Because 99.99% is frickin easy

That's almost one full hour of complete downtime per year.

It only gets hard in the 99.9999+ range ... And you rarely meet that range with cloud providers either as requests still fail for some reason, like random 503 when a container is decommissioned or similar

kbrkbr• 9 months ago

It still can be true that the uptime is better, or am I overlooking something?

aiiizzz2• 9 months ago

Nah you're definitely correct.

notepad0x90• 9 months ago

I don't know how often Hetzner has similar outages, but outages at the rack and server level, including network outages and device failure happen for individual customers. If you've never experienced this, it is probably just survivor's bias.

Aws/cloud has similar outages too, but more redundancy and automatic failover/migrations that are transparent to customers happen. You don't have to worry about DDOS and many other admin burdens either.

YMMV, I'm just saying sometimes Aws makes sense, other times Hetzner does.

lleu• 9 months ago

Been using OVH here with no complaints.

nonethewiser2• 9 months ago

Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony.

That's not necessarily ironic. Seems like you are suffering from recency bias.

stepri• 9 months ago

“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.”

It’s always DNS.

commandersaki• 9 months ago

Someone probably failed to lint the zone file.

DrewADesign2• 9 months ago

DNS strikes me as the kind of solution someone designed thinking “eh, this is good enough for now. We can work out some of the clunkiness when more organizations start using the Internet.” But it just ended up being pretty much the best approach indefinitely.

bayindirh• 9 months ago

Even when it's not DNS, it's DNS.

JoBrad• 9 months ago

Sometimes it’s BGP.

shamil0xff• 9 months ago

Might just be BGP dressed as DNS

Nextgrid13• 9 months ago

I wonder how much of this is "DNS resolution" vs "underlying config/datastore of the DNS server is broken". I'd expect the latter.

huflungdung• 9 months ago

I don’t think it is DNS. The DNS A records were 2h before they announced it was DNS but _after_ reporting it was a DNS issue.

wdfx• 9 months ago

... wonders if the dns config store is in fact dynamodb ...

babarjaana• 9 months ago

Dumb question but what's the difference between the two? If the underlying config is broken then DNS resolution would fail, and that's basically the only way resolution fails, no?

paulddraper2• 9 months ago

Those aren't really that different.

That's a major way your DNS stops working.

koliber2• 9 months ago

It's always US-EAST-1 :)

us0r• 9 months ago

Or expired domains which I suppose is related?

oneeyedpigeon2• 9 months ago

Downtime Never Stops!

indycliff• 9 months ago

the answer is always DNS

dexterdog3• 9 months ago

That's why they wrote the haiku

nijave6• 9 months ago

I don't think that's necessarily true. The outage updates later identified failing network load balancers as the cause--I think DNS was just a symptom of the root cause

I suppose it's possible DNS broke health checks but it seems more likely to be the other way around imo

lkjdsklf• 9 months ago

I don’t work for aws, but a different cloud provider so this is not a description of this incident, but an example of the kind of thing that can happen

One particular “dns” issue that caused an outage was actually a bug in software that monitors healthchecks.

It would actively monitor all servers for a particular service (by updating itself based on what was deployed) and update dns based on those checks.

So when the health check monitors failed, servers would get removed from dns within a few milliseconds.

Bug gets deployed to health check service. All of a sudden users can’t resolve dns names because everything is marked as unhealthy and removed from dns.

So not really a “dns” issue, but it looks like one to users

tomaytotomato• 9 months ago

Slack, Jira and Zoom are all sluggish for me in the UK

kalleboo3• 9 months ago

I wonder if that's not due to dependencies on AWS but all-hands-on-deck causing far more traffic than usual

disposable2020• 9 months ago

I seem to recall other issues around this time in previous years. I wonder if this is some change getting shoe-horned in ahead of some reinvent release deadline...

devttyeu• 9 months ago

Can't update my selfhosted HomeAssistant because HAOS depends on dockerhub which seems to be still down.

grk• 9 months ago

Does anyone know if having Global Accelerator set up would help right now? It's in the list of affected services, I wonder if it's useful in scenarios like this one.

XorNot3• 9 months ago

Well that takes down Docker Hub as well it looks like.

AshLeece• 9 months ago

Yep, was just thinking the same when my Kubernetes failed a HelmRelease due to a pull error…

killingtime74• 9 months ago

Signal is down for me

miduil• 9 months ago

Yes. https://status.signal.org/

    >  Signal is experiencing technical difficulties. We are working hard to restore service as quickly as possible.

Edit: Up and running again.

grenran• 9 months ago

seems like services are slowly recovering

kitd• 9 months ago

O ffs. I can't even access the NYT puzzles in the meantime ... Seriously disrupted, man

littlecranky674• 9 months ago

Maybe unrelated, but yesterday I went to pick up my package from an Amazon Locker in Germany, and the display said "Service unavailable". I'll wait until later today before I go and try again.

HighGoldstein• 9 months ago

I wonder why a package locker needs connectivity to give you a package. Since your package can't be withdrawn again from a different location, partitioning shouldn't be an issue.

jcgl• 9 months ago

Generally speaking, it's easier to have computation (logic, state, etc.) centralized. If the designers didn't prioritize scenarios where decentralization helped, then centralization would've been the better option.

bstsb• 9 months ago

glad all my services are either Hetzner servers or EU region of AWS!

gbalduzzi• 9 months ago

Twilio is down worldwide: https://status.twilio.com/

croemer4• 9 months ago

Coinbase down as well

croemer4• 9 months ago

Coinbase down as well: https://status.coinbase.com/

littlecranky674• 9 months ago

Best option for a whale to manipulate the price again.

00deadbeef• 9 months ago

It's not DNS

There's no way it's DNS

It was DNS

foobar19622• 9 months ago

That or a Windows update.

secondcoming2• 9 months ago

Or unattended-upgrades

o1o1o1• 9 months ago

I'm so happy we chose Hetzner instead but unfortunately we also use Supabase (dashboard affected) and Resend (dashboard and email sending affected).

Probably makes sense to add "relies on AWS" to the criteria we're using to evaluate 3rd-party services.

countWSS• 9 months ago

Reddit itself breaking down and errors appear. Does reddit itself depends on this?

alvis• 9 months ago

Why would us-east-1 cause many UK banks and even UK gov web sites down too!? Shouldn't they operate in the UK region due to GDPR?

Nextgrid13• 9 months ago

2 things:

1) GDPR is never enforced other than token fines based on technicalities. The vast majority of the cookie banners you see around are not compliant, so it the regulation was actually enforced they'd be the first to go... and it would be much easier to go after those (they are visible) rather than audit every company's internal codebases to check if they're sending data to a US-based provider.

2) you could technically build a service that relies on a US-based provider while not sending them any personal data or data that can be correlated with personal data.

Ylpertnodi2• 9 months ago

GDPR is never enforced Yes it is.
other than token fines based on technicalities. Result!

GoblinSlayer4• 9 months ago

Integration with USA for your safety :)

assimpleaspossi2• 9 months ago

I'm thinking about that one guy who clicked on "OK" or hit return.

rvitorper2• 9 months ago

Somebody, somewhere tried to rollback something and it failed

starkindustries3• 9 months ago

It has started recovering now. https://www.whatsmydns.net/#A/dynamodb.us-east-1.amazonaws.c... is showing full recovery of dns resolutions.

mk892• 9 months ago

It's fun to see SRE jumping left and right when they can do basically nothing at all.

"Do we enable DR? Yes/No". That's all you can do. If you do, it's a whole machinery starting, which might take longer than the outage itself.

They can't even use Slack to communicate - messages are being dropped/not sent.

And then we laugh at the South Koreans for not having backed up their hard drives (which got burnt by actual fire, a statistically way less occurring event than an AWS outage). OK that's a huge screw up, but hey, this is not insignificant either.

What will happen now? Nothing, like nothing happened after Crowdstrike's bug last year.

XorNot3• 9 months ago

Signal seems to be dead too though, which is much more of a WTF?

GoblinSlayer4• 9 months ago

A decentralized messenger is Tox.

assimpleaspossi2• 9 months ago

As of 4:26am Central Time in the USA, it's back up for one of my services.

croemer4• 9 months ago

martinheidegger• 9 months ago

Designed to provide 99.999% durability and 99.999% availability Still designed, not implemented

riknos3143• 9 months ago

The real challenge is that implementations aren't static.

Just because today's implementation has 4 9s that doesn't mean tomorrow's will...

gritzko• 9 months ago

idiocracy_window_view.jpg

the_duke• 9 months ago

Slack now also down: https://slack-status.com/

jpfromlondon• 9 months ago

This will always be a risk when sharecropping.

shakesbeard• 9 months ago

Slack (canvas and huddles), Circle CI and Bitbucket are also reporting issues due to this.

raspasov• 9 months ago

02:34 Pacific: Things seem to be recovering.

karel-3d3• 9 months ago

Slack is down. Is that related? Probably is.

rob• 9 months ago

My Alexa is hit or miss at responding to queries right now at 5:30 AM EST. Was wondering why it wasn't answering when I woke up.

comrade1234• 9 months ago

I like that we can advertise to our customers that over the last X years we have better uptime than Amazon, google, etc.

eska• 9 months ago

Just yesterday I saw another Hetzner thread where someone claimed AWS beats them in uptime and someone else blasted AWS for huge incidents. I bet his coffee tastes better this morning.

VBprogrammer• 9 months ago

I honestly wonder if there is safety in the herd here. If you have a dedicated server in a rack somewhere that goes down and takes your site with it. Or even the whole data center has connectivity issues. As far as the customer is concerned, you screwed up.

If you are on AWS and AWS goes down, that's covered in the news as a bunch of billion dollar companies were also down. Customer probably gives you a pass.

stavros3• 9 months ago

To be fair, my Hetzner server had ten minutes of downtime the other day. I've been a customer for years and this was the second time or so, so I love Hetzner, but everything has downtime.

tgv• 9 months ago

The Register calls it Microsoft 364, 363, ...

SteveNuts4• 9 months ago

365 "eights" of uptime per year.

kingstnap• 9 months ago

Reported uptimes are little more than fabricated bullshit.

They measure uptime using averages of "if any part of a chain is even marginally working".

People experience downtime however as "if any part of a chain is degraded".

uyzstvqs• 9 months ago

I'd say that this is true for the average admin who considers PaaS, Kubernetes and microservices one giant joke. Vendor-neutral monolithic deployments keep on winning.

notahacker• 9 months ago

Feel bad for the Amazon SDR randomly pitching me AWS services today. Although apparently our former head of marketing got that pitch from four different LinkedIn accounts. Maybe there's a cloud service to rein them in that broke ;)

Aldipower5• 9 months ago

My minor 2000 users web app hosted on Hetzner works fyi. :-P

mlrtime7• 9 months ago

But how are you going to web scale it!? /s

Aldipower5• 9 months ago

Web scale? It is an _web_ app, so it is already web scaled, hehe.

Seriously, this thing runs already on 3 servers. A primary + backup and a secondary in another datacenter/provider at Netcup. DNS with another AnycastDNS provider called ClouDNS. Everything still way cheaper then AWS. The database is already replicated for reads. And I could switch to sharding if necessary. I can easily scale to 5, 7, whatever dedicated servers. But I do not have to right now. The primary is at 1% (sic!) load.

There really is no magic behind this. And you have to write your application in a distributable way anyway, you need to understand the concepts of stateless, write-locking, etc. also with AWS.

aembleton3• 9 months ago

Right up until the DNS fails

Aldipower5• 9 months ago

I am using ClouDNS. That is an AnycastDNS provider. My hopes are that they are more reliable. But yeah, it is still DNS and it will fail. ;-)

seeg• 9 months ago

quay.io was down: https://status.redhat.com

gramakri22• 9 months ago

quay.io is down

schaum• 9 months ago

It was in read only mode,now available again

dddfdfdfdfdf2• 9 months ago

Hello world

dddfdfdfdfdf2• 9 months ago

hello world

drcongo• 9 months ago

Snow day!

throw-10-133• 9 months ago

this is why you avoid us-east-1

philipwhiuk• 9 months ago

And yet you're still impacted because it hosts IAM

danielpetrica2• 9 months ago

In this moments I think devs should invest in vendor independence if they can. While I'm not to that stage yet (cloudlfare dependence) using open technologies like docker (or Kubernetes), Traefik instead of managed services can help in this disaster situations by switching to a different provider in a faster way than having to rebuild from zero. as a disclosure I'm not still to that point on my infrastructure But I'm trying to slowly define one for my self

pinkgolem2• 9 months ago

Fyi: traffic also had problems bringing new devices online

danielpetrica2• 9 months ago

I'm speacking of the self hosted version you can install on your own vps not the managwment version. I don't like using managed services if possible.

shawn_w2• 9 months ago

One of the radio stations I listen to is just dead air tonight. I assume this is the cause.

myself248• 9 months ago

A physical on-air broadcast station, not a web stream? That likely violates a license; they're required to perform station identification on a regular basis.

Of course if they had on-site staff it wouldn't be an issue (worst case, just walk down to the transmitter hut and use the transmitter's aux input, which is there specifically for backup operations like this), but consolidation and enshittification of broadcast media mean there's probably nobody physically present.

shawn_w2• 9 months ago

Yeah, real over the air fm radio. This particular station is a Jack one owned by iHeart; they don't have DJs. Probably no techs or staff in the office overnight.

JCM917• 9 months ago

US-East-1 is literally the Achilles Heel of the Internet.

rvitorper2• 9 months ago

Exactly

sofixa6• 9 months ago

You would think that after the previous big us-east-1 outages (to be fair there have been like 3 of them in the past decade, but still, that's plenty), companies would have started to move to other AWS regions and/or to spread workloads between them.

JCM917• 9 months ago

It’s not that simple. The bit AWS doesn’t talk much about publicly (but will privately if you really push them) is that there’s core dependencies behind the scenes on us-east-1 for running AWS itself. When us-east-1 goes down the blast radius has often impacted things running in other regions.

It impacts AWS internally too. For example rather ironically it looks like the outage took out AWS’s support systems so folks couldn’t contact support to get help.

Unfortunately it’s not as simple as just deploying in multiple regions with some failover load balancing.

BaudouinVH• 9 months ago

canva.com was down until a few minutes ago.

codebolt2• 9 months ago

Atlassian cloud is also having issues. Closing in on the 3 hour mark.

ryanmcdonough2• 9 months ago

Now, I may well be naive - but isn't the point of these systems that you fail over gracefully to another data centre and no-one notices?

spicybright• 9 months ago

It should be! When I was a complete newbie at AWS my first question was why do you have to pick a region, I thought the whole point was you didn't have to worry about that stuff

nicce2• 9 months ago

As far as I know, region selection is about regulation and privacy and guarantees on that.

sofixa6• 9 months ago

another data centre

Yes, within the same region. Doing stuff cross-region takes a little bit more effort and cost, so many skip it.

pjmlp5• 9 months ago

Assuming that the service actually bothered to have multiple regions as fallbacks configured.

tosh2• 9 months ago

SES and signal seem to work again

jacquesm20• 9 months ago

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?'

Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength.

hvb210• 9 months ago

The internet got its main strengths from the fact that it was completely decentralized.

Decentralized in terms of many companies making up the internet. Yes we've seen heavy consolidation in now having less than 10 companies make up the bulk of the internet.

The problem here isn't caused by companies chosing one cloud provider over the other. It's the economies of scale leading us to few large companies in any sector.

jacquesm20• 9 months ago

I think one reason is that people are just bad at statistics. Chance of materialization * impact = small. Sure. Over a short enough time that's true for any kind of risk. But companies tend to live for years, decades even and sometimes longer than that. If we're going to put all of those precious eggs in one basket, as long as the basket is substantially stronger than the eggs we're fine, right? Until the day someone drops the basket. And over a long enough time span all risks eventually materialize. So we're playing this game, and usually we come out ahead.

But trust me, 10 seconds after this outage is solved everybody will have forgotten about the possibility.

lentil_soup• 9 months ago

Decentralized in terms of many companies making up the internet

Not companies, the protocols are decentralized and at some point it was mostly non-companies. Anyone can hook up a computer and start serving requests which was/is a radical concept, we've lost a lot, unfortunately

psychoslave3• 9 months ago

Well, that is exactly what resilient distributed network are about. Not that much the technical details we implement them through, but the social relationship and balanced in political decision power.

Be it a company or a state, concentration of power that exeeds the needs for their purpose to function by a large margin is always a sure way to spread corruption, create feedback loop in single point of failure, and is buying everyone a ticket to some dystopian reality with a level of certainty that beats anything that a SLA will ever give us.

chasd0011• 9 months ago

Decentralized with respect to connectivity. If a construction crew cuts a fiber bundle routing protocols will route around the damage and packets keep showing up at the destination. Or, only a localized group of users will be affected. That level of decentralization is not what we have at higher levels in the stack with AWS being a good example.

Even connectivity has it's points of failure. I've touched with my own hands fiber runs that, with a few quick snips from a wire cutter, could bring sizable portions of the Internet offline. Granted that was a long time ago so those points of failure may no longer exist.

padjo5• 9 months ago

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

jacquesm20• 9 months ago

Thank you for illustrating my point. You didn't even bother to read the second paragraph.

antihero5• 9 months ago

My website running on an old laptop in my cupboard is doing just fine.

davedx2• 9 months ago

I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

Absurd claim.

Multi-region and multi-cloud are strategies that have existed almost as long as AWS. If your company is in anything finance-adjacent or critical infrastructure then you need to be able to cope with a single region of AWS failing.

mlrtime7• 9 months ago

Yes it does happen but very rarely to the tune of a few hours every 5-10 years.

It is rare, but it happens at LEAST 2-3x a year. AWS us-east-1 has a major incident affecting multiple services (that affect most downstream aws services) multiple times a year. Usually never the same root cause.

Not very many people realize that there are some services that still run only in us-east-1.

lucideer2• 9 months ago

to the tune of a few hours every 5-10 years

I presume this means you must not be working for a company running anything at scale on AWS.

sreekanth8503• 9 months ago

Depends on how serious you are with SLA's.

kelseydh3• 9 months ago

It seems like this can be mostly avoided by not using us-east-1.

psychoslave3• 9 months ago

This is planning future based on best things in the past. Not completely irrational and if you can't afford plan B, okayish.

But thinking AWS SLA is granted forever, and everyone should put all its eggs in it because "everyone do it" is neither wise not safe. Those who can afford it, and there are many businesses like that out there, should have a plan B. And actually AWS should not necessarily be plan A.

Nothing is forever. Not the Roman empire, not Inca empire, not china dynasties, not USA geological supremacy. That's not a question of if but when. It doesn't need to be through a lot of suffering, but if we don't systematically organise for a humanity which spread well being for everyone in a systematically resilient way, we will make it through more lot of tragic consequences when this or that single point of failure finally falls.

Waterluvian3• 9 months ago

Without having a well-defined risk profile that they’re designing to satisfy, everyone’s just kind of shooting from the hip with their opinions on what’s too much or too little.

DiffEq• 9 months ago

Maybe; but Parlar had no plan and are now nothing....because AWS decided to shut them off. Always have a good plan...

nucleardog2• 9 months ago

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years.

Not only that, but as you're seeing with this and the last few dozen outages... when us-east-1 goes down, a solid chunk of what many consumers consider the "internet" goes down. It's perceived less as "app C is down" and more is "the internet is broken today".

YouAreWRONGtoo3• 9 months ago

More like 2-3 times per year and this is not counting smaller outages or simply APIs that don't do what they document.

kxrm• 9 months ago

Completely agree, but I think companies need to be aware of the AWS risks with third parties as well. Many services were unable to communicate with customers.

Hosting your services on AWS while having a status page on AWS during an AWS outage is an easily avoidable problem.

snowwrestler• 9 months ago

I would take the opposite view, the little AWS outages are an opportunity to test your disaster recovery plan, which is worth doing even if it takes a little time.

It’s not hard to imagine events that would keep AWS dark for a long period of time, especially if you’re just in one region. The outage today was in us-east-1. Companies affected by it might want to consider at least geographic diversity, if not supplier diversity.

zaphirplane• 9 months ago

tune of a few hours every 5-10 years

You know that’s not true, is-east-1 last one was 2 years ago? But other services have bad days and foundational one drag others a long

Spooky23• 9 months ago

Sure, if your blog or whatever goes down who cares. But otherwise you should thinking about disaster planning and resilience.

AWS US-East 1 has many outages. Anything significant should account for that.

lumost• 9 months ago

This depends on the scale of company. A fully functional DR plan probably costs 10% of the infra spend + people time for operationalization. For most small/medium businesses its a waste to plan for a once per 3-10 year event. If you’re a large or legacy firm the above costs are trivial and in some cases it may become a fiduciary risk not to take it seriously.

pyrale3• 9 months ago

What if AWS dumps you because your country/company didn't please the commander in chief enough?

If your resilience plan is to trust a third party, that means you don't really care about going down, does it?

Besides that, as the above poster said, the issue with top tier cloud providers (or cloudflare, or google, etc) is not just that you rely on them, it is that enough people rely on them that you may suffer even if you don't.

temperceve• 9 months ago

Depends on the business. For 99% of them this is for sure the right answer.

throw0101d3• 9 months ago

Planning for an AWS outage […]

What about if your account gets deleted? Or compromised and all your instances/services deleted?

I think the idea is to be able to have things continue running on not-AWS.

maerF0x0• 9 months ago

Using AWS instead of a server in the closet is step 1.

Step 2 is multi-AZ

Step 3 is multi-region

Step 4 is multi-cloud.

Each company can work on it's next step, but most will not have positive EROI going from 2 to 3+

dangoldin• 9 months ago

I worked at an adtech company where we invested a bit in HA across AZ + regions. Lo and behold there was an AWS outage and we stayed up. Too bad our customers didn't and we still took the revenue hit.

Lesson here is that your approach will depend on your industry and peers. Every market will have their won philosophy and requirements here.

delfinom• 9 months ago

In before meteor strike takes a AWS region and they cant restore data.

coffeebeqn3• 9 months ago

We started that planning process at my previous company after one such outage but it became clear very quickly that the costs of such resilience would be 2-3x hosting costs in perpetuity and who knows how many manhours. Being down for an hour was a lot more palatable to everyone

indoordin0saur6• 9 months ago

Been doing this for about 8 years and I've worked through a serious AWS disruption at least 5 times in that time.

croes3• 9 months ago

Telefonica is moving it 5G core network to AWS

https://aws.amazon.com/blogs/industries/o2-telefonica-moves-...

A few hours could be a problem.

Not to mention it creates valuable a single point of failure for a hostile attack.

raincole2• 9 months ago

Most companies just aren't important enough to worry about "AWS never come back up." Planning for this case is just like planning for a terrorist blowing up your entire office. If you're the Pentagon sure you'd better have a plan for that. But most companies are not the Pentagon.

Frieren• 9 months ago

Most companies just aren't important enough to worry about "AWS never come back up."

But a large enough of "not too big to fail" companies become a too big to fail event. Too many medium sized companies have a total dependency on AWS or Google or Microsoft, if not several at the same time.

We live in a increasingly fragile society, one step closer to critical failure because big tech is not regulated in the same way than other infrastructure.

paulddraper2• 9 months ago

Exactly.

And FWIW, "AWS is down"....only one region (out of 36) of AWS is down.

You can do the multi-region failover, though that's still possibly overkill for most.

anal_reactor3• 9 months ago

First, planning for AWS outage is pointless. Unless you provide service of national security or something, your customers are going to understand that when there's global internet outage your service doesn't work either. The cost of maintaining a working failover across multiple cloud providers is just too high compared to potential benefits. It's astonishing that so few eningeers understand the fact that maintaining a technically beautiful solution costs time and money, which might not make a justified business case.

Second, preparing for the disappearance of AWS is even more silly. The chance that it will happen are orders of magnitude smaller than the chance that the cost of preparing for such an event will kill your business.

Let me ask you: how do you prepare your website for the complete collapse of western society? Will you be able to adapt your business model to post-apocalyptic world where it's only cockroaches?

jacquesm20• 9 months ago

Let me ask you: how do you prepare your website for the complete collapse of western society?

How did we go from 'you could lose your AWS account' to 'complete collapse of western society'? Do websites even matter in that context?

Second, preparing for the disappearance of AWS is even more silly.

What's silly is not thinking ahead.

psychoslave3• 9 months ago

Let me ask you: how do you prepare your website for the complete collapse of western society?

That's the main topic that going through my mind lately, if you replace "my website" with "Wikimedia movement".

We need a far better social, juridical and technical architecture regarding resilience as hostil agendas are in the rise at all level agaisnt sourced trackable global volunteer community knowledge bases.

csomar• 9 months ago

Now imagine for a bit that it will never come back up.

Given the current geopolitical circumstances, that's not a far fetched scenario. Especially for us-east-1; or anything in the D.C. metro area.

pmontra• 9 months ago

In the case of a costumer of mine the AWS outage manifested itself as Twilio failing to deliver SMSes. The fallback plan has been disabling the rotation of our two SMS providers and sending all messages with the remaing one. But what if the other one had something on AWS too? Or maybe both of them have something else vital on Azure, or Google Cloud, which will fail next week and stop our service. Who knows?

For small and medium sized companies it's not easy to perform an accurate due diligency.

freetanga• 9 months ago

Additionally I find that most Hyperscalers are trying to lock you in, by tailoring services which are industry standard with custom features which end up building roots and making a multi-vendor or lift and shift problematic.

Need to keep eyes peeled at all levels or the organization as many of these enter through day-to-day…

jacquesm20• 9 months ago

Yes, they're really good at that. This is just 'embrace and extend'. We all know the third.

invalidusernam3• 9 months ago

What if the fall-back also never comes back up?

ho_schi2• 9 months ago

The internet is a weak infrastructure, relying on a few big cables and data centers. And through AWS and Cloudflare it has become worse? Was it ever true, that the internet is resilient? I doubt.

Resilient systems work autonomously and can synchronize - but don't need to synchronize.

    * Git is resilient.
    * Native E-Mail clients - with local storage enabled - are somewhat resilient.
    * A local package repository is - somewhat resilient.
    * A local file-sharing app (not Warp/ Magic-Wormhole -> needs relay) is resilient if it uses only local WiFi or Bluetooth.

We're building weak infrastructure. A lot of stuff shall work locally and only optionally use the internet.

CaptainOfCoit6• 9 months ago

The internet seems resilient enough for all intents and purposes, we haven't had a global internet-wide catastrophe impacting the entire internet as far as I know, but we have gotten close to it sometimes (thanks BGP).

But the web, that's the fragile, centralized and weak point currently, and seems to be what you're referring to rather.

Maybe nitpicky, but I feel like it's important to distinguish between "the web" and "the internet".

smaudet2• 9 months ago

If you take into account the "the web" vs "the internet" as others have mentioned.

Yes the Internet has stayed stable.

The Web, as defined by a bunch of servers running complex software, probably much less so.

Just the fact that it must necessarily be more complex means that it has more failure modes...

bombcar3• 9 months ago

The Internet was much more resilient when it was just that - an internetwork of connected networks; each of which could and did operate autonomously.

Now we have computers that shit themselves if DNS isn’t working, let alone LANs that can operate disconnected from the Internet as a whole.

And partially working or indicating this it works (when it doesn’t) is usually even worse.

rco8786• 9 months ago

If AWS goes down unexpectedly and never comes back up it's much more likely that we're in the middle of some enormous global conflict where day to day survival takes priority over making your app work than AWS just deciding to abandon their cloud business on a whim.

apexalpha• 9 months ago

Or Trump decided your country does not deserve it.

CaptainOfCoit6• 9 months ago

Can also be much easier than that. Say you live in Mexico, hosting servers with AWS in the US because you have US customers. But suddenly the government decides to place sanctions on Mexico, and US entities are no longer allowed to do business with Mexicans, so all Mexican AWS accounts get shut down.

For you as a Mexican the end results is the same, AWS went away, and considering there already is a list of countries that cannot use AWS, GitHub and a bunch of other "essential" services, it's not hard to imagine that that list might grow in the future.

chasd0011• 9 months ago

what's most realistic is something like a major scandal at AWS. The FBI seizes control and no bytes come in our out until the investigation is complete. A multi-year total outage effectively.

Keyframe2• 9 months ago

At least we've got github steady with our code and IaaC, right? Right?!

OfflineSergio• 9 months ago

I don't think its worth it, but lets say I did it, what if others that I depend on dont do it? I still won't be fully functional and only one of us have spent a bunch of money.

bschne2• 9 months ago

I find this hard to judge in the abstract, but I'm not quite convinced the situation for the modal company today is worse than their answer to "what if your colo rack catches fire" would have been twenty years ago.

jacquesm20• 9 months ago

"what if your colo rack catches fire"

I've actually had that.

https://www.webmasterworld.com/webmaster/3663978.htm

rglover• 9 months ago

It would behoove a lot of devs to learn the basics of Linux sysadmin and how to setup a basic deployment with a VPS. Once you understand that, you'll realize how much of "modern infra" is really just a mix of over-reliance on AWS and throwing compute at underperforming code. Our addiction to complexity (and burning money on the illusion of infinite stability) is already and will continue to strangle us.

kbar13• 9 months ago

the correct answer for those companies is "we have it on the roadmap but for right now accept the risk"

saltyoldman2• 9 months ago

Contrast this with the top post.

JamesSwift3• 9 months ago

What good is jumping through extraordinary hoops to be multi cloud if docker, netlify, stripe, intercom, npm, etc all go down along with us-east-1?

fisf3• 9 months ago

Because you should not depend on one payment provider and pull unvendored images, packages, etc directly into your deployment.

There is no reason to have such brittle infra.

bongodongobob• 9 months ago

You simply cannot avoid it. There are so many applications and services that use AWS. Companies cant sit on 100% in-house software stacks.

greatgib• 9 months ago

When I follow the link, I arrive on a "You broke reddit" page :-o

JCM917• 9 months ago

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere.

The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

GoblinSlayer4• 9 months ago

But then you will be affected by outages of every dependency you use.

caymanjim2• 9 months ago

This is the real problem. Even if you don't run anything in AWS directly, something you integrate with will. And when us-east-1 is down, it doesn't matter if those services are in other availability zones. AWS's own internal services rely heavily on us-east-1, and most third-party services live in us-east-1.

It really is a single point of failure for the majority of the Internet.

cmiles84• 9 months ago

This. When Andy Jassy got challenged by analysts on the last earnings call on why AWS has fallen so far behind on innovation in areas his answer was a hand wavy response that diverted attention to say AWS is durable, stable, and reliable and customers care more about that. Oops.

judahmeek• 9 months ago

behind on innovation how exactly?

wrasee• 9 months ago

Please tell me there was a mixup and for some reason they didn’t show up.

tete2• 9 months ago

The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

This is and was never true. I've done setups in the past where monitoring happened "multi cloud" with also multiple dedicated servers. Was pretty broad so you could actually see where things broke.

Was quite some time ago so I don't have the data, but AWS never came out on top.

It actually matched largely with what netcraft.com put out. Not sure if they still do that and release those things to the public.

chaostheory2• 9 months ago

This makes sense given all the open source projects coming out of Netflix like chaos monkey.

testplzignore• 9 months ago

Netcraft confirmed it? I haven't heard that name since the Slashdot era :)

eric-hu• 9 months ago

Which cloud provider came out on top?

ifwinterco3• 9 months ago

Everything except us-east-1 is generally pretty reliable. At $work we have a lot of stuff that's only on eu-west-1 (yes not the best practice) and we haven't had any issues, touch wood

hnfong• 9 months ago

Just curious, what's special about us-east-1?

ttul3• 9 months ago

My impression is that `us-east-1` has the worst reliability track record of any region. We've always run our stuff in `us-west-2` and there has never been an outage that took us down in that region. By contrast, a few things that we had in `us-east-1` have gone down repeatedly.

1-62• 9 months ago

Glad that you're taking the first step toward resiliency. At times, big outages like these are necessary to give a good reason why the company should Multicloud. When things are working without problems, no one cares to listen to the squeaky wheel.

BoredPositron3• 9 months ago

Still no serverless inference for models or inference pipes that are not available on bedrock, still no auto scaling GPU workers. We started bothering them in 2022...crickets

llmslave3• 9 months ago

AWS has been in long term decline, most of the platform is just in keeping the lights on mode. Its also why they are behind on AI, alot of would be innovative employees get crushed under red tape and performance management

nextworddev• 9 months ago

Good thing they are the biggest investor into Anthropic

radium3d3• 9 months ago

Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.

CobrastanJorji• 9 months ago

Multi-cloud. It's fairly unlikely that AWS and Google Cloud are going to fail at the same time.

cakeday• 9 months ago

Host in house?

Yes, mostly.

grogers2• 9 months ago

Certainly if you aren't even multi-region, then multi-cloud is a pipe dream

bean4692• 9 months ago

What are you going to do? Host in house?

Yep. Although it's just anecdata, it's what we do where I work - haven't had a slightest issue in years.

Breza3• 9 months ago

Why not host in house? If you have an application with stable resource needs, it can often be the cheaper and more stable option. At a certain scale, you can buy the servers, hire a sysadmin, and still spend less money than relying on AWS.

If you have an app that experiences 1000x demand spikes at unpredictable times then sure, go with the cloud. But there are a lot of companies that would be better off if they seriously considered their options before choosing the cloud for everything.

lootgraft• 9 months ago

The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud.

If an internal "AWS team" then this translates to "I am comfortable using this tool, and am uninterested in having to learn an entirely new stack."

If you have to diversify your cloud workloads give your devops team more money to do so.

jen202• 9 months ago

I would be interested in a follow up in 2-3 years as to whether you've had fewer issues with a multi-cloud setup than just AWS. My suspicion is that will not be the case.

ej_campbell5• 9 months ago

Aren't you deployed in multiple regions?

FlynnLivesMattr• 9 months ago

How did the call go?

morshu90012• 9 months ago

This was a single region outage, right? If you aren't cross-region, cross-cloud is the same but harder

TrackerFF2• 9 months ago

Lots of outage in Norway, started approximately 1 hour ago for me.

nextaccountic• 9 months ago

Is this why reddit is down? (https://www.redditstatus.com/ still says it is up but with degraded infrastructure)

krowek• 9 months ago

Shameless from them to make it look like it's a user problem. It was loading fine for me one hour ago, now I refresh the page and their message states I'm doing too many requests and should chill out (1 request per hour is too many for you?)

anal_reactor3• 9 months ago

I remember that I made a website and then I got a report that it doesn't work on newest Safari. Obviously, Safari would crash with a message blaming the website. Bro, no website should ever make your shitty browser outright crash.

etothet2• 9 months ago

Never ascribe to malice that which is adequately explained by incompetence.

It’s likely that, like many organizations, this scenario isn’t something Reddit are well prepared for in terms of correct error messaging.

kaptainscarlet• 9 months ago

I got a rate limit error which didn't make sense since it was my first time opening reddit in hours.

ryanchants• 9 months ago

Could be a bunch of reddit bots on AWS are now catching back up as AWS recovers and spiking hits to reddit

TrackerFF2• 9 months ago

Lots of outage happening in Norway, too. So I'm guessing it is a global thing.

ryanmcdonough2• 9 months ago

Now, I may well be naive - but isn't the point of these systems that you fail over gracefully to another data centre and no-one notices?

codeulike• 9 months ago

I get the impression that this has been thought about to some extent, but its a constantly changing architecture with new layers and new ideas being added, so for every bit of progress there's the chance of new Single Points Of Failure being added. This time it seems to be a DNS problem with DynamoDB

weberer• 9 months ago

Llama-5-beelzebub has escaped containment. A special task force has been deployed to the Virginia data center to pacify it.

sota_pop• 9 months ago

Amazon devs to ClaudeCode: “That didn’t fix it. The service is down, pls fix. Make no mistakes. pls.”

cmiles84• 9 months ago

US-East-1 and its consistent problems are literally the Achilles Heel of the Internet.

Danborg• 9 months ago

r/aws not found

There aren't any communities on Reddit with that name. Double-check the community name or start a new community.

xodice3• 9 months ago

Major us-east-1 outages happened in 2011, 2015, 2017, 2020, 2021, 2023, and now again. I understand that us-east-1, N. VA, was the first DC but for fucks sake they've had HOW LONG to finish AWS and make us-east-1 not be tied to keeping AWS up.

hvb210• 9 months ago

First, not all outages are created equal, so you cannot compare them like that.

I believe the 2021 one was especially horrific because of it affecting their dns service (route53) and the outage made writes to that service impossible. This made fail overs not work etcetera so their prescribed multi region setups didn't work.

But in the end, some things will have to synchronizes their writes somewhere, right? So for dns I could see how that ends up in a single region.

AWS is bound by the same rules as everyone else in the end... The only thing they have going for them that they have a lot of money to make certain services resilient, but I'm not aware of a single system that's resilient to everything.

xodice3• 9 months ago

If AWS fully decentralized its control planes, they’d essentially be duplicating the cost structure of running multiple independent clouds and I understand that is why they don't however as long as AWS is reliant upon us-east-1 to function, they have not achieved what they claim to me. A single point of failure for IAM? Nah, no thanks.

Every AWS “global” service be it IAM, STS, CloudFormation, CloudFront, Route 53, Organizations, they all have deep ties to control systems originally built only in us-east-1/n. va.

That's poor design, after all these years. They've had time to fix this.

Until AWS fully decouples the control plane from us-east-1, the entire platform has a global dependency. Even if your data plane is fine, you still rely on IAM and STS for authentication and maybe Route 53 for DNS or failover CloudFormation or ECS for orchestration...

If any of those choke because us-east-1’s internal control systems are degraded, you’re fucked. That’s not true regional independence.

spwa4• 9 months ago

Reddit seems to be having issues too:

"upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout"

ivad• 9 months ago

Seems to have taken down my router "smart wifi" login page, and there's no backup router-only login option! Brilliant work, linksys....

ta988• 9 months ago

Happened to lots of commercial routers too (free wifi with sign-in pages in stores for example) and that's way outside us-east-1

kristopherleads• 9 months ago

Was just on a Lufthansa and then United flight - both of which did not have WiFi. Was wondering if there was something going on at the infrastructure level.

rvba2• 9 months ago

What if they use the same router inside AWS and now they cannot login too?

hmry• 9 months ago

WiFi login portal (Icomera) on the train I'm on doesn't work either.

shinycode• 9 months ago

It’s that period of the year when we discover AWS clients that don’t have fallback plans

waste_monk2• 9 months ago

♫ It's the most blunderful time of the year

There'll be much admin moaning

And servers not glowing

and the NOC crew in tears

It's the most blunderful time of the year ♫

okr2• 9 months ago

Btw. we had a forced EKS restart last week on thursday due to Kubernetes updates. And something was done with DNS there. We had problems with ndots. Caused some trouble here. Would not be surprised, if it is related, heh.

0x002A6• 9 months ago

As Amazon moves from day-1 company as it claimed once, to be the sales company like Oracle focusing on raking money, expect more outages to come, and longer to be resolved.

Amazon is burning and driving away the technical talent and knowledge knowing the vendor lock-in will keep bringing the sweet money. You will see more sales people hoovering around your c-suites and executives, while you will face even worse technical support, that seem not knowing what they are talking about, yet alone to fix the support issue you expect to be fixed easily.

Mark my words, and if you are putting your eggs in one basket, that basket is now too complex and too interdependent, and the people who built and knew those intricacies are driven away with RTOs, move to hubs. Eventually those services; all others (and also aws services themselves) heavily dependent on, might be more fragile than the public knows.

zht• 9 months ago

Do you have data suggesting AWS outages are more frequent and/or take longer to resolve?

0x002A6• 9 months ago

This is a prediction, not a historical pattern to be observed now. Only future data can verify if this prediction was correct or not.

hopelite• 9 months ago

That is why technical leaders’ role wouldn’t demand they not only gather data, but also report things like accurate operational, alternative, and scenario cost analysis; financial risks; vendor lock-in; etc.

However, as may be apparent just from that small set, it is not exactly something technical people often feel comfortable with doing. It is why at least in some organizations you get the friction of a business type interfacing with technical people in varying ways, but also not really getting along because they don’t understand each other and often there are barriers of openness.

0x002A6• 9 months ago

I think business types vs technical types inherently have different perspectives especially for american companies. One has the "get it done at all costs" the other has "this can't be done since impossible/it will break this".

When a company moves from engineering/technical driven to sales/profit/stock price/shareholders satisfaction driven, once it was not possible to cut (technical) corners, now becomes the de facto. If you push the L7s/L8s out of the discussion room, who would definitely stop or veto circular dependencies, and replace with sir-yes-sir people, now you've successfully created short term KPI wins for the lofty chairs but with a burning fuse of catastrophic failures to come.

joncrane4• 9 months ago

You will see more sales people hoovering around your c-suites and executives, while you will face even worse technical support, that seem not knowing what they are talking about, yet alone to fix the support issue you expect to be fixed easily.

WILL see? We've been seeing this since 2019.

0x002A6• 9 months ago

it will get worse

0x002A6• 9 months ago

https://www.theregister.com/2025/10/20/aws_outage_amazon_bra... quoting from this

"And so, a quiet suspicion starts to circulate: where have the senior AWS engineers who've been to this dance before gone? And the answer increasingly is that they've left the building — taking decades of hard-won institutional knowledge about how AWS's systems work at scale right along with them."

...

"AWS has given increasing levels of detail, as is their tradition, when outages strike, and as new information comes to light. Reading through it, one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow. To be clear: I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time."

....

"This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. "

...

"I want to be very clear on one last point. This isn't about the technology being old. It's about the people maintaining it being new. If I had to guess what happens next, the market will forgive AWS this time, but the pattern will continue."

hubertzhang• 9 months ago

I cannot pull images from docker hub.

chistev• 9 months ago

What is HN hosted on?

Bender• 9 months ago

2 physical servers, one active and one standby running BSD at M5 internet hosting.

kkfx• 9 months ago

Honestly anyone do have outages, that's nothing extraordinary, what's wrong is the number of impacted services. We choose (at least almost choose) to ditch mainframes for clusters also for resilience. Now with cheap desktop iron labeled "stable enough to be a serious server" we have seen mainframes re-created sometimes with a cluster of VM on top of a single server, sometimes with cloud services.

Ladies and Gentleman's it's about time to learn reshoring in the IT world as well. Owning nothing, renting all means extreme fragility.

Ygg2• 9 months ago

Ironically enough I can't access Reddit due to no healthy upstream.

me551ah3• 9 months ago

We created a single point of failure on the Internet, so that companies could avoid single points of failure in their data centers.

nonethewiser2• 9 months ago

China is unaffected.

Spivak• 9 months ago

It's actually kinda great. When AWS has issues it makes national news and that's all you need to put on your status page and everyone just nods in understanding. It's a weird kind of holiday in a way.

tonypapousek• 9 months ago

Looks like they’re nearly done fixing it.

Oct 20 3:35 AM PDT
The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution. If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches. We will provide an update by 4:15 AM, or sooner if we have additional information to share.

chibea2• 9 months ago

It's a bit funny that they say "most service operations are succeeding normally now" when, in fact, you cannot yet launch or terminate new EC2 instance, which is basically the defining feature of the cloud...

rswail3• 9 months ago

In that region, other regions are able to launch EC2s and ECS/EKS without a problem.

jamwil3• 9 months ago

Is that material to a conversation about service uptime of existing resources, though? Are there customers out there that are churning through the full lifecycle of ephemeral EC2 instances as part of their day-to-day?

archon8102• 9 months ago

Except it just broke again.

shkkmo• 9 months ago

Still not fixed and may have gotten worse...

ceroxylon• 9 months ago

In the last hour, I have seen the number of impacted services go from 90 to 92, currently sitting at 97.

pantulis• 9 months ago

Now I know why the documents I was sending to my Kindle didn't go through.

CTDOCodebases• 9 months ago

I'm getting rate limit issues on Reddit so it could be related.

testemailfordg2• 9 months ago

Seems like we need more anti-trust cases on AWS or need to break it down, it is becoming too big. Services used in rest of the world get impacted by issues in one region.

arielcostas• 9 months ago

But they aren't abusing their market power, are they? I mean, they are too big and should definitely be regulated but I don't think you can argue they are much of a monopoly when others, at the very least Google, Microsoft, Oracle, Cloudflare (depending on the specific services you want) and smaller providers can offer you the same service and many times with better pricing. Same way we need to regulate companies like Cloudflare essentially being a MITM for ~20% of internet websites, per their 2024 report.

SergeAx• 9 months ago

How much longer are we going to tolerate this marketing bullshit about "Designed to provide 99.999999999% durability and 99.99% availability"?

coleca• 9 months ago

That is for S3 not AWS as a whole, AWS has never claimed otherwise. AFAIK S3 has never broken the 11 9s of durability.

sitzkrieg• 9 months ago

it is very funny to me that us-east-1 going down nukes the internet. all those multiple region reliability best practices are for show

geye1234• 9 months ago

Potentially-ignoramus comment here, apologies in advance, but amazon.com itself appears to be fine right now. Perhaps slower to load pages, by about half a second. Are they not eating (much of) their own dog food?

bananapub3• 9 months ago

those sentences aren't logically connected - the outage this post is about is mostly confined to `us-east-1`, anyone who wanted to build a reliable system on top of AWS would do so across multiple regions, including Amazon itself.

`us-east-1` is unfortunately special in some ways but not in ways that should affect well-designed serving systems in other regions.

julianozen4• 9 months ago

They are 100% fully on AWS and nothing else. (I’m Ex-Amazon)

It seems like the outage is only effecting one region so AWS is likely falling back to others. I’m sure parts of the site are down but the main sites are resilient

rwky6• 9 months ago

I was getting 500 errors a few hours ago on amazon.com

altairprime2• 9 months ago

Kindle downloads and Amazon orders history were wholly unavailable, with rapid errors from the responsive website.

aiiizzz2• 9 months ago

Slack was acting slower than usual, but did not go down. Color me impressed.

chibea2• 9 months ago

One main problem that we observed was that big parts of their IAM / auth setup was overloaded / down which led to all kinds of cascading problems. It sounds as if Dynamo was reported to be a root cause, so is IAM dependent on dynamo internally?

Of course, such a large control plane system has all kinds of complex dependency chains. Auth/IAM seems like such a potentially (global) SPOF that you'd like to reduce dependencies to an absolute minimum. On the other hand, it's also the place that needs really good scalability, consistency, etc. so you probably like to use the battle proof DB infrastructure you already have in place. Does that mean you will end up with a complex cyclic dependency that needs complex bootstrapping when it goes down? Or how is that handled?

cowsandmilk• 9 months ago

Many AWS customers have bad retry policies that will overload other systems as part of their retries. DynamoDB being down will cause them to overload IAM.

joncrane4• 9 months ago

Which is interesting because per their health dashboard,

We recommend customers continue to retry any failed requests.

wwdmaxwell• 9 months ago

I think Amazon uses an internal platform called Dynamo as a KV store, it’s different than DynamoDB, so im thinking the outage could be either a dns routing issue or some kind of node deployment problem.

Both of which seem to prop up in post mortems for these widespread outages.

oofbey2• 9 months ago

They said the root cause was DNS for dynamoDB. inside AWS relying on dynamoDB is highly encouraged so it’s not surprising that a failure there would cascade broadly. The fact that EC2 instance launching is effected is surprising. Loops in the service dependency graph are known to be a bad idea.

plandis• 9 months ago

Dynamo is AFAIK, not used by core AWS services.

cyberax7• 9 months ago

When I worked at AWS several years ago, IAM was not dependent on Dynamo. It might have changed, but I highly doubt this. Maybe some kind of network issue with high-traffic services?

Auth/IAM seems like such a potentially (global) SPOF that you'd like to reduce dependencies to an absolute minimum.

IAM is replicated, so each region has its own read-only IAM cache. AWS SigV4 is also designed to be regionalized, if you ever wondered why the signature key derivation has many steps, that's exactly why ( https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_s... ).

devmor• 9 months ago

I find it very interesting that this is the same issue that took down GCP recently.

julianozen4• 9 months ago

There was a very large outage back in ~2017 that was caused by DynamoDB going down. Because EC2 stored its list of servers in DynamoDB, EC2 went down too. Because DynamoDB ran its compute on EC2, it was suddenly no longer able to spin up new instances to recover.

It took several days to manually spin up DynamoDB/EC2 instances so that both services could recover slowly together. Since then, there was a big push to remove dependencies between the “tier one” systems (S3, DynamoDB, EC2, etc.) so that one system couldn’t bring down another one. Of course, it’s never foolproof.

planckscnst3• 9 months ago

I don't remember an event like that, but I'm rather certain the scenario you described couldn't have happened in 2017.

The very large 2017 AWS outage originated in s3. Maybe you're thinking about a different event?

https://share.google/HBaV4ZMpxPEpnDvU9

bdangubic2• 9 months ago

you put your sh*t in us-east-1 you need to plan for this :)

amai2• 9 months ago

We are on Azure. But our CI/CD pipelines are failing, because Docker is on AWS.

amai2• 9 months ago

The internet was once designed to survive a nuclear war. Nowadays it cannot even survive until tuesday.

kuon2• 9 months ago

I realize that my basement servers have better uptime than AWS this year!

I think most sysadmin don't plan for AWS outage. And economically it makes sense.

But it makes me wonder, is sysadmin a lost art?

fooqux• 9 months ago

But it makes me wonder, is sysadmin a lost art?

I dunno, let me ask chatgpt. Hmmm, it said yes.

tripplyons• 9 months ago

ChatGPT often says yes to both a question and its inverse. People like to hear yes more than no.

tredre32• 9 months ago

But it makes me wonder, is sysadmin a lost art?

Yes. 15-20 years ago when I was still working on network-adjacent stuff I witnessed the shift to the devops movement.

To be clear, the fact that devops don't plan for AWS failures isn't an indication that they lack the sysadmin gene. Sysadmins will tell you very similar "X can never go down" or "not worth having a backup for service Y".

But deep down devops are developers who just want to get their thing running, so they'll google/serveroverflow their way into production without any desire to learn the intricacies of the underlying system. So when something breaks, they're SOL.

"Thankfully" nowadays containers and application hosting abstracts a lot of it back away. So today I'd be willing to say that devops are sufficient for small to medium companies (and dare I say more efficient?).

Centigonal• 9 months ago

But deep down devops are developers who just want to get their thing running, so they'll google/serveroverflow their way into production without any desire to learn the intricacies of the underlying system. So when something breaks, they're SOL.

Depends on the devops team. I have worked with so many devops engineers who came from network engineering, sysadmin, or SecOps backgrounds. They all bring a different perspective and set of priorities.

archon8102• 9 months ago

That's not very surprising. At this point you could say that your microwave has a better uptime. The complexity comparison to all the Amazon cloud services and infrastructure would be roughly the same.

dijit12• 9 months ago

So, we need to centralise all our compute onto one provider because economies of scale and its too complicated to do it well…

… but we should not compare them to self-hosting because hosting that much data and compute is complicated.

The emperor has no clothes.

asah• 9 months ago

Does this include your SPoF Internet connection?

kuon2• 9 months ago

I have a redundant fiber with two providers, I'm a LIR and I do my BGP, my internet is fine.

nla• 9 months ago

I still don't know why anyone would use AWS hosting.

nemo44x• 9 months ago

Someone’s got a case of the Monday’s.

skywhopper3• 9 months ago

There are plenty of ways to address this risk. But the companies impacted would have to be willing to invest in the extra operational cost and complexity. They aren’t.

randomtoast• 9 months ago

Thing is us-east-1 the primary region for many services of AWS. DynamoDB is a very central offering used by many service. And the issue that has happend is very common[^1].

I think no matter how hard you try to avoid it, in the end there's always a massive dependency chain for modern digital infrastructure[^2].

[1] https://itsfoss.community/uploads/default/optimized/2X/a/ad3...

[2] https://xkcd.com/2347/

karel-3d3• 9 months ago

Slack was down, so I thought I will send message to my coworkers on Signal.

Signal was also down.

alimbada• 9 months ago

E-mail still exists...

pjmlp5• 9 months ago

It just goes to show the difference between best practices in cloud computing, and what everyone ends up doing in reality, including well known industry names.

mrbungie• 9 months ago

It just goes to show the difference between best practices in cloud computing, and what everyone ends up doing in reality,

Well, inter-region DR/HA is a expensive thing to ensure (whether on salaries, infra or both), specially when you are in AWS.

esafak• 9 months ago

Does AWS follow its own Well-Architected Framework!?

spyspy4• 9 months ago

Eh, the "best practices" that would've prevented this aren't trivial to implement and are definitely far beyond what most engineering teams are capable of, in my experience. It depends on your risk profile. When we had cloud outages at the freemium game company I worked at, we just shrugged and waited for the systems to come back online - nobody dying because they couldn't play a word puzzle. But I've also had management come down and ask what it would take to prevent issues like that from happening again, and then pretend they never asked once it was clear how much engineering effort it would take. I've yet to meet a product manager that would shred their entire roadmap for 6-18 months just to get at an extra 9 of reliability, but I also don't work in industries where that's super important.

pjmlp5• 9 months ago

Indeed, yet one would expect AWS to lead by example, including all of those that are only using a single region.

Thaxll• 9 months ago

Best practice does not include plan for when AWS going down. Netflix does not plan for it and they have a very strong eng org.

pjmlp5• 9 months ago

It was only one region.

immibis12• 9 months ago

Did they stop their Chaos Gorilla, which simulates a region outage?

mcphage• 9 months ago

It shouldn’t, but it does. As a civilization, we’ve eliminated resilience wherever we could, because it’s more cost-effective. Resilience is expensive. So everything is resting on a giant pile of single point of failures.

Maybe this is the event to get everyone off of piling everything onto us-east-1 and hoping for the best, but the last few outages didn’t, so I don’t expect this one to, either.

Trasmatta• 9 months ago

The irony is that true resilience is very complex, and complexity can be a major source of outages in and of itself

lanstin2• 9 months ago

I have enjoyed this paper on such dynamics: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...

It is kind of the child of what used to be called Catastrophe Theory, which in low dimensions is essentially a classification of folding of manifolds. Now the systems are higher dinemsional and the advice more practical/heuristic.

mschuster914• 9 months ago

Maybe this is the event to get everyone off of piling everything onto us-east-1 and hoping for the best, but the last few outages didn’t, so I don’t expect this one to, either.

Doesn't help either. us-east-1 hosts the internal control plane of AWS and a bunch of stuff is only available in us-east-1 at all - most importantly, Cloudfront, AWS ACM for Cloudfront and parts of IAM.

And the last is the one true big problem. When IAM has a sniffle, everything else collapses because literally everything else depends on IAM. If I were to guess IAM probably handles millions if not billions of requests a second because every action on every AWS service causes at least one request to IAM.

TheNewsIsHere• 9 months ago

The last re:Invent presentation I saw from one of the principals working on IAM quoted 500 million requests per second. I expect that’s because IAM also underpins everything inside AWS, too.

foinker2• 9 months ago

No shot that happens until an outage breaks at least an entire workday in the US timezones. The only complaint I personally heard was from someone who couldn't load reddit on the train to work.

geodel• 9 months ago

Well by the time it really happens for a whole day Amazon leadership will be brazen enough to say "OK, enough of this my site is down, we will call back once systems are up so don't bother for a while". Also maybe responsible human engineers would fired by then and AI can be infinitely patient while working through insolvable issues.

BeFlatXIII• 9 months ago

No complaints from folks who couldn’t load reddit at work?

2d8a875f-39a2-4• 9 months ago

we’ve eliminated resilience wherever we could, because it’s more cost-effective. Resilience is expensive.

You are right. But alas, a peek at the AMZN stock ticker suggests that the market doesn't really value resilience that much.

Cthulhu_4• 9 months ago

Stocks stopped being indicative of anything decades ago though.

agos2• 9 months ago

when did we have resilience?

BoredPositron3• 9 months ago

Cold War was pretty good in terms of resilience.

JCM917• 9 months ago

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions. Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans.

AWS doesn’t talk about that much publicly, but if you press them they will admit in private that there are some pretty nasty single points of failure in the design of AWS that can materialize if us-east-1 has an issue. Most people would say that means AWS isn’t truly multi-region in some areas.

Not entirely clear yet if those single points of failure were at play here, but risk mitigation isn’t as simple as just “don’t use us-east-1” or “deploy in multiple regions with load balancing failover.”

helsinkiandrew2• 9 months ago

US-East-1 is more than just a normal region. It also provides the backbone for other services, including those in other regions

I thought that if us-east-1 goes down you might not be able to administer (or bring up new services) in other zones, but if you have services running that can take over from us-east-1, you can maintain your app/website etc.

I haven’t had to do this for several years but that was my experience a few years ago on an outage - obviously it depends on the services you’re using.

You can’t start cloning things to other zones after us-east-1 is down - you’ve left it too late

cmiles84• 9 months ago

Well that sounds like exactly the sort of thing that shouldn’t happen when there’s an issue given the usual response is to spin things up elsewhere, especially on lower priority services where instant failover isn’t needed.

sgarland7• 9 months ago

It depends on the outage. There was one a year or two ago (I think? They run together) that impacted EC2 such that as long as you weren’t trying to scale, or issue any commands, your service would continue to operate. The EKS clusters at my job at the time kept chugging along, but had Karptenter tried to schedule more nodes, we’d have had a bad time.

Yeul• 9 months ago

Internet was supposed to be a communication network if the East Coast was nuked.

What it turned into was Daedalus from Deus Ex lol.

t_sawyer3• 9 months ago

Yeah because Amazon engineers are hypocrites. They want you to spend extra money for region failover and multi-az deploys but they don't do it themselves.

ajkjk3• 9 months ago

They absolutely do do it themselves..

falcor842• 9 months ago

That's a good point, but I'd just s/Amazon engineers/AWS leadership/ , as I'm pretty sure that there's a few layers of management removed between the engineers on the ground at AWS, those who deprioritise any longer-term resilience work needed (which is a very strategic decisioN), and those those who are in charge of external comms/education about best practices for AWS customers.

ZeroCool2u• 9 months ago

They can't even bother to enable billing services in GovCloud regions.

bravetraveler20• 9 months ago

Call me crazy, because this is, perhaps it's their "Room 641a". The purpose of a system is what it does, no point arguing 'should' against reality, etc.

They've been charging a premium for, and marketing, "Availability" for decades at this point. I worked for a competitor and made a better product: it could endure any of the zones failing.

voxadam2• 9 months ago

perhaps it's their "Room 641a".

For the uninitiated: https://en.wikipedia.org/wiki/Room_641A

jf• 9 months ago

Interesting. Langley isn’t that far away

nevir3• 9 months ago

It's really not that nefarious.

IAD datacenters have forever been the place where Amazon software developers implement services first (well before AWS was a thing).

Multi-AZ support often comes second (more than you think; Amazon is a pragmatic company), and not every service is easy to make TRULY multi-AZ.

And then other services depend on those services, and may also fall into the same trap.

...and so much of the tech/architectural debt gets concentrated into a single region.

Anon1096• 9 months ago

It's possible that you really could endure any zone failure. But I take these claims people make all the time with a grain of salt, unless you're working on AWS scale (basically just 3 companies) and have actually run for years and seen every kind of failure mode claiming to be higher availability is not something that's able to be accurately evaluated.

(I'm assuming by zone you mean the equivalent of an AWS region, with multiple connected datacenters)

stronglikedan• 9 months ago

Was that competitor priced competitively with AWS? I think of the project management triangle here - good, fast, or cheap - pick two. AWS would be fast and cheap.

gchamonlive2• 9 months ago

Been a while since I last suffered from AWS arbitrary complexity, but afaik you can only associate certificates to cloudfront if they are generated in us-east-1, so it's undoubtedly a single point of failure for all CDN if this is still the case.

kokanee3• 9 months ago

I worked at AMZN for a bit and the complexity is not exactly arbitrary; it's political. Engineers and managers are highly incentivized to make technical decisions based on how they affect inter-team dependencies and the related corporate dynamics. It's all about review time.

xbar• 9 months ago

This set of facts comes to light every 3-5 years when US-East-1 has another failure. Clearly they could have architected their way out of this blast radius problem by now, but they do not.

Why would they keep a large set of centralized, core traffic services in Virginia for decades despite it being a bad design?

firesteelrain4• 9 months ago

It’s probably because there is a lot of tech debt plus look at where it is - Virgina. It shouldn’t take much of imagination to figure out why that is strategic

AtlasBarfed2• 9 months ago

What is the motivation of an effective Monopoly to do anything?

I mean look at their console. Their console application is pretty subpar.

cyberax7• 9 months ago

AWS _had_ architected away from single-region failure modes. There are only a few services that are us-east-1 only in AWS (IAM and Route53, mostly), and even they are designed with static stability so that their control plane failure doesn't take down systems.

It's the rest of the world that has not. For a long time companies just ran everything in us-east-1 (e.g. Heroku), without even having an option to switch to another region.

api2• 9 months ago

My contention for a long time has been that cloud is full of single points of failure (and nightmarish security hazards) that are just hidden from the customer.

"We can't run things on just a box! That's a single point of failure. We're moving to cloud!"

The difference is that when the cloud goes down you can shift the blame to them, not you, and fixing it is their problem.

The corporate world is full of stuff like this. A huge role of consultants like McKinsey is to provide complicated reports and presentations backing the ideas that the CEO or other board members want to pursue. That way if things don't work out they can blame McKinsey.

raw_anon_11115• 9 months ago

You act as if that is a bug not a feature. As hypothetically someone who is responsible for my site staying up, I would much rather blame AWS than myself. Besides none of your customers are going to blame you if every other major site is down.

aurareturn7• 9 months ago

Or it is a matter of efficiency. If 1 million companies design and maintain their servers, there would be 1 million (or more) incidents like these. Same issues. Same fixes. Not so efficient.

qaq• 9 months ago

even if us-east-1 was a normal region there is not enough spare capacity to take up all the workloads from us-east-1 in other regions so t's a moot point

nevir3• 9 months ago

It also doesn't help that most companies using AWS aren't remotely close to multi-region support, and that us-east-1 is likely the most populated region.

einrealist2• 9 months ago

It sounds like they want to avoid split-brain scenarios as much as possible while sacrificing resilience. For things like DNS, this is probably unavoidable. So, not all the responsibility can be placed on AWS. If my application relies on receipts (such as an airline ticket), I should make sure I have an offline version stored on my phone so that I can still check in for my flight. But I can accept not to be able to access Reddit or order at McDonalds with my phone. And always having cash at hand is a given, although I almost always pay with my phone nowadays.

I hope they release a good root cause analysis report.

immibis12• 9 months ago

It's not unavoidable for DNS. DNS is inherently eventually consistent anyway, due to time-based caching.

masfuerte• 9 months ago

Amazon are planning to launch the EU Sovereign Cloud by the end of the year. They claim it will be completely independent. It may be possible then to have genuine resiliency on AWS. We'll see.

louthy• 9 months ago

Then it will be eu-east-1 taking down the EU

samcat116• 9 months ago

This is the difference between “partitions” and “regions”. Partitions have fully separate IAM, DNS names, etc. This is how there are things like US Gov Cloud, the Chinese AWS cloud, and now the EU sovereign cloud

seany• 9 months ago

gov, iso*, cn are also already separate (unless you need to mess with your bill, or certain kinds of support tickets)

belter4• 9 months ago

Thus simply being in another region doesn’t protect you from the consistent us-east-1 shenanigans.

Well it did for me today...Dont use us-east-1 explicitly just other regions and I had no outage today...( I get the point about the skeletons in the closet of us-east-1 ...maybe the power plug goes via Bezos wood desk? )

thayne• 9 months ago

There are hints at in their documentation. For example ACM certs for cloudfront and KMS keys for route53 DNSSEC have to be in the us-east1 region.

everfrustrated5• 9 months ago

However these services don't need high write uptime.

cyberax7• 9 months ago

FWIW, I tried creating a DNSSEC entry for one of my domains during the outage, and it worked just fine.

helsinkiandrew2• 9 months ago

The incident underscores the risks associated with the heavy reliance on a few major cloud service providers.

Perhaps for the internet as a whole, but for each individual service it underscores the risk of not hosting your service in multiple zones or having a backup

runako2• 9 months ago

Even though us-east-1 is the region geographically closest to me, I always choose another region as default due to us-east-1 (seemingly) being more prone to these outages.

Obviously, some services are only available in us-east-1, but many applications can gain some resiliency just by making a primary home in any other region.

joncrane4• 9 months ago

What services are only available in us-east-1?

tom1337• 9 months ago

IAM control plane for example:

There is one IAM control plane for all commercial AWS Regions, which is located in the US East (N. Virginia) Region. The IAM system then propagates configuration changes to the IAM data planes in every enabled AWS Region. The IAM data plane is essentially a read-only replica of the IAM control plane configuration data.

and I believe some global services (like certificate manager, etc.) also depend on the us-east-1 region

https://docs.aws.amazon.com/IAM/latest/UserGuide/disaster-re...

tomchuk• 9 months ago

IAM, Cloudfront, Route53, ACM, Billing...

runako2• 9 months ago

In addition to those listed in sibling comments, new services often roll out in us-east-1 before being made available in other regions.

I recently ran into an issue where some Bedrock functionality was available in us-east-1 but not one of the other US regions.

cyberax7• 9 months ago

This is the right move. 10 years ago, us-east-1 was on the order of 10x bigger than the next largest region. This got a little better now, but any scaling issues still tend to happen in us-east-1.

The AWS has been steering people to us-east-2 for a while. For example, traffic between us-east-1 and us-east-2 has the same cost as inter-AZ traffic within the us-east-1.

fsto• 9 months ago

Ironically, the HTTP request to this article timed out twice before a successful response.

aeon_ai• 9 months ago

It's not DNS

There's no way it's DNS

It was DNS

ajross2• 9 months ago

It's "DNS" because the problem is that at the very top of the abstraction hierarchy in any system is a bit of manual configuration.

As it happens, that naturally maps to the bootstrapping process on hardware needing to know how to find the external services it needs, which is what "DNS" is for. So "DNS" ends up being the top level of manual configuration.

But it's the inevitability of the manual process that's the issue here, not the technology. We're at a spot now where the rest of the system reliability is so good that the only things that bring it down are the spots where human beings make mistakes on the tiny handful of places where human operation is (inevitably!) required.

allarm• 9 months ago

hardware needing to know how to find the external services it needs, which is what "DNS" is for. So "DNS" ends up being the top level of manual configuration.

Unless DNS configuration propagates over DHCP?

jamesbelchamber5• 9 months ago

This website just seems to be an auto-generated list of "things" with a catchy title:

5000 Reddit users reported a certain number of problems shortly after a specific time.
400000 A certain number of reports were made in the UK alone in two hours.

ktosobcy2• 9 months ago

Uhm... E(U)ropean sovereigny (and in general spreading the hosting as much as possbile) needed ASAP…

BirAdam• 9 months ago

Well, except for a lot of business leaders saying that they don't care if it's Amazon that goes down, because "the rest of the internet will be down too."

Dumb argument imho, but that's how many of them think ime.

tjwebbnorfolk2• 9 months ago

because... EU clouds don't break?

https://news.ycombinator.com/item?id=43749178

Cthulhu_4• 9 months ago

Nah, because European services should not be affected by a failure in the US. Whatever systems they have running in us-east-1 should have failovers in all major regions. Today it's an outage in Virginia, tomorrow it could be an attack on undersea cables (which I'm confident are mined and ready to be severed at this point by multiple parties).

ktosobcy2• 9 months ago

No, it's called "diversification". Applies both to stock/currency/investments as well as enything else :P

d_burfoot• 9 months ago

I think AWS should use, and provide as an offering to big customers, a Chaos Monkey tool that randomly brings down specific services in specific AZs. Example: DynamoDB is down in us-east-1b. IAM is down in us-west-2a.

Other AWS services should be able to survive this kind of interruption by rerouting requests to other AZs. Big company clients might also want to test against these kinds of scenarios.

davidrupp2• 9 months ago

AWS Fault Injection Service: https://docs.aws.amazon.com/fis/latest/userguide/what-is.htm...

jrochkind13• 9 months ago

At some point AWS has so many services it's subject to a version of xkcd Rule 34 -- if you can imagine it, there's an AWS service for it.

bob1029• 9 months ago

One thing has become quite clear to me over the years. Much of the thinking around uptime of information systems has become hyperbolic and self-serving.

There are very few businesses that genuinely cannot handle an outage like this. The only examples I've personally experienced are payment processing and semiconductor manufacturing. A severe IT outage in either of these businesses is an actual crisis. Contrast with the South Korean government who seems largely unaffected by the recent loss of an entire building full of machines with no backups.

I've worked in a retail store that had a total electricity outage and saw virtually no reduction in sales numbers for the day. I have seen a bank operate with a broken core system for weeks. I have never heard of someone actually cancelling a subscription over a transient outage in YouTube, Spotify, Netflix, Steam, etc.

The takeaway I always have from these events is that you should engineer your business to be resilient to the real tradeoff that AWS offers. If you don't overreact to the occasional outage and have reasonable measures to work around for a day or 2, it's almost certainly easier and cheaper than building a multi cloud complexity hellscape or dragging it all back on prem.

Thinking in terms of competition and game theory, you'll probably win even if your competitor has a perfect failover strategy. The cost of maintaining a flawless eject button for an entire cloud is like an anvil around your neck. Every IT decision has to be filtered through this axis. When you can just slap another EC2 on the pile, you can run laps around your peers.

vidarh3• 9 months ago

The takeaway I always have from these events is that you should engineer your business to be resilient

An enduring image that stays with me was when I was a child and the local supermarket lost electricity. Within seconds the people working the tills had pulled out hand cranks by which the tills could be operated.

I'm getting old, but this was the 1980's, not the 1800's.

In other words, to agree with your point about resilience:

A lot of the time even some really janky fallbacks will be enough.

But to somewhat disagree with your apparent support for AWS: While it is true this attitude means you can deal with AWS falling over now and again, it also strips away one of the main reasons people tend to give me for why they're in AWS in the first place - namely a belief in buying peace of mind and less devops complexity (a belief I'd argue is pure fiction, but that's a separate issue). If you accept that you in fact can survive just fine without absurd levels of uptime, you also gain a lot more flexibility in which options are viable to you.

The cost of maintaining a flawless eject button is indeed high, but so is the cost of picking a provider based on the notion that you don't need one if you're with them out of a misplaced belief in the availability they can provide, rather than based on how cost effectively they can deliver what you actually need.

jskopek• 9 months ago

I would argue that you are still buying peace of mind by hosting on AWS, even when there are outages. This outage is front page news around the world, so it's not as much of a shock if your company's website goes down at the same time.

jrochkind13• 9 months ago

The cranked tills (or registers for the Americans) is an interesting example, because it seems safe to assume they don't have that equipment anymore, and could not so easily do that.

We have become much more reliant on digital tech (those hand cranked tills were prob not digital even when the electricity was on), and much less resilient to outages of such tech I think.

alpinisme• 9 months ago

Some of the peace of mind comes just from knowing it’s someone else’s (technical) problem if the system goes down. And someone else’s problem to monitor the health of it. (Yes, we still have to monitor and fix all sorts of things related to how we’ve built our products, but there’s a nontrivial amount of stuff that is entirely the responsibility of AWS)

chasd0011• 9 months ago

An enduring image that stays with me was when I was a child and the local supermarket lost electricity. Within seconds the people working the tills had pulled out hand cranks by which the tills could be operated.

what did the do with the frozen food section? Was all that inventory lost?

jezzamon• 9 months ago

Tech companies, and in particular ad-driven companies, keep a very close eye on their metrics and can fairly accurately measure the cost of an outage in real dollars

bilekas2• 9 months ago

These things happen when profits are the measure everything. Change your provider, but if their number doesn't go up, they wont be reliable.

So your complaints matter nothing because "number go up".

I remember the good old days of everyone starting a hosting company. We never should have left.

epistasis• 9 months ago

It certainly doesn't take profit for bad planning to happen...

And everybody starting a hosting company is definitely a profit driven activity.

bilekas2• 9 months ago

And everybody starting a hosting company is definitely a profit driven activity.

Absolutely, nobody was doing it out of charity, but there is more diversity in the market and thus more innovation and then the market decides. Right now we have 3 major providers, and that makes up the lion's share. That's consolidation of a service. I believe that's not good for the market or the internet as a whole.

DanHulton2• 9 months ago

I forget where I read it originally, but I strongly feel that AWS should offer a `us-chaos-1` region, where every 3-4 days, one or two services blow up. Host your staging stack there and you build real resiliency over time.

(The counter joke is, of course, "but that's `us-east-1` already! But I mean deliberately and frequently.)

otterley5• 9 months ago

AWS already offers Fault Injection Service, which you can use in any region to conduct chaos engineering: https://aws.amazon.com/fis/

DanHulton2• 9 months ago

Oh my god that's great, thank you for pointing this out!

time0ut• 9 months ago

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute.

The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS control plane. By the time we'd gotten the root credentials out of the vault, things were coming back up.

Good reminder that you are only as strong as your weakest link.

shawabawa38• 9 months ago

for what it's worth, we were unable to login with root credentials anyway

i don't think any method of auth was working for accessing the AWS console

kondro9• 9 months ago

Sure it was, you just needed to login to the console via a different regional endpoint. No problems accessing systems from ap-southeast-2 for us during this entire event, just couldn’t access the management planes that are hosted exclusively in us-east-1.

nijave6• 9 months ago

Like the other poster said, you need to use a different region. The default region (of course) sends you to us-east-1

e.x. https://us-east-2.console.aws.amazon.com/console/home

1970-01-013• 9 months ago

I remember Facebook had a similar story when they botched their BGP update and couldn't even access the vault. If you have circular auth, you don't have anything when somebody breaks DNS.

crote2• 9 months ago

Wasn't there an issue where they required physical access to the data center to fix the network, which meant having to tap in with a keycard to get in, which didn't work because the keycard server was down, due to the network being down?

bcrl3• 9 months ago

That's similar to the total outage of all Rogers services in Canada back on July 7th 2022. It was compounded by the fact that the outage took out all Rogers cell phone service, making it impossible for Rogers employees to communicate with each other during the outage. A unified network means a unified failure mode.

Thankfully none of my 10 Gbps wavelengths were impacted. Oh did I appreciate my aversion to >= layer 2 services in my transport network!

vladvasiliu2• 9 months ago

Identity Center and only put it in us-east-1

Is it possible to have it in multiple regions? Last I checked, it only accepted one region. You needed to remove it first if you wanted to move it.

AndrewKemendo4• 9 months ago

Correct. That does make it a centralized failure mode and everyone is in the same boat on that.

I’m unaware of any common and popular distributed IDAM that is reliable

raverbashing• 9 months ago

Security people and ignoring resiliency and failure modes: a tale as old as time

ttul3• 9 months ago

There is always that point you reach where someone has to get on a plane with their hardware token and fly to another data centre to reset the thing that maintains the thing that gives keys to the thing that makes the whole world go round.

beefnugs• 9 months ago

So sick of billion dollar companies not hiring that one more guy

SOLAR_FIELDS3• 9 months ago

This reminds me of the time that Google’s Paris data center flooded and caught on fire a few years ago. We weren’t actually hosting compute there, but we were hosting compute in AWS EU datacenter nearby and it just so happened that the dns resolver for our Google services elsewhere happened to be hosted in Paris (or more accurately it routed to Paris first because it was the closest). The temp fix was pretty fun, that was the day I found out that /etc/hosts of deployments can be globally modified in Kubernetes easily AND it was compelling enough to want to do that. Normally you would never want to have an /etc/hosts entry controlling routing in kube like this but this temporary kludge shim was the perfect level of abstraction for the problem at hand.

citizenpaul• 9 months ago

temporary kludge shim was the perfect level of abstraction for the problem at hand.

Thats some nice manager deactivating jargon.

jordanb2• 9 months ago

Couldn't you just patch your coredns deployment to specify different forwarders?

nahumba• 9 months ago

This is the en of the thread of the first comment. Now i can find below the second comment

hinkley20• 9 months ago

Too much armor makes you immobile. Will your security org be held to task for this? This should permanently slow down all of their future initiatives because it’s clear they have been running “faster than possible” for some time.

Who watches the watchers.

barbazoo• 9 months ago

Wow, you really *have* to exercise the region failover to know if it works, eh? And that confidence gets weaker the longer it’s been since the last failover I imagine too. Thanks for sharing what you learned.

shdjhdfh• 9 months ago

You should assume it will not work unless you test it regularly. That's a big part of why having active/active multi-region is attractive, even though it's much more complex.

ej_campbell5• 9 months ago

That wouldn't have even caught that, most likely unless they verified they had no incidental tie ins with us-east-1.

jpollock• 9 months ago

The last place I worked actively switched traffic over to the backup nodes regularly (at least monthly) to ensure we could do it when necessary.

We learned that lesson by having to do emergency failovers and having some problems. :)

ej_campbell5• 9 months ago

Totally ridiculous that AWS wouldn't by default make it multi-region and warn you heavily that your multi-region service is tied to a single region for identity.

The usability of AWS is so poor.

skywhopper3• 9 months ago

They don’t charge anything for Identity Center and so it’s not considered an important priority for the revenue counters.

reenorap• 9 months ago

It's a good reminder actually that if you don't test the failover process, you have no failover process. The CTO or VP of Engineering should be held accountable for not making sure that the failover process is tested multiple times a month and should be seamless.

sroussey• 9 months ago

If you don’t regularly restore a backup, you don’t have one.

ct520• 9 months ago

I always find it interesting how many large enterprises have all these DR guidelines but fail to ever test. Glad to hear that everything came back alright

ozim• 9 months ago

Sounds like a lot of companies need to update their BCP after this incident.

michaelcampbell2• 9 months ago

"If you're able to do your job, InfoSec isn't doing theirs"

ransom1538• 9 months ago

People will continue to purchase Mutli-AZ and multi-region even though you have proved what a scam it is. If east region goes down, ALL amazon goes down, feel free to change my mind. STOP paying double rates for multi region.

mrbluecoat• 9 months ago

due to an "operational issue" related to DNS

Always DNS..

rose-knuckle17• 9 months ago

aws had an outage. Many companies were impacted. Headlines around the world blame AWS. the real news is how easy it is to identify companies that have put cost management ahead of service resiliency.

Lots of orgs operating wholly in AWS and sometimes only within us-east-1 had no operational problems last night. Some that is design (not using the impacted services). Some of that is good resiliency in design. And some of that was dumb luck (accidentally good design).

Overall, those companies that had operational problems likely wouldn't have invested in resiliancy expenses in any other deployment strategy either. It could have happened to them in Azure, GCP or even a home rolled datacenter.

aurareturn7• 9 months ago

Redundancy is insanely expensive especially for SaaS companies where the biggest cost is cloud.

Are customers willing to pay companies for that redundancy? I think not. Once every few years outage for 3 hours is fine for non critical services.

skopje2• 9 months ago

> Redundancy is insanely expensive especially for SaaS companies

That right there means the business model is fucked to begin with. If you can't have a resilient service, then you should not be offering that service. Period. Solution: we were fine before the cloud, just a little slower. No problem going back to that for some things. Not everything has to be just in time at lowest possible cost.

AtNightWeCode4• 9 months ago

In general it is not expensive. In most cases you can either load balance across two regions all the time or have a fallback region that you scale out/up and switch to if needed.

dehrmann• 9 months ago

Three nines might be good enough when you're Fornite. Probably not when you're Robinhood.

immibis12• 9 months ago

The part that makes no sense is - it's not cost management. AWS costs ten to a hundred times MORE than any other option - they just don't put it in the headline number.

dangoodmanUT2• 9 months ago

Reminder that AZs don't go down

Entire regions go down

Don't pay for intra-az traffic friends

avi_vallarapu• 9 months ago

This is the reason why it is important to plan Disaster recovery and also plan Multi-Cloud architectures.

Our applications and databases must have ultra high availability. It can be achieved with applications and data platforms hosted on different regions for failover.

Critical businesses should also plan for replication across multiple cloud platforms. You may use some of the existing solutions out there that can help with such implementations for data platforms.

- Qlik replicate - HexaRocket

and some more.

Or rather implement native replication solutions available with data platforms.

zoklet-enjoyer• 9 months ago

Is this why Wordle logged me out and my 2 guesses don't seem to have been recorded? I am worried about losing my streak.

LightBug1• 9 months ago

Remember when the "internet will just route around a network problem"?

FFS ...

mentalgear• 9 months ago

Amazon Alexa: routines like pre-set alarms were not functioning.

It's ridiculous how everything is being stored in the cloud, even simple timers. It's past high time to move functionality back on-device, which would come with the advantage of making it easier to de-connect from big tech's capitalist surveillance state as well.

Itanu• 9 months ago

exactly why it won't happen :)

freedomben2• 9 months ago

Exactly.

I half-seriously like to say things like, "I'm excited for a time when we have powerful enough computers to actually run applications on them instead of being limited to only thin clients." Only problem is most of the younger people don't get the reference anymore, so it's mainly the olds that get it

ibejoeb5• 9 months ago

This is just a silly anecdote, but every time a cloud provider blips, I'm reminded. The worst architecture I've ever encountered was a system that was distributed across AWS, Azure, and GCP. Whenever any one of them had a problem, the system went down. It also cost 3x more than it should.

manishsharan• 9 months ago

You mean multi-cloud strategy ! You wanna know how you got here ?

See the sales team from Google flew out an executive to NBA Finals, Azure Sales team flew out another executive to NFL superBowl and the AWS team flew out yet another executive to Wimbledon finals. And thats how you end up with multi-cloud strategy.

kevstev• 9 months ago

Eh, businesses want to stay resilient to a single vendor going down. My least favorite question in interviews this past year was around multi-cloud. Because imho it just isn't worth it- the increased complexity, the trying to like-like services across different clouds that aren't always really the same, and then just the ongoing costs of chaos monkeying and testing that this all actually works, especially in the face of a partial outage like this vs something "easy" like a complete loss of network connectivity... but that is almost certainly not what CEOs want to hear (mostly who I am dealing with here going for VPE or CTO level jobs).

I could care less about having more vendor dinners when I know I am promising a falsehood that is extremely expensive and likely going to cost me my job or my credibility at some point.

ibejoeb5• 9 months ago

In this particular case, it was resume-oriented architecture (ROAr!) The original team really wanted to use all the hottest new tech. The management was actually rather unhappy, so the job was to pare that down to something more reliable.

spyspy4• 9 months ago

I've seen the exact same thing at multiple companies. The teams were always so proud of themselves for being "multi-cloud" and managers rewarded them for their nonsense. They also got constant kudos for their heroic firefighting whenever the system went down, which it did constantly. Watching actually good engineers get overlooked because their systems were rock-solid while those characters got all the praise for designing an unadulterated piece of shit was one of the main reasons I left those companies.

koakuma-chan2• 9 months ago

Were you able to find a better company? If yes, what kind of company?

waynesonfire• 9 months ago

multi-cloud... any leader that approves such a boondoggle should be labelled incompetent. These morons sell it as a cost-cutting "migration". Never once have I seen such a project complete and it more than doubles complexity and costs.

worik5• 9 months ago

Watching actually good engineers get overlooked because their systems were rock-solid while those characters got all the praise for designing an unadulterated piece of shit

That is the computing business. There is no actual accountability, just ass covering

AnimeLife• 9 months ago

looks like very few get it right. A good system would have few minutes of blip when one cloud provider went down, which is a massive win compared to outages like this.

steveBK1236• 9 months ago

They all make it pretty hard, and a lot of resume-driven-devs have a hard time resisting the temptation of all the AWS alphabet soup of services.

Sure you can abstract everything away, but you can also just not use vendor-flavored services. The more bespoke stuff you use the more lock in risk.

But if you are in a "cloud forward" AWS mandated org, a holder of AWS certifications, alphabet soup expert... thats not a problem you are trying to solve. Arguably the lock in becomes a feature.

ferguess_k• 9 months ago

I think the problems are:

1) If you try to optimize in the beginning, you tend to fall into the over-optimization/engineering camp;

2) If you just let things go organically, you tend to fall into the big messy camp;

So the ideal way is to examine from time and time and re-architecture once the need arises. But few companies can afford that, unfortunately.

avs733• 9 months ago

i'll bet there are a large number of systems that are dependent on multiple cloud platforms being up without even knowing it. They run on AWS, but rely on a tool from someone else that runs on GCP or on Azure and they haven't tested what happens if that tools goes down...

Common Cause Failures and false redundancy are just all over the place.

luhn2• 9 months ago

Case in point is recent-ish Google Cloud downtime, which ended up taking down Cloudflare and half the internet with it.

smb06• 9 months ago

On the flip side, our SaaS runs primarily on GCP so our users are fine. But our billing and subscription system runs on AWS so no one can pay us today.

bgwalter• 9 months ago

Probably related:

https://www.nytimes.com/2025/05/25/business/amazon-ai-coders...

"Pushed to use artificial intelligence, software developers at the e-commerce giant say they must work faster and have less time to think."

Every bit of thinking time spent on a dysfunctional, lying "AI" agent could be spent on understanding the system. Even if you don't move your mouse all the time in order to please a dumb middle manager.

add-sub-mul-div• 9 months ago

Keep going

renegade-otter• 9 months ago

If we see more of this, it would not be crazy to assume that all this compelling of engineers to "use AI" and the flood of Looks Good To Me code is coming home.

Cthulhu_4• 9 months ago

Big if, major outages like this aren't unheard of, and so far, fairly uncommon. Definitely hit harder than their SLAs promise though. I hope they do an honest postmortem, but I doubt they would blame AI even if it was somehow involved. Not to mention you can't blame AI unless you go completely hands-off - but that's like blaming an outsourcing partner, which also never happens.

vivzkestrel• 9 months ago

stupid question: is buying a server rack and running it at home subject to more downtimes in a year than this? has anyone done an actual SLA analysis?

alphabettsy• 9 months ago

That depends on a lot of factors, but for me personally, yes it is. Much worse.

Assuming we’re talking about hosting things for Internet users. My fiber internet connection has gone down multiple times, though relatively quickly restored. My power has gone out several times in the last year, with one storm having it out for nearly 24 hrs. I was sleep when it went out and I didn’t start the generator until it was out for 3-4 hours already, far longer than my UPSes could hold up. I’ve had to do maintenance and updates both physical and software.

All of those things contribute to a downtime significantly higher than I see with my stuff running on Linode, Fly.io or AWS.

I run Proxmox and K3s at home and it makes things far more reliable, but it’s also extra overhead for me to maintain.

Most or all of those things could be mitigated at home, but at what cost?

amelius2• 9 months ago

Maybe if you use a UPS and Starlink then ...

dboreham• 9 months ago

Unanswerable question. Better to perform a failure mode analysis. That rack in your basement would need redundant power (two power companies or one power company and a diesel generator which typically won't be legal to have at your home), then redundant internet service (actually redundant - not the cable company vs the phone company that underneath use the same backhaul fiber).

pluto_modadic4• 9 months ago

so, funny story, my fiber got cut (backhoe) and it took then 12 hours to restore it.

If you had /two/ houses, in separate towns, you'd have better luck. Or, if you had cell as a backup.

Or: if you don't care about it being down for 12 hours.

Forricide2• 9 months ago

So for me, extremely anecdotally, I host a few fairly low-importance things on a home server (which is just an old desktop computer left sitting under a desk with Ubuntu slapped on it): A VPN (WireGuard), a few Discord bots, a Twitch bot + some auth stuff, and a few other services that I personally use.

These are the issues I've ran into that have caused downtime in the last few years:

- 1x power outage: if I had set up restart on power, probably would have been down for 30-60 minutes, ended up being a few hours (as I had to manually press the power button lol). Probably the longest non-self-inflicted issue.

- Twitch bot library issues: Just typical library bugs. Unrelated to self-hosting.

- IP changes: My IP actually barely ever changes, but I should set up DDNS. Fixable with self-hosting (but requires some amount of effort).

- Running out of disk space: Would be nice to be able to just increase it.

- Prooooooobably an internet outage or two, now that I think about it? Not enough that it's been a serious concern, though, as I can't think of a time that's actually happened. (Or I have a bad memory!)

I think that's actually about it. I rely fairly heavily on my VPN+personal cloud as all my notes, todos, etc are synced through it (Joplin + Nextcloud), so I do notice and pay a lot of attention to any downtime, but this is pretty much all that's ever happened. It's remarkable how stable software/hardware can be. I'm sure I'll eventually have some hardware failure (actually, I upgraded my CPU 1-2 years ago because it turns out the Ryzen 1700 I was using before has some kind of extremely-infrequent issue with Linux that was causing crashes a couple times a month), but it's really nice.

To be clear, though, for an actual business project, I don't think this would be a good idea, mainly due to concerns around residential vs commercial IPs, arbitrary IPs connecting to your local network, etc that I don't fully pay attention to.

freedomben2• 9 months ago

I've been dabbling in this since 1998, It's almost always ISP and power outages that get you. There are ways to mitigate those (primary/secondary ISPs, UPSes, and generators) but typically unless you're in a business district area of a city, you'll just always be subject to problems

moralestapia• 9 months ago

Curious to know how much does an outage like this cost to others.

Lost data, revenue, etc.

I'm not talking about AWS but whoever's downstream.

Is it like 100M, like 1B?

nivekney• 9 months ago

Wait a second, Snapchat impacted AGAIN? It was impacted during the last GCP outage.

menomatter• 9 months ago

What are the design best practices and industry standards for building on-premise fallback capabilities for critical infrastructure? Say for health care/banking ..etc

skopje2• 9 months ago

A relative of mine lived and worked in the US for Oppenheimer Funds in the 1990's and they had their own datacenters all over the US, multiple redundancy for weather or war. But every millionaire feels entitled to be a billionaire now, so all of that cost was rolled into a single point of cloud failure.

Reminds me of a great Onion tagline:

"Plowshare hastily beaten back into sword."

saltyoldman2• 9 months ago

GCP is underrated

thinkindie2• 9 months ago

Today’s reminder: multi-region is so hard even AWS can’t get it right.

draxil• 9 months ago

I was just about to post that it didn't affect us (heavy AWS users, in eu-west-1). Buut, I stopped myself because that was just massively tempting fate :)

mannyv• 9 months ago

This is why we use us-east-2.

muttantt• 9 months ago

us-east-2 remains the best kept secret

jdlyga2• 9 months ago

Time to start calling BS on the 9's of reliability

esskay12• 9 months ago

Er...They appear to have just gone down again.

lexandstuff3• 9 months ago

Yep. I don't think they ever fully recovered, but status page is still reporting a lot of issues.

jrochkind13• 9 months ago

My systems didn't actually seem to be affected until what I think was probably a SECOND spike of outages at about the time you posted.

The retrospective will be very interesting reading!

(Obviously the category of outages caused by many restored systems "thundering" at once to get back up is known, so that'd be my guess, but the details are always good reading either way).

indoordin0saur6• 9 months ago

Mine are more messed up now (12:30 ET) than they were this morning. AWS is lying that they've fixed the issue.

1970-01-013• 9 months ago

Someone, somewhere, had to report that doorbells went down because the very big cloud did not stay up.

I think we're doing the 21st century wrong.

Johnny555• 9 months ago

My Ring doorbell works just fine without an internet connection (or during a cloud outage). The video storage and app notifications are another matter, but the doorbell itself continues to ring when someone pushes the button.

lapetitejort2• 9 months ago

Someone, somewhere, had to report that rock throwers went down because the very big cloud did not stay up.

I think we're doing the 16th century wrong.

wartywhoa232• 9 months ago

Except we're not doing the 16th century right now.

amelius2• 9 months ago

Medium also.

nokeya• 9 months ago

Serverless is down because servers are down. What an irony.

thebruce87m• 9 months ago

Serverless is just someone elses server, or something

bicepjai• 9 months ago

Is this the outage that took Medium down ?

mmmlinux• 9 months ago

Ohno, not Fortnite! oh, the humanity.

TriangleEdge• 9 months ago

Severity - Degraded...

https://health.aws.amazon.com/health/status

https://downdetector.com/

mlhpdx8• 9 months ago

Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway).

My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other regions.

The service itself is also native multi-region and has multiple layers where failover happens (DNS, routing, destination selection).

Nothing’s perfect and there are many ways this setup could fail. It’s just cool that it worked this time - great to see.

Nothing I’ve done is rocket science or expensive, but it does require doing things differently. Happy to answer questions about it.

SteveNuts4• 9 months ago

Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted

This seems like such a low bar for 2025, but here we are.

immibis12• 9 months ago

You're also betting that CloudFront isn't one of the several AWS services that only works when us-east-1 is up.

AndrewKemendo4• 9 months ago

How did you do resilient auth for keys and certs?

mlhpdx8• 9 months ago

We use AWS for keys and certs, with aliases for keys so they resolve properly to the specific resources in each region. For any given HTTP endpoint there is a cert that is part of a the stack in that region (different regions use different certs).

The hardest part is that our customers' resources aren't always available in multiple regions. When they are we fall back to a region where they exist that is next closest (by latency, courtesy of https://www.cloudping.co/).

zild3d2• 9 months ago

active/active? curious what the data stack looks like as that tends to be the hard part

mlhpdx8• 9 months ago

The data layer is DynamoDB with Global Tables providing replication between regions, so we can write to any region. It's not easy to get this right, but our use case is narrow enough and rate of change low enought (intentionally) that it works well. That said, it still isn't clear that replication to us-east-1 would be perfect so we did "diff" tables just to be sure (it has been for us).

There is some S3 replication as well in the CI/CD pipeline, but that doesn't impact our customers directly. If we'd seen errors there it would mean manually taking Virginia out of the pipeline so we could deploy everyehere else.

ta124310• 9 months ago

Paying for resilience is expensive. not as expensive as AWS, but it's not free.

Modern companies live life on the edge. Just in time, no resilience, no flexibility. We see the disaster this causes whenever something unexpected happens - the Evergiven blocking Suez for example, let alone something like Covid

However increasingly what should be minor loss of resilience, like an AWS outage or a Crowdstrike incident, turns into major failures.

This fragility is something government needs to legislate to prevent. When one supermarket is out that's fine - people can go elsewhere, the damage is contained. When all fail, that's a major problem.

On top of that, the attitude that the entire sector has is also bad. People thing IT should tail once or twice a year and it's not a problem. If that attitude affect truly important systems it will lead to major civil projects. Any civilitsation is 3 good meals away from anarchy.

There's no profit motive to avoid this, companies don't care about being offline for the day, as long as all their mates are also offline.

webdoodle• 9 months ago

I in-housed an EMR for a local clinic because of latency and other network issues taking the system offline several times a month (usually at least once a week). We had zero downtime the whole first year after bringing it all in house, and I got employee of the month for several months in a row.

fogzen• 9 months ago

Great. Hope they’re down for a few more days and we can get some time off.

twistedpair3• 9 months ago

Wow, about 9 hours later and 21 of 24 Atlassian services are still showing up as impacted on their status page.

Even @ 9:30am ET this morning, after this supposedly was clearing up, my doctor's office's practice management software was still hosed. Quite the long tail here.

https://status.atlassian.com/

BiraIgnacio• 9 months ago

It's scary to think about how much power and perhaps influence the AWS platform has. (albeit it shouldn't be surprising)

megous• 9 months ago

I didn't even notice anything was wrong today. :) Looks like we're well disconnected from the US internet infra quasi-hegemony.

indoordin0saur6• 9 months ago

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

wavemode• 9 months ago

SEV-0 for my company this morning. We can't connect to RDS anymore.

assholesRppl2• 9 months ago

Yep, confirmed worse - DynamoDB now returning "ServiceUnavailableException"

claudiug• 9 months ago

ServiceUnavailableException hello java :)

dutzi• 9 months ago

Here as well…

perching_aix2• 9 months ago

In addition to those, Sagemaker also fails for me with an internal auth error specifically in Virginia. Fun times. Hope they recover by tomorrow.

autophagian• 9 months ago

Yeah. We had a brief window where everything resolved and worked and now we're running into really mysterious flakey networking issues where pods in our EKS clusters timeout talking to the k8s API.

cj00• 9 months ago

Yeah, networking issues cleared up for a few hours but now seem to be as bad as before.

JCM917• 9 months ago

Agree… still seeing major issues. Briefly looked like it was getting better but things falling apart again.

ljdtt• 9 months ago

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

parliament323• 9 months ago

Yes, although it's military in origin.

vishnugupta2• 9 months ago

It used to be quite common but has fallen out of usage.

strictnein• 9 months ago

FUBAR: Fucked Up Beyond All Recognition

Somewhat common. Comes from the US military in WW2.

sorentwo• 9 months ago

It is an old US military term that means “F*ked Up Beyond All Recognition”

loudmax2• 9 months ago

"FUBAR" comes up in the movie Saving Private Ryan. It's not a plot point, but it's used to illustrate the disconnect between one of the soldiers dragged from a rear position to the front line, and the combat veterans in his squad. If you haven't seen the movie, you should. The opening 20 minutes contains one of the most terrifying and intense combat sequences ever put to film.

whaleofatw2022• 9 months ago

Dangerous curiosity ask, is whether the number of folks off for Diwali is a factor or not?

I.e. lots of folks that weren't expected to work today and/or trying to round them up to work the problem.

junon4• 9 months ago

Seeing as how this is us-east-1, probably not a lot.

loudmax2• 9 months ago

Northern Virginia's Fairfax County public schools have the day off for Diwali, so that's not an unreasonable question.

In my experience, the teams at AWS are pretty diverse, reflecting the diversity in the area. Even if a lot of the Indian employees are taking the day off, there should be plenty of other employees to back them up. A culturally diverse employee base should mitigate against this sort of problem.

If it does turn out that the outage was prolonged due to one or two key engineers being unreachable for the holiday, that's an indictment of AWS for allowing these single points of failure to occur, not for hiring Indians.

hinkley20• 9 months ago

Seems like a lot of people missing that this post was made around midnight PST time and thus it would be more reasonable to ping people at lunch in IST before waking up people in EST or PST.

hinkley20• 9 months ago

Sometimes I miss my phone buzzing when doing yard work. Diwali has to be worse for that.

Forricide2• 9 months ago

Definitely seems to be getting worse, outside of AWS itself, more websites seem to be having sporadic or serious issues. Concerning considering how long the outage has been going.

busymom03• 9 months ago

That's probably why Reddit has been down too

PeterCorless• 9 months ago

Downdetector had 5,755 reports of AWS problems at 12:52 AM Pacific (3:53 AM Eastern).

That number had dropped to 1,190 by 4:22 AM Pacific (7:22 AM Eastern).

However, that number is back up with a vengeance. 9,230 reports as of 9:32 AM Pacific (12:32 Eastern).

Part of that could be explained by more people making reports as the U.S. west coast awoke. But I also have a feeling that they aren't yet on top of the problem.

rogerrogerr• 9 months ago

Where do they source those reports from? Always wondered if it was just analysis of how many people are looking at the page, or if humans somewhere are actually submitting reports.

outworlder• 9 months ago

I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close.

Our company decided years ago to use any region other than us-east-1.

Of course, that doesn't help with services that are 'global', which usually means us-east-1.

lordnacho2• 9 months ago

Is there some reason why "global" services aren't replicated across regions?

I would think a lot of clients would want that.

jedberg7• 9 months ago

Some AWS services are only available in us-east-1. Also a lot of people have not built their infra to be portable and the occasional outage isn't worth the cost and effort of moving out.

sleepybrett• 9 months ago

So did a previous company i worked at, all our stuff was in west-2.. then east-1 went down and some global backend services that aws depended on also went down and effected west-2.

I'm not sure a lot of companies are really looking at the costs of multi-region resiliency and hot failovers vs being down for 6 hours every year or so and writing that check.

perching_aix2• 9 months ago

cheapest + has the most capacity

andrewl-hn• 9 months ago

Several reasons, really:

1. The main one: it's the cheapest region, so when people select where to run their services they pick it because "why pay more?"

2. It's the default. Many tutorials and articles online show it in the examples, many deployment and other devops tools use it as a default value.

3. Related to n.2. AI models generate cloud configs and code examples with it unless asked otherwise.

4. It's location make it Europe-friendly, too. If you have a small service and you'd like to capture European and North American audience from a single location us-east-1 is a very good choice.

5. Many Amazon features are available in that region first and then spread out to other locations.

6. It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks. In hybrid cloud scenarios where you want to connect bits of your infrastructure running on AWS and on some physical hardware by a set of dedicated fiber optic lines us-east-1 is the place to do it.

7. Yes, for AWS deployments it's an experimental location that has higher risks of downtime compared to other regions, but in practice when a sizable part of us-east-1 is down other AWS services across the world tend to go down, too (along with half of the internet). So, is it really that risky to run over there, relatively speaking?

It's the world's default hosting location, and today's outages show it.

indoordin0saur6• 9 months ago

We have discussions coming up to evict ourselves from AWS entirely. Didn't seem like there was much of an appetite for it before this but now things might have changed. We're still small enough of a company to where the task isn't as daunting as it might otherwise be.

nijave6• 9 months ago

For us, we had some minor impacts but most stuff was stable. Our bigger issue was 3rd party SaaS also hosted on us-east-1 (Snowflake and CircleCI) which broke CI and our data pipeline

Eridrus• 9 months ago

This was a major issue, but it wasn't a total failure of the region.

Our stuff is all in us-east-1, ops was a total shitshow today (mostly because many 3rd party services besides aws were down/slow), but our prod service was largely "ok", a total of <5% of customers were significantly impacted because existing instances got to keep running.

I think we got a bit lucky, but no actual SLAs were violated. I tagged the postmortem as Low impact despite the stress this caused internally.

We definitely learnt something here about both our software and our 3rd party dependencies.

oofbey2• 9 months ago

One advantage to being in the biggest region: when it goes down the headlines all blame AWS, not you. Sure you’re down too, but absolutely everybody knows why and few think it’s your fault.

mjrpes• 9 months ago

Down detector agrees: https://downdetector.com/status/amazon/

Amazon says service is now just "degraded" and recovering, but searching for products on Amazon.com still does not work for me. https://health.aws.amazon.com/health/status

ilamont• 9 months ago

Search, Seller Central, Amazon Advertising not working properly for me. Attempting to access from New York.

When this is fixed, I am very interested in seeing recorded spend for Sunday and Monday.

jmuguy• 9 months ago

Yeah we were fine until about 1030 eastern and have been completely down since then, Heroku customer.

markus_zhang2• 9 months ago

It has been quite a while, wondering how many 9s are dropped.

365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.

codeduck3• 9 months ago

I'm sure they'll find some way to weasel out of this.

Veserv• 9 months ago

You added a zero. There are ~8760 hours per year, so 8 hours is ~1 in 1000, 99.9%.

rdtsc• 9 months ago

9s don’t have to drop if you increase the time period! “We still guarantee the same 9s just over 3450 years now”.

hvb210• 9 months ago

It's a single region?

I don't think anyone would quote availability as availability in every region I'm in?

While this is their most important region, there's a lot of clients that are probably unaffected if they're not in use1.

They COULD be affected even if they don't have anything there because of the AWS services relying on it. I'm just saying that most customers that are multi region should have their east region out and are just humming along.

oxfordmale• 9 months ago

But is is a partial outage only, so it doesn't count. If you retry a million times everything still works /s

beachy• 9 months ago

I haven't done any RFP responses for a while but this question always used to make me furious. Our competitors (some of who had had major incidents in the past) claimed 99.99% availability or more, knowing they would never have to prove it, and knowing they were actually 100% until the day they weren't.

We were more honest, and it probably cost us at least once in not getting business.

hinkley20• 9 months ago

Where were you guys the other day when someone was calling me crazy for trying to make this same sort of argument?

procaryote• 9 months ago

it's a matter of perspective... 9.9999% is real easy

mvdtnz2• 9 months ago

The problems now seem mostly related to starting new instances. Our capacity is slowly decaying as existing services spin down and new EC2 workloads fail to start.

steveBK1236• 9 months ago

Agreed, every time the impacted services list internally gets shorter, the next update it starts growing again.

A lot of these are second order dependencies like Astronomer, Atlassian, Confluent, Snowflake, Datadog, etc... the joys of using hosted solutions to everything.

hinkley20• 9 months ago

Before my old company spun off, we didn’t know the old ops team had put on-prem production and our Atlassian instances in the same NAS.

When the NAS shit the bed, we lost half of production and all our run books. And we didn’t have autoscaling yet. Wouldn’t for another 2 years.

Our group is a bunch of people that has no problem getting angry and raising voices. The whole team was so volcanically angry that it got real quiet for several days. Like everyone knew if anyone unclenched that there would be assault charges.

throwaway-aws9• 9 months ago

You have to remember that health status dashboards at most (all?) cloud providers require VP approval to switch status. This stuff is not your startup's automated status dashboard. It's politics, contracts, money.

hinkley20• 9 months ago

Which makes them a flat out lie since it ceases to be a dashboard if it’s not live. It’s just a status page.

davedx2• 9 months ago

Andy Jassy is the Tim Cook of Amazon

Rest and vest CEOs

hinkley20• 9 months ago

Don’t insult Tim Cook like that.

He got a lot of impossible shit done as COO.

They do need a more product minded person though. If Jobs was still around we’d have smart jewelry by now. And the Apple Watch would be thin af.

napolux2• 9 months ago

worst of all: ring alarm unstoppable siren because the app is down and the keyboard was removed by my parents and put "somewhere in the basement".

bartread7• 9 months ago

Is it hard wired? If so, and if the alarm module doesn’t have an internal battery, can you go to the breaker box and turn off the circuit it’s on? You should be able to switch off each breaker in turn until it stops if you don’t know which circuit it’s on.

If it doesn’t stop, that means it has a battery backup. But you can still make life more bearable. Switch off all your breakers (you probably have a master breaker for this), then open up the alarm box and either pull the battery or - if it’s non-removable - take the box off the wall, put it in a sealed container, and put the sealed container somewhere… else. Somewhere you can’t hear it or can barely hear it until the battery runs down.

Meanwhile you can turn the power back on but make sure you’ve taped the bare ends of the alarm power cable, or otherwise electrically insulated them, until you’re able to reinstall it.

belter4• 9 months ago

This looks like one their worst outage in 15 years and us-east-1 still shows as degraded but I had no outages, as dont use us-east-1. Are you seeing issues on other regions?

https://health.aws.amazon.com/health/status?path=open-issues

The closest to their identification of a root cause seems to be this one:

"Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations."

hinkley20• 9 months ago

I wonder how many people discovered their autoscaling settings went batshit when services went offline, either scaling way down or way up, or went metastable and started fishtailing.

tlogan• 9 months ago

I noticed the same thing and it seems to have gotten much worse around 8:55 a.m. Pacific Time.

By the way, Twilio is also down, so all those login SMS verification codes aren’t being delivered right now.

baubino• 9 months ago

Basic services at my worksite have been offline for almost 8 hours now (things were just glitchy for about 4 hours before that). This is nuts.

indoordin0saur6• 9 months ago

Have not gotten a data pipeline to run to success since 9AM this morning when there was a brief window of functioning systems. Been incredibly frustrating seeing AWS tell the press that things are "effectively back to normal". They absolutely are not! It's still a full outage as far as we are concerned.

jread• 9 months ago

Lambda create-function control plane operations are still failing with InternalError for us - other services have recovered (Lambda, SNS, SQS, EFS, EBS, and CloudFront). Cloud availability is the subject of my CS grad research, I wrote a quick post summarizing the event timeline and blast radius as I've observed it from testing in multiple AWS test accounts: https://www.linkedin.com/pulse/analyzing-aws-us-east-1-outag...

jonplackett2• 9 months ago

The problem is now that, what’s anyone going to do? Leave?

I remember a meme years ago about Nestle. It was something like: GO ON, BOYCOT US - I BET YOU CAN’T - WE MAKE EVERYTHING.

Same meme would work for Aws today.

MaKey• 9 months ago

Same meme would work for Aws today.

Not really, there are enough alternatives.

hinkley20• 9 months ago

It’s amazing how much you can avoid them by eating food that still looks like what it started as though. They own a lot of processed food.

ivape• 9 months ago

There are entire apps like Reddit that are still not working. What the fuck is going on?

htrp• 9 months ago

thundering herd problems.... every time they say they fix it something else breaks

altbdoor• 9 months ago

Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS service. I asked if it was a single point of failure. The whole room laughed, I rest my case.

Elidrake24• 9 months ago

If you were dependent upon a single distribution (region) of that Service, yes it would be a massive single point of failure in this case. If you weren't dependent upon a particular region, you'd be fine.

zimbu668• 9 months ago

Of course lots of AWS services have hidden dependencies on us-east-1. During a previous outage we needed to update a Route53(DNS) record in us-west-2, but couldn't because of the outage in us-east-1.

wubrr• 9 months ago

Some 'regional' AWS services still rely on other services (some internal) that are only in us-east-1.

antinomicus• 9 months ago

Even Amazon’s own services (ie ring) were affected by this outage

ta124310• 9 months ago

Relying on AWS is a single point of failure. Not as much as relying on a single AWS region, but it's still a single point.

It's fairly difficult to avoid single points of failure completely, and if you do it's likely your suppliers and customers haven't managed to.

It's about how much your risk level is.

AWS us-east-1 fails constantly, it has terrible uptime, and you should expect it to go. A cyberattack which destroyed AWSs entire infrastructure would be less likely. BGP hijacks across multiple AWS nodes are quite plausible though, but that can be mitigated to an extent with direct connects.

Sadly it seems people in charge of critical infrastructure don't even bother thinking about these things, because next quarters numbers are more important.

I can avoid London as a single point of failure, but the loss of Docklands would cause so much damage to the UK's infrastructure I can't confidently predict that my servers in Manchester connected to peering points such as IXman will be able to reach my customer in Norwich. I'm not even sure how much international connectivity I could rely on. In theory Starlink will continue to work, but in practice I'm not confident.

When we had power issues in Washington DC a couple of months ago, three of our four independent ISPS failed, as they all had undeclared requirements on active equipment in the area. That wasn't even a major outage, just a local substation failure. The one circuit which survived was clearly just fibre from our (UPS/generator backed) equipment room to a data centre towards Baltimore (not Ashburn).

aeve890• 9 months ago

Similar experience here. People laughed and some said something like "well, if something like AWS falls then we have bigger problems". They laugh because honestly is too far-fetched to think the whole AWS infra going down. Too big to fail as they say in the US. Nothing short of a nuclear war would fuck up the entire AWS network so they're kinda right.

Until this happen. A single region in a cascade failure and your saas is single region.

stephenlf• 9 months ago

They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook.

lxgr• 9 months ago

Nothing short of a nuclear war, a bad deploy, or some operational oopsie, and everybody knows how rare all these things are!

steveBK1236• 9 months ago

I mean given what we've seen with these AWS failures impact, wouldn't any enemies first target be to hit us-east-1 ? Imagine if it just disappeared?

dvsgaevsvsgavsv• 9 months ago

Amazing. So you will build your own load balancer that sends loads between AWS and Gcloud and make it the single point of failure instewd?

bigbuppo2• 9 months ago

Whose idea was it to make the whole world dependent on us-east-1?

nemomarx• 9 months ago

The NSA might be happy everything runs through a local data center to their Virginia offices

kondro9• 9 months ago

The most recent public count of datacenters for AWS in us-east-1 is 159. I suspect that’s even an unwieldy number for NSA to spy on.

lenerdenator2• 9 months ago

1) People in us-east-1.

2) People who thought that just having stuff "in the cloud" meant that it was automatically spread across regions. Hint, it's not; you have to deploy it in different regions and architect/maintain around that.

3) Accounting.

wting• 9 months ago

Eh, us-east-1 is the oldest AWS region and if you get some AWS old timers talking over some beers they'll point out the legacy SPOFs that still exist in us-east-1.

rickmode• 9 months ago

Isn’t it the cheapest AWS region? Or at least among the cheapest. If I’m correct, this incentivizes users to start there.

everfrustrated5• 9 months ago

All countries are more expensive than the US, but many US regions are the same (cheapest) price. (eg Ohio, Oregon).

us-east-1 is more a legacy of it being the first region so by virtue of being the default region for the longest time most customers built on it.

alchemism• 9 months ago

AWS often deploys its new platform products and latest hardware (think GPUs) into us-east-1, so everyone has to maintain a footprint in us-east-1 to use any of it.

So as a result, everyone keeps production in us-east-1. =)

bigbuppo2• 9 months ago

Not to mention, even if you went through the hassle of a diverse multi-cloud deployment, there's still something in your stack that has a dependency on us-east-1, and it's probably some weird frontend javascript module that uses a floatilla of free tier lambda services to format dates.

redwood• 9 months ago

Surprising and sad to see how many folks are using DynamoDB There are more full featured multi-cloud options that don't lock you in and that don't have the single point of failure problems.

And they give you a much better developer experience...

Sigh

pardner• 9 months ago

Darn, on Heroku even the "maintenance mode" (redirects all routes to a static url) won't kick in.

JCM917• 9 months ago

At 3:03 AM PT AWS posted that things are recovering and sounded like issue was resolved.

Then things got worse. At 9:13 AM PT it sounds like they’re back to troubleshooting.

Honestly sounds like AWS doesn’t even really know what’s going on. Not good.

vishnugupta2• 9 months ago

This is exacerbated by the fact that this is Diwali week which means the most of Indian engineers will be out on leave. Tough luck.

Waterluvian3• 9 months ago

I know there's a lot of anecdotal evidence and some fairly clear explanations for why `us-east-1` can be less reliable. But are there any empirical studies that demonstrate this? Like if I wanted to back up this assumption/claim with data, is there a good link for that, showing that us-east-1 is down a lot more often?

everfrustrated5• 9 months ago

The unreliability claim is driven by two factors.

1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. Most services start with us-east-1 first.

2. us-east-1 is MASSIVE and considerably larger than the next largest region. There's no public numbers but I wouldn't be surprised if it was 50% of their global capacity. An outage in any other region never hits the news.

iLoveOncall• 9 months ago

1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time.

This is true.

Most services start with us-east-1 first.

This is absolutely false. Almost every service will FINISH with the largest and most impactful regions.

kixiQu• 9 months ago

a pipeline which pushes change to regions one at a time
When AWS deploys updates to its services, deployments to Availability Zones in the same Region are separated in time to prevent correlated failure.

https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

jppope2• 9 months ago

I don't think its fair to dismiss a lot of anecdotal evidence, much of human experience is based off of it, and just being anecdotal doesn't make it incorrect. For those of us using aws for the last decade, there have been a handful of outages that are pretty hard to forget. Often those same engineers have services in other regions - so we witness these things going down more frequently in us-east-1. Now can I say definitively that us-east-1 goes down the most - nope. Have I had 4 outages in us-east-1 I can remember and only 1-2 us-west-2, yep.

Waterluvian3• 9 months ago

Where are you getting the sense that anecdotal evidence is being dismissed?

Liftyee2• 9 months ago

Damn. This is why Duolingo isn't working properly right now.

redeux2• 9 months ago

It’s a good day to be a DR software company or consultant

sineausr931• 9 months ago

On a bright note, Alexa has stopped pushing me merchandise.

neuroelectron• 9 months ago

Sounds like a circular error with monitoring is flooding their network with metrics and logs, causing DNS to fail and produce more errors, flooding the network. Likely root cause is something like DNS conflicts or hosts being recreated on the network. Generally this is a small amount of network traffic but the LBs are dealing with host address flux, causing the hosts to keep colliding host addresses as they attempt to resolve to a new host address which are being lost from dropped packets and with so many hosts in one AZ, there's a good chance they end up with a new conflicting address.

czhu12• 9 months ago

Our entire data stack (Databricks and Omni) are all down for us also. The nice thing is that AWS is so big and widespread that our customers are much more understanding about outages, given that its showing up on the news.

busymom03• 9 months ago

For me Reddit is down and also the amazon home page isn't showing any items for me.

aaronbrethorst• 9 months ago

My ISP's DNS servers were inaccessible this morning. Cloudflare and Google's DNS servers have all been working fine, though: 1.1.1.1, 1.0.0.1, and 8.8.8.8

Readerium• 9 months ago

99.999 percent lol

motbus3• 9 months ago

Always a lovely Monday when you wake just in time to see everything going down

suralind• 9 months ago

I wonder how their nines are going. Guess they'll have to stay pretty stable for the next 100 years.

itqwertz• 9 months ago

Did they try asking Claude to fix these issues? If it turns out this problem is AI-related, I'd love to see the AAR.

ecommerceguy• 9 months ago

Just tried to get into Seller Central, returned a 504.

twistedpair3• 9 months ago

I just saw services that were up since 545AM ET go down around 12:30PM ET. Seems AWS has broken Lambda again in their efforts to fix things.

wcchandler• 9 months ago

This is usually something I see on Reddit first, within minutes. I’ve barely seen anything on my front page. While I understand it’s likely the subs I’m subscribed to, that was my only reason for using Reddit. I’ve noticed that for the past year - more and more tech heavy news events don’t bubble up as quickly anymore. I also didn’t see this post for a while for whatever reason. And Digg was hit and miss on availability for me, and I’m just now seeing it load with an item around this.

I think I might be ready to build out a replacement through vibe coding. I don’t like being dependent on user submissions though. I feel like that’s a challenge on its own.

postexitus• 9 months ago

Most Reddit API is down as well.

kccqzy• 9 months ago

Reddit itself is having issues. I have multiple comments fail to post. And the Reddit user page leads me to a 404.

qingcharles2• 9 months ago

Yeah, Reddit has been half-working all morning. Last time this happened I had an account get permabanned because the JavaScript on the page got stuck in a no-backoff retry loop and it banned me for "spamming." Just now it put me in rate limit jail for trying to open my profile one time, so I've closed out all my tabs.

ryanisnan• 9 months ago

Anecdotally, I think you should disregard this. I found out about this issue first via Reddit, roughly 30 minutes after the onset (we had an alarm about control plane connectivity).

midtake• 9 months ago

Reddit is worthless now, and posting about your tech infrastructure on reddit is a security and opsec lapse. My workplace has reddit blocked at the edge. I would trust X more than reddit, and that is with X having active honeypot accounts (it is even a meme about Asian girls). In fact, heard about this outage on X before anywhere else.

Krasnol• 9 months ago

I found it pretty fast on the /r/signal sub and went from there.

busymom03• 9 months ago

For me, Reddit fails to load. Gives some "upstream connection error"

Bolwin• 9 months ago

Try lemmy, it's currently in the front page, albeit on a meme sub

solatic• 9 months ago

And yet, AMZN is up for the day. The market doesn't care. Crazy.

wartywhoa232• 9 months ago

Someone vibecoded it down.

stego-tech2• 9 months ago

Not remotely surprised. Any competent engineer knows full well the risk of deploying into us-east-1 (or any “default” region for that matter), as well as the risks of relying on global services whose management or interaction layer only exists in said zone. Unfortunately, us-east-1 is the location most outsourcing firms throw stuff, because they don’t have to support it when it goes pear-shaped (that’s the client’s problem, not theirs).

My refusal to hoard every asset into AWS (let alone put anything of import in us-east-1) has saved me repeatedly in the past. Diversity is the foundation of resiliency, after all.

mcintyre19942• 9 months ago

as well as the risks of relying on global services whose management or interaction layer only exists in said zone.

Is this well known/documented? I don't have anything on AWS but previously worked for a company that used it fairly heavily. We had everything in EU regions and I never saw any indication/warning that we had a dependency on us-east-1. But I assume we probably did based on the blast radius of today's outage.

captainkrtek3• 9 months ago

Some of the “global” and “edge” services depend on us-east-1.

See: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

“In the aws partition, the IAM service’s control plane is in the us-east-1 Region, with isolated data planes in each Region of the partition.“

Also, intra-region, many of the services use eachother, and not in a manner where the public can discern the dependency map.

toephu2• 9 months ago

Half the internet goes down because part of AWS goes down... what happened to companies having redundant systems and not having a single point of failure?

jppope2• 9 months ago

Ironically for most companies its cheaper to just say if AWS goes down half of the internet goes down so people will understand

rdm_blackhole2• 9 months ago

My app deployed on Vercel and therefore indirectly deployed on us-east-1 was down for about 2 hours today then came back up and then went down again 10 minutes ago for 2 or 3 minutes. It seems like they are still intermittent issues happening.

lawlessone• 9 months ago

Am i imagining it or are more things like this happening in recent weeks than usual?

Isuckatcode• 9 months ago

Man , I just wanted to enjoy celebrating Diwali with my family but been up from 3am trying to recover our services. There goes some quality time

melozo3• 9 months ago

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

NelsonMinar• 9 months ago

I saw a quote from a high end AWS support engineer that said something like "submitting tickets for AWS problems is not working reliably: customers are advised to keep retrying until the ticket is submitted".

willsmith722• 9 months ago

It seems reasonable to me that Amazon (retail) would build better AZ redundancy into their services than say Snapchat or a bank

melozo3• 9 months ago

Sure, but it’s not reasonable that internal collaboration platforms built for ticketing engineers about outages doesn’t work during the outage. That would be something worth making multi-region at a minimum.

nettlin• 9 months ago

The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform.

Some parts of amazon.com seem to be affected by the outage (e.g. product search: https://x.com/wongmjane/status/1980318933925392719)

TheJoeMan• 9 months ago

Reviewing order history was also spotty. According to my wife “keep hitting refresh getting different dogs each time until it works”

stevejb• 9 months ago

Amazon's revenue in 2024 was about the size of Belgium's GDP. Higher than Sweden or Ireland. It makes a profit similar to Norway, without drilling for offshore oil or maintaining a navy. I think they've got plenty of juice left.

melozo3• 9 months ago

You’re right about that. I guess what I mean is, how long will people be enthusiastic about AWS and its ability to innovate. But AWS undeniably has some really strong product offerings - it’s just that their pace of innovation has slowed. Their managed solutions for open source applications is generally good, but some of their bespoke alternatives are lacking over the last few years (ecs kinesis code* tools) - it wasn’t always like that (sqs ddb s3 ec2).

notJim• 9 months ago

The navy comment is a bit unfair, as it's well-known that Amazon is more of an airpower (hence "the cloud" etc.)

shepherdjerred• 9 months ago

You could argue Amazon's security is an irregular military force

underlipton• 9 months ago

The universe's metaphysical poeticism holds that it's slightly more likely than it otherwise would be that the company that replaced Sears would one day go the way of Sears.

qingcharles2• 9 months ago

Amazon customer support had a banner saying it was unavailable for most of the day. Couldn't get anything less than 5 day shipping on any item today.

neon_me• 9 months ago

"serverless"

JPKab• 9 months ago

The length and breadth of this outage has caused me to lose so much faith in AWS. I knew from colleagues who used to work there how understaffed and inefficient the team is due to bad management, but this just really concerns me.

llmslave3• 9 months ago

"Tech people" are long gone, most projects are death marches of technical debt

0x5345414e• 9 months ago

This is having a direct impact on my wellbeing. I was at Whole Foods in Hudson Yards NYC and I couldn’t get the prime discount on my chocolate bar because the system isn’t working. Decided not to get the chocolate bar. Now my chocolate levels are way too low.

pewpew_• 9 months ago

I was attempting to use self checkout for some lunch I grabbed from the hotbar and couldn’t understand why my whole foods barcode was failing. It took me a full 20 seconds to realize the reason for the failure.

tonymet10• 9 months ago

"alexa turn on coffee pot" stopped working this morning, and I'm going bonkers.

jdlyga2• 9 months ago

Alexa is super buggy now anyway. I switched my Echo Dot to Alexa+, and it fails turning on and off my Samsung TV all the time now. You usually have to do it twice.

dewarrn1• 9 months ago

This is a fun example, but now you've got me wondering: has anyone checked on folks who might have been in an Amazon Go store during the outage?

nxpnsv2• 9 months ago

Life indeed is a struggle

burnt-resistor• 9 months ago

First World treatlerite problems. /s What's going to suck years after too many SREs/SWEs will have long been fired, like the Morlocks & Eloi and Idiocracy, there won't be anyone left who can figure out that plants need water. There will be a few trillionaires surrounded by aristocratic, unimaginable opulence while most of humanity toils in favelas surrounded by unfixable technology that seems like magic. One cargo cult will worship 5.25" floppy disks and their arch enemies will worship CD-Rs.

https://xkcd.com/2347/

TZubiri• 9 months ago

That's it, internet centralization has gone too far, call your congress(wo)man

colechristensen• 9 months ago

I had to buy a donut and the gas station with cash, like a peasant.

dabinat• 9 months ago

My site was down for a long time after they claimed it was fixed. Eventually I realized the problem lay with Network Load Balancers so I bypassed them for now and got everything back up and running.

YouAreWRONGtoo3• 9 months ago

I don't get how you can be a trillion dollar company and still suck this much.

jjice• 9 months ago

We got off pretty easy (so far). Had some networking issues at 3am-ish EDT, but nothing that we couldn't retry. Having a pretty heavily asynchronous workflow really benefits here.

One strange one was metrics capturing for Elasticache was dead for us (I assume Cloudwatch is the actual service responsible for this), so we were getting no data alerts in Datadog. Took a sec to hunt that down and realize everything was fine, we just don't have the metrics there.

I had minor protests against us-east-1 about 2.5 years ago, but it's a bit much to deal with now... Guess I should protest a bit louder next time.

motiejus2• 9 months ago

Too big to recover.

jrm4• 9 months ago

Hey wait wasn't the internet supposed to route around...?

BryanBeshore• 9 months ago

https://www.youtube.com/shorts/liL2VXYNyus

cpncrunch• 9 months ago

"The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers."

https://health.aws.amazon.com/health/status?path=service-his...

vmnb• 9 months ago

Ah, it's just what I thought. An underlying internal subsystem.

Aldipower5• 9 months ago

altavista.com is also down!

the-chitmonger• 9 months ago

I'm not sure if this is directly related, but I've noticed my Apple Music app has stopped working (getting connection error messages). Didn't realize the data for Music was also hosted on AWS, unless this is entirely unrelated? I've restarted my phone and rebooted the app to no avail, so I'm assuming this is the culprit.

raw_anon_11115• 9 months ago

From the great Corey Quinn

Ah yes, the great AWS us-east-1 outage.

Half the internet’s on fire, engineers haven’t slept in 18 hours, and every self-styled “resilience thought leader” is already posting:

“This is why you need multi-cloud, powered by our patented observability synergy platform™.”

Shut up, Greg.

Your SaaS product doesn’t fix DNS, you're simply adding another dashboard to watch the world burn in higher definition.

If your first reaction to a widespread outage is “time to drive engagement,” you're working in tragedy tourism. Bet your kids are super proud.

Meanwhile, the real heroes are the SREs duct-taping Route 53 with pure caffeine and spite.

https://www.linkedin.com/posts/coquinn_aws-useast1-cloudcomp...

Rooster61• 9 months ago

This wins all the internets today. Probably not a day where internets are particularly valuable, but it wins them nonetheless.

raw_anon_11115• 9 months ago

He once said about an open source project that I was the third highest contributor on at AWS “This may be the worst AWS naming of 2021.” It was one of the proudest moments in my career.

Yes I know it’s sad…

t1234s• 9 months ago

Do events like this stir conversations in small to medium size businesses to escape the cloud?

FigurativeVoid• 9 months ago

It would have to be catastrophic for most businesses to make think about escaping the cloud. The cost of migration and maintenance are massive for small and medium businesses.

rovr138• 9 months ago

Depends how small.

I have clients and I’ve heard “even Amazon is down, we can be down” more than once.

tonymet10• 9 months ago

This isn't a "cloud failure". All of these apps would be running now had they spent the additional 5% development costs to add failover to another region.

decimalenough4• 9 months ago

us-east1 is supposed to consist of a number of "availability zones" that are independent and stay "available" even if one goes down. That's clearly not happening, so yes, this is very much a cloud failure.

ilikecakeandpie2• 9 months ago

Probably, but they're usually dropped after they come back up

tonymet10• 9 months ago

I don't think blaming AWS is fair, since they typically exceed their regional and AZ SLAs

AWS makes their SLAs & uptime rates very clear, along with explicit warnings about building failover / business continuity.

Most of the questions on the AWS CSA exam are related to resiliency .

Look, we've all gone the lazy route and done this before. As usual, the problem exists between the keyboard and the chair.

dijit12• 9 months ago

Not sure any of their SLA’s are covered here.

If they don’t obfuscate the downtime (they will, of course), this outage would put them at, what, two nines? Thats very much out of their SLA.

People also keep talking about it as if its one region, but there are reports in this thread of internal dependencies inside AWS which are affecting unrelated regions with various services. (r53 updates for example)

tonymet10• 9 months ago

Sounds like your lesson is "yes we should continue shaming AWS rather than fix our app"

Tractor8626• 9 months ago

They supposed to exceed their SLA. SLA is guaranteed worst case.

tonymet10• 9 months ago

If you don't break anything, you aren't making anything valuable.

chermi• 9 months ago

Stupid question, why isn't the stock down? Couldn't this lead to people jumping to other providers and at the very least require some pretty big fees for do dramatically breaking SLA? Is it just not a biggest fraction of revenue to matter?

Archonical• 9 months ago

Maybe since Amazon is reporting Q3 numbers soon and this will only show up in Q4 numbers?

ilikecakeandpie2• 9 months ago

Lots of stock brokerages consumer offerings are served through....

....AWS!

chasd0011• 9 months ago

Robinhood is down ;)

Capricorn2481• 9 months ago

Non-technical people don't really notice these things. They hear it and shrug, because usually it's fixed within a day.

CNBC is supposed to inform users about this stuff, but they know less than nothing about it. That's why they were the most excited about the "Metaverse" and telling everyone to get on board (with what?) or get left behind.

The market is all about perception of value. That's why Musk can tweet a meme and double a stocks price, it's not based in anything real.

chasd0011• 9 months ago

wow I think most of Mulesoft is down, that's pretty significant in my little corner of the tech world.

8cvor6j844qw_d6• 9 months ago

That's unusual.

I wss under the impression that having multiple available zones guarantees high availability.

It seems this is not the case.

ykl• 9 months ago

One of the open secrets of AWS is that even though AWS has a lot of regions and availability zones, a lot of AWS services have control planes that are dependent on / hosted out of us-east-1 regardless of which region / AZ you're using, meaning even if you are using a different availability zone in a different region, us-east-1 going down still can mess you up.

worik5• 9 months ago

This outage is a reminder:

Economic efficiency and technical complexity are both, separately and together, enemies of resilience

_pvzn• 9 months ago

Can confirm, also getting hit with this.

EbNar• 9 months ago

May be because of this that trying to pay with PayPal on Lenovo's website has failed thrice for me today? Just asking... Knowing how everything is connected nowadays it wouldn't surprise me at all.

hippo77• 9 months ago

Finally an upside to running on Oracle Cloud!

haunter• 9 months ago

The Premier League said there will be only limited VAR today w/o the automatic offside system becasue of the AWS outage. Weird timeline we live in

https://www.bbc.com/news/live/c5y8k7k6v1rt?post=asset%3Ad902...

magarnicle• 9 months ago

A silver lining to this cloud (outage).

iammrpayments• 9 months ago

Why is VAR connected to the internet? Are they trying to gather data on offside players customers to improve recommedations?

ashikns• 9 months ago

I worked in a similar system. The raw data from the field first goes to a cloud hosted event queue of some sort, then a database, then back to whatever app/screen on field. The data doesn't just power on-field displays. There's a lot of online websites, etc that needs to pull data from an api.

afavour• 9 months ago

I wouldn't be at all surprised if people pay for API access to the data. I've worked with live sports data before, it's a very profitable industry to be in when you're the one selling the data.

Of course in a sane world you'd have an internal fallback for when cloud connectivity fails but I'm sure someone looked at the cost and said "eh, what's the worst that could happen?"

l33tnull• 9 months ago

I can't do anything for school because Canvas by Instructure is down because of this.

arrty88• 9 months ago

I expect gcp and azure to gain some customers after this

kevinsundar• 9 months ago

AWS pros know to never use us-east-1. Just don't do it. It is easily the least reliable region

j45• 9 months ago

More and more I want to be could agnostic or multi-cloud.

rwke• 9 months ago

With more and more parts of our lives depending on often only one cloud infrastructure provider as a single point of failure, enabling companies to have built-in redundancy in their systems across the world could be a great business.

Humans have built-in redundancy for a reason.

IOT_Apprentice• 9 months ago

Apparently IMDb, an Amazon service is impacted. LOL, no multi region failover.

iwontberude• 9 months ago

worst outage since xmas time 2012

artyom• 9 months ago

Amazon has spent most of its HR post-pandemic efforts in:

• Laying off top US engineering earners.

• Aggressively mandating RTO so the senior technical personnel would be pushed to leave.

• Other political ways ("Focus", "Below Expectations") to push engineering leadership (principal engineers, etc) to leave, without it counting as a layoff of course.

• Terminating highly skilled engineering contractors everywhere else.

• Migrating serious, complex workloads to entry-level employees in cheap office locations (India, Spain, etc).

This push was slow but mostly completed by Q1 this year. Correlation doesn't imply causation? I find that hard to believe in this case. AWS had outages before, but none like this "apparently nobody knows what to do" one.

Source: I was there.

1970-01-013• 9 months ago

Completely detached from reality, AMZN has been up all day and closed up 1.6%. Wild.

Loughla• 9 months ago

Until this impacts their bottom line, how is that unexpected?

Will we see mass exits from their service? Who knows. My money says no though.

How many companies can just ride the "but it's not our fault" to buy time with customers until it's fixed?

homeonthemtn• 9 months ago

"We should have a fail back to US-West."

"It's been on the dev teams list for a while"

"Welp....."

nullorempty• 9 months ago

It won't be over until long after AWS resolves it - the outages produce hours of inconsistent data. It especially sucks for financial services, things of eventual consistency and other non-transactional processes. Some of the inconsistencies introduced today will linger and make trouble for years.

president_zippy2• 9 months ago

I wonder how much better the uptime would be if they made a sincere effort to retain engineering staff.

Right now on levels.fyi, the highest-paying non-managerial engineering role is offered by Oracle. They might not pay the recent grads as well as Google or Microsoft, but they definitely value the principal engineers w/ 20 years of experience.

valdiorn• 9 months ago

I missed a parcel delivery because a computer server in Virginia, USA went down, and now the doorbell on my house in England doesn't work. What. The. Fork.

How the hell did Ring/Amazon not include a radio-frequency transmitter for the doorbell and chime? This is absurd.

To top it off, I'm trying to do my quarterly VAT return, and Xero is still completely borked, nearly 20 hours after the initial outage.

hacker_homie• 9 months ago

It's always DNS

AtomicOrbital• 9 months ago

https://m.youtube.com/watch?v=KFvhpt8FN18 clear detailed explanation of the AWS outage and how properly designed systems should have shielded the issue with zero client impact

dorongrinstein• 9 months ago

Anyone needing multi-cloud WITH EASE, please get in touch. https://controlplane.com

I am the CEO of the company and started it because I wanted to give engineering teams an unbreakable cloud. You can mix-n-match services of ANY cloud provider, and workloads failover seamlessly across clouds/on-prem environments.

Feel free to get in touch!

rsanheim• 9 months ago

I wonder what kind of outage or incident or economic change will be required to cause a rejection of the big commercial clouds as the default deployment model.

The costs, performance overhead, and complexity of a modern AWS deployment are insane and so out of line with what most companies should be taking on. But hype + microservices + sunk cost, and here we are.

newZWhoDis• 9 months ago

Honest answer? The outage would need to last about a week.

babl-yc• 9 months ago

I don't expect the majority of tech companies to want to run their own physical data centers. I do expect them to shift to more bare-metal offerings.

If I'm a mid to large size company built on DynamoDB, I'd be questioning if it's really worth the risk given this 12+ hour outage.

I'd rather build upon open source tooling on bare metal instances and control my own destiny, than hope that Amazon doesn't break things as they scale to serve a database to host the entire internet.

For big companies, it's probably a cost savings too.

baobabKoodaa2• 9 months ago

For big companies, it's probably a cost savings too.

For any sized company, moving away from big clouds back onto traditional VPS or bare-metal offerings will lead to cost savings.

a-dub• 9 months ago

i am amused at how us-east-1 is basically in the same location as where aol kept its datacenters back in the day.

fastball• 9 months ago

One of my co-workers was woken up by his Eight Sleep going haywire. He couldn't turn it off because the app wouldn't work (presumably running on AWS).

cranberryturkey2• 9 months ago

Sling still down at 11:42PM PST

cranberryturkey2• 9 months ago

Kraken can't do deposits either at 3:38am PST

ronakjain90• 9 months ago

we[1] operate out of `us-east-1` but chose to not use any of the cloud based vendor lockin (sorry vercel, supabase, firebase, planetscale etc). Rather a few droplets in DigitalOcean(us-east-1) and Hetzner(eu). We serve 100 million requests/mo, few million user generated content(images)/mo at monthly cost of just about $1000/mo.

It's not difficult, it's just that we engineers chose convenience and delegated uptime to someone else.

[1] https://usetrmnl.com

throw-10-133• 9 months ago

imagine spending millions on devops and sre to still have your mission critical service go down because amazon still has baked in regional dependencies

teunlao• 9 months ago

us-east-1 down again. We all know we should leave. None of us will.

michaelcampbell2• 9 months ago

Anthem Health call center disconnected my wife numerous times yesterday with an ominous robo-message of "Emergency in our call center"; curious if that was this. Seems likely, but what a weird message.

ky_vulnerable• 9 months ago

Do we know what caused the outage yet?

AbstractH248• 9 months ago

Are there websites that do post-mortems for how the single points of failure impacted the entire internet?

Not just AWS, but Cloudflare and others too. Would be interesting to review them clinically.

jodrellblank• 9 months ago

Another time to link The Machine Stops by E.M. Forster, 1909: https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/th...

“The Machine,” they exclaimed, “feeds us and clothes us and houses us; through it we speak to one another, through it we see one another, in it we have our being. The Machine is the friend of ideas and the enemy of superstition: the Machine is omnipotent, eternal; blessed is the Machine.”

"she spoke with some petulance to the Committee of the Mending Apparatus. They replied, as before, that the defect would be set right shortly. “Shortly! At once!” she retorted"

"there came a day when, without the slightest warning, without any previous hint of feebleness, the entire communication-system broke down, all over the world, and the world, as they understood it, ended."

0xbadcafebee• 9 months ago

We never went down in us-east-1 during this incident. We have tons of high-traffic sites/services. Not multi-region, not multi-cloud.

You're gonna hear mostly complaints in this thread, but simple, resilient, single-region architecture is still reliable as hell in AWS, even in the worst region.

AWS multiple services outage in us-east-1

Related Stories