Tell HN: CloudFront Is Having an Outage

https://news.ycombinator.com/item?id=36775816

by klinskyc • 3 years ago

53 24 3 years ago

news.ycombinator.com

Sites hosted via Cloudfront

(e.g https://app.circleci.com/) are having an outage and returning 421s and 500s currently

snacktaster• 3 years ago

All our HTTPS (ACM) sites are returning:

421 ERROR
The request could not be satisfied.
The distribution does not match the certificate for which the HTTPS connection was established with. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

gregmfoster• 3 years ago

Seeing 421 on our AWS amplify splash page as well

ssabev2• 3 years ago

Well, that explains wtf is happening. You guys seeing this on us-east-1 or across?

romanhotsiy• 3 years ago

us-east-1 only for us

flangola7• 3 years ago

Seeing it across 3 regions at minimum

Aaronstotle• 3 years ago

saw it across us-east-1 mainly

neuronexmachina• 3 years ago

As the saying goes: Friends don't let friends rely on us-east-1. (It was the very first region, and it shows)

hinkley• 3 years ago

I will never understand why anyone in our infrastructure team thought for a moment that putting our backup datacenter in us-east-1. And why nobody else tried to get him fired.

That's like buying a backup generator from a guy that wants to meet you in the Walmart parking lot.

messe• 3 years ago

Having worked at AWS: us-east-1 is probably the easiest region to deploy in (it's old, it's popular, every AWS service is available there), but might just be the worst region to rely on (it's old, it's popular, and every AWS service is available there—but only most of the time).

I vividly remember when AWS Fault Injection Simulator first launched, posting a meme in the #aws-memes slack channel posting a drake meme with the first panel being "Using AWS Fault Injection Simulator", and the second "Deploying in us-east-1".

dave4420• 3 years ago

My employer’s sites are hosted via CloudFront and seem to be fine. The underlying buckets are in Dublin, in case that makes any difference.

akhayam2• 3 years ago

AWS Status page doesn't show anything yet: https://health.aws.amazon.com/health/status

NullInvictus• 3 years ago

Short of something within a few hundred feet of Bezo's desk being actually on fire, AWS's health page never seems to actually report issues.

scrum-treats2• 3 years ago

Bezos is no longer the CEO. It's Andy Jassy.

I agree, AWS (and Amazon Alexa + Ring) seem to be skirting and/or choosing active silence on that "Lambda" outage that took out us-east-1. Alexa had at the very least two full scale outages this year, and Ring was successfully hacked as well.

Amazon's response: "looks like a Lambda... more later" then nothing, nothing, and "it must be a 3rd party skill."

AWS health page: GREEN.

AWS Post-event summaries: non-existent after 2021.

This is quite troubling when you consider what AWS holds in its coffers, along with the number of customers AWS currently has. No transparency, not even strategic transparency, is a definite red flag.

pgn674• 3 years ago

Got posted a minute ago.

Operational issue - Amazon CloudFront (Global) Service Amazon CloudFront Severity Informational RSS Elevated Error Rates Jul 18 10:26 AM PDT Between 9:37 AM and 10:13 AM PDT, we experienced elevated error rates for request serviced by the CloudFront Origin Shield and Regional Edge Cache in the US-EAST-1 region. The issue has been resolved and service is operating normally.

ssabev2• 3 years ago

"elevated error rates" is now my favourite euphemism for a downtime

temp0826• 3 years ago

FYI, it takes some pretty high-level approval at AWS for someone to make a change on their "status" page (signaling SLA breaches and money stuff). Not an actual live status page.

klinskyc• 3 years ago

It looks like they are back up now

akhayam2• 3 years ago

Lots of users seem to be impacted for a short time period: https://isdown.app/integrations/aws/amazon-cloudfront

sgammon• 3 years ago

Seems stable now

Tell HN: CloudFront Is Having an Outage

Related Stories