Cloudwatch just stopped triggering our Lambdas

https://twitter.com/bugaralife/status/1331604999802122240

by artembugara • 6 years ago

47 27 6 years ago

twitter.com

jonathannorris• 6 years ago

This seems like a larger system outage, we are seeing issues with Kinesis and Cloudwatch.

athorax• 6 years ago

Cloudwatch, cloudwatch events, and kinesis events in us-east-1 have been down for about 2 hours now

dageshi3• 6 years ago

Had Service Unavailable for about 10 mins on Cloudwatch events. Just corrected itself for me at least. us-east-1

tootie2• 6 years ago

AWS status board is reporting a major outage in Kinesis for us-east-1. Causing big problems for us right now.

edoceo• 6 years ago

For folks who go in on managed services because self-hosted has the danger of an outage - how does this factor? I self-host lots of my company services and frequently catch heck from my peers for it: if your "x" goes down it costs more than the managed. (It's more than just cost tho)

Seems like folk factor managed as if it was zero-downtime game and that self-hosted is an huge burden.

Maybe these monitors aren't critical but what about when it is?

dageshi3• 6 years ago

In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.

abraxas• 6 years ago

In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.

briffle• 6 years ago

There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done.

If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.

NullPrefix• 6 years ago

There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.

I feel like this statement perfectly embodies the problems that people face when choosing a closed source OS vs Linux/BSD. When stuff doesn't work on Windows there's pretty much nothing you can do so you have to just accept it and pretend that you wanted it to not work this way. As opposed to Linux where you can make it work your way, but it costs a lot of time to do that.

staticassertion• 6 years ago

a) Don't deploy prod to us-east-1

b) Don't stick to a single region/ AZ. Cross-region/AZ failures are quite rare

c) If AWS goes down a huge portion of the internet is fucked. Customer expectations are totally different. When 90% of the internet isn't loading images no one's coming to you to complain.

d) The fire is elsewhere. Someone else is on it. At worst we spend some time trying to work around it, at best we work on other things while we wait.

markbnj• 6 years ago

> For folks who go in on managed services because self-hosted has the danger of an outage

Seems like an overly simplistic and narrow view of the tradeoffs. The mainstream cloud services providers operate at very high levels of availability, but they nonetheless still have the "danger of an outage." There are many reasons why managed infrastructure may make sense for an organization, from ease of scaling to the completeness of the available solution space and cost efficiencies, but "absolutely avoiding any chance of downtime" isn't one of them.

PHGamer• 6 years ago

if big companies go down. the ops guy can blame it on the hosting provider esepcially if its amazon. no place can guarantee 100% uptime but if you pick some joe crabs hosting you will be blamed but if you pick amazon or azure the company will be like oh well shit happens.

mandelbrotwurst• 6 years ago

If I'm using a managed service, it's more that I am expecting to get better reliability and less work on our part than it is expecting perfect reliability and no work on our part, both of which are a pipe dream no matter which route you go.

Zanfa• 6 years ago

From their own status page:

"We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD."

hyperdimension• 6 years ago

"It's not posted on SHD as the issue has impacted our ability to post there."

At the risk of stating the obvious...why host your status page of a service on something that depends on that service?!

smt882• 6 years ago

It has been disappointing to see how many people on HN have business-critical production deployments in a single AZ. I thought we learned not to trust any one AWS location years ago when Netflix learned for us.

risaacs99• 6 years ago

This is a regional outage, not a single AZ.

smt882• 6 years ago

You're right, I misspoke. I meant across regions.

All the abstractions AWS offers (like RDS) make it easy to replicate across regions, but of course not everyone uses those.

tootie2• 6 years ago

I recently joined an org that has all their eggs in the us-east-1 basket. I was warned as much by the devops lead that it was at least a partially conscious decision that the cost of multi-region would outweigh the risk. I think I'm going to start questioning that point of view.

jlmorton• 6 years ago

This is affecting all of us-east-1, not just a single availability zone.

Over the past few years, it's become easier to host an application in multiple regions, but it's still a heavy lift, requiring a lot of thought not just from site reliability engineers, but application architecture.

Nexeo• 6 years ago

Is anyone else getting "Capacity unavailable" when trying to add tasks in Fargate?

Cloudwatch just stopped triggering our Lambdas

Related Stories