Today is when the Amazon brain drain sent AWS down the spout

https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/

by raw_anon_1111 • 9 months ago

1018 649 9 months ago

theregister.com

jqpabc123• 9 months ago

Nothing gets sold or fixed without people who know how it's built.

nine_zeros• 9 months ago

But it is more important to keep the PIP, stack-ranking, ladder-climbing game running than keeping the people /s

bwfan123• 9 months ago

cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2].

[1] https://pages.cs.wisc.edu/~remzi/Naur.pdf

[2] https://x.com/elonmusk/status/1980221072512635117

intermerda• 9 months ago

Management learns this the hard way [2]

I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?

ChrisArchitect• 9 months ago

More discussion: https://news.ycombinator.com/item?id=45640838

ortusdux• 9 months ago

"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums.

The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”

- Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview

ASalazarMX• 9 months ago

Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.

JustExAWS10• 9 months ago

It was a nothingburger. Apple sold the same GSM iPhone 4 for three years with no design changes and nothing else was said about it three months later.

Mistletoe2• 9 months ago

I’d add a corollary to Steve’s often quoted idea, that became even more relevant after Covid. Everyone competent that makes tons of money retired early. We are left with the dregs at all these companies, the newbies and those that didn’t budget well and plan for early retirement.

Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.

nebula88042• 9 months ago

It was going to happen Covid or not. People get old...what are you going to do? Is it really that bad if a bunch decide, "you know what? I'll take the financial hit and retire a few years early because I have been reminded of the fragility of life".

rkomorn• 9 months ago

Maybe, in retrospect, all those brilliant folks who made all that money would've deserved it more if they'd built cultures that could survive 5 years.

deepsquirrelnet• 9 months ago

FWIW this has happens in consulting too, not just product companies. Just swap “product” for “delivery”.

Gormo7• 9 months ago

Is this actually a verbatim quote? The typo at the end makes that seem unlikely.

behnamoh2• 9 months ago

It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues. Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns.

So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.

Worthless article.

samrus• 9 months ago

Just because its speculation doesnt mean its worthless. But yeah it should be taken as speculation rather than a validated and tested hypithesis

behnamoh2• 9 months ago

Anything that wastes my time and only reveals its half-assed reasoning half way through the article is indeed worthless.

JustExAWS10• 9 months ago

I worked at AWS and still have friends who work there. I don’t know any L5s who wouldn’t jump at a chance to leave if they even got a slightly worse offer than what they are making now. I know a few L6s and L7s that would stick around out of momentum.

But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.

I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.

I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.

When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.

I took the latter and had three offers within 3 weeks. This was late 2023.

azemetre• 9 months ago

How long did you stick it out? Were you close toward completing your plan?

dijit6• 9 months ago

Maybe its speculation, but I mean drawing conclusions becomes easy when 40% of devops staff being laid off by AWS was in the news three days ago.

https://blog.stackademic.com/aws-just-fired-40-of-its-devops...

placardloop5• 9 months ago

AWS doesn’t even have a “devops team” nor even any devops job roles. AWS also does not use Terraform (which is what the article says everyone was replaced with) at any significant scale, so this article is similar junk.

jongjong2• 9 months ago

Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS.

It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.

The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.

madaxe_again• 9 months ago

To bake an apple pie from scratch, you must first overthrow capitalism.

chicagobuss2• 9 months ago

perfection

jongjong2• 9 months ago

I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system.

All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.

This kind of thinking seems to permeate most other parts of society... It's gotta stop.

"Oh but what if someone steals it"

This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.

"Oh but we don't want people to own domains permanently or else they will take all the good domains"

Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.

I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.

tpmoney2• 9 months ago

People could host websites on the Blockchain from beyond the grave...

This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.

tpmoney2• 9 months ago

People could host websites on the Blockchain from beyond the grave...

yreg• 9 months ago

Also the potential for a domain to get irreversibly stolen is not a good feature for the security of the users.

tombert2• 9 months ago

I think this would make the squatting problem that we already have way worse. There would be bots buying every single remotely usable domain, and there would be no incentive for them to sell it unless they get an absurdly large offer.

I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.

Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.

[1] https://web.archive.org/web/20160219161720/http://www.hugedo...

grugagag3• 9 months ago

Yes, having some cost disincentivies some abuses that completely free attracts. If email cost a few cents to send there would probably be a lot less spam around

bombcar3• 9 months ago

Let’s solve rent-seeking by building a new system that’s even more rent-seeking!

At least currently death dissolves bonds.

pluto_modadic• 9 months ago

DNS is:

- simple

- battle hardened

- distributed

- affordable

blockchains are:

- essoteric, backwards, and not easily implemented

- new and unproven, frequently hacked

- effectively a ploy to centralize / redo Web 1.0 but owned by one blockchain

- ...waaaaaaay more about money and "owning something" than DNS is.

chicagobuss2• 9 months ago

internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics"

https://forums.theregister.com/forum/all/2025/10/20/aws_outa...

int3trap• 9 months ago

There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call.

RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.

mschuster917• 9 months ago

So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job"

Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.

foobiekr3• 9 months ago

The problem is that if it was 8 in 10 suggestions or even 8 in 20, then yes, terrible. On the other hand, if it was 8 needles in a haystack of garbage, at some point you do, in fact, need to tell people to STFU so you can work.

pdonis• 9 months ago

One thing I love about El Reg is that they never shirk from calling a spade a spade.

freedomben• 9 months ago

I also love the humor and personality that authors are able to inject into their pieces

mrec• 9 months ago

I miss Lewis Page. It's a rare day when I'm so downhearted that the memory of https://www.theregister.com/2008/12/30/german_beaver/ can't raise at least a wan smile.

furyofantares• 9 months ago

They never shirk from calling anything a spade.

yen223• 9 months ago

I nearly missed that this article was written by Corey Quinn, guy who writes a lot about AWS

add-sub-mul-div3• 9 months ago

Amazon has reportedly been a shitty place to work forever, so using issues that happen to be popular today to explain turnover is disingenuous.

sc68cal2• 9 months ago

From the article.

At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia

Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.

bombcar3• 9 months ago

Arguably we had a perfect example with Twitter.

mlhpdx• 9 months ago

This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so.

Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.

Suspending disbelief won’t make downsizing work better.

add-sub-mul-div3• 9 months ago

I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.

RobRivera2• 9 months ago

I observed a similar thing in the post Goldwater active duty officer corps reformations.

bcrosby95• 9 months ago

Nah, the decades old crop of "new" big tech companies are just entering their IBM phase.

goldchainposse• 9 months ago

Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.

CobrastanJorji• 9 months ago

Seems like it worked fine. They laid off a quarter of their junior principal engineers, the stock went up. They had a massive outage a few months later, the stock went up again. Everything's working out fine for their strategy so far.

few• 9 months ago

I remember comments saying the stock went up because the average joe didn't realize how much of the internet was powered by AWS until all their day to day apps started failing. To most people Amazon is an online shopping site.

array_key_first• 9 months ago

It continues to work until eventually the debt is so high the company implodes.

See: general electric, RCA, Xerox, GM

foobarian• 9 months ago

You would think this would eventually show up on the balance sheets, right? Presumably a lot of their big customers have SLAs with money penalties, so maybe next quarter earnings? Or quarter after that?

b00ty4breakfast• 9 months ago

The entire thing reminds me of Wile E Coyote suspended in mid-air before he looks down and plummets to the bottom of a ravine

croes• 9 months ago

Yes, development tools are better every day

Are they?

nijave4• 9 months ago

It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.

edoceo• 9 months ago

I thought the recovery was early AM Seattle time (like 4am). Where I think start-of-day is like 9am. Maybe recovery started early (6am) New York time?

nijave4• 9 months ago

[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.

[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...

[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...

[12:11 AM PDT] <declared outage>

They claim not to have known the root cause for ~8hr

technofiend2• 9 months ago

huh.. maybe publicly communicated recovery was then. I was seeing knock-on effects hours later and didn't see full recovery until late afternoon EST.

rr808• 9 months ago

I noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.

NKosmatos• 9 months ago

This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online.

P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)

add-sub-mul-div3• 9 months ago

It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.

christhecaribou3• 9 months ago

There’s clear data that shows an increase in the frequency and severity of LSEs post RTO3 in 2023, and it looks like RTO5 has accelerated that trend.

orbital-decay• 9 months ago

The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.

christhecaribou3• 9 months ago

“Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.

Dig1t• 9 months ago

Amazon remained the single largest H-1B sponsor, increasing approvals from 9,257 in 2024 to 10,044 in 2025, an addition of 787 visas.

https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m...

I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India.

Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.

browningstreet• 9 months ago

It depends on how the resources are assigned, what projects they’re asked to focus on, and which strategic tech debt initiatives aren’t approved.

hluska2• 9 months ago

We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss.

However, talent is a very small part of shipping a project. How that talent is resourced is far more important.

dafelst• 9 months ago

Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?

grugagag3• 9 months ago

ouch, that’s what a Diwali outage looks like..

citizenpaul2• 9 months ago

Its almost like institutional knowledge is a real thing that you cannot put on some BIG BRAIN MBA spreadsheet.

Groxx• 9 months ago

how many lines of code is it though? or how many tokens? we need something we can measure if we're gonna fire people!

immibis• 9 months ago

g-b-r• 9 months ago

For those who worked there recently, how much does the comment at [1] reflect the current state of things?

[1] https://forums.theregister.com/forum/all/2025/10/20/aws_outa...

bithead• 9 months ago

What - their AI couldn't find it sooner? Better get those RAGs in order.

charcircuit• 9 months ago

This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess.

I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.

That information is under NDA, so it's only natural you aren't privy to it.

Esophagus4• 9 months ago

I’ve seen this happen with startups as well -

They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat.

The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.

le-mark2• 9 months ago

I once saw a layoff that was followed by a week long outage because no one remaining knew how to deploy to prod, and no one knew how to recover after the failed deployment. I felt bad for the people remaining who had to go through that,but it was hilarious.

pinkmuffinere• 9 months ago

one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow

Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.

estimator7292• 9 months ago

75 minutes is damn good turnaround for any major problem, IMO

bberrry• 9 months ago

75 minutes to diagnose what's failing is not.

__loam2• 9 months ago

Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue

Freedom2• 9 months ago

What if the SRE talent has a lot of real-life experience but can't pass LeetCode puzzles that have nothing to do with the job?

raverbashing2• 9 months ago

The sre talent moved elsewhere once the rto bs started

Eridrus• 9 months ago

I dunno man, what part of the AWS experience leaves you thinking the software is amazing?

It's good enough, but there's no real evidence it's the best, simply the largest.

ecnahc515• 9 months ago

Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.

michaelt• 9 months ago

Is 75 minutes really considered that long of a time? [...] When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.

One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.

But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.

Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.

Of course, I don't know how common well designed, properly resourced business critical systems are.

jofzar• 9 months ago

If you a regular company, no. If you are the biggest provider in the world and your change is breaking large parts of the Internet, yes.

jmaestrooper• 9 months ago

Is 75 minutes really considered that long of a time?

From my experience in setting up and running support services, not really. It's actually pretty darn quick.

First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.

Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.

Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.

I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.

Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.

But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...

snowwrestler• 9 months ago

Sorry, are you saying you worked at Amazon and this is how they handle major outages? Just snooze and wait for a ticket to make its way up from end user support? No monitoring? No global time zone coverage?

Because if so, this seems like about the most damning thing I could learn from this incident.

shepherdjerred• 9 months ago

AWS operates completely than what you're describing.

Alerts and monitoring will results in automatic pages to engineers. There is no human support before it gets escalated.

If an engineer hasn't taken a look within a few minutes, it escalates to their manager, and so on.

lljk_kennedy3• 9 months ago

Wholly inaccurate. AWS Systems Engineers would have been paged by automated monitoring systems once alert thresholds were breached. No escalation through Support needed.

Cyberdogs73• 9 months ago

For a service like AWS, 75 mins is going to result in a LOT of COE's for people on way it wasn't mitigated quicker. A Sev 1 like this has an SLA of 20 mins to mitigate impact. Writing about these failures will consume a dozen peoples time for the next 6 weeks.

I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.

rags2riches• 9 months ago

LOT

Just capitalised for emphasis, right?

COE

Center of Excellence? Council of Europe? Still wondering even after Googling.

SLA

Service Level Agreement. This I knew beforehand.

SDM

Service Delivery Manager?

Magmalgebra• 9 months ago

Depends on what you're measuring.

Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.

75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.

lljk_kennedy3• 9 months ago

It's 75 minutes to _communicate_ the message to customers. Definitely internal teams were ahead of this before it was posted to the AWS Health Dashboard. Status Page posts are lagging indicators of incident progress.

jeffrallen• 9 months ago

I work in an incident management team where the turnaround from "we've decided to take x action, to y metric shows it is working, to z is posted on the status page" can be 1-2 minutes.

It is possible with professionals, institutional knowledge, drills, and good tools.

Merad• 9 months ago

The web operates in a very different world if you've invested in good tooling. I used to be lead on a modestly sized payment processing back end to the tune of about 100 transactions/second (we were essentially Stripe for the client facing apps at the company). In many cases our monitoring and telemetry let us identify root cause in a matter of minutes. Not saying that is or should be the norm for all web apps, but what we had was not too far off from a read-only debugger view of the back end app's state throughout the request and it was very powerful. Of course for us more often than not the root cause was "the bank we depend on is having a problem" so our knowledge couldn't do much other than help the company shape customer communications about the incident.

rkozik1989• 9 months ago

A few years back I was working at a software company that provided on-site sensor sensor networks to hospitals, pharmacies, etc. Our product required them to physically install a server on-site, but we were starting to get disrupted by cloud-based solutions. Essentially what we did was alert medical staff when blood, organs, etc. refrigeration temperatures went out of range. If the right people involved did not get notifications on time for these issues people will die. Its not hyperbole, you have to wait years for liver transplant. Their aren't just new livers available for everyone if a handful of them spoil.

With that being said, the problem here isn't that it took 75 minutes to find the root cause, but rather that the fix took hours to propagate through the us-east-1 data center network. Which is completely unacceptable for industries like healthcare where even small disruptions are a matter of life and death.

causal• 9 months ago

Yeah. They will identify the cause, but not the cause behind the cause.

president_zippy23• 9 months ago

Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there.

Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.

It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.

https://www.youtube.com/watch?v=fFihTRIxCkg

steve-atx-7600• 9 months ago

Seriously. I don’t know any half way decent engineer that would ever work there twice.

president_zippy23• 9 months ago

I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week.

I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.

Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.

I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.

That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.

To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.

stouset• 9 months ago

After having heard horror stories from friends who've worked there, you couldn't pay me enough.

wppick• 9 months ago

The formula is usually more money and ability to work special team isolated from the usual toxic orgs. I think A9 was probably somewhat like that, and AWS probably used to be at some point long ago

VirusNewbie5• 9 months ago

I know some people who are fine working there. No one seems thrilled but if you're an above average engineer who is just getting by at 140k a year and suddenly you're looking at 350k a year as an SDEIII or something, that can be a life changing amount of money.

However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.

However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.

wkat42424• 9 months ago

We do a lot of business with them and we have workshops with them sometimes, and the one thing I notice is how they're all so evangelical. They wouldn't say a bad thing about their company. I couldn't be like that, when I work something I sell my knowledge but not my soul. I'd always speak freely (not always appreciated but usually it's not a problem if it's true).

But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.

Maybe it depends on the country but it feels like this is just their culture.

swaits2• 9 months ago

Hi, I am a half decent engineer. I say that as objectively as one can say something like this about themselves.

I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.

I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.

And the culture? It was life changing for me, especially when I first joined. In all the best ways.

And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.

They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.

The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.

ajkjk• 9 months ago

I had a decent time there a decade ago. In retail, though, not AWS.

Razengan2• 9 months ago

Well don't you want to be able to say "I work on the DEATH STAR" when asked what you do? Think of all the Twi'lek headtail that pulls

awill2• 9 months ago

I've worked there since 2015, and this simply isn't true.

There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).

jesse_dot_id2• 9 months ago

I'm an awesome engineer and I'd never even apply to a FAANG company. I value stability over pretty much all else.

BoredPositron• 9 months ago

Yep, everyone is excited until they leave after 1 1/2 or 2 years. There are always outliers but my personal experience is that the churn rate is incredible high.

tjwebbnorfolk2• 9 months ago

I'm barely a halfway-decent engineer, working at a M7 big-tech co, and my next job requirements consist of: anywhere but Amazon.

The stories I hear there is just not the style of work I'm interested in

lo_zamoyski2• 9 months ago

I'm sure the intersection between sadomasochists and Amazon employees is above average.

nradov• 9 months ago

Are there really many illegal aliens working in the warehouses? I know that Amazon does verify employment eligibility and checks documents. There may be some committing identity theft but I doubt that it's a large proportion.

ahi2• 9 months ago

I assume the parent was being hyperbolic. Illegal immigrants is the barrel scraping bottom of the work force.

terminalshort3• 9 months ago

Not just firings. I sure got a deluge of Amazon recruiter emails right after they went full RTO.

saghm• 9 months ago

I still occasionally get them even though I literally was one of the people who left after they tried to make us go in office (I don't like to use "RTO" because no one on my team had actually worked out of an office for Amazon before since the project we were on was fairly new). My wife (at the time fiancee) has an autoimmune issue that makes it much safer for me not to commute, and although my manager suggested I could get an exemption, he didn't actually know what the process was because it all happened so quickly that no one seemed to have actually defined what that process was up front. I had a little less than a month to figure what to do and get that exemption before they expected me to either have that exemption, be in an office in another city three days a week, or transfer to a local team and be in the office in my current city three days a week. I ended to deciding that it wasn't worth the effort to try to figure out how to convince them to let me stay.

sumanthvepa• 9 months ago

There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.

I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.

I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.

Eric_WVGG• 9 months ago

I believe his point is that, even if you expand the available pool of developers outside of the US, that pool is still exhaustible.

president_zippy23• 9 months ago

I never implied that, let alone said it.

The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.

If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.

If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.

That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.

hopelite2• 9 months ago

I think you’re not considering the other side of that perspective. I am sure you are very happy for your fortune to have been plucked out of India and been given the opportunity to work at Amazon and presumably live in America, which put you in the place you described that seems to be in a really good position today. The issue is that not only was the H1-B meant for highly specialized people that cannot be found in the USA, it has very long been absolutely abused by American corporations and politicians that have been betraying their own people for several decades now by engaging and ignoring this abuse that was really just about undermining salaries of Americans by giving the opportunity to you rather than Americans, while it was really mostly about enriching the rich. You were essentially just a method for the rich to get richer.

I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.

I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.

It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.

What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.

I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.

phatfish• 9 months ago

But why not be a superstar engineer in India to start with? Why the round trip to another country to disrupt their society?

RobRivera2• 9 months ago

There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.

I fail to reach this interpretation in this thread.

cudgy4• 9 months ago

Well at least you’re a humble outcoder.

rirze• 9 months ago

circa mid-nineties

I'm interpreting this as mid-1990s, in which case I very much believe in your technical ability. My dad came over late-1990s and he worked at mid-sized companies ever since. Even then, he and his H1B peers were decently intelligent.

I would caution your defense of today's H1B/L1s/OPT workers; I'd say the quality of Indian engineers in the US has halved every 10 years.

Today's Indian engineers come to the US because they can't enroll in a decent college in India and/or obtain a upper-middle class salary from a job. It is an entirely different mechanism for which people are migrating over. It used to be brain drain, now it is sewage drain.

The H1Bs in the big tech companies are maybe 50/50 technically decent, but everywhere else, they are just taking contracting spots. It is a very corrupt and bloated system that has to go because they are not providing valuable work.

rester3242• 9 months ago

It seems amazon itself is aware of this issue. The linked engadget article even mentions this:

"The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."

Yokolos2• 9 months ago

That's wild. Does any other company have this problem?

palmotea• 9 months ago

It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."

This article? https://www.engadget.com/amazon-attrition-leadership-ctsmd-2...

It's from 2022, so it'd be interesting to see an update.

citizenpaul2• 9 months ago

“deplete the available labor supply in the US” in certain metro regions within a few years.

AWS Manager interpretation. "Few years?, not my problem"

So I'm guessing things will get worse. It took a long time but I remember when "cloud" started getting big lots of people voiced concern about being at the whim of Amazon/Bezos for your business critical infrastructure. Took longer than most people though but we are getting there.

Edit: I see its from 2022, so maybe it is the end stage?

beAbU• 9 months ago

tens of millions of illegal immigrant warehouse workers

You have a source for that claim?

typpilol• 9 months ago

I think you misunderstood. He's not saying they have 10m illegal workers.

He's saying even if they had 10m illegal workers they would burn though them all too

pavlov• 9 months ago

The claim is not that Amazon would be using illegal warehouse workers today, but that there is theoretically a pool of tens of millions of people available. Which is still kind of dubious.

casenmgreen• 9 months ago

Yes.

I would never work for AWS, given what I've heard, and consistently, of their internal culture.

Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.

grafmax• 9 months ago

Given their self-imposed labor supply issues, notably the awful working conditions, I hope the workforce there figures out how to effectively organize against this cesspool of a corporation.

ChrisMarshallNY9• 9 months ago

They pay very well, for some roles.

I have a friend who recently started there. He just brought a Mercedes, and a second house, and he’s still in his early thirties.

They keep him busy, though.

JCM94• 9 months ago

Amazon’s not the only company on the planet that pays well. While they’re above average they’re far from top of the market. If you’re talented enough one can make a lot of money and skip the toxic culture. Double win.

coredog642• 9 months ago

Come back after your friend hits the 4 year cliff. OY2 is a killer.

JCM94• 9 months ago

From what folks at AWS tell me that’s basically already happened. The best and brightest won’t even apply to work there anymore. For many key functions they’ve legit run out of people to recruit and thus have to go down market compared to competitors. Thats very much true in hot sectors like AI where AWS has “C” and at best “B” team players and leadership.

derangedHorse• 9 months ago

I’ve also heard key pieces of their infrastructure has sloppily written code and that communication between teams is horrible. Even with their insane salary offers, most people don’t think it’s worth it. Especially given their 6% “unregretted attrition”[1]

[1] https://fortune.com/2024/03/20/amazon-layoffs-performance-re...

iamleppert• 9 months ago

It's only a matter of time before ICE starts cracking down on Amazon Fulfillment Centers...

agoodusername63• 9 months ago

There was another comment that pointed out how unlikely this is to happen because Amazon is just too big to bring the law down on now.

When you're the, what. Second? Third? Largest employer in the US, enforcing the law now becomes a meaningful hit to economic velocity. And as much as Trump hates brown people, his administration has begrudgingly revealed that there are moves that his billionaire buddies Will Not Allow.

I'm no fan of ice or this administrations deportation strategy, but it's a serious problem that even enforcing the law on Amazon is now an economic liability so much that nobody dares to try

typeofhuman• 9 months ago

Not totally on topic, but I recently passed the tipping point with Amazon shopping. I now go to Temu. They have US warehouses shipping in a couple days which was the only thing keeping me on Amazon. Plus everything on Amazon is basically the same stuff on Temu but with a markup!

NoMoreNicksLeft5• 9 months ago

I seem to remember an internal memo being leaked in which middle management was complaining that they would eventually/soon burn through all the laborers in that region, after which they'd suffer immense difficulty in staffing that warehouse.

alexpotato• 9 months ago

There was a tweet the other day about Walmart.

They raised worker pay which, unsurprisingly, did not make Wall Street happy in the short term.

However, over the next couple years there were multiple benefits:

- lower turnover

- less employee theft

- cleaner stores

- more same store sales

etc

SilverElfin• 9 months ago

That doesn’t seem convincing as far as evidence. Is there something by more formal showing raising pay led to those specific outcomes?

garblegarble• 9 months ago

I went looking for a source on this, looks to be: https://www.wsj.com/business/retail/walmart-employee-treatme... / https://archive.is/fPAlB

foobiekr3• 9 months ago

Every single engineer I know who went to Amazon except one lasted under 3 years and to this day, often ten+ years later, they all will mention how much they hated it.

The one exception is an engineer who stopped engineering, switched into product, and transferred to China to hit on the women there.

Some Amazon practices actually sound great to me (short documents, read before the meeting) but so many things just sound needlessly, relentlessly cheap.

michaelcampbell• 9 months ago

I know of 1 tech person at Amazon that claims to have liked it there; the husband of a co-worker (albeit 6 years ago). He was some in-house consultant type role though and few all over the world to help the internal teams straighten out whatever AWS mess they'd gotten into, so that's not quite the role that people think about when talking in a FAANG context.

y-c-o-m-b• 9 months ago

I was on one of the core AWS teams. I lasted 3 years and 1 month, to your point hah. I left about a year ago. My stress levels were through the roof during the time I was there. It truly was one of the most toxic stressful places I've ever worked, second only to Intel.

The largest contributor of stress being on-call rotations where getting paged between 12am-6am each night was basically a guarantee. God help you if it was a holiday and you got a high sev page, where the people that you really need are all out of pocket. The many many many instances of their security "regime" relentlessly paging us in the middle of the night for things like having an S3 bucket for static website assets; despite numerous exceptions given by L7+ leadership.

I disagree with the notion around "short documents", not only were they quite lengthy at times, but they actually made the process of "busywork" worse by adding more overhead to trivial matters.

Add on the layoffs and "return to office" horse-shit excuses and it's no wonder nobody wants to go back.

dls2016• 9 months ago

I got a job at AWS/EFS from a post here on hacker news. Stayed there almost 2 years until RTO took its toll (left early 2024). If not for that, I'd still be there... and I went in with full knowledge of all the horror stories. Perhaps the EFS org was just a diamond in the rough, but it was honestly one of the best jobs I've had. Even the on call wasn't so bad, with management taking an extremely hands-on and proactive approach to reducing operational burden. Extremely high technical bar which taught me a ton about building and operating large distributed systems. I do wonder if EFS is still run so well.

I've since been at Oracle/OCI (absolute dog shit with the worst on call I've ever seen, and I've been in the military lol), and now at Microsoft/Azure, which so far seems like a decent workplace.

gtirloni• 9 months ago

tens of millions of illegal immigrant warehouse workers

Sorry, what?

HardCodedBias• 9 months ago

Amazon culture is entirely powered by recruiting.

Even L6 managers feel this, but it becomes more clear as one goes up in levels. Recruiting is job one.

If Amazon runs out of recruitable engineers (unlikely, they are one of the most prestigious firms in the world) then they will simply lower the bar. HC must be filled.

The company is well structured, it will survive.

mmonaghan• 9 months ago

Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve.

I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.

christhecaribou3• 9 months ago

The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat.

This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.

mrguyorama• 9 months ago

Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling.

We had business as usual today.

anigbrowl• 9 months ago

One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.

the_real_cher• 9 months ago

AWS has been having issues like this for years.

benjaminclauss• 9 months ago

big "who is John Galt" vibes in these comments lol

sporkland• 9 months ago

Garbage reporting: 1. AWS had an outage 2. AWS has lost a lot of employees

Conclusion: The brain drain lead to the outage...

I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.

I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.

homeonthemtn• 9 months ago

There's an argument to be made that each event generates new institutional knowledge for those that are there.

Bit of a double edged sword.

flowerthoughts• 9 months ago

Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.

neilv6• 9 months ago

AWS is still my overall favorite cloud provider, and I use it very effectively.

I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:

1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)

2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?

3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.

4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)

Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.

VirusNewbie5• 9 months ago

I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.

kn0where• 9 months ago

Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.

kmac_• 9 months ago

Whether it involves FAANG companies or not, a job is ultimately just a job. While it's nice to have such a company on a CV and to gain the experience, it is, in essence, similar to any other employment.

Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.

Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.

JustExAWS10• 9 months ago

The question is why is AWS a shining star for you even if all of those issues are addressed? Having AWS on your resume as a developer doesn’t set you apart from hundreds of thousands of other developers who were able to grind enough leetcode to make it in BigTech.

True, it opened a lot of doors for me - some I chose not to go through. But I went into ProServe working in the consulting division, became a major contributor to a popular open source “AWS Solution” in its niche and made industry contacts by working with outside companies.

You don’t have to memorize LPs. You just have to know how to answer standard behavioral questions like I’ve done a million times before and since - 30 years across 10 jobs.

Every tech company that pays top of market requires coding screens and all of the BigTech companies are forcing RTO and they all have a toxic work culture.

It’s par for the course if you want to make the money they offer.

Google is no better if you look at the leveling guidelines. It’s still based on politics and working on the new and shiny - the reason they have so many products that go nowhere and at one point had 5 or 6 messaging apps simultaneously.

I think you are looking at BigTech - any of them - through rose colored glasses

neilv6• 9 months ago

You have good points. I wanted to highlight a couple bits:

Google is no better if [...]

Google has significant differences from Amazon, from everything I've heard. And in some ways a better cultural fit for me.

Two Google problems are the forward-facing frat pledging and egos during hiring, and then misaligned careerism by many on the job, with both problems supported from the top. (Well, and the adtech ethics, with a veneer of "California nice" and rationalizing "we're the good guys, so it's OK to do this"; partly mitigated by the veneer being self-fulfilling to some degree.)

I know less about Amazon, but my impression is that the problems are more like (let's call it) "corporate roughness", and the gradual effects of who survives or thrives in that vs. who leaves. But complicated, because not all teams seem to feel that rough side of corporate culture as much as others. On the good side, the company often does great work, and they have some good earlier ideas about engineering and product culture (but maybe coasting or backsliding in parts).

I haven't seriously considered most of the other big tech companies, and I think many of them have much worse problems.

Having AWS on your resume as a developer doesn’t set you apart from [...]

That's not much of a goal for me. Resume prestige (and also resume-driven-development keywords) are goals for most developers today, though. Too much so. Which I think is a sign of a serious culture problem across our field, and I think it shows in our field's output.

jnaina• 9 months ago

When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts.

When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.

It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.

malfist2• 9 months ago

I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient

donavanm3• 9 months ago

Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic.

Ex a sloppy as hell and inconsistent premise.

engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause

its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”

Or this stupid citation if were talking about senior/long term AWS tech roles:

Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.

The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.

So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?

malfist2• 9 months ago

As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day

donavanm3• 9 months ago

And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently.

Again, Im not saying things are good (I left for reasons!). But use better data and arguments.

aussieguy12342• 9 months ago

The PIP culture of AWS sounds horrifying. As a decent engineer, I would not work there unless that is addressed.

I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world.

There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.

wkat42424• 9 months ago

What's PIP?

frankzinger• 9 months ago

Performance Improvement Plan, a formal process for employees whose performance is not meeting job expectations.

losteric• 9 months ago

Every good Amazon org has sacrificial bad employees. Managers know a team of all-stars creates risky team dynamics under PIP

rester3242• 9 months ago

This sounds toxic on it's own right. What a horrible managament practice

Tsarbomb• 9 months ago

Years ago I interviewed for Amazon. I found out that on a desired team size of 8, there were only 3 left, where the other 5 who were no longer there, as I was told, were poor performers. I didn't take the job.

aussieguy12342• 9 months ago

I guess at AWS, "poor performer" could potentially mean one of the best engineers in the world.

coredog642• 9 months ago

Not to defend PIP, but it is applied at a scale greater than a team, and that below a certain size an org is immune to stack ranking requirements.

neilv6• 9 months ago

This is The Register, a respected journalistic outlet.

Yes, but they bristle at the thought. :)

vehementi• 9 months ago

Mere hours into the incident, before there's any RCA, someone rushes to discredit themselves with a simplistic explanation

crmd2• 9 months ago

I wish to understand the virtue of Amazon culture.

It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.

It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.

I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.

sbt• 9 months ago

It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.

It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.

dash2• 9 months ago

I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.

jatins2• 9 months ago

I think it comes down to demand and supply for jobs.

The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can

woooooo• 9 months ago

It's not as conscious as that, its an emergent outcome of the snake pit.

Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.

valbaca• 9 months ago

It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.

It doesn't need to be sustainable in the long run: just needs to get to the next quarter and there continues to be enough desperate people in the US or India willing to be ground up in the machine for a chance to buy a house in a major metro

(Source: I was at Amazon for 10 years, finally quit last month)

andrewflnr3• 9 months ago

Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.

alpb• 9 months ago

Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.

andrewflnr3• 9 months ago

Organizations like this survive these brain brains.

True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.

dredmorbius2• 9 months ago

Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't."

From TFA.

strunz• 9 months ago

The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.

ed_elliott_asc• 9 months ago

It’s a bit of a guess though isn’t it?

andrewflnr3• 9 months ago

It's about the time between the announcements about finding the cause. I find that to be thin evidence. There are far too many alternate explanations. It's not even that I find the idea to be implausible, but I don't think the article's doom-saying confidence level is warranted.

jjk1663• 9 months ago

Seems logical to assume that could be because there aren't many people left who know how all the systems connect.

It's only logical presupposing a lot of other conditions, each of which is worthy of healthy skepticism. And even then, it's only a hypothesis. You need evidence to go from "this could have contributed to the problem" to "this caused the problem."

Based on what little is given in the article, it seems to go strongly against this hypothesis. For example it links to multiple past findings that Amazon's notification times need improvement going back to 2017. If something has been a problem for nearly a decade, it's hard to imagine it is a result of any recent personnel changes.

TFA does not establish how many AWS workers have left or been laid off, nonetheless how many of those were actually undesirable losses of highly skilled individuals. Even if we take it on faith that a large number of such individuals were lost, it is another bridge further to claim that there was neither redundancy in that skillset which remained, nor that any vacancies have been left unfilled since.

No evidence is given that indicates that if a more experienced team were working on the problem it would have been identified and resolved faster. The article even states something to the opposite effect:

AWS is very, very good at infrastructure. You can tell this is a true statement by the fact that a single one of their 38 regions going down (albeit a very important region!) causes this kind of attention, as opposed to it being "just another Monday outage." At AWS's scale, all of their issues are complex; this isn't going to be a simple issue that someone should have caught, just because they've already hit similar issues years ago and ironed out the kinks in their resilience story.

Indeed, the article doesn't even provide evidence that the response was unreasonably slow. No comparison to similar outages either from AWS in the past, before the hypothecated brain drain, nor from competitors. Note that the author has no idea what the problem actually was, or what AWS had to do to diagnose the issue.

cryptonector2• 9 months ago

Twice I've had to deal with outages where the root cause took a long time to find because there were several distinct root causes interacting in ways that made it difficult or impossible to reproduce the problem in an isolated way, or to even reason about the problem until we started figuring out that there were multiple unrelated root causes. All other outages I've dealt with were the source where experienced engineers and institutional knowledge were sufficient to quickly find the cause and fix it.

Which is to say: it's entirely possible that the inferences drawn by TFA are just wrong. And it's also possible that TFA is wrong but also right to express concern with how Amazon manages talent.

lgregg• 9 months ago

I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.

lazyasciiart• 9 months ago

That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"

whateveracct• 9 months ago

glad a company who did RTO got fucked

hope it only gets worse for them

jadenPete• 9 months ago

This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.

philistine• 9 months ago

I think you missed the point of the article. The layoffs are blamed for the poor response, not the outage itself.

Trisell• 9 months ago

I’m pretty sure at this point I know more about AWS and AWS internals than my account solution architect and I’ve never worked for AWS.

shelled• 9 months ago

It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do.

If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.

raverbashing2• 9 months ago

Still, by the time the failure start, that would have been the beginning of the day in Europe, so I suppose those teams got the page

Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there

1vuio0pswjnm7• 9 months ago

""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. "

I use stored DNS data.^1 The data is collected periodically and stored permanently

I seem to be unaffected by DNS-based outages

I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits

1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy

ccozan• 9 months ago

For super critical entries, we put it in /etc/hosts and also at the domain provider as fixed entries.

The combination never fails.

whatever13• 9 months ago

For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.

vbezhenar• 9 months ago

For internal addressing, why do we use DNS?

Why not? It's standard protocol for name-address mapping.

In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.

Client just performs DNS query before HTTP query or client caches DNS response for some time. It's solved problem and "pull" approach usually preferable to "push" approach. For example, what if that broadcast gets lost (UDP is not reliable protocol)? How application handles initial discovery?

Of course it is possible to replace DNS with something else. But why? If you're afraid that your DNS server will fail, the same could happen with your custom service discovery server.

thaack• 9 months ago

That's a lot of broadcast traffic at scale.

How would you handle the ack of the broadcast? UDP can be unreliable. If you do send an ack from the destination, that singular endpoint that sent the message is going to get a number of responses equal to the number of devices on the network, which would nuke it at large scale.

fnordpiglet• 9 months ago

With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.

donavanm3• 9 months ago

Dont write cbells hagiography yet. I wouldnt put the gift of fear, “no lone zone”, and rolling stone up there as top tier cultural engineering successes. They all have reasonable reasons, bits of truth, and i liked cbell overall… but the tech culture was not so consistently great.

whatever13• 9 months ago

Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over.

You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.

At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.

AbstractH24• 9 months ago

I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be.

If AI is to tech what outsourcing was to manufacturing, then your analogy has me concerned for the future.

whatever13• 9 months ago

Good point. They can start offering 95% availability for services, initially for a better price. Then just bring the market expectation to 95% availability and raise prices.

SolarNet• 9 months ago

I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be.

I mean this is true, but we aren't talking about the consumer here. We're talking about the industry which is to say the powerful people who own and run all these companies.

What has happened is that those overseas countries now have all the experienced engineers over there and they know it. So you see things like the Trump admin begging Korean companies to keep their workers in the US because they understand how to actually do these things. And the reason the Trump admin did that is because they owe favors to the rich people who want to profit off of factories in the US.

random9749832• 9 months ago

https://geohot.github.io/blog/jekyll/update/2025/09/13/get-o...

zer0tonin2• 9 months ago

I am waiting for the author to lead the way.

1dom• 9 months ago

We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop.

By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.

The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!

liampulles2• 9 months ago

This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity.

As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.

I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.

rhetocj232• 9 months ago

Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them

liampulles2• 9 months ago

Absolutely.

ChrisMarshallNY9• 9 months ago

When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle.

…

This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.

Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like someone restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.

It’s nice to know that Apple (or some other middleman) backs up the torrents of spam that I get.

Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.

One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.

igleria• 9 months ago

Dunno about brain drain but some departments seem to be having a mandate of "must have AI" when procuring products. Pump and keep pumping!

ferguess_k3• 9 months ago

IMO the "must have AI" mandate exposes the culture of:

- Get things done as fast as possible, regardless of even short-term consideration;

- Implicitly but loudly discourage people to figure out the issues or write code all by themselves, because it is much slower;

throw-10-13• 9 months ago

aws is s globally centralized point of failure, it should not be allowed to exist

tcgv• 9 months ago

According to the article, the issue was caused by:

"engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause"

Interestingly, we found matching errors in our own logs:

System.Net.WebException
The remote name could not be resolved: 'dynamodb.us-east-1.amazonaws.com'

Occurrences were recorded on:

- 2025-04-30

- 2025-05-29

- 2025-06-17

- Yesterday

We had logged this as a low-priority bug since previous incidents only affected our AWS testing environments (and never our production env which is on Azure). At the time, we assumed it was some CI/CD glitch.

It now seems that the underlying cause was this DNS issue all along, and only yesterday did it start impacting systems outside of AWS.

znpy3• 9 months ago

You just made me realize we had random DNS failures using ElastiCache last weeks... Totally randomly, some elasticache endpoints would fail to resolve within our VPC, bringing down some of our services.

At this point I wonder if it's related.

WesolyKubeczek• 9 months ago

We don't care. We don't have to. We are the cloud company.

FrustratedMonky4• 9 months ago

Out of touch. Is Amazon going through some turmoil? Why are people leaving?

I mean in software. I Know warehouses are pretty bad.

ewuhic• 9 months ago

Brain drain implies they went somewhere else where's better. Where did they go?

JCM94• 9 months ago

It’s no secret that AWS has been seeing a mass talent exodus. Probably 90% of the folks I know that were the best and brightest in what they do and worked at AWS are no longer there. It’s beyond a blip, but a full blown exodus, with the talent bar severely lowered.

The writing was on the wall for a bit now that something like yesterday would happen.

roncesvalles• 9 months ago

Largely because Amazon tarnished its reputation among tech workers with its toxic work culture, to the point where it had to introduce a new Leadership Principle called "Earth's Best Employer" as damage control. Meta is on the same track.

pjjpo• 9 months ago

I think it's important Amazon remains stable and a quicker resolution would have been great.

That being said, if many important services (the article mentions banking) are still single-point-of-failure in us-east-1, the least stable but cheapest region, there seems to be a problem far greater than Amazon here.

sporkit150• 9 months ago

I’m ex Amazon. The company promotes, hires, and fires based on everything _except_ merit. I saw many projects fail due to under qualified teammates and leadership. Amazon is an incredible company but it was only a matter of time till its activism caught up with itself.

oaiey• 9 months ago

I am reading the book Children of Time where descendents of mankind try to keep their tech running long after the ship started.

We are now coming into an age in which standing application and infrastructure systems have to run long past their original creators are on the ship.

In my opinion, as a industry we are not mature enough for that and we need to become better.

ferguess_k3• 9 months ago

I have to quote one of the comments:

"Hopefully today will serve as a massive wake-up call for AWS"

I wouldn't hold your breath. There will be incident reviews, meetings, assessments, analysis etc. but basically boil down to what can we do to stop this from happening again without actually spending any more money. So no, not hiring fresh talent or retaining that talent already in play, no to radical overhaul of process and knowledge. No to remediation of known issues if it involves expenditure. Instead it will be do more with less. Beat the employees harder, enforce more and more diligence and output from less and less people for the same or less money. Spin it like mad with catchy titles like knowledge sharing, centers of excellence, efficiency improvement initiatives, agile resilience, and continuous operational excellence.

There’ll be shiny PowerPoint decks about empowering ownership and shifting left, while the remaining engineers are shifting caffeine straight into their bloodstream at 3 a.m.

Next quarter, they’ll unveil a bold new policy called Focus Fridays which will be promptly filled with mandatory incident retrospectives. Someone will suggest replacing ancient tooling, only to be told, “We’ll revisit that next fiscal year,” which is code for never.

Then come the internal awards: “Unsung Hero of the Outage” goes to the one poor sod who rebooted the wrong thing but accidentally fixed it.

HR will roll out a “Resilience Recognition” badge on the intranet. This will be marketed with great fanfare and excitement, showcasing how the company truly values it's employees and recognized their contribution because badges are cheap. Leadership will congratulate themselves for “learning from adversity,” and by the time the next blackout happens, they’ll have a snazzy new dashboard to watch it fail in real time along side their investment portfolio dashboard that takes up a greater fraction of their attention.

But don’t worry!!!! There’ll be a T-shirt. “I survived the 2025 AWS outage.” Comes in gray. Just like morale. If it wasn't for the negative impacts on the employees and customers the word Schadenfreude would be very applicable.

And it's a sad indictment on current management practices and in particular the MBA brigade* that this is all by design, acceptable losses on the alter of profit, albeit short-term profit. Efficiency theatre as far as the eye can see.

*Yes, the same people who think Jack Welch was a misunderstood visionary rather than the spiritual father of mass layoffs, short-termism, and shareholder-value human sacrifices. The kind who see burnout as a KPI and chaos as a “scaling opportunity.”

Next they’ll launch a “Transformation Task Force” whose primary transformation will be renaming the same broken process from post-mortem to value realization review. A new acronym, a new logo, and boom, problem solved at a low low cost, honest, the consultants said so. Until the next outage, at which point someone will quote Sun Tzu in Slack.

znpy3• 9 months ago

Amazon has officially started their day-2 era.

mynameisjoseph• 9 months ago

I've been experiencing a similar problem after 30% layoff at my company. 30% does not tell who is expert or not. This amount of layoff includes very exprienced engineers inevitably, which causes very slow speed of problem solving, piling up of issues.

Today is when the Amazon brain drain sent AWS down the spout

Related Stories