Prime Down: Amazon’s sale day turns into fail day

https://techcrunch.com/2018/07/16/prime-down-amazons-sale-day-turns-into-fail-day/

by koolba • 8 years ago

646 348 8 years ago

rpeden• 8 years ago

If only they had access to some kind of scalable cloud hosting service, they could've completely avoided this sort of outage. :)

Jokes aside, I admire the work of the team(s) responsible for Amazon's web site. I use it so often and encounter glitches so rarely that it really stands out when something does go wrong.

coryfklein7• 8 years ago

Serious question: I've heard that Bezos's approach with building out commercial units is to break down each part of the vertical into separate commercially-viable components. Idea being if AWS doesn't make sense for 3rd parties to use then it may not be economical to use it internally.

Now the question part: would Amazon ever secretly run Amazon.com in a multi-cloud setup, balancing between AWS, GCE, Azure, etc?

nevir• 8 years ago

As someone that worked at Amazon a long time ago—back when AWS was just getting started—can confirm (historically). Publicly there were a myriad of AWS services; internally all we could use was S3 (for many years), if we were lucky. AWS being born of Amazon's "spare capacity" is an urban legend.

Nowadays, I hear it's quite different, and much of AWS is more rapidly dogfooded.

lev994• 8 years ago

Now the question part: would Amazon ever secretly run Amazon.com in a multi-cloud setup, balancing between AWS, GCE, Azure, etc?

I'm not sure how secretive it would be. AWS 'bids' on Amazon.com's business and Amazon.com is under no obligation to use AWS as it's cloud service provider.

camtarn5• 8 years ago

Worked for Amazon a few years back.

I can't see Amazon ever using external cloud hosting for anything except the most trivial of tasks. They're absolutely, utterly paranoid about any sort of confidential information, and I think even with encryption the perceived risk would be too high.

Spooky23• 8 years ago

I doubt it. These companies compete brutally, they would never allow Azure or Google to know their use patterns.

IDontWorkAtAmzn• 8 years ago

That's just marketing BS.

One of Amazon Retail's big internal goals this year is "finally get everyone off of Oracle".

Bezos opposed the creation of AWS. Almost everything that was early AWS (EC2, S3) was done over the objection of Seattle leadership. Look where the teams were based.

After they shipped AWS, it took eighty zillion years for Retail to use anything AWS offered beyond S3, and, as mentioned above, it's not like their 100% on DDB or RDS now: they have dependencies on freakin' _Oracle_ all over the place.

I mean, this is not a criticism of Bezos or Amazon at all, but at the end of the day, even if Bezos is a supergenius (and I see no reason to doubt that he is. Although I often wonder how he keeps himself motivated to continue to work so hard on building a Walmart competitor with the precious few years he has left on Earth given that he could do literally anything with his time), Amazon is still a company made of tens of thousands of people. It's not a 4 dimensional chess-ballet. It's got probably 95% of the same chaos and disorder that every big organization made of humans has. It just turns out that cutting it 5% in the right markets has incredible returns.

lev994• 8 years ago

The (hypothetical) day when AWS & Amazon.com goes down on Prime Day and Amazon stock tanks 50%.

deevolution• 8 years ago

Their stocks started declining around 2:30 eastern time

nodesocket• 8 years ago

It is beyond absurd people who sell $AMZN on this news. Amazon users will just keep refreshing until it comes back online, and then continue buying as normal.

adreamingsoul5• 8 years ago

Theoretically, everything has a limit. Including scalable cloud computing.

coryfklein7• 8 years ago

Hasn't this happened previously? It may become part of the celebration each year.

jbeales• 8 years ago

Just tried to do a search, got no results. Google search found me the product page, (which worked).

If you can't even search for products, there's a problem!

sj4nz3• 8 years ago

After reading Ryan Holiday's "Trust Me I'm Lying" and knowing how well AWS usually handles surges of traffic, I am not so sure that this isn't a marketing stunt that gets them an amazing amount of press. Cynical?

odorousrex• 8 years ago

Any publicity is good publicity!

coryfklein7• 8 years ago

It's happened again [1], and in less than 15 minutes!

That's it, I'm terming this the Klein effect.

[1] https://news.ycombinator.com/item?id=17499447

sj4nz3• 8 years ago

This made me smile. :) This is very close to to Poe's law as well, because after all, who can tell for sure if something that simply just went-wrong was intentional or not?

jerf• 8 years ago

There are cheaper ways to get marketing press that don't involve kneecapping one of your two primary revenue generation engines. They've gotten at least two major news cycles just out of vaporware promises to deliver packages by drone, for instance. (It may not be vaporware forever but it certainly isn't coming soon enough to justify breathless press from a few months ago.) It's not like they don't have any idea how to drum up press.

Jach• 8 years ago

Could also be an A/B test of some new infrastructure design to see if it's ready to deploy in November. I have no idea if they still hold everything together with Perl (via this templating system: http://www.masonhq.com/) but it wouldn't surprise me, and it also wouldn't surprise me that there'd be occasional pushes to replace it with something "modern" (or at least friendlier to the revolving door of new grads).

sj4nz3• 8 years ago

This would make a lot of sense, they essentially get a "game day" with actual sales, make a lot of money and a few months to prepare for the next wave of seasonal mob-shopping.

I don't think they're using Perl anymore however.

jimbofisher1• 8 years ago

This would go down as one of the worst A/B tests in history if that is the case, which I highly doubt.

sodafountan• 8 years ago

Nah, not at all. The vast majority of people don't care about the inner workings of a website, they just want to buy cheap shit and have it all work. Prime Day pretty much markets itself, I just recently got a package from amazon and it was plastered with Prime Day advertising, my co-workers are all talking about it, it's the centerpiece of their website. You'd have to work pretty hard NOT to know about Prime Day today.

adreamingsoul5• 8 years ago

lol. Seriously?

bluedino• 8 years ago

This is the page everyone seems to be getting

https://i.imgur.com/vpIHDpA.jpg?1

plandis• 8 years ago

Same on mobile. Perhaps Amazon retail is transition to a dog photos as a service business model?

eat_veggies• 8 years ago

did Dapps stand for Dog apps the whole time?

alternatively, Amazon rebrands their entire cloud lineup to PAWS

hmcm55• 8 years ago

changing the url to smile.amazon.com works

98codes• 8 years ago

For browsing -- search still seems completely hosed for me

degenerate• 8 years ago

People should be using Amazon Smile all the time, outage or no outage :)

I use this Chrome extension that rewrites all amazon pages to use it: http://www.smilealways.io/

FF equivalent: https://addons.mozilla.org/en-US/firefox/addon/auto-smile/

52-6F-62• 8 years ago

It doesn't appear to be available for Canada, unfortunately. I didn't even know it existed.

ovi256• 8 years ago

FWIW, it worked fine in Western Europe since 1 GMT, 8 hours ago.

oflannabhra• 8 years ago

Is there enough historical, public data available to estimate the amount of money Amazon is losing per second?

CamelCaseName3• 8 years ago

Last year they sold $2.4B, and this year I've seen estimates around $3.4B - $4B.

Prime day is 36 hours long, but I bet sales are weighted heavily in the first few hours.

So 3.7B / 36 / 60 / 60 = $28.6K per second, and then maybe double or triple that for the first hour or two after 3PM and that'll give you an idea of the scale.

There's also knock on effects, like reduced trust in Amazon/less orders for the rest of prime week, but also positive effects, like people who will just defer their shopping.

For what it's worth -- my sales are below my 30 day average. Glad I didn't go all out this year in terms of advertising.

mikestew3• 8 years ago

It probably doesn't help that the mobile app appears to load a random picture of a cute dog every time I press the "retry" button. So you can guess what I'm doing, trying to get it to load a new "Dogs of Amazon" pic.

Probably should have gone with goatse, reduce the load.

EDIT: do NOT search for "goatse" on your work connection. That alone, even if you've never heard the word, should tell you why I suggested it as an alternative.

tzs• 8 years ago

Maybe they figure that people who like dogs enough to want to see more will also click the link [1] to find out more about the "Dogs of Amazon", which seems to be unaffected by their problems. There they will find a story about Amazon's dog-friendly offices, 30+ pictures of Amazon dogs, and a video showing many of them. The time to consume all that is time the person is not hitting refresh on the shopping site, so maybe is a net win for Amazon.

[1] https://blog.aboutamazon.com/working-at-amazon/how-much-does...

ergothus5• 8 years ago

Interestingly, despite not being a dog person, I found the first instance of a dog to be a cute, human-relatable inclusion. On later pages I found it frustrating and annoying.

I'm sure there's a fundamental UX principle or two at work there, but I won't pretend to truly know what they are.

794CD01• 8 years ago

It's called the Duck Hunt principle.

alpb• 8 years ago

Dammit I made the mistake of searching "Goatse" at work.

mikestew3• 8 years ago

Oh, shoot, I forgot there were yungin's in the room. I'll edit that.

(Another "makes you feel old" moment when someone says they had to look up "goatse".)

benrapscallion• 8 years ago

The dog pictures are cached locally in the app - can confirm because they also show up when traveling through a cellphone-signal-free zone everyday.

mikestew3• 8 years ago

The pictures might be local, but I sure hope that retry button actually generates a request to the server.

pssflops• 8 years ago

I appreciate the edit for anyone whom has not experienced goatse as it also put my age in check.

blueside• 8 years ago

I can't help but feel terrible for the team of people there that ultimately gets blamed for this, I hope they can get some sleep tonight.

matte_black2• 8 years ago

A night of missed sleep didn’t hurt anybody.

coryfklein7• 8 years ago

It's not so much the missed sleep, as it is the 48 hours of intense heart-rending stress.

LocalPCGuy• 8 years ago

That...just isn't true. Missed sleep costs millions of dollars a day and worse, numerous human lives in sleep related accidents (car, medical and more).

Cofike• 8 years ago

The bathroom schedule will be tightened up by 15 minutes as punishment.

baud1472582• 8 years ago

Well, with this punishment, there won't be any bathroom break left!

wgerard• 8 years ago

I thought Amazon (or at least the AWS side of things) was supposedly fairly good about blameless post-mortems?

I remember people were praising them as such when this outage happened: https://aws.amazon.com/message/41926/

ben509• 8 years ago

They are pretty good about it, and the "Correction of Error" for this one will be pretty epic.

Generally, the rule at Amazon is that any particular f*-up is forgivable... once. (Especially if you can show that you had preventative measures, documented procedures and redundancy in place.)

That said, there will be finger pointing and blame because you're dealing with human beings.

smn1234• 8 years ago

People likely won’t be blamed. It’ll most probably be a review of process and a correction of errors to prevent future occurrence.

tres2• 8 years ago

Funny how this comes on the heels of aggressively expanding their workforce and trying to leverage themselves in a hundred different directions...

Maybe it's just me and my confirmation bias at work, but it seems that the core value proposition that Amazon provided -- high value, low margins on products -- has been eroding before our eyes.

Seems so much like the transition Microsoft made... too much focus on "synergies" and leveraging... not enough on keeping the bilges dry and the engine running.

It's funny... Fred Brooks wrote about this in 1975... and we're still making the same mistakes forty years later. There are real limitations to how quickly any organization can grow. Even awesome companies who are excellent at building organizations -- places like like Amazon and Microsoft -- can't organize this law of software development away.

klodolph3• 8 years ago

Seems so much like the transition Microsoft made... too much focus on "synergies" and leveraging... not enough on keeping the bilges dry and the engine running.

The companies that just keep the bilges dry and the engine running are the ones that we love, but they’re gone because they got made irrelevant. Or they got absorbed into something larger. Microsoft has a bunch of failed initiatives (Windows Phone, Zune) plus a bunch of successful ones (Azure, Xbox, Office 365).

If you’re up for classic books like Brooks check out The Innovator's Dilemma. You have to try to expand in a hundred different directions because you don’t know which one of those hundred directions will be relevant next decade, and you have to be unafraid of cannibalizing your core business because if you don’t eat yourself then someone else will eat you instead.

tres2• 8 years ago

Thanks for the recommendation!

I think the hard part is to walk the line between stagnation & over-expansion. This is a dilemma we all face in organizations large and small as well as individually.

Building systems and processes (a.k.a Habits) that allow us to assume and integrate some new set of "stuff" without having to think about them (so we can move on to the next new "stuff") are what sets these companies apart; Amazon has been brilliant at this; however, from my perspective, looking at all the high-rise office buildings going up, it seems like maybe Icarus has flown too high...

But again, I could be building a narrative to fit my preconceived ideas... I'm definitely no authority here. Maybe this is just another blip... I think more troubling to me is the overall degradation in quality of the things I have formerly taken for granted in Amazon -- the quality of the products and ratings.

marcosdumay2• 8 years ago

The Innovator's Dilemma is certainly not about synergies between products. By its conclusion, it's worthless to try to diversify within the same structure, you'd better create a new company and turn the old one into a holding.

jethro_tell2• 8 years ago

I think that would seem relevant to me if this was the first time amazon.com had been unavailable. Or the first time any big website has gone down.

antonvs4• 8 years ago

This seems like a bit of an overreaction. Amazon has had big outages before (particularly at AWS), and they invariably solve the problem and move on. As someone else observed, events like this are mainly notable because of how infrequently they happen.

camtarn5• 8 years ago

Kinda wish I still worked there and could see the post-outage writeup.

My bet's on some unforeseen bottleneck that affects search and static pages. Almost everything within Amazon is crazy scaleable, but there are some bits where you scale them up and their behaviour changes radically. For instance, a service's cache misses might skyrocket as customers get distributed over a wider set of servers, causing service response times to increase just a little bit on average, tipping a dependent service over into more frequent timeouts, causing its downstream service to blow a timeout-percentage 'software fuse' and stop using that service... etc etc.

Given that each of those services (and many more possibly-related ones) will have an on-call engineer paged into a conference call when the manure hit the rotating ventilation apparatus, there are going to be a lot of unhappy people cancelling their weekend plans right now. I definitely don't miss that aspect of the job!

coryfklein7• 8 years ago

Thanks for the comment and I love all the insight but serious question about:

there are going to be a lot of unhappy people cancelling their weekend plans right now.

The week just started... are you saying that you can already anticipate the war room for an event like this to extend through this coming weekend?

camtarn5• 8 years ago

Haha, no, oops. My last few days have been a bit crazy and I'm on holiday, so I just thought it was Sunday ;)

adreamingsoul5• 8 years ago

oncall is the worst

cheeze4• 8 years ago

Not oncall for Amazon but I don't particularly mind it. It sucks to have to be near a computer for the time, but I actually find digging in and finding that needle in a haystack pretty fun. On a good team, the life interruption is minimal.

zerotolerance2• 8 years ago

I prefer to be oncall as long as I'm empowered to prioritize software improvements that reduce ops burden and improve customer experience.

zwischenzug• 8 years ago

We reduced the stress of on-call by introducing processes that supported creativity and reduced panic:

https://zwischenzugs.com/2017/04/04/things-i-learned-managin...

throwaway427• 8 years ago

smile.amazon.com was working fine during the outage, if that helps narrow it down...

azhenley• 8 years ago

It wasn't though. I saw an item being listed as a Prime Day deal on the normal Amazon site, then I searched for it through smile.amazon.com and it wasn't there. Went back to non-smile and it was there (but throwing errors so I couldn't click it anyway...)

gnat• 8 years ago

Not for me: a page that gave a fast 503 on amazon.com would spin indefinitely on smile.amazon.com

aviv3• 8 years ago

My bet is they maxed out on the SQS FIFO 300 messages per second limit.

camtarn5• 8 years ago

I have no idea really (I stopped working on the retail websites in 2013) but my gut feel is that that's about a couple of orders of magnitude too low at least.

But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...

adrianmonk• 8 years ago

Weekend plans? Prime Day is going to be over in the early part of the week.

Does Amazon expect a fix to be deployed ASAP after the immediate crisis is averted?

kaspm2• 8 years ago

My guess was either that

OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems

OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data

OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.

zerotolerance2• 8 years ago

The systems at Amazon are too large to leverage autoscaling. When I was there (in Marketplace) it was generally estimated that Amazon used about 80X the capacity of their entire AWS public cloud.

HankB992• 8 years ago

When I checked it out, the site seemed responsive but search was broken. I searched "usb flash drive" and was presented with 2 results. I pondered this for a moment and then realized that there were 400 pages with 2 results/page (as far as I checked.) Perhaps there is some load shedding algorithm that reduces the result count per page to reduce load. It did discourage me from further searching so I guess it worked. ;)

lanius• 8 years ago

I actually managed to add an item on sale to my cart, but my cart is now empty after refreshing. Oh well, maybe next year.

phy62• 8 years ago

This made me laugh more than it should have.

jcampbell1• 8 years ago

I'm seeing automatic reloading the page every second or so. Maybe some bad javascript, though it isn't adding entries to the history. Looks like they have a script that is DDoS themselves.

phy62• 8 years ago

Same, the search results page keeps refreshing, void of any items.

itsEtai• 8 years ago

Help strikers in Europe by NOT purchasing from Amazon today. Thank you.

ryeguy• 8 years ago

But I want things.

middleclick• 8 years ago

I have completely stopped using Amazon, with the final nail in the coffin being the stories about how they treat their workers. I think it's just a matter of dependence and most of the stuff I used to get, I can find it elsewhere.

kasparsklavins• 8 years ago

Explain?

highwind• 8 years ago

Maybe this? https://www.washingtonpost.com/business/2018/07/16/amazon-pr...

mlrtime• 8 years ago

"Europe" strikes as often as the wind changes.

antonvs4• 8 years ago

Which only raises the question, why doesn't the US? Working conditions must be so fantastic here that no-one feels the need to strike.

It can't be that companies have crippled unions so that they can treat wage and working conditions as a one-sided negotiation. Surely not...

AndyMcConachie2• 8 years ago

Damn right we do. And it's why we don't live in a capitalist dystopia.

baud1472582• 8 years ago

Like the f*cking rail service in France, where they've been striking 2 days every three day, to protect outdated privileges and political relevance.

AndyMcConachie2• 8 years ago

https://www.eldiario.es/economia/principales-reivindicacione...

cpuguy83• 8 years ago

Wouldn't it help them to purchase from Amazon today? If there are lots of orders but no one to fulfill it seems like that would be greater pressure on Amazon.

all_blue_chucks2• 8 years ago

Yeah, if everybody stopped shopping there, the strikers would be muuuuch better off...

rednerrus• 8 years ago

My guess is this is an intentional marketing ploy. Think of how much press this is generating. Frontpage of HN. No way we are talking about this otherwise.

NathanCH2• 8 years ago

Amazon is the front page nearly every day for some reason or other (:

wetpaws• 8 years ago

If you are big enough it's hard to miss you.

positr0n• 8 years ago

They are losing millions in revenue every minute their site is down. Way too expensive for marketing.

ergothus5• 8 years ago

Not sure I follow.

"Hey did you know Amazon is unreliable and unusable?" "Really? (checks) Yup, you're right!"

(somewhere in the distance, a marketing guru cackles with delight)

rconti3• 8 years ago

yes, this is the first I'm hearing about this whole amazon thing!

adreamingsoul5• 8 years ago

#tinfoilhat

krapp2• 8 years ago

No. I work in one of the biggest fulfillment centers in the network. Shit broke.

rajathagasthya• 8 years ago

What happens to people responsible for the crash today (infra, culprit services)? Does Amazon take some kind of "action" since Prime Day is a huge, once-a-year event for Amazon?

chronid5• 8 years ago

They will have to write a detailed post-mortem (with many people with titles starting from director watching it every week). Based on what comes out from the post-mortem automation/testing will be implemented to remove/mitigate the failure from the equation in the future.

Unless this is something egregious (ex: a manager not allowing the team to "scale up" in preparation for the event) no one will get fired. Tempers may flare a bit if it is something stupid (it's usually not).

Nothing different from what happened with the (very public) DynamoDB and S3 failures of yesteryear.

wenc2• 8 years ago

I wonder how Alibaba Cloud handles similar events [1], where there are bursts of 256k/s transactions and ~1bil packages being shipped out.

Do they just do brute-force massive scale out?

Amazon's US market is big, but my understanding is that number of online users in China (> 400mil) exceeds the population of the US (~325mil), which makes me wonder if the folks there think about data architecture a little differently than we do.

[1] https://qz.com/1127087/singles-day-crazy-stats-from-alibabas...

rconti3• 8 years ago

Could be. But also, Amazon is doing Prime Day in Europe. Albeit they've got a time zone offset.

paxy• 8 years ago

Doesn't seem like it was a scaling problem at all in Amazon's case.

cheeze4• 8 years ago

What? How do you jump to this conclusion!?

rannn• 8 years ago

Probably every day at Alibaba is a prime day in terms of transactions...

akvadrako• 8 years ago

True but they still have Singles Day, so it's not like they have constant load.

jamesblonde• 8 years ago

Alibaba build on Apache Flink. Today is a good day for Apache Flink.

Taniwha• 8 years ago

remember that much of Alibaba is essentially a front for lots of small companies - they don't do the shipping themselves

wenc2• 8 years ago

Oh I included that number to indicate the scale of orders that have to be handled by Alibaba's data systems -- I understand that they don't ship the packages themselves.

Also, I just read that as of 2017, there are 700mil internet users in China, 90% on mobile. The scale there is just staggering.

jjeaff2• 8 years ago

Would Alibaba actually ship anything close to 1 billion packages. Most listings I see on Alibaba are 3rd party.

mrep• 8 years ago

It's up, aaaaannnnddddd now it's back down...

rc-1140• 8 years ago

... Is there anything even good for Prime Day this year? Or the year before? Two years ago I remember seeing at least some Dell Workstations that could be repurposed into cheapo home servers. Most of the stuff seems to be odds, ends, and stuff that the various Chinese product-clone companies couldn't get rid of.

JacobJans• 8 years ago

I got a baby car seat for half off, which saved a lot of money, actually.

rootbear• 8 years ago

The web site is a bit of a mess. And it seems that, just since this morning, my entire Wish List has vanished. Foolish me for not having a backup...

antonvs4• 8 years ago

Check again, mine was missing but it's back now. It's doubtful they'll actually lose data, issues are probably just due to services that are offline.

XalvinX2• 8 years ago

If you aren't being a little sarcastic, you are lucky. The bug saved you money for shit that you obviously are not wishing very hard for. Kids at Christmas time certainly don't need a backup....

mh-• 8 years ago

I wouldn't blame yourself too hard.. Amazon has strategically made it very easy to add items to your lists (bookmarklets, chrome extensions, etc.) while offering _no_ supported way to get the list back out.

hartator2• 8 years ago

It's interesting that you can have hired the best talents in the world, but still have major outages. I wonder if there a way to ensure more people = more stability. Sounds a bit stupid, but maybe if each datacenter has its own software team working on the same issue, it will be very redundant work, but maybe it will be more organic in its failure?

gm-conspiracy• 8 years ago

Or a team of 9 pregnant women making a baby in only one month?

hartator2• 8 years ago

Should work because 9 months / 9 = 1 month.

You probably want a distributed team though, with maybe fetus being thrown around by pneumatic mail.

HillaryBriss3• 8 years ago

I've heard this analogy before but I believe it could be improved. Thinking in terms of months leads to an awkward divisor.

It would be better to go with a divisor of 10 so as to assign precisely one finger and one toe to each of the distributed work groups.

(But, I have to admit, you may want yet another group to coordinate and assemble it all.)

aviv3• 8 years ago

Yikes AWS console down

biztos• 8 years ago

What about the advertising money already spent?

I read somewhere (sorry, forgot where) that Amazon had been pushing sellers to spend like mad on ads within Amazon.com for Prime Day, apparently it gave you a big edge over whatever the algorithms suggest.

Those sellers will have missed their sales targets, and will consider the ad spend to have been wasted. Will they get it back?

grivescorbett• 8 years ago

I’d like to read that comms doc... “how to explain to your client what happened on pre day.”

giarc• 8 years ago

The ad dollars are only spent if the ad is shown to users. If the site isn't up, ads can't be displayed.

tssva• 8 years ago

A lot of the ads were shown prior to Prime Day . They were to drive traffic to Amazon on Prime Day and not to promote product the day of.

fencepost• 8 years ago

And vendors who boosted supply based on anticipated sales (both discounts and purchases driven by discounts). Are vendors going to find themselves with thousands of extra widgets on-hand but without the anticipated purchasing frenzy they were counting on to sell them?

CamelCaseName3• 8 years ago

Yup, and if you have too much inventory, you can either remove it or sell it at a discount, both are expensive options.

Otherwise you'll get Amazon telling you, "Hey buddy! You sure are using up a bunch space and not selling much. Why don't you not do that? We're limiting the amount of storage space you can use for Q3.

kolleykibber• 8 years ago

It's also impossible to log into the AWS console. Surely something of such importance should be separate to the ecommerce site.

andrewguenther• 8 years ago

AWS console works just fine for me.

cloakandswagger• 8 years ago

I can log in without issue. If you're having problems, it's probably unrelated to the Prime Day outage.

lawnchair• 8 years ago

Down for me also and it's being reported on https://status.aws.amazon.com/

helper2• 8 years ago

I can't login to my root aws account right now. It's pretty annoying that the root account login for aws is tied to amazon.com retail accounts.

rconti3• 8 years ago

yeah, I'm pretty sure this could be causing some serious issues with plenty of folks. not sure if it's just the web console or if APIs are affected too, but I sure can't get in.

philsnow• 8 years ago

I've always been pretty sore about this, that my retail and AWS accounts are linked, with no clear way to disassociate them. If I had known that the accounts were going to be joined at the hip, I would have created my AWS account with a different email.

all_blue_chucks2• 8 years ago

Using the root account is a poor practice anyway...

helper2• 8 years ago

Is it really so difficult to imagine that I needed to perform an action that can only be done from the root account?

For example, the pen test authorization request form can only be filled out from the root account.

blairanderson• 8 years ago

I am unable to buy/bid on my Amazon AMS ads still as of 2:12PM.

rezashirazian3• 8 years ago

Something no one has mentioned yet, could it be that the engineering force at Amazon is no longer what it used to be?

I can personally point to two friends who I consider top notch engineers and designers that have left Amazon because of its toxic culture. I'm sure I'm not the only with these anecdotal examples, we've all heard the stories. At the end of the day years of unbalanced work/life balance, overly aggressive management and frugal approach to everything makes for a weak argument for A players to stick around.

Could this be an example of crumbling engineering standard at Amazon?

tills13• 8 years ago

Considering they cleared $2.6 billion on last year's Prime Day, and membership is up YOY, I'd say this comment is 100% knee-jerk/over analysing. Stuff breaks, things go wrong, planning for the worst sometimes isn't enough.

chc• 8 years ago

I don't see how "they sold a lot a year ago" and "membership is up YOY" is meant to dispute the idea that engineering talent is leaving. I agree we don't have enough evidence to say for sure, but those figures don't seem relevant.

rezashirazian3• 8 years ago

My comment has nothing to do with Amazon's current financial success or growth. It's regarding their engineering team and their ability to recruit and retain top talent.

It is not a binary state of absolute destitute and top notch brilliance, it's a trend that can move one way or another and will show itself in more frequent outages, poorly rolled out products, lazy design and etc.

The plumping can keep on working for a few years even after it begins to erode. Historically there are many examples of this.

HillaryBriss3• 8 years ago

have we reached peak Amazon?

deegles2• 8 years ago

My gut feel is that they did everything right to prepare based on forecasts from last year but still got swamped by demand.

ergothus5• 8 years ago

I saw the page reload multiple times when doing a search before finally dying...I can only imagine that means additional load generated by the initial triggering failure.

Which in turn tells me they didn't test the failure case. Now, Amazon is a huge and complicated beast so I don't want to imply this was a "dumb" mistake, but (assuming I'm correct) it is a failure that risked making MORE failure, so it's not demand alone to blame.

ams6110• 8 years ago

I have Amazon Prime, and I had no idea that today was Prime day. Haven't seen any marketing for it, and nothing in email until this morning.

throwaway2893552• 8 years ago

AWS employee and bar raiser here.

Something no one has mentioned yet, could it be that the engineering force at Amazon is no longer what it used to be?

In many regards, yes. The bar had to be lowered to meet the demands of growth. We've also taken in a lot of hires from companies that have brought their culture and friends with them. The culture at Amazon is not what is was even 2 years ago. It is in many places day 2.

No one also seems to notice that Amazon retail often suffers widespread issues like this. We can count on SEV1's happening during peak as things blow up badly. This has happened several years in a row, and sadly the themes are pretty much the same across all: forgot to scale (yes...really) or some stupid system bottleneck. It doesn't help that Amazon retail has a good amount of its workforce based in India and seemingly disconnected from the Seattle based leadership.

nikofeyn• 8 years ago

what is a bar raiser?

and when has amazon ever been an engineering force? i have always felt the website and service experience is a relic of the 2000s. more often than not, i get the answer “our system can’t do that” from customer service.

dabockster• 8 years ago

The bar had to be lowered to meet the demands of growth.

Not only that, but I've noticed that Amazon has started using way more contractors (not actual firms, but more Mechanical Turk/UpWork like contractors, if not just straight up misclassification) and agencies within the past year or so. I can't say that I'm surprised that Amazon is having technical issues now that they've grew in numbers but not in technical chops.

On a side note, it is a great time to become a recruiter in Seattle with all these agencies popping up. /s

ergothus5• 8 years ago

It is in many places day 2.

For those unfamiliar with internal Amazon lingo, a big deal is made of it always being "day 1"

https://www.fool.com/investing/2017/04/13/jeff-bezos-says-it...

weka• 8 years ago

It doesn't help that Amazon retail has a good amount of its workforce based in India and seemingly disconnected from the Seattle based leadership.

Could you expand on this a bit, please?

yeukhon2• 8 years ago

No load testing ever?

duxup• 8 years ago

I got the impression from some former Amazon employees that they were burnt out and put their time in and got Amazon on their resume and were just done with life at Amazon so they moved on.

Do you sense any attrition like that?

rrjanbiah• 8 years ago

FWIW, if you can look into https://github.com/rrjanbiah/amazon-india-issue-tracker

ainiriand• 8 years ago

I can see what you mean. Take for example the email I received to promote Prime. A mix of spanish and english with some very bad translation problems in spanish. It was a bit embarrasing to read it.

convivialdingo2• 8 years ago

Absolutely. The fact that this is ongoing hours later is pretty much a clear sign that there’s problems.

I understand that it’s not always easily fixable. But honestly this looks pretty bad. Could be anything from bad code to a flakey cable.

I got out of high volume websites - too much marketing, unpaid overtime, and horrible work-life balance.

sadamznintern3• 8 years ago

Could this be an example of crumbling engineering standard at Amazon?

As someone that's been going through quite a bit of depression because Amazon was the best offer I got (as opposed to Google or Facebook because I'm still bad at coding interviews) I'm afraid this is the straw that's going to break the camel's back for me. This is exactly what I was afraid of and exactly what I have waiting for me when I join next week.

There's no point anymore.

throwaway080383• 8 years ago

Chin up. You're probably in the top 5% of the country in income. If you're in Washington, feel free to tack on another 9% or so to your income in state taxes you don't have to pay.

in_cahoots• 8 years ago

100% serious here: you should probably see a therapist. Life is so much more than your job, and your job is so much more than the brand name of your company. Having worked at a failing business and one of the companies you mentioned, I’m no happier at the latter than I was at the former. If you define yourself by the names on your resumé one day you’ll wake up and realize how much of life you’ve missed out on.

kbyatnal• 8 years ago

Hey, I've seen a few of your posts around here and just wanted to encourage you to keep your head up. Amazon is an amazing place and even though I don't know anything about your specific situation, there are a lot of incredibly smart people there and I promise that you'll learn a ton.

symlinkk• 8 years ago

I would fucking love to work at Amazon. You’re doing better than almost everyone I know.

manfredo• 8 years ago

People say the same thing about every big tech company - Google and Facebook included. Trust me, employers will still be impressed with Amazon on your resume.

autokad• 8 years ago

new person to amazon.

they have a lot of really amazing and smart people. but like anything in life, its what you make of it. I'd say put your best into the position and try to learn as much as you can from others. no good reason to do less

throawayamzn2• 8 years ago

I was in the exact same boat - ex Amazon intern, felt I was a pretty solid engineer, but didn't quite nail the FB interview. Ended up joining Amazon because I was sick of interviewing.

Let me tell you: it will be OK, and the other posts about making the best of your situation are true. Amazon is an incredible learning opportunity if you stay open to it. I'm still working there a year later and I'm building far higher impact projects than my friends at Goog and FB because like another poster said, Amazon lets SDE1s work on just about everything. The growth potential is tremendous if you show you're competent and willing to learn.

Chin up, you're in a good position :)

barnstorm• 8 years ago

You can't let an outage or some alleged talent gap be a reason for wanting to tap out.

You need to realize that your doing great, folks would kill to get a job at Amazon, the scale and the challenges are mind bending compared to what most other companies deal with and the interview is grueling and you made it.

Technology is a word that describes something that doesn't work yet and Amazon thinks that you are a person that can help tip the balance.

codeonfire3• 8 years ago

In addition to the shit that is not politically correct to say, Amazon relies heavily on interns for production code and ops. Bezos has this view that the unskilled can get him 80% of what he wants and then a few top people can smooth out the rest. In reality the few top people never even come in contact with the kid that is trying to keep amazon.com running overnight. Successive waves of interns and SDE 1's fuck up the same shit over and over. Bezos is stupidly applying this same strategy at his space company.

fjsolwmv5• 8 years ago

I always wonder how stupid people can accumulate $100Billion being stupid.

Analog24• 8 years ago

This is complete nonsense, at least in my org. Interns are given projects that are as far from production code as possible. I'm sure there are managers out there who have done this but it does not extrapolate to the entire company and it's ridiculous to think that it's a company wide practice to make interns responsible for anything that would affect customers.

wiredone2• 8 years ago

This is complete BS. Interns are basically on a really well paid holiday and given random features that may/may not ever make it to production.

fjsolwmv5• 8 years ago

No. Amazon has had outages on holiday shopping days throughout its history.

Amazon has always been toxic and frugal, so obviously it didn't interfere with whatever mythical software quality it had in the past.

And today's failures might be because past engineers built unstable unmaintainable systems and then ran away.

And of course amazon is 10, 100, 1000 times larger tech system than it used to be.

downrightmike• 8 years ago

I mean, I see it fail from the fact that there aren't any deals I want. Maybe if their fire tablets ran full android, but there isn't really anything on sale.

dawhizkid• 8 years ago

I’m curious what the reprocusions are, if any, once a post mortem is completed and teams or individuals that contributed to the outage are identified. Is “causing” this a fireable offense?

beckler• 8 years ago

lol, like employers need a real reason to fire someone in this day and age

NathanCH2• 8 years ago

It shouldn't be unless there was malicious intent.

selectiveshift• 8 years ago

Not unless there was malicious intent or willful negligence. Amazon is a data driven company. The data shows that a “blame” culture results in more incidents. (Airline industry taught us this: https://www.faa.gov/about/initiatives/maintenance_hf/library...)

smt884• 8 years ago

Do you know this from experience? Or are you just guessing?

smt884• 8 years ago

My friend was fired years ago for an AWS outage that took down Netflix. I don't know any other details, but should ask him.

dlgeek• 8 years ago

No. Amazon's culture is super big on blaming the system. I know of incidents where operators made mistakes doing stuff that directly caused really big outages - the postmortems were entirely about the (lack of) safety of the tooling involved that allowed a single person to make such a mistake, and involved examining all the tooling of entire business units to try to identify other tools that could have such an impact. No individual blaming is involved - and in fact, postmortems always refer to "the engineer" or "the operator" and never include names.

russellbeattie• 8 years ago

I work in a non-retail part of Amazon and I'm on vacation. Hasn't stopped friends and family from texting me about this though. As if I personally can go in and reboot a server or something. Hope we get it sorted soon!

If you're affected by this, please accept my unofficial thanks for your patience and understanding. (If you're a coworker in retail, good luck getting things up and running!) :-)

macshaggy• 8 years ago

Well, everything I wanted to buy is simply not on sale. If Prime Day isn't supported by those things that I want or need then why would I participate?

fma• 8 years ago

I've participated before only to be disappointed. I didn't even Beyer looking at their 'deals' this year. I consider this their garage sale to get rid of junk...unless you want to buy Amazon products like Ring.

ComputerGuru• 8 years ago

I know TFA is about technical failures, but the deals themselves are also incredibly lacklustre. I was expecting at least the Warehouse Deals part of Prime Day to come through, typically 15 to 20% off all used offerings. This year however, Amazon restricted it to only select listings which translated to a few hundred items total. Very sad.

HillaryBriss3• 8 years ago

I didn't see anything that appealed to me either. One other person told me the same.

Maybe Amazon's overloaded system was caused by shoppers checking back more frequently than in past years because they can't find really good deals this year either but keep looking harder and harder anyway?

Jaruzel• 8 years ago

I've come to the conclusion that 'Prime Day' is just a clearance sale. Unless you find something that is exactly what you've been planning on buying, it's all just 'tat' and not worth buying even at Prime Day prices.

XalvinX2• 8 years ago

I just realized the influence of Hacker News. I read techcrunch every day and it is rare for an article, any article, to get more than 2 or 3 comments (although, to be fair, they just added commenting in the last few months)...this one got 35 already!

Anyways, I think that this is NOT a fail for Amazon at all, but a major win. They obviously created all kinds of attention! I'm sure they will get it right next time, and maybe, if they think fast, can reverse this situation by offering those who were disappointed a 2nd chance at an even-greater discount...at least that's what I would do.

anigbrowl• 8 years ago

Industrial action is cool and good https://twitter.com/kadybat/status/1018926864767676416

dingo_bat• 8 years ago

TFA is a super buggy webpage.

oh-kumudo• 8 years ago

Amazon is worse than Alibaba even, what a failure today.

kazishariar• 8 years ago

It's hush-hush in the industry that CAT5 and it's predecessor the CAT5+1, both are shielded with peanut butter. It's also a little-known fact that most datacenters or PCs as we in the NOC tend to call them. Are also built around tropical rainforests, which provide both security from the common man, as well as naturally cool down the millions of heatsinks needed for the cloud. But an even lesser known fact is that monkeys, yes monkies love peanut butter. --I'll leave the rest to your imagination. But one of those fat guys, drinking beer, with a hoodie, and in need of a bath left one of the windows open. The rest... well, the rest was prime.

mirimir• 8 years ago

Huh? Amazon seems OK now.

Edit: But hmmm, Quora just went down.

504. Gateway Timeout.
Quora is temporarily unavailable.
Please wait a few minutes and try again.

And they use AWS, right?

Edit: I just got as far as search, order, cart. But no account as Mirimir, so ...

Edit: Re Quora - http://downdetector.com/status/quora

I wonder what other AWS stuff is down. If that's it, anyway.

anonu• 8 years ago

Consumerism at its best... Probably for the better than Amazon Prime is down (at least for me - I'll speak for myself).

Honestly, my life got better when I stopped getting packages to my door. Only buy the most essential things you need - sell the stuff you don't. Better to live an uncluttered life without "things"...

wheeler5x5• 8 years ago

How about some love for their marketing guy who made it all happen?

Jeff probably said "make it rain, let's see if your hordes can take down Amazon.com," and this guy basically accepted, and succeeded at, the challenge.

Prime Down: Amazon’s sale day turns into fail day

Related Stories