AWS down again?

https://aws.amazon.com

by raffraffraff • 5 years ago

271 232 5 years ago

aws.amazon.com

I've been getting the "We're sorry!" error for at least 15 minutes. No idea about regions or anything, I was just about to look up their docs about moving accounts between orgs etc. Anyone affected?

Edit: Seems to be flapping between the 'sorry' error and a blank page. Thoughts and prayers with the SREs, if they call 'em that over there.

bierjunge• 5 years ago

Same here. And as always, AWS status page [0] is completely green and useless.

[0] https://status.aws.amazon.com/

my123• 5 years ago

Management console seems to be accessible fine to me, maybe it's just the aws.amazon.com website that's down.

Thaxll2• 5 years ago

Because everything works fine? So a landing page does not work you need to put defcon 3?

johnisgood• 5 years ago

Which means people suggesting one another to compare their downtime with AWS' should stop suggesting it, as you cannot rely on their data at all.

reidfromhome• 5 years ago

The only thing that was down was the marketing home page at aws.amazon.com, which is not a “service” and is not on the SHD. All of the actual services are fine, hence the green.

markus_zhang4• 5 years ago

We are sorry page is all I got.

x0xrx• 5 years ago

I wonder why the error message is also in … French? (I’m not located anywhere French speaking).

Désolés!
Une erreur s'est produite lorsque nous avons tenté de traiter votre requête. Soyez assuré que nous travaillons déjà à la résolution du problème que nous pensons trouver très rapidement.

jpgvm• 5 years ago

Mine is in Japanese:

申し訳ありません。
リクエストの処理中に問題が発生しました。現在問題を調査しておりますので、解決するまでもう少々お待ちください。

markus_zhang4• 5 years ago

I'm in Canada so sort of make sense to show bilingual page.

julianlam2• 5 years ago

Unless you're Quebecois, in which case the province would prefer French appear first. _smirk_

tapan_jk• 5 years ago

I am in India and my message is in English and Japanese. Never seen a Japanese message on aws before.

leto_ii• 5 years ago

Same in the Netherlands.

myth2018• 5 years ago

I'm in Brazil and also read the message both in english and french

redwood2• 5 years ago

Still no post mortem from Monday's right? We're all still in the dark?

davewritescode• 5 years ago

When I've worked with AWS and other providers before, they've typically given early indications about potential root causes under NDA. If you're a small customer, you probably get to wait with the rest of us but the big folks probably have some idea of the root cause.

tr33house2• 5 years ago

Can confirm this

naikrovek• 5 years ago

Still

it's usually a month or more after a large outage to see the full breakdown on what happened. people expecting to see it the same week are kinda not living in reality.

redwood2• 5 years ago

Fair I guess I would have expected more in the way of official comms though. Most of what I've seen is based on deduction and speculation rather than from the horse's mouth

markus_zhang4• 5 years ago

btw status page doesn't show anything

CliffJumper• 5 years ago

OFC not. They'll deny there's a problem until it goes away.

woeirua• 5 years ago

Amazon’s going to have to start paying out on some SLAs if this goes on for much longer…

zwily• 5 years ago

There’s no SLA for the console, IIRC.

js4ever• 5 years ago

I see no impact on my infrastructure and the dashboard is still accessible on my side. So I'm not sure about exactly what is down.

markus_zhang4• 5 years ago

Guess it's just the homepage then.

rswail• 5 years ago

I just logged in via standard console and things seem to be fine.

That direct URL aws.amazon.com gives me a weird error, but everything else seems to be fine.

kchoudhu• 5 years ago

Yup, my traders are getting the "sorry!" error when trying to log into QuickSight, so looks like another great day at AWS.

kumarvvr• 5 years ago

Yes. I am trying to access the console and there is an error message. I am from India.

kl4m• 5 years ago

https://console.aws.amazon.com up.

SCHiM2• 5 years ago

I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself).

Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political pressures and weaselly worded SLAs (how do you mean you had an outage? Only 49% of your requests were failing!).

tinalumfoil• 5 years ago

Even if you can manage more uptime on your own than through the cloud (which I doubt), being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing.

Your customers will be more understanding if your outage is part if a wider outage that makes national news.

Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops to about 98%. It doesn't drop if the downtime is perfectly correlated.

Even if you don't directly integrate, your customer's workflow might. They see many services down and say, "cool, time to get caught up on laundry". If only you're down it's more aggrevating.

justbaker• 5 years ago

Your customers will be more understanding if your outage is part if a wider outage that makes national news.

Yes they definitely are.

savant_penguin• 5 years ago

Or they could see their entire product line break at the same time, in such a way that they cannot mitigate

ocdtrekkie2• 5 years ago

My desktop PC has better uptime than the cloud right now. So does my datacenter. In fact, I am not sure I have anything electronic that doesn't have better uptime than AWS or Azure.

y4mi2• 5 years ago

Even if you can manage more uptime on your own than through the cloud (which I doubt)

Getting higher uptime is super easy for smallish inhouse deployments.

You just don't install any updates and let the server run, trusting your VPN to shield you from possible security issues.

The maintenance burden is the reason why people often prefer the cloud services, not the uptime. Because maintaining the instance with updates, reading all patch notes and steps for migration, keep every health metric monitored and respond quickly on issues without getting stuck googling for possible reasons is quite a bit of work and quickly forces you to employ n+1 people.

5e92cb50239222b• 5 years ago

Wha? We host everything on our hardware (which is nothing special) and haven't had any downtime in this year (yet). And we're just another run of the mill dev shop, very far from "superstars" who work on these (supposedly extremely stable) platforms.

SCHiM2• 5 years ago

That's usually a good thing.

For any individual company able to offload the blame, that's great. It's not so great if half the countries' doorbells, robot cleaners, various home streaming service setups, the baby camera, the fridge, the smart TV and your phone stop working... All at the same time.

In my opinion, the 'downtime' really should be measured in $NUM_SERVICES_STOPPED X $TIME, instead of just $TIME. And in this case I think any long time Amazon outage is orders of magnitudes worse than your regular old slow IT company outage.

albertopv2• 5 years ago

An our on premise linux server recently reached an uptime of 1000 days. Yes, days.

jve2• 5 years ago

If you happen not to be on the major cloud platform where everybody is enjoying donwtimeshare, we provide you with some downtime monkeys that will pull the lever EXACTLY when your platform should go down!

Let no more have your integrations with systems in the cloud frustrate your customers when they are unreachable - let the news explain the downtime. JOIN the Downtime Umbrella NOW and receive 5 downtime lever pulls for FREE! /s

ipaddr• 5 years ago

Why would I want to be down when everyone else is down? That's a period where I can get customers from everyone else.

bob1029• 5 years ago

I'd rather have control over my circumstances than the ability to assign blame for them.

"my outage means the customer is probably also down so they maybe don't care" is not a viable way to run a business in my mind.

gerbilly6• 5 years ago

being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing.

A long time ago, I used to circulate snarky little emails at work.

One of them was responding to this very concept.

My managers were throwing out our working and mature UNIX servers (implementing DNS, Mail, and other services) in favour of NT.

The new system crashed a lot, we had some security breaches, but at least it was 'industry standard.'

Managements' response was that with the UNIX stuff we had no one to pin our outages on. Now we could blame Microsoft, and call their support line.

I circulated an email making fun of this justification, which promoted a fictional product called 'Blame Studio' which would help you map out the blame path for any of your products or services.

It would help to make sure that none of the blame ever landed on you, but rather was always redirected onto some other company.

nabla9• 5 years ago

Also, not all uptime is equal or worth of the price.

What would be more valuable to you:

1. 99.5% uptime where unscheduled downtime is max 10 minutes vs.

2. 99.5% uptime where unscheduled downtime comes in 2-5 hour chunks.

ocdtrekkie2• 5 years ago

Another key point of this: Selecting your downtime. If you own your own stuff, your major changes and maintenance are done at times favorable for your business. AWS or Azure configuration fat-fingers happen when best for their business, not yours.

gtirloni3• 5 years ago

I don't see any momentum in that direction at all. If anything, it's quite the opposite.

These outages have almost zero impact on anyone deciding to use a cloud provider.

csw-001• 5 years ago

Agreed. In fact, I've heard government decision makers actually swayed to go to the cloud due to outages with logic along the lines of, "If Amazon can't keep AWS running, what chance does our 30 person, chronically underfunded, team have?" There is also safety in having the outage be somebody else's fault, and having that outage effect a huge swath of the economy - "Sorry customer, but it's not just us." Reminds me a bit of the "nobody gets fired for picking IBM" mentality.

tuldia3• 5 years ago

The learned helplessness in the cloud is stupefying, so many outages and downtime that could have been avoided by a competent admin.

spamizbad• 5 years ago

One problem facing on-prem orgs today is the sad state of commodity server hardware; poor quality control, buggy firmware (and vendors who won’t help), BMCs with massive attack surfaces (that have already claimed one VPN provider), and commodity network switches that can push lots of packets but aren’t terribly flexible for people who are pushing more challenging payloads around their DC (video, etc).

"Hyperscalers" like Amazon, Microsoft, Facebook and Google build their own hardware and are able to avoid many of these problems. Unfortunately, none of this stuff is available off the shelf to mere mortals.

There’s a startup trying to fix this problem (Oxide) which I think is launching their racks next year. Will be interesting to see what happens.

jjav• 5 years ago

Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages).

Not even so sure about that. I've had a ton more downtime ("degraded" in AWS speak) with AWS than any self-hosted systems. And that's with more than half my career on self-hosted.

If a major disaster strikes, like the whole rack catching fire and melting everything, then it's true that AWS could recover quicker than self-hosted. But most problems are not of that sort.

artembugara• 5 years ago

Guess whos status page is all green?

darrena092• 5 years ago

This still seems to work for me, replace region with your region:

https://<region>.console.aws.amazon.com/

burai2• 5 years ago

Testes this and it works fine. Thanks!

raffraffraff3• 5 years ago

I don't think you said what you think you said

spresso• 5 years ago

This worked for me too, thanks for the tip!

trevorriles• 5 years ago

I'm still able to log in to the console. Seems to just be the AWS landing page that is down.

tyingq4• 5 years ago

Me too. Though it does seem to be any page for that specific domain. They all return an HTTP 500 (Internal Server Error).

Google "site: aws.amazon.com" and try any of the links.

Sosh1012• 5 years ago

It seems like AWS and Google have had more outages than normal recently. I guess this is to do with COVID disruptions? (remote working and such).

acdha• 5 years ago

This morning I’d bet more on a rushed log4j patch. They use it heavily.

x86_64Ubuntu• 5 years ago

It's kind of terrible that a logging library can lead to such downtime.

stjohnswarts5• 5 years ago

why would remote workers matter? AWS is basically all remote.

Sosh1012• 5 years ago

Because change can be disruptive.

hvgk• 5 years ago

Turning my phone off right fucking now!

ale42• 5 years ago

Is web.archive.org using itself AWS? I tried archiving the error page, but I only get a "Sorry / Job failed" page...

ransom1538• 5 years ago

When will people stop paying for Multi-AZ ops. You basically pay double for this - and - when one region goes down - aws is hosed.

tomcooks2• 5 years ago

Remind me why this is better than a 10$/year VPS

iso16313• 5 years ago

You can't possibly hope to match the reliability of AWS with a tiny VPS

BenjiWiebe2• 5 years ago

Mine has exceeded AWS' uptime in the last 5 years. I think. I wish I had the actual numbers. If not, it's only because I actually didn't care very much about uptime, and did an in-place system upgrade on the live server late in the evening, and when it broke I waited till the next morning to fix it.

Also, the rather rare and brief maintenance window from the provider is always middle of the night for all my customers.

easton• 5 years ago

$10… per year? Where?

rei_ayanami2• 5 years ago

Yeah, lowest DO is $5 per month. Not even Vultr is that low.

FrankSansC• 5 years ago

(from France)

at first:

  We're sorry!
  An error occurred when we tried to process your request. Rest assured, we're already working on the problem and expect to resolve it shortly.

and now:

  This page isn’t working
  aws.amazon.com is currently unable to handle this request.
  HTTP ERROR 503

clavicat• 5 years ago

Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down.

Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?

tyingq4• 5 years ago

I think also just finally getting to the point where they have a tech debt burden comparable to everyone else. Starting fresh has its advantages...for a while.

papito3• 5 years ago

Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.

kaidon• 5 years ago

I was thinking something similar. The FAANG interview has been optimized to punish candidates with actual experience in these problems in favor of those that can solve leetcode problems and regurgitate design architectures from youtube. This filters out a lot of engineers with grit that can actually work through tough real world (IT) problems.

gonzo41• 5 years ago

Those people, the gritty ones who can't make it through the interview hoops are busy integrating companies into the cloud providers. Now they're taking it easy whilst the new kids are getting a first class lesson in what grit is.

aeyes• 5 years ago

I don't even work at FAANG but even we have our own in-house grown compute platforms with layers and layers of abstractions and enormous complexity which were built over several years by hundreds of engineers. These don't resemble public cloud or open source solutions at all, ideas are similar at best. You can have a lot of real world small company knowledge but the moment you walk through the door this experience is worthless, all that counts is how fast you can adapt and how good your understanding of the basics is.

Look at some of the code that was open sourced by Yahoo years ago. Other large engineering organizations today have technology which far exceeds the complexity of what was made public by Yahoo a decade ago. Unfortunately Yahoo is pretty much the only example of a large company open sourcing big portions of their platform code.

ceejayoz• 5 years ago

https://en.wikipedia.org/wiki/Availability_bias

Twitter used to go down so frequently the "fail whale" was a whole cultural thing.

Big AWS outages are rare, but hardly new.

https://www.theregister.com/2017/03/01/aws_s3_outage/

https://arstechnica.com/information-technology/2012/10/amazo...

SamuelAdams• 5 years ago

Yep I remember the same issues with Reddit. They’re fairly stable now, but up until about 2015 they had outages maybe once a month or so.

mschuster91• 5 years ago

I'd say the scale is the problem, not the people.

Every service you mentioned (maybe sans Gmail, the only additions I remember in the last twelve years are a new UI and Hangouts) has bolted on so many features over the last years: AWS was virtual machines + SDNs in the beginning as Amazon only intended to sell spare capacity on their own servers, now it's a global one-stop-shop for everything that can be done on the Internet. Facebook was a social media feed, now it's event coordination, groups, chat, image and media hosting at global scale.

And apparently, no one at these organizations ever thought about re-working their infrastructure with "lessons learned over the last decade" in mind. Every new feature was simply bolted on, on top of an infrastructure that was hardly even envisioned to ever become the scale they are today. And that sort of refactoring costs serious amounts of money and developer time, not to mention that it doesn't make sense to develop new features on a code base that's going to be shut down in a year, so management doesn't approve it out of a fear they will be "out-featured" by a competitor and cannot react (=copy, like Instagram's Stories that were a clear rip-off from Snapchat) in time or that their own PKI/OKR goals and with it their bonus payments won't get hit.

That mindset/scale issue is also why IBM mainframes are still so common, why travel PIRs seem to be stuck in formats over half a century old or why "put CSV files on an FTP server" is the standard on bank transfers... big corporate/government clients pay a shitload of money for virtualized mainframes on new hardware that still can run the 70s-era code and even more money for people able to speak COBOL, because that is still cheaper than the alternative - reworking everything from scratch on a modern foundation, testing data integrity and edge cases, revise interfaces to hundreds or thousands of clients. Hell, even Internet standards have the same problem... we are still using protocols like BGP that have been around since before I was born, and tacked on security only a few years ago after a couple of fat-finger incidents.

Modernization in such entities only tends to happen when laws or regulatory frameworks change, and then it can become a real shitshow for those at the lowest rungs of the IT ladder that have to implement them - simply take same-sex marriages and try to shoehorn them into a database that was labeled for "husband and wife", or trans/inter people with gender data represented by a boolean field.

jboggan• 5 years ago

As the layers of the stack increase in depth the probability that any one piece breaks the system approaches one. There will be a Singularity but it will be a horizontal asymptote instead of a vertical one.

armchairhacker• 5 years ago

maybe we actually have a case for decentralized “web3” networks. At least ones which really need reliability, but can’t can use whatever e.g. airplanes use (telehealth? online exams?)

Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.

colechristensen• 5 years ago

The whole of Amazon isn’t falling, just some services in one region, it is easy enough to build region redundant services but most people don’t because the few hours of downtime per year don’t seem worth it.

You usually just don’t need 100% uptime. “Sorry were closed, come back later” is fine.

stjohnswarts5• 5 years ago

I wonder if we're hearing the full story. Could they be being attacked by foreign hacker farms in Russia and China just to make the US infrastructure look bad?

404mm• 5 years ago

Status page is all green. All must be ok! /s

justintrtle• 5 years ago

I'm not seeing any issues with my services is useast1 or uswest2.

PedroBatista• 5 years ago

Only the most informed and rational companies will see how and why most of this "in the cloud" thing is a bad idea.

Most will accept it as a fact of life and continue to pay for it both directly and indirectly, as long as there's cheap money going around the cloud business can't do wrong.

But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

eterm• 5 years ago

On the contrary, downtime is inevitable and I think I'd rather have downtime when all my competitors do too.

cnity• 5 years ago

How about having uptime while your competitors have downtime?

jeremyjh2• 5 years ago

Do you think people who manage their own data-centers never have outages?

marginalia_nu• 5 years ago

When they do, they at least have the ability to do something about it.

emptybottle4• 5 years ago

Running your own data center is on the far other end of the spectrum. There are numerous options between solely using AWS and building a data center.

papito3• 5 years ago

Try calling "your ops guy" during Thanksgiving or when he is sloshed in a bar when your colocation servers stop responding. No one wants this job anymore. Current technology jobs have already enough unnecessary complexity to deal with uptime as well.

emptybottle4• 5 years ago

On the other hand try calling "AWS" in the same scenario. My money is on sloshed guy coming back with the faster response.

412092• 5 years ago

Would you rather have to fix your own data center, or wait 4 hours.

AWS works 99% of the time,plus it's someone else's problem

iso16313• 5 years ago

In my industry, five nines is the starting level. You're proposing something 1,000 times worse.

emptybottle4• 5 years ago

You don’t need to build your own data center to host outside AWS.

You can fully manage your hardware and software fleet while still paying a host to provide data center service, and networking too.

El_RIDO• 5 years ago

I absolutely prefer to have the option to go into a datacenter in a hurry and actually fix stuff and be in charge, then be stuck with having to wait an indefinite amount of time, twiddle my thumbs, apologize to customers and hope for the best.

While I considered myself a decent Windows NT admin, 20 years ago, the reason I went all in on Linux and FLOSS software at the turn of the century was because I dreaded the powerlessness that these proprietary solutions gave me, when they failed. You'd call the vendors, pored through logs, finding obscure, undocumented error codes, etc. With FLOSS and self-hosting youve got all the information at your fingertips. And if you encountered bugs and you can dig into the sources, patch them, re-compile and fix things - and share them with others and feel that you're contributing to our profession.

When I got the chance to do cloud projects over the last 5 - 10 years, I always took these opportunities, hoping to ensure that I keep up with the tech. At first I was hopeful to offload the boring ops tasks, take our config management to the next level and automate even more. With every platform I got to work with, AWS, Azure and GCP so far, we kept finding bugs in their APIs, outdated or otherwise incorrect documentation and very unpredictable performance, unless you actually can run stacks at scale to average it out (as in more then just 10 - 20 instances or a larger clustered SaaS of the cloud vendor). Many times we also encountered undocumented limits that required requesting support and waiting for approval by the cloud vendor, to get even their mid-sized resources allocated. It all works very nicely on the free-tier-eligible, smallest instances and services, if as slow and high latency as is to be expected, but as soon as you actually need some decently sized storage, compute or bandwidth, it becomes quickly more expensive than what you can put in two or three datacenters for redundancy yourself, if you look at the yearly costs. So far none of the PoCs I was involved with ever got approved long term. They mostly end up as reference implementations for our customers or show case material for the corporate blog. :-(

No thanks, I don't want to go back to feel that powerless as I did on closed systems ever again. Luckily, although many seem to think that cloud is the only option to run at a global scale, you can still provide lots of valuable services on the internet using robust hardware, housed in well connected datacenters.

Jenk• 5 years ago

Have you heard about the "No True Scotsman" logical fallacy?

tyingq4• 5 years ago

But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

As far as I can tell, there really hasn't been an AWS outage where you couldn't have avoided issues with multi-region and some careful selection of which products you're using. Which isn't much different from what you would have to do with your own infrastructure.

Flankk2• 5 years ago

Are you arguing that purchasing and maintaining your own servers is cheaper than using AWS? Let's hear the rationale, then.

mac-chaffee2• 5 years ago

Yes, on-prem is generally cheaper in the long-run.

For example, my company recently purchased $400k worth of hardware that would cost $80k/month if it was all EC2. There's amortized costs of the supporting infra (cooling, power) and ongoing maintenance/depreciation, but paying for the hardware in 5 months can't be beat.

stunt• 5 years ago

I don't use Cloud for their uptime, it's all about their managed services for me.

christophilus3• 5 years ago

I wonder if it's the log4j thing... AWS is built on Java, and I'd be very surprised if log4j isn't used heavily there.

trollied• 5 years ago

I think it is them frantically patching it. There are some Amazon staffers posting in the RCE reddit thread: https://www.reddit.com/r/programming/comments/rcxehp/rce_0da...

Our slack group for this issue is at 3,400 people, haha. It'd be funny if I wasn't one of them. Where do y'all work that has 5000 employees on a single issue?? One that has an arrow under it's name.

zinekeller• 5 years ago

Just went up for me, but it was indeed down earlier.

ff7c11• 5 years ago

${jndi:ldap://AWS Down}?

desdiv• 5 years ago

For those out of the loop: this is a reference to the Log4j RCE story that's also currently on the HN frontpage.

Amazon uses a lot of Java, so the two stories might be actually connected somehow.

[0] https://news.ycombinator.com/item?id=29504755

netsec_burn• 5 years ago

Spoiler: They are. AWS and Amazon both used Log4j exploitable from the homepage (/) of both sites through headers.

Source: me

sydney6• 5 years ago

Perhaps AWS just hasn't paid their networking egress fees to Amazon, you know?

tr33house2• 5 years ago

The worst part of AWS outages is that their status page is always green. It's OK to fail but it's not OK to hide the failure and expect all the customers to stumble upon the failure especially for a company as big as AWS. Not OK!

kreitje• 5 years ago

I get the "We're sorry!" page on this link, but when I go to console.aws.amazon.com it works.

koprulusector• 5 years ago

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work.

Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the good ol’ days (perhaps with some “member berries” https://youtu.be/mPs8-ZBSjok), or you’ve just straight up forgotten what it takes to build and run your own infra from the ground up.

And to those who happen across this comment before posting your own: please contribute something to the discussion beyond the above. Thanks.

technobabbler5• 5 years ago

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem.

Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

christophilus3• 5 years ago

I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.

londons_explore• 5 years ago

I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend.

Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.

ksec3• 5 years ago

making downtime somebody else's problem.

It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

gerbilly6• 5 years ago

Well at least you said please, however:

I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud.

The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

Some of us old timers argued against this only to be shouted down by people who don't even understand TCP/IP.

The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design.

Today we'd just cruft together an MVP, solve only the interesting parts (or more likely the easy parts) and then move on, letting dashboards which lie to cover it up.

Many of us have been taking shots at those 'big boys' since the start of this trend.

Now that because of recent events and we have a chance to be heard you're telling us to be quiet. Why?What are you afraid of?

gtirloni3• 5 years ago

Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.

julianlam2• 5 years ago

That's an interesting position to take, because I imagine if a team of developers decided to invent the internet in a vacuum today, it'd be a hell of a lot more secure than the "let's hope nobody uses this protocol maliciously" attitude prevalent in the early days of the internet.

Not that that is a bad thing, but just something to think about.

nonameiguess• 5 years ago

None of that stuff is legacy. It's just centralized. Economies of scale. Go work for an infrastructure provider, the same way recruiters largely work for recruiting firms instead of all shops having their own in-house.

The fact that the people who could invent the Internet mostly work for a few giants doesn't mean they no longer exist in the current generation.

tuldia3• 5 years ago

This!

Not to mention the amount of garbage in the cloud, the constant learned helplessness that we have to endure even knowing that the situation could have been avoided or even mitigated/solved if the access to the box was possible.

The status-quo of the cloud is uninspiring to say the least...

PragmaticPulp2• 5 years ago

we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

New cloud products are targeted at new companies and services.

If a company has already invested all of the R&D into building self-hosting and they've got it running properly with well-defined and measurable economics, it doesn't really make sense to upend it all and rebuild in the cloud.

But for new services, embracing hosted platforms is a proven accelerant for development. Skip past the solved problems like hosting, get straight to work on the interesting problems your business is trying to solve.

The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did.

Oh please. This is just "back in my day" ranting about "kids these days" and how you think one generation is superior to the other. Give it up.

tjpnz• 5 years ago

Interesting. The infra engineer at my previous company fits your description, yet he was the one pushing for our AWS migration.

zdragnar• 5 years ago

Now that because of recent events and we have a chance to be heard

Is there anything about this particular incident that is new that contributes to your position?

Maybe we run in different tech circles, but I feel like the "on-prem vs. cloud" has been litigated fairly extensively here and elsewhere. In fact, as you said yourself:

"Many of us have been taking shots at those 'big boys' since the start of this trend."

Consultant32452• 5 years ago

I see this as akin to how factory work has changed with modernization. In a modern highly automated factory you need a much smaller number of highly specialized engineers to maintain the robots. In the analogy, these fewer highly specialized engineers are analogous to the ones who need to know understand TCP/IP in depth and now run AWS. The rest of the workers, now replaced by robots, can move on to different productive work.

Spivak• 5 years ago

(Assumedly) much much younger person here who has spent the majority of her career on prem, has lifted multiple shops to AWS, and right now works full time on an all AWS stack.

Your experience with UNIX isn't worthless, but it's worthless to anyone who's working "further up the stack" than you. If you feel like your skills are degrading then you need to find a job somewhere that's actually building infra that shops will build on top of. Your skills aren't day-to-day anymore and that's a good thing. Your generation made all that junk turnkey in the way that you probably think about dealing with Ethernet frames. Taking my first AWS job literally obsoleted all my system's knowledge -- super humbling experience -- it wasn't completely worthless but none of the problems I've spent years working out solutions even existed anymore.

You're confusing people building products with people building infrastructure -- the devops role makes this messy because you're usually "using" infra tools like a dev rather than building them. If you're working on foundational elements then literally nothing has changed. If you're shipping products then absolutely retool around a cloud provider, the infra isn't your secret sauce and if you have to move back on prem because of cost it will be good problems to have.

I mean this completely sincerely, take a job at a company that's providing hosting/services to people. All the old timers with deep deep systems knowledge are gods.

strken• 5 years ago

This conflates two very different things: cut-throat KPI-driven development practices at companies like Amazon, and the entire current generation of software developers. You're taking problems caused by the environment and attributing them to a moral deficiency in the people, which is neither fair nor helpful.

jrochkind1• 5 years ago

Hm, my understanding/memory of the history of internet technologies has a lot more "let's try this and see if it works" than your comment about the role of specing it out and up-front design in the internet would suggest. I think there was actually quite a bit of people in labs just doing things, coordinating informally with people they knew personally in other labs.

Yes, there was also, at various points and times, specs and big-picture thinking, sure.

organic_popcorn• 5 years ago

The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

Losing skills you worked on for years is just part of this space. We are continually building on new abstractions so that we can focus on building solutions.

This really feels like a rant of kids these days. SCRUM doesn’t mean you can’t do upfront design.

justapassenger• 5 years ago

The current generation couldn't invent the internet.

As an old fart myself, this is very cheap argument. Old generation wasn’t superior. Old generation couldn’t invent transistor, how useless we are!

You always move up the stack as tech progress and that’s a good thing.

iso16313• 5 years ago

I use "cloud" services, but I ensure that my systems continue even if they fail. If AWS is down, maybe I can't do some analysis on historic logs until it comes back, that's a known failure

I look at the components of a system and think "what happens if this is turned off // breaks in an unusual way // goes slow", and ensure that the predicted effects are known and acceptable based on the likelihood of failure.

That's the same whether it's an AWS managed DNS service, storage bucket, or a raspberry pi on my desk. As a systems engineer I know what that component does, what happens when it doesn't "do", and ensure the business knows how to work around it when it breaks.

If your business can't cope with an AWS outage (even if it's not as efficient) then you've got problems.

Plan for failure, and it doesn't take you by surprise.

pwinnski• 5 years ago

The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design.

This is a staggering bit of revisionism. As someone who was around at the time, I remember that many RFCs were written based on already-working code. They had some advantages: nobody much cared what they were doing, so they didn't have to answer to multiple levels of management, and they clearly gave almost no thought to security from bad actors, but if you think there aren't people today--including at those very cloud providers you disdain--doing work at least as well-thought-out as those early pioneers, you haven't been paying any attention at all.

I'll stay off your lawn, but maybe take off those rose-colored glasses and stop pretending the past was rosy.

phailhaus• 5 years ago

I mean, because you're wrong? Even with the recent outages, AWS still has far higher uptime and support than anything that someone could cobble together in their own small company. That's the advantage of using cloud infrastructure financed by billions of dollars.

The current generation couldn't invent the internet.

Come on now, this is an overdone, lame argument and I can't believe you're seriously suggesting this. Do you also lament the fact that kids these days can't bind their own books? The point of tools is to be built upon, not to sit around marveling at your own genius. If you build a good tool, the folks that come after you don't have to think about it. That's how you make progress.

technobabbler5• 5 years ago

IMHO these are entirely different skillsets, and a division-of-labor question rather than some sort of insurmountable generation gap. It's not like infrastructural know-how isn't relevant anymore; they just became CDN engineers or DevOps or senior scaling or reliability engineers. Their jobs are no easier than before, especially when you have to consider network traversals across layers of virtualized/containerized services across multiple data centers owned by disparate parties and maintained by different vendors.

Virtualization aside, we haven't abandoned basic infrastructure, but centralized it in the hands of a few huge, expert providers. IMO this is a good thing, and was both necessary and natural as the Web grew to offer more and more opportunities to more new professionals. In detaching HTML from HTTP from ARP, etc. we gave rise to entire new professions like full-time UX (which arguably the Old Guard was never good at beyond a small audience of academics and engineers), or various flavors of front-end developer, or serverless ecosystems.

The Web and associated technologies advanced so quickly it was impractical for a single IT or network department to know all of it anymore, and some of the newer webapps wouldn't have been possible if that same team or company had to also manage all of their own basic infrastructure like it was the 90s still.

Now you can be a front-end only shop, or a UX consultant, or a network engineer who never has to touch HTML, or, or, or... maybe big enterprises always had and could always have all of those in-house, but the division of labor has been a huge boon for small businesses and startups and nonprofits, who just don't have the same resources.

As someone who grew up configuring zmodem and running BBSes and having to (mis)configure NetBIOS all the time, I am so, so glad I never have to worry about OSI layers and such ever again. It's boring to me, and the experts at it are SO much better at it, might as well let them handle it. Especially when the cost of that outsourcing is often like <$100/mo. Well worth both the time and money... and sanity. The division of concerns lets you focus on the things you're either interested in and/or good at.

Our professionals haven't gotten worse. The stack has gotten much deeper.

dimmke• 5 years ago

Thank you for saying this.

ignoramous• 5 years ago

oh boy! Time to rethink this whole cloud thing!

The problem isn't the cloud, the problem is all of the web properties going down all at once, exacerbated by the fact that AWS charges a premium for any viable cross-region / multi-cloud architecture (given its relatively high egress fees). This is discounting the inter-dependence among AWS services themselves, some of which aren't multi-region (afaik).

PragmaticPulp2• 5 years ago

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work.

Engineers like to build stuff. Historically, running your own infrastructure and configuring servers from the ground up were the building blocks of the internet. These new cloud services took away those building blocks and they had the nerve to charge for it.

In the real world, there are more interesting problems to solve than setting up and maintaining your own servers so it isn't really a complaint. It's actually more fun to get the hosting stuff out of the way and focus on the problem at hand.

HN comments are basically notorious for exaggerating the downsides of hosted services while overplaying the ease and benefits of DIY. A lot of the comments here are similar to the famous Dropbox comment thread where users couldn't understand why anyone would want to use Dropbox when they could simply set up a complicated self-hosted rsync contraption to sync their own files.

emptybottle4• 5 years ago

It's not "weasels" scampering out of the wood work.

This is a community of highly technical users expressing legitimate frustration and concern that the biggest player in cloud hosting is

  a) having a service impacting event, and
  b) the status page says everything is up

That's why people talk about the beneits of alternate hosting arrangements. If the status page said what was happening, there would be no need to ask "is AWS down?" the status would be clear.

unethical_ban• 5 years ago

You're correct that the knee jerk reactions are a bit over the top, but what is so interesting is how little the benefits of cloud are discussed here on HN. People who haven't used cloud native services think it's just EC2 and someone else's computer. But the idea of compute and data all being controlled through a single set of APIs, controlled by the same security syntaxes, is pretty incredible. Of course I'm a fan of the idea of on-prem alternatives, but the unified architecture is nice.

doublerabbit• 5 years ago

Serverless is just buzzword for a container on a virtual machine on a server. The Cloud is just a buzzword for a company with a data centre selling virtual servers from other big servers.

You’re either so new to this stuff to have no experience to remember the days before cloud

I do, and it was much more peaceful.

Everything you can do on "in the cloud" you can do in colocation. Which in the long-run is cheaper, more secure* and it's yours! Including the data.

There are caveats, network, component failure. Investment in to these and you can have a pretty king setup.

The cloud enabled magnitude of email spam, brute-forcing, botnets, security vulnerabilities and much more. Operators are lazy don't want to combat it. Providers are bias, then again you can say that about any business.

People flock to the cloud like it's the greatest thing, when all your buying in to is a expensive price-plan for a company who will happily knock you off their service if you somehow brush up the wrong way and then charge you for a closed account.

I do laugh when something goes wrong for FANG. Partly, I'm cynical and want to see the world burn, but it's also these companies exploit their userbase, their staff and the environments resources. When Facebook locked themselves out of their own offices due to the BGP issue, now that's funny.

My colocation costs are:

$5000 covers for a three-year 1Gbit 2u server racking space in two different DCs. Where I have full-control, as many services as I desire, allowed to host what I desire and where the internet space is actually mine. It may be a small cube of internet but I know it's my network, my traffic.

Cloud is whitewash for me, I won't buy in to it. It has it purposes and if your happy with it, fine. But for me Colo for life.

gerbilly6• 5 years ago

Preach it brother!

Also, you should consider renting out space on your cube.

You could even design APIs around the internet protocols so they can create their own DNS records, mailboxes. Wait...

stjohnswarts5• 5 years ago

There is nothing wrong with pointing out the warts. I wonder if you've ever been at the mercy of google/aws and their lack of customer service if you aren't representing the business $$ of a fortune 500 company?

hintymad• 5 years ago

I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up. Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.

Now, how many people think they can do better than Uber?

greedo4• 5 years ago

Most of those issues sound like management/process issues, not technical issues. Our staff can't provision AWS instances without extensive review...

cosmotic• 5 years ago

I don't think anyone is saying they could do better than AWS at making their own infrastructure. I think the complaint is that it's a single point of failure for a huge swath of the internet.

dudeinjapan• 5 years ago

Disagree on two counts.

1) Like any utility, the cloud is predicated on the assumption that it never goes down. If AWS itself goes down, it causes knock-on effects more most mid-sized (i.e. 1 region, multi-AZ) services on their platform, so we can't have it.

2) If we're going to have to accept downtime (1) then yes, it IS time we rethink the cloud--either how we can achieve no downtime, or how we achieve fault-tolerance in some other way.

gerbilly6• 5 years ago

The whole thing reminds me of the early nineties.

There was a whole mature infrastructure built around UNIX and open standards at the time.

We were told to scrap that and replace it with Windows NT and its descendants.

While UNIX wasn't perfect¹, Windows was an order of magnitude worse.

Compared to our UNIX machines, those systems got the internet protocols wrong, were full of security holes and crashed all the time. Objectively worse performance and security was tolerated for a decade or more, just because we all convinced each other that this was the way things were going.

It took Microsoft maybe ten fifteen years to work their problems out. And now there's a Linux env built into Windows.

This cloud thing isn't the end of computing history, just like Windows on the server wasn't the end of history.

Cloud isn't better than on prem. It's just popular right now.

1: https://en.wikipedia.org/wiki/Morris_worm

robomartin• 5 years ago

Ah, the debate between managing your own infrastructure or renting it from someone else. Always a fun debate, often centering around uptime far more so than cost.

I'll go ahead and make what will seem like a weird comparison. I sometimes think about this as I do about Microsoft Office.

What?

Over the years the MS Office suite has reached an asymptotic limit with regards to functionality users would actually be willing to pay for. I'll say that somewhere between 2013 and 2016 the suite had everything most users would need to do the usual stuff one does with Word, Excel, PowerPoint, etc. In other words, it became good enough and maybe even better than good enough.

This is the parallel I see with Linux servers and the Internet infrastructure needed to run a reliable site or service. Not just the OS, but the system as a whole has, over time, approached an asymptotic limit on functionality, reliability, uptime, ease of use, capacity, bandwidth, management, etc. Not sure we are at that limit yet. It sure feels like we must be close. As many have echoed, today one can deploy a range of servers in multiple configurations and achieve extremely good uptimes with as much functionality as needed without necessarily having to go to cloud service providers.

Is it the same? Not sure. I haven't operated at the highest scales in building web services, so I don't know. My gut feeling is that short of things like seriously large DDoS attacks, a self-owned infrastructure could do just as well as something hosted by a cloud provider. With knowledgeable engineers running the show there should not be any issues in setting-up, managing, maintaining and supporting such a structure.

Note that I am not hating on cloud services. No. What I am saying is that because technology tends to get better over time, it is only natural that the alternative approach has gotten better and better to the point that it would be perfectly sensible for a business to consider rolling their own rather than automatically resorting to cloud providers.

hdjjhhvvhga• 5 years ago

I know it is not recommended, but for small (-to-medium) websites you can use DNS failover. I run non-critical services but still I like to have above-average uptime, so I have them configured in two separate physical locations by two different providers. From the point of view of visitors, they had zero downtime in the last 7 years, in spite of several hardware failures (disks, motherboard, power supply - all resolved within an hour). I know it because I tested it multiple times from various location checking what happens if one of them dies.

Of course if you have millions of visitors a day, you'll do better with a more robust setup, but mine costs me €80/month.

not1ofU• 5 years ago

log4j? :-)

FerdSlav• 5 years ago

Have a couple LightSail instances that all went offline around the time that the homepage did and still cannot connect. Everything in the dashboard says they are healthy and online but all attempts to access them (either just access the webpage or SSH) timeout.

Edit: SSH access has been restored if using their web client, everything else still times out

Rudisimo• 5 years ago

Every time I see a situation like this one, I tell myself: "What a wonderful opportunity to go outside and smell the roses".

P.S. A comment from an exhausted engineer to another.

P.S.S. Seriously, I need to get out of the house more.

AWS down again?

Related Stories