S3 was down

http://status.aws.amazon.com/

by rabidonrails • 9 years ago

359 174 9 years ago

status.aws.amazon.com

stuffaandthings• 9 years ago

yup

EB interface is affected too

parthdesai• 9 years ago

Yup, it is down for our site. Some of the images are not being loaded. (US-east-1)

Console says: Failed to load resource: the server responded with a status of 503 (Slow Down)

sotojuan• 9 years ago

Happening here too (NYC, us-east-1).

toddwprice• 9 years ago

Down for us as well. `An error occurred with the message 'Please reduce your request rate.' when writing an object`

zedpm3• 9 years ago

Kudos for Amazon sorting out their status page. The last time this happened, the status page didn't show anything until hours after the outage began. I just noticed this one maybe 15 minutes ago, and 5-10 minutes later they acknowledged the problem.

On the other hand, I'm feeling a strong case of "Not this shit again". Wondering if US-East-1 is more trouble than it's worth, as these outages seem to happen mostly there.

sitharus• 9 years ago

Us-east-1 is where all the shiny new features are released and where most customers are. From experience the other regions are much more reliable.

lightbyte• 9 years ago

I thought us-east-1 had the most issues because it is the oldest region and thus uses the oldest hardware.

linsomniac• 9 years ago

Oregon seems to have most or all of the features. We have a task to move from us-east-1 to oregon, but haven't completed it yet.

Twirrim• 9 years ago

Us-east-1 is where all the shiny new features are released

Not true at all. It'll be among the first for newer features, in as much as the US regions usually are among the first, but it's certainly not the first place code is deployed to.

All services in AWS start out software deployments in smaller regions where the blast radius of a software bug is likely to be smaller.

moduspol• 9 years ago

That may be changing. FIFO SQS queues were only available in the Ohio region until several months later.

Looks like November 17 to June 14.

[1] https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sq...

[2] https://aws.amazon.com/about-aws/whats-new/2017/06/amazon-sq...

rayboy1995• 9 years ago

We just switched to US West Oregon and I'm not looking back.

sorenbs• 9 years ago

It has been tribal knowledge for at least five years to stay away from us-east-1

Oregon sees almost the same speed of adoption of new features with way better reliability.

imrehg• 9 years ago

Definitely seems like that... And we should back off and reduce our request rates, HN, otherwise we'll make it worse!

rficcaglia3• 9 years ago

I tried CTRL+ALT+DEL but it didn't help!! :)

drusepth• 9 years ago

Turning it off and back on again as we speak.

klinskyc2• 9 years ago

Thought I was going crazy. We're seeing 'Please reduce your request rate.' for a lot of our requests as well

rficcaglia3• 9 years ago

Definitely affecting S3 bucket ops, static site hosting on s3, and even things like CloudTrail. Anything that uses S3 East VA my guess. Rather severe impact. Maybe less focus on avocados is what's called for...

crgwbr• 9 years ago

We're see 'Please reduce your request rate' frequently on read and write operations.

maccam912• 9 years ago

"Slow Down"s for me. We started seeing some weird S3 behavior since about 11:45 PST.

CityLims• 9 years ago

Russia? China? Jealous of our cat pics much?!

barillax• 9 years ago

We're seeing the same thing. Been tracking it via Twitter for the last 15 minutes or so: https://twitter.com/search?q=s3%20slow%20down

alexandros• 9 years ago

Lots of people confirming this on Twitter[1], and we're also seeing it at resin.io

[1] https://twitter.com/AlecSanger/status/908402829349572608

fernandopj4• 9 years ago

From status page: (https://status.aws.amazon.com)

11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

fernandopj4• 9 years ago

12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.

fernandopj4• 9 years ago

Now it is CodeCommit too:

12:28 PM PDT We are investigating increased error rates for Git Push, Git Pull and API calls in the US-EAST-1 Region.

rhelsing2• 9 years ago

I'm getting issues as well. Here's to hoping this issue is just a hiccup and not like the last outage.. otherwise, it may be time to seriously consider alternatives. Anyone know of a good solution for a self-hosted s3-esque service?

benwilber0• 9 years ago

it may be time to seriously consider alternatives

they have multi-az support if you use it

xenophonf• 9 years ago

How does one set that up?

rficcaglia3• 9 years ago

true, but some internal AWS features themselves rely on S3 and apparently us-east-1. Those are hosed right now, too. You can't even modify bucket metadata via AWS Console or cli tools. So like checking permissions or logging right now is not possible, or CloudTrail.

zedpm3• 9 years ago

Not exactly. S3 doesn't have AZs at all, it's only split on regions. Further, a bucket can exist in only one region. You can set up cross-region replication, but you of course need to flip the bucket coordinates in all your applications to fail over. It's not nearly as easy as Multi-AZ support in things like RDS.

BillinghamJ3• 9 years ago

S3 is natively cross-AZ. You can’t opt out of that.

Active/active cross-region is possible, but far more complex.

sumitkumar• 9 years ago

good time for a filecoin plug?

toomuchtodo• 9 years ago

https://www.minio.io/

https://www.openstack.org/software/releases/ocata/components...

https://github.com/ceph

If you're serving a static site out of S3, put Cloudflare (ugh) in front of it and enable the feature to serve your cached content when the origin is down [1]. When S3 goes sideways, as everyone is learning, it goes down hard and you're probably not going to be able to make changes to objects, read object metadata, etc.

[1] https://support.cloudflare.com/hc/en-us/articles/200172256-H...

rhelsing2• 9 years ago

minio looks great for my purposes. thanks!

mnutt• 9 years ago

When serving files from s3, if you care about availability it's worth using your own subdomain rather than pointing directly at S3. You can't have a subdomain point to different buckets in different regions directly, but you can point your subdomain to a CDN and have your CDN's origin point to different buckets in different regions.

res0nat0r• 9 years ago

I would turn on bucket replication and sync everything to another region, or move your primary bucket to us-west-2 or something other than us-east-1 and replicate somewhere else.

mulmen• 9 years ago

How are you going to self-host for less money and higher reliability than S3 at anywhere near a competitive price?

xfax2• 9 years ago

Does anyone else have some of their buckets missing from the console list?

ceejayoz2• 9 years ago

I'd fully expect an S3 outage to affect that display.

quickConclusion• 9 years ago

Our buckets are there, just we cannot GET content, with error messages about reducing the rate.

klinskyc2• 9 years ago

Seeing the "reduce your requests" error message on our end

xfax2• 9 years ago

Does anyone else have some of their buckets missing from the console list?

pierrebeaucamp2• 9 years ago

Seems like Github might be affected as well? Can't comment or merge PRs.

pierrebeaucamp2• 9 years ago

nah, works now. Probably unrelated

pm90• 9 years ago

Github isn't hosted on AWS btw.

jwilk• 9 years ago

At least downloads are hosted on S3.

coreyw• 9 years ago

We are seeing increased error rates for GET and HEAD requests. So far we have not been having any issues with PUT requests. We are in us-east-1.

andrewaylett• 9 years ago

In us-east-1. Don't worry me like that! It's definitely not down _globally_ or else I'd have been paged by now...

joombaga• 9 years ago

And we're only getting it for some buckets in us-east-1

hsod• 9 years ago

Ha, I got paged anyway since one of our monitors hits the S3 API which is hosted in us-east-1

gboudrias• 9 years ago

Does this mean your infrastructure is actually dependent on multiple regions? Just curious.

scrollaway9• 9 years ago

Quick tip for those using static (jekyll, hugo...) sites on s3: If you have cloudflare in front of it, you can turn on aggressive html caching by creating a page rule: example.com/* => Cache level => Cache everything

https://support.cloudflare.com/hc/en-us/articles/200172256-H...

This downtime made me realize we weren't caching any html on cloudflare. I just turned it on and all our static sites are doing fine now (and our bills are smaller!).

If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.

kccqzy• 9 years ago

Programmatically purging caches is slow and costs extra. Just use normal cache-busting techniques like a different URL after a deployment.

EDIT: Oops. Misread cloudflare/CloudFront.

hamandcheese• 9 years ago

Thats fine for assets, but not at all ideal for pages.

sgl753• 9 years ago

Changing URLs is not really an option for html pages of a static site.

manigandham• 9 years ago

Cloudflare and all other modern CDNs do this instantly and for free. Only the legacy ones take longer (usually minutes at most) and might charge a small extra.

tokenizerrr• 9 years ago

No it's not? It's free to just purge everything.

teej• 9 years ago

Won't help if your cache is empty and Cloudflare can't access S3. I originally discovered this service outage when an image in cloudflare was returning a 500 error about 40 minutes ago.

scrollaway9• 9 years ago

Yes and no. In outage periods, S3 access is often degraded because throttling is greatly increased, leading to 503s. Meaning if you have that cache layer on top of it, you dodge the issue.

Where it doesn't help is if Cloudflare itself is having issues (all it does is move the weak link one layer up in this case). But that is easy enough to disable (assuming the cloudflare API or dashboard is up).

sfeng• 9 years ago

This is also a good time to get your static files deployed onto Azure or Google Cloud as well, and use your CDN or Cloudflare to failover.

samstave• 9 years ago

This should be a default feature of Cloudflare going forward if they experience massive S3 issues through their customers...? Right?

scrollaway9• 9 years ago

I'm not sure why it would be a default feature. HTML caching should most definitely not be enabled by default.

However, I do agree that it should be possible to turn on without pagerules for these sorts of scenarios.

shallot_router• 9 years ago

Definitely not. If HTML caching were enabled by default, I would've run into all sorts of weird issues with a lot of my Cloudflare-protected sites.

pmlnr2• 9 years ago

Or you can host a mirror at home on a raspberry pi and redirect the dns when you have to.

dna_polymerase2• 9 years ago

Because DNS record propagation is absolutely great for this. Works within seconds!\s

dalanmiller• 9 years ago

Does this require paid subscription to Cloudflare?

scrollaway9• 9 years ago

It does not! It does take up a page rule though. You can have up to three page rules per domain for free.

seanwilson2• 9 years ago

Quick tip for those using static (jekyll, hugo...) sites on s3: If you have cloudflare in front of it, you can turn on aggressive html caching
If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.

You need to be really careful with this.

I'm not sure how you've got things set up but isn't this going to lead to issues unless you have a cache busting or revalidation strategy? You can purge Cloudflare's cache but that's not going to purge the cache on browsers that already visited your site and cached a page already. You might get cases where an old cached HTML page is asking for page resources that don't exist anymore or pages break because the user is seeing the old HTML with the new CSS/JavaScript.

Also, if you can log in to your site, it can potentially cache one user's logged in page content and share it with others if your Cache-Control settings don't include "private".

You could set things up to make sure the browser and then the Cloudflare cache always asks your server first if a page has been updated recently (revalidation) but your server has to be configured right for this (e.g. etag or modified date usage).

Cloudflare is actually meant to have a feature that keeps a static version of your site to show in the event of outages:

https://www.cloudflare.com/always-online/

scrollaway9• 9 years ago

Unfortunately afaik Always Online does not work in the event of S3 errors because S3 is available, merely answering with HTTP error codes.

milankragujevic• 9 years ago

I don't think this is a good idea if you have lots of visits, as Cloudflare might ask you to upgrade to a Business account which is very expensive. It's excellent for small sites and blogs which are static, but for anything visited often, like a Docs site for some popular project, it might get dropped from CF's network. Don't quote me on that, I haven't experienced it, but I heard that they do ask people to upgrade, for example Brian Fritz of omdbapi.com, who had to close down the service to the public because of huge amounts of traffic, and that's just with text, not serving images or video...

temuze• 9 years ago

Now let's see who learned from the last major outage 200 days ago :)

pilom• 9 years ago

This was us! Added CloudFlare in front of every bucket with our own "cdn.example.com" DNS name and changed every reference to s3 in code. Didn't even know S3 had problems today until I saw this on HN. (note: This outage taught us to set up monitoring of the S3 assets separate from the "cdn")

bm1362• 9 years ago

They've fixed the status indicator, it's now a static asset on the status page:

* http://status.aws.amazon.com/images/status1.gif

cordite2• 9 years ago

Can't receive orders :D

ksenzee2• 9 years ago

Nice to see this getting reported on the AWS status page during the actual event, for once.

mi100hael• 9 years ago

It was around 10 mins before anything showed up, actually, and 'status1' for a while after that.

ksenzee2• 9 years ago

For anyone else, I'd say that's letting us know a bit late. For AWS? Huge improvement.

jaysunn• 9 years ago

We are seeing errors in US-EAST:

error: S3ServiceException:Please reduce your request rate.,Status 503,Error SlowDown,Rid

scrollaway9• 9 years ago

https://news.ycombinator.com/item?id=15251374

jaredstenquist• 9 years ago

Unable to deploy with Elastic Beanstalk currently. Internal 500 error. They put up the Blue Diamond icon, so this must be serious.

cordite2• 9 years ago

Can't create a new app / env with elastic beanstalk either.

Internal validation error

baq• 9 years ago

knowing them the datacenter must be flooding with lava.

rabidonrails• 9 years ago

New update from AWS: 12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.

veb• 9 years ago

Is Slack using S3? I can't seem to upload anything.

giarc• 9 years ago

I had a long "Upload Processing" flashing across screen in web app. I restarted tab and image was uploaded but in wrong order.

acejam• 9 years ago

Yes, Slack uses S3 for attachments and shared items.

jsjohnst5• 9 years ago

Yes, they do.

lasermike026• 9 years ago

Oh boy.

1001101• 9 years ago

I can see our us-east-1 buckets, and just pulled a file out. WFM, FWIW. IAM got me a few weeks ago though. :(

Achshar• 9 years ago

I am having trouble on my vultr VPN as well. Any chance it's related?

namidark• 9 years ago

stormcode• 9 years ago

It's effecting heroku deploys that rely on build-packs hosted on S3 (like ruby build-pack).

insomniacity• 9 years ago

That's really disappointing - why isn't something so critical to heroku deployments distributed across multiple AZs and regions?

chosenken3• 9 years ago

Yup, its taking all our stuff down now too. We are working on moving our static site to S3 behind cloudfront, testing it now. Its down.

Our Airflow tasks that use S3, they are all down.

Basically, we are down :)

scrollaway9• 9 years ago

Unrelated to the downtime but what do you use Airflow for? We've been considering using it for our ETL, but I'm even wondering if we could replace our Jenkins instance with it (Jenkins currently only runs cronjobs and one-off tasks; for CI we use travis).

chosenken3• 9 years ago

We use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOps team, running our infrastructure and working on back end systems, so my knowledge of airflow is more high level. I just keep the system up so it can run ;P FYI I work for a startup out of Cincinnati, OH called Astronomer, you can find us at https://www.astronomer.io

Also, we weren't like all down, we just saw lots of time out issues when reading/writing to S3.

rdtsc• 9 years ago

Still 99.99% availability just the averaging time was increased by another 100 years :-)

ceejayoz2• 9 years ago

S3 commits to being 99.9% available (~9 hours a year). You may be thinking of the object durability numbers.

hyperanthony• 9 years ago

Confirmed this is also impacting CodeDeploy in us-east-1, makes sense since it has S3 dependency on revision location.

tuna• 9 years ago

https://www.whoownsmyavailability.com/

ajoy• 9 years ago

Most of the recent issues seems to be in this particular colo (us-east-1), which is also their oldest facility. Maybe its prudent to move resources to other newer colos or pay for multi-zone availability.

twistedpair• 9 years ago

Amen. We moved everything to US-West-2 years ago and never looked back. When we see VA on fire, we just sigh.

yeukhon3• 9 years ago

Well, if your users are in the east cost / London, you want to build something closer without spinning up one in Europe yet, us-east-1 is your choice. So.. YMV.

redthrowaway• 9 years ago

We're getting 503s on s3 assets. Definitely having issues. Cloudfront is still good.

philhartmanonic• 9 years ago

I've gotten a lot of those errors while using the AWS CLI for S3, but it was intermittent. I was using the sync command, so I just kept kicking it back off again as soon as it finished, and eventually got everything up there.

mrb2• 9 years ago

Every time there is an outage at some cloud provider, I enjoy knowing that my site has maintained 100% availability since its launch. I run 3 redundant {name servers, web servers} pairs on 3 VPS hosted at 3 different providers on 3 different continents. Even if the individual providers are available only 98% of the time—7 days of downtime per year—my setup is expected to still provide five nines availability (details: http://blog.zorinaq.com/release-of-hablog-and-new-design/#co...)

Edit: It's not about bragging. It's not about the ROI. I want to (1) experiment & learn, and most importantly (2) show what is possible with very simple technical architectures. HN is the ideal place to "show and tell" these sorts of projects.

devhead3• 9 years ago

to be fair, it's a static html blog; while most of us who are now experiencing issues are running a bit more complicated set of services.

edit: punctuation

mrb2• 9 years ago

Actually dynamic (blog comments.)

But yeah, you are right it's a very, very simple site.

d3ad1ysp0rk• 9 years ago

The ROI on keeping a personal blog up at 5 9s is awful... I understand the desire from a geeky perspective, but it's only really useful for personal enjoyment of the challenge, or bragging.

scott_karana2• 9 years ago

S3 is only three nines, and EC2 is 99.95%.

That's almost certainly lower than combining reputable VPS providers geo-redundantly, and likely a much higher cost for the "convenience".

And it's not just for geeky pride. You learn the most when things break. Far too many funded startups run poorly architected apps in a single AWS EZ, unlike GP.

https://aws.amazon.com/ec2/sla/

https://aws.amazon.com/s3/sla/

293984j29384• 9 years ago

This blog hurts my eyes to look at...

brandon2722• 9 years ago

I guess the part I'm confused about here are the DNS records and DNS pinning. If your zone returns 1.1.1.1 and 2.2.2.2 and 3.3.3.3 as IP addresses to use, and I'm browsing your site while resolving to 1.1.1.1 and 1.1.1.1 goes down -- your site will appear down for me, correct?

My browser won't automatically try 2.2.2.2 or 3.3.3.3, or would it?

fulafel• 9 years ago

Not the op, but won't browsers fall back to the other IPs? In olden days some browsers would not handle multiple IPs per DNS record correctly, hopefully it works now.

ErikDub• 9 years ago

Yes both Chrome and Firefox would automatically try the next IP when I checked a couple of months ago. We used to use this for load balancing, returning the IPs of our servers in random order. Even when one of the servers would go down we wouldn't loose any traffic as browser would just use the next one.

javabean22• 9 years ago

Who cares?

mattrjacobs• 9 years ago

If you want to understand the impact of an S3 outage (or many other kinds) ahead of time, Gremlin (http://gremlininc.com) has built a tool to run these scenarios on your infrastructure.

Happy to answer any questions about the tool, either here or over email.

(Disclaimer: I work for Gremlin)

runesoerensen• 9 years ago

"12:49 PM PDT We are now seeing recovery in the throttle error rates accessing Amazon S3. We have identified the root cause and have taken actions to prevent recurrence."

enkay• 9 years ago

Not sure if related but had unusual SES timeouts and "too many requests" errors around the same time this was posted.

vim_wannabe• 9 years ago

The great thing about AWS is that you shift the burden of responsibility to Amazon.

notyourday3• 9 years ago

https://www.whoownsmyavailability.com/

discordianfish2• 9 years ago

I've just launched https://pub.latency.at/ which give you free Prometheus metrics for various cloud endpoints like S3. Doesn't look too bad right now though.

It's also hosted on S3 but still up and running. (The main service is independent of S3 anyway after setup)

discordianfish2• 9 years ago

I've added a play-with-docker button to give you a prometheus+grafana instance scraping this.

Spun up this few hours ago, will be gone soon but if anyone wants to check it out without waiting for the metrics to trickel in:

http://pwd10-0-33-3-3000.host3.labs.play-with-docker.com/das...

cddotdotslash• 9 years ago

Good time to setup cross region replication with multiple CouldFront origins. I made a tutorial site to show you how: https://spwa.io

Slippery_John• 9 years ago

As always I recommend people practice Chaos Engineering [0] to minimize the impact of such events. Even if the complexity of cross-region failover is to expensive, having some sort of graceful failure is preferable to simply dying. Netflix's Simian Army [1] toolset is particularly useful here, though I don't see an OSS version of Chaos Kong (which simulates the failure of an entire region) sadly.

[0] http://principlesofchaos.org/ [1]: https://github.com/Netflix/SimianArmy/wiki

sgl753• 9 years ago

Seems to be resolved.

daxorid• 9 years ago

Status page claims resolution, but errors are still ongoing for us.

devhead3• 9 years ago

looks like s3 apis are starting to respond correctly now.

hopefully this raises awareness on how important planning for failure is before you make a design choice to introduce a dependency.

ak217• 9 years ago

What do people use for mocking S3 behavior with high fidelity, to test exponential backoff and other resilience strategies in this situation?

I've seen a handful of test libraries but none of them seem to make realistic up-to-date error injection a priority.

silasb• 9 years ago

Maybe minio.

YawningAngel• 9 years ago

We use S3-proxy internally and are pretty happy with it.

rob-olmos• 9 years ago

https://github.com/andrewgaul/s3proxy ? (no hyphen)

bdibs• 9 years ago

Everything seems to be running normally again, at least for me.

vacri• 9 years ago

I wish that AWS would settle on a standard timezone (preferably UTC). Troubleshooting the fallout had me mentally converting their PDT status pages with console graphs in both UTC and 'local browser' time[1]. All for a region located in EDT.

[1] I think they even have a graph somewhere whose axes are in UTC, but whose tooltips are in local browser time, but I can't recall for sure.

RomanPushkin• 9 years ago

It happens pretty often, so I started awesome list: https://github.com/ro31337/awesome-aws-alternatives

Please contribute if you know alternatives.

S3 was down

Related Stories