Ask HN: Is S3 down?

https://news.ycombinator.com/item?id=13755673

by iamdeedubs • 9 years ago

2589 1025 9 years ago

news.ycombinator.com

I'm getting

{ "errorCode" : "InternalError" }

When I attempt to use the AWS Console to view s3

mtdewulf• 9 years ago

Yep, same here.

josephlord2• 9 years ago

Looks like it. Brief panic caused here.

baconomatic• 9 years ago

Seeing it here as well.

jacobevelyn• 9 years ago

We're getting errors indicative of an S3 outage too.

TheVip• 9 years ago

Same problem bro...

mwambua2• 9 years ago

Not sure if it's related... but I'm having issues with Amazon Cloud-drive.

iamdeedubs• 9 years ago

I would assume, I couldn't even share a screen shot of my evidence to my team on slack!

mwambua2• 9 years ago

Lol... It's interesting how much depends on Amazon's infrastructure at the moment.

kangman3• 9 years ago

any one get more info from AWS?

rhelsing2• 9 years ago

Having issues as well.. big issues..

alexleclair• 9 years ago

Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved?

[1] https://status.aws.amazon.com/

tuna-piano2• 9 years ago

Still green now, 8 minutes in.

bpicolo2• 9 years ago

I've had a few non-Amazon providers tell me AWS things are not working in the last 5 minutes, no note from Amazon though.

Nice.

fudged71• 9 years ago

... still green

cheeze• 9 years ago

Just went yellow

Edit: nevermind

matthuggins• 9 years ago

Still green for me

schneidmaster2• 9 years ago

Did it? Still fields of green for me.

jonstaab2• 9 years ago

http://downdetector.com/status/aws-amazon-web-services looks like a reasonable alternative place to check/report downtime.

eicnix3• 9 years ago

Which is coincidently down.

zedpm3• 9 years ago

I just check Twitter, since Amazon's status is always a lie. My personal dashboard is still showing no problems. It's bad enough that the main public status is always green even when there's clearly a problem, but you'd think they could at least make the private status accurate.

jonstaab2• 9 years ago

Gah. It was up 3 minutes ago. Anyone have any suspicion this is another ddos episode? I saw that SO was down last night too: https://twitter.com/StackStatus/status/836450836322516992

vjdhama• 9 years ago

Apparently, that's down too. Sigh.

talawahdotnet3• 9 years ago

There was an alert on the personal health dashboard[1] a second ago, it said S3 Operational issue in us-east-1 but when I tried to view the details it showed an error.

Then I refreshed and the event disappeared altogether.

[1] https://phd.aws.amazon.com/phd/home?region=us-east-1#/dashbo...

socialentp2• 9 years ago

Same here. But it is in the general status dashboard: http://status.aws.amazon.com/

hartleybrody• 9 years ago

I'm seeing green checkmarks across the board, but they just added a notice to the top of the page:

Increased Error Rates
We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

syntheticcdo• 9 years ago

I guess sub-1% to 100% failure rate is technically an "increase".

samstave• 9 years ago

the worst thing is when your system cant handle these "increased error rates" as your control plane cascades failure due to something like this....

The worst "increased error rate" problem I had was when the API was failing and my autoscale system couldnt deal and launched thousands of instances because it couldnt tell when instances were launched (lack of API access) and the instances pummelled the fuck out of all other parts of the system and we basically had to reboot the entire platform....

Luckily, amazon is REALLY forgiving with respect to costs in these (and actually most) circumstance....

tlogan2• 9 years ago

These service health boards are more like advertisement page then actual status of the service.

mwfj• 9 years ago

I guess their bizarre thinking is something along the lines of: "unless we have proof that noone can access the service, we won't change the indicator from green to yellow.

Seriously: I don't understand why you guys stay with AWS.

matwood• 9 years ago

I always joke that if one of those statuses ever went to red, it means the zombie apocalypse has begun.

cperciva4• 9 years ago

The good news is, if Amazon's services are marked as offline, you're allowed to use Amazon Lumberyard to control nuclear power plants.

paulddraper5• 9 years ago

The number of non-green marks is the number of ICBMs currently in flight towards an AWS data center.

chrisan2• 9 years ago

In case anyone wants to see what mysterious the red icon looks like: https://status.aws.amazon.com/images/status3.gif

At best when there are problems (not like now I guess) I will see the "note" green icon https://status.aws.amazon.com/images/status1.gif

krylon• 9 years ago

I've heard (on the Fnord new show on the most recent CCC congress, so take it with a grain of salt and a bucket of humor) that Amazon's TOS are more or less void when a Zombie Apocalypse breaks out.

They had some convoluted but fairly specific wording in their TOS, whoever wrote must have had a lot of fun.

obsurveyor• 9 years ago

Then I guess it has begun, the page is now showing red. I'd put a picture on imgur but it's not loading.

clamprecht• 9 years ago

Calling @jeffbarr

https://news.ycombinator.com/user?id=jeffbarr

adrenalinelol• 9 years ago

Looks like his personal site isn't loading... :)

nostromo• 9 years ago

The red check mark is hosted on S3...

AtheistOfFail4• 9 years ago

Comment of the year.

ak21962• 9 years ago

The truth is stranger than fiction.

https://twitter.com/awscloud/status/836656664635846656

ceejayoz3• 9 years ago

Me right now: https://www.youtube.com/watch?v=_cHa063Mwos

emrekzd• 9 years ago

In December 2015 I received an e-mail with the following subject line from AWS, around 4 am in the morning:

"Amazon EC2 Instance scheduled for retirement"

When I checked the logs it was clear the hardware failed 30 mins before they scheduled it for retirement. EC2 and root device data was gone. The e-mail also said "you may have already lost data".

So I know that Amazon schedules servers for retirement after they already failed, green check doesn't surprise me.

smoodles• 9 years ago

So just as a FYI the reason that probably happened to you is that the underlying host was failing. I am assuming they wanted to give you a window to deal with it but the host croaked before then. I've been dealing w/ AWS for a long long time and I've never seen a maintenance event go early unless the physical hardware actually died...

problems8• 9 years ago

That's completely ridiculous, get some fucking RAID Amazon.

I order drives off newegg directly to my DC and I'm yet to lose data with the cheapest drives available in RAID10.

amaks• 9 years ago

That what happens when cloud provider doesn't support live migration for VMs.

ak21962• 9 years ago

We have a slack emoji for it called greenish. It's the classic AWS green checkmark with an info icon in the bottom. Apparently it's NOT an outage if you don't acknowledge it. It's called alt-uptime.

foxylion5• 9 years ago

I really liked it. But when trying to add it to my HipChat group it failed to upload. Why? S3 outage, what an irony.

nhumrich• 9 years ago

AWS internal lingo calls this the "green-i"

Fiahil• 9 years ago

So, global S3 outage for more than an hour now. Still green, still talking about "US East issue". I'm amazed.

gtsteve4• 9 years ago

It doesn't appear to be global; my app in eu-west-1 appears unaffected.

It's possible that the console won't work however as I believe that's served from us-east-1.

tuna-piano2• 9 years ago

It's crazy how much better the communication (including updates and status pages) is of the companies that rely on AWS than AWS' communication itself.

https://status.heroku.com/incidents/1059

busterarm2• 9 years ago

Also, https://status.pantheon.io

tcsf• 9 years ago

Blake Gentry gave a full accounting of Heroku's response process here - http://www.heavybit.com/library/video/every-minute-counts-co...

Amazon should take notes.

all_usernames4• 9 years ago

I feel for them. Imagine, 40 or 50 different engineering teams all responsible for updating their statuses. At this moment on the AWS status page I see random usage of the red, yellow, and green icons, even though all the status updates are "Increased error rates." What that tells me is that there's no unified communication protocol across the teams, or they're not following it. And just imagine what it's like being on the S3 team right now.

I notice even Cloudflare is starting to have problems serving up pages now.

SnowingXIV2• 9 years ago

Font Awesome went out for me for a bit, but they did a great job getting back up and keeping their users in the loop.

https://status.fortawesome.com/

gordon_freeman• 9 years ago

Looks like they have fixed the issue with their health dashboard now.

From https://status.aws.amazon.com/ : Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

zitterbewegung• 9 years ago

Can't access my website which is hosted on s3 (http://joshuajherman.com).

leesalminen8• 9 years ago

FreshDesk makes extensive use of S3 and it's been unbearably slow to load for the past hour or so. All on S3 requests.

willcodeforfoo• 9 years ago

Uh-oh. Same here... and tried taking a screenshot of pinging s3.amazonaws.com and Slack upload hung.

cwe• 9 years ago

Dropbox using this? Can't seem to sync

rckclmbr• 9 years ago

They used to, but i think they are off it now.

GabeIsman• 9 years ago

Yes.

meddlepal• 9 years ago

Totally fucked.

jgacook• 9 years ago

Yup - dead in the water

c4urself2• 9 years ago

It is, of course the checkmark will stay green throughout this as Amazon doesn't care about actually letting its customers know they have a problem.

danielmorozoff• 9 years ago

Yea seeing the same thing

scrollaway6• 9 years ago

Down in US-East-1 as of 17:40 GMT. Amazon SES also down in US-East-1 as of a few minutes later.

Hearing reports of EBS down as well.

St-Clock• 9 years ago

I confirm for SES being down in US-East-1 :-(

kureikain• 9 years ago

EBS is down for 30% of my servers as well

_callcc• 9 years ago

SES also down.

pmalynin2• 9 years ago

Down from the outside; The internal access (from within EC2) APIs still work.

65827• 9 years ago

Dead as a doornail for me

djb_hackernews• 9 years ago

Anyone else seeing ELB/ALB issues?

asolove• 9 years ago

Yes

redthrowaway• 9 years ago

Yup. Some of our machines in us-east-1e dropped out of the load balancers.

cryreduce• 9 years ago

in the s3 web interface requests to S3 backend end with 503 Service Unavailable

booleanbetrayal2• 9 years ago

S3 and Elastic Beanstalk (S3 dependencies) ... no issues with RDS at the moment

4wmturner• 9 years ago

Is cli working for anyone else? I can't use the console UI, but aws s3 ls and get commands seem to be working fine.

ARolek• 9 years ago

Same here. US East (N. Virginia)

xvolter• 9 years ago

Seeing the same here

twiss• 9 years ago

S3 Ireland (eu-west-1) seems to be doing fine at first sight.

amcrouch• 9 years ago

It appears to be down. My website runs on S3 and my monitors are going nuts!

talawahdotnet3• 9 years ago

Yup it looks so. My console says I have zero buckets, my Lambdas are timing out and https://aws.amazon.com/ returns a big:

"500 The server encountered an error processing your request." message

alanning• 9 years ago

Our lambda functions are also unavailable. Lucky for us we didn't move any of our critical functionality to lambda yet although we are planning to once we have an EC2 backup in place...

ec1096856• 9 years ago

Don't know if share911 is still your product (its about page is down), but you could run your critical services on top of something like PagerDuty to give you the reliability you need.

andrewfong• 9 years ago

Yeah, we host on S3 (US-East-1 I think) with Cloudfront for caching / SSL. Some of our requests get through but it's been intermittent. Lots of 504 Gateway Time-Outs when retrieving CSS, JS.

rawrmaan2• 9 years ago

Incredible how much stuff this affected for me. Opbeat is not loading and I can't even deploy because CircleCI seems to depend on S3 for something and my build is "Queued". This seems so dangerous...

piquadrat2• 9 years ago

Hi, Beni from Opbeat here. Our community site is indeed down unfortunately, but the rest should be unaffected. Where do/did you experience issues?

rawrmaan2• 9 years ago

Hi Beni! My dashboard wasn't loading (the JS assets from cloudfront.net seemed to be throwing an access-control-origin error) but it is now working again. Very slow, but it works. Fortunately nothing critical is happening right now so no worries. Love your service :)

vidarh• 9 years ago

circleci.com won't even load for me.

tbeutel• 9 years ago

I'm getting this using s3cmd:

$ s3cmd ls WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 6 sec...

greenhathacker• 9 years ago

"I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened."

ethanpil2• 9 years ago

What kills me is that their status page still shows nothing is wrong.

https://status.aws.amazon.com/

jeffijoe• 9 years ago

Their status page probably can't refresh because S3 is down.

amazon_throw• 9 years ago

correct.

jontro• 9 years ago

I get this in my aws console.

Increased API Error Rates

09:52 AM PST We are investigating increased error rates in the US-EAST-1 Region.

Event data Event S3 operational issue Status Open Region/AZ us-east-1 Start time February 28, 2017 at 6:51:57 PM UTC+1 End time - Event category Issue

jrs2353• 9 years ago

They don't show it on the status dashboard at https://status.aws.amazon.com/ (at least at the time I originally posted this comment).

But if you go to your personal health dashboard (https://phd.aws.amazon.com/phd/home#/dashboard/open-issues) they report an S3 operational issue event there.

Edit: Mine is reporting region us-east-1

Edit 2: And now the event disappeared from my personal health dashboard too. But we are still experiencing issues. WTH.

chickenfries• 9 years ago

Also seeing the intermittent event on the personal dashboard. Wonderful.

jrs2353• 9 years ago

Apparently they need a special/separate status page for everyone's personal health dashboard too. SMH.

the_arun2• 9 years ago

Same here

kyleblarson• 9 years ago

Any specific regions? us-west-2 seems fine to me. [edit] now I can't see any of my buckets in the web interface.

jrs2353• 9 years ago

Appears to be us-east-1

magic_beans3• 9 years ago

Definitely experiencing non-loading for dependencies hosted on S3 at the moment...

jhaile• 9 years ago

All of our S3 assets are unavailable. Cloudfront is accessible but returning a 504 status with the message: "CloudFront is currently experiencing problems with requesting objects from Amazon S3."

samgranieri• 9 years ago

I'm running into timeouts trying to download elixir packages, and I'm willing to bet this is the cause

stevefram• 9 years ago

Yes, affecting elb in us-east-1 right now. web services are down and unable to bring up the elb screen in the aws console.

chrisan2• 9 years ago

Down for us as well. We have cloudfront in front of some of our s3 buckets and it is responding with

    CloudFront is currently experiencing problems with requesting objects from Amazon S3.

Can I also say I am constantly disappointed by AWS's status page: https://status.aws.amazon.com/ it seems whenever there is an issue this takes a while to update. Sometimes all you see is a green checkmark with a tiny icon saying a note about some issue. Why not make it orange or something. Surely they must have some kind of external monitor on these things that could be integrated here?

edit: Since posting my comment they added a banner of

"Increased Error Rates

We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."

However S3 still shows green and "Service is operating normally"

gtsteve4• 9 years ago

To mitigate the effect of S3 going down I use cross-region replication to replicate objects to another S3 region. If S3 went down in my primary region I could update the app config to write back to the backup region and update the CDN configuration to use the backup region as an origin.

I did that out of paranoia but it turns out this could happen to us. Does that sound like a sensible approach?

Fortunately all my company's stuff is in eu-west-1 which still seems to be fine.

kaishiro2• 9 years ago

I'm certainly not an expert here, but just to make you feel good (if nothing else), this is exactly what we did this morning (Melbourne time). Woke up to a bunch of flailing Lambda funcs on us-east-1. Luckily we're using Apex so cross deploying them to Singapore took all of 30 seconds. We were concerned about API Gateway since it was also sitting in us-east-1 but ended up not being an issue. Realized that redeploying S3 and Lambda across regions can be done in practically no time, but we would have been in trouble had we needed to replicated our APIs in another region. Going to start exploring spec'ing up our existing gateways in swagger to help with this.

kangman3• 9 years ago

what's the SLA for s3?

nomadicactivist2• 9 years ago

Here it is, in typical SLA language... only AWS knows what it means https://aws.amazon.com/s3/sla/

jpalomaki• 9 years ago

There's logic behind the complicated rules. The idea is that you can't cause the calculated availability to go down by generating lots of requests when the service is down.

nomadicactivist2• 9 years ago

And my favourite part: "To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center." -- for a company that has built itself on automation, surely you could automate some bill credits based on the SLA.

bdcravens3• 9 years ago

Many companies don't give out refunds and you have to proactively request. (alas my employer's value prop, in a completely different industry of course)

ucaetano• 9 years ago

Does the AWS Support Center also run on AWS?

kesor• 9 years ago

You can open tickets in support with API calls ... just sayin.

BlackjackCF• 9 years ago

Yes. Have heard confirmation from Amazon that this outage is affecting us-east-1.

agotterer• 9 years ago

Not sure if its related or not (I'll just assume it is), but dockerhub is down as well. Haven't been able to push or pull for the last 15 minutes, some other folks complaining of the same thing.

eggie54• 9 years ago

confirmed: dockerhub is down too

jchmbrln2• 9 years ago

Yeah, bad morning. I started by trying to pull a Docker image. When I couldn't, I tried to push some stuff to S3. Now of course I'm checking HN. :p

serialpreneur• 9 years ago

Can't pull images from either Dockerhub or Quay right now

notheguyouthink3• 9 years ago

Yea my self hosted concourse is down. Feels quite bizarre to have our internal CI so crippled by S3... though i'm still investigating, perhaps the workers are just hung after all of the failed docker pulls.

140am• 9 years ago

https://twitter.com/aws_shd/status/836635812020158464

"Increased API Error Rates - 9:52 AM PST We are investigating increased error rates in the US-EAST-1" "S3 operational issue - us-east-1"

FLGMwt• 9 years ago

Would love to know the threshold for "Increased" -_-

SteveNuts• 9 years ago

"more than one"

JepZ• 9 years ago

Yeah and still a green check mark :D

Rapzid• 9 years ago

Upload failing for me from Sacramento --> us-east-1

notheguyouthink3• 9 years ago

Same here, i mistakingly went to the dashboard first too. Silly me.

0xCMP2• 9 years ago

SES is also down

davidsawyer• 9 years ago

Same

pfela• 9 years ago

Their status page images are hosted on S3, so will be a while for the green checkmarks to update

indytechcook• 9 years ago

My EC2 Servers are also not provisioning.

amasad2• 9 years ago

Same here. I can't find any information about that though.

partisan• 9 years ago

Unable to log into my servers. They are still up and taking traffic, but no contact. Also unable to provision new servers at this late hour.

linsomniac3• 9 years ago

We got timeouts to our bucket address from every location we tried starting at 10:37 Mountain time (GMT-7). Slack uploads started failing, imgur isn't working, and the landing page for the AWS console is showing a 500 error in the image flipper in the middle of the page. The Amazon status page has been all green, but there is a forum post about people having problems at https://forums.aws.amazon.com/thread.jspa?threadID=250319&ts...

In the last couple of minutes that forum post has gone from not existing to 175 views and 9 posts.

newsat134• 9 years ago

Yup, same here. For a moment, I was worried that the UI showed 0 buckets. Gave me a heart attack.

AndyKelley• 9 years ago

This seems like an appropriate time as any... Anyone want to list some competitors to S3? Bonus if it also provides a way to host a static website.

josephlord2• 9 years ago

Rackspace's Cloudfiles. Does support static websites.

leesalminen8• 9 years ago

I use both RS Cloud Files and Google's Cloud Storage. Google's is superior in nearly every way.

The only con is that it is a Google product that could be deprecated at any point in time. But, with all the acquisition stuff happening over at RS, I'd be lying if I said I wasn't worried about them killing of their cloud offering.

alfredxing• 9 years ago

Google Cloud Storage comes to mind. From what I recall they can host static websites as well.

BrandonY2• 9 years ago

Yes, we can. You can host static content in GCS in the same way that you would using S3, or you can Google Cloud Load Balancer to do more complex setups, such as mixing static GCS content and compute URLs on the same domain.

If you're interested: https://cloud.google.com/solutions/web-serving-overview#host... https://cloud.google.com/compute/docs/load-balancing/http/us...

Disclaimer: I work on Google Cloud Storage.

connorshea3• 9 years ago

GitLab Pages [0] is pretty good for static websites IMO, you can use GitLab CI to build more complex sites as well [1].

Disclosure: I work for GitLab

[0]: pages.gitlab.io [1]: https://about.gitlab.com/2016/12/07/building-a-new-gitlab-do...

sotojuan• 9 years ago

GitLab really needs to provide a way to force HTTPS for domains that have it set up. I have to manually link with `https://` everywhere.

trashcan• 9 years ago

Slightly related, FYI all of the package downloads for Gitlab are timing out with a 504 error from Cloudfront ("Cloudfront is having difficulty accessing S3". The registry is also showing an error.

I'm trying to downgrade to an older version because our install is not working but can't get the DEB unfortunately.

danohu• 9 years ago

Backblaze's B2 is an S3 near-clone: https://www.backblaze.com/b2/cloud-storage.html

I don't know about reliability, but it's a fraction of the price of S3.

boulos11• 9 years ago

As I've said previously: I love the Backblaze folks, but B2 is apparently hosted in a single building. So that's not a question of if, but when.

Disclosure: I work on Google Cloud.

newsat134• 9 years ago

Use cloudron.io and install Ghost/Wordpress/Write your own. Just run the server wherever you want.

puddintane• 9 years ago

Interesting that they still host the cloudron.io site on Amazon platform

ping cloudron.io -> 54.192.7.94 -> server-54-192-7-94.dfw3.r.cloudfront.net (Amazon Technologies) [1]

[1] http://whatismyipaddress.com/ip/54.192.7.94

ruchit472• 9 years ago

If its static site, put it anywhere and just cache it with Cloudflare. (turn on the always on feature)

dokument• 9 years ago

You also get your ssl sessions backed up on global webcaches as an added feature.

PuffinBlue2• 9 years ago

To do this you'd create a 'Page Rule' that specifically tells Cloudflare to 'cache everything' for a URL pattern like https://examplesite.tld/* .

You can set a cache invalidation time too.

Always online is a slightly different feature I believe.

solotronics• 9 years ago

IBM Bluemix offers file storage[NFS], block storage[iSCSI], and cloud storage [S3 API, Swift API].

we also offer a large number of boilerplates such as Flask, ASP,net, node.js

we have been making lots of changes lately check us out!

static page deploy guide: https://www.ibm.com/blogs/bluemix/2014/08/deploying-static-w...

https://console.ng.bluemix.net

Entangled• 9 years ago

And Swift.

edgartaor2• 9 years ago

Google Cloud Storage or its Firebase hosting version.

kylemathews• 9 years ago

Netlify is great for hosting static sites. Free for non-commercial sites.

frakkingcylons• 9 years ago

Netlify's status page and pre-rendering service are down due to the S3 outage.

https://twitter.com/netlifystatus/status/836643259053023232

pas• 9 years ago

Dream Objects https://www.dreamhost.com/cloud/storage/

technofide• 9 years ago

An open source project minio.io implements Amazon S3 APIs, and allows you to create your own in-house S3 like infra.

ec1096856• 9 years ago

Any other region of AWS would also have worked around this one.

methurston• 9 years ago

Down for me.

linsomniac3• 9 years ago

The AWS status page is still showing all green but how has a header saying they are investigating increased error rates. https://status.aws.amazon.com/

bkruse2• 9 years ago

This is one of the times that I am glad to be running my own distributed object storage. I'm sure it's not as robust as Amazon, but......

jasonjayr• 9 years ago

What are you using?

bkruse2• 9 years ago

Openstack Swift - easy to get setup and reliable. Takes a little expertise but was well worth the investment in learning it!

rhelsing2• 9 years ago

Is this only affecting US-EAST-1?

learc83• 9 years ago

One of my heroku apps is down, and I cant' log into the heroku dashboard to check it out. I'm guessing this is related.

robeastham• 9 years ago

Me too. This couldn't have come at a worse time. Just launched a new site.

jpmw• 9 years ago

It is, they are downloading slugs from S3.

prab97• 9 years ago

Quora is down too.

jahrichie• 9 years ago

same here, east us seems non-responsive

vinayan33• 9 years ago

Yes it's down for me. I can't access files stored on S3. Also, the service I run is hung trying to store files on S3.

sz4kerto• 9 years ago

Dropbox is down as well. This is going to be gud.

gtrubetskoy2• 9 years ago

they're down to 3 nines

edit: for the year, it only takes 52.57 minutes

jflowers45• 9 years ago

trello and giphy both seemed affected

rrggrr2• 9 years ago

Its unreal watching key web services fall like dominoes. Its too bad the concept of "too big to fail" applies only to large banks and countries.

elastic_church• 9 years ago

"To big [to allow] to fail" is what that term means.

This would extend to a service like amazon actually, where survival of the service would be an extraordinary effort in case this problem lasted for a long time.

The way you imagined it, as 100% uptime, is incorrect.

phil21• 9 years ago

I didn't get that at all from the OP. His comment I'm fairly certain is of the "why the heck are we centralizing the web to 2 or 3 infrastructure companies" for what amounts to a minor amount of convenience.

We've seen this story play out in other industries and it never works out well for average people. It's been astounding for me to watch the pace of this centralizing and who is helping it along.

The tldr; point is that a single service provider should not have the amount of control Amazon does over the Internet. At least that's my take.

I know my opinion on this wildly differs from the HN crowd and SV "decision makers" these days - what is so curious to me is that this is a complete 180 from that same demographic even 10 years ago.

nialv7• 9 years ago

The fact that we even have "key" services is absurd.

balls187• 9 years ago

Thanks for posting this. I've passed this information through my network.

Slack image uploads are hanging.

jsanroman• 9 years ago

It's down :(

headcanon• 9 years ago

Still down for us. S3 seems to be the only thing affected - our mobile apps work fine (EC2 and RDS backend)

obeattie2• 9 years ago

Best thing about incidents like these: post-mortems for systems of this scale are absolutely fascinating. Hopefully they publish one.

magic_beans3• 9 years ago

Have they even acknowledged the mortem..?

malchow2• 9 years ago

Everything is green. There are no tanks in Baghdad center.

obeattie2• 9 years ago

Yes.

"We've identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue."

officelineback3• 9 years ago

Agreed. AWS' postmortems are fascinating.

exodos• 9 years ago

Yah getting the same error in multiple regions as of 1:12 EST

vpeters25• 9 years ago

Apologizes for the "me too" post:

It appears to be impacting gotomeeting, I get this error when trying to start a 12pm meeting here:

CloudFront is currently experiencing problems with requesting objects from Amazon S3.

Edit: ironically, my missed 12pm meeting was an Azure training session.

tech4all• 9 years ago

Yes serious API problems started about 15 minutes ago. Around noon central.

gamache4• 9 years ago

A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You're signing up for the oldest hardware and the most frequent outages.

For legacy customers, it's hard to move regions, but in general, if you have the chance to choose a region other than us-east-1, do that. I had the chance to transition to us-west-2 about 18 months ago and in that time, there have been at least three us-east-1 outages that haven't affected me, counting today's S3 outage.

EDIT: ha, joke's on me. I'm starting to see S3 failures as they affect our CDN. Lovely :/

xbryanx• 9 years ago

I'm getting the same outage in us-west-2 right now.

ngtvspc• 9 years ago

I can confirm this as well.

STRML3• 9 years ago

Seeing it in eu-west-1 as well. Even the dashboard won't load. Shame on AWS for still reporting this as up; what use is a Personal Health Dashboard if it's to AWS's advantage not to report issues?

gamache4• 9 years ago

Huh, I'm not seeing it on my us-west-2 services. Interesting.

firloop• 9 years ago

The dashboard doesn't load, nor does content using the generic S3 url [1], but we're in us-west-2 and it works fine if you use the region specific URL [2]. In practice this means our site on S3/Cloudfront is unaffected.

[1] https://s3.amazonaws.com/restocks.io/robots.txt

[2] https://s3-us-west-2.amazonaws.com/restocks.io/robots.txt

codelitt• 9 years ago

IIRC the console for S3 is global and not region specific even though buckets are.

WaxProlix• 9 years ago

Same here, and it's 100% consistent, not 'increased error rates' but actually just fully down. I'd just stop working but I have a demo this afternoon... the downsides of serverless/cloud architectures, I guess.

Ph4nt0m3• 9 years ago

Same outage in ca-central-1

all_usernames4• 9 years ago

Our services in us-west-2 have been up the whole time.

I think the problem is globally accessible APIs are impacted. As others have noted, if you can use region/AZ-specific hostnames to connect, you can get though to S3.

CloudFront is faithfully serving up our existing files even from buckets in US-East.

notheguyouthink3• 9 years ago

That's a really good point!

traskjd• 9 years ago

Reminds me of an old joke: Why do we host on AWS? Because if it goes down then our customers are so busy worried about themselves being down that they don't even notice that we're down!

nabla9• 9 years ago

Reminds me of an even older joke (from 80's or 90's):

Q: Why computers don't crash at the same time?

A: Because network connections are not fast enough.

(I think we are starting to get there)

jchmbrln2• 9 years ago

Probably valid, though in this case while us-west-1 is still serving my static websites, I can't push at all.

compuguy• 9 years ago

Is us-east-2 (Ohio) any better (minus this aws-wide S3 issue)?

mullen• 9 years ago

us-east-2 is brand new and us-east-1 is the oldest region. Any time there is an issue, it is almost always us-east-1. If possible, I would migrate out of us-east-1.

twistedpair3• 9 years ago

Amen. We setup our company cloud 2 years ago in US-West-2 and have never looked back. No outage to date.

jacquesm6• 9 years ago

If you have a piece of unvarnished wood handy...

nola-radar• 9 years ago

The s3 outage covered all regions.

movedx6• 9 years ago

Really? Even Australia? Can you provide evidence of this so I know for any clients that call me today? :)

EDIT: Found my answer. "Just to stress: this is one S3 region that has become inaccessible, yet web apps are tripping up and vanishing as their backend evaporates away." -- https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...

bischofs• 9 years ago

It shouldnt be technically possible to lose S3 on every region, how did amazon screw this up so bad?

boulos11• 9 years ago

I believe the reports here are misleading: if you try to access your other regions through the default s3.amazonaws.com it apparently routes through us-east first (and fails), but you're "supposed to" always point directly at your chosen region.

Disclosure: I work on Google Cloud (and didn't test this, but some other comment makes that clear).

movedx6• 9 years ago

My advice is: don't keep your eggs in one basket. AZs a localised redundancy, but as Cloud is cheap and plentiful, you should be using two or more regions, at least, to house your solution (if it's important to you.)

EDIT: less arrogant. I need a coffee.

jacquesm6• 9 years ago

Two different vendors if you can afford it. It's a bit of a hassle though.

gamache4• 9 years ago

But now you're talking about added effort. Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much. It's easy to say things like this, but people who can do ops are not cheap and plentiful.

jefe_• 9 years ago

Getting Issues with Citrix Sharefile api (which I've suspected to run in S3). Seems to only be impacting writes in preliminary assessment.

eggie54• 9 years ago

yes, confirmed.

fletom5• 9 years ago

what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page.

they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."

increased error rates? really?

Amazon, everything is on fire. you are not fooling anyone

edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

evtothedev• 9 years ago

The AWS Status page will lie to you: https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-...

fletom5• 9 years ago

I like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.

idlewords2• 9 years ago

@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."

scarlac• 9 years ago

While that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.

paulddraper5• 9 years ago

FYI, in seriousness you can see the fabled red status icon here:

https://status.aws.amazon.com/images/status3.gif

It does exist, apparently.

mpetrovich2• 9 years ago

There are some real gems @Pinboard too: "Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava https://status.aws.amazon.com"

jasoncrawford• 9 years ago

Apparently this is not a joke:

“The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.”

https://twitter.com/awscloud/status/836656664635846656

tlrobinson• 9 years ago

I thought this was a joke, but apparently not too far off:

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

https://twitter.com/awscloud/status/836656664635846656

artursapek• 9 years ago

I don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).

fletom5• 9 years ago

when you start investigating an outage, that is exactly when you should change your checkmark to yellow if not red. if you're as big as AWS there should not be any more than a minute or two between when your service goes down and when you actually update your status page to show that.

hobofan4• 9 years ago

After having seen multiple AWS outages/service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.

gtrubetskoy2• 9 years ago

the non-green icon is probably hosted on s3 (i'm not trying to be funny)

j2kun• 9 years ago

Amazon, everything is on fire. you are not fooling anyone

Fun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.

wildmusings2• 9 years ago

I'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?

stretchwithme• 9 years ago

What we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues.

This would address issues that are only visible from the outside.

laughfactory• 9 years ago

And system monitoring which isn't dependent on itself. Kind of a "duh" kind of thing...

ATsch• 9 years ago

http://outage.report/ does this pretty much, except for the "verified customers" part.

kevin_b_er2• 9 years ago

If this isn't good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don't know what is.

rinze• 9 years ago

edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

So this is what centralization looks like.

skywhopper• 9 years ago

I hear that their process for updating the status page involves S3.

fletom5• 9 years ago

it would appear that you are correct

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates." https://twitter.com/awscloud/status/836656664635846656

knaik942• 9 years ago

So... https://twitter.com/awscloud/status/836656664635846656

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

ckozlowski2• 9 years ago

(Disclaimer: I work for AWS.)

The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that.

Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

http://status.aws.amazon.com/

perlgeek• 9 years ago

Maybe you could encourage your colleagues to host the status page outside of AWS?

danappelxx2• 9 years ago

Could you go more in depth? What does S3 have anything to do with it?

Ph4nt0m3• 9 years ago

For some reason, reading about "believe we understand root cause" made me think of: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

whafro• 9 years ago

"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard."

Yep.

_ao7892• 9 years ago

I love it, like that fixes the problem! ..now fix the REAL problem

twistedpair3• 9 years ago

Why is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.

bseabra• 9 years ago

Same here. We are seeing issues.

afshinmeh2• 9 years ago

yeah, looks like Travis CI is down, too: https://www.traviscistatus.com

freyr• 9 years ago

And Trello

Eyes• 9 years ago

My website is not down.

thadjo2• 9 years ago

same

ethanpil2• 9 years ago

Corporate language is entertaining while we all pull out our hair.

"We are investigating increased error rates for Amazon S3" translates to "We are trying to figure out why our mission critical system for half the internet is completely down for most (including some of our biggest) customers."

dyeje• 9 years ago

Experiencing issues with Elastic Beanstalk and Cloudfront as well.

seibelj• 9 years ago

I cannot eb init or deploy to us-east-1

dfischer• 9 years ago

Wow this is a fun one. I almost pooped my pants when I saw all of our elastic beanstalk architecture disappear. It's so relieving to see it's not our fault and the internet feels our pain. We're in this together boys!

I'm curious how much $ this will lose today for the economy. :)

b01t• 9 years ago

yup

thadjo2• 9 years ago

heroku API is down for me

smmnyc• 9 years ago

They placed their API into maintenance mode: https://status.heroku.com/incidents/1059

Raphmedia2• 9 years ago

Same here in US EAST

gaia• 9 years ago

Sometimes refreshing the console gives this error instead of showing ZERO buckets https://pbs.twimg.com/media/C5xZVGKUYAAXYGj.jpg:large

oaktowner• 9 years ago

Apparently app updates on iOS are failing right now, too. Could be related?

ondrae• 9 years ago

We're on AWS GovCloud and our S3 is all good. GovCloud is its own region.

neom2• 9 years ago

Shame we pay a bazillion dollars for it. Anyway, curious to know how you've thought about mitigating something like this? I worry if it happens to us govcloud users we obviously have much fewer options for redundancy.

bandrami2• 9 years ago

That sound you hear is every legacy hosting company firing up its marketing machine

yichi2• 9 years ago

I've always wondered why people dismiss dedicated hosting without a second thought. It's actually cheaper than AWS if you factor in all of the performance you get.

socialentp2• 9 years ago

Same here. I can log in to the new S3 console UI, but all of my buckets/resources are missing. Same error as you in the old UI. Also unable to connect through the AWS CLI (says, "An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied"). Fun.

STRML3• 9 years ago

It's not just us-east-1! They're being extremely dishonest with the green checkmarks. We can't even load the s3 console for other regions. I would post a screenshot, but Imgur is hosed by this too.

ceejayoz3• 9 years ago

IIRC the Console is mostly hosted out of US-EAST-1. Direct API calls to other S3 regions are likely to work, but it's not surprising the Console's having trouble.

Beacon11• 9 years ago

Works for me, in us-west-2.

joshuahaglund• 9 years ago

you sure? it's not for me.

austinkurpuis• 9 years ago

Same here. Also having trouble publishing to S3 via CLI and API.

dbg314152• 9 years ago

Yes, appears to be.

grimmdude• 9 years ago

Making an already troublesome day worse. Yeehaw

ianopolous• 9 years ago

I'm seeing the same error on eu-west as well.

eggie54• 9 years ago

is this affecting dockerhub for anyone?

sklarsa• 9 years ago

Yep, can't push new images right now

newsat134• 9 years ago

Yes, docker pulls are failing

Globz2• 9 years ago

Yes Trello is down and they are using S3 :(

aarondf• 9 years ago

Yes

tomharrisonjr• 9 years ago

We're seeing queries using Athena against S3 fail in us-east-1

alfg• 9 years ago

Yeah, same here on US-WEST-2. Unable to use the S3 Console, but I can still upload/get content via the API it seems.

mijustin• 9 years ago

Started a list of "things to do when S3 is down."

https://justinjackson.ca/s3/

What else should I add?

mxuribe• 9 years ago

Have a philosophical debate with yourself (just within your mind) as to whether all this interweb/webternet stuff is a worthwhile pursuit...knowing that it would not survive should a comet hit the earth again...or, maybe it will? ;-)

vinayan33• 9 years ago

Yes. Go for a walk and interact with people in the community. You might meet someone interesting and learn something as well!

contingencies4• 9 years ago

My suggestion would be to rediscover the clarity and focus of thinking about systems and code on paper.

remx3• 9 years ago

Migrate all your stuff to Cloudfront.

S3 is not a CDN!

hobofan4• 9 years ago

Or instead of Cloudfront, you could migrate to a service provider that provides meaningful status pages.

beanland• 9 years ago

I'm going to go do my laundry.

coreywstone• 9 years ago

It's winter - hit the slopes!

vacri8• 9 years ago

I'm loving #16 right now...

ganesharul• 9 years ago

Sendgrid, Twilio, Quora is also down. Is this related to S3. Entire world depends on AWS

leesalminen8• 9 years ago

Twilio's outgoing SMS seems to be working fine for me.

yodon• 9 years ago

Sendgrid being down particularly hurts - we need it to notify our users of the problem

jyriand• 9 years ago

That's quite ironic. http://isitdownrightnow.com is also down.

ruchit472• 9 years ago

I have in the middle of thoughts of moving out of AWS and having a dedicated provider as our billing has increased a lot with the scale. The only thing which was holding me was the uptime confidence. Now I feel it's not a bad idea.

renzy• 9 years ago

getting the same...

oculusthrift• 9 years ago

Yes. and Now my HN profile page is down as well.

grzm• 9 years ago

I've been getting sporadic "Bad Gateway" errors on HN for the past half hour or so, not apparently associated with any one feature (such as profile pages). My suspicion is that it's unrelated to anything happening at AWS (HN doesn't have any AWS dependencies, does it?), though maybe everyone checking HN to see what the status is has increased load.

adamveld12• 9 years ago

Where is that "Show HN" that will let me check if a site is affected by an S3 outtage?

ryanmarr2• 9 years ago

My ELBS and EB related instances are also down. I can't even get to Elastic Beanstalk or Load Balancers in the web console. Anyone else having this issue?

JBerryMedX2• 9 years ago

Yes, we're experiencing this issue as well. Are your ELBs also in us-east-1?

caravel• 9 years ago

But wait. Isn't S3 "the cloud". Everyone promised the cloud would never go down, ever. It has infinite uptime and reliability.

Well good thing I have my backups on [some service that happens to also use S3 as a backend].

ceejayoz3• 9 years ago

Everyone promised the cloud would never go down, ever.

No, they didn't. Large portions of AWS's documentation details how you, the developer, are responsible for using their tools to engineer a fault-tolerant, highly available system. Everything goes down. AWS promises varying amounts of nines everywhere, not 100%.

djhworld• 9 years ago

I know your comment is in jest, but Amazon do say their API SLA for S3 is 99.99% available [1]

[1] https://aws.amazon.com/s3/sla/

M4v3R• 9 years ago

Which is less than 5 minutes per month, I guess they've already used that :).

hobofan4• 9 years ago

After more than an hour we are now at ~99.8%, so everybody affected should be able to claim a 10% discount, right?

dragonwriter4• 9 years ago

Isn't S3 "the cloud". Everyone promised the cloud would never go down, ever.

S3 is not the cloud, it's one system running in the cloud. The cloud is not down, S3 and services dependent on (and possibly related to) it are.

One of the selling points of the cloud is that dynamically provisioned services from multiple providers enable engineering fault tolerant systems that are relatively secure against the failure of any single backend. But, yeah, if you are dependent on one infrastructure vendor's service -- particularly running in one particular region/zone -- you are probably better off than running on a single server for reliability against failures, but you aren't anywhere close to immune to failures. I don't think even cloud vendors have been particularly reluctant to make that point.

draw_down3• 9 years ago

Who told you that? I'm not sure anyone ever told you that.

vocatus_gate5• 9 years ago

We used to have a word for "the cloud" back in my day. It was called "outsourcing."

davidcollantes• 9 years ago

Azure is also down. Related?

Theodores• 9 years ago

What, to Yahoo Mail being down? Can't see the connection myself. Even with HN I had the twitter status down page before I reloaded five seconds later, in disbelief... Maybe it is not S3.

framebit• 9 years ago

Wow, amazing to watch stuff go down as this problem ripples out!

vegasje2• 9 years ago

We're in US-West-2 and our ELBs are dropping 5XXs like there's no tomorrow. This is definitely cascading.

soccerdave• 9 years ago

We're in US-West-2 and not seeing any issues. Are the instances behind your ELB trying to access S3 in their application logic?

vegasje2• 9 years ago

Nope. No S3 logic behind the scenes. A few of the ELBs are fine, and a few are not. Seems random.

The EC2 instances themselves are fine, but the affected ELBs are spitting out 500s.

twistedpair3• 9 years ago

All our systems are running just fine in us-west-2 right now.

dageshi• 9 years ago

Huh, I wonder if that's why Origin (EA's Steam competitor) cloud sync just stopped working

travelton• 9 years ago

I can't get to my Amazon Orders page. "There's a problem displaying some of your orders right now."

palad1n• 9 years ago

Yeah, it looks like that was part of the failure.

ryanmarr2• 9 years ago

My EB instances and Load Balancers are also down. I can't even get to load balancers in ec2 web console or to elastic beanstalk in web console. It's been almost an hour now.

rnhmjoj• 9 years ago

Wow, S3 is a much bigger single point of failure than I have imagined. Travis CI, Trello, Docker Hub, ... I can't even install packages because the binary cache of NixOS is down. Love living in the cloud.

krallja• 9 years ago

Trello API is up, so you can use any Trello client except the web client.

stri8ed• 9 years ago

Twilio call recordings appear to be missing as well.

booleandilemma• 9 years ago

Thank you, HN, for giving me the answer the AWS Service Health Dashboard could not.

mabramo• 9 years ago

Thanks for sharing. I overheard someone on my team say that a production user is having problems with our service. The team checked AWS status, but only took notice of the green checkmarks.

Through some dumb luck (and desire to procrastinate a bit), I opened HN and, subsequently, the AWS status page and actually read the US-EAST-1 notification.

HN saves the day.

davewritescode• 9 years ago

Same here, was eating lunch and browsing HN when hipchat started lighting up with customer complaints.

foxylion5• 9 years ago

On thing I learned here. When something seems horribly wrong, check HN first, it may be "global" problem.

nodefortytwo• 9 years ago

Not seeing any errors from eu-west-1

Exuma3• 9 years ago

Yep, currently have over 20,000 people on site seeing no images. Wonderful

verelo• 9 years ago

Years ago when we launched our product i decided to use the US-WEST-2 region as our primary region and to build fail over to US-EAST-1 (Anyone here remember the outage of 2011? Yeah, that was why).

There is something to be said about not being located in the region where everything gets launched first, and where most the customers are not [imo all the benefits of the product, processes and people, but less risk].

Good luck to everyone impacted by this...crappy day.

kopy• 9 years ago

Looks like they store the statuses on S3

Rockastansky2• 9 years ago

Is anyone else also seeing 500 errors for cognito on us-east-1?

kevindong3• 9 years ago

It really is amazing how many web services are dependent on S3. For instance, the Heroku dashboard is currently down for me. Along with all of my services that are on Heroku.

TravelTechGuy• 9 years ago

Same here, but worse. Some of the apps I have hosted on Heroku (including APIs) are showing "Application Error". Like you, tried logging into dashboard and got a Heroku error page.

mcjiggerlog• 9 years ago

Same here :(. Not sure why serving a connection to my dyno depends on S3 being up...

kevindong3• 9 years ago

Turns out that not all Heroku dynos are hosted on US-East. One of my friend's dyno is still up and running great.

olegkikin• 9 years ago

Quora is down.

natashabaker• 9 years ago

For those using Heroku - any workarounds to at least put the app in maintenance mode? It seems their entire platform, including API, is down.

kolemcrae• 9 years ago

Yup. Every single image on my site is hosted there.... eek! :|

paulddraper5• 9 years ago

No, it's not https://status.aws.amazon.com/rss/s3-us-standard.rss

lancefisher• 9 years ago

Looks like the RSS feed hasn't been updated for an hour and a half.

Rockastansky2• 9 years ago

Anybody else seeing 500 errors with AWS Cognito for us-east-1?

They are consistent for me.

bkanber• 9 years ago

I'm having issues with CloudWatch and related monitoring services; eg auto-scaling groups are unable to scale up or down.

etse• 9 years ago

Anyone want to share their real experience with their reliability of Google Cloud Storage.

bandrami2• 9 years ago

And they've just broken four-9's uptime (53 minutes). They must be pretty busy, since they still haven't bothered to acknowledge a problem publicly...

simook• 9 years ago

yes it is.

qaq2• 9 years ago

Cmon but the cloud is magic and very reliable let's move everything to the cloud

Exuma3• 9 years ago

Great, all my billing services on Heroku are turned off. Why do they need S3 access for me to access my web dynos?

I'd rather my app load but appear broken so I can show my own status rather than just shutting down every single app...

buildbuildbuild2• 9 years ago

Same here. S3 as a point of failure makes zero sense for dyno uptime, I'm very frustrated with Heroku. (the dyno was already running, no need to download a new slug in my opinion)

janlukacs• 9 years ago

We're down too with www.paymoapp.com - pretty frustrated that the status page shows everything is up and running.

k__• 9 years ago

1. Announce security vulnerability

2. People push updates as fast as possible to fix security

3. No tests, so everything blows up

chx2• 9 years ago

hex.pm and docker hub are both failing, a lot of projects can't CI because of these. The house of cards we built.

benwilber0• 9 years ago

Notice how Amazon.com itself is unaffected. They're a lot smarter than us.

officelineback3• 9 years ago

Browsing works but some functionality is broken; for example you can't view your order history.

yichi2• 9 years ago

I do recall reading somewhere that Amazon.com isn't actually hosted or fully leveraging on the AWS platform, mostly due to the political struggle between the AWS and the merchant department.

Raphmedia2• 9 years ago

Sales Department: "We require 100% uptime! Can you do that?"

AWS Department: "Wellll, if we don't change the status to red, it's as if we were up all the time!"

p0rkbelly• 9 years ago

There are public talks on Youtube from Amazon.com titled "Drinking our own Champagne" where they say the opposite.

bpicolo2• 9 years ago

The ability is there for any company to take advantage of multiple regions. Takes time and money, but it's doable.

orn• 9 years ago

I'm trying to reach S3 hosted website, no luck

j_shi• 9 years ago

Is there a list of all apps/services that rely on S3?

mierenga• 9 years ago

Would be easier to compile the inverse.

scottlinux• 9 years ago

- launching AMIs (they are stored on s3)

- cloudfront

- ses

- ebs

- rds (snapshots/backups on s3)

- lambda functions (appear to be stored in s3)

perhaps others

Trisell• 9 years ago

US-West(Oregon) just went down as well.

deboflo• 9 years ago

us-west-2 (Oregon) is still up for me from the CLI.

ahmetcetin2• 9 years ago

The same here

happyrock• 9 years ago

Anyone doing a region failover? Any issues so far? We are making plans to flip to us-west-1

whorleater• 9 years ago

We did a failover to us-west-2 and it seemed to work.

ahmetcetin2• 9 years ago

The same here still

garindra• 9 years ago

DockerHub is down as well. DockerHub was down in Oct 2015 because S3 was down in US-EAST. They should have known to cache images in multiple S3 regions since then.

homakov2• 9 years ago

Was just pentesting it, and have some minor result. If you are using S3 browser uploads, make sure parameters you supply to Presign do not contain \n or it can lead to format injection https://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthenti...

Many aws SDK libs don't remove \n for you.

(I hope it wasn't me who broke it lol)

buildbuildbuild2• 9 years ago

"Was just pentesting it" ... hopefully with their permission. Be careful.

homakov2• 9 years ago

It wasnt heavy pentesting, just some params jungling. No way it could cause anything :) still funny coincidence

malchow2• 9 years ago

<% if(service.isUp || true) { renderGreenButton() } %>

dhairya• 9 years ago

region-west2 is also down

zerotolerance• 9 years ago

Do the engineering thing and build fault tolerant systems. Maybe adopt features that have been around since 2015:

https://aws.amazon.com/blogs/aws/new-cross-region-replicatio...

krlkv• 9 years ago

S3 is down? Official Twitter feed is also "unaware" https://twitter.com/awscloud

philliphaydon2• 9 years ago

I'm confused, just logged into work account, and site, and some contract stuff I do. All use S3 / Cloudfront... no errors...

awsoutage• 9 years ago

https://twitter.com/sadserver/status/818937064552951809

Interesting tweet from last month.

JBerryMedX2• 9 years ago

My company's ELBs in us-east-1 are experiencing massive amounts of latency causing the instances to be marked unhealthy.

atombender• 9 years ago

It's interesting to note the cascading effects. For example, I was immediately hit by three problems:

* Slack file sharing no longer works, hangs forever (no way to hide the permanently rolling progress bar except quitting)

* Github.com file uploads (e.g. dropping files into a Github issue) don't work.

* Imgur.com is completely down.

* Docker Hub seems to be unavailable. Can't pull/push images.

Ph4nt0m3• 9 years ago

* Hipchat file sharing no longer works, hangs forever * CircleCI cannot access artifacts

mcphilip2• 9 years ago

A doctor's office that's unable to process patients due to the outage:

https://mobile.twitter.com/drjincali/status/8366578638879989...

cookiecaper• 9 years ago

I mean, that's not really AWS's problem, is it? Outages happen. If you have a mission-critical service like health care, you really shouldn't write systems with single points of failure like this, especially not systems that depend on something consumer-grade like S3.

This appears to be a normal doctor's office where there are routine appointments. Emergencies would be referred to the ER anyway. And while I obviously don't know the details of how his office is run, you'd think that you could get by on a pen-and-paper fallback to manage the office. Maybe that's an advantage to keeping experienced office staff on board.

chrisper• 9 years ago

Heroku has issues too.

cylinder• 9 years ago

AWS - too big to fail.

azernik• 9 years ago

My own service was running just fine, until I tried to push out a (fairly critical) bugfix. Sadly, my deploy procedure relies on my build server pushing docker images to quay.io, and prod servers pulling them back down from quay.io, and quay.io hosts their Docker registry in S3. Time to make some apologies and excuses to my users...

jasonsync• 9 years ago

We're still up and running (Sync.com) if you need to share some files.

pweissbrod• 9 years ago

i was unable to purchase cat food at my local pet store with a credit card because their POS software was down

monksy• 9 years ago

On the upside.. no one is stealing more data from the CloudPets thing we were talking about earlier today.

kevin2r• 9 years ago

I was also trying to open gotomeeting web client, no luck.

smackfu• 9 years ago

Zoom was down too.

dgelks• 9 years ago

Getting the same error on the GUI but the aws cli and sdk seem to be working fine (our site is still up too)

hyperanthony• 9 years ago

Experiencing issues with S3 and ELB for over an hour now.

tzaman• 9 years ago

It appears Docker Hub is hosted on S3 as well, none of the official images can be pulled.

nickstefan12• 9 years ago

bets as to the cause? internal DDoS against their dynamo clusters backing s3? DNS issues between amazon's services?

spacecadets• 9 years ago

There goes my Trello to do list. Now I'm lost. Oh well.

knaik942• 9 years ago

"We’re continuing to work to remediate the availability issues for Amazon S3 in US-EAST-1. AWS services and customer applications depending on S3 will continue to experience high error rates as we are actively working to remediate the errors in Amazon S3." Last Update 1:54pmEST

It shows up in the event log now too.

soheil4• 9 years ago

I try not to put all my eggs in one basket, that's why for images I use imgur. They have a great API and it's 100% free. There is a handy ruby gem [1] which takes a user uploaded image and sticks it on imgur and returns its URL with dimensions etc. On top of that you don't have to pay for traffic to those assets.

[1] https://github.com/soheil/imgur

arrty88• 9 years ago

imgur doesn't use S3 behind the scenes?

BoorishBears• 9 years ago

It uses it, and it's completely down right now.

bastawhiz• 9 years ago

Using imgur over a service that you pay for (with an SLA) is, as my college CS professor used to call it, "skating on thin ice".

Narretz• 9 years ago

Well, imgur.com is currently down, too. I assume they are still using AWS: http://blog.imgur.com/2013/06/04/tech-tuesday-our-technology...

K2L8M11N2• 9 years ago

Imgur's down right now as well (except direct links to images) and it's not lossless (they resave images).

SubiculumCode3• 9 years ago

I admire your sword of sarcasm +1

soheil4• 9 years ago

draw_down3• 9 years ago

You guys are so funny sometimes.

aytekin• 9 years ago

Never depend your business on a single provider.

maxerickson2• 9 years ago

Coincidence?

https://twitter.com/homakov/status/836649802842591232

I've been fuzzing S3 parameters last couple hours...

And now it's down.

remx3• 9 years ago

Post about S3 not being a CDN hosted on an S3-powered blog:

https://jdorfman.posthaven.com/medium-bitcoin-660x493-dot-jp...

The irony

maccard• 9 years ago

My fire tv stick is totally unusable too. Seems I can't access any applications (even Lodi or Netflix)

mystcb• 9 years ago

Been unreliability informed about 1 hour ETA for a fix. fingers crossed

khamoud• 9 years ago

I think this explains why the docker registry is down as well.

http://status.docker.com/

sweddle• 9 years ago

yeah still all green in AWS status.... maybe their red and yellow icons are kept on S3. :-)))

camperman• 9 years ago

Many a true word is spoken in jest - that's exactly what the problem is :)

BrandonM• 9 years ago

Additionally, Zendesk is apparently failing to process new tickets, so our users can't report the errors they're encountering.

jliptzin• 9 years ago

Thank god I checked HN. I was driving myself crazy last half hour debugging a change to S3 uploads that I JUST pushed to production. Reminds me of the time my dad had an electrician come to work on something minor in his house. Suddenly power went out to the whole house, electrician couldn't figure out why for hours. Finally they realized this was the big east coast blackout!

stevehawk• 9 years ago

irc.freenode.net / ##aws (must be registered with nickserv to join)

outage first reported around 11:35CST.

Havoc• 9 years ago

Disadvantage of being in the detail I guess. My thinking was Imgur seems broken today >>> Something major on the intertubes must be fk'd.

TeMPOraL2• 9 years ago

Precisely how I discovered it. Imgur down. Imgur is almost like a piece of critical Internet infrastructure. That + some other site misbehaving tipped me off that something very wrong is happening...

skiril• 9 years ago

they already admitted it: https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...

draw_down3• 9 years ago

"bit-barn bods" is some of the worst alliteration I've ever seen.

edcoffin• 9 years ago

If you've ever felt the AWS health dashboard was dubious before now...

dorianm2• 9 years ago

Heroku API/Dashboard is down, Bugsnag is down, etc.

mrep2• 9 years ago

quite ironic that 'isitdown.com' is also down

Humdeee• 9 years ago

you should always try 'isisitdown.com/down' first

notriddle• 9 years ago

isup.me is fine.

dorianm2• 9 years ago

Heroku API/Dashboard is down, Bugsnag is down, etc.

soheil4• 9 years ago

wow even services like Intercom are affected, I can't see who is on my website right now.

machinarium• 9 years ago

Omg I wish I googled this earlier. Wasted hours debugging :(

SnowingXIV2• 9 years ago

Yep. I was wondering why my heroku deploys were hanging so I was looking into every possible issue on my end.

And then I see the news.

manmal• 9 years ago

Our static site hosted on eu-central-1 is still up: http://www.creativepragmatics.com.s3-website.eu-central-1.am...

thepumpkin1979• 9 years ago

is it just us-east-1? could it be prevented by using a different region?

JustinAiken• 9 years ago

For our app, both S3 and SES have been completely down in us-east-1 for hours now.

SubiculumCode3• 9 years ago

Mr. Robot live shoot? :)

slack file services down too

shifted316• 9 years ago

The status page is stored in s3. It can't be updated. The page you see is cached in cloudfront. They are working on updating the status page.

devenrl• 9 years ago

Sorry, my simplistic mind is only thinking this right now:

http://alessandrobender.com.br/wp-content/uploads/2015/07/fi...

ignaces2• 9 years ago

Heroku apps are also down because of this!

SubiculumCode3• 9 years ago

news.ycombinator.com seems really slow right now. s3 dependencies?

johngalt2• 9 years ago

Sysadmin: I can forgive outages, but falsely reporting 'up' when you're obviously down is a heinous transgression.

Somewhere a sysadmin is having to explain to a mildly technical manager that AWS services are down and affecting business critical services. That manager will be chewing out the tech because the status site shows everything is green. Dishonest metrics are worse than bad metrics for this exact reason.

Any sysadmin who wasn't born yesterday knows that service metrics are gamed relentlessly by providers. Bluntly there aren't many of us, and we talk. Message to all providers: sysadmins losing confidence in your outage reporting has a larger impact than you think. Because we will be the ones called to the carpet to explain why <services> are down when <provider> is lying about being up.

dragonwriter4• 9 years ago

Because we [sysadmins] will be the ones called to the carpet to explain why <services> are down when <provider> is lying about being up.

But isn't that the whole point of lying: to the less technical manager (often the only person whose view matters at major customers), the status board saying "up" means the problem is the sysadmins, not the vendor.

mjcl• 9 years ago

That works in the vendor's favor in the short term, but can screw them in the long term because you get staff who go the extra mile to avoid the vendor in the future, including structuring requirements to avoid them.

For example, by experience and gossip I know Wind stream has awful reliability, but they handwave that away. By including a requirement I knew they couldn't meet (dynamic E911), they were knocked out of a 200 site VoIP RFP early.

MaxfordAndSons• 9 years ago

It's unbelievable that the status page is still showing green checkmarks, almost what, 2 hours into the outage?

edit: oh, it is actually because of the outage! So if they can't get a fresh read on the service status from s3, they just optimistically assume it's green... even though the service failing to provide said read... is one of the services they're optimistically showing as green XD

purplecones• 9 years ago

Hey can't change it due to the S3 issue. See their twitter post: https://twitter.com/awscloud/status/836656664635846656

sitkack• 9 years ago

Same related flaw as Three Mile Island. Fail closed and measure the output, not the intent.

carbocation• 9 years ago

People were joking about this but it turns out to be true: they host the status icons on their service: https://twitter.com/awscloud/status/836656664635846656

paulddraper5• 9 years ago

The icons aren't hosted there (or if they are, they are cached). https://status.aws.amazon.com/images/status3.gif

The status information is hosted there.

kangman3• 9 years ago

ins3ption

swearfu2• 9 years ago

Saw that too, sounds like a convenient excuse for being caught in a lie.

AWS Employee #1: Hey, people are catching on that our status page isn't accurate

AWS Employee #2: Tell them it's cause of S3

buryat• 9 years ago

Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard.

devy3• 9 years ago

Due to HN's flaky Cloudflare 503 Bad Gateway error, I noticed that Cloudflare is also being affected by S3 being down in a similar but subtle way. See their status page's broken logo on the upper left hand corner.[1] It was actually directly linking to a S3 URL: https://s3.amazonaws.com/statuspage-production/pages-transac...

[1] https://www.cloudflarestatus.com/

taobility• 9 years ago

if that simple, why the text desc for Details also didn't reflect the incident?

jaequery• 9 years ago

Is there any service that distributes your files to multiple cloud services at the same time? With this recent S3 outage, I'm now feeling uneasy to store files on S3 for mission critical apps.

jpwgarrison2• 9 years ago

I like how HN (and others) handle this - there should be a static link to a 3rd party source, like a twitter feed, at the top of any status page.

sirn• 9 years ago

While status icon being hosted on S3 is funny, I think it's more likely that it's not the icon itself that caused the status page to not getting updated, but rather the fault of service information (say, a JSON file) that used to generate the status page that was stored on S3. The banner could probably be configured locally, so they choose to update that for the time being (e.g. while moving the status bucket somewhere else).

rdiddly• 9 years ago

It's not a lie, it's an "alternative fact" about how totally like awesome AWS is!

rabidonrails2• 9 years ago

Just commenting here because hopefully people can see: AWS status page updated: 1:44 CST

paulddraper5• 9 years ago

Hurry, look now, so you can tell your grandchildren!!!

Greenish ELB, RDS.

Yellow EC2, Lambda.

Red S3, Auto Scaling.

EDIT: A few dozen services in us-east-1 are down/degraded.

jnordwick2• 9 years ago

Any is always the manager that is the bad guy in these scenarios? Haven't we grown up yet?

jnordwick2• 9 years ago

Why is always the manager that is the bad guy in these scenarios? Haven't we grown up yet?

johngalt2• 9 years ago

The manager is not the bad guy. They are doing everything they should do in the scenario I presented. Checking into an outage affecting a critical system. Criticizing the sysadmin's findings based on the evidence that Amazon's status page disagrees. I don't expect a non-technical party to believe me over Amazon.

The bad guys are the providers who report false positives to preserve metrics.

rdtsc• 9 years ago

but falsely reporting 'up' when you're obviously down is a heinous transgression.

When SLA's are in play and so are job performance scores and bonuses there is probably a strong incentive to fudge numbers. It can be done officially ("Ah but sub-chapter 3 of chapter X in the fine print explains this wasn't technically an outage") or unofficially.

vocatus_gate5• 9 years ago

When I worked in Antarctica any outage affecting users that lasted over 50 minutes was considered an official "outage" and had to be reported to mission command. So of course ALL maintenance was rolled back/backed out if it came anywhere even close to 50 minutes, just so we wouldn't have to fill out the stupid outage paperwork.

primitivesuave2• 9 years ago

Thank you for the insight. Could you and/or any sysadmin on here elaborate on what a "nail in the coffin" situation might look like? For example, is this current outage with inaccurate status updates enough to seriously consider migrating to another CDN provider? If so, which one would you migrate to?

i336_2• 9 years ago

Disclaimer, not a job-toting sysadmin quite yet, but here's my 2¢:

- Architectural SPOFs (single points of failure) need to be carefully weighed up in any design, and "ALL our files are on $single_provider" is one such huge red flag. Unfortunately these considerations are all too frequently drowned out by the ease of going with the least path of resistance.

For example GitHub occasionally goes down, which breaks a remarkable amount of infrastructure: a huge number of people don't know how to use Git, do full clones from scratch each time, and have no idea how to work without a server (even though Git is built to work locally); CI systems tend to want to do green-field rebuilds, so start out with empty directory trees and need to do full clones each build (I'm not sure if any CI systems come with out-of-the-box Git caching); GH-powered authentication systems fall apart; etc. Kinda crazy, scary and really annoying, but yeah.

In terms of "nail in the coffin", that depends on a lot of factors, including a subjective analysis of how much local catastrophe was caused by the incident; subjective opinions about the provider's reaction to the issue, what they'll do to mitigate it, perhaps how transparent they are about it; etc.

Ultimately, the Internet likes to pretend that AWS and cloud computing is basically rock-solid. Unfortunately it's not, and stuff goes down. There were some truly redundant architecture experiments in the 80s (for example, the Tandem Nonstop Computer, one of which was recently noted to have been running continuously for 24 years: https://news.ycombinator.com/item?id=13514909) but x86 never really went there, and superscalar computing is built on a sped-up version of the same ideas that connect desktop computers together, so while there are lots of architectural optical illusions, well, stuff falls apart.

- Everyone in this thread is talking about Google Compute Engine, but it really depends on your usage patterns and requirements. GCE is pretty much the single major competitor to AWS, although the infrastructure is _completely_ different - different tools, different APIs, different pricing infrastructure. The problem is that it's not like like MySQL vs PostgreSQL or Ubuntu vs Debian; it's like SQL vs Redis, or Linux vs BSD. Both work great, but you basically have to do twice the integration work, and map things manually. With this said, if you don't have particularly high resource usage, VPS or dedicated hosting may actually work out more cost-effectively.

TL;DR: you go back to the SPOF problem, where _you_ have to foot the technical debt for the reliability level you want. Yay.

discreditable• 9 years ago

This is why I always set up my own monitoring for services in addition to the provider's status page. Simple SmokePing graphs have saved me a ton of time when it comes to troubleshooting provider outages. It especially helps when I can show them exactly when there are problems.

dang2• 9 years ago

All: I hate to ask this, but HN's poor little single-core server process is getting hammered and steam is coming out its ears. If you don't plan to post anything, would you mind logging out? Then we can serve you from cache. Cached pages are updated frequently so you won't miss anything. And please do log back in later.

(Yes it sucks and yes we're working on fixing it. We hate slow software too!)

rbirkby• 9 years ago

Can anyone get Alexa to play music? Is this related?

huac• 9 years ago

Canvas (the educational software platform) is down, and my friends/students are in bad shape now. 'sso.canvaslms.com' returns 504, assume from this S3 outage.

valine• 9 years ago

Apple's iCloud is having issues too, probably stemming from AWS. Ironically Apple's status page has been updated to reflect the issue while Amazon's page still shows all green. https://www.apple.com/support/systemstatus/

brandon272• 9 years ago

I can't stream music from my iCloud library.

chiph• 9 years ago

I'm seeing problems with Kindle downloads.

geerlingguy• 9 years ago

From Amazon: https://twitter.com/awscloud/status/836656664635846656

    The dashboard not changing color is related to S3 issue.
    See the banner at the top of the dashboard for updates.

So it's not just a joke... S3 being down actually breaks its own status page!

etler• 9 years ago

For this kind of page it might be best for them to use a data URI image to remove as many external resources as possible.

Symbiote• 9 years ago

Unicode characters would work fine, and be even smaller.

Warning sign, octagonal sign, no Entry (all filtered by HN).

There are plenty of possibilities.

JangoSteve• 9 years ago

I was thinking they should host the little green check mark icons on s3.

DocK• 9 years ago

Even Kindle books aren't able to be served; download attempts hang.

ayemeng• 9 years ago

Funny, status page is incorrect because of S3

https://twitter.com/awscloud/status/836656664635846656

edgartaor2• 9 years ago

QUESTION. There could be data lost from this failure?

LeonM• 9 years ago

Just posted on their Twitter:

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

dangle• 9 years ago

AWS is updating twitter here. No red icons on the status page IS an AWS issue:

https://twitter.com/awscloud/status/836656664635846656

rebornix• 9 years ago

Alexa smart home component stopped working, if you try to reinstall the Alexa app on your phone, you'll find that you can't even login anymore.

eggie54• 9 years ago

all of your jokes about the dashboard not turning red b/c the icon is hosted on US EAST are true:

Amazon Web Services‏Verified account @awscloud 8m8 minutes ago More The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.

koolba4• 9 years ago

I bet the outage is related to the new color coded CloudWatch metrics: https://twitter.com/awscloud/status/836630468778864640

As part of the release they wanted to make sure everybody gets a chance to see "red" metrics.

jsperson• 9 years ago

Just finished reading The Everything Store... I bet a "?" email went out.

ajmarsh• 9 years ago

via AWS twitter account "The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

axg• 9 years ago

Amazon: The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.

https://twitter.com/awscloud/status/836656664635846656

DenisM4• 9 years ago

Now might be a good time to ponder a lasting solution. Clearly, we cannot trust AWS, or any other single provider, to stay up. What is the shortest, quickest to implement, path to actual high availability?

You would have to host your own software which can also fail, but then at least you could do something about it. For example, you could avoid changing things during critical times of your own business (e.g. a tradeshow), which is something no standard provider could do. You could also dial down consistency for the sake of availability, e.g. keep a lot of copies around even if some of them are often stale - more often than not this would work well enough for images.

alexbilbie• 9 years ago

High availability is improved by hosting in multiple AWS regions.

S3 offers alternative region replication functionality and you can use Cloudfront of another CDN to load balance between buckets

Cshelton• 9 years ago

Yup, all of our fail overs worked flawlessly. You wouldn't even be able to tell if you didn't know.

DenisM4• 9 years ago

But do you always serve files from S3? Wasn't the entire S3 down today? Or was it just some regions? I couldn't even connect to s3.amazonaws.com ...

temuze• 9 years ago

How can you use Cloudfront to point to multiple buckets? I'm pretty sure each origin needs a path and Cloudfront serves the first path that matches.

z3t42• 9 years ago

I use a master and four mirror slaves. Then use round robin for reads. I used to have three slaves, but when two of them was down at the same time I got nervous and set up one more.

DenisM4• 9 years ago

How does that work with S3? Your master and slaves are presumably just SQL databases, right? Or?

vacri8• 9 years ago

Quickest path? Use AWS (or similar). Outages happen. The more robust you want to be against outage, the more prep work you have to do. In order to have 'no outage', you have to do an incredible amount of work. There is no quick path.

myth_drannon• 9 years ago

Looks like SoundCloud is hosting the tracks on S3 , can't program without my music...

yuxt• 9 years ago

this still works for me https://musicforprogramming.net

bas• 9 years ago

"Amazon CloudFront: Service is operating normally"

This is bullshit if you're using an S3 origin in your distribution.

jotaen• 9 years ago

Does anyone have trouble with the Cloud Console? The JS assets for the CloudFront dashboards seem broken, so unfortunately it’s not possible to change the behaviours of the Distributions (e.g. to point them to another bucket)

trakl• 9 years ago

http://venturebeat.com/2017/02/28/aws-is-investigating-s3-is...

https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...

murphy522• 9 years ago

We host with TierPoint and they are reporting a massive DDOS attack

zedpm3• 9 years ago

Can anyone comment on mitigating issues like this with S3 Cross-region replication? I'm reading up on it now while one of my services is dead in the water.

thraway2016• 9 years ago

The only appropriate comment is that this issue is affecting all of our buckets, both in us-west and us-east. Replicating to another region would yield no useful benefits in this specific failure scenario.

nolite• 9 years ago

Can't agree with this. Buckets in eu-west-1 are fine

zedpm3• 9 years ago

I did some digging and experimentation, and so far it looks like you could keep a backup bucket in another region and use bidirectional replication [0] to keep the two buckets in sync. If something like this happened again, you could point your app(s) at the bucket in another region and keep accepting data. The objects would eventually get replicated back to the original bucket, and you could cut over again when service was restored. There does seem to be an appreciable replication lag, so you could run into problems during your cut where some objects had not yet been replicated, but your app ought to handle things like that gracefully anyway.

[0] https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html

murphy522• 9 years ago

TierPoint, a large hosting service, is reporting a massive DDOS attack on their infrastructure.

ignaces2• 9 years ago

source?

xtus• 9 years ago

After few requests timed out, started to dig a bit. The CNAME for a bucket endpoint was pointing to s3-1-w.amazonaws.com with a TTL of at least an other 5600 secods. Doing a full trace was giving back a new s3-3-w.amazonaws.com The IP related to s3-1-w was/is timing out, all cool instead for the s3-3-w.

Animats8• 9 years ago

Amazon outage just reported on NBC News.[1]

AMZN stock down $3.45 (0.41%).

[1] http://www.nbcchicago.com/news/national-international/Amazon...

Animats8• 9 years ago

Just reported on USA Today.[1]

[1] http://www.usatoday.com/story/tech/news/2017/02/28/amazons-c...

Animats8• 9 years ago

Fox News now has the story.[1]

[1] http://www.fox5ny.com/news/238689310-story

BWStearns2• 9 years ago

It's been down that level from before this started happening. Surprised this outage hasn't moved it yet.

CephalopodMD• 9 years ago

Target's stock price absolutely tanked today. I bet that, while the s3 outage will have a negative effect on AMZN, Target reporting way under estimates is probably creating a positive effect that evens it out.

gist• 9 years ago

AMZN stock down

YES! Buy on rumor, sell on fact as the saying goes.

johansch11• 9 years ago

https://twitter.com/Schmidt_RB/status/836641520321179648

"I know I'm piling on here, but Amazon's stock price is a better uptime indicator than their status page. #AWS #S3 #awscloud"

fjabre• 9 years ago

I never understood why so many devs flocked to AWS. I actually find their abstraction of services gets in the way and slows down my dev instead of making it easier like so many devs claim it does. I prefer Linode.

Sanddancer2• 9 years ago

It lets them pretend that ops isn't a skillset you need people to specialize in. Just throw more devs and servers at the problem instead of building a good infrastructure.

robxu9• 9 years ago

New update:

"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue."

fletom5• 9 years ago

are you openly admitting that the AWS service status page runs on AWS? because that is far more embarrassing than this downtime ever could be

elwell• 9 years ago

maybe just the image icon? idk...

dang2• 9 years ago

That's piling on, and rudely at that. Please don't be rude on HN.

We detached this subthread from https://news.ycombinator.com/item?id=13756997 and marked it off-topic.

flavor8• 9 years ago

Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

"Believe" is not inspiring.

nlightcho• 9 years ago

https://twitter.com/awscloud/status/836630468778864640

At least now we can see all the network failures in full RGB.

AtheistOfFail4• 9 years ago

We have a red error, finally!

Source: https://status.aws.amazon.com/

After two hours, they have finally updated their dashboard.

c4urself2• 9 years ago

We have now repaired the ability to update the service health dashboard -- AWS Status

Well that explains all the green checkmarks /s

oneeyedpigeon2• 9 years ago

Finally! The status page admits something's up.

contingencies4• 9 years ago

What a shame they took down MegaUpload! Clearly we need greater competition in the wholly-owned-infrastructure, file-hosting-as-a-service space.

rabidonrails2• 9 years ago

Updated:

Amazon Elastic Compute Cloud (N. Virginia) Increased Error Rates less 11:38 AM PST We can confirm increased error rates for the EC2 and EBS APIs and failures for launches of new EC2 instances in the US-EAST-1 Region. We are also experiencing degraded performance of some EBS Volumes in the Region.

Amazon Elastic Load Balancing (N. Virginia) Increased Error Rates more

Amazon Relational Database Service (N. Virginia) Increased Error Rates more

Amazon Simple Storage Service (US Standard) Increased Error Rates more

Auto Scaling (N. Virginia) Increased Error Rates more

AWS Lambda (N. Virginia) Increased Error Rates more

amasad2• 9 years ago

Where are you seeing these updates?

dlb_• 9 years ago

https://status.aws.amazon.com/

Svenskunganka3• 9 years ago

This can't be only the US-EAST-1 region. I'm a european resident and most things are down for me too.

samat2• 9 years ago

I've checked eu-west-1 and it works fine for both reads and writes.

Svenskunganka3• 9 years ago

I'm not an AWS customer, but if it is comparable to Google Cloud's Multi-regional Storage it should be geo-redundant. Doesn't S3 replicate the data across regions, so in case a region goes offline it won't affect the service?

nicpottier3• 9 years ago

SES seems to be downf or us as well.

nicpottier3• 9 years ago

SES seems to be down for us as well.

gmisra• 9 years ago

FYI to S3 customers, per the SLA, most of us are eligible for a 10% credit for this billing period. But the burden is on the customer to provide incident logs and file a support ticket requesting said credit (it must be really challenging to programmatically identify outage coverage across customers /s)

https://aws.amazon.com/s3/sla/

machbio4• 9 years ago

thats for below 99.9% - they are at 99.997% .. you are never getting that 10% credit..

christop• 9 years ago

0.1% of 28 days is 40 minutes, so it seems likely to happen.

joatmon-snoo4• 9 years ago

You got your orders of magnitude wrong ;)

    99.9964583 = 100 - 153/(30*24*60)
    99.6458333 = 100 * (1 - 153/(30*24*60))

primitivesuave2• 9 years ago

The 10% savings of ~$10 does not compare to time/potential business lost, but thanks for the tip :)

mabbo3• 9 years ago

potential business lost

My startup's op team had a great discussion today because of this that basically boils down to "if we hit our sales goals, an incident like this a year from now would end our company".

Looks like our plans to start prepping for multi-cloud support will be a higher priority.

nicpottier3• 9 years ago

SES seems to be down for us as well in Virginia. Of course nothing on the status page.

rrecuero2• 9 years ago

Is anybody else having trouble loading http://platform.twitter.com/widgets.js? It is probably hosted on S3 I assume

ladybro• 9 years ago

Fine for me.

rrecuero2• 9 years ago

Thanks. It's back up now :)

fernandopj• 9 years ago

Update[1]: AWS Status dashboard now showing icons other than green. https://status.aws.amazon.com/

[1] https://twitter.com/awscloud/status/836662601090134017

joatmon-snoo4• 9 years ago

According to the personal health dashboard, they've root-caused the S3 outage and are working to restore.

In the meantime, EC2, ELB, RDS, Lambda, and autoscaling have all been confirmed to be experiencing issues.

KurtMueller• 9 years ago

You can always check by going to www.isitdownrightnow.com/

Oh wait. The site sits on S3. Never mind.

magic_beans3• 9 years ago

Dashboard has been updated, finally!

mpetrovich2• 9 years ago

Update: AWS dashboard has been fixed and is now showing outages https://status.aws.amazon.com/

l0c0b0x• 9 years ago

Google DNS 8.8.8.8 was (for the first time that I've noticed) spotty about 30 minutes ago. Something big is happening: http://map.norsecorp.com/#/

gcoguiec• 9 years ago

We have now repaired the ability to update the service health dashboard.

It seems their status page is hosted ... as a S3 static website.

tudorconstantin• 9 years ago

"We have now repaired the ability to update the service health dashboard. " - full of yellow red icons now indeed https://status.aws.amazon.com/

ttttytjj• 9 years ago

It's fixed... I mean the status page https://status.aws.amazon.com/

robineyre• 9 years ago

Hi all. I came across this forum on Google. I have the same error - and it's all a bit beyond me. I'm not a techie or coder but set up Amazon S3 several months ago to backup my websites and it generally works fine - and has saved my bacon on a couple of occasions. (Also back up in Google Drive.)

As someone who's really only a yellow belt (assuming you're all black belts!), just so I understand ('cos I'm cacking myself!) ...

I'm seeing the same issue. Does this mean there's a problem with Amazon? I can't access either of my S3 accounts even if I change the region, and I'm concerned it may be something I've done wrong, and deleted the whole lot. It was working yesterday!!!

Would be massively grateful for a heads up. Thanks in advance.

z4chj• 9 years ago

Yes this is an issue with Amazon and there is little you can do besides wait until it has been resolved

leesalminen8• 9 years ago

Now at the top of Drudge http://drudgereport.com/

artur_makly• 9 years ago

https://twitter.com/xiaodown/status/836656364965371904

https://twitter.com/ArturMakly/status/836665379233628161

samat2• 9 years ago

One of a really rare times when it's good to be in Europe (s3 works here).

manshoor• 9 years ago

finally status are updated https://goo.gl/wCINaC

redm• 9 years ago

It looks like the S3 outage is spreading to other systems or the root cause of the S3 problem is affecting different services. There are at least 20 services listed now. [1]

[1] http://status.aws.amazon.com/

simplehuman• 9 years ago

It's still down. All morning! So much business lost.

mixedbit• 9 years ago

'Increased Error Rates' is a bit harsh, couldn't they call it 'Sub-prime Success Rates'?

FussBudget86• 9 years ago

You think this is bad? Just look at what's happening in Sweden...

kardashev• 9 years ago

You'll remember me when the west wind moves

Upon the fields of barley

You'll forget the sun in his jealous sky

As we walk in fields of green

oshoma• 9 years ago

The status page shows a lot of yellow and red now.

From http://status.aws.amazon.com/ Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

mmansoor78• 9 years ago

Per AWS :

For S3, we believe we understand root cause and are working hard at repairing. Future updates across all services will be on dashboard.

https://twitter.com/awscloud/status/836666548311859200

frik2• 9 years ago

  Increased Error Rates

  Update at 11:35 AM PST: We have now repaired the ability to 
  update the service health dashboard. The service updates 
  are below. We continue to experience high error rates with 
  S3 in US-EAST-1, which is impacting various AWS services. 
  We are working hard at repairing S3, believe we understand 
  root cause, and are working on implementing what we believe 
  will remediate the issue.

Amazon hosted their status page on their failing service, ouch. Now they fixed the status page, after more than one hour.

  The dashboard not changing color is related to S3 issue. 
  See the banner at the top of the dashboard for updates.

https://twitter.com/awscloud/status/836656664635846656

jpwgarrison2• 9 years ago

I am having trouble sending attachments in the Signal app - seems unlikely, but could this be related?

[edit- looks like they do have a pretty heavy reliance on S3, per https://github.com/WhisperSystems/Signal-Server/blob/master/... and various other sources.]

mayneack• 9 years ago

https://twitter.com/whispersystems/status/836651250842124288

Globz2• 9 years ago

My Atom keep crashing and the log says it can't resolve :

https://atom-installer.github.com/

is there a part of this hosted on S3? I cannot open Atom anymore, it keep crashing on the check for updates screen...

beeftime• 9 years ago

http://downforeveryoneorjustme.com/isitdownrightnow.com

RIP

cdnsteve• 9 years ago

Look like the dashboard has been updated to no longer use S3: AWS is having a major meltdown right now

http://status.aws.amazon.com/#ecr-us-east-1_1488312155

soheil4• 9 years ago

It doesn't look that bad, think about it S3 is such a critical part of almost any web application, it is treated like a realtime micro-service. So looks like most of the Internet in the U.S. is affected but nevertheless no one is dead yet and the world has not ended. So even if hypothetically let's say China attacked us using cyber-warfare it wouldn't be so bad after all... This was kind of like a test.

thomassharoon• 9 years ago

Is S3 down outside of Us-East too? I can't seem to create a bucket in US-West or EU

officelineback3• 9 years ago

Other regions still work, but the web console relies on us-east-1 so you should use the API to create new buckets until the issue is resolved.

rajangdavis2• 9 years ago

Hate to ask, but does anybody now of an alternative storage solution? Also, anyone have any alternative to Heroku for now?

moreisee• 9 years ago

https://cloud.google.com/storage/

wsh91• 9 years ago

Google Cloud Storage (as mentioned) for storage, Google App Engine for PaaS. :)

(I work on Cloud, specifically Datastore.)

rajangdavis2• 9 years ago

Thanks for the recommendation; is there a way to do back ups?

For example, I have images and various assets stored on S3; would there be a way to change the storage provider on the fly on a website?

The other case is could I have apps hosted on Heroku and set up a service to duplicate the app code and database over to Google for redundancy? This isn't super critical as the apps are not customer focused, but they generate content that is customer focused.

mmaunder• 9 years ago

So much has broken thanks to this. Web apps, slack uploads, parts of Freshdesk etc. I don't love you right now AWS.

https://status.aws.amazon.com/

aabajian• 9 years ago

Leap day bug?

kyled• 9 years ago

And yet people think I'm crazy for wanting to wrap get time functions so code can be tested...

vanpupi2• 9 years ago

Any opinions you can post on http://wp.me/p7HKNy-5h as well

Exuma3• 9 years ago

vanpupi2• 9 years ago

Post your opinion on http://wp.me/p7HKNy-5h

the_arun2• 9 years ago

Quora is down too. Getting 504. Gateway Timeout.

Is it related to S3??

bdcravens3• 9 years ago

I was listening to sessions from AWS Re:invent last night. What jumped out at me was the claim of 11 9's for S3. How many of those 9's have they blown through with this outage?

ta_wh2• 9 years ago

That's a durability target, not an availability SLA. Durability != Availability.

bdcravens3• 9 years ago

That makes more sense (since I was listening to it in the context of a conference session, I'm not sure if I heard a distinctive term being used, though I'm sure they're careful about the language they use)

wintermute-_-• 9 years ago

That's for data retention, S3 only guarantees three 9's availability.

Fej• 9 years ago

Okay, it's been a few hours and this is starting to get ridiculous. When was the last time that we had a core infrastructure outage this major, that lasted for this long?

samaysharma2• 9 years ago

From https://status.aws.amazon.com/: "Update at 12:52 AM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour."

(I think the AM means PM)

boulos11• 9 years ago

And now:

Update at 1:12 PM PST: S3 object retrieval, listing and deletion are fully recovered now. We are still working to recover normal operations for adding new objects to S3.

uranian• 9 years ago

Heroku seems to suffer from this too

AzzieElbab• 9 years ago

Netflix is up. Enjoy

Animats8• 9 years ago

AWS is claiming that Simple Storage (US Standard) is starting to come back up as of 12:54 PM PST.

cyberferret2• 9 years ago

Well, at least our decision to split services has paid off. All of our web app infrastructure is on AWS, which is currently down, but our status page [0] is on Digital Ocean, so at least our customers can go see that we are down!

A pyrrhic victory... ;)

[0] http://status.hrpartner.io

EDIT UPDATE: Well, I spoke too soon - even our status page is down now, but not sure if that is linked to the AWS issues, or simply the HN "hug of death" from this post! :)

EDIT UPDATE 2: Aaaaand, back up again. I think it just got a little hammered from HN traffic.

insomniacity• 9 years ago

HTTP 500 :(

crack-the-code• 9 years ago

Plot twist: Digital Ocean is secretly hosted on AWS

cyberferret2• 9 years ago

LOL...I just got notice that our status server is down now too! Maybe DO is just a rebranded offshoot of AWS after all... :D

AtheistOfFail4• 9 years ago

Could be worse, your entire infrastructure could be hosted on Heroku.

You don't use S3 but because they do, your entire infrastructure crumbles.

lambdasquirrel• 9 years ago

I don't see why this is being downvoted. It's a pretty legitimate concern.

bananabill• 9 years ago

I didn't realize Heroku used s3 until today, when my heroku app failed. Makes me wonder why I'm using heroku instead of just aws.

ExactoKnight2• 9 years ago

The biggest change heroku needs to make is support different regions.

jariz• 9 years ago

When even the status page is down, panic.

afshinmeh2• 9 years ago

Heroku as well: https://status.heroku.com/

linsomniac3• 9 years ago

We are starting to see recoveries, our SES emails have mostly gone out and our data synchronization has updated 2 of our 3 feeds. Amazon has posted a message that they expect "improved error rates" in the next 45 minutes.

outericky• 9 years ago

StatusPage.io survived. Thanks gents.

sk2code• 9 years ago

Now what kind of business choose to remain down for 2 hours plus during the peak business hours?

Seems cloud computing still has a lot to learn.

shiven• 9 years ago

Status page is lit up like a Christmas tree! Looks like AWS finally found the myriad non-green icons.

devy3• 9 years ago

So S3's been down for at least 3 hours. Does AWS break this year's S3 durability & reliability promise of eleven 9s by now? [1][2]

[1] https://aws.amazon.com/s3/details/

[2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

noir_lord3• 9 years ago

11 9's is durability not availability.

dcosson• 9 years ago

That's durability (data loss) not availability.

Here's [1] their official SLA. This outage so far brings them to less than 3 nines of uptime this month (43.8 minutes) but still more than 2 nines (7.2 hours) so it sounds like everyone gets 10% off their S3 bill.

Very curious if Amazon will apply this automatically or only if you complain.

Edit: from further down the same page, it looks like only if you write in to support do you get these broken SLA credits. Kind of lame since everything else about their billing is so precise and automatic.

[1] https://aws.amazon.com/s3/sla/

kseifried• 9 years ago

I have gotten several credits from AWS for under a dollar due to incorrect billing calculations on their end that they caught, notified me about and sent a credit for. So I will assume yes, they do this automatically (if they don't I'd be quite surprised).

kccqzy• 9 years ago

I don't think this has anything to do with durability & reliability (they probably haven't lost data). It's about availability.

melor• 9 years ago

Only limited impact to Aiven services due to service migration capability http://help.aiven.io/announcements/aiven-customer-notice-aws...

vacri8• 9 years ago

So this is particularly weird - one of my instances was showing 0% CPU in CloudWatch (dropped from 60% at the start of the event), but the logs were saying 'load 500'. I ssh'd in... and the problem resolved itself. The only thing I did was run htop to look at the load, and it dropped from 500 (reported in htop) to it's normal level. Just ssh'ing in fixed that issue.

benevol• 9 years ago

Here we go again:

Technology leads to technology (and wealth) monopolies, in other words: more centralization. Which has always been bad.

Just like with Cloudflare leaking highly sensitive data all over the Internet, a couple of days ago.

ta_wh2• 9 years ago

Wouldn't go that far. It's always been the case in the cloud that if you're not region/provider replicated, you're susceptible to localized outages.

___start• 9 years ago

but the sentiment is valid imo (albeit arguably benign)...

jasonl99• 9 years ago

I can't download purchased MP3's from amazon's own site, I get "We’re experiencing a problem with your music download. Please try downloading from Your Orders or contact us."

When I go to my orders I get "There's a problem displaying some of your orders right now. If you don't see the order you're looking for, try refreshing this page, or click "View order details" for that order."

It seems that Amazon is eating its own dog food.

francesco19752• 9 years ago

Yes it is down

https://status.aws.amazon.com/

francesco19752• 9 years ago

Yes it is down

https://status.aws.amazon.com/

Half internet is down the data center in Virginia the one with the cloud is totally dead apparently. Enjoy the cloud bullshit :)

splatcollision• 9 years ago

I just spent the last hour trying to figure out why in the hell I can't update the function code on a lambda instance. Next time I will remember to check HN first!

boulos11• 9 years ago

Disclosure: I work on Google Cloud.

Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4):

https://cloud.google.com/storage/docs/interoperability

and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to storage.googleapis.com), and it works like a charm. Resumable uploads work differently between us, but honestly since we let you do up to 5TB per object, I haven't needed to bother yet.

Again, Disclosure: I work on Google Cloud (and we've had our own outages!).

masterleep• 9 years ago

Competition is great for consumers!

JPKab• 9 years ago

I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support)

It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.

espeed3• 9 years ago

Me too. We switched to Google Cloud years ago at its inception and have never looked back -- always viewed it as a competitive advantage due to its solid, more advanced infrastructure -- faster network, reliable disks, cleaner UI that's easier to manage. Just a cleaner operation all the way around.

eknkc2• 9 years ago

We were on GCP for around a year, it was my decision I really wanted to love GCP and I initially did. But we recently switched to AWS.

I think there is little GCP does better than AWS. Pricing is better on paper, but performance per buck seems to be on par. Stability is a lot worse on GCP, and I don't just mean service outages like this one (which they had their fair share) but also individual issues like instances slowing down or network acting up randomly. Also lack of service offerings like no PostgreSQL, functions never leaving alpha, no hosted redis clusters etc... Support is also too expensive compared to AWS.

Management interfaces are better on GCP and sustained use discount is a big step up against AWS reservations. Otherwise, I think AWS works better.

advisedwang2• 9 years ago

I work in GCP support. I'm really curious: what do you feel changed that led to such improved support? I'd like to make sure we keep doing it.

vacri8• 9 years ago

My experience of support with Google Apps for Business makes me very wary of using anything Google for critical business infra. Google products are nice, but as soon as you hit a problem or edge case, you're on your own in my experience.

hkmurakami2• 9 years ago

(Before that, you guys didn't at all have your shit together when it came to customer support)

Sounds like it basically coincides with Diane Greene coming on board to run the show -- which is great news for all of us with increased competition on not just the technical front but also support (which is often the deal maker/breaker)

ehsankia• 9 years ago

Honestly, if you're a big service that millions of people use, you should not put all your eggs in a single basket and should probably use a mix, in case one of the clouds goes down like in this case.

snackai• 9 years ago

What indeed is bad taste is your choice of Google Cloud over AWS. No I really like GCP, use it at core of many apps, but if people really want a decentralized web we need to use more than one provider. Don't "convert". Use booth, redundancy ffs.

jamesblonde2• 9 years ago

I just wrote a piece reflecting on the s3 outage and the limitations of s3 metadata/replication:

https://medium.com/@jim_dowling/reflections-on-s3s-architect...

themihai3• 9 years ago

GCP has always felt like a forever beta product. On top of that you get a lot of lockin so I would never recommend GCP for a long term project.

sachinag2• 9 years ago

Opportunistic, sure. But I did not know about the API interoperability. Given the prices, makes sense to store stuff in both places in case one goes down.

khc• 9 years ago

I am surprised more people don't know about it. I get questions like https://github.com/kahing/goofys/issues/158 every now and then and to be fair I don't think they market it well: https://cloud.google.com/storage/docs/migrating

Disclosure: I don't work for google but have an upcoming interview there.

danielvf• 9 years ago

I had no idea this was an option. Great to know!

ptrptr• 9 years ago

Now that's what I call a shameless plug!

zoloateff2• 9 years ago

boulous not in bad taste at all - happy google convert and gcs user works very well for us ymmv

zoloateff2• 9 years ago

boulous is app engine datastore the preferred way to store data or cloud sql or something else, do you mind throwing some light on this thanks

espeed3• 9 years ago

What software are you using for your NAS box?

mbrumlow• 9 years ago

What is your NAS box doing with S3/GCS ?

boulos11• 9 years ago

Remote backup (Synology). I've asked them more than once to directly support GCS, or even just to accept my damn patch ;).

simonebrunozzi• 9 years ago

I keep telling people that in my view, Google Cloud is far superior to AWS from a technical standpoint. Most people don't believe me... Yet. I guess it will change soon.

natbobc3• 9 years ago

Google Cloud is the Betamax of cloud... while it might be technically superior it's not the only factor to consider. :)

notyourwork3• 9 years ago

One service outage determines superiority? I prefer a lot more data than a single point.

scrollaway6• 9 years ago

We would definitely seriously consider switching to GCS more if your cloud functions were as powerful as AWS Lambda (trigger from an S3 event) and supported Python 3.6 with serious control over the environment.

vikiomega9• 9 years ago

On a curious note, how do you guys use lambda?

boulos11• 9 years ago

Is there something about the GCS trigger that doesn't work for you? I hear you on Python 3, but I'm also curious about "serious control over the environment". Can you be more specific?

rynop• 9 years ago

"fraction of the cost" - how do you figure? Or are you just saying from a cost-to-store perspective?

Your Egress prices are quite a bit more compared to CloudFront for sub 10TB (.12/GB vs .085/GB).

The track record of s3 outages vs time your up and sending Egress seems like S3 wins in cost. If all your worried about is cross region data storage, your probably a big player and have AWS enterprise agreement in place which offsets the cost of storage.

boulos11• 9 years ago

Sorry, my comparison is our Multi Regional storage (2.6c/GB/month) versus S3 Standard plus Cross-Regional Replication. That's the right comparison (especially for outages like this one).

As to our network pricing, we have a drastically different backbone (we feel its superior, so we charge more). But as you mention CloudFront, the right comparison is probably Google Cloud CDN (https://cloud.google.com/cdn/) which has lower pricing than "raw egress".

blantonl• 9 years ago

I have a component in my business that writes about 9 million objects a month to Amazon S3. But, to leverage efficiencies in dropping storage costs for those objects I created an identical archiving architecture on Google Cloud.

It took me about 15 minutes to spin up the instances on Google Cloud that archive these objects and upload them to Google Storage. While we didn't have access to any of our existing uploaded objects on S3 during the outage, I was able to mitigate not having the ability to store any future ongoing objects. (our workload is much more geared towards being very very write heavy for these objects)

It it turns out this cost leveraging architecture works quite well as a disaster recovery architecture.

twakefield2• 9 years ago

The brilliance of open sourcing Borg (aka Kubernetes) is evident in times like these. We[0] are seeing more and more SaaS companies abstract away their dependencies on AWS or any particular cloud provider with Kubernetes.

Managing stateful services is still difficult but we are starting to see paths forward [1] and the community's velocity is remarkable.

K8s seems to be the wolf in sheep's clothing that will break AWS' virtual monopoly on IaaS.

[0] We (gravitational.com) help companies go "multi-region" or on-prem using Kubernetes as a portable run-time.

[1] Some interesting projects from this comment (https://news.ycombinator.com/item?id=13738916)

* Postgres automation for Kubernetes deployments https://github.com/sorintlab/stolon

* Automation for operating the Etcd cluster:https://github.com/coreos/etcd-operator

* Kubernetes-native deployment of Ceph: https://rook.io/

dankohn1• 9 years ago

Note that Kubernetes "builds upon 15 years of experience of running production workloads [on Borg] at Google" [0], but is different code than Borg.

In addition to Rook, Minio [1] is also working to build an S3 alternative on top of Kubernetes, and the CNCF Landscape is a good way of tracking projects in the space [2].

[0] https://kubernetes.io/ [1] https://www.minio.io/ [2] https://github.com/cncf/landscape

Disclosure: I'm the executive director of CNCF, which hosts Kubernetes, and co-author of the landscape.

013a• 9 years ago

Is there any way built in to Kubernetes to go multi-AZ, multi-region, or even multi-cloud? Is federation the answer to this?

I remember reading somewhere in the K8s documentation that it is designed such that nodes in a single cluster should be as close as possible, like in the same AZ.

justicezyx• 9 years ago

K8s is a better borg! It leaps forward and build upon many years experience of operating the system.

joshontheweb4• 9 years ago

I'm in the process of moving to GCS mostly based on how byzantine the AWS setup is. All kinds of crazy unintuitive configurations and permissions. In short, AWS makes me feel stupid.

joshontheweb4• 9 years ago

I should add that someone from the AWS team reached out to me in response to this comment asking for feedback on how they can improve their usability. So I give them credit for that.

i336_2• 9 years ago

Hopefully you're still there even though S3 is back up. I have an interesting question I really, really hope you can answer. (Potential customer(s) here!!)

There are a large number of people out there looking intently at ACD's "unlimited for $60/yr" and wondering what that really means.

I recently found https://redd.it/5s7q04 which links to https://i.imgur.com/kiI4kmp.png (small screenshot) showing a user hit 1PB (!!) on ACD (1 month ago). If I understand correctly, the (throwaway) data in question was slowly being uploaded as a capacity test. This has surprised a lot of people, and I've been seriously considering ACD as a result.

On the way to finding the above thread I also just discovered https://redd.it/5vdvnp, which details how Amazon doesn't publish transfer thresholds, their "please stop doing what you're doing" support emails are frighteningly vague, and how a user became unable to download their uploaded data because they didn't know what speed/time ratios to use. This sort of thing has happened heaps of times.

I also know a small group of Internet archivists that feed data to Archive.org. If I understand correctly, they snap up disk deals wherever they can find them, besides using LTO4 tapes, the disks attached to VPS instances, and a few ACD and GDrive accounts for interstitial storage and crawl processing, which everyone is afraid to push too hard so they don't break. One person mentioned that someone they knew hit a brick wall after exactly 100TB uploaded - ACD simply would not let this person upload any more. (I wonder if their upload speed made them hit this limit.) The archive group also let me know that ACD was better at storing lots of data, while GDrive was better at smaller amounts of data being shared a lot.

So, I'm curious. Bandwidth and storage are certainly finite resources, I'll readily acknowledge that. GDrive is obviously going to have data-vs-time transfer thresholds and upper storage limits. However, GSuite's $10/month "unlimited storage" is a very interesting alternative to ACD (even at twice the cost) if some awareness of the transfer thresholds was available. I'm very curious what insight you can provide here!

The ability to create share links for any file is also pretty cool.

andmarios3• 9 years ago

As far as I understand the S3 API of Cloud Storage is meant as a temporary solution until a proper migration to Google's APIs.

The S3 keys it produces are tied to your developer account. This means that if someone gets the keys from your NAS, he will have access to all the Cloud Storage buckets you have access to (e.g your employer's).

I use Google Cloud but not Amazon. Once I wanted a S3 bucket to try with NextCloud (then OwnCloud). I was really frightened to produce a S3 key with my google developer account.

dividuum• 9 years ago

Is there any equivalent to the Bucket Policies that AWS provides (http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucke...). Cloud Storage seems to be limited to relatively simple policies without conditionals. For a few AWS IAM keys I set up a policy that limits write/delete access to a range of IPs (among other things). Something like that doesn't seem possible with what Google offers. Or do I miss something?

BrandonY2• 9 years ago

The HMAC credential that you'd use with the S3-compatible GCS API, also called the "XML API", does need to be associated with a Google account, but it doesn't need to be the main account of the developer. It can be any Google user account. I suggest creating a separate account and granting it only the permissions it needs. It'd be nice if service accounts (aka robot accounts) could be given HMAC credentials, that's not supported. Service accounts can, however, sign URLs with RSA keys.

As another option, you can continue using the XML API and switch out only the auth piece to Google's OAuth system while changing nothing else.

There's a lot more detail available at: https://cloud.google.com/storage/docs/migrating

Disclaimer: I work on Google Cloud Storage.

stef25• 9 years ago

OwnCloud

Kicked the tires, not impressed at all. Notes went missing from the interface could only get them back after manually digging through folders via FTP.

nodesocket3• 9 years ago

Not poor taste at all. Love GCP. I actually host two corporate static sites using Google Cloud Storage and it is fantastic. I just wish there was a bucket wide setting to adjust the cache-control setting. Currently it defaults to 1 hour, and if you want to change it, you have to use the API/CLI and provide a custom cache control value each upload. I'd love to see a default cache-control setting in the web UI applying to the entire bucket.

I also want to personally thank Solomon (@boulos) for hooking me up with a Google Cloud NEXT conference pass. He is awesome!

7ewis2• 9 years ago

How did you get the pass?

Been trying to get one for IO (can't attend NEXT unfortunately)

dward• 9 years ago

Out of curiosity, are you also using the cloud CDN?

https://cloud.google.com/compute/docs/load-balancing/http/us...

DenisM4• 9 years ago

If you made a .NET library that allows easily connecting to both AWC and GCS by only changing the endpoint I would certainly use that library instead of Amazon's own.

Just saying, it gets you a foot in the door.

Spunkie• 9 years ago

So this is more compute related but do you know if there are any plans on supporting the equivalent of the webpagetest.org(WPT) private instance AMI on your platform?

Not only is webpagetest.org a google product but it's also much better suited for the minute by minute billing cycle of google cloud compute. For any team not needing to run hundreds of tests an hour the cost difference between running a WPT private instance on EC2 versus on google cloud compute could easily be in the thousands of dollars.

pmarreck• 9 years ago

Classy parley. I'll allow it.

philliphaydon2• 9 years ago

Does GCS support events yet?

hyperpallium• 9 years ago

As Relay's chief competitor in this region, we of Windsong have benefited modestly from the overflow; however, until now we thought it inappropriate to propose a coordinated response to the problem.

gaul• 9 years ago

S3 applications can use any object store if they use S3Proxy:

https://github.com/andrewgaul/s3proxy

malloryerik• 9 years ago

Would use Google but I just can't give up access to China. Sad because I also sympathize with Google's position on China.

sandGorgon• 9 years ago

i have had problems integrating apache spark using google storage. especially because s3 is directly supported in spark.

if you are api compatible with s3, could you make it easy /possible to work with google storage inside spark?

remember i may or may not run my spark on Dataproc.

bluedonuts• 9 years ago

You can use the Google cloud storage connector (https://cloud.google.com/hadoop/google-cloud-storage-connect...) which works with hadoop (and therefore spark).

NiekvdMaas2• 9 years ago

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services:

Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue fixed. During the downtime, there is nothing you can do to speed this up - you're waiting helplessly. Because Cloud SQL is a hosted service, you can not connect to a shell or access any filesystem data directly - there is nothing you can do, other than wait for the Google engineers to resolve the problem.

When the Cloud SQL instance was up&running again, support confirmed that there is nothing you can do to prevent a filesystem crash, it "just happens". The workaround they offered is to have a failover set up, so it can take over in case of downtime. The worst part is that GCS refused to offer credit, as according to their SLA this is not considered downtime. The SLA [1] states: "with respect to Google Cloud SQL Second Generation: all connection requests to a Multi-zone Instance fail" - so as long as the SQL instance accepts incoming connections, there is no downtime. Your data can get lost, your database can be unusable, your whole system might be down: according to Google, this is no downtime.

TL;DR: make sure to check the SLA before moving critical stuff to GCS.

[1] https://cloud.google.com/sql/sla

TekMol• 9 years ago

21 hours sounds pretty long to me. What type of data was it and how long would you have waited until you continued with a backup of the data on a different machine?

fidget3• 9 years ago

The GCS being referred to by the GP is Google Cloud Storage, not Cloud Sequel. You really do need failover set up though. That's true for basically any MySQL installation, managed or not.

lbill• 9 years ago

Not using a failover is a bold choice (not stupid, just bold). A failover is like a good insurance policy: you pay for it, you hope that you'll never need it, but when shit happens you are very happy to have it!

adwf• 9 years ago

That isn't just a Google issue though. You'd have had the exact same trouble with AWS/RDS if you're running with no replica. The lack of filesystem access is a security "feature" for both. If you have no HA setup then you have no recourse but to restore to a new server from backup, or wait for your cloud provider to fix it.

thejosh• 9 years ago

How about giving a timeline of when Australia will be launching? I see you're hiring staff, and have a "sometime 2017" goal on the site, but how about a date estimate? :)

carimura• 9 years ago

We're seeing recovery across our services now.

jedicoder107• 9 years ago

Status Pages (Services & Products affected by S3 outage)

- https://status.heroku.com/

- https://status.aws.amazon.com/

- https://medium.statuspage.io/

- https://status.slack.com/

- http://status.filestack.com/

- http://www.trellostatus.com/

- https://health.autodesk.com/

- http://status.ifttt.com/

- http://status.imgur.com/

- http://status.docker.com/

tyingq• 9 years ago

Pretty good list of other affected sites/services in this article: http://venturebeat.com/2017/02/28/aws-is-investigating-s3-is...

Some big names and services popular with HN mentioned there. Quora, AirBnb, SendGrid, Downdetector(heh).

tjpaudio• 9 years ago

Interestingly, I placed an order on amazon.com and while the order appears when I look at my account, none of the usual automated emails have come. I wonder how deeply this is effecting their retail customers.

netvisao• 9 years ago

Looks like our dashboard is still sustaining it https://acedashboard.cbp.dhs.gov/

poofyleek• 9 years ago

This is truly serverless computing at work.

phildougherty• 9 years ago

I wrote a quick post discussing this outage. I figured I should share here https://blog.containership.io/aws-got-you-down

newman314• 9 years ago

There is no cloud, there is only someone else's computer.

Animats8• 9 years ago

Negative comment on all this in Forbes.[1] Too much centralization. CEOs read that.

[1] https://www.forbes.com/sites/ryanwhitwam/2017/02/28/amazon-s...

Animats8• 9 years ago

"Inc." is quoting comments from here.[1]

[1] http://www.inc.com/sonya-mann/amazon-web-services-outage.htm...

nvarsj• 9 years ago

eu-west-1 is doing great. Obviously European ops are superior to their US counterparts.

cperciva4• 9 years ago

S3 is currently (22:00 UTC) back up.

The timeline, as observed by Tarsnap:

    First InternalError response from S3: 17:37:29
    Last successful request: 17:37:32
    S3 switches from 100% InternalError responses to 503 responses: 17:37:56
    S3 switches from 503 responses back to InternalError responses: 20:34:36
    First successful request: 20:35:50
    Most GET requests succeeding: ~21:03
    Most PUT requests succeeding: ~21:52

josephb• 9 years ago

Thanks for taking the time to post a timeline from the perspective of an S3 customer. It will be interesting to see how this lines up against other customer timelines, or the AWS RFO.

kaishiro2• 9 years ago

Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?

greenleafjacob• 9 years ago

Could be anything. Most likely scenario is the internal error is a load shedding error and the 503s were when the system became completely unresponsive. If it was a configuration issue then it is more likely that it would have directly recovered rather than going 'internal error -> 503 -> internal error'.

hmottestad• 9 years ago

503 is typically what we see when our proxy can't connect to the backend server. We usually get 500 with internal server error when we've messed up the backend server.

So it's likely that the first 500s were the backend for s3 failing, then they took the failing backends offline causing the load balancers to throw 503 because they couldn't connect to the backend.

cperciva4• 9 years ago

My guess based on the behaviour I've seen is that internal nodes were failing, and the 503 responses started because front-end nodes didn't have any back-end nodes which were marked as "not failing and ready for more requests". When Amazon fixed nodes, they would have marked the nodes as "not failed", at which point the front ends would have reverted to "we have nodes we can send traffic to" behaviour.

Twirrim2• 9 years ago

S3 is not a monolithic architecture, Amazon is a strong proponent of Service Oriented Architecture for producing scalable platforms.

There are a number of services behind the front end fleet in S3's architecture that handle different aspects of returning a response. Each of those will have their own code paths in the front end, very likely developed by different engineers over the years. As ever, appropriate status codes for various circumstances are something that always seems to spur debate amongst developers.

The change in status code would likely be a reflection of the various components entering unhealthy & healthy states, triggering different code paths for the front end... which suggests whatever happened might have had quite a broad impact, at least on their synchronous path components.

reiichiroh• 9 years ago

Any truth to this being a DOS by some kiddies named Phantom Squad?

thenewregiment2• 9 years ago

soundcloud uses aws s3. it is still down.

gopalakrishnans• 9 years ago

Just in case if anybody needs :) https://www.microsoft.com/developerblog/real-life-code/2016/...

mcheshier• 9 years ago

If I listen closely, I think I can hear the pagers going off in South Lake Union from Downtown Seattle.

cdevs• 9 years ago

I think there was some fontawesome loading issues related to this, I also noticed a site trying to load twitter messages but couldn't Get the JavaScript loaded during that time today.

metafunctor2• 9 years ago

Based on reports from the field, it looks like S3 was down for about three hours for most of their customers.

S3 promises four nines of availability (11 nines of durability), so today we got about 3-4 years worth of downtime in one fell swoop. Oops.

machbio4• 9 years ago

durability is different from availability

IAmGraydon2• 9 years ago

Which is why he made the distinction between the two.

cowkingdeluxe• 9 years ago

Where do you see four nines for S3 SLA?

https://aws.amazon.com/s3/sla/ shows 99.9%

metafunctor2• 9 years ago

It was from memory. I distinctly remember it being advertised as "four nines". Perhaps they've adjusted their marketing. The S3 FAQ still says [1]: "S3 Standard is designed for 99.99% availability".

[1] https://aws.amazon.com/s3/faqs/

sonnyhe2002• 9 years ago

Is it down again?

nodesocket3• 9 years ago

Meanwhile engineers across the globe scramble to fix outages due to AWS s3, $AMZN is unaffected on the stock market. Just shows the disconnect between emotions and reality.

https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&...

IAmGraydon2• 9 years ago

Why would this affect Amazon's stock? Amazon generates $136B per year. AWS comprises $8B of that. Even if 5% of customers left because of this (and that's never going to happen), it would barely create as much as a tiny ripple in their ocean of revenue. Investors care not about Amazon's cloud services.

ec1096856• 9 years ago

AWS's profit helps fund the rest of their business.

ianamartin2• 9 years ago

This is why it's important to write code that doesn't depend on only a single service provider. S3 is great. But it's better to set up a Riak cluster on AWS than to actually use S3, if you can.

The only services my team uses directly are EC2 and RDS, and I'm thinking of moving RDS over to EC2 instances.

We are entirely portable. We can move my entire team's infrastructure to a different cloud host really quickly. Our only dependency is a Debian box.

I flipped the switch today and cloned our prod environment, including VPN and security rules, over to a commodity hosting provider.

Change the DNS entry for the services, and we were good to go. We didn't need to do anything because everyone was freaking out about everything else being down. But our internal services were close to unaffected.

At least for my team.

Obviously, we aren't Trello or some of the other big people affected. And we don't have the same needs they do. But setting up the DevOps stuff for my team in the way that I think was correct to begin with (no dependencies other than a Debian box) really shined today. Having a clear and correct deployment strategy on any available hardware platform really worked for us.

Or at least it would have if people weren't so upset about all our other external services being down that they paid no attention to internal services.

Lock-in is bad, mmkay?

If your company is the right size, and it makes sense, do the extra work. It's not that hard to write agnostic scripts that deploy your software, create your database, and build your data from a backup. This can be a big deal when some providers are flipping out.

All-your-junk-in-one-place is really overrated, in my opinion. Be able to rebuild your code and your data at any given point in time. If you don't have that, I don't really know what you have.

ec1096856• 9 years ago

Not knowing your situation exactly, but there could be a cost of running your own infrastructure and not taking advantage of their services? For example, are the chances of losing data higher in riak (take into account disaster and operational bugs that could result in data loss or availability issues) than in one of amazon's supported data stores.

I don't necessarily disagree with what you are saying but there is cost of doing everything yourself.

You would have been equally protected if you had been in more than one region.

ianamartin2• 9 years ago

Yes. There's a cost to every decision you make. Sometimes it's the cost of s3 being down. Sometimes it's the cost of some developer time to make services agnostic. The value of that isn't immediately obvious, perhaps.

But the developer cost here (my time) was worth it. Our shit wasn't down, while everyone else's was.

I also want to point out that I spent minimal time setting this up. We can deploy to GCE or commodity VPCs at a moment's notice, and that a project I did over a couple of weekends piggybacking on the ansible playbooks I wrote for AWS.

It's not that hard. You have to get your developers on board with being provider agnostic, and you have to be agnostic yourself. But it is not insurmountable.

It also help when you're the lead dev or your team and also have a good relationship with the devops guy. :)

all_usernames4• 9 years ago

As of 4:30PM Pacific, we're still having trouble with EC2 autoscaling API operations in US-East-1. Basically very long delays in launching new instances or terminating old ones.

sc30317• 9 years ago

Still no RCA? I'd love to hear what the issue was for this. A couple of coworkers and I are betting that it was a networking issue of some sort.

ibmcloud• 9 years ago

Our IBM cloud- Softlayer provides secure and stable cloud environment with private network, for baremetal,dedicated, private and public cloud. Leave a comment if you want to learn more. also HIPAA ready.

mvindahl• 9 years ago

We all laughed at the notion om moon people dropping rocks at the earth.

Then they started dropped rocks on S3 and who is laughing now?

julenx• 9 years ago

Outage as a Service

nomadic_09• 9 years ago

I'm curious to see the postmortem for this.

kfkhalili• 9 years ago

So why did the outage occur?

Ask HN: Is S3 down?

Related Stories