Amazon DNS error

https://www.amazon.com

by tanto • 12 years ago

156 66 12 years ago

UPDATE: RESOLVED according to the AWS status page -- 6:24 PM PST Between 4:12 PM and 6:02 PM PST, users experienced elevated error rates when making DNS queries for CloudFront distributions. The service has recovered and is operating normally. [2]

Cloudfront DNS is hosed. Doing a DNS lookup on my cloudfront distribution fails. Many folks on Twitter also see the issue too [1]. Maybe a failed upgrade or something, their whois info was updated today @ 2014-11-26T16:24:49-0800.

  [~]$ host d1cg27r99kkbpq.cloudfront.net
  Host d1cg27r99kkbpq.cloudfront.net not found: 2(SERVFAIL)

[1] https://twitter.com/search?q=cloudfront

[2] http://status.aws.amazon.com/

dice2• 12 years ago

The AWS status page not showing anything isn't unusual. They'll probably update in half an hour to describe it as a partial failure then revise it to an "all OK" 10 minutes before it's actually fixed then retroactively downgrade it to a minor quality of service disruption.

Not that I've noticed they tend to lie through their teeth on the status page or anything....

teraflop• 12 years ago

Apparently, "DNS resolution errors" are considered neither a "service disruption" nor a "performance issue", but instead are categorized as an "informational message."

aidos• 12 years ago

Seeing issues with aws console (also looks cloudfront DNS related).

flippyhead• 12 years ago

seeing the same :/

jokamoto• 12 years ago

Looks like anything running off of AWS DNS is dead right now.

  $ dig @ns-666.awsdns-19.net cloudfront.net

Times out for that server as well as for all of the nameservers listed for cloudfront.net.

tootie• 12 years ago

My Route53 domains are ok.

rustyconover• 12 years ago

Route53 still is returning answers for records that are not aliases to Cloudfront.

If you have an alias to a cloudfront distribution the answers aren't being provided by the Route53 servers.

You can verify this with a dig:

dig mx [your domain like domain.com] - will probably still work.

dig a [name of alias like www.domain.com] - isn't working.

jordanthoms5• 12 years ago

Yep, seeing strange intermittent DNS errors for CF here.

johngd• 12 years ago

Amazon.com's content cdn is returning 0 byte replies for images.

stephenc_c_• 12 years ago

My site is down, I'm using Route53, S3 and CloudFront so I guess that's why.

zuccs• 12 years ago

Same setup here except Cloudflare instead of Route53. So I guess it's S3/CloudFront related?

thanhquanky• 12 years ago

I think it's fixed

general_failure• 12 years ago

None of the images are loading

atom_enger3• 12 years ago

Also experiencing issues with the AWS Console. Still not resolved for me.

juliann• 12 years ago

Not loading any images for me. Browsing from Argentina.

ejdyksen• 12 years ago

Amazon appears to be up for me, but a number of places are still affected:

- All of Vox Media's properties (The Verge, Polygon, Vox.com, SBNation, etc)

- All of Atlasssian's services (Bitbucket, Jira OnDemand, etc)

- Flowdock

- Instagram

- aws.amazon.com has no assets

Edit: console.aws.amazon.com has no assets, either, so it's also currently worthless.

It's probably worth having a DNS failover strategy for Route53 (if that's what you're using) that doesn't involve the UI on console.aws.amazon.com.

jbinto• 12 years ago

Amazon.com is up for me, but is entirely missing product images.

stevekemp• 12 years ago

It's probably worth having a DNS failover strategy for Route53 (if that's what you're using) that doesn't involve the UI on console.aws.amazon.com.

Which is one of the reasons I setup https://dns-api.com/ - A way of updating Route53 DNS via git hooks.

bbnnt2• 12 years ago

yep all crashed :/ is that a regular behavior from the amazon status page, being all green when a bunch of users experience the same problem ?

dice2• 12 years ago

It's flapping in some locations. Seems to be hard down in others. From a service monitoring Cloudfront on various geographically distributed Nagios monitors (times are PST):

  Columbus:

  [11-26-2014 16:46:24] SERVICE ALERT: public-www;CDN - Logo;OK;SOFT;2;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.165 second response time
  [11-26-2014 16:45:34] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known
  [11-26-2014 16:39:24] SERVICE ALERT: public-www;CDN - Logo;OK;SOFT;3;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.030 second response time
  [11-26-2014 16:38:34] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;2;Name or service not known
  [11-26-2014 16:37:34] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known
  [11-26-2014 16:25:24] SERVICE ALERT: public-www;CDN - Logo;OK;SOFT;2;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.030 second response time
  [11-26-2014 16:24:34] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known
  [11-26-2014 16:21:24] SERVICE ALERT: public-www;CDN - Logo;OK;SOFT;2;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.066 second response time
  [11-26-2014 16:20:24] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known

  Portland:

  [11-26-2014 16:49:40] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known
  [11-26-2014 16:43:40] SERVICE ALERT: public-www;CDN - Logo;OK;SOFT;2;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.148 second response time
  [11-26-2014 16:42:40] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known
  [11-26-2014 16:39:40] SERVICE ALERT: public-www;CDN - Logo;OK;HARD;3;HTTP OK: HTTP/1.1 200 OK - 2960 bytes in 0.186 second response time
  [11-26-2014 16:21:40] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;HARD;3;Name or service not known
  [11-26-2014 16:20:40] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;2;Name or service not known
  [11-26-2014 16:19:41] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known

  Santa Clara:

  [11-26-2014 16:24:26] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;HARD;3;Name or service not known
  [11-26-2014 16:23:25] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;2;Name or service not known
  [11-26-2014 16:22:25] SERVICE ALERT: public-www;CDN - Logo;CRITICAL;SOFT;1;Name or service not known

maknz• 12 years ago

Our CloudFront distribution is down. Route53 seems fine, but it could just be caching. Looks like we're in good company though.

cjreyes• 12 years ago

Is there a RoR Gem / configuration that will serve assets locally if an external asset host name doesn't resolve or times out?

bbnnt2• 12 years ago

that'd be genius

Hengjie2• 12 years ago

There isn't, but there's a frontend library that you can use to add fallbacks to <script> tags:

https://github.com/shinnn/script-fallback-from-urls

tbfrench• 12 years ago

Not a gem, but from jQuery days:

lukeschlather• 12 years ago

It's a decent idea, but it would be better to do client-side. For this sort of event, knowing that the name resolves on the server doesn't give you any confidence it will resolve for the client.

What would be really nice is if you could specify a fallback host in your DNS prefetch, and the browser would make it "just work."

thejosh• 12 years ago

Exactly, if a POP for Australia is down but works in Germany, better to serve this on the client side.

jacobsenscott• 12 years ago

You can assign a proc to config.asset_host, so you could easily do it -

  config.asset_host = -> {
    cdn_up? ? "http://mycdn.com" : "http://mydomain.com"
  }

KhalilK• 12 years ago

DDoS?

ranman• 12 years ago

Status page is updated: http://status.aws.amazon.com/

msaffitz• 12 years ago

We were first alerted to an issue by our monitoring at 4:20 Pacific, 40 minutes before AWS updated their status page. It's infuriating that AWS doesn't communicate more promptly and transparently when an issues like this occurs.

pauloschilling• 12 years ago

fletchowns2• 12 years ago

Is it any different if you shell out the big bucks for support?

jordanthoms5• 12 years ago

Half the internet is down, Amazon.com is flapping, the AWS website doesn't have any assets, and status.aws has a Green checkmark with a little (i), 40 minutes after the problems start?!

I love AWS but they really need to improve their procedures for communicating during outages. If your parent companies' billion dollar site can be affected in any way on the night before black friday, and even your own site is down, and you are not acknowledging that the service is FUBAR - you have a problem.

cperciva• 12 years ago

Hey, if you're able to load status.aws.amazon.com then clearly it's working!

jordanthoms5• 12 years ago

It's just as well they don't use CF for anything critical on the status site...

erichmond• 12 years ago

Exactly this. In our experience Amazon's status page doesn't reflects actual outages we are having, Cloudwatch often sends false positives (systems being down, when in fact, they are not) and SNS messages get lost in the ether.

We've built our systems to recover from failure states once we know there is a problem. AWS's inability to do that reliably is forcing us to own the problem ourself, and as a result, we will probably migrate away to use cheaper boxes in the cloud.

On the positive side, RDS has been solid.

felixgallo• 12 years ago

Imagine what kind of conditions would have to exist for Amazon to use a red indicator.

corobo• 12 years ago

Greentick [i]: Amazon AWS is closing its doors forever, we apologise for the inconvenience

Essa• 12 years ago

Amazon right now http://i.imgur.com/Ed2jtn2.jpg

pestaa• 12 years ago

I too like jokes, but saying nothing else leads to superficial discussions.

timfernando• 12 years ago

Having lots of issues on the EU side too (Route 53 + Cloudfront).

Noticed: - trello.com - import.io - intercom.io

as some sites as well as ours who we've noticed issues with.

anonymuse3• 12 years ago

AWS Status shows Cloudfront DNS issues, which is reflected in our page's assets not loading. Kinda makes me wish we were using something like https://github.com/etsy/cdncontrol/ but that's a fight for another day!

  <title type="text">Informational message: DNS Resolution      errors </title>

  <link>http://status.aws.amazon.com</link>

  <pubDate>Wed, 26 Nov 2014 17:00:39 PST</pubDate>
	   <guid>http://status.aws.amazon.com/#cloudfront_1417050039</guid>

  <description>We are currently investigating increased error   rates for DNS queries for CloudFront distributions.  </description>

anonymuse3• 12 years ago

Anecdotally, our site is now loading quickly for me: http://canary.is/.

However, digging the Cloudfront name servers times out intermittently:

  $ dig +short @ns-666.awsdns-19.net cloudfront.net
  ;; connection timed out; no servers could be reached

theophrastus• 12 years ago

One wonders if this is same matter currently afflicting a certain infamous torrent site ("..downtime appears to be a routing issue as the site is still reachable in most parts of the world"): http://torrentfreak.com/the-pirate-bay-goes-down-locally-141...

atom_enger3• 12 years ago

Seems like the best way to verify an AWS outage is via Twitter (#AWS) ... This doesn't feel right ..

rynop• 12 years ago

The Route53 (AWS DNS service) is 100% SLA: http://aws.amazon.com/route53/sla/

I would hope they honor this dns issue under the same guidelines although its technically not the route53 service we are paying for.

andrea_s• 12 years ago

I use Route53 with no Cloudfront involvement, and everything looks good... So I think it's not really related to Route53

sumnulu• 12 years ago

Not fixed yet, CloudFront is down. DNS resolution not working.

robryan2• 12 years ago

I wonder if we should except s3 to degrade pretty badly if this isn't fixed soon. Going to be seeing a much higher than normal rate of requests.

rdl• 12 years ago

I'm curious if the root cause of this is software failure or external attack. Aws is clearly at the point in size and competence that hardware failures are unlikely as root cause.

toyg• 12 years ago

Aws is clearly at the point in size and competence that hardware failures are unlikely as root cause.

The opposite might also be true: Amazon might now have reached a point in size where they can't scale further upwards without losing visibility and control of part of their hardware.

It's all speculation anyway.

ams6110• 12 years ago

But aren't big, complex systems more likely to fail in unexpected ways?

joeevans1000• 12 years ago

The status page icon for CloudFront turns red when HN comments reach 100.

jordanthoms5• 12 years ago

Actually I think red is reserved for a nuclear attack on N. Virginia by ze Russians...

anonymuse3• 12 years ago

This is good, let's get this sort of hiccup out of the way before the predictably massive traffic from Black Friday and Cyber Monday.

What's everyone using to monitor external asset hosts? Is anyone dynamically switching between them, or failing back to local assets?

atom_enger3• 12 years ago

This was linked elsewhere in the thread https://github.com/etsy/cdncontrol/ and Etsy seemed to be up through the entire thing. So perhaps they're doing something right :)

jpetitbon2• 12 years ago

From what I can see myself and from twitter feed, the issue looks resolved. But the AWS status page is still not reporting any update.

jpetitbon2• 12 years ago

"5:57 PM PST Error rates for DNS queries of CloudFront distributions are currently recovering."

jonperl• 12 years ago

NewRelic synthetics caught this for me.

Hengjie2• 12 years ago

6:24 PM PST Between 4:12 PM and 6:02 PM PST, users experienced elevated error rates when making DNS queries for CloudFront distributions. The service has recovered and is operating normally.

Amazon DNS error

Related Stories