Cloudflare DNS problem

http://support.cloudflare.com/

by sinak • 11 years ago

60 35 11 years ago

support.cloudflare.com

sinak• 11 years ago

More details here on Cloudflare's status page here: http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

packetized• 11 years ago

Dyn also appears to be having a real, real bad day. Glue records vanished like a fart in the wind.

  $ dig @8.8.8.8 ns2.p29.dynect.net

  ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6 <<>> @8.8.8.8 ns2.p29.dynect.net
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 2690
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

  ;; QUESTION SECTION:
  ;ns2.p29.dynect.net.		IN	A

  ;; AUTHORITY SECTION:
  dynect.net.		600	IN	SOA	ns0.dynamicnetworkservices.net. hostmaster.dyndns.com. 2015070600 10800 1800 604800 1800

  ;; Query time: 2 msec
  ;; SERVER: 8.8.8.8#53(8.8.8.8)
  ;; WHEN: Mon Jul  6 21:42:48 2015
  ;; MSG SIZE  rcvd: 120

mattbeckman• 11 years ago

Does CloudFlare use Dyn or are these unrelated issues?

rdl5• 11 years ago

The affected zones are those cnamed from CloudFlare to Dyn.

kelseydh5• 11 years ago

I love how even https://support.cloudflare.com/ is down by this.

jonny_eh• 11 years ago

Makes me feel good as an impacted user. I know that the problem won't be ignored!

colinbartlett3• 11 years ago

They do at least use a different DNS provider for their status page: http://www.cloudflarestatus.com

Someone12343• 11 years ago

I do love how everything is operational...

I see the big blurb about DNS at the top. But it still feels like the table is misleading. When they're having a major DNS outage and every site is "operational" it feels misleading.

Although if this was Amazon's AWS console we wouldn't even have the blurb at the top...

xxdesmus• 11 years ago

ZenDesk uses Dyn. Hence, it was down. Sorry about that.

rdl5• 11 years ago

Sorry -- our ops and engineering teams are looking into this and will update/resolve as quickly as possible.

https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

iaw• 11 years ago

Good luck, this is really rough. I encountered a 101 on a random site before I saw this and was a bit baffled.

kelseydh5• 11 years ago

Lately my number one bloody cause of downtime has been DNS lookup failures. I switched to Cloudflare hoping to avoid this. If I can't trust Cloudflare, then who the heck can I trust?

Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup? I don't like being at the mercy of their support team while my site has literal downtime.

rdl5• 11 years ago

We should have it fixed very shortly; 99.9% on the cause.

colinbartlett3• 11 years ago

I use DNSimple and they have a feature that lets you sync to another provider.

I have not used it but, as I understand it, the feature helps you setup slave servers which respond to Notify messages from DNSimple. That way, if they go down, your slave provider will keep things up.

laumars• 11 years ago

You could always run your own DNS server. Of course, you'd then lose the other services Cloudflare offer like basic DDoS protection, but that's the trade off you'd have to consider.

elmin• 11 years ago

I've never had any trouble with Amazon Route53.

zrail• 11 years ago

That's because Route53 is basically bullet proof, such that their SLA guarantees 100% availability. I have switched everything I touch to Route53 and have had zero issues.

rdl5• 11 years ago

In this case you would have been unaffected if you used CloudFlare for DNS, rather than using Dyn for DNS and CNAME from CloudFlare to it. Using CloudFlare for DNS is our preferred configuration.

toomuchtodo2• 11 years ago

Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup?

Learn basic troubleshooting so you can determine which vendor is the problem (ie not Cloudflare in this case).

troydavis• 11 years ago

We had the same question and concluded that the only reasonable way to say more than "Well, we hope this doesn't happen again" was using 2 different authoritative DNS providers (both anycasted). Details: http://blog.papertrailapp.com/dns-outage-on-monday-december-...

It's not a question of trust; at some point, any single provider will get DDoSed or have an operational bug.

The only question is whether you're willing to accept that. If you are, then try to pick one great provider.

If you're not, and are willing to spend a little time to avoid this problem, try to pick two great providers. Since then, DNSimple has made primary/secondary providers even easier: https://support.dnsimple.com/articles/secondary-dns/

Basically, if you don't want this problem, don't pick one provider. It's not bulletproof, but it's a lot closer.

kelseydh5• 11 years ago

If you're on Heroku, one immediate way around this problem is to guide your users towards your heroku domain. I've been doing this for clients contacting me who absolutely need access right this moment.

E.g. https://myappname.herokuapp.com

Someone12343• 11 years ago

That's a nice tip.

Can I ask a question while we wait for Cloudflare to fix this... Heroku looks nice. But also quite expensive after you add on dyno, Postgre/Redis, and "addons." What about Heroku do you like/appreciate that justifies the slightly higher costs? Or do you just like that all of the devops-style stuff is handled by them (as opposed to AWS/Azure/Linode, etc where you'd have to do it yourself).

I am genuinely asking, I have no reason to dislike Heroku, and I know it is quite popular. Just trying to understand what the "value add" is over other cloud services (or why they're dissimilar).

Please don't feel obliged to answer, but if you have the time it would be appreciated/interesting.

kelseydh5• 11 years ago

No problem, here's my two (relatively unorganized) two cents:

With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop.

On Heroku you definitely pay for convenience and ease of use, knowing that you get a benefit in doing so by being able to focus on the development of your app, the part that matters. The costs grow if you get traffic, but through careful management (such as packing multiple processes into one dyno), you can reduce costs. Even if many find Heroku's prices high, they are still much lower than what you see in enterprise land, or even on AWS if you aren't careful.

Heroku's documentation is also incredibly good, and this helps you a lot to move fast. Almost by accident, simply by following Heroku documentation I have found that developers end up following good cloud server practices almost by default (i.e. because Heroku forces you to build your app to follow certain conventions, conventions needed for making your app horizontally scaleable; such as delegating clock processes to a separate process, and not allowing dynos to manipulate or rely on unique file memory specific only to them).

If I architectured or configured my own servers it would have taken 10x as long, exposed me to far more security issues, and I would have probably screwed it up any way. The lesson from this is that I think as developers we need far less freedom to configure than we think we need.

So in conclusion, if you're a small to medium-sized app that does not have complex server or bandwidth needs (i.e. you are not Imgur or Pied Piper), then I highly recommend it. I'm sure many Oracle salesman will be unhappy with this opinion, but take my word for it that the vast majority of applications do not need complex, overengineered server systems to be successful. (After all as a startup, you should not be wasting months planning on how you will scale to support 10 million concurrent users. I've seen that hubris distract failing startups before.)

jgrahamc3• 11 years ago

This appears to not be a problem with CloudFlare, but a problem with Dyn: https://www.dynstatus.com/. Of course, Dyn and CloudFlare have mutual customers and so if a Dyn has a problem it's possible for a mutual customer to have a problem.

hosay123• 11 years ago

Looks like your support site might need a little.. support http://i.imgur.com/X76QdaC.png

You might want to try putting it behind a CDN to cope with the load

jgrahamc3• 11 years ago

We use Zendesk. Zendesk uses Dyn. Boom!

stockliasteroid2• 11 years ago

Strange, we are using CNAME apex resolution with cloudflare to point to Heroku apps, so I'm not aware that Dyn is used anywhere in that chain, but resolution is still down for those apps.

edit: resolved now.

jgrahamc3• 11 years ago

    $ dig +short NS heroku.com    
    ns1.p19.dynect.net.
    ns4.p19.dynect.net.
    ns3.p19.dynect.net.
    ns2.p19.dynect.net.

rdl5• 11 years ago

Sorry about this: just posted a blog post. Appears to be a Dyn issue, but affects a lot of our customers (including support.cloudflare.com which is Zendesk)

https://blog.cloudflare.com/the-internet-is-a-cooperative-sy...

Cloudflare DNS problem

Related Stories