DNSimple target of DDoS attack

https://twitter.com/dnsimple/status/341538008941613056

by markprovan • 13 years ago

64 28 13 years ago

twitter.com

dexcs• 13 years ago

That makes sense. rubygems.org was down for me for a few minutes....

davidradcliffe• 13 years ago

We're working on adding some redundancy!

thejosh• 13 years ago

This seems to happen quite often to DNS hosts, I remember ClouDNS getting hit often.

It's easier to hit these sorts of "smaller player DNS hosts" if the website you want to take down is otherwise protected?

relix• 13 years ago

In the three years I've been with DNSMadeEasy [0], haven't had an outage yet.

[0] https://cp.dnsmadeeasy.com/u/62796 (affiliate link)

yesimahuman3• 13 years ago

Ah, got bitten by this. Just added Route 53 as redundancy, should have done that a long time ago.

whafro3• 13 years ago

Are you doing this manually? I make solid use of DNSimple's ALIAS record type, which is also supported by Route 53, but DNSimple says they don't support AXFRs (http://support.dnsimple.com/articles/master-slave-support).

yesimahuman3• 13 years ago

Honestly, I'm not a DNS pro, but I added more nameservers in our registrar to point to R53 nameservers, then setup R53 to point to our Heroku app w/ alias support by following this guide: https://devcenter.heroku.com/articles/route-53

nate2• 13 years ago

Are you using route53 to do an alias AND have the root domain use SSL? The setup at route53 is confounding me. I thought I'd setup a CNAME www.domain.com that points to sub.herokussl.com, and than an A record Alias that points domain.com to www.domain.com. But Amazon seems to not avow that the CNAME record is a "Record Set" that I can use for this.

whafro3• 13 years ago

That's interesting. Have you tried it the other way around?

- domain.com ALIAS sub.herokussl.com - www.domain.com CNAME domain.com

yesimahuman3• 13 years ago

Did you set up the S3 bucket with static file hosting redirecting to the www subdomain, before you try adding the alias in 53?

soci2• 13 years ago

DNSimple seems the best way to go if you want to host your service with Heroku using a root domain (no www at the beginning of the domain name) [1].

Unfortunately, DNSimple is now the weakest layer of our stack. And at http://KiteBit we are suffering it right now!

[1]https://devcenter.heroku.com/articles/custom-domains#root-do...)

robotmay4• 13 years ago

Yep, I just got whacked by this at https://www.photographer.io (www is working at least). Guess I've learnt my lesson about using the root domain.

However I'm glad this happened whilst I was still beta testing! My CloudFront stack was pointing at the root domain, which was stupid. Fixed that now.

soci2• 13 years ago

We were not so lucky.

We have a URL forwarding set to www that points to out root domain so we are completely down.

jwarzech• 13 years ago

Our site (and others) seem to still be working through the 'www' domain. We have had nothing but great experiences with DNSimple up to this point and will probably stay customers, just sort of frustrating as we wait for our domain to resolve to another dns provider as a quick fix...

robotmay4• 13 years ago

I'm having an interesting issue where my root domain is currently getting its nameserver details from DNSimple, whereas www. is getting them from iWantMyName. Hopefully that rotates around too at some point soon :\

randall• 13 years ago

Anyone know how to make DNS redundant? Is it as simple as adding them as extra nameservers, and then copying all the records? I'm thinking about using Linode's DNS as a failover.

latch• 13 years ago

Yes. You can do master-slave with AXFR but it doesn't seem very well supported nowadays. I remember 1 provider that let you put in your AWS credentials and they'd sync to Route53 using the API, though I can't find which one right now. Different providers offer different solutions...a lot don't offer anything.

Of course, you can always do it manually..which is fine if you have few records and they are static.

edit: Kept googling and I did find http://www.dnsly.net/ neat enough that I wanted to share it, even though I think dns providers should be doing this as part of their existing packages.

Also, although it's no guarantee, dns providers that use anycast are less susceptible (but not invulnerable) to ddos.

colmmacc• 13 years ago

There are two important steps;

1. Adding NS records to the parent zone via your registrar. E.g. if you are using example.com, when you add nameservers with your registrar they add them to the ".com" zone.

2. Update the NS records in your own copies of the zone on your DNS providers.

If your registrar is also one of your DNS providers, then both of these steps are sometimes handled in one action from your registrar - but you still need to update the NS records on the other provider.

"NS" record sets are special in DNS in that there is a copy of the NS record for a particular zone in both the parent zone and the child zone. About 8% of resolvers consider the parent zone's copy the one that matters, the other 92% honour whatever is in the child zone's.

This can lead to confusing cases where you have different NS configurations on different providers - the resolver may "stick" to whichever one it found first (as long as both providers are in the parent zone). DNS can be maddening!

Full-disclosure: I'm a Route 53 developer.

robotmay4• 13 years ago

This is unfortunately a bit tricky as DNSimple doesn't let you update your NS records for the root domain with them. My computer seems to be going through DNSimple still, but Pingdom went via iWantMyName to find my new Route 53 nameservers. Not sure what I can do for now though.

whafro3• 13 years ago

It's not a great long-term solution, but DNSimple has been disabling their ALIAS support, which many Heroku and AWS users are likely to depend on, especially if you use SSL.

Short term, keep your ALIAS record and add an additional A record for your root domain pointing to one of the IPs indicated by your hostname. DNSimple says they'll treat the A record as a fallback when ALIAS isn't working, and will return both sets of records when it is (https://twitter.com/dnsimple/status/341574753276002304).

For the next 3/12/24/96 hours or however long it takes for the threat to subside, this should increase your availability, and the likelihood that your A record will work for that time is probably reasonable. Longer term, you'll want to get rid of the A record.

zrail3• 13 years ago

Is anyone else seeing a sustained level of trashy DNS queries to their own servers? I've been seeing a sustained level to mine that's way above normal, for the last few days. I wonder if this is a broader problem than just DNSimple.

devicenull• 13 years ago

That's probably someone attempting to you for a DDOS reflection. Take a look at http://openresolverproject.org/ and make sure you're not providing an open resolver to the internet

zrail3• 13 years ago

Definitely not in the list for either server.

dsl• 13 years ago

Can you give an example of what 'trashy' is?

You could be an open resolver being used for reflection. You could be running DNSSEC and providing an amplification vector. You could be getting queries for another DNS server that used to have your IP. The possibilities are endless.

zrail3• 13 years ago

I'm running tinydns so it's definitely not DNSSEC or open resolving. The queries are for my domains, so it's not that they're questionable. The problem is that there are millions of them all from the same IP. So, it could be that it's some kind of attack, but I really don't know.

I'm planning on moving hosting over to Route53 at some point in the next few days, because the fact that I can't figure this out other than "turn off logging" tells me I probably shouldn't be running DNS servers.

the9to5• 13 years ago

Bravo to the folks at DNSimple for being on top of communications during this time. It's something that they themselves brought up less then a year ago during the Zerigo DDOS (https://news.ycombinator.com/item?id=4280515) so it'd good to see them sticking to it.

But it also seems as though the same advice proposed in that thread should have been used by their customers: Namely, utilize multiple DNS providers to mitigate risk, and choose providers with IP anycast. Heck, even setting up your own secondary DNS on a $5/mo cloud server would keep your site up (unless of course your site is the main target of the DDOS).

DNSimple target of DDoS attack

Related Stories