Let's Make DNS Outage Suck Less

http://kvz.io/blog/2013/03/27/poormans-way-to-decent-dns-failover/

by kvz • 13 years ago

37 14 13 years ago

kvz.io

trotsky• 13 years ago

dnsmasq --interface=lo --all-servers --server=172.16.0.23 --server=8.8.8.8 --server=4.2.2.2

echo "nameserver 127.0.0.1" > /etc/resolv.conf

http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

--all-servers : By default, when dnsmasq has more than one upstream server available, it will send queries to just one server. Setting this flag forces dnsmasq to send all queries to all available servers. The reply from the server which answers first will be returned to the original requester.

Dnsmasq also has avoidance of unresponsive servers built in, this is a bit more of a blunt instrument.

kvz5• 13 years ago

Thanks for the link! I did find dnrd (http://dnrd.sourceforge.net/), but I really wasn't looking into proxying DNS requests and introducing another daemon / dependency, that could at some point fail or get confused, requiring even more tech to keep it alive, or provide insight in it's workings/reasoning.

Really wanted to solve this using as transparent & archaic tech as possible in the hope this still stands when big player's resolving servers fall.

Granted, it is blunt; archaic tools often are ; )

sigil• 13 years ago

The example you gave won't stick for most people:

    echo "nameserver 127.0.0.1" > /etc/resolv.conf

If you use dhcp3-client (chances are you do), add this line to /etc/dhcp/dhcplient.conf to try your local DNS cache first:

    prepend domain-name-servers 127.0.0.1;

Note that this will slow down stuff like captive portals at airports that want to push you to their "accept terms" page first.

I use djb's dnscache instead of dnsmasq, but dnsmasq works fine too.

http://cr.yp.to/djbdns/run-cache.html

jefe782• 13 years ago

They're using DNS for DB communication. That's a major faux pas.

kvz5• 13 years ago

Using IPs for communication to Amazon RDS instances is inadvisable. You'd have to use WAN addresses outside of EC2, and LAN addresses inside of EC2. Addressing them by domain resolves this. Additionally I don't think Amazon guarantees your box will be accessible on the same IP addresses after Multi-AZ failovers.

jefe782• 13 years ago

You're definitely doing it wrong.

datums3• 13 years ago

If you use the rotate option with timeout you'll quickly hop to the next working resolver. http://edwin.io/optimized-resolv-conf

Do you have data you can share regarding "Amazon EC2 resolving nameserver (172.16.0.23) is unreachable too often" ?

dsr_• 13 years ago

You don't need to do rotation, either -- suppose the first nameserver, when up, is consistently faster than the others. Just setting a timeout of 1 is appropriate, then.

kvz5• 13 years ago

A timeout of 1 can result in false positives. 3 seconds is recommended according to the manpage. Should the occasional false positive be none of your concern, then you still add at least a second to every request. If you make many, that's many seconds.

datums3• 13 years ago

This is more of a performance optimization and less about a HA solution. With rotation you also avoid being throttled. If that's not a concern based on the # of request you make then you can simply timeout and go to the next resolver on the list.

kvz5• 13 years ago

From the manpage: "[rotate] causes round robin selection of nameservers from among those listed". This means in your configuration, the broken nameserver will be hit once every 3 times, resulting in a 1second timeout before trying the other 2. So in every 9 requests your setup results in 3 seconds delay for as long as the nameserver is down.

In case of Amazon, you actually prefer their nameserver as it resolves names to LAN IPs where possible. With your strategy, ~66% of my instance hostnames would be resolved to the less efficient WAN IPs, that's with all 3 nameservers being reachable.

As for data, there's papertrail and Premium Support tickets with Amazon admitting to their downtimes. In most cases (e.g. the one on Jan 14, 2013 11:27 PM PST) "there was a problem with the underlying host".

Remember it doesn't even need to be the service itself going down, there's weaker links in this chain like misconfiguration of the virtual host. Cables can get tripped over. Anycast is not possible here.

I'm not blaming them. This article was about mitigating suckyness in case of outages, not proving that such a thing even exists : )

datums3• 13 years ago

So there are 2 different things being addressed. There's a HA solution (handle failovers) and then there's operational performance. Introducing cron to manage this solution, means cron needs to be running and monitored at all times. Also worse case scenario if cron dies and you have a now down nameserver configured.

Agreed on aws I would use their resolver to request the internal ip. Another option is setting up your environment in VPC. With VPC you'll be able to assign a static internal ip to your instance.

btw I did find many reports on the 172.16.0.23 lookup issues.

3amOpsGuy• 13 years ago

You know, sometimes the simpler solutions work best. I like this one.

I'm sitting here contrasting this against my normal approach* and the approach here is:

    1) easier to explain to others
    2) self documenting
    3) just as effective as running a caching nameserver

* i have vm hosts configured in CM to have a bunch of promises applied, one of them is to run a caching nameserver, these hosts are the only ones allowed to do zone transfers. The vm instances running on top of these have a promise applied which has them use their underlying dom0 for dns queries.

Let's Make DNS Outage Suck Less

Related Stories