How and why I run my own DNS servers

http://bugsplat.info/2012-12-31-how-i-run-my-own-dns.html

by zrail • 14 years ago

145 100 14 years ago

fleitz• 14 years ago

As for the SSH attempts just move it to a different port, disable password logins. SSH shouldn't really be accessible on the same IP as any publicly facing service for your domain. It invites this kind of automated scanning, which is actually a huge risk unless you're on top of your patches. Also change your SSH config to misreport the OS/SSH version.

The IP for SSH access should then be heavily filtered to only SSH and the requisite ICMP packets.

Have you considered using Route53 / Cloudflare?

zrail10• 14 years ago

Thanks, that's a good tip, I'll work on that today.

I had considered Route53 but it's actually in the same ballpark as this setup, money wise, and it's less flexible for what I want it to do. I haven't evaluated Cloudflare at all yet.

pfg• 14 years ago

I'm using CloudFlare (without the proxy/CDN feature) just because of their awesome DNS management. Haven't had any issues and it's got a really neat interface.

pwelch• 14 years ago

Another solution I like is adding fail2ban. It adds a rule to iptables to block the IP after a set number of incorrect attempts and you can set when to unblock it.

ChuckMcM3• 14 years ago

Or fail2ban which puts in iptables rules to ban IPs that fail to login. The ssh scrapers try a variety of user accounts, and it is pretty easy to spot them (for example root can't login on my system from anywhere except the physical console, so trying to login as root,bin,games,demo,Etc from ssh is an instant ban.

Of course you need to not run an open DNS resolver which means 'don't do recursive lookups for anything except your trusted addresses'. I saw one setup that worked really well which was to only do recursive lookups from loopback, then client would create a vpn tunnel to the dns server, get their dns service that way. Seemed a bit extreme but it allows off site DNS service.

jstalin• 14 years ago

I use fail2ban as well. I honestly don't get why people get so worked up about SSH. As long as you have a long, secure password, how is it a security vulnerability?

leephillips• 14 years ago

I removed fail2ban after it prevented me from logging in to my server from a Panera cafe. It took me a long time to figure out what was going on after logging in from a second server: someone had been trying to break in some days before from the IP that I happened to be using. So fail2ban could be a problem if you work from the road.

jacquesm5• 14 years ago

Having a fake open 22 is actually a great way to build up a blocklist.

haxxorfreak• 14 years ago

Until you need to login from a different machine, forget the -p flag and get yourself blacklisted. :)

ramidarigaz2• 14 years ago

I've been running kippo (ssh honeypot) with a really weak root password. It's really entertaining to watch people break in and try to muck around.

ramidarigaz2• 14 years ago

How do you change the reported OS/SSH version? I didn't see any obvious lines in sshd_config that do that.

shredfvz• 14 years ago

+1.

SendEnv in ssh_config? AcceptEnv in sshd_config? No specifics cited in the man pages.

Nick_C2• 14 years ago

Banner and VersionAddendum. Defaults to "none" for both.

[Later] Actually, those two have no effect on my remote Debian 6 VPS. Possibly a Debian "fine-tuning".

Sami_Lehtinen4• 14 years ago

I decided to enable port knocking. So SSH is not available on any port, without right kind of packets being sent beforehand.

hackerboos2• 14 years ago

Security through obscurity.

larrys4• 14 years ago

"To host your own DNS servers the registrars require you to list two IP addresses "

Registrar here. This may be a requirement of some registrars but not a requirement of all registrars. And in any case you can fake this (see warning) by simply using the same IP address with a different host name or by using a different IP address with a made up IP address.

Warning: Obviously you should have two valid DNS servers no doubt. I am simply pointing out that there are work arounds to this especially if you are trying to get up and running and won't have the 2nd dns operational for a short period of time you can at least get started with DNS that should work (since it only takes one and assuming that one is working you will be serving up records.)

koide• 14 years ago

And there are some registrars that offer a secondary DNS free of additional charge just for this case.

onetwothreefour2• 14 years ago

Some ccTLDs actually require 4 NS servers on different subnets.

I'll leave you to guess which one. :)

thaumaturgy4• 14 years ago

Mind if I ask which registrar? Just curious. My email address is in my profile.

jaytaylor• 14 years ago

I'd also like to know which reg..

Jay [at> jaytaylor dot com

dsl2• 14 years ago

As a registrar you should know this is actually a requirement of the registry and varies based on TLD policy. (maybe you meant you're a reseller?)

larrys4• 14 years ago

The registry has one requirement.

The registrar could have another one.

.com .net "require" only 1 nameserver (Verisign). PIR/Afilias require two (hence the need to fake or use a real one).

Please note also that my parent comment was with regards to the statement "the registrars require you to list two IP addresses" not why that may or may not have been required by the registry.

It's possible of course that even though .com .net only require one nameserver, a registrar might require two for some reason. It's also possible (but unlikely) that even though .org requires two the registrar could take care of this by putting in a dummy record if they wanted and only requiring one of the customer.

End users don't deal with the registry. They deal with the registrar.

We are an ICANN accredited registrar.

jstanley• 14 years ago

"To host your own DNS servers the registrars require you to list two IP addresses with the idea that you'll be providing redundant service. The one thing you don't want is downtime with DNS, it screws everything up."

I've got around this by having two hostnames for one machine.

The way I see it, since all my services are on this one machine anyway, I'll have bigger problems than DNS if the machine is down: there will be nothing at the address the names are pointing to, so who cares if they resolve properly?

zrail10• 14 years ago

Fair point. I've got email and DNS distributed onto both machines so I don't lose anything, and prgmr was pretty clear that I couldn't have a second IP just to fake out DNS so I went with a whole second server.

eCa• 14 years ago

That sounds weird (what prgmr said). I've had dedicated servers running dns at several providers for ten years, and all I've ever done is get a second ip for the same machine and bind it to a different hostname.

uxp• 14 years ago

Also, if your email box goes down and is unreachable then the sending server should retry later when it can't get a response. If that box is also your DNS server, and the sending email server is unable to lookup the MX record it will automatically bounce. Having a slave/secondary email server should mitigate this to some extent though.

Many applications do plan for unreachable services, but throwing the fault down the stack screws up how the error is handled. You can still use a primary application server box as a DNS server, but I would be certain that it's a secondary/slave server, not the primary server.

lsc• 14 years ago

I wasn't trying to sell you another server. conventional wisdom is that you should have your secondary DNS on a different server, on a different network, run by a different provider, not another server with me.

andreasvc3• 14 years ago

I think it's fairer to say you got around to it because they're not checking.

Even with a single machine it's still better to have redundant DNS. You want to be able to move around the machine to a new connection / IP, without having to wait for the registrars to update the IP of the DNS server; similarly when your one DNS provider is DDOSed, your site will stop working, but not if you had 2 different providers. So to answer your question: you as the operator will care, because you want to get things back up quickly ...

I think for small servers it's much preferable to run a hidden primary (not publicly listed in DNS), and let secondary DNS servers update from there. That way you have complete control over your DNS, even when your machine goes down.

rachelbythebay• 14 years ago

Those "machines" seem huge for what they're doing (DNS services for 32 zones). They add up to more resources than all of my machines put together from just a few years ago.

I guess this is progress. It just seems strange to me.

zrail10• 14 years ago

Yeah they're way too beefy for what I'm using them for at the moment. My ongoing plan is to move various services onto these machines but it's been slow going.

tom_fitzhenry3• 14 years ago

To further improve reliability, consider using a secondary-as-a-service (as well as your self-hosted secondary name service). This will only work if tinydns supports zone transfers (AXFR/IXFR), which as far as I can tell, it doesn't.

I don't use a self-hosted secondary. I use a secondary-as-a-service as my secondary /and primary/, which has the advantage of my VPS going down (for less than 4 weeks) wouldn't impact my DNS's uptime.

I blogged about it this month: https://tom-fitzhenry.me.uk/blog/2012/12/host-your-own-dns-w...

zrail10• 14 years ago

Before I picked up the Ramnode server I was using BuddyNS as my primary with the prgmr node as the stealth primary. It worked fine but it was less straightforward than I wanted.

joe_bleau• 14 years ago

I'm using prgmr and buddyns as secondaries to my djbdns primary. axfrdns (part of the djbdns package) handles the zone transfer requests, and I can force a transfer with a really simple perl script.

tom_fitzhenry3• 14 years ago

Though the stealth primary solution involves more moving pieces, and introduces zone transfers, which are more complex than provisioning via puppet, I've had no noticeable issues.

I say "noticeable issues", because although my DNS is monitored, the monitoring isn't granular enough to detect sub-minute outages, for example.

Did you have any issues in particular with the stealth primary solution?

perssontm• 14 years ago

Any idea where ramnodes datacenter is? And perhaps an ip address to try some traceroutes there?

arantius• 14 years ago

> This will only work if tinydns supports zone transfers (AXFR/IXFR), which as far as I can tell, it doesn't.

http://cr.yp.to/djbdns/axfrdns.html

It does. I'm doing this currently, I host primary DNS myself with tinydns, and the makefile sends a notify to the registrar-hosted slave DNS, which gets data via this daemon (with tcp server configured to only allow that IP).

Sami_Lehtinen4• 14 years ago

Why bother using Postfix for mail forwarding? Why not to use Gmail MX directly?

I run my own servers with postfix, dovecot and roundcube. Therefore I don't need Gmail for anything. (It's also huge privacy issue.)

zrail10• 14 years ago

I don't use google apps for my domains so Gmail MX wouldn't really work, plus I have some aliases that I think are maybe too complicated for Gmail. What do you think of roundcube? I haven't tried it yet, since php puts me off.

Sami_Lehtinen4• 14 years ago

Yeah, roundcube is the only reason I have installed PHP. I really would prefer not to use PHP. I don't personally like roundcube too much. For sure it's not great, but it does still work okish. I'm usually using proper mail client(s), so roundcube is just a backup. Btw. roundcube is also the reason why I'm running MySQL too. I'm using Python 3 & MongoDB with all of my own apps.

aioprisan4• 14 years ago

What aliases could be too complicated for Gmail? Roundcube is an OK web email client, but let's be honest, it doesn't even match up what Gmail can offer you, in terms of integrated services like Google Docs, free phone dialing, video conferencing, etc.

andreasvc3• 14 years ago

RoundCube is fine. I wonder though, why there isn't a good webmail client in Python or Ruby.

kalleboo• 14 years ago

I switched from running my own servers to GMail. It's easily worth $5/mo to not have to worry about security bulletins, kernel updates, etc. Self-hosting is fine for stuff like my personal website, but I live and die by my email.

madsushi• 14 years ago

I run my mail through Postfix and have it listen for special code words/triggers and then perform special actions (like restarting nginx or Apache).

kjackson2012• 14 years ago

I ran my own DNS server from my own home DSL static IP address 10+ years ago. It's fairly innocuous as long as you only respond to DNS requests for your own domain and don't forward requests. But it ends up being more hassle than its worth.

As recently as a few months ago, I hosted my own DNS server on a Linksys SLUG, but after I suffered a power outage and realized that there's no reason for hosting this stuff myself, I just decided to use namecheap.com for their Free DNS service.

iSloth• 14 years ago

Personally I would have gone with a dedicated server from somewhere like OVH, about the same price as those VPS's with similar resources (but they are dedicated to you...)

As for SSH, just move it on to a port other than 22, that will fix 99% of the bots trying to guess your password.

MidnighToker• 14 years ago

been using ovh for years and really enjoy their self-manage tools (though I actually work for a different provider).

fail2ban || denyhosts are also good ways of stopping ssh brute force :)

aioprisan4• 14 years ago

Route53 is a much better alternative to running your own service. I used to have a similar setup for taskup.com and other sites that I'm hosting but for the $0.20 that I pay every month to Amazon, I get weighted DNS responses based on the region that the request comes from and requests get served to the local server for that region, redundancy in 12+ zones, etc. The latency alone for a DNS lookup from India was about 300ms to our server in PA, but now it's under 30ms. Having said that, it's a good learning opportunity.

onetwothreefour2• 14 years ago

There's literally no reason to ever run your own DNS when Route53 and DNSMadeEasy exist.

lesterbuck• 14 years ago

My reading of the Route53 prices is that you pay $.50/month/domain, plus query charges after a million. How do you pay $0.20/month? I find it very irritating that Amazon says "Pay only for what you use. There is no minimum fee." when there obviously is a minimum fee of $.50/month, even if zero queries are resolved. There are entire low end VPS servers that cost only marginally more ($14/yr) than what Route53 costs for one lousy domain, i.e. one entry in a database table.

aioprisan4• 14 years ago

sorry, that was supposed to be $1.50/month

aioprisan4• 14 years ago

with Route53, it's not just another entry in a DNS zone file, you actually get multiple-zone IP addresses for each domain hosted, by default. Also, you get WRR and anycast, as well as latency based routing. Not to mention that you can create aliases that are not visible to resolvers but point to internal Amazon resources that you have access to.

rscale• 14 years ago

I switched to Route 53 (from TinyDNS with custom RoR/Postgres frontend) and I'm completely satisfied. The bills are tiny, the service has been reliable, the console lets me make all our desired records without a lot of hoop-jumping, and somebody else maintains everything. Migration was a one-day project, including DR.

There are some added bonuses if you happen to be using other AWS services, but even without those, I think Route53 is one of Amazon's better offerings.

chernevik2• 14 years ago

I do this myself off a Linode. It's neat to understand how this stuff actually works, it expands the imagination of what's possible. I like being able to quickly stand up a subdomain for development / testing / staging. I really want to get an email server running, I think the ability to spark up email addresses and servers for automated reporting or file distribution opens up some useful possibilities.

dotBen• 14 years ago

As it goes, Linode has an amazing DNS service bundled with their instances - replicated between all their international datacenters.

If you host with Linode I can't see why you wouldn't use it, unless you are just futzing/curious to roll your own.

vivab0rg• 14 years ago

+1 for Linode's DNS service. The best part is that you can access it via API too.

chernevik2• 14 years ago

I'm sure it's awesome, I like Linode a lot, but the point is to learn how all this works.

thaumaturgy4• 14 years ago

I also run my own domain name servers. I use PowerDNS with a MySQL backend; setting up replication between the two MySQL servers wasn't too hard, which gives me redundant name servers in two data centers. I have a few more than 32 zones, but the name services stuff has almost zero impact on server resources.

Updating and querying the name records is a piece of cake thanks to the MySQL database, and if I ever get around to finishing it, I can have my own web-based front-end for it.

Best of all, I can have a nice, short ttl on all of my entries, and PowerDNS is configured to always check the database, so I have a two minute refresh period on any changes made to entries on my name servers -- that alone has been well worth running my own. (Client: "Don't I have to wait like a few hours or a day or something for this to spread over the internet or something?" "Nope. 'Bout two minutes in most cases, little bit more for AT&T's customers.")

I definitely didn't do it right the first time, though. DNS has a few gotchas, like making sure you disable axfr requests, making sure you have an SOA record for appropriate zones, and so on. I'm probably still doing something wrong, but I don't know what it is at this point. Also, the default recommended PowerDNS MySQL setup is needlessly complex, as far as I can tell.

larrys4• 14 years ago

"and if I ever get around to finishing it, I can have my own web-based front-end for it."

If you finish it let me know. I would be interested in possibly buying the front end from you (even if it's rough wouldn't be customer facing.)

druiid3• 14 years ago

There's several existing MySQL management solutions for powerDNS, including: https://www.poweradmin.org/trac/ https://github.com/averna-syd/PowerdnsTango

PowerDNS Tango is the prettiest looking one.

basilgohar2• 14 years ago

I've rolled my own very basic PHP frontend to my own SQLite-backended self-hosted PowerDNS setup. If there is serious interest, I might just throw it up on Gitorious or Github as I've always intended.

mrb• 14 years ago

I use PowerDNS too, but with BIND flat files as a backend. Setting up replication is even easier than MySQL: I just rsync the flat files from a central server to my 2 DNS servers whenever I make changes. PowerDNS automatically reloads the files based on their mtime.

thaumaturgy4• 14 years ago

Yeah, I'd considered that, and there are some serious advantages to flat files. To be honest, relying on MySQL for production anything makes me nervous. But: SQL queries occasionally save my butt when needing to make lots of changes all at once, instead of having to figure out sed again or edit lots of files; and, I wanted something that would sync automatically, since I'm an idiot and sometimes forget to do something like an rsync command after a change.

jcampbell12• 14 years ago

I don't understand why you use short TTLs. Do you care that every visitor has to wait an extra 100ms or so to see the page they came for? I personally want all my DNS records at the ISP.

Sure, Google can get away with a 600 second TTL because there is so much traffic that only one person in a million will have to wait.

thaumaturgy4• 14 years ago

I already explained why I use short TTLs. Let me try putting it another way: because it solves more problems than it creates. That's the same yardstick I use for all my infrastructure decisions, because sure 'nuff, no matter what I decide, someone is going to think it was the wrong decision and that I should care about their personal opinion.

Do I care that every visitor has to wait a little bit more every few pages or so? Yes, of course I do. But you fail to understand our business or customers at all when you think that a delay caused by a short ttl in dns is a problem that we need to address right now.

A much bigger problem for our customers is when they call or email and ask for help with some kind of hosting change -- moving onto or off of our servers or changing some service or some such thing -- and they get told that there will be a period of minutes or hours or maybe even a day or two where visitors will see either the new version or the old version and no, we can't say which.

When we tell people that, yes, we can change their mail services for them, and by the way, that change should be instantaneous for most people, they are delighted. And delighting our customers is what's most important to us.

Further, you fail to realize that most of our hosting customers are coming from services like BlueHost or GoDaddy, where their sites have probably been living on overprovisioned servers and so they get better performance with us anyway, which also delights them.

You also don't realize that our average customer, so far, is running a poorly optimized WordPress or Joomla installation, and that either of those systems introduces page delays that are far more severe than a DNS lookup. In other words, our customers are not currently people that are looking for the fastest possible website; they merely want it to be "fast" -- which usually means, "renders in about a second or less on my crappy 1.5Mb DSL" -- but mostly, they want it to always be up and they never want to worry about backups or hackers or phishing or viruses or anything else.

So, yeah, once short ttls start to cause problems -- a la an email from a customer saying, "hey, I just profiled my site, and I noticed it's taking an extra 132 ms to load because of a dns lookup, can you check into that" (a message which we haven't even come close to receiving yet) -- then that'll become a bigger issue and I'll address that instead of whatever else I'm working on at the time.

I know I shouldn't get irritated at comments like yours -- I should simply ignore them -- but they bug the hell out of me because you probably don't even realize what the effect is. You're not sharing helpful information; you're not telling me something about short ttls that I didn't already know; you're not telling me about some bug in PowerDNS caused by ttls less than 194 or that ttls less than 233 can cause issues with MySQL if you have more than pi requests/second. The only thing that comments like this do is further demotivate me from sharing information like this in the future. My thought process before commenting in a thread about server configuration is already something like, "Hey, should I bother to mention how we do this? It was a bit fiddly and there's some bad documentation out there. Someone might ask a question that I could answer. Someone might mention something I don't know about. That could be good. But probably someone's just going to tell me I'm wrong. Am I in the mood for that? Nah, not really."

Sorry for slamming you with a wall-o-text response. I had thought about the last time KeepAlive was mentioned in an HN thread before posting my comment about PowerDNS; you just happened to make me feel like an idiot for not listening to my instincts and deciding to not comment in this thread.

druiid3• 14 years ago

For many many years we ran our own DNS servers using PowerDNS with MySQL backends. That's all fine and well and it's a very powerful architecture and relatively reliable and strong to even high amounts of traffic... that was until we started regularly seeing 10Gbps+ DDoS attacks, so we put a DDoS protection service in front of the sites, but there wasn't really anything at the time capable of protecting the DNS servers as well.

So a few months go by, everything has been fine and then another DDoS hits. Looking through the web-servers, not hitting there, all is fine. Study firewall traffic through the network and note that the majority of incoming traffic (that was making it through to the network that is), was headed to the DNS servers. The attackers were sending the DDoS to the DNS servers, requesting some of our root domains and given that it appeared as valid traffic there was not much to be done to filter any of it. The easy solution in this case happened to be blocking all Chinese and Russian IP's for a couple days which mitigated enough of it to solve. After that, we stopped hosting our own DNS.

Moral of the story: DNS services are a great learning tool to use and PowerDNS is probably where you want to start, but at scale you may or may not want to actually run your own.

larrys4• 14 years ago

(Thanks for the links you provided to the front ends in your other comment).

Would like to point out that ddos is a ymmv item obviously for anyone scared off by the potential for this which is the same as the potential for any ddos attack and mitigation. A company affiliated with ours has been managing dns for a customer since 1996 with 10k zones. Back when you had to read Cricket Liu Oreilly book. (movie.edu if anyone remembers). They've never experienced a problem. All run on pretty low powered hardware for that matter.

That said there are things to know about dns while not difficult time could be better spent elsewhere if there is no compelling reason to diy.

dsl2• 14 years ago

DDoS risk is based on the type of business you're running (online casino, ecommerece, etc) and not really much to do with if you are running infrastructure in house or not.

For that matter, I don't know of a reputable DDoS mitigation service that doesn't provide DNS hosting as part of the package. It's pretty core to the whole thing.

druiid3• 14 years ago

Indeed, but what I was mostly getting at is that you have another possible attack vector if your run your own DNS, while services that provide it for you are basically pennies on the dollar compared to potential losses.

jspthrowaway2• 14 years ago

You'd be surprised how often your first assertion doesn't hold, actually.

Nick_C2• 14 years ago

Has anyone tried the PaperTrail service? I'm tempted except for the tiny window for queries, only a week. I'd like, say, a month at least.

zrail10• 14 years ago

Once you sign up you can customize your plan as much as you want, setting search retention to a month is a possibility. It's not cheap, though.

spc476• 14 years ago

I run my own DNS, but in a rather unique way. The registrar record points to my hosting company's DNS servers, but they are slaved off my DNS server; the only IPs that can hit my DNS server are the hosting company's. I make the changes I want, they're pushed out to the actual Internet facing DNS servers.

lvh• 14 years ago

Twisted[1] comes with a DNS server built-in.

For non-authorative local DNS caches:

    twistd dns --recursive --cache

For authorative servers, to serve a BIND-style zonefile:

    twistd dns --bindzone ZONEFILE

But, of course, that's pretty boring. The fun stuff is when you have a Python source file as your zonefile! See: https://twistedmatrix.com/documents/current/names/howto/name...

twisted.names (that's the name of the DNS package) is simple enough to use for DNS serving (demonstrated above) and flexible enough to get it to do pretty much whatever you want.

How and why I run my own DNS servers

Related Stories