Network Solutions' DNS was down

http://www.networksolutions.com

by mikegirouard • 13 years ago

72 61 13 years ago

networksolutions.com

dholowiski5• 13 years ago

Oh crap. Today is going to be a bad day for me.

jryce2• 13 years ago

If the mail server goes down again it'll be a bad day for us too.

dholowiski5• 13 years ago

We run our own mail server, but the MX records come from network solutions :(

jryce2• 13 years ago

Their mail server appears to be online.

billsinc• 13 years ago

Seeing this as well, appears to be DNS only at this point.

mikegirouard3• 13 years ago

You're right. Title updated.

esmale• 13 years ago

I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.

mikegirouard3• 13 years ago

Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP.

It doesn't matter how long I do this... DNS problems always get me.

dedward5• 13 years ago

Have you thought about setting up with an external DNS monitoring service? It's cheap and really comes in handy over the years.

astrodust• 13 years ago

And to think you paid extra for this service.

supine2• 13 years ago

If your name servers are *.worldnic.com it appears you are affected.

First mention on Twitter was an hour ago https://twitter.com/thefrost/status/357483942082920449

mikegirouard3• 13 years ago

This[0] FB post is the best I can come up with as an explanation.

Yesterday, some Network Solutions customer sites were compromised.

The funny thing is, they have a link in their post going back to their site, which of course doesn't work.

[0] https://www.facebook.com/networksolutions/posts/101514668014...

muraiki2• 13 years ago

That explains my friend's site yesterday. Requests to his site sometimes resulted in the display of a banner page by "Islamic Ghosts Team," but not all the time. I noticed that Network Solutions was apparently running Apache 2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).

theg2• 13 years ago

We're slowly coming back online, there are reports of a (D)DOS attack against them but I can't find a source.

TranceMan2• 13 years ago

Some talk here [1] of a recorded message from their support number that they are currently experiencing a DOS attack.

1. http://www.webhostingtalk.com/showthread.php?t=1285835

pilif2• 13 years ago

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers.

Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them).

When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data.

Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.

dholowiski5• 13 years ago

I thought that even if you self host - isn't it your registrar that points the dns records to your dns servers? Would someone with self-hosted DNS still be working in this situation? (doesn't the DNS query first go to network solutions, who refers it to your name servers?).

I'm not trying to argue, I genuinely need to know the answer as I'll have to explain it to many other people, later today...

mgkimsal2• 13 years ago

I self-host for most things. I live on the edge and don't have backup DNS. For most situations, you'd want backup DNS as well.

ns1.kimsal.com ns2.kimsal.com nsA.offsite1.net nsB.offsite1.net

etc. the kimsal ones are my primary/secondary, but having the others offsite, slaving from the primary, would mean that even if my primaries went down, the info would be at the 'offsite' ones. Those 'offsite' DNS servers would need to be registered with the registrar as well, and become part of your record (you can check that with a whois).

jbert• 13 years ago

No, they just go into the zone.

You run a server for example.net. Your server will respond to any requests sent to it for records in that zone.

Someone attempting to reach your site will query the parent zone (.com) for NS records for your zone. So they will query for NS records for example.com against a server which is authoratitive for .com.

(How do they find .com server? They ask the root servers. How do they find the root servers? That's the bootstrap info.)

supine2• 13 years ago

No, this is for domains where .worldnic.com is the authorative server. If you registered your name through NetSol but hosted the DNS servers elsewhere then you would just be relying on the respective TLD servers (e.g. .gtld-servers.net for .com domains) to refer resolvers to the correct authorative servers.

dedward5• 13 years ago

One aspect of Network Solutions business is managing some of the root zones. Those are the servers that ultimately tell the internet what other servers to go look at to resolve queries. These are the "root" nameservers. So while Network Solutions is in charge of the root zones themselves, and does run a bunch of root nameservers, there are other root nameservers run by Various organizations and collectively they can handle a massive amount of work. That is not what is being attacked. If somone DID successfully DDOS a large enough portion of the root nameserver infrastructure, the internet would largely grind to a halt. It is designed suitably sized, and run by large enough players that this is not likely.

The DNS service that is being attacked is the one they offer end users, usually along with domain registration. There is no particular reason you need to use their services when registering a domain with them, although, like most rigistrars, their website will make this deliberately confusing, and may require you to use their DNS service if you also want to use some of their other value-added services along with your registration (like email forwarding, websites, stuff like that).

Unless you are using other services from NetSol that require you to use their DNS, you are free to use whatever DNS infrastructure you like, including your own.

(If DDOS is a business concern, I'd go with someone big who deliberately does just DNS.. Neustar or one of those)

esw• 13 years ago

This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.

pilif2• 13 years ago

I would assume that sites affected by a Network Solutions downtime would qualify as small(ish) sites because the big ones have other solutions anyways. Also, it's much easier to deal with a DOS on DNS than with a DOS on your app servers. If you can handle it for your app servers, you'll be able to handle if for DNS.

If not, well, then you're down anyways.

dsl2• 13 years ago

FYI, Zerigo DNS is a joke.

They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707

thejosh3• 13 years ago

What happens when someone doesn't like you and decides to DDoS your DNS servers?

Though if it's only your site on the DNS Servers, they might as well just DDoS your website.

nevir• 13 years ago

Well, what happens if someone doesn't like your registrar and DDoS' them?

Do both.

druiid3• 13 years ago

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was still enough to saturate the connection with our provider at the time.

This is only one of many possible scenarios with self-hosted DNS. Essentially there are services out there (generally Anycast are best) that do only DNS and have had very high up-time. I'd suggest if you are anything over a 'small' target to look into these services. Sadly it's one of the few things I basically suggest not self-hosting at this point simply due to the unrealistic requirements to scale it yourself under attack scenarios.

ChuckMcM2• 13 years ago

This has been my experience as well. Stupid bots being denied recursive requests. I modified my fail2ban script to autoban them which helped considerably.

rhizome• 13 years ago

This DDoS argument really is a premature optimization, and it's very simple to implement an external primary/secondary if the problem ever does occur.

dedward5• 13 years ago

To Rhizomes point: Yes, it's relatively easy to re-point them to somewhere else.. and if your business can tolerate the interruption that's fine.

Those changes, though, take time to cache out, and depending on your registrar's procedures and whatnot, you could be looking at hours or a couple of days to get things to stabilize back to normal; not to mention finding a new provider and getting things set up. You'll need ot make sure you have the zone data on hand ready to get set up with a new provider, and so on. You could have a procedure in place to do this, sure, with everything tested and validated, but at that point it's much easier to just have a DNS provider of the proper class who can handle these things for you. And keep in mind, if it's your domain they are trying to shut down, and not your DNS provider specifically, the attack will very likely just follow the change, leaving you in quite a mess.

So the short answer: self-host if it's not revenue impacting. If there is money at stake, spend a little bit for a top-notch provider.

dedward5• 13 years ago

What's important is that you treat the DNS management aspect of your business as a proper thing to be managed by itself. Whether you host it yourself, or contract it out to a service that specifically does this for you is a matter of risk tolerance and budget.

A common setup is a private nameserver that you host yourself, wherever, that isn't publicly used. You then contract UltraDNS/Neustar or their brethren and set them up as your public servers, doing zone transfers from your private ones. You get the advantages of direct management of your zones, and the global scale infrastructure needed to largely eliminate nameserver issues.

asr2bd• 13 years ago

Our site has been sporadically up and down for the past few hours. Glad I know the culprit finally. Hope this gets resolved soon

nodata• 13 years ago

Site down? Don't link to it. Bad netiquette. Jesus christ.

simonsarris• 13 years ago

Honest question from a networking beginner:

So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues.

If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?

mgkimsal2• 13 years ago

yes

area51org• 13 years ago

Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.

jeltz• 13 years ago

Also you as always have to be careful and make sure that all DNS servers have the same content or else you will get weird bugs.

thejosh3• 13 years ago

Wouldn't a delay be better than full downtime?

colmmacc3• 13 years ago

(I work on Route 53).

We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers.

In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider.

That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.

smountcastle• 13 years ago

It depends on the implementation of the resolver, but in general when no information is known about a nameserver, the resolver will pick one at random. Overtime, it'll keep track of the latency and pick one (usually randomly within a band specifying a range of latencies). See this paper for more info: https://www.dns-oarc.net/files/workshop-201203/OARC-workshop...

throwit1979• 13 years ago

In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.

TranceMan2• 13 years ago

Their facebook page has been updated:

Network Solutions is experiencing a Distributed Denial of Service (DDOS) attack that is impacting our customers as well as the Network Solutions site. Our technology team is working to mitigate the situation. Please check back for updates.

https://www.facebook.com/networksolutions/posts/101514681058...

ottoflux• 13 years ago

Back up.

SubZero• 13 years ago

Why did I click on the link to a down website? What did I expect? Shame on me.

Tenhundfeld• 13 years ago

TechZone360 (ugh, so many ads) article with a little more information:

Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/...

muraiki2• 13 years ago

A comment in the article pointed out that this one seems very similar to a Cisco article from a month ago: http://blogs.cisco.com/security/hijacking-of-dns-records-fro...

dangayle• 13 years ago

Just what I wanted to wake up to. Client emails asking me why their sites were down. Sigh.

dholowiski5• 13 years ago

This appears to be a problem again this afternoon (1:44PM MST)

Network Solutions' DNS was down

Related Stories