DDoS Attack Against Dyn Managed DNS

https://www.dynstatus.com/incidents/nlr4yrr162t8

by owenwil • 10 years ago

1563 655 10 years ago

dynstatus.com

_ar7• 10 years ago

Almost every website I visit except HN seems to be down...

chrisabrams• 10 years ago

Same here, that's odd isn't it ;)

pyvpx6• 10 years ago

well, first thing one does to use CloudFlare is migrate to their DNS. A CF-hosted site isn't going to be using Dyn...

tbarbugli4• 10 years ago

Github does not work for 100% the time

agotterer• 10 years ago

GitHub is working for me from NYC.

geekamongus3• 10 years ago

It's down in NC.

bananicorn• 10 years ago

Weird, works for me - from Italy (not sure if there isn't just some caching going somewhere down the line and I can see it because of that) edit: nevermind, it's almost certain i've got it cached

owenwil4• 10 years ago

Definitely a DNS cache on your computer (Or even in Chrome)

jtmarmon4• 10 years ago

I'm updating a list of confirmed outages as I see them here https://news.ycombinator.com/item?id=12759520

So far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this can even happen

owenwil4• 10 years ago

Yep! Intercom is down as well

olegkikin• 10 years ago

They all work fine for me.

owenwil4• 10 years ago

Caching is a magical thing :)

mcjiggerlog• 10 years ago

Me too. I'm in Spain - definitely seems to be geographical.

colanderman6• 10 years ago

I'm surprised; I would have thought such large sites would use more than one DNS provider? I mean:

    $ host -t NS twitter.com
    twitter.com name server ns4.p34.dynect.net.
    twitter.com name server ns3.p34.dynect.net.
    twitter.com name server ns2.p34.dynect.net.
    twitter.com name server ns1.p34.dynect.net.

I would have expected at least one of those to be somewhere else. What is the reason they would not have a backup provider?

corobo3• 10 years ago

I'd guess the reasoning is that DNS providers these days are all anycast-style DNS. A DDoS would usually just be a blip on a few servers around the world depending on where the attacks originate.

I'm not saying it's a good reason but it's a reason.

komali23• 10 years ago

I know a lot about some things, but almost nothing about networking, so excuse me if this is a really dumb question but - would your physical location determine what hosts you returned from that query? Like if you were in Asia would you get different ones back?

wongarsu3• 10 years ago

All of these work for me from Germany, and querying their authorative nameservers works just fine (so definitely no caching effect). Anycast for the win!

Symbiote2• 10 years ago

They don't work from Denmark, at 16:17 UTC.

elwell7• 10 years ago

https://www.whatsmydns.net/#A/twitter.com

lucaspiller• 10 years ago

Isn't a major feature of DNS that it can be cached? Why aren't records being returned by ISP/Google DNS/OpenDNS servers? Is their TTL set that low?

dudul3• 10 years ago

Damn, I've spent the past 30 minutes trying to update my DNS and playing with my router config! :)

No GitHub, well, it's gonna be a fun Friday...

piquadrat• 10 years ago

putting something like

    192.30.253.113 github.com

into your /etc/hosts (or other appropriate location for your OS) should get you going again.

geekamongus3• 10 years ago

Careful...some people may be trying to get out of work today.

profmonocle• 10 years ago

Just remember to take it out when this is resolved. IPs shuffle around all the time, and that IP might not always work for the main github.com domain.

6d6b735• 10 years ago

Isn't it the whole idea of git to make user independent from place like github? After all if you have the repository on your local machine you can continue working as if nothing happened, right?

dudul3• 10 years ago

Build server (e.g. CircleCI, Travis, etc), reviewing/merging pull requests, accessing 3rd party library READMEs. You know, doing software stuff.

forgottenpass• 10 years ago

Isn't it the whole idea of git to make user independent from place like github?

Yup. But the whole point of GitHub is to make you dependent on GitHub.

They've been slightly successful.

dudul3• 10 years ago

And there is no twitter to tweet about it!!!

wnm• 10 years ago

Heroku also seems to be affected. I'm getting this when I run 'heroku status':

> We are seeing a widespread DNS issue affecting connections to our services both internally and externally.

artursapek2• 10 years ago

Twitter, Reddit, wow. I was so confused for a moment. Thankfully HN is here to explain.

tunap2• 10 years ago

I had several, sporadic 'secure connection could not be established' yesterday while trying to open HN, amongst others. Painfully slow page load times across the board, too(Craigslist, Monoprice,weather.gov, etc) Still may be my buggy phone SIM...

dlg1416• 10 years ago

wait, buggy phone sims is a thing?

mjpa• 10 years ago

Is it really an internet wide outage?

Only 2 of the points in the US are affected on https://www.whatsmydns.net/ for the domains we've got on Dyn - same for Twitter etc

owenwil4• 10 years ago

If it's under a denial-of-service it's possible that it may respond correctly part of the time.

toast02• 10 years ago

Since many (all?) of Dyn's authoritative server IPs are anycast, attack traffic is probably not well distributed either. If you're routed to a server that's getting a lot of attack traffic, you're likely to have problems, but a server without much attack traffic will work fine.

geekamongus3• 10 years ago

"Widespread" might be a better term.

scrollaway3• 10 years ago

Relevant (or at least a-propos) post by Bruce Schneier, from a month ago: "Someone Is Learning How to Take Down the Internet"

https://www.schneier.com/blog/archives/2016/09/someone_is_le...

Edit: And to be clear: I don't mean to imply there's any connection :)

jgrahamc• 10 years ago

Really? It's unclear there's any connection between that vague article and Dyn's problems today.

FartyBumMcPoops• 10 years ago

http://dyn.com/blog/post-mortem-attack-to-dyn-standard-dns-n...

curiousgal• 10 years ago

We don't know who is doing this, but it feels like a large nation state. China or Russia would be my first guesses.

Why not the USA?

manojlds• 10 years ago

With the elections coming, USA would be on the top of my list.

amalcon• 10 years ago

It doesn't make a whole lot of sense for the USA to take down the internet, as they benefit the most from it. A significant fraction of that economy is based on it, much larger than in the cases of China and Russia. It would be like the owner of a coal mine campaigning for a carbon emissions tax: maybe there's something we don't know, but from the information we have it seems unlikely.

Note that this wouldn't rule out the USA as such. First, it could be a longshot preparedness thing, with no expectation that it would ever be used. Second, they could be red-teaming the thing (looking for weaknesses so that they can arrange for them to be shored up).

In either of these scenarios, it's no less likely that the USA would be doing it than anyone else. If you assume that whoever is doing this is planning to use their knowledge, however, the economic argument makes the USA less likely to be involved.

AsyncAwait2• 10 years ago

The usual thinking goes something like; well, the US created the internet so why would they want to take it down? Yes, NSA spies and all that, but they need the internet up to do that and also as bad as NSA is, it's nowhere near as bad as China or Russia where they ... (ranges from censorship to eating babies alive)

ckozlowski• 10 years ago

Because we are not indiscriminate like that.

mzw_mzw4• 10 years ago

Why the USA?

balls1872• 10 years ago

Russia retaliating for the US taking out the ESA Mars Drone.

briandear• 10 years ago

Why would they attack themselves?

WorldMaker• 10 years ago

From the context of the paper, because the USA could just send a three-letter-agency agent of some sort to Dyn, a US-based company, and ask what their infrastructure looks like? (Presuming of course some weird scenario where they weren't already tracking it, which seems unlikely.)

nickbauman4• 10 years ago

Prediction: A massive, sustained attack will occur on key US Internet infra on election night in an attempt to debase the US election results.

mtgx• 10 years ago

It's okay. James Comey, the FBI chief, said the US electoral system is such a mess, it would be too hard for an attacker to hack it or damage its integrity in any way. It's all good.

https://www.techdirt.com/articles/20160912/16553435504/fbi-d...

Of course, he said nothing about internal rigging:

https://twitter.com/TweetBrettMac/status/789372518436052992

MaxfordAndSons• 10 years ago

This is terrifying. Thankfully I don't think much actual voting infra is network reliant. But it could probably delay the results from being finalized for days, and allow Trump to spew further allegations of rigging.

Though if they targeted electric grid, water, and public transport, starting early in the day and choosing the regions by their populations political leaning, it could easily have an effect on the result itself.

heartbreak• 10 years ago

Assuming you're referring specifically to targeting media companies reporting on the results and not the electric grid like someone else mentioned, wouldn't they have to DDoS Google itself for that to work? I don't really see a DDoS of Google being effective.

rybosome5• 10 years ago

That was exactly my thought. This may be unrelated, or it may be a test run. But a large scale attack on Election Day that crippled communications would stir up unrest for a variety of reasons. Although I think that's highly unlikely to change the outcome, unrest after such a contentious election is not good.

codedokode2• 10 years ago

Is it really important who wins it there are only two candidates that share common view on many problems? And you don't need Internet to count votes anyway.

pure_ambition• 10 years ago

Probably not a great idea. If the internet went down at my work, none of us would be able to do anything, so we'd probably all head out to the polls just because we have nothing better to do. Unintentionally increased turnout.

lxknvlk• 10 years ago

prediction: in some time they will probe Google. That will be fascinating.

jbyers• 10 years ago

I think it's safe to assume they've been probing Google for 15+ years. One documented example: https://en.wikipedia.org/wiki/Operation_Aurora

corvus_sapiens2• 10 years ago

The Department of Homeland Security told CNBC that it is "looking into all potential causes" of the attack.

http://www.cnbc.com/2016/10/21/major-websites-across-east-co...

Is this par for course for all large DDOS attacks or did something tip them off?

krinchan• 10 years ago

More like for the first time in a long time, serious negative economic impact is occurring. I sincerely wish this was a wake up call, but it won't be.

464192002d7fe1c4• 10 years ago

From what I know of the situation (don't trust me, I'm not going to offer citations or sources), this attack wasn't particularly large in terms of gigabits/second. It was, however, very large in terms of economic impact.

I would assume that when a large number of big enterprise-y things go down, HSI takes notice. When other providers get attacks that are 20x larger (gbit/sec), but have much less widespread impact and impact on less enterprise-y things, they don't care so much.

0xmohit• 10 years ago

Maybe they are also using deep learning.

superzadeh• 10 years ago

Maybe some Deep Learning experiment actually created Skynet and it is probing us.

AmrEldib4• 10 years ago

Let's try to put this DDoS attack in some context aside from the technical part.

As @scrollaway mentioned, 6 weeks ago, Bruce Schneier posted that several companies told him that they're detecting attempts to probe their networks and find ways to bring it down https://www.schneier.com/blog/archives/2016/09/someone_is_le...

Now let's look at the progress of events:

- Hillary Clinton's personal email server was hacked a while ago.

- A lone hacker published a document obtained by hacking the DNC servers. The document includes opposition research on Donald Trump and how Hillary can attack him in the election.

- Wikileaks published emails obtained by hacking the DNC

- US intelligence agencies confirmed that Russia was behind the DNC hack

- It was reported that the CIA is starting a cyber attack against Russian targets. http://www.nbcnews.com/news/us-news/cia-prepping-possible-cy...

- This is happening while the war in Syria and Iraq is growing. The Russians are there to "fight ISIS" but they have deployed an air defense system even though ISIS doesn't have any air force.

- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.

https://www.theguardian.com/world/2016/oct/20/russian-fleet-...

https://www.theguardian.com/world/2016/oct/19/convoy-of-russ...

- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.

http://www.businessinsider.com/r-finland-sees-propaganda-att...

- US ships were attacked near Yemen after they're bombed some targets the belong to the rebels. https://www.theguardian.com/us-news/2016/oct/13/us-enters-ye...

- US election is in 3 weeks and Donald Trump is openly in love with Putin. Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.

Say Hello to World War III, everybody!

heurist2• 10 years ago

I just want to sell my software, why does everyone have to fight?!

Thank you for these links. I'm trying not to get wrapped up in conspiracies but am increasingly worried by the mounting conflict. I'd love to hear a calm, reasoned response from someone more knowledgable than me on these topics.

fokinsean3• 10 years ago

- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.

If they were really gearing up for war why would they move their only carrier away from the mother land. Your article even says it is more of a "show of force" than start of war.

So how did you jump to WW3?

leesalminen5• 10 years ago

US election is in 3 weeks and Donald Trump is openly in love with Putin.

He states that he's never met Putin nor has any holdings in Russia. He has stated that he is open to positive relationships with the Russian government.

Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.

I believe he stated that he wants NATO to "pay their fare share" in the costs of maintaining the organization.

I'm not a Trump supporter but we shouldn't believe everything we read.

kettu228• 10 years ago

- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.

The Finns actually have still quite good relationship with Russia (better than other neighbors) and nobody's actually questioning Finland's independence. Baltic countries is a different story.

Source: A Finn here.

knz2• 10 years ago

Say Hello to World War III, everybody!

Is this sabre rattling or the prelude to a global conflict? Surely at worst it will (continue to) be a proxy war between NATO and Russia in Syria and nothing more? What motive is there for Russia or NATO to engage in open warfare? I'm not sure that a slow and prolonged lead up to an open war would even be effective in this situation.

Perhaps it should be "Say hello to Cold War v2.2017"?

matwood2• 10 years ago

- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.

I read they were passing in international waters. Is that not the case? It's clearly a show of force, but no need for the hyperbole if it is not true.

DominikR• 10 years ago

- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.

Finland isn't worried, they have had stable relations for half a century as both sides agreed to not mess with each other. They have even refused to join NATO because it is actually safer for Finland and vice versa.

Cowboys with missiles stationed on Russia's border making hyperbole statements (like you do) - now that would be a real threat. (the same was also true the other way around with the Soviets stationing missiles in Cuba)

asclepi• 10 years ago

If this is supposed to be "taking down the internet", then I'm not impressed. Using cached DNS still gives access to any service. I'm even typing here on HackerNews.

If this is another practice run, then I'm still not impressed. Taking down one provider is not that hard. Good luck finding the resources to do this DDoS to ALL large DNS providers out there.

Maybe it's not really fair to link to that post every time a DDoS with more than average payload happens. Especially since the post doesn't mention any specifics, because well, "protect my sources". It's like the "buy gold now" guy starring in the 2 AM infomercial predicting an economic recession within the next 5 years, without adding what the exact cause is going to be. He is probably going to be right, but that doesn't make him a visionary.

drinchev3• 10 years ago

I work as a freelancer and today I didn't get paid and that's just me. Companies probably lost millions today. By just one DDoS to one DNS provider.

Yeah it's not the whole Internet, but how do you define "taking down the Internet" anyway. Is it every connected computer or just a huge amount of interconnected big websites? Because the latter is happening right now.

almostimplement• 10 years ago

Schneier: "I have received a gazillion press requests, but I am traveling in Australia and Asia and have had to decline most of them. That's okay, really, because we don't know anything much of anything about the attacks.

If I had to guess, though, I don't think it's China. I think it's more likely related to the DDoS attacks against Brian Krebs than the probing attacks against the Internet infrastructure, despite how prescient that essay seems right now. And, no, I don't think China is going to launch a preemptive attack on the Internet."

[1] https://www.schneier.com/blog/archives/2016/10/ddos_attacks_...

CarVac• 10 years ago

Using Google Public DNS fixed things for me.

ssebastianj• 10 years ago

I'm using Google Public DNS too. I don't really know if there is a relation with DynDNS but I'm still experiencing issues in GitHub and Twitter, like partial loading of images.

dichardson• 10 years ago

Perhaps Google had old (but valid) records still in their cache for a while. Google DNS was working for me for a while, and then stopped. Apparently Dyn has the problem fixed, but maybe there is some TTL based propagation delays still. I updated my internal network to use Dyn's internet guide/public DNS and the problem is fixed.

Maybe this is their strategy: we break it, you buy it ;)

https://help.dyn.com/internet-guide-setup/

If you can't load that page, the public DNS servers are: 216.146.35.35, 216.146.36.36

zappo2938• 10 years ago

Explains why the Heroku API is down.

jread2• 10 years ago

Seems to be impacting POPs in US East most severly. We use Ripe Atlas to assess the impact of DNS outages, and in the past hour have measured about 50-60% recursive query failure from a few hundred probes in that region: https://cloudharmony.com/status-for-dyn

jread2• 10 years ago

Now impacting multiple regions including both US East and US West with very query failure ratios - 50-70%

Legogris• 10 years ago

Microsoft's visualstudio.com's build servers fail to resolve Github and New Relic. So much for my Friday night deploy to staging.

cultavix• 10 years ago

Not sure if related but circleci.com is down for us do to a "DNS issue" !

jessemillar3• 10 years ago

Definitely related. Can confirm.

altyus• 10 years ago

For me redirecting my DNS to Google public DNS 8.8.8.8 and 8.8.4.4 did the trick.

mbym• 10 years ago

Same story for me here today (reporting from Cork, Ireland)

dankohn1• 10 years ago

I'm a Verizon FIOS customer in NYC and was unable to reach nytimes.com and several other sites this morning. Switching my DNS to Google's (8.8.8.8 and 8.8.4.4) seemed to fix the problem, but I don't understand why yet.

bpicolo• 10 years ago

I was on Google dns until I joined my VPN a bit ago. The public DNS was failing to hit lots of things. (NY, USA)

dijit2• 10 years ago

that's not going to help much if the authoritative name servers (which is what dyn is, btw) go down for more than a day.

Max record cache time is 86400s (24h), so if the attackers can keep it down for 24h then google will have to have custom instructions in place (or cache more aggressively than the RFC allows)

wongarsu3• 10 years ago

Since the attacked dyndns DNS servers are evidently anycast, the google server you are reaching might connect to a different dyndns server than you do. If google has luck to reach a less overloaded server, they might get an answer where you get none.

honkhonkpants• 10 years ago

Is there any reason why Dyn has to be "down" from Google's perspective? Is it possible that the large DNS providers maintain private network between each other, such that DDoS attacks that are effective against the public are ineffective against the private network?

aeharding• 10 years ago

Not only does it work (for now), many of the sites that are "down" are noticeably faster. :-)

elwell7• 10 years ago

less users connected

j_s• 10 years ago

I added the following to my hosts file for today:

  #8:07 AM 10/21/2016
  199.16.156.70 twitter.com
  104.244.43.231 abs.twimg.com
  104.244.43.231 pbs.twimg.com
  192.30.253.113 github.com
  151.101.24.133 assets-cdn.github.com

after giving up on modifying DNS timeouts. https://blogs.technet.microsoft.com/stdqry/2011/12/14/dns-cl...

alexmorenodev2• 10 years ago

Here in Brazil things are pretty slow.

"Oh, maybe its our shitty ISP screwing up everything again."

No, it's in a bigger scale.

sly0102• 10 years ago

I am confused. Are so many big websites using Dyn, or does Dyn have some special role in the DNS chain in the US?

aeden• 10 years ago

Dyn has no special role, they just provide DNS for a large number of high-traffic sites and services.

redm• 10 years ago

They sell premium services, have a large sales team, and are very aggressive. I get emails from them weekly discussing millisecond savings of their DNS solutions and the value increase in customers and sales.

Squeaky wheels get grease and their sales team squeaks a lot.

dijit2• 10 years ago

Realistically they compete with Neustar which is shockingly expensive and has less features and is harder to use.

I chose Dyn over Neustar (UltraDNS) when it was time to renew contracts because it was 60% cheaper, had a better latency, their support was great and the interfaces were clear.

Not a fanboy or anything, I really don't like how aggressively they hound me now (even though I have nothing to do with DNS for my current employer), but it's cheap and effective so it's not surprising people use them.

blantonl• 10 years ago

Boy, you aren't kidding about an aggressive sales team. They are relentless.

Ironically, a quick search of my Gmail mailbox came up with this gem in the subject line from Dyn.

"Did you know the average cost of a single DDoS outage is $882K?"

ljosa3• 10 years ago

They're widely used because they were one of the few providers of geo-aware DNS service for a long time. (These days there are other, cheaper options, including Amazon Route53.)

planetix• 10 years ago

Other option would have been Akamai's geo-aware DNS that has been available for a long time.

RijilV• 10 years ago

It's pretty wild that AWS is having impacts for the dyn outage given the shoutout to Route53. Wonder what they know that they don't?

bogomipz3• 10 years ago

They offer Anycast and have POPs around the globe. They also have some other nice features such as intelligent failover and extra GEO IP features. Things that you would otherwise have to build yourself.

They have been around a long time. For years they had a free product called DynDNS that would allow you get an A record for your dynamic IP at home.

adamrights• 10 years ago

We were affected @WSJ as well.

meshko3• 10 years ago

Very funny guys, can you stop now? We have a demo in 4 minutes.

kiostech• 10 years ago

Maybe run in localhost bro

ChadMoran• 10 years ago

Or bra

Reply

iagooar• 10 years ago

How did the demo go?

Reply

hueving3• 10 years ago

Like all live demos.

Reply

meshko3• 10 years ago

Well, the parts that relied out outside services hooked up via SSO were not demoed, but majority of it worked fine because demo server was misconfigured to not actually rely on the external services. It is pretty funny.

Reply

erowtom• 10 years ago

We had a demo at the exact same time. (Internal weekly product demo, not that critical). We did it on local host, the only one reliable 100% of the time.

Reply

mudge2• 10 years ago

I had a client ready to pay. Freshbooks was down.

Reply

maherbeg• 10 years ago

I find having a demo video backup is always a good idea.

Reply

ilostmykeys• 10 years ago

The Internet is so resilient. LOLz.

Reply

wongarsu3• 10 years ago

The internet is resilient against being completely taken down (as demonstrated by everything working just fine for me from Germany). It's explicitly not resilient against taking parts of it down. You can take some continents off the internet entirely by cutting half a dozend cables, but it's extremely hard to make the internet unusable for everyone.

Reply

RRRA• 10 years ago

Those distributed alternatives look better everyday... if only there was a working group and a transitional path.

Reply

ljosa3• 10 years ago

AWS says "We are investigating elevated errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region." Is that coincidental, or are they being DDoSed also?

Reply

jpdlla• 10 years ago

Have some errors logged on our side of problems resolving DNS hostname for SES email-smtp.us-east-1.amazonaws.com

Reply

cyberferret2• 10 years ago

That might explain why we are down - most of our EC2 instances are in us-east-1. Looks like Amazon SQS is impacted too. We are getting a stream of undeliverable messages, and our 'dead letter' queue is filling up!

Reply

colanderman6• 10 years ago

Apparently us-east-1 is backed by Dyn (and only Dyn) as well?

    $ host -t NS us-east-1.amazonaws.com
    us-east-1.amazonaws.com name server ns3.p31.dynect.net.
    us-east-1.amazonaws.com name server ns1.p31.dynect.net.
    us-east-1.amazonaws.com name server ns2.p31.dynect.net.
    us-east-1.amazonaws.com name server ns4.p31.dynect.net.

That's… utterly bizarre to me. us-east-2 has a more diverse selection:
$ host -t NS us-east-2.amazonaws.com us-east-2.amazonaws.com name server u4.amazonaws.com. us-east-2.amazonaws.com name server u6.amazonaws.com. us-east-2.amazonaws.com name server u3.amazonaws.com. us-east-2.amazonaws.com name server u2.amazonaws.com. us-east-2.amazonaws.com name server u1.amazonaws.com. us-east-2.amazonaws.com name server u5.amazonaws.com. us-east-2.amazonaws.com name server ns2.p31.dynect.net. us-east-2.amazonaws.com name server ns1.p31.dynect.net. us-east-2.amazonaws.com name server pdns1.ultradns.net. us-east-2.amazonaws.com name server pdns5.ultradns.info. us-east-2.amazonaws.com name server ns3.p31.dynect.net. us-east-2.amazonaws.com name server ns4.p31.dynect.net. us-east-2.amazonaws.com name server pdns3.ultradns.org.
Not that anyone should be running a service whose availability they care about solely in us-east-1 anyway…

Reply

marcoslater• 10 years ago

Same exact thing with EU-WEST-1/2.

  $ dig ns eu-west-1.amazonaws.com +short
  ns3.p31.dynect.net.
  ns1.p31.dynect.net.
  ns4.p31.dynect.net.
  ns2.p31.dynect.net.
  
  $ dig ns eu-west-2.amazonaws.com +short
  u6.amazonaws.com.
  u5.amazonaws.com.
  u2.amazonaws.com.
  u4.amazonaws.com.
  u1.amazonaws.com.
  u3.amazonaws.com.
  pdns1.ultradns.net.
  pdns3.ultradns.org.
  pdns5.ultradns.info.
  ns2.p31.dynect.net.
  ns1.p31.dynect.net.
  ns4.p31.dynect.net.
  ns3.p31.dynect.net.

I wonder why this is, considering the more extended usage that -1 on each region usually gets. :S

Reply

rdrey• 10 years ago

Ex-Amazonian here. I worked on the EC2 API for just over a year, and this could simply be a legacy thing. See https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service....

us-east-1 is the oldest region and predates Route 53. Not adding extra DNS providers to the older regions is probably an oversight.

(The EC2 API team requests load balancers from a separate load balancer team. The load balancer team probably didn't exist as a separate team when some of these regions were created.)

Reply

sandinmyjoints• 10 years ago

AWS may have updated this, I now see

    $ host -t NS us-east-1.amazonaws.com
    us-east-1.amazonaws.com name server pdns5.ultradns.info.
    us-east-1.amazonaws.com name server ns3.p31.dynect.net.
    us-east-1.amazonaws.com name server pdns1.ultradns.net.
    us-east-1.amazonaws.com name server pdns3.ultradns.org.
    us-east-1.amazonaws.com name server ns4.p31.dynect.net.
    us-east-1.amazonaws.com name server ns1.p31.dynect.net.
    us-east-1.amazonaws.com name server ns2.p31.dynect.net.
    us-east-1.amazonaws.com name server u1.amazonaws.com.
    us-east-1.amazonaws.com name server u2.amazonaws.com.
    us-east-1.amazonaws.com name server u3.amazonaws.com.
    us-east-1.amazonaws.com name server u4.amazonaws.com.
    us-east-1.amazonaws.com name server u5.amazonaws.com.
    us-east-1.amazonaws.com name server u6.amazonaws.com.

Reply

paulddraper2• 10 years ago

Not that anyone should be running a service whose availability they care about solely in us-east-1 anyway

Don't confuse regions with availability zones. (Though in this case, the availability zones don't help...)

Reply

nkozyra• 10 years ago

If that were the reason I wouldn't expect this update:

6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.

Reply

piker• 10 years ago

CNN.com is knocked out by this attack as well. I could see that as a useful target.

Reply

pmuk2• 10 years ago

I'm getting DNS errors on my PS4 when trying to download stuff, I guess it's related!

Reply

pmuk2• 10 years ago

Switching to Google's public DNS seems to have fixed it!

Reply

jssjr2• 10 years ago

I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

Reply

afshinmeh2• 10 years ago

Just being curious, why don't you use different DNS servers?

Reply

pcai• 10 years ago

Most services, even if they aren't the size of Github, can't change their DNS provider on a dime.

Reply

toast02• 10 years ago

(I'm not Github, but I work for a Dyn customer) Using multiple DNS providers has technical and organizational issues.

From a technical perspective, if you're doing fancy DNS things like geo targetting, round robin though more A records than you'll return to a query, or healtchecks to fail out ips from your rotations, using multiple providers means they're likely to be out of sync, especially if the provider capabilities don't match. That may not be terrible, because some resolvers are going to cache DNS answers for way longer than the TTL and you have to deal with that anyway. You'll also have to think about what to do when an update applied successfully to one provider, but the second provider failed to apply the update.

From an organizational perspective, most enterprise DNS costs a bunch of money, with volume discounts, so paying for two services, each at half the volume, is going to be significantly more expensive than just one. And you have to deal with two enterprise sales teams bugging you to try their other products, asking for testimonials, etc, bleh.

Also, the enterprise DNS I shopped with all claimed they ran multiple distinct clusters, so they should be covered for software risks that come from shipping the same broken software to all servers and having them all fall over at the same time.

Reply

cddotdotslash2• 10 years ago

I'm curious why you don't host your status page on a different domain/provider? When checking this AM why GitHub was down, I also couldn't reach the status page.

Reply

ToastyMallows• 10 years ago

The only way that I could check to see if Github knew they were having problems was by searching Google for "github status", and then seeing from the embedded Twitter section in the results page that there was a tweet about having problems. Twitter also being down for me didn't help the situation either.

Reply

joseluisq• 10 years ago

+1 Logical question!

Reply

jcl• 10 years ago

If it helps any, this link seems to work for me to reach the github status page (requires https certificate override, of course):

https://107.22.212.99/

Reply

peterbonney• 10 years ago

Lots of companies use Twitter for that sort of real-time status reporting, whose own up/down status one would think is sufficiently uncorrelated... unfortunately the internet is complicated.

Reply

3pt14159• 10 years ago

If this is consistently a problem why doesn't Github have fallback TLDs that use different DNS providers? Or even just code the site to work with static IPs. I tried the Github IP and it didn't load, but that could be for an unrelated issue.

Reply

sly0102• 10 years ago

If this is consistently a problem why doesn't Github have fallback TLDs I don't believe this has been consistently a problem in the past. But after today big services probably will have fallback TLDs.

Reply

lanna2• 10 years ago

This is what you can do to restore your GitHub access:

    grep github ~/.ssh/known_hosts
    sudo vim /etc/hosts
    sudo killall -HUP mDNSResponder
    ping github.com

Reply

lanna2• 10 years ago

Why am I being downvoted for providing useful information? I don't understand HN...

Reply

lukasm• 10 years ago

I added

192.30.253.112 github.com

but https://assets-cdn.github.com is failing

EDIT: Use 192.30.253.112 github.com 151.101.24.133 assets-cdn.github.com

or try 8.8.8.8 DNS

Reply

ionbara90• 10 years ago

seems like the right thing to do. however the ip address itself won't respond either

Reply

BlackGuyCoding2• 10 years ago

I love how the White House & GH posted a statement on Twitter.. that we can't access since its down.

Reply

lambda• 10 years ago

Twitter's working fine for me. This attack is affecting different people differently; as a DDOS, attacking a distributed system (DNS) with a lot of redundancy, it's possible for some people to be affected badly while others not affected at all.

I briefly lost access to GitHub, but Twitter has been working fine every time I've checked. Posting status messages in multiple venues helps to ensure that even if one channel is down, people might be able to get status from another channel.

Reply

JoshGlazebrook• 10 years ago

I wish you guys used statuspage or at least allowed email updates for the status of GitHub services.

Reply

jssjr2• 10 years ago

Another status update from GitHub: "We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates."

We're maintaining yellow status for the foreseeable future while the changes to our NS records propagate. If you have the ability to flush caches for your resolver, this may help restore access.

Latest status message: https://twitter.com/githubstatus/status/789565863649304576

Reply

jtmarmon4• 10 years ago

Semi related: I noticed this incident right when it began, but not because I was trying to access a website. This started happening to me: http://imgur.com/PPlaY5o

Then when I went to push to github out of fear my computer was about to soil itself, that failed too, and I noticed the outage.

Does anyone know if the above errors could be related to the outage? I'm using vim inside tmux with zsh as my shell. Maybe zsh does some kind of communication with gh while running?

I restarted my computer and it's still happening

Reply

scrollaway3• 10 years ago

The zsh default git plugin definitely doesn't touch github, or the network in general.

Are you using some oh-my-zsh github plugin by any chance?

Reply

jtmarmon4• 10 years ago

plugins=(git rbenv nvm gitfast zsh-autosuggestions github)

Reply

cyberferret2• 10 years ago

Hmm... Seems to be quite widespread. Some of our Amazon AWS services (located in the US) that rely on SQS are reporting critical errors. Intercom.io is also down at present, which we use for support for our web apps. Not looking very good from here (in Australia).

Reply

d--b• 10 years ago

Looks like at least some of it is resolved. spotify is back

Reply

im3w1l• 10 years ago

Fascinating weak spot!

Reply

r1ch• 10 years ago

Surprised to see so many big names relying on a single provider. DNS is designed to be distributed, it should be possible to avoid a single point of failure.

Reply

mayli• 10 years ago

This question came to my mind when I saw this post. The possibility might be the management cost. Synchronize between different providers can go wrong and might hard to debug when end users get different replies.

Reply

mdtancsa• 10 years ago

Anyone know any details of what the attack looks like ? I had a quick look in my (albeit small) network to look for odd flows going to their ASN33517, but didnt see much that looked odd on first glance...

Reply

foobarbecue• 10 years ago

According to Fortune, Hacker News "reported" on the incident. Are we journalists now?

"Popular tech site Hacker News reported many other sites were affected including Etsy, Spotify, Github, Soundcloud, and Heroku." -- http://fortune.com/2016/10/21/internet-outages/

Reply

mxuribe• 10 years ago

No, not necessarily journalists; rahter, an information source...Fortune - a site/company known for journalism/reporting - now just gave HackerNews more legitimacy as an official information source...Now with this power, please use it responsibly. ;-)

Reply

stingraycharles• 10 years ago

Too bad that the majority of the readers will think that HackerNews is somehow related to the "Hackers" that took down the internet.

Reply

Raphmedia• 10 years ago

No but we are a hivemind of smart individuals that correctly upvote important information and downvote irrelevant information. Most of the time, you can be sure that the top HN listings are going to be relevant.

Reply

melvinram• 10 years ago

I wonder if they used "Hacker News" as the source on that because it contains the word "hacker" and they wanted to say "oh, hackers read this site"... as in "the blackhat, break into stuff, steal your money, deface your website" folks.

Reply

pocoforte• 10 years ago

Perhaps they're confusing Hacker News with thehackernews.com.

Reply

exolymph• 10 years ago

"Reported" is a generic word; it has a journalism-specific usage but also a general usage.

Reply

hueving3• 10 years ago

If a headline says, "Users reported having bouts of explosive diarrhea", it doesn't make them medical journalists.

Reply

nbrempel• 10 years ago

It's a strange coincidence that Hover DNS was down for same reason a week ago.

http://hoverstatus.com

Reply

mirekrusin2• 10 years ago

They should do it once a year and call it Friday without Internet Day.

Reply

Asparagirl• 10 years ago

Better yet, have one day a year that is "Red Team Day" where people hunt for vulnerabilities so that assessments can be done, and companies can later fix any issues noted. Like how earlier this week there was a statewide earthquake drill in California, local emergency sirens were sounded, schoolkids practiced hiding under desks, etc. The Internet needs periodic tests like that too.

Reply

jessemillar3• 10 years ago

I absolutely love this idea! Would be a bit tricky to implement, but would definitely improve security in the long run.

Reply

metaverse2• 10 years ago

While my app isn't resolved using DYN, we are relying on APIs on our EC2 backend that use their DNS. Is there a Linux DNS caching server that will serve from a local cache primarily, and do lookups in the background instead to update the local cache? During the period DYN was down, it would've continued severing from the local cache and retried the background lookups, keeping my app up. I can also see it improving performance as my servers currently do lookups to the EC2 DNS on each http request...

Reply

nixgeek2• 10 years ago

If you're in us-east-1 then you potentially do actually rely on Dyn even for the amazonaws.com instance hostnames.

https://gist.github.com/agh/4e20df0d2d3bfa189477569b77f72e24

Reply

metaverse2• 10 years ago

Seems then that ELB has a local cache because http requests were reaching my app servers throughout the outage.

Reply

CodeSheikh• 10 years ago

Let's assume, that foreign countries such as Russia or China would be trying to sabotage our elections on Nov 8th night. What are the severe economic and political backlash that we can deal with if we cut off the traffic coming in from those region (not in a "we control the internet" kinda way)? I am sure they already have nodes operating within the USA. A lot of major tech companies use CDNs that can still serve traffic globally to the consumers of those countries. Even better, how about we regulate and slow down all of incoming traffic for say half day on election day? Is it even possible?

Reply

transfire3• 10 years ago

But then why would they be doing it right now? I'm sure they already know if they can do it or not. I don't think they need to do a large scale test run that would put people on high alert. They'd keep their head down until election day.

But then, what is China or Russia going to get out of doing something like this? It isn't going to change anything. Hillary is the next president regardless. Hell, even if no votes could be counted I am sure the Supreme Court wouldn't have a problem calling it for her.

So to me the idea that China and Russia is doing this for political reasons doesn't make any sense.

Reply

ohblahitsme• 10 years ago

Twitter and Github are still down here in LA (and confirmed on isup.me)

Reply

jessemillar3• 10 years ago

Both are down here in Utah too.

Reply

flowless• 10 years ago

New attack incoming https://www.dynstatus.com/incidents/nlr4yrr162t8

Reply

trusche• 10 years ago

Same from Ireland.

Reply

runn1ng• 10 years ago

They were up most of the day in here (Prague, Czech Republic, Europe), but it's down now (started about 20 minutes ago). It seems to be another wave of the attack.

Reply

stephankoelle• 10 years ago

Can confirm, europe was up until then

Reply

elwell7• 10 years ago

and mixpanel and zendesk

Reply

devnull42• 10 years ago

Dyn reporting another attack started at 15:52 UTC.

Reply

raemike123• 10 years ago

USA cyber defenses are NOT up to the task of defending our critical electronic infrastructure. Letting every company that runs critical services decide their own security posture is not scalable and has left us vulnerable. While no one is getting hurt, we are taking cyber missile hits from our enemies and eventually the damage will be worse. Other countries with more central controls will be less vulnerable than we are to crippling infrastructure take downs.

Reply

bluejekyll3• 10 years ago

No. What we need are new techniques for creating back-pressure to all the routers which are forwarding on this type of attack. The issue is that our Routing technology does not give downstream nodes any way to push back on the flood of packets.

Cisco could step up to the plate here. And no, I'm not talking about firewalls. We need newer ICMP type packets to create this back-pressure, so that we can stop floods like this.

Reply

andrewmchen• 10 years ago

It looks like we used to have something similar but it got deprecated. https://tools.ietf.org/html/rfc6633

Reply

bogomipz3• 10 years ago

"What we need are new techniques for creating back-pressure to all the routers which are forwarding on this type of attack"

What is this type of attack? TCP/UDP/ICMP has no notion of of a compromised host. Or even that it was crafted packet.

Back pressure already exists in TCP, see slow-start and window sizes, flow control is part of the "control part". When a router's porst buffers are full the router drops the packets on the floor. It does not do further processing of those packets.

What would ICMP do here? If I have a million compromised hosts and each sends a single SYN packet towards a destination host, how would ICMP help?

">Cisco could step up to the plate here" What would Cisco do? Cisco doesn't control the ICMP protocol.

I think you are not understanding ICMP. The job of ICMP is to report error conditions. ICMP serves as a helper to IP which is itself unreliable and has no form of error control or checking. A router or host being overrun is not a network error condition it is a resource condition. No ICMP type is ever going to be able to stop a host from originating UDP/TCP/ICMP towards a destination. Even if it could you would just overwhelm it in the outbound direction by replying to potentially millions of hosts.

Reply

castis• 10 years ago

we are taking cyber missile hits

A better analogy would be 'mocked cyber mass protests' seeing as how no infrastructure will need to be rebuilt after this passes.

Reply

etherealflux• 10 years ago

Thankfully, we don't leave in the Star Trek universe, where hacking a computer causes it to explode in a shower of white-hot shrapnel.

Reply

floatboth• 10 years ago

Who the hell uses the word "cyber" and especially "cyber missiles" non-ironically these days? (Government people, for some reason, but yeah.)

Critical infrastructure MUST NOT rely on PUBLIC networks like the Internet. It's a TERRIBLE idea. If you're working on anything actually critical, build your own fucking ISOLATED network with your own fucking cables.

Reply

pbarnes_1• 10 years ago

Zzzz.

The USG is not responsible for a corporation that runs DNS services.

Reply

eredi93• 10 years ago

and the attacker are back. DDoS v2 is here

Reply

middleman90• 10 years ago

Shopify is down

Reply

chromaton2• 10 years ago

I can't currently get resolution on www.paypal.com.

$ dig @8.8.8.8 www.paypal.com

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17925 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;www.paypal.com. IN A

;; Query time: 29 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 21 12:35:33 2016 ;; MSG SIZE rcvd: 32

Reply

robbyt• 10 years ago

Me either. I'm in NYC on TWC. I can't resolve PayPal with Google's DNS or with TWC dns servers.

Also, I'm unable to resolve Twitter on TWC dns.

Reply

heyts• 10 years ago

I'm in New-York too and can't resolve Paypal, Etsy, Soundcloud, Github, Netflix, Heroku or Twitter

Reply

devy2• 10 years ago

I'm in NYC too. Github.com is resolving/working fine. Netflix.com is resolved but all assets (probably) weren't loading. Additionally Zendesk is also affected.

Reply

oAlbe• 10 years ago

Quote from the status page:

This attack is mainly impacting US East and is impacting Managed DNS customers in this region.

I'm in Italy, using my provider's default DNSs (not Google) and I can't reach paypal.com, thenextweb, twitter, spotify etc either.

Reply

teoruiz• 10 years ago

Exactly the same right now from Spain. It seems the attackers are now targeting Dyn's European servers.

Reply

sschueller• 10 years ago

Interesting. 8.8.8.8 is not able to provide me with a record. However ns1.p57.dynect.net and ns3.p57.dynect.net give an answer where as ns2.p57.dynect.net and ns4.p57.dynect.net hang.

Shouldn't 8.8.8.8 query another name server if one fails to respond or takes too long?

EDIT: Is there an inherent flaw in how secondary records are queried? And as bhauer mentioned, is there a possibility to fall back to last known record?

Reply

zeronzr• 10 years ago

https://www.whatsmydns.net/#A/art.nflximg.net

http://status.aws.amazon.com/

Seems like it is affecting another regions now.

Reply

reb• 10 years ago

Chiming in as a TWC user on the West Coast:

Github out, Etsy out, Paypal out, Twitter out, Soundcloud out, Crunchbase out, Heroku out, Spotify intermittent, Netflix only loads a white page with plaintext "who's watching" list and no functionality.

Reply

chromaton2• 10 years ago

And it's back again. I'm on AT&T in Atlanta.

  $ dig @8.8.8.8 www.paypal.com

  ; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40999
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

  ;; QUESTION SECTION:
  ;www.paypal.com.      IN  A

  ;; ANSWER SECTION:
  www.paypal.com.    266  IN  CNAME  www.paypal.com.akadns.net.
  www.paypal.com.akadns.net. 29  IN  CNAME  ppdirect.paypal.com.akadns.net.
  ppdirect.paypal.com.akadns.net.  299 IN  CNAME  wlb.paypal.com.akadns.net.
  wlb.paypal.com.akadns.net. 29  IN  CNAME  www.paypal.com.edgekey.net.
  www.paypal.com.edgekey.net. 20  IN  CNAME  e3694.a.akamaiedge.net.
  e3694.a.akamaiedge.net.  19  IN  A  23.73.8.114

  ;; Query time: 146 msec
  ;; SERVER: 8.8.8.8#53(8.8.8.8)
  ;; WHEN: Fri Oct 21 13:05:48 2016
  ;; MSG SIZE  rcvd: 198

Reply

roland-s• 10 years ago

Paypal and others still down for me at University of California (I think we're our own ISP?)

Reply

cm32• 10 years ago

Just to be clear, this is a DDoS against Dynect's NS hosts, right?

I'm confused because of the use of "dyn dns", which to me means dns for hosts that don't have static ip addresses.

I'm actually surprised so many big-name sites rely on Dynect, which I hadn't heard of, but more importantly don't seem to use someone else's NS hosts as 2nd or 4th entries.

Reply

detaro5• 10 years ago

The company is just called "Dyn", DynECT is a product name, but yes.

Reply

cm32• 10 years ago

Thanks, I was really confused when I read the title "Massive Dyn DNS outage" and how that affects Twitter or Github.

Reply

atsidi• 10 years ago

I've been having the same problem accessing github in particular. Just for fun, I opened the Opera browser and activated the built-in VPN. That got everything going again. At least for browsing, not so useful for my git pulls and pushes.

Reply

andmarios• 10 years ago

OpenDNS servers seem the only ones that still work. Kudos.

It may not be the proper action but this kind of soft-fail scenario (use the old DNS until you can contact the DNS servers and get new ones) is much better.

  echo "nameserver 208.67.222.222" | sudo tee -a /etc/resolv.conf

Reply

adobrawy• 10 years ago

Twitter and GitHub is down on Scaleway (AS12876) and Tiktalik (Warsaw, Poland, Europe, AS198717) network too (no response from dynect.net).

Reply

mirekrusin2• 10 years ago

Github doesn't work again for me :(

Reply

danyork9• 10 years ago

https://cloudharmony.com/status-for-dyn is now (12:43pm EDT) showing Dyn's "US East" and "US West" centers as being down. Anyone know anything about this Cloudharmony service? How often does it update? and what is it monitoring?

Reply

llamataboot• 10 years ago

Heroku is still having problems as well

Reply

emmet• 10 years ago

Is it just me or are these kind of attacks becoming way more frequent recently? This kind of widespread outage seems so new, but again, that might just be me.

Reply

bhauer14• 10 years ago

Out of curiosity, why do caching DNS resolvers, such as the DNS resolver I run on my home network, not provide an option to retain last-known-good resolutions beyond the authority-provided time to live? In such a configuration, after the TTL expiration, the resolver would attempt to refresh from the authority/upstream provider, but if that attempt fails, the response would be a more graceful failure of returning a last-known-good resolution (perhaps with a flag). This behavior would continue until an administrator-specified and potentially quite generous maximum TTL expires, after which nodes would finally see resolution failing outright.

Ideally, then, the local resolvers of the nodes and/or the UIs of applications could detect the last-known-good flag on resolution and present a UI to users ("DNS authority for this domain is unresponsive; you are visiting a last-known-good IP provided by a resolution from 8 hours ago."). But that would be a nicety, and not strictly necessary.

Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside.

Reply

drinchev3• 10 years ago

All will break the moment that one of the websites that you access makes a server-side request to another website ( think about logging-services, server-clusters, database servers, etc - they all either have IPs or most-likely some domains. )

Reply

bhauer14• 10 years ago

I'm not sure I understand what you're saying.

The scenario is that my local network's caching DNS resolver retains resolutions beyond the authority-provided TTL in the event that a TTL-specified refresh at expiration fails. Therefore, my web browser may—in the very rare situation where this arises—make an HTTP request to an IP address of a server that has been intentionally moved by a service provider (let's assume they did so expecting their authoritative TTL to have expired). Since this scenario only arises because my caching resolver wasn't able to reach the authority, I'm not seeing a downside.

But if I understand your reply correctly, you are saying that the web server I've contacted may, in turn, be using a DNS resolver that is similarly configured to provide last-known-good resolution when its upstream provider/authority cannot provide resolution. This would potentially result in that web server making an HTTPS API request to a wrong IP, again only in the rare case where we have defaulted to a last-known-good resolution. I'm not really seeing the problem here except that the HTTPS request might fail (if the expected service was moved and no longer exists at the last-known-good IP), but how is that worse than the DNS resolution having failed? In both cases, the back-end service request fails.

Reply

beachstartup• 10 years ago

i seem to remember that dns has generally been reliable (until recently, i guess), probably nobody has ever thought that to be necessary.

you could write a cron script that generates a date-stamped hosts file based on a list of your top-used domain names, and simply use that on your machine(s) if your dns ever goes down. that's basically a very simple local dns cache.

if you feel like living dangerously, have it update /etc/hosts directly.

Reply

caleblloyd• 10 years ago

i seem to remember that dns has generally been reliable (until recently, i guess)

Probably because people used to use long TTLs (1 hour, 4 hours, whatever) and now the default behavior in services like Amazon Route 53 is to use 5 minutes.

Reply

mej102• 10 years ago

This company, for example, has a 99.999% uptime SLA. Thats roughly 5 minutes per year.

Reply

the_mitsuhiko2• 10 years ago

Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside.

Because you would keep old DNS records around forever if a server goes away for good. So you need to have a timeout for that anyways.

Reply

bhauer14• 10 years ago

Yes, but:

1) Memory and disk are cheap. My caching DNS resolver can handle some stale records.

2) I suggested above that this behavior would continue until an administrator-specified and potentially quite generous maximum TTL expires. That is, I could configure my caching DNS resolver to fully purge expired records after, for example, 2 weeks.

Reply

Kalium16• 10 years ago

Historically, doing this has been a source of a truly awe-inspiring amount of pain.

Reply

gmaslov• 10 years ago

Aw, don't leave us hanging like that. What problems did it cause?

Reply

davidu3• 10 years ago

OpenDNS does this: https://support.opendns.com/hc/en-us/articles/227987767-Dyna...

It's called SmartCache.

Reply

bhauer14• 10 years ago

Awesome! Is this available as software I can install on my network? Sorry, probably a dumb question.

Reply

tedmiston4• 10 years ago

Anyone know if Google Public DNS does?

Reply

conatico• 10 years ago

A shame OpenDNS used to redirect me to some spam webpage every time I tried to resolve a domain that didn't exist--they earned a spot on my black list forever. :(

Reply

gwu78• 10 years ago

I do this, too.

It's called HOSTS and djb's cdb constant database.

And one does not need to use a recursive cache to get the IP addresses. Fetching them non-recursively and dumping them to a HOSTS and a cdb file can sometimes be faster; I have a script that does that. Fetching them from scans.io can be even faster.

   cd||exit
   [ -c null ]||mknod null c 2 2 
   
   case $# in
   0)
   {
    sed '
         /#/d;
         /^[0-9]/!d;
    ' /etc/hosts \
     |{ 
        while read a b c d;
        do 
        echo +${#b},${#a}:$b-\>$a;
        done;
      }
    echo;
   } \
    |exec awk '!($0 in a){a[$0];print}' \
    |exec cdbmake $0.cdb $0.t||exit
   exec cdbdump < $0.cdb
   
   ;;1)
   test ${#0} = 2 ||
   exec cdbget $1 < $0.cdb >null;
   exec cdbget $1 < $0.cdb;

   esac

   usage: $0  
   usage: $0 domainname

First usage compiles and dumps database to screen. Second usage checks for presence of domainname and exits 0 if present otherwise exits 100. Third usage is if $0 is only two characters it will check for presence of domainname and if present print the IP and domainname in HOSTS format.
http://cr.yp.to/cdb.html
With all due respect to the enormous reliance on it that has built up over the past decades, DNS is not the internet. It is just a service heavily used for things like email and web. This does not mean, in an emergency, email and web cannot work without DNS. They once did and they still can.
The internet runs just fine without DNS. Some software may refuse to honour HOSTS and rely on solely on DNS. But that is a vulnerability of the software, not the internet. (And in such cases, e.g., qmail, I just serve my own zone via tinydns, which again is just a mirror of HOSTS.)

Reply

bluejekyll3• 10 years ago

I've been thinking of adding this exact feature to my DNS framework that I've been working on (if github was resolving): https://github.com/bluejekyll/trust-dns

If you have any feedback, I'd love to hear it.

Reply

bhauer14• 10 years ago

Sounds promising. I'll want to take a look when GitHub is resolvable again. :)

Reply

Kalium16• 10 years ago

To be perfectly honest, a "feature" like this has no business being in a safe and secure DNS server. You should fail-safe, rather than serving stale data of unknown safety.

Serving data you cannot verify is a dangerous failure state.

Reply

zwily2• 10 years ago

This can be pretty bad in a world where AWS ELB IP addresses change regularly.

Reply

cjbprime4• 10 years ago

Why? The OP is only proposing using a cached result when there's no updated record available.

Reply

jasimp3• 10 years ago

I can guarantee you that popular DNS resolvers (think 500b+ transactions a day) do have this feature!

Don't want to say much more due to it being my job, and I don't want to give away too much.

EDIT: https://www.google.com/patents/US8583801

Reply

imsofuture• 10 years ago

What is the point of this comment?

Reply

pyvpx6• 10 years ago

you mean opendns?

Reply

JoshTriplett• 10 years ago

It'd be nice to have a "backup TTL" included, to allow sites to specify whether and how long they wanted such caching behavior.

Also, that cache would need to only kick in when the server was unreachable or produced SERVFAIL, not when it returned a negative result. Negative results returned by the authoritative server are correct, and should not result in the recursive resolver returning anything other than a negative result.

Reply

bhauer14• 10 years ago

Also, that cache would need to only kick in when the server was unreachable or produced SERVFAIL, not when it returned a negative result. Negative results returned by the authoritative server are correct, and should not result in the recursive resolver returning anything other than a negative result.

Precisely. I am not suggesting any change to how a caching resolver comprehends valid responses from the authoritative server for a domain. For example, if the authoritative server says, "No such domain," then the domain is understood to be gone. At that point, the domain being gone is in fact the last-known-good resolution.

Reply

Godel_unicode4• 10 years ago

This is why TTL is a variable. If you want to have your records last longer, you can. If you want to later shorten them, you can. People screw DNS up enough already, let's not make it worse by adding layers of TTL.

Reply

spc476• 10 years ago

It might be a stretch to use the information, but the SOA RR does contain an EXPIRE field, defined as "A 32 bit time value that specifies the upper limit on the time interval that can elapse before the zone is no longer authoritative." It's an additional request, but the SOA RR does contain the type of information you are asking for.

Reply

DivineTraube• 10 years ago

HTTP has a good solution/proposal for this: the server can include a stale-on-error=someTimeInSeconds header in addition to the TTL and then every cache is allowed to continue serving stale data for the specified time while the origin is unreachable. Probably a good idea to include such a mechanism in DNS, too.

https://tools.ietf.org/html/rfc5861

Reply

LordHumungous2• 10 years ago

I think a problem that you might be overlooking is that DNS lookups aren't just failing, they are also very slow when a DDOS attack is underway on the authority servers. This introduces a latency shock to the system which causes cascading failures.

Reply

jedisct1• 10 years ago

You can install EdgeDNS locally. It does that, among other things.

https://github.com/jedisct1/edgedns

Reply

devy2• 10 years ago

Is Zendesk being affected? Their status page is reporting external DNS provider is having DNS issue [1] and most of their sites are being affected.

[1] https://status.zendesk.com/

Reply

jorts3• 10 years ago

Yes, they were affected.

Reply

darkmouth• 10 years ago

and its down again

Reply

transfire3• 10 years ago

Oo oo, I know! Iran did it!

Reply

chatmasta• 10 years ago

I'd like to see proof of this attack from an outside network observer.

Is it possible the government could force a DNS provider to pretend to fall victim to a DDoS attack, as a form of a false flag cyber attack?

Reply

transfire3• 10 years ago

Must be trying to stop the latest Julian Assange leak.

Reply

alasdair_• 10 years ago

The Wikileaks twitter feed is putting out some really weird stuff lately. They are claiming their "supporters" are behind the attacks, which makes little sense.

Reply

leesalminen5• 10 years ago

Boulder here. Can't resolve Wufoo or PayPal using 8.8.8.8

Reply

patmcguire2• 10 years ago

Any quick script to see if a given domain ultimately resolves to them? My SaaS company has a lot of custom domains from whatever DNS servers pointed at us and I'd like to be able to tell people whether it's our fault or not.

Reply

stevekemp3• 10 years ago

`whois $domain`.

But it should be "obvious" if your users report "Server not found" vs. "Cannot connect" or "Page not found" style errors.

Reply

mdavidn• 10 years ago

`dig NS $domain`

Query for the root domain, without any subdomains like www. That is, you need to check the "zone apex," the shortest name purchased from a registrar and potentially delegated to Dyn. Look for dynect.net in the list of authoritative name servers.

Reply

patmcguire2• 10 years ago

Yeah, I tried that. I don't see dnyect in a lot of domains that are failing, and it's clearly related somehow, they didn't all break at the same time by coincidence.

Reply

kilroy123• 10 years ago

Interesting. Lots of sites have been down for me, here in Mexico City. Twitter. Github. Loads of other random sites. When I turned on my US based VPN. It all started working again.

Reply

Raed667• 10 years ago

You can add Netflix to the list.

    GET https://art-s.nflximg.net net::ERR_NAME_RESOLUTION_FAILED

    GET https://assets.nflxext.com net::ERR_NAME_RESOLUTION_FAILED

Reply

BlackGuyCoding2• 10 years ago

Anyone having any issues with WhatsApp? Mobile text seems to work fine but all images fail, Desktop & web browser aren't connecting at the moment (west coast)

Reply

Supersam654• 10 years ago

OpenDNS DNS Servers (208.67.222.222 and 208.67.220.220) are still resolving websites while my typical fallback to 8.8.8.8 is not.

Reply

foxhop2• 10 years ago

I noticed the same pattern.

Reply

nodesocket3• 10 years ago

I've been singing the praise of AWS Route53 for a long time, they up and running. I can't believe major multi-million dollar companies (Twitter, GitHub, Soundcloud, Pagerduty) would not run a mix of multiple DNS providers.

Also what is happening is a cascade effect, where a 3rd party being down effects others.

Reply

crftr• 10 years ago

I've been singing the praise of AWS Route53 for a long time, they up and running.

I'm a fan of Route53, too.

But can we say that it weathered the attack? Or was it just lucky that its systems weren't targeted?

Reply

cloudjacker• 10 years ago

Or was it just lucky that its systems weren't targeted?

I was wondering that too

Reply

stevekemp3• 10 years ago

One of the reasons why Route53 is good is because they give different nameservers to each hosted zone - unless you choose to use a branded record-set.

I've seen them be hit by dDos attacks in the past, but never had any significant impact.

(I wrap Route53 and handle storing DNS records in a git repository over at https://dns-api.com/ Adding support for other backends is my current priority to allow more redundancy.)

Reply

halayli• 10 years ago

github.com seems to be down because of this.

Reply

newsat132• 10 years ago

Switch to OpenDNS servers - 208.67.222.222 and 208.67.220.220. Even google NS are down it seems. Heroku works after switching to opendns.

Reply

bitskits• 10 years ago

Google's OpenDNS servers are not down, they are recursive resolvers, and are not getting data from Dyn's authoritative resolvers. OpenDNS caches DNS data for longer than other recursive DNS providers (they call it smartcache, IIRC), which is why they are still working.

Reply

deathanatos• 10 years ago

Google's DNS has been working all day here. The problem is that Dyn's DNS server is being DDoS'd; if you request a record that the authoritative DNS server for is hosted by Dyn, then when you query Google's DNS for that record, then Google's server needs to make a query to Dyn, which is down, and thus, your query fails. But queries to Google for non-Dyn domains will continue to work just fine.

OpenDNS works because, as another poster notes, that, for better or worse, they don't strictly obey TTLs: https://news.ycombinator.com/item?id=12762429

Reply

bostonvaulter2• 10 years ago

Here's the link that verifies OpenDNS's addresses (if you're wary of trusting a single HN comment): https://use.opendns.com/

Reply

newsat132• 10 years ago

Sorry for the confusion about saying Google NS is down :-) I meant dig heroku.com @8.8.8.8 does not work (as pointed by some other poster it is because google NS honors TTL but opendns does not)

Reply

Kaedon• 10 years ago

What other providers would you recommend than Dyn? Route53? Cloudflare? Something else?

Reply

bgentry• 10 years ago

If you're having issues with people accessing your running Heroku apps, it's likely because you're running your DNS through herokussl.com (with their SSL endpoint product) which is hosted on Dyn.

If you can update your DNS to CNAME directly to the ELB behind it, it should at least make your site accessible.

Reply

asdf3335• 10 years ago

thanks for the tip! how did you determine the ELB address behind the ssl endpoint?

edit: figured it out. What i did was do:

nslookup your-SSL-endpoint.herokussl.com

then you'll see the elb address.

Switch to the openDNS servers helpfully pointed out by someone above first...

Reply

Artemis22• 10 years ago

Presumably with something like `dig @208.67.220.220 -t CNAME <your site>.herokussl.com`. This uses the OpenDNS nameservers, that people have been reporting as working. Haven't tested it as I am on the go.

Reply

gkop• 10 years ago

Another option may be to switch from the SSL endpoint add-on to the new, free SNI-based SSL termination feature, which will mean CNAMEing to your-domain.herokudns.com. , which seems not to be affected by today's issues.

Reply

throwaway5345• 10 years ago

Nice, this is working well for us too. We were able to get the CNAME of the ELB by doing a `dig whatever.ourdomain.com` in an EC2 instance we launched in São Paulo (which presumably worked since Dyn's outage is primarily affecting their east coast PoPs.)

Reply

jrochkind12• 10 years ago

Is it time for everyone to actually start using secondary name servers/DNS resolvers too from a different provider from primary? DNS _is_ built for this, for the very purpose of handling failure of the primary resolver, isn't it? Just most people don't seem to do it -- including major players?

Or would that not actually solve this particular scenario?

Reply

sinap• 10 years ago

The attack is on the authoritative name servers, not a DNS resolver. A public DNS resolver will query the authoritative name server for a record if it doesn't exist in it's cache.

Reply

mgamble• 10 years ago

Agreed, but there is nothing stopping you from having the authoritative name servers for a domain with different providers. As someone previously said, DNS was designed for this.

Reply

jrochkind12• 10 years ago

Can you have secondary name servers too though? And would it have worked to avoid outage for domains doing such in this case?

Reply

danyork9• 10 years ago

Yes, I think this attack has brought to everyone's attention that many companies have gone away from what used to be the extremely common practice of having your authoritative DNS serving shared across multiple DNS hosting providers. This would have addressed the issue... and we're seeing that by the end of the day many of these sites have gone to having multiple providers.

Reply

paulddraper2• 10 years ago

I thought DNS (particularly public) was basically immune to DDoS?

If one DNS server is down, use the cached result or another server.

DNS is some of the most distributable, cachable data I can imagine.

Reply

snug5• 10 years ago

Depends on how many PoPs they have. Looks like they have 4 easter US.[0] If they are seeing large attacks that Krebs saw a few weeks ago, that could certainly be enough to take down one or two, and then causing redirected traffic to take down the other two.

I used to work for a DNS/DDoS provider, and this was a very real problem. Leave the PoPs that are being affected out, or risk overloading the other PoPs by overloading real traffic.

Before moving the other traffic, you also have to worry about blocking the DDoS traffic otherwise you're just redirecting them to the other PoPs. Mitigating DDoS attacks are not fun, and hard to block.

[0]http://dyn.com/dns/network-map/

Reply

acdha• 10 years ago

Some sites intentionally disable that, however, by setting a short TTL on replies. The idea is usually that it allows them to very quickly adjust to hardware failures or load across datacenters but it has the consequence of making your infrastructure comparatively brittle.

Reply

octoploid• 10 years ago

It is spreading to other DNS providers, too: https://status.fastly.com/

www.ft.com is unreachable for example.

Reply

snug5• 10 years ago

Fastly is simply putting up a status page so they aren't contacted about issues, and letting them know it's about DYN. And they are having internal issues with communications like zendesk.

Reply

Kluny• 10 years ago

Highrise seems to be having problems, as seen by email errors when we forward email to Highrise dropboxes.

Reply

elwell7• 10 years ago

To get on github you can add to your /etc/hosts:

    192.30.253.113  github.com
    151.101.32.133  assets-cdn.github.com

And it seems faster than normal right (less users).
Edit; for profile pics include:
151.101.32.133 avatars0.githubusercontent.com 151.101.32.133 avatars1.githubusercontent.com 151.101.32.133 avatars2.githubusercontent.com 151.101.32.133 avatars3.githubusercontent.com 151.101.32.133 avatars4.githubusercontent.com 151.101.32.133 avatars5.githubusercontent.com

Reply

xyclos• 10 years ago

how about npm?

Reply

puddintane• 10 years ago

I was able to access everything by changing DNS as mentioned in the other posts [1].

[1] https://news.ycombinator.com/item?id=12762841

edit Of course this is if your local policy allows you to change this!

Reply

elwell7• 10 years ago

you can get the ip from a different location using this: https://www.whatsmydns.net/

Reply

MattConfluence• 10 years ago

How about *.github.io?

Edit: saw your other reply and looked it up myself, it's 23.235.33.133

Reply

elwell7• 10 years ago

I don't think /etc/hosts will work with wildcard subdomains.

Reply

edcastano• 10 years ago

The great irony: http://www.isitdownrightnow.com/yesware.com.html

Reply

dEnigma• 10 years ago

I was not aware of the attacks going on until this happened:

1. Tried to download "Unknown Horizons" (game featured recently on Hacker News) binary, github-link doesn't work.

2. Think "Ok, might be an old link", google their github-repository, github appears down.

3. Try accessing github status website, is down.

4. Interested, try to visit github status twitter account, twitter is down.

Really weird experience, normally at least the second source of news on a downed website I try during an attack works.

Reply

jeppebemad• 10 years ago

Had a similar experience. When I went to confirm on twitter, that was down too. I was able to acces Twitter from my phone though, where I found a ton of tweets saying "Twitter down!". Strange.

Reply

alexmorenodev2• 10 years ago

Funny how we're able to tell that twitter's down on twitter, but here in Brazil, when whatsapp was down, nobody could use their own whatsapp to ask or tell if whatsapp was down.

Reply

x2398dh1• 10 years ago

Currently I am able to get into every site on the web, including GitHub, by using a VPN service based in Hong Kong.

Reply

Artemis22• 10 years ago

PayPal, Braintree, Spreedly down. Some companies are going to lose money today...

Reply

tedmiston4• 10 years ago

Anyone else spend the morning thinking the problem was their setup? I've been flushing my system DNS cache, Chrome's DNS cache, changing DNS servers, rebooting my router, turning VPN on/off, etc.

Reply

mudge2• 10 years ago

Yep, I did the same.

Reply

stuntmachine• 10 years ago

Yeah. :/

It happened to be at the same time I was getting things configured to connect to a new VPN that I hadn't used before for the first time. Until about 7am today my home network was a 10.0.0.0/8 network. VPN kept bombing in the last phase of connecting and I couldn't figure out why, so I thought it was an IP conflict with my internal network range.

So naturally, I then went into my router and changed my subnet for my entire home network to the more common 192.168.1.0/24 range to see if it'd help. It didn't. Until suddenly VPN "just worked" -- which makes me wonder if I needed to change my network at all to begin with.

Then I started experiencing all sorts of weird issues where the Internet seemed to disappear from one minute until the next.

Then I hit IRC when things finally stabilized and see "Did you hear about Dyn?".

My reaction: wut.

TL;DR: I rearchitected my home network at 7am for no reason.

Reply

arp• 10 years ago

Here's how to add static mappings temporarily to survive through the outage:

https://www.reddit.com/r/sysadmin/comments/58o5mp/dyn_dns_dd...

Reply

Animats11• 10 years ago

"digikey.com", the big electronic part distributor, is currently inaccessible. DNS lookups are failing with SERVFAIL. Even the Google DNS server (8.8.8.8) can't resolve that domain. Their DNS servers are "ns1.p10.dynect.net" through "ns4.p10.dynect.net", so it's a Dyn problem.

This will cause supply-chain disruption for manufacturers using DigiKey for just-in-time supply.

(justdownforme.com says the site is down, but downforeveryoneorjustme.com says it's up. They're probably caching DNS locally.)

Reply

invisiblep• 10 years ago

Why not use:

OpenDNS - recursive DNS

Cloudflare (DNS only) - authoritative DNS

Both services are free and distributed across the world.

Reply

q3k• 10 years ago

Dyn was supposed to be distributed too.

The takeaway here should be to use multiple authoritative DNS providers, not a single (even if better) one.

Reply

foxhop2• 10 years ago

So I had hardcoded my DNS server to googles, aka:

    dig @8.8.4.4 github.com +short

I was not getting an answer.
However using my routers/dhcp/ISP to set my DNS server, I am able to get answers:
dig github.com +short 192.30.253.112

Reply

snug5• 10 years ago

Cached locally?

dig +trace github.com

Reply

brooklyndude• 10 years ago

Why does it always have to be a "Nation State", have been hanging out with 17 year old's that knew far more about DNS configs than a room of "Cyber-Security-Professisonals", they were clueless, these kids could run circles around them.

Kids.

Reply

tbarbugli4• 10 years ago

I am very surprised this is not getting that much attention on national news.

Reply

ifelsehow• 10 years ago

Reposting imglorp's comment on the root of the comment tree, as it's buried currently. This should restore service for those desperately needing to access Github etc ;)

....point your machine or router's DNS to use opendns resolvers instead of your regular ones: 208.67.222.222 and 208.67.220.220

Reply

fatherzeus• 10 years ago

For people in need of the IPs for their respective services. You can find them here: ipaddress.com or any of the other similar services

Reply

tim_armandpour• 10 years ago

I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues. We sincerely apologize for the inconvenience and thank our customers for their support and patience. Real-time updates on all incidents can be found on our status page and on Twitter at @pagerdutyops and @pagerduty. In case of outages with our regular communications channels, we will update you via email directly.

In addition you can reach out to our customer support team at support@pagerduty.com or +1 (844) 700-3889.

Tim Armandpour, SVP of Product Development, PagerDuty

Reply

kirizt2• 10 years ago

I appreciate the update, but your service has been unavailable for hours already. This is unacceptable for a service whose core value is to ensure that we know about any incidents.

Reply

nickbauman4• 10 years ago

Flush your DNS like the parent said.

Reply

saltyhiker• 10 years ago

Given that a large swath of SaaS services, infrastructure providers, and major sites across the internet are impacted, this seems harsh. Are you unhappy with PagerDuty's choice of DNS provider, or something else they have control over? I don't think anyone saw this particular problem coming.

Reply

AlanBoyce69• 10 years ago

How am i meant to see twitter status updates when twitter is down?

Reply

jorts3• 10 years ago

Please check our status page as an alternative method for updates. Unfortunately, it's also been encountering the same issue so we're sending out an email with the latest updates.

Reply

cjbprime4• 10 years ago

The outage started more than eight hours before you posted this message..

Reply

pfarnsworth2• 10 years ago

To be fair, Hacker News isn't exactly the first place where companies post status messages. I actually applaud him for posting his message here.

Reply

pjlegato2• 10 years ago

"Challenges" is exactly the sort of Dilbertesque euphemism that you should never say in a situation like this.

Calling it a "challenge" implies that there is some difficult, but possible, action that the customer could take to resolve the issue. Since that is not the case, this means either you don't understand what's going on, or you're subtly mocking your customers inadvertently.

Try less to make things sound nice and MBAish, and try more to just communicate honestly and directly using simple language.

Reply

patrickg_zill2• 10 years ago

Sorry if this sounds dickish, but renting 3 servers @ $75 apiece from 3 different dedicated server companies in the USA, putting TinyDNS on them, and using them as backup servers, would have solved your problems hours ago.

Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K/second performance on 700Mhz PIII CPUs: https://lists.isc.org/pipermail/bind-users/2001-June/029457....

EDIT to add: and lengthening TTLs temporarily would mean that those 10K queries would quickly lessen the outage, since each query might last for 12 hours; and large ISPs like Comcast would cache the queries for all their customers, so a single successful query delivered to Comcast would have (some amount) of multiplier effect.

Reply

nodesocket3• 10 years ago

That's not how that sound be done. Just use a mix of two providers. Using your own servers and TinyDNS is silly for million/billion dollar companies.

See MaxCDN for example who uses a mix of dns providers (AWS Route53 and NS1):

    ns-5.awsdns-00.com.   ['205.251.192.5']   [TTL=172800] 
    ns-926.awsdns-51.net.   ['205.251.195.158']   [TTL=172800] 
    ns-1762.awsdns-28.co.uk.   ['205.251.198.226'] (NO GLUE)   [TTL=172800] 
    ns-1295.awsdns-33.org.   ['205.251.197.15'] (NO GLUE)   [TTL=172800] 
    dns1.p03.nsone.net.   ['198.51.44.3']   [TTL=172800] 
    dns2.p03.nsone.net.   ['198.51.45.3']   [TTL=172800] 
    dns3.p03.nsone.net.   ['198.51.44.67']   [TTL=172800] 
    dns4.p03.nsone.net.   ['198.51.45.67']   [TTL=172800]

Curious, are you the kind of person that runs their own smtp email server and complains about GitHub pricing being too expensive?

Reply

464192002d7fe1c4• 10 years ago

+1 for TinyDNS.

I just wish it scaled to multiple cores :(

Reply

nodesocket3• 10 years ago

Running multiple DNS providers is not actually that difficult and certainly not cost prohibitive. I am sure after this, we will see lots of companies adding multiple DNS providers and switching to AWS Route53 (which has always been solid for me).

Reply

pfarnsworth2• 10 years ago

I had the privilege of being on-call during this entire fiasco today and I have to say I was really really disappointed. It's surprising how broken your entire service was when DNS went down. I couldn't acknowledge anything, and my secondary on-call was getting paged because it looked like I wasn't trying to respond. I was getting phone calls for alerts that wasn't even showing up on the web client, etc. Overall, it caused chaos and I was really disappointed.

Reply

tedmiston4• 10 years ago

Hopefully you have a nice SLA with them.

Reply

zippergz• 10 years ago

How does the service you're responsible for work when DNS stops functioning?

Reply

JPHPJ• 10 years ago

"It's surprising how broken your entire service was when DNS went down." lol

Reply

recycle• 10 years ago

PagerDuty outage is the real low point of this whole situation. Email alerts from PagerDuty that should have alerted of the outage in the first place, only got delivered hours later after the whole mess cleared out.

Reply

Rapzid2• 10 years ago

Looks like github and braintree both got AWS dns servers mixed in about the same time. Did they both switch over or is Dyn working with AWS on this?

Reply

Animats11• 10 years ago

Github is currently inaccessible. Can you still compile Rust programs that depend on Github files?

Reply

clay_to_n• 10 years ago

If you need to, you can edit your hosts file to include github's IP addresses (mentioned elsewhere in this thread):

192.30.253.113 github.com

151.101.4.133 assets-cdn.github.com

Reply

steveklabnik• 10 years ago

First of all, the only thing that'd matter is for modifying your dependencies at the moment. If you've previously built the project, and don't touch your deps, GitHub won't be hit.

Second, Cargo only depends on GitHub for the index. For more: http://integer32.com/2016/10/08/bare-minimum-crates-io-mirro...

That includes a link to a mirror run by integer32.

Reply

kakarot• 10 years ago

Don't be a dick. I'm sure their staff has a giant collective migraine right now.

Reply

Mizza• 10 years ago

Although I don't like to to recommend Google products, they provide a provide a public DNS-over-HTTPS interface that should be useful for people who want to add specific entries into their /etc/hosts files: https://dns.google.com/query?name=github.com&type=A&dnssec=t...

Reply

djhworld• 10 years ago

At work earlier we was seeing hostname resolution errors with applications trying to contact amazon s3 from on premises infrastructure.

This was in eu-west-1, but it coincided with a bunch of other systems in the organisation having problems at the same time.

Additionally CloudWatch logs seemed to be completely broken for about 30 minutes on the Amazon Console.

Reply

Animats11• 10 years ago

So who was prepared for this? Pornhub:

pornhub.com:

    Name Server: ns1.p44.dynect.net
    Name Server: ns2.p44.dynect.net
    Name Server: ns3.p44.dynect.net
    Name Server: ns4.p44.dynect.net
    Name Server: sdns3.ultradns.biz
    Name Server: sdns3.ultradns.com
    Name Server: sdns3.ultradns.net
    Name Server: sdns3.ultradns.org

ultradns.biz:
Name Server: PDNS196.ULTRADNS.ORG Name Server: ARI.ALPHA.ARIDNS.NET.AU Name Server: ARI.BETA.ARIDNS.NET.AU Name Server: ARI.GAMMA.ARIDNS.NET.AU Name Server: ARI.DELTA.ARIDNS.NET.AU Name Server: PDNS196.ULTRADNS.NET Name Server: PDNS196.ULTRADNS.COM Name Server: PDNS196.ULTRADNS.BIZ Name Server: PDNS196.ULTRADNS.INFO Name Server: PDNS196.ULTRADNS.CO.UK

Reply

Animats11• 10 years ago

Looks like Pagerduty just dumped Dyn:

pagerduty.com:

    Name Server: NS-219.AWSDNS-27.COM
    Name Server: NS-1198.AWSDNS-21.ORG
    Name Server: NS-1569.AWSDNS-04.CO.UK
    Name Server: NS-739.AWSDNS-28.NET

Pagerduty annoucement: "If you are having issues reaching any pagerduty.com address please flush your DNS cache to resolve the issue."

Reply

Animats11• 10 years ago

Github just added AWS DNS:

github.com:

    Name Server: ns2.p16.dynect.net
    Name Server: ns-1283.awsdns-32.org.
    Name Server: ns-1707.awsdns-21.co.uk.
    Name Server: ns-421.awsdns-52.com.
    Name Server: ns1.p16.dynect.net
    Name Server: ns4.p16.dynect.net
    Name Server: ns3.p16.dynect.net
    Name Server: ns-520.awsdns-01.net.

Reply

razster2• 10 years ago

ultradns.biz has been down as well.

Reply

Animats11• 10 years ago

"ultradns.biz" is not responding to pings, but their DNS servers are responding to DNS queries properly:

    nslookup
    > server pdns196.ultradns.biz
    Default server: pdns196.ultradns.biz
    Address: 156.154.66.196#53
    Default server: pdns196.ultradns.biz
    Address: 2610:a1:1015::e8#53
    > pornhub.com
    Server:		pdns196.ultradns.biz
    Address:	156.154.66.196#53
    Name:	pornhub.com
    Address: 31.192.120.36

Right now, if your site is in trouble, I'd suggest getting UltraDNS service and AWS DNS service, and some obscure service as well, and put them them all in your domain registration. DNS service is cheap. Get some redundancy going. We have no idea how long this DDOS attack will last. It's not costing the attackers anything. They might leave it running for days.

Reply

BinaryIdiot• 10 years ago

When your business depends on your infrastructure being up and running you try to prepare for anything. Then again Twitter is down so...

Reply

Animats11• 10 years ago

Twitter is still 100% on Dyn.

twitter.com:

    Name Server: NS1.P34.DYNECT.NET
    Name Server: NS4.P34.DYNECT.NET
    Name Server: NS2.P34.DYNECT.NET
    Name Server: NS3.P34.DYNECT.NET

They're probably using some geographically based DNS distribution scheme which they can't quickly move to other DNS servers.

Reply

tibu• 10 years ago

Amazon was Using Dyn, now they added also UltraDNS too:

  Name Server: pdns1.ultradns.net 
  Name Server: pdns6.ultradns.co.uk 
  Name Server: ns3.p31.dynect.net 
  Name Server: ns1.p31.dynect.net 
  Name Server: ns4.p31.dynect.net 
  Name Server: ns2.p31.dynect.net

Reply

Animats11• 10 years ago

Twilio just dumped Dyn, and is now available again.

twilio.com:

    Name Server: ns3.dnsmadeeasy.com
    Name Server: ns2.dnsmadeeasy.com
    Name Server: ns4.dnsmadeeasy.com
    Name Server: ns1.dnsmadeeasy.com
    Name Server: ns0.dnsmadeeasy.com

Reply

Animats11• 10 years ago

Digikey just dumped Dyn, and is now back up.

digikey.com:

    Name Server: cbru.br.ns.els-gms.att.net
    Name Server: ns2.digikey.com
    Name Server: cmtu.mt.ns.els-gms.att.net
    Name Server: ns1.digikey.com

Reply

elmigranto2• 10 years ago

I've managed to (seemingly) save my browsing with Yandex DNS:

    77.88.8.8
    77.88.8.1

https://dns.yandex.ru

Reply

elwell7• 10 years ago

I'm sure yandex is safe, but I'm wary of using a anything dns.*.ru to route my traffic to potential phishing versions of sites.

Reply

elmigranto2• 10 years ago

If it makes you more at ease, use this one :)

https://dns.yandex.com

Reply

dev_1024• 10 years ago

How come you can access these sites from some countries? I imagine there are lots of name servers and that the attackers are specifically targeting servers for US?

Reply

azaydak2• 10 years ago

Quick question for you all. Just two days ago I registered two domain names at dynu (not dyn). Early this morning I a cold call from a company in India who knew the domain names and my phone number and was calling to ask if I wanted them to help me manage my website cheaply. Also, this morning I got a spam text from someone who claimed to by godaddy offering the same thing. Now I protect my number really well so this is the first time in 5+ years that I ever got spam texts or calls to my number. Do you think Dynu was also hacked?! Or maybe Dynu sells client numbers (which is how the guy in India claimed to get my number) and it was just by random chance that this happened at the same time as the Dyn hack.

Reply

shortstuffsushi2• 10 years ago

Fwiw, this isn't a hack, this is a DDoS (denial of service). It seems almost certain that your information was either given out by dynu, or your WHOIS record isn't protected. Check your domains out with your favorite WHOIS tool first. Otherwise... time for an awkward conversation with dynu.

Reply

azaydak2• 10 years ago

Right, hack wasn't the right word. Anyway, thanks for the info.

Reply

danyork9• 10 years ago

Agree with shortstuffsushi that this is just someone getting your domain name info and spamming you. It sadly happens all the time.

Go to http://whois.icann.org/en and enter your domain name and see what info is public about you. If all your info is public, you may want to see if your registrar offers "private" registration where your info does not appear in WHOIS.

Reply

peatmoss22• 10 years ago

Need to get in to dyn.com to download your zone files add this to your hosts file: 204.13.248.106 www.dyn.com 204.13.248.106 dyn.com 216.146.41.66 manage.dynect.net 151.101.33.7 static.dyn.com

Reply

DenisM2• 10 years ago

The DDoS problems, at least those not related to spoofing IPs, could be curtailed if we provide a strong incentive to the ISPs to work on it.

Let's hold the ISPs financially liable for the harmful traffic that comes from their network. If a client reports a harmful IP to the ISP, every bit of subsequent traffic sent from that IP to this client carries a penalty.

Yeah, I know, routing tables are small, yada yada. If we put thumbscrews to the ISPs they will find a way to block a few thousands IPs of the typical botnet, even it requires buying new switches from Cisco & co.

Incentives drive behavior.

Reply

Florin_Andrei2• 10 years ago

Put the thumbscrews on the IoT manufacturers instead, so they don't release widgets with bad security, so the problem is eliminated at its root.

You wouldn't allow car manufacturers to sell cars with faulty airbags, why do we allow device manufacturers to provide plentiful firepower for bad actors?

Reply

DenisM2• 10 years ago

With ISPs it's a lot easier - you know who your ISP is, so either they respect your blacklist or they automatically owe you money.

How would you even start chasing a manufacturer of a cheap IP cam from China? How many of them can you chase at once?

Reply

mmaunder• 10 years ago

Third attack underway: https://twitter.com/AlexJamesFitz (as of 10 mins ago)

Reply

lips• 10 years ago

How many DNS services ala Dyn exist? Is it not still massively significant that a successful attack can be launched on even one of these?

Reply

dmalvarado2• 10 years ago

This may be dumb, but someone enlighten me:

If this kind of attacking does escalate, wouldn't it be possible to simply cut off requests from outside the United States at the points of entry? Basically, turning the US into an intranet?

Reply

danyork9• 10 years ago

We don't know yet, but the attack very easily could have been coming from a botnet of devices entirely inside the US. Geographic borders don't matter much at all for the Internet.

Reply

dmalvarado2• 10 years ago

But even if it were, the creator of the botnet would first have to gain control and then issue a command, right? How would either of those things be possible from outside if there was no connection into the US?

Reply

shortstuffsushi2• 10 years ago

In (well, after) attacks like this, and really any other massive DDOS, shouldn't it be possible to identify potential botnets and try to take them out (notify their owners that they're being used, notify their hosting providers, etc) so that they can't be used again in the future?

Reply

LeanderK2• 10 years ago

Can someone explain why this is so bad? I think the internet handled the downtime of Dyn pretty great, not reaching github wasn't exactly pleasing, but i added the ip temporary to /etc/hosts and the problem was solved. Isn't the best strategy to accept that attacks will continue and systems may go down and design for resilience? If so this attack can serve as a warning and as a check that we can handle these types of attacks. I am a bit exaggerating, but i would imagine that constant attacks keep the internet resilient and healthy. An unchallenged internet may be the greater risk.

Reply

rak00n• 10 years ago

You're assuming we'll build immunity fast enough. What if we don't?

Attacks at this scale can bring a significant part of Internet down. The economic affect can be just as bad as a war.

Reply

LeanderK2• 10 years ago

but just pretending no one will try bad things won't help either. It may speed up the progress and will prevent bringing a significant part of the internet down in the future and also remind everybody that these things can (and probably will) happen.

Reply

edgartaor• 10 years ago

I'm curious. What kind of infrastructure you need to make this massive attack?

Reply

saidajigumi• 10 years ago

via [1]: "Dyn says today's DDoS are in part being caused by Mirai botnet, which recently caused record-sized attack [2]"

[1] https://twitter.com/AlexJamesFitz/status/789562789920636928

[2] https://krebsonsecurity.com/2016/10/source-code-for-iot-botn...

tl;dr: a substantial part of this attack is a botnet of IoT devices.

Reply

Animats11• 10 years ago

Analysis of the Mirai botnet: [1]

This is worth reading. It has links to copies of the code and names the known control servers. Quite a bit is known now about how this thing works.

The bots talk to control servers and report servers. The attacker appears to communicate with the report servers over Tor.

[1] http://blog.level3.com/security/grinch-stole-iot/

Reply

rybosome5• 10 years ago

I'm wondering, from a regulatory perspective, what might be done to mitigate DDoS attacks in the future?

From comments made on this and other similar posts in the past, I've gathered the following:

1) Malicious traffic often uses a spoofed IP address, which is detectable by ISPs. What if ISPs were not allowed to forward such traffic?

2) There is no way for a service to exert back pressure. What if there was? e.g. send a response indicating the request was malicious (or simply unwanted due to current traffic levels), and a router along the way would refuse to send follow up requests for some time. There is HTTP status code 429, but that is entirely dependent on a well-behaved client. I'm talking about something at the packet level, enforced by every hop along the way.

3) I believe it is suspected that a substantial portion of the traffic is from compromised IoT devices. What if IoT devices were required to continually pass some sort of a health check to make other HTTP requests? This could be enforced at the hardware/firmware level (much harder to change with malware), and, say, send a signature of the currently running binary (or binaries) to a remote server which gave the thumbs up/down.

Reply

tim333• 10 years ago

One thing that occured to me regulations wise is to require IoT devices to have some minimum level of security such as a unique hard password rather than it just being "admin" or some such. You could enforce it for items sold in the US or EU and the Chinese manufacturers would probably follow so their goods could be sold easily.

Reply

nkristoffersen2• 10 years ago

I've noticed a lot of wifi routers are doing this now. Which is pretty great. All appear to be unique passwords for each router.

Reply

thomask02• 10 years ago

Let ISP's shutdown traffic of customers with compromised devices.

Reply

nkristoffersen2• 10 years ago

Lets avoid giving ISP's more power than they have. Next thing we will see is "oh we thought that person was using a compromised device" for any disagreement.

Reply

chrismarlow92• 10 years ago

Regarding point 2, I can think of a few ways to utilize that mechanism itself as a way to DDoS something. Sometimes the security mechanisms themselves are the attack vectors.

Reply

rybosome5• 10 years ago

Can you explain how? Not asking in a challenging way, I'd like to learn for my own edification.

Reply

amluto• 10 years ago

What if IoT devices were required to continually pass some sort of a health check to make other HTTP requests?

Even better, what if IoT devices were required to pass some health check to operate at all. This could be as simple as a verified boot plus a forcible reboot every now and then.

Reply

codedokode2• 10 years ago

Today the peering agreements are made so that ISPs get paid for whatever traffic they pass through. They have no financial motivation to change anything. And as the Internet is decentralized you cannot order them to do anything. So everyone has to protect from DDOS by themselves.

Reply

rybosome5• 10 years ago

Today the peering agreements are made so that ISP's get paid for whatever traffic they pass through. They have no financial motivation to change anything.

That seems believable.

And as the Internet is decentralized you cannot order them to do anything.

...that doesn't. Being decentralized doesn't render them immune to regulation. If all major networks responsible for large scale peering were required not to pass on a certain type of traffic, it would be quite difficult to route around that. Yes, if only some did, this would be routed around.

Reply

pawal• 10 years ago

DNS was designed so that you can have multiple operators for your authoritative name servers.

Who would have thought adding a spof to your infrastructure would ever be a problem?

Reply

danyork9• 10 years ago

Journalist and security researcher Brian Krebs believes this is someone doing a DDoS as payback for research into questionable "DDoS mitigation services" that he and Dyn's Doug Madory did. Doug just presented his results yesterday at NANOG and Krebs believes this is payback. Read more: https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twit...

Reply

LordHumungous2• 10 years ago

If so that's a quick turnaround.

Reply

danyork9• 10 years ago

Well, Krebs sees this as an extension of the attacks that took down his site a few weeks ago after he wrote about this research. So he wrote about it, attackers take down his site. His co-author Doug Madory speaks about it, attackers take down Madory's employer's site.

Krebs indicates in an update at the end that a source had heard rumors in criminal channels that an attack against Dyn was being planned.

Doug Madory's presentation was on the agenda for NANOG and so attackers would have had plenty of time to know about it.

Reply

wweiss1230• 10 years ago

How can I, a proficient web developer but one with little experience working directly with its underlying infrastructure, help in whatever effort is being down to thwart this and related attacks? I feel a moral obligation to help as these attacks seem a grave threat to our economy and could cause unrest given the current political climate. Thanks.

Reply

dmourati• 10 years ago

Read all the analysis you can to form a better understanding of how this all works. Use that information to design and run more resilient services in the future. Teach what you have learned to others.

Reply

danyork9• 10 years ago

Other HN threads on related articles:

Krebs on Security: https://news.ycombinator.com/item?id=12761859

NY Times: https://news.ycombinator.com/item?id=12765652

Bloomberg: https://news.ycombinator.com/item?id=12763501

Dyn: https://news.ycombinator.com/item?id=12764650

Reply

danyork9• 10 years ago

There's a bit of exquisite irony in the fact that just yesterday an article on the Dyn blog was:

Recent IoT-based Attacks: What Is the Impact On Managed DNS Operators? - http://hub.dyn.com/traffic-management/recent-iot-based-attac...

It's a good piece about how IoT-based DDoS attacks are carried out. And now Dyn has the answer...

HN thread about that article at: https://news.ycombinator.com/item?id=12764650

Reply

anonymousjunior• 10 years ago

No idea if this would work, but could people theoretically just ping flood the IOT devices involved to mitigate the attack?

They run some sort of web server since most devices provide some web interface, so clearly there's a port open which could be hit if the IP is know, and with the shoddy security in these devices I'd wonder if their local (likely low performance) hardware would be susceptible to something as simple as a ping flood attack.

Reply

wav-part• 10 years ago

Why is there even a concept of managed DNS ? Arnt we already paying >$1M/yr so that we can get 32 bit integer from a string ? This does not make sense.

Reply

dboreham• 10 years ago

What this event shows is that using DNS as a load routing/balancing mechanism is a bad idea (because that's why folks have low TTL and an inability to specify truly redundant secondaries).

Reply

DDoS Attack Against Dyn Managed DNS

Related Stories