What happens when DNS in China is poisoned and pointed at your server

http://furbo.org/2015/01/22/fear-china/

by aaronbrethorst • 12 years ago

158 55 12 years ago

for those wondering, extended title could be along the lines of "Experiencing the effects of a DNS poisoning attack"

acdha• 12 years ago

This is why I really like having Varnish in front of anything Internet-facing: for something like returning a 400 for an un-approved Host header, you're going to be effectively bottlenecked by the network interface.

thaumaturgy• 12 years ago

That's cute, the IP address for his demo curl command is ... daringfireball's web server.

pronoiac• 12 years ago

And if you run it, you get the Daring Fireball homepage, which surprised me - it's not configured for this problem.

Anthony-G• 12 years ago

I noticed that as well. However, they seem to be friends / colleagues / associates. A web search for "Craig Hockenberry daringfireball" shows that Craig Hockenberry is (or has been) a contributor to John Gruber's show and John Gruber has recommended one of Craig's apps [1].

[1] http://safetylightapp.com/

coldcode• 12 years ago

I wonder if this could really be harnessed in China as a weapon.

brazzledazzle• 12 years ago

That's the only reason I can imagine they'd respond with anything else besides 127.0.0.1 or some internal web server with warnings to obey their overlords.

beachstartup2• 12 years ago

why is 'out' inbound, and 'in' outbound?

faster2• 12 years ago

Because it's from the perspective of the router; traffic out of the router is TO his network.

beachstartup2• 12 years ago

if he owns/operates the router, it still doesn't make any sense.

in our network, 'out' means outbound from the edge, and 'in' means in from the edge, in all cases.

fallous3• 12 years ago

We were the lucky targets of China's DNS poisoning on Monday the 19th. Any requests from clients in China to Edgecast's CDN was instead thrown at our public IP. After deducing what was going on, we ended up blacklisting large (class A in some cases) APNIC address blocks assigned to China.

Interestingly, it wasn't our webservers that were overwhelmed but instead the Cisco firewall that sits in front of them. 25K concurrent connections made it decidedly unhappy.

meowface• 12 years ago

My company's web server saw a similar unusual surge in traffic with host headers and URLs of Chinese sites for an 8 hour window a few months ago. All the IPs were from a single Chinese ISP, though the volume of the traffic was a lot lower than what OP observed. Seems like these events aren't uncommon.

jlmorton• 12 years ago

Same here. We received well over 150mbit/s, also on Monday the 19th, also from China. We also saw occassional spikes starting on January 9th. We saw the same BitTorrent /announce traffic, and lots of other random traffic.

We ended up mitigating it by moving our IP address on that host, and blocking all input on the old address.

themoogle• 12 years ago

you are lucky you only got 150mbit/s

Here is what I had to deal with on the 9nth Ramming my server...

http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png

Yes that is almost 1000mbit/s All coming from china...

broknbottle• 12 years ago

asa 5505 with upgraded license?

fallous3• 11 years ago

Yeah, rackspace-provided unit.

smilekzs3• 12 years ago

They (the GFW) intend to have every server owner ban China IPs so that they could claim "you guys are also banning us, what position are you in to blame us for GFW?"

pavel_lishin• 12 years ago

Big ol' [citation needed] on that one.

smilekzs3• 12 years ago

/* That's what he said */

Well, do you know what the Communist Party thinks?

bm212• 12 years ago

this seems like a clever way of censoring internet access from within China - get the site owners to do the censorship for you.

emiunet• 12 years ago

Thanks a lot for the post. What suprised me the most was the fact that Google is serving 40K (now 47K in live statistic) requests per second. That was way below what I thought. Each Apache server without real optimization running on a typical physical machine would be able to serve 40-50 requests/second; so they only need a thousand of servers to do 50K requests/second. Or am I missing something?

Narkov• 12 years ago

Yeah, the actual search bit.

Serving a static file is pretty simple at 40-50 req/sec but searching an entire index of the Internet @ 40-50 req/sec is harder.

intr1nsic• 12 years ago

Your missing factor is "the fact that Google is serving 40K (now 47K in live statistic) requests per second" is wrong.

"The number of requests peaked out at 52 Mbps. Let’s put that number in perspective". Perspective is 52 Mbps isn't a small number, but not massive either.

noonespecial• 12 years ago

Not particularly feasible but it would be awesome to simply proxy all of the requests to their intended destination.

IgorPartola• 12 years ago

Oh man. That would be pretty cool. Somebody like CloudFlare could do that since they proxy for lots of smaller sites. Then again, HTTPS will add even more complexity.

hobs• 12 years ago

Even if it likely impossible for anyone smaller than CF, it might even mean they would move the attack off sooner since it wasnt doing anything :)

throwawayaway• 12 years ago

that would be good, or proof of something not widely known in china that is officially denied by china.

bombless• 12 years ago

I live in China. We have a department named 工信部, it blocks a site by polluting the DNS and point the domain name to random foreign IP. I think this explains what happened here.

zaroth• 12 years ago

A random foreign IP... Wow, yeah, that just makes perfect sense.

Clearly all requests from China with an incorrect host header should just be served a Goatse-style 'hello.jpg'.

methou• 12 years ago

A Chinese here, I'm deeply ashamed by this malicious action at a national level. though DNS poisoning wasn't new for Chinese netizens but it was always poisoning A records to varies foreign military sites like DoD or some Korea sites (which also makes no sense at all). At scale of this big, those shameless bastards running state censorship machines has been outrageously attacking on random innocent site owner, this is far beyond unacceptable.

And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse.

Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.

sorpaas• 12 years ago

Also Chinese here, GFW is really a shame for us.

jasonm23• 12 years ago

Many of us outside GFW/Cn are aware of the "decoration" (we call it Astroturfing in the US) - it's so transparent that it's often joked about on Reddit and similar sites.

fallous3• 12 years ago

I have seriously contemplated a special request handler for Chinese traffic that is being directed to the wrong host on our servers. Something along the lines of "Chinese government censors broke your internet" or the like.

I figure it either helps inform the citizens as to why this link isn't working or the message itself causes the GFW to ban our IP as well.

haosdent• 12 years ago

Today, GFW start block VPN. So it only would become worse in the future..............

userbinator• 12 years ago

Seeing /announce reminded me of this:

https://torrentfreak.com/bittorrent-users-ddos-websites-with...

Using DHT is also a particularly effective way of getting massive amounts of UDP traffic directed at you, which does not stop even after stopping the torrents. It subsides eventually, but can be quite irritating.

kesor3• 12 years ago

I wrote a blog post about a DDoS on a website by the Chinese at http://dvps.me/ddos-attack-by-torrent. Falsely I assumed that it was torrent tracker injection, while in fact this looks like a spike in the affect of DNS poisoning in China.

http://serverfault.com/questions/656093/mod-security-block-r... describes the problem as well.

https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... also describes the same thing.

As well as http://www.webhostingtalk.com/showthread.php?p=9351951

kesor3• 12 years ago

One possibility is to block requests coming with a different 'Host:' header than your website. Or just block the URLs like described in dvps.me/ddos-attack-by-torrent

kesor3• 12 years ago

http://en.wikipedia.org/wiki/Golden_Shield_Project

avinassh• 12 years ago

and also: http://www.youtube.com/watch?v=rH8UitdFb2o

pilif• 12 years ago

Nitpick:

One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases:
>If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address.

ServerName is the name Apache reports in Error messages and using for redirects. It has nothing to do with the Host-Header sent by the client. If it did, a reverse lookup definitely wouldn't be of any help either.

His remedy was still ok though - just for the wrong reasons.

xorcist• 12 years ago

It's not a nitpick. It's the difference between making stuff up and reading the documentation. It does matter when reason about how to configure your server.

chockenberry• 12 years ago

I'm the author of the post linked above: thanks for the clarification about ServerName. I spent a lot of time reading that documentation and got pretty confused about how it was actually being used.

marcosdumay• 12 years ago

Complementing it, when there is no Host HTTP header, apache displays the default virtual server for the port used. Unless you've set it, it's the first entry on your config file.

bastih• 12 years ago

New @ GFW: Now providing infrastructure stress testing for the western world.

nl• 12 years ago

Surely at this level of traffic it would make sense to get in contact with your provider and see if they can block it upstream?

thspimpolds• 12 years ago

This is why i cloud front personal hosts, and fastly work hosts. I never open my server to the worlds with cdn' being so cheap or free now

19890604• 12 years ago

A Chinese here. I would suggest, if possible, use 301 to redirect the malicious requests(which are without recognized hostnames in HTTP header) to <www.gov.cn>, which is the government's face.

smilekzs3• 12 years ago

Good idea!

antihero• 12 years ago

thousands of Apache child processes

Were you still using preforking MPM or something? Surely switching to threaded or even better, event MPM would have mitigated this to a degree...

cesarb• 12 years ago

Or put a nginx in front of the Apache, disable keepalives in the Apache server (letting nginx deal with keepalive from the clients), and reduce MaxClients so even if they're all active the server does not go into swap.

(I've done exactly that for a friend, it made a huge difference in how much traffic his box can handle).

m11r• 12 years ago

This is something I was wondering, too. Using nginx, I can easily top 50,000 req/s (45Mbit/s) on smaller, static/cached content, using 4 cores and < 1GB RAM.

WillHuxtable• 12 years ago

Maybe China are trying a new approach to censorship. Instead of blocking things themselves, perhaps they're trying to get sysadmins to block all of China. I imagine it would save them a lot of work.

pdoconnell• 12 years ago

We started seeing this around the 9th I believe. We were not the only people as well, based on a somewhat late ISC post[1].

We thought it was a new form of intelligent blackholing. Instead of sending traffic to IPs that could easily be blacklisted by tools to get around the firewall, the Great Firewall would start sending them to random "good" IPs for the same result. Others seem to think the same thing[2].

[1] https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... [2] https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visi...

sneakest• 11 years ago

I had one of my clients have the same thing happen to them 3 nights in a row about 10 days ago. All of the sudden they got dumped a ton of traffic out of no where. Torrent tracker updates and what seemed like legitimate traffic routed to the wrong ip. Then after about an hour most of it stopped. During the attack and after we changed some firewall settings and clamped down our request per second and connection per ip limits on the web servers. One big change was having the web servers respond 444 to any host request that was not configured (i.e. default). So if someone came to the ip looking for say google.com they would get a 444 response as it is obviously misrouted traffic.

Then the next day the same thing happened but much more traffic. The 444 change helped some but there was just too much traffic for the web servers to handle quickly so they bogged down to a crawl. Luckily we were able to figure out through severfault and some other searches that it appears to be DNS poisoning coming from China. We ended up banning the entire country.

The third day the same thing happened but because we had blocked the traffic from China at our outside firewall the servers were unaffected. Since then we have seen some flashes of traffic being blocked but not nearly as much as before.

In all cases the increased volume of traffic only lasted about an hour. The only thing I can surmise about the length of time is DNS records only being cached for about an hour. So after that time the poisoned cached DNS records were replaced by the real resolving ip address. In the case of the article's author their servers suffered a much longer attack than we did. I am not sure if it is the Chinese government doing it or an attack for hire scheme using holes in many Chinese based DNS servers.

TLDR; Remove default server settings from your web servers and have the default server block respond 444 or 404. This may help stave off the attack until your incoming traffic takes up all available resources of the web servers. Of course you could always block all traffic based in China like we did.

Here is info on the status codes incase you are curious: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_C...

What happens when DNS in China is poisoned and pointed at your server

Related Stories