Level 3 are now hijacking failed DNS requests for ad revenue on 4.2.2.x

http://james.bertelson.me/blog/2014/01/level-3-are-now-hijacking-failed-dns-requests-for-ad-revenue-on-4-2-2-x/

by doctorshady • 12 years ago

150 74 12 years ago

james.bertelson.me

mcpherrinm• 12 years ago

Yep, I'm seeing what should be NXDOMAIN results returning the IP 198.105.254.11 which brings me to a page like http://searchguide.level3.com/search/?q=http://198.105.254.1...

Does anyone know if actual Level3 customers see this page, or is it only for off-network requests? Up until the end of last year, my employer had a Level3 internet connection and we legitimately used 4.2.2.1 as our DNS recursive resolver. I'd be pretty pissed if they returned spammy results to their customers, but to non-customers, well, I don't care: That's what you get. Use a DNS server that somebody says you're allowed to (8.8.8.8, maybe)

jlgaddis3• 12 years ago

Querying from one of my personal servers on a Level3 DIA circuit, I am getting NXDOMAINs for non-existent hostnames.

_Lemon_• 12 years ago

I heard years ago that Level 3 were trying to encourage people (non-customers?) not to use these DNS servers. I guess this is one way to ask people not to use them.

Having said that, 8.8.8.8, Google DNS, has been planted firmly in my memory as my go to "is this machine up?" IP.

tracker1• 12 years ago

My issue is that level3's dns has always been very fast, and more importantly, up... When google's dns is slow, level3 is fast... when my isp's dns goes down or wonky, level3's is up... I'd pay them $10/year to use them without the ads.

karlshea2• 12 years ago

Why not use OpenDNS then? If you have an account you can configure them to behave however you'd like.

dsl6• 12 years ago

http://www.opendns.com/home-internet-security/premium-dns/

intslack2• 12 years ago

When their DNS seemed to go down a few days ago I also noticed this behavior and immediately switched to a local independent ISP's DNS that namebench spit out.

Doesn't seem too out of character, if they're returning this for actual customers, from the company that most likely allowed the US government to tap into Google and Yahoo's fiber lines.

http://www.nytimes.com/2013/11/26/technology/a-peephole-for-...

dsl6• 12 years ago

That is a pretty big, and very uneducated, accusation.

Since the 1970s the CIA has been installing taps on undersea cables using specialized submarines. This is well documented, and it is well known that the NSA prefers to use methods that involve the least amount of interaction from uncleared individuals even if it is at a much greater expense.

pyvpx2• 12 years ago

the NSA is the one installing the taps. CIA tried to do a radio tower once -- or maybe that was the FBI. anyway, when it comes to sigint, it's NSA all day every day.

intslack2• 12 years ago

The article sources "three people with knowledge of Google’s and Yahoo’s systems who spoke on the condition of anonymity," so I wouldn't really call it uneducated.

Is the accusation that Level 3 has a 'black room' agreement with the NSA totally out of the realm of possibility? We know AT&T has such an agreement after Mark Klein exposed Room 641A years ago.

skymt• 12 years ago

Seems reasonable. Those servers were never intended for widespread public use, so they may as well make back some funds for upkeep, and maybe encourage some more-technical users to switch away.

Here's a blog post with some background on these servers: http://www.tummy.com/articles/famous-dns-server/

oasisbob• 12 years ago

I'm fairly sure that the "4.2.2.x was never meant to be public" line is a myth. Though the NANOG thread cited in that post is good historical background, it is contradicted by more modern sources:

"[...] DNS infrastructure is largely split into two types; open (public) and closed (private). Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

- Mark Taylor, VP at Level3

http://blog.level3.com/level-3-network/a-flawed-study-of-cdn...

I can't find any cite where anyone else who I would consider a reliable source in the DNS world (Vixie, &c) repeat this claim. To the contrary, Level 3 is often grouped with Google, OpenDNS and others in discussions of open public resolvers [1][2][3], and those in the know never seem to speak up and say otherwise in these discussions.

That being said, I have absolutely no personal knowledge on any of this.

[1] http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_M...

[2] http://markmail.org/message/gh7f2wvfbn5mpvuq

[3] http://www.circleid.com/posts/87143_dns_not_a_guessing_game/...

rdtsc• 12 years ago

Presumably it takes non-0 costs to maintain 4.2.2.x DNS servers. While I'd want to believe Google and L3 just try to help out the public at large with free DNS services, I suspect they are not doing that just purely out of altruism.

inglor• 12 years ago

I'm surprised Google hasn't done it yet

gwu78• 12 years ago

"Those servers were never intended for widespread public use, ..."

What is the test for "intended for widespread public use"?

If a server is configured to accept queries from any IP address, can we conclude anything about "intent"?

If the server admin does nothing to stop widespread public use (which is trivial to do of course), can we conclude anything about intent?

Is there some other clue we need to look for?

Under your reasoning, it's "reasonable" to provide open resolvers and hijack NXDOMAIN responses, so long as there is no "intent" for widespread public use.

Interesting.

reedloden• 12 years ago

Is this only affecting Level3's 4.2.2.x nameservers (when it happens), or are their 209.244.0.x ones doing this as well?

5teev• 12 years ago

Comcast also did this to me. Not one of the several tech support people I talked with seemed to be aware of Comcast's non-hijacking DNS servers at 75.75.75.75 and 75.75.76.76.

pudquick• 12 years ago

Fortunately they haven't done this for 2 years. They killed it when they flipped on DNSSEC because the practice of NXDOMAIN hijacking is incompatible:

http://dns.comcast.net/index.php/help#faq2

Now no customers from Comcast suffer this.

... JavaScript and HTML injection when you reach a cap limit in a throttled market or when you get a cease and desist for pirating, however, is another matter.

gergles• 12 years ago

Why Am I Here?

The Example Net Web Helper has been enabled to provide helpful searches from web address errors. You entered an unknown name that the Example Net service used to present site suggestions which you may find useful. Clicking any of these suggestions provides you with Yahoo! search results, which may include relevant sponsored links. Why should I use this?

The Example Net Web Helper makes finding what you are looking for easier and more convenient. The service uses the entered non-existing website name to determine useful search results. Often, you will see a desired website or page that meets your needs. Do you track my Internet usage?

No. The Example Net Web Helper simply redirects queries to non-existing domain names to a useful search results page instead of a cryptic error message page or browser-defined page.

The "Example Net" huh?

kbar13• 12 years ago

don't use L3's 4.2.2.x resolvers, as they aren't meant for public use, unlike google's public dns

__david__2• 12 years ago

You say that, but if they weren't meant for public use, they wouldn't be accessible to the public. Time Warner's DNS, for example, are not available from outside their network.

eli3• 12 years ago

And if your car is unlocked it's meant for joyriding?

jlgaddis3• 12 years ago

Some of us pay Level3 four or five figures a month. I checked and receive NXDOMAINs (like I should) but I would immediately open a ticket if they were fucking with my DNS.

That said, we run our own recursive servers and don't rely on Level3's.

dsl6• 12 years ago

To be perfectly clear: It is not hijacking when you are sending them queries for which you should have no reasonable expectation that they service. If you are actually a Level 3 customer, call your sales rep, but I believe this is only for non-customers.

EDIT: By the way, this is the actual company operating the "service" behind the scenes for Level 3 http://www.xerocole.com/searchguide/

wpietri2• 12 years ago

If you're not going to service a query, there are perfectly good ways to do that. Since they didn't do one of those, I'm happy to call this hijacking.

pyvpx2• 12 years ago

and one of them is returning bogus junk. why should they (the NSP) care? especially if you aren't paying them for a network service? it's not hijacking -- 4.2.2.2 isn't a public resolver and never was. it just happened to become one.

ballard2• 12 years ago

Yes, it can screw up caches and return unexpected content.

Inserting an ad where an error should be is detrimental because the implementation often break standards endpoint and intermediate apps depend on (http code, DNS query answer/s). An app that was expecting just JSON now dies in some horrible (maybe silent or end-user confusing stacktrace) way.

nodata• 12 years ago

It's hijacking. Try and talk your way around it all you like, but call it what it is.

dsl6• 12 years ago

I'm going to go sit in your car and wait for you to start driving to the store, then start screaming bloody murder that you are hijacking the car and not taking me to work.

matthewbadeau2• 12 years ago

I have to plug OpenNIC[1] anytime I hear of a DNS hijacking story. OpenNIC is a peer run network of DNS servers that are open for public use.

[1]http://www.opennicproject.org/

aroman• 12 years ago

This looks cool, but why should I use and trust this over something like Google's Public DNS?

matthewbadeau2• 12 years ago

Honestly, I have no argument for OpenNIC over Google's DNS that will convert you right now. It really depends on who you trust more.. strangers over the internet or a large corp who was the target of espionage. That's really up to you to decide the lesser of two evils.

A cool thing about OpenNIC is that they offer alternative TLDs that aren't part of ICANN's gTLDs. Also, the owners of the public servers strive to be as open as possible with their policies and features, such as no logging or using DNSCrypt. One of them even offers DNS level ad blocking, though I don't like it because I prefer the internet at its purest form and that policy doesn't seem to flow well with their anti-censorship mantra.

diakritikal• 12 years ago

I'm rather curious. I thought this kind of predatory network shenanigans was par for the course in the US?

nly2• 12 years ago

Its common in the UK as well. Virgin Media (pretty much the only significant cable ISP in the UK) hijack NXDOMAIN, but let you turn it off via a link from the pages it serves.

nknighthb• 12 years ago

By residential ISPs, sure. Level 3 is not a residential last-mile provider. I'd be surprised if you could get any sort of service out of them for less than $1k/month.

mfincham2• 12 years ago

Suggestion: if your network provider's recursive DNS service sucks so much that you cannot bear to use it (and even if it doesn't, quite frankly) your next best bet is probably to install unbound (https://unbound.net/) listening on localhost on your workstation.

Not only does this give you known-good DNS resolution, but you can also enable DNSSEC validation and be fairly confident that it'll actually do its job in preventing your local machine from resolving poisoned zones.

aidenn0• 12 years ago

I set up a caching recursive DNS server on my lan, but had to add a second entry for a caching public DNS server since:

1) Most DNS entries these days seem to have very short TTLs

2) Occasionally the recursive queries would fail

mfincham2• 12 years ago

Would you mind clarifying what you mean by "second entry"?

ballard2• 12 years ago

I use mdnsresponder nomulitcastannounce -> dnsmasq 127.0.0.1#53 -> dnscrypt-proxy 127.0.0.1#54 -> an encrypted DNS proxy that does dnssec. All of which is locked down by a minimal whitelist leak-preventing fw ruleset like little snitch. I have a script which checks for authentic internet access to allow captive portals to work which leaks temporarily (which I prob need to toggle between rulesets to only allow the captive portal agent to work and deny everything else).

skrause• 12 years ago

Can everyone else reproduce this problem? People from different locations and ISPs should try it.

I'm not a Level 3 customer in a any way and I'm on a German VDSL connection provided by Deutsche Telekom. And here the Level 3 resolvers still return normal NXDOMAIN answers:

  ; <<>> DiG 9.8.3-P1 <<>> thisprobablydoesntexist.com @4.2.2.2
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 44948
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

nwh• 12 years ago

I can not replicate from Australia.

    ; <<>> DiG 9.8.3-P1 <<>> nxdomain.horse @4.2.2.
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58067
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

bjackman• 12 years ago

Not sure if I'm misunderstanding this but my ISP (Virgin Media - not recommended, by the way) has always done this.. I assumed all ISPs did that?

Angostura• 12 years ago

If you're in the UK and using Virgin you can opt out. Link on the error page, ISTR

dsl6• 12 years ago

Lots of well meaning kids go and setup Level 3 resolvers on grandma's home network, then the ISP has to deal with support calls when someone elses DNS servers go down.

As a result quite a number of networks "hijack" 4.2.2.0/24 and route it locally to their own resolvers.

ds93• 12 years ago

How can one detect whether this is happening?

pyre2• 12 years ago

On the flipside, when the local ISP's DNS goes down, many a time will the support person have no knowledge of it, and run you thought the "do you own a computer? did you turn it off and on again?" spiel.

You act as if most ISP support actually admits to problems on their own network and/or that troubleshooters have access to this information.

chmars• 12 years ago

I have used the DNS servers of the Swiss Privacy Foundation for some time. The IP addresses are not easy to remember but it is great to have uncensored DNS from a Swiss non-profit organization:

77.109.138.45 (Ports: 53, 110; DNSSEC), 77.109.139.29 (Ports: 53, 110; DNSSEC) and 87.118.85.241 (Ports: 53, 110; DNSSEC).

https://www.privacyfoundation.ch/de/service/server.html

(The Swiss Privacy Foundation operates Tor exit nodes too.)

escapologybb• 12 years ago

I have a newbie question, what would an end user do with the (Ports: 53, 110; DNSSEC) information?

I've set my machine to use those three IP addresses as the DNS servers, is there something else I'm missing? Thanks!

ds93• 12 years ago

Normally DNS is on port 53, but if your ISP is preventing you from DNS requests to servers other than theirs on port 53 you can use the other one.

'DNSSEC' means DNSSEC is supported by the server if your resolver can use it - it's a digital signature regime to prevent DNS forgery (disclaimer: look up criticisms of it as well as selling points).

ck2• 12 years ago

Can't really blame them since they have been telling the public not to use their service forever.

Still, I am guilty of using them too.

Not really thrilled with the idea of using Google DNS.

y0ghur7_xxx• 12 years ago

Not really thrilled with the idea of using Google DNS.

You can use your own dns server. Just install a recursive dns server on your own network like https://www.powerdns.com/recursor.html or https://unbound.net/

dmourati• 12 years ago

unbounds https cert is broken. The identity of this website has not been verified. • Server's certificate is not trusted. • Server's certificate cannot be checked.

Not a good first sign.

mp3geek• 12 years ago

The tracker should now be blocked also in Easyprivacy.

https://hg.adblockplus.org/easylist/rev/303e65d3a2bd

nly2• 12 years ago

I can't reproduce this behavior from the UK. In any case, DNSmasq (the DNS cache daemon that is part of OpenWRT) has an option to filter bogus NXDOMAIN responses if you can get a list of the IPs.

Alternatively, run your own recursive resolver and cache, it's worth it.

userbinator• 12 years ago

I noticed this a few weeks ago (not sure exactly, but I don't really mistype domain names all that often either) and I can remember the first time that 198.x IP showed up I was rather shocked since the domain I mistyped was my own site's (!), but they seem to have stopped doing it now.

Perhaps L3 themselves have a lot of stuff both within and outside their subnet that depends on these servers behaving correctly.

d0ugie• 12 years ago

While in search of a new DNS server because of this and for lower latency, namebench is your friend: https://code.google.com/p/namebench/

sdkmvx2• 12 years ago

At the least it’s leaking, in clear text on the wire, things that I expected to be sent to an encypted DDG search. If there was sensitive search terms or information in that query, it just dropped into Level3′s logfiles.

He must not realize that even if the DNS server was working correctly, the original request that should result in NXDOMAIN is also passed in clear text over the wire and naturally potentially logged by the DNS server. The lesson is not to rely on DNS security. Your ISP can see what servers (IPs) you communicate with anyway.

Level 3 are now hijacking failed DNS requests for ad revenue on 4.2.2.x

Related Stories