NTP Pool Bad Actors: The Rising Sophistication of Network Scanning

http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

by tshtf • 10 years ago

208 76 10 years ago

netpatterns.blogspot.com

jlgaddis2• 10 years ago

There is (starting to be) discussion on a few mailing lists as well, including pool@lists.ntp.org [0].

[0] http://lists.ntp.org/pipermail/pool/2016-January/007758.html

adekok4• 10 years ago

We've gone from anonymous people scanning you, to semi-trusted people scanning you.

The mantra of the new Internet is "Trust no one". :(

dspillett6• 10 years ago

I don't see the people running hosts in pool.ntp.org's pool as any less anonymous than the general internet at large.

adekok4• 10 years ago

You trust them to give you a correct time. That's at least a bit more trust than you give any other random IP address.

fweespeech• 10 years ago

NTP Pools are effectively anonymous.

jms7033• 10 years ago

No one objects to IRC servers scanning when you connect. This isn't a trust issue. If you sent packets to a free service, you have no rigts over whatever trace you left behind.

Tepix3• 10 years ago

IRC servers only scan you to see if you run an open proxy. They have a good reason to scan you. There is no reason why a NTP server should scan you.

devicenull4• 10 years ago

I'm not really seeing a vulnerability here. The entire IPv4 internet is scanned probably hundreds of times a day. Was anyone really counting on IPv6 addresses being longer to add security?

http://www.internetsociety.org/deploy360/blog/2015/02/ipv6-s...

qjighap2• 10 years ago

Could be wrong, but part getting through a firewall is establishing open ports and mapping the machines behind the firewall and since you have a verified machine name this would save bad actors a significant amount of time being that they can make the assumption that the computer exists rather than timeouts to nothing. While this goes in on the hiding an IP (disable ping) is not proper security argument it is valid that you would want to stop disclosure on such a public level.

devicenull4• 10 years ago

If you have a machine behind a firewall, why do you care that it's IP is secret? Do you also worry about people finding out your IPv4 IP?

dspillett6• 10 years ago

A machine with a random address that makes no (direct) contact with the outside world will be a lot harder to find in a 64-bit address space then a 32. As soon as it talks to the outside world though (by talking to public NTP servers in this case) that difference is rendered moot.

clinta• 10 years ago

Unless it's using Privacy Extensions in which case it will change it's outbound IP in a matter of hours and once again be lost in the huge address space.

Unfortunately privacy extensions are not enabled by default on most linux distributions and server operating systems.

snowy• 10 years ago

At least most IPv4 addresses on a LAN are behind NAT. It's not a firewall but probably has saved some people.

NelsonMinar• 10 years ago

I've wondered that too. IP addresses have never been secrets.

fapjacks• 10 years ago

You're absolutely right, I agree. I think this reaction ("Oh no! My 'private' IPs are known and being scanned!") is probably one that comes from having spent a lot of time with a personal network behind the NAT of an IPv4 router, which necessarily provides a wall to internal network scans. Using IPv6, NAT is no longer necessary, and therefore it can surprise people that aren't expecting that behavior. I thoroughly enjoyed the write-up though!

dspillett6• 10 years ago

If someone is detecting which IPv6 addresses are in use in your range from NTP requests, that implies they are all talking to the external NTP servers directly.

Surely if all your internal hosts are talking directly to the external NTP servers you are doing it wrong? My gateway box sets itself by pool.ntp.org and the internal ones set themselves by it. I thought that was how you should do things (even if it isn't a rule, it is only polite to try not overuse a public resource).

These addresses are 128 bits in length

Only 64 are relevant here: once you make outgoing connections from any address a scanner knows there is at least one active host in that /64 and there may be more. Though of course a 64 bit address space is still impractical to scan.

toast03• 10 years ago

Surely if all your internal hosts are talking directly to the external NTP servers you are doing it wrong? My gateway box sets itself by pool.ntp.org and the internal ones set themselves by it. I thought that was how you should do things (even if it isn't a rule, it is only polite to try not overuse a public resource).

Depends on how many internal hosts you have, and ease of configuring them (there is a DHCP option for ntp server, but not everything will use it), and available hardware to setup ntp servers: ideally you would have each client syncing from three servers so the clients drop a broken server.

dspillett6• 10 years ago

ideally you would have each client syncing from three servers

For a small network (a home or small office network for instance) you likely have one incoming router anyway, so if that dies clock setting is the least of your worries until it comes back up, meaning setting the internal hosts by that one clock is as fine as anything else.

For a larger network you have redundant everything, including edge servers that can act as your multiple NTP sources for internal hosts rather than every internal host poking the public pool.

dfc• 10 years ago

You are surely doing something wrong with ntp if you only have one time source configured. The only configuration that I can think of that would be worse is to have a setup with two time sources.

imglorp2• 10 years ago

https://www.ntpsec.org/

brandmeyer• 10 years ago

Improving the security of the Internet's go-to time transfer protocol is certainly useful. However, switching to a different protocol would not mitigate this port scanning attack. According to this researcher, just sending an empty packet to an NTP server's port is sufficient to result in a port scan back.

imglorp2• 10 years ago

... Which is the sort of bugs ntpsec effort is fixing.

toast03• 10 years ago

That doesn't really help, if you make an NTP request to a remote server, the server and all elements of the network path will know the source IP you used. You can't avoid that without also avoiding getting the response from the request.

Using a broadcast time source (CDMA/GSM, GPS/Galileo/GLONASS, WWV/WWVH/international equivilents?, ATSC broadcasts, etc), or choosing a trusted time server would be alternatives; you could also use a telephone time service and leak your phone number instead of your ip address.

tptacek• 10 years ago

I am not optimistic about this project, which seems to involve people who don't have much of a track record in secure programming trying to refactor a giant blob of C code.

ChuckMcM2• 10 years ago

Very interesting post, I've suggested to Brad that he consider probes with a decreasing ttl in the packet to see if the harvesting is happening at an interstitial node or directly on the ntp server. If you had access to a compromised Juniper router it would be straight forward to add a rule to mirror packets which were headed to the NTP pool addresses.

Oddly after being an unwitting participant in one of the NTP amplification attacks I set up my own stratum 0 NTP server based on the beaglebone black, the adafruit GPS module, and the PPS time keeper code. So all of my machines only talk internally for time, although initial installs still use what ever the distro packed into them. I brought up Debian on a Dragonboard 410c and it sets the time via an NTP call during the initial startup process. (or fails to set the time as NTP is blocked egress/ingress from the firewall)

rudolf0• 10 years ago

It seems pretty clear it's the NTP servers themselves. All the servers are in the same small subnet, and 2 of them even have rDNS hostnames indicating they're owned by shodan.io.

arca_vorago• 10 years ago

I have been interested in using the dragonboard with linux as a small controller for some embedded things, how is that working for you?

On subject, I have always though having a master internal ntp under strict controls should be used, so good on you for that, I dont see ntp payed attention to far too often in the enterprise.

ChuckMcM2• 10 years ago

Support for the DragonBoard is much more limited than other platforms. Android seems to work best although with a stupid "giant phone" sort of design language which is silly. If they had the level of documentation that the BeagleBone (or even the RasPi has) they might be a contender but my estimate is that the platform won't go anywhere.

ansible• 10 years ago

I've recently acquired a Nvidia Jetson TK1 development board. It is $200 USD, and basically a Linux desktop system based on an ARM processor platform, running Ubuntu 14.04. You can easily hook up mass storage and other stuff. Just 2GB of RAM though.

Rebelgecko• 10 years ago

I have a Dragonboard that I haven't really had a chance to do anything with yet. However I've noticed two main drawbacks:

1. You have to use some awful hacks if you want to ssh into it (IIRC it ignores ARP requests for some reason) 2. The GPS only works on Android

That said it's still a nice upgrade hardware-wise from something like a Raspberry Pi. The software support is just spotty in a few areas

scurvy• 10 years ago

FWIW, I've had really good luck with running Endrun CDMA stratum 1 timeservers. You can pick them up off of ebay for a few hundred. In theory it's more stable than GPS, especially if you get one with OCXO or rubidium option.

fl0wenol• 10 years ago

While it is pretty freaky, I think the point of Shodan scanning IPV6 devices that use the default pool is to detect linux-based IoTs that use the default NTP settings, which is kind of the reason why Shodan exists.

It is unlikely that Shodan is going to hack your Raspberry Pi specifically because then people will go after them. But hackers of many hat colors will use the freely available information it gathers for their own purposes, so act with this in mind.

CrankyBear• 10 years ago

This is interesting, but really, why are you not running a firewall on incoming requests to the NTP port? No fuss, no muss, no scan.

throwaway20482• 10 years ago

emumeration of ipv6 hosts is likely to be the goal here, not a scan of NTP ports.

The normal ipv4 methods of "scan the entire address space" fail in an ipv6 world, so shodan needs some way of mass havresting ips for them to gather scan results for...

dspillett6• 10 years ago

This isn't talking about incoming request for the NTP service - it is fuller scans to valid IPv6 addresses that have been harvested because hosts using those addresses talked to an NTP service that logged them.

Johnny5553• 10 years ago

Isn't this article just saying that "security through obscurity" doesn't work, and you can't count an obscure IPv6 address to keep you hidden?

If your server isn't meant to be contacted by the world, then put it behind a firewall. Don't count on an obscure address to keep you hidden.

devicenull4• 10 years ago

Yep, that's exactly what it's saying.

__jal• 10 years ago

For folks only looking for a cautionary tale, who don't worry about larger security trends and only want to keep their own machine safe, that's a fine takeaway.

For people who do care about network security, it is a very interesting post for a few reasons.

If someone is harvesting IPs from NTP queries sent to Debian infrastructure for intelligence gathering, that in itself is a big deal.

It is also a somewhat novel technique - if an attacker is placed such that they can see the first NTP queries sent by a new install, they are well placed to target that device before it is fully hardened/patched, because they likely see some of the first packets sent by that install. It is really rather clever.

It also points to the sort of correlation we need to be getting better at - orchestration isn't just for devops weenies anymore, and this sort of thing is only going to become more sophisticated.

I've been noodling about with making a tool for making this sort of analysis easier, but there are a number of problems to fix, not least of which is the sheer volume of data generated watching the wire, even on my home network (which is a bit absurd for your average home, but tiny compared to a business of any size).

devicenull4• 10 years ago

If someone is harvesting IPs from NTP queries sent to Debian infrastructure for intelligence gathering, that in itself is a big deal.

Debian doesn't run the NTP pool.

jms7033• 10 years ago

Web servers "harvest" IPs. Why can't NTP?

noselasd2• 10 years ago

The more interesting thing it's also saying is that some of the *.pool.ntp.org servers are bad apples, and results in your NTP request, or at least its source IP address being collected and results in a scan back at you.

Johnny5553• 10 years ago

Is there any expectation that a public address (even if it's buried in a huge IPv6 address space) is in any way secret? Once a packet leaves your network, you have to assume that the world knows your IP address. If you want to keep it secret, don't let it leave your network.

w8rbt2• 10 years ago

Unless I missed something in the article, I'm not sure this is odd or malicious behavior. The NTP servers know your IPs because you sent packets to them from those IPs. So you basically told them what your big random 128-bit IPv6 address is.

And the fact that they sent packets back to you (after you sent them packets) is not surprising either. However, if you can show that a full-blown port scan occurred after you sent them packets, then that would be odd. I did not see evidence of that in the article... did I miss that?

linuxguy2• 10 years ago

Yes, I think you missed that. From the article:

"It takes less than five seconds for your address to be harvested and scanned. The entire scan takes less than one second and scans over 100 common TCP and UDP ports. "

w8rbt2• 10 years ago

OK. I'd like to see a full packet capture. And technically, the address was not 'harvested'. He gave it to them when he sent them packets. The story strikes me as very dramatic and over the top because of statements such as that.

tyingq4• 10 years ago

It seems like many aren't quite getting what's going on here...here's a brief overview:

Shodan.io fancies itself as a search engine where you can search for IOT things (webcams, refrigerators, etc) that are on the internet.

They have no technical issues doing this in the IPV4 space, because it's easy enough to scan every single address in the space.

This isn't practical for IPV6. So, it seems they wanted a way to identify every IOT device in the IPV6 space without having to scan all of it.

At least one approach they found was to join the ntp.org pool[1], and effectively donate server time. Since pool.ntp.org is the default NTP server listed for many linux distros (and thus, probably IOT devices), they now are getting live connections from the exact devices they want to index on their search engine.

Once you connect, they scan you back on 100 ports on so (ports unrelated to NTP) to see if you are a webcam, router, or whatever else they want to put in their index.

Pretty shady. Kind of like volunteering for a charity so that you can raid their internal mailing list for spam purposes.

[1]http://www.pool.ntp.org/join.html

Cymen• 10 years ago

So next up ntp clients that support blacklists?

the84722• 10 years ago

just spin up a separate network namespace and macvlan device to run your ntpd in. it'll get a separate v6 address via router advertisement and ntpd will be the only thing listening on that address.

just wrap ntpd in firejail[1].

ipv6-per-networked-process pretty much brings an end to portscanning.

[1] https://firejail.wordpress.com/

mey• 10 years ago

Could switch over to NIST at time.nist.gov See http://tf.nist.gov/tf-cgi/servers.cgi for info/details

Granted that means trusting NIST over someone else.

The other options are getting time from a cellular modems, GPS or the most extreme running your own atomic clock.

nvarsj• 10 years ago

Pretty shady.

Sort of, but in the global order of shadiness, this ranks pretty far below what your typical multinational corporation does to track an individual. It's pretty ingenious actually.

jms7033• 10 years ago

What a novel idea. There is nothing wrong with this. Why be afraid of scans?

tyingq4• 10 years ago

They are taking the data from the scans and building a nice easy to use searchable signature database for script kiddie hackers. At the very least, it's impolite.

Tepix3• 10 years ago

If they know what software you are running, the minute there is a new vulnerability revealed, they can break into your system. It gives you less time to patch your stuff. You think this is the stuff of fantasy? Read https://www.drupal.org/PSA-2014-003

morgante• 10 years ago

I don't really see what's shady about it. If you connect to a server, it's accepted and expected that they'll get your IP address.

tyingq4• 10 years ago

They run a port scan against your ip and publish the discovered info publicly. You may find that okay, others don't.

Tepix3• 10 years ago

The real-life equivalent is a pizza delivery guy who, after delivering the pizza, goes around your house, checks which doors and windows are locked, unlocked or open, what brand the locks and the burglar alarm are and then proceeds to sell this data.

I hope the NTP pool terms of service will disallow such behaviour in the future, otherwise I'm removing my server from the pool.

Lazare2• 10 years ago

The lesson here, I think, is that security intuitions developed in an IPv4 world with NAT everywhere may need re-examining in an IPv6 world.

If your device has a publicly routable IP address, it should probably be behind a firewall. If it's not behind a firewall, it's going to get scanned. Relying on security through address space size is stupid.

What was most puzzling was the fact that the devices that were targeted had randomized IPv6 addresses and were not published in DNS or any public record. For all intents and purposes they were hidden safely within my lab network.

That is just wrong, and it's wrong in a very glaring, obvious way: The devices were not "hidden safely within [his] lab network", because he was not using NAT. They were connected to the internet with publicly routable IP addresses, which they were using to communicate with other hosts on the internet.

Edit: That being said, it's a great writeup, and some good technical work was done to figure out what was going on, and I enjoyed reading it. But the starting premise seemed to be "how could this be happening, my devices are hidden!" and I feel like it should have been "oh, it makes sense that someone would do this since my devices are no longer hidden".

mentat• 10 years ago

Even if they were NATed I'd assert they weren't safely hidden but that's a more obscure discussion.

Lazare2• 10 years ago

You're quite right; people significantly overestimate the safety of NAT in my experience.

But what I was getting at is that however illusory the safety of NAT is, the author seemed to be instinctively assuming that his IPv6 devices had it, because we've spent 20 years training ourselves to assume that everything is behind NAT. It's a new world.

ams6110• 10 years ago

More like an unlisted phone number. The addresses weren't published in any deliberate way, but they were still visible on the "caller ID" when they connected to another system.

yAnonymous• 10 years ago

So I finally found an NTP server where I can set the update interval to a few seconds.

vetrom• 10 years ago

Nice evidence for why site operators should be running their own NTP servers. If you run 1 server set per site which doesn't even need a strong local clock, you mitigate this information leakage.

I'd think this is something that most network designers/engineers would get. That said, I'd be a rich man if I had a nickel for every time I saw NTP misconfigured.

NTP Pool Bad Actors: The Rising Sophistication of Network Scanning

Related Stories