Ntpd won't save you from one particular rogue bit

http://rachelbythebay.com/w/2017/09/27/2153/

by dantiberian • 9 years ago

233 79 9 years ago

rachelbythebay.com

simcop2387• 9 years ago

I wonder if any of the other ntp implementations (chrony et al) suffer from this same issue?

jlgaddis4• 9 years ago

Yeah, it's an issue with NTP the "protocol" not NTP the "program".

grogers• 9 years ago

To expand on this, NTP uses timestamps with 32 bit seconds (plus fractions of a second), so if you manually step your clock some multiple of ~136 years, the protocol inputs will be the exact same so you'd have no way of knowing you were off from the server.

However, you could imagine an NTP implementation which hardcodes the approximate starting time to get the right era. You'd only have to recompile every lifetime or so to keep it up to date.

jlgaddis4• 9 years ago

It won't save you from this particular case but you're safe so long as your clock is off by no more than ~68 years.

cgsmith• 9 years ago

The other use case is manufactured hardware with their time set for the unix epoch and depend on it for updating.

whouweling• 9 years ago

Interesting edge case! This is why it may be sensible to do some extra checks in things like product expiration batch jobs, to check if the previous run was not to far in the future or past and refuse to run in that case.

Scary to think what might happen if some database purge process is running after this bit flip!

viraptor• 9 years ago

This is going to give me actual nightmares. Not only database cleanups, but there are quite a few automated backup cycling scripts which would happily throw out terabytes of data this way...

nolok4• 9 years ago

As parent said, it's usually a very good idea to check for time delay and ensure it "makes sense" before doing anything destructive. Doesn't cost much, can save a lot, and adding a command switch to bypass it allows to skip the "haven't booted in a couple week and now everything refuses to run" problem.

A much more common case of when such a check help is "last time run is in the future", that you face every time there is a clock issue. Some scripts... Don't react too well about that.

poizan424• 9 years ago

If the resulting time that ntpd sees is more than a few milliseconds off, it'll step the clock, and that will clear out the future time.

It shouldn't do that, otherwise it's going to break in 2036.

(The "future time" it's going to clear off will set the time back to 1900 once we are past February 7, 2036)

Edit: I don't know if I didn't explain myself well enough. Because the protocol only has 32 bits of seconds it cannot tell apart September 27, 2017 and November 4, 2153. This means that ntpd absolutely must trust that it is in the correct 68-year span. But according to the author ntpd "clears out the future time" if it is more than a few milliseconds off. This violates the spec and is also inconsistent as it happily keeps up with the future date as long as it doesn't have to step the clock.

jlgaddis4• 9 years ago

It won't, this is accounted for in the spec; cf. "pivot dates" (IIRC).

poizan424• 9 years ago

I know that it shouldn't, but the author said that it did in the blog post.

nolok4• 9 years ago

I often see this type of comment on many subject and I find it super weird in a way. I mean, if it took you less than a minute after reading it to spot the issue despite not being familiar with the details before, it's usually safe to assume whoever made the specs and spent days on it also did.

poizan424• 9 years ago

Where did I disagree with the spec?

petecooper5• 9 years ago

I get numerous requests to my (macOS) computer for ntpd to connect to shady subnets when I'm connected to a particular commercial VPN:

https://twitter.com/petecooper/status/911946604759977984

https://pbs.twimg.com/media/DKficrvW4AA1Hxm.jpg:large

Numerous hosts across numerous networks, perhaps two or three an hour.

I've wondered what exactly would be gained by resetting a clock to a different time – this is a useful article.

gog2• 9 years ago

Which VPN provider are you talking about?

Are you sure that it is not just NTP servers being served via DHCP once the connection is established and your computer trying to use those provided by your VPN?

petecooper5• 9 years ago

It's PureVPN.

I'm not 100% sure about any of the ntpd connection requests, they're not predictable in their appearance. Some sessions are very quiet (zero requests), others I get a bunch of incoming connection requests for smbd, and other odd things. I really should start taking notes rather than just deny the connections.

geofft2• 9 years ago

That sounds like

1. your VPN provider is giving you an actual public IP address (??)

2. people are scanning your computer for NTP vulnerabilities or something (this happens if you have a public IP, regardless of network)

3. NTP is using UDP and so connectionless, and so Little Snitch can't distinguish "ntpd wants to reply to someone who contacted it" from "ntpd wants to connect to someone"

An alternative explanation for 1/2 is that your VPN provider is not isolating you from other VPN users (less surprising than giving you your own public IP) and someone else on the VPN is trying to conduct NTP amplification attacks using you: https://blog.cloudflare.com/understanding-and-mitigating-ntp...

In either case, the solution is basically to make your ntpd not listen for requests from other machines and only handle time from your local computer + initiating requests to time.apple.com or whatever your chosen NTP server is. It shouldn't be trying to reply at all to unexpected packets, even to send a refusal message (again, because UDP is connectionless, it's easy for an attacker on your LAN to send spoofed packets and convince you to send replies to some random computer on the internet, and I guess on this VPN, other customers are your LAN). I'm surprised that macOS's default NTP server isn't configured this way out-of-the-box, though.

petecooper5• 9 years ago

An alternative explanation for 1/2 is that your VPN provider is not isolating you from other VPN users (less surprising than giving you your own public IP) and someone else on the VPN is trying to conduct NTP amplification attacks using you:

This, I think, is most likely.

jandrese3• 9 years ago

It seems strange that the firewall would block the outbound packet after letting the udp packet from some completely random host in.

userbinator• 9 years ago

This means you can't actually persist the bad time to your RTC.

I wonder where that limit comes from, since the RTC in a standard PC stores both the year(00-99) and century(also 00-99) in BCD, so a date in 2153 should be representable.

This reminds me, one interesting thing I've noticed over the years is that early PC's RTCs were pretty accurate, but the most recent ones, as in the past few years, are horrible (as much as +/- several seconds per day), and the ones in smartphones even worse. Maybe because they're assuming NTP, so have cut costs by using less accurate crystals? I've had systems in storage for many years, and the RTC was within a few seconds of the current time when turned on again.

lucaspiller• 9 years ago

I have a heating thermostat that has optional ‘smart’ features that allow you to control it via your phone. I tried it when I first got it, but it seemed a bit pointless, so I turned off the feature and the wifi connection. That was a month ago, and the onboard clock has now drifted by around 15 minutes...

dspillett• 9 years ago

I can't say I've ever had a PC with a particularly accurate RTC, though in recent years I've run NTP on everything I can so not noticed any effects.

Anecdotally clocks in small mobile devices like phones do seem less accurate though, at least when they don't have access to the network to update themselves. I've not looked into it but assumed temperature variances, which a device like a phone will experience to a larger extent, were part of the problem.

Of course phones syncing with the network relies on their clocks agreeing with the rest of the world - for some months my provider seemed to be a minute or so behind which meant I had to adjust a little when gauging if I was just going to make train I was rushing towards or if I'd likely missed it leaving already...

jofer• 9 years ago

I've noticed the same thing, but I'm always surprised by it. Particularly with smartphones.

A GPS is basically just a very, very precise clock and a radio receiver.

Why don't smartphones use the GPS clock?

I'm sure there's a good reason (and some probably do), but it sill surprises me. (Presumably it's battery related -- the GPS clock can't be used unless GPS is fully enabled or something similar.)

Lx1oG-AWb6h_ZG0• 9 years ago

The precise clock is located on the satellites, not receivers. The receivers figure out the correct time by listening for the signals from multiple satellites and matching them against each other and it’s own internal tables. This is the expensive part.

wruza2• 9 years ago

and the ones in smartphones even worse

Oh, I bet my tooth that I can hear my phone playing music slightly faster than usual sometimes. Never experienced that on PC. Maybe it is not connected to RTC and is purely biological condition (never cared enough to test two devices side by side), but if someone experienced that too and has an explanation, it would be great to know.

etatoby• 9 years ago

Yes, I've definitely heard it.

Which does not prove it's a fault in the device—it could be a fault in my fried brain—but anecdotally, I've never noticed it on any other music playing device, since the time of the original walkman (and in that case the music would usually slow down due to attrition or other mechanical faults, but rarely if ever speed up.)

wyldfire3• 9 years ago

The RTC is not a factor for problems in music playback on phones or elsewhere. It's only for "wall clock" time, and generally only for when the system boots. The OS timer tick that drives the scheduler is often generated by the CPU and some operating systems don't use it anymore -- Linux has offered a tickless config for a while that I think many android configs capitalize on for the sake of conserving battery. The scheduler could be a culprit for playback discontinuity (stalls) but not problems with the tempo.

I'm wondering -- "What might cause music playback to be faster than normal?" I don't think I know enough to tell. Perhaps the clocks on the audio playback device are skewed? Most SoCs use a dedicated DSP which doesn't seem terribly different from PC's. Not sure, but we can say for certain that it's definitely immune to changes in wall clock time (whether in the RTC or its representation in the OS).

cellularmitosis• 9 years ago

If the pitch sounds higher, then your phone is actually playing back the song faster. If it sounds faster without the pitch being higher, then its just a perception-of-reality thing.

LinuxBender• 9 years ago

Try disabling the power management features for a bit and see if your drift is the same. On NTP servers, I always have to disable cpuspeed, c-states, intel power mgmt, etc.

jandrese3• 9 years ago

This would also explain why phone hardware is worse off than PC hardware, it is much more aggressive about managing power.

JdeBP• 9 years ago

A "standard PC" is not in fact required to have a century register. One can obtain from ACPI information where it is if the hardware has one but this is an optional feature and PCs are not required to have a valid register number in their where-the-century-register-is fields. Indeed, the entire MC146818 compatibility system is optional as far as modern ACPI is concerned. There's an FADT flag that signals that it is not there.

monochromatic4• 9 years ago

cut costs by using less accurate crystals

Maybe... but how much money could they possibly be saving? Quartz oscillators like the ones used in cheap digital watches are plenty accurate and completely abundant. I can't imagine they cost very much.

Maybe the issue has to do with miniaturization? If you're trying to make smaller and smaller devices, there could be an acceptable tradeoff between size and precision.

agoetz• 9 years ago

Most RTCs do use the same crystals that are in wristwatches, which is actually the source of their error.

The tuning fork crystal design used in wrist watches has a parabolic temperature coefficient, which means that the clock is only really accurate at room temperature. This isn't a problem for wristwatches, because, presumably your wrist is at approximately room temperature, but it does become a problem for electronics that operate with large temperature swings. (like the inside of a phone or computer, for instance).

https://www.maximintegrated.com/en/app-notes/index.mvp/id/58

dragonwriter• 9 years ago

This reminds me, one interesting thing I've noticed over the years is that early PC's RTCs were pretty accurate, but the most recent ones, as in the past few years, are horrible

Insofar as sensitivity to thermal conditions is an issue, early PCs tended to be big, airy, ventilated boxes and have less bits that would ramp up to high temperature under load but drop down in other operating regimes. So that may be a factor.

tonyg• 9 years ago

We will continue to suffer problems like this so long as we continue to use languages which offer machine words in place of general integers.

wruza2• 9 years ago

Time is hard.

I learned that once tried to implement "universal" datetime library for fun and education. There is wall clock, which leaps for political decisions; universal time coordinated which leaps on schedule to ease astronomical differences; atomic time of few sorts; numerous computer clocks; time zones which can go back and forth for north and south, land and sea; non-gregorian calendars and the fact that there is no 0 AD and no 0 BC; that dates were offset for many days few times in gregorian history; special and general relativity errors; and of course integer overflow issues.

I may have missed few points, but still, time is hard.

moviuro• 9 years ago

See this[0] article which does a good tour of numerous issues with time.

[0]http://FalsehoodsAboutTime.com

Dylan168074• 9 years ago

universal time coordinated which leaps on schedule to ease astronomical differences

"Schedule" is a bit generous there.

Oh and there's TAI for just counting seconds but you're discouraged from using it that way because of some slightly-suspicous reasoning about retrospective calibration.

deepsun• 9 years ago

That's interesting, could you elaborate, or give a hyperlink on why TAI is discouraged?

mnarayan01• 9 years ago

Unless you're running it with additional configuration (e.g. the -g option), I don't think ntpd will save you from any bit flips other than the 10 least significant: You'll be outside the 1000s panic threshold.

Fnoord• 9 years ago

Its not recommended to use ntpd with -g argument in production. An attacker can MITM NTP protocol. The 1000s threshold severely limits this attack. The attack can be used e.g. in defeating TOTP.

I'm not sure if this rogue bit can be used to attack TOTP. Can anyone clarify?

gerdesj2• 9 years ago

Well that was fun! I tried out the timeshift program and my PC duly went a bit mad. ntpq -p showed all being well.

Chromium threw a fit because news.ycombinator.com's SSL cert had expired and offered to reset the clock which it could not do, given that I'm not in the habit of running apps as root. My Kerberos tickets all expired so Evolution lost contact and so did quite a few other things. MariaDB dumped core. systemd's timers went berserk so things like my LetsEncrypt cert tried renew itself.

I'm still hunting through some odd looking log files but overall things seem to be back to normal when I reran timeshift and returned to current time.

winkywooster• 9 years ago

Similarly with my Mac, within moments it became unusable. Even trying to reboot from the command line and I got a message that it was waiting to acquire a lock on /. Hard reset and everything is back to normal.

netsharc• 9 years ago

iOS devices had a bug that hard locks it if you set the time to 1/1/1970 (on US timezones, that sets it to negative territory), I can't believe they don't do any tests for their OSes to survive this...

ifoundthetao2• 9 years ago

Hm, I wasn't able to replicate it. I'm using a Kali Linux VM. I'll try some other OSes and see if it works on there.

Was anyone else successful with the PoC code?

sp332• 9 years ago

Is your VM host resetting the time in the guest?

ifoundthetao2• 9 years ago

That's possible. I'll try it on a physical system in a bit. See what I can do.

gerdesj2• 9 years ago

It works flawlessly on a physical system. There was rather a lot of red text in journalctl -r 8)

wyldfire3• 9 years ago

One source of RTC weirdness is the fact that inb/outb to/from the RTC and other legacy parts of the ISA aren't protected by mutex on linux. So if you're unlucky enough to collide with some other program doing something innocuous with the RTC you can accidentally set bits that will never be set by the RTC. Many RTCs decide not to tick anymore when that happens.

If all your programs use /dev/rtc and ioctl()s you are probably a little safer because there will be coarse locks around the RTC itself, those will serialize the activity. But IIRC the inb/outb stuff can be done from user space (as superuser) and even if you're only reading you have to write to the address register which could break a write-in-progress by sending its output to the wrong RTC field.

exikyut2• 9 years ago

Many RTCs decide not to tick anymore when that happens.

Of course I laughed when I read that, but then I realized I was interpreting that line to mean "it declares shenanigans and stops reporting time so you realize something broke."

Just wanted to clarify - do you mean the above, or "of course software cannot kill hardware!!1" "won't tick anymore"?

wyldfire3• 9 years ago

Yeah, IIRC it really stopped ticking (until you write a valid value at which point it would resume).

Note that it doesn't stop reporting the time in this condition it will just give you back all the garbage fields that you wrote before. So an interesting thing happens when the system tries to transform that into a UTC wall clock basis and it usually ends up with a really wild interpretation of the date (decades/centuries off, similar to the problem described in TFA).

dwyerm• 9 years ago

I've been chasing exactly this issue for weeks!! The problem I'm seeing is that something changes the binary/BCD flag. It turns out that changing this flag doesn't convert the current representation of time inside the RTC; it just changes the logic used to advance the counter fields and apply the next tick.

So if you change that flag without also setting the time, then many of the fields are now invalid. But the tick logic doesn't care. It just sets the invalid fields back to zero and keeps on ticking.

The year 20 17 becomes 1 4 1 1.

The RTC clock is a 1984 part that is still getting embedded, more or less unchanged, into today's PCs. It is maddening.

gumby• 9 years ago

This is a nice bug! And not really a clock issue -- many programs/protocols could have such a bug.

Ntpd won't save you from one particular rogue bit

Related Stories