HN user

mlichvar

107 karma
Posts1
Comments39
View on HN

That's what the chrony tempcomp directive is for. But you would have to figure out the coefficients, it's not automatic.

An advantage of constantly loading at least one core of the CPU might be preventing the deeper power states from kicking in, which should make the RX timestamping latency more stable and improve stability of synchronization of NTP clients.

The major Linux distributions replaced ntp with ntpsec. A better question would be who is still running ntp. I know about FreeBSD and NetBSD.

ntpsec as a project seems to be doing ok. They are releasing new versions, fix reported issues, accept patches, and develop the code publicly. While ntp still has a huge list of acknowledged but unfixed CVEs.

I agree with most of what you said.

The author has other posts in the series where he tried to measure the accuracy relative to the PHC (not system clock) using PPS: https://scottstuff.net/posts/2025/06/02/measuring-ntp-accura...

Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets.

So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this

That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).

chrony can be configured to encapsulate NTP messages in PTP messages (NTP over PTP) in order to get the delay corrections from switches working as one-step PTP transparent clocks. The current NTPv5 draft specifies an NTP-specific correction field, which switches could support in future if there was a demand for it.

The switches could also implement a proper HW-timestamping NTP server and client to provide an equivalent to a PTP boundary clock.

PTP was based on a broadcast/multicast model to reduce the message rate in order to simplify and reduce the cost of HW support. But that is no longer a concern with modern HW that can timestamp packets at very high rates, so the simpler unicast protocols like NTP and client-server PTP (CSPTP) currently developed by IEEE might be preferable to classic PTP for better security and other advantages.

Y292B Bug 2 years ago

The Linux kernel keeps time as a 64-bit integer, but it's in nanoseconds, not seconds. It's a Y2262 problem.

My current understanding of the wireless technologies wrt to home automation:

- wifi is most reliable, secure, easiest to debug, but usable only for mains-powered devices due to higher energy consumption

- bluetooth LE has lowest energy consumption, best for unreliable broadcasting of data (e.g. temperature sensors), but has shorter range

- zigbee is best for battery-powered devices where reliable communication is needed and is initiated by the device (e.g. switch, window/door sensor)

- zwave is best for battery-powered devices which need to quickly receive data (e.g. door lock or directly controller radiator valves), but security seems problematic according to some reports

To me that looks like they are reinventing NTP, but not addressing all the issues of PTP.

A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.

It seems they didn't replace the PTP offset calculation assuming a constant delay (broadcast model). That doesn't work well when the distribution of the delay is not symmetric, e.g. errors in hardware timestamping on the NIC are sensitive to network load. They would need to measure the actual error of the clock to see that (the graphs in the article seem to show only the offset measured by SPTP itself, a common issue when improvements in time synchronization are demonstrated).

I think a better solution taking advantage of existing PTP support in hardware is to encapsulate NTP messages in PTP packets. NICs and switches/routers see PTP packets, so they provide highly accurate timestamps and corrections, but the measurements and their processing can be full-featured NTP, keeping all its advantages like resiliency and security. There is an IETF draft specifying that:

https://datatracker.ietf.org/doc/draft-ietf-ntp-over-ptp/

An experimental support for NTP-over-PTP is included in the latest chrony release. In my tests with switches that work as one-step transparent clocks the accuracy is same as with PTP (linuxptp).

Fuzzing needs to cover all important bits of the code to be useful. The problem I see is that incomplete coverage creates a false sense of security. Projects have some minimal fuzzing coverage (e.g. in oss-fuzz) and care less about quality of the code, thinking fuzzing will catch all security bugs.

Rust code needs proper fuzzing too. It takes a lot of effort to ensure everything is covered and stays covered as the code is developed. Crashing libraries or applications can be a denial of service. Sure, it's lower impact than an RCE due to a buffer overflow, but it is still a security issue.

A better solution to secure bootstrapping of time would be NTP+NTS (RFC 8915) using self-signed certificates with unlimited time validity, which can be preloaded with the OS and updated via normal OS updates if the server key is compromised. They would probably need to run their own servers. There are some public NTS servers (e.g. Cloudflare and Netnod), but I have not seen any using long-term certificates specifically for this use case.

A major problem in synchronization of the system clock is PCIe. Hardware can timestamp PPS signal or PTP/NTP packets with accuracy of a few nanoseconds if everything is well compensated, but the PCIe bus between the CPU and the timestamping HW has a latency of hundreds of nanoseconds with potentially large asymmetry, degrading the accuracy significantly.

Measuring that error is difficult. A possibility is to run a program periodically making random reads from memory (avoiding the CPU cache) to generate a PPS signal on the memory bus, which can be observed with a scope. There is a lot of noise due to other memory activity, RAM refresh, etc. From the configured memory speed and tRCD+tCL timings the uncertainty of the error can be reduced.

This might improve with new hardware. There is a feature called Precision Time Measurement (PTM), which is a hardware implementation of an NTP-like protocol in PCIe, but so far I have seen this working only on some onboard NICs.

There's no networking hardware timestamp support for NTP because NTP has nothing to do with hardware timestamps.

Both NTP and PTP don't care (as protocols) where the timestamps are coming from. That's an implementation detail.

NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions

That was maybe 20-30 years ago, but not today. The wikipedia article needs an update. If you don't hit a routing asymmetry, in my experience it's usually milliseconds over Internet and tens of microseconds in local network if using SW timestamping. Please note that NTP clients by default use long polling intervals to avoid excessive load on public servers on Internet, so they need to be specifically configured for better performance in local networks.

You can find some measurements with HW timestamping here: https://chrony.tuxfamily.org/examples.html

Note that this is for the system clock, which has to be synchronized over PCIe to the hardware clock of the NIC. That adds hundreds of nanoseconds of uncertainty. It doesn't matter if the hardware clock is synchronized by PTP or NTP.

If you care only about the hardware clocks, it's easy to show how accurate is the synchronization by comparing their PPS signals on a scope. NTP between two directly connected NICs, or a with a hub, can get to single-digit nanosecond accuracy. I have seen that in my testing. It's just timestamps, it doesn't matter how they are exchanged.

No, NTP and PTP are two different protocols. They can both use hardware timestamps and reach single-digit nanosecond accuracy in ideal conditions. The main difference is in existing support in switches and routers, which is needed to avoid the impact of asymmetric delay between ports (typically tens of nanoseconds per switch).

PTP has good support in higher-end switches and routers, but it's difficult to secure and make resilient to failures. It was designed for automation and control networks in factories etc. NTP is a better fit for computer networks, but there doesn't seem to be any switches or routers with HW NTP support. If you really need the best accuracy with NTP, you can find old 100Mb/s hubs on ebay and create a separate network.

The article missed an opportunity to describe how spectacularly can things break when the 32-bit time_t overflows in Y2038.

If you still have such a machine (preferably without any valuable data), try setting the date right before the overflow with this command

date -s @$[2**31 - 10]

and see if you can recover the system without reboot.

I have seen different daemons stopped responding and just consuming CPU, NTP clients flooding public servers, and other interesting things. I suspect many of these systems will still be running in Y2038 and that day the Internet might break.

FWIW, from those 11 chrony CVEs:

- 8 were found within the project itself (mostly by me)

- none are memory-safety issues in the NTP-specific code

- the last memory-safety issue is from 2015 and it was in the custom management protocol (exploitable only by authenticated users), which was since then greatly simplified and made stateless

The project now has an excellent fuzzing coverage, it was audited, and I'm quite confident there are no remotely reachable memory-safety issues. I'll buy you a drink if you find one :).

NTP as a network protocol is extremely simple. There is no complex data, almost everything has a constant length. A minimal server+client implementation can be written in a few hundred lines of code. I wrote one in Rust, but the reason was server performance, not security.

Most of the complexity related to NTP is on the client side, in the processing of measurements provided by NTP. I don't think the language matters much here. However, if major operating systems will start switching to services written in Rust, I don't see a reason why chrony couldn't be rewritten in Rust, first the small parts related to networking and later everything.

RFC 7808 specifies a TZDIST service.

But most operating systems already have their own mechanism to update their timezone database. This is not a problem. Shifting timezones by 15 minutes every few thousand years would work with current software.

All we need to do is to agree that UTC will have no more leap seconds.

Some models that are known to work well with chrony are Intel I210, I350, and X550.

Those don't care about the protocol as they can provide hardware timestamps for all received packets.

Other popular NICs like the Intel X540 or XXV/XL710 are limited to timestamping of PTP event messages in order to limit the rate of timestamps which needs to be handled by the driver. For those chrony supports an NTP-over-PTP protocol which forces the hardware to trigger the timestamping by wrapping NTP messages in PTP.

Sub-microsecond accuracy is certainly possible. Here is an example with 3 network switches: https://chrony.tuxfamily.org/img/client-hwts-3switch-f323.pn...

The accuracy is limited by asymmetries in network switches.

In any case, whatever algorithms Clockwork is using with their protocol, I'm sure they could be used with NTP too. If additional information needs to be exchanged between the hosts, extension fields can be specified for that.

The unicast PTP support can be limited to boundary clocks and the enterprise profile doesn't require transparent clocks to support the unicast mode.

Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.

Do the transparent clocks in your datacenter support unicast PTP?

There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.

The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.

That can work, but probably not very well. Without real-time resampling some buffer in the path will underrun or overrun depending on whether the recording soundcard is sampling slower or faster than the playback soundcard. The clocks are not synchronized.

FPGA NTP Server 5 years ago

Yes, NICs with support for hardware timestamping are common (it's typically in the MAC, not PHY), but switches that have a good support for PTP, either as a boundary clock, or transparent clock, are not cheap. At least I have not seen one yet. Do you have any examples?

Some switches support NTP as a server and client (equivalent to the PTP boundary clock), but there don't seem to be any using hardware timestamping. It's just the classic ntpd using software timestamps, good to few tens of microseconds at best.

And yes, NTP could definitely perform as well as PTP if the switches had a proper support. In my tests with directly connected NICs the synchronization is stable to few nanoseconds, same as with PTP. At the protocol level, they use the same timestamps.

FPGA NTP Server 5 years ago

The main difference between PTP and NTP is that PTP relies on hardware support in switches and routers. Those are not cheap. If they had the same support for NTP, it would perform as well.

A highly accurate stratum-1 NTP server can be build with a common computer NIC. No need to mess with FPGAs (unless that's your thing). The Intel I210 is about $50. It has a PPS input and output. With some calibration, the timestamping can be accurate to few tens of nanoseconds.

NTP can work very well between directly connected NICs. But without hardware support in the switches/routers, that accuracy degrades quickly in the network. A single switch can easily add hundreds of nanoseconds worth of jitter and tens of nanoseconds worth of asymmetry.

Do most TLS-based protocols and applications use session tickets? From the gnutls documentation it looks like it's something that needs to be supported in the code, on both servers and clients. Is a client that cannot resume a session vulnerable to this issue?

Yes, but NTP as a time service (client/server mode) is safe. A request has a single response and their lengths are symmetric (that's actually a requirement for accurate synchronization). The problem with amplification is in the optional monitoring/control modes of the protocol (modes 6 and 7 as used by the ntpq and ntpdc utilities respectively), which should be disabled on public servers. Unfortunately, there are still some old misconfigured servers causing problems for a lot of people.

In PTP the problem is in the synchronization protocol itself. A master in the unicast mode is basically a programmable packet generator. It sends sync/announce messages at a rate and duration specified by its slaves, and the address can be spoofed.