One second per second is harder than it sounds

https://rachelbythebay.com/w/2014/06/14/time/

by luu • 12 years ago

225 97 12 years ago

ars• 12 years ago

I had an S5 motherboard (ASRock P55 Delux), after pulling out my hair I finally gave up on it and bought a new motherboard.

And I made a resolution to never ever buy the V1 of anything. (It was a brand new Intel chipset, and a brand new motherboard design to go with it.)

If everyone did like me and never bought V1 then there would never be a V2, I don't have a solution for that, but I refuse to be the guinea pig.

makomk• 12 years ago

I had a desktop PC a few years back that exhibited S5. Can't remember the exact details, but I think it turned out that Linux was using a time source which wasn't stable on that particular hardware and switching to a different time source fixed it. There's been a lot of issues like that in the past, most of which are thankfully no more.

spydum2• 12 years ago

Since the advent of virtualization (most notoriously VMware ESX) time keeping has gotten to be much more difficult. I know back in my unix admin days, I spent more than a few days on problematic Linux kernels on VMs under ESX (made far worse when hosts were over provisioned and VMs were starved for CPU readiness). I know later enhancements like tick-less kernels may have helped, but I do not envy anyone who has to wrestle these beasts.

joevandyk• 12 years ago

Every so often, my linux vm running on VMware fusion gets really out of date. I tried the "sync time" feature in VMware, didn't fix it.

Eventually I had ntpdate running every minute via cron. Pretty annoying.

isomorphic• 12 years ago

I run various production systems in both VMware ESX and VirtualBox, and I have time issues in all VMs, regardless of hypervisor vendor. The VM tools are installed and operating "properly." The host machine clocks are synced correctly with ntp. Yet the VMs will sometimes get tens of minutes out of alignment in short order.

At this point I'm thinking I need to run ntp on the physical hardware with the least jitter, then just run ntpdate in a cron job on all the VMs. This would work better than ntp clients in the VMs or the VM tools.

ghshephard• 12 years ago

I spent about 2+ weeks in early 2003/2004 trying to get VMware Server to properly keep the time in Sync. I had memorized the VMware whitepaper on the topic, and had followed every possible bit of advice they offered (including not using NTPD) - nothing worked. Eventually just cronned ntpdate to run every minute and that resolved the issue on all our systems.

This, by the way, horrified all our NTPD theoreticians, who made it clear that running ntpdate was going to cause catastrophic things to occur in our Operations environment - but, given that our logs were all getting progressively more and more useless as we were unable to correlate times for events between servers - the worst case scenario (in my mind) had already occurred.

I don't recall any particularly negative side effects as a result of our ntpdate sledgehammer.

I presume things have gotten better in the last 10 years with VMware.

bradknowles4• 12 years ago

Somewhat.

Years ago, the answer was to run NTP on the host and use VMWare Tools to help ensure that good quality time was served to the clients. That's when the earlier versions of the page at <http://support.ntp.org/bin/view/Support/KnownOsIssues#Sectio... were written.

Since then, that advice has apparently been recanted.

The ultimate problem with any virtualization system is the frequency and accuracy of clock updates to the client, and the loss of clock interrupts.

To the degree you can solve that problem, protocols like NTP can work reasonably well to monitor and correct for various types of known clock issues.

MichaelGG• 12 years ago

Windows time sync only cares about being accurate to 5 !minutes, for Kerberos. HyperV and makes Windows not even able to be that accurate. Oddly enough, Linux guests have no trouble with subsecond accuracy.

I'm told the w32time codebase is not something MS employees like to look at.

72deluxe• 12 years ago

I don't think that w32time is something that users like to look at! I recall a situation where you run w32time on a server and it runs fine and doesn't report an error, but still silently ignore the option you set as it is using another time source from another machine that it has designated a master.

I could be wrong, but I was disappointed that the utility didn't return any errors when it knew that it was going to ignore you.

dm22• 12 years ago

28 people killed because a clock was off by .34 seconds (system hadn't been restarted in 100 hours)

http://fas.org/spp/starwars/gao/im92026.htm

kranner• 12 years ago

Having followed the link, I'd like to mention that a Scud missile was also involved.

lttlrck• 12 years ago

And a war.

manojlds• 12 years ago

Isn't it actually "28 lives couldn't be saved" because of the clock issue. Also, the article states that the defense was never tried against SCUD and could have failed nevertheless.

oakwhiz2• 12 years ago

I'm guessing that the people wouldn't have been standing within range of missile attacks if they didn't expect that the missile defense system would protect them.

spingsprong• 12 years ago

It probably would have failed.

The Gulf War Patriot was designed to shoot down Mach 2 jets, not Mach 5 Scuds.

Detonating slightly late against an aircraft is actually a good thing, since it makes the Patriot explode near the aircraft's middle instead of its nose.

A Scud is a short range ballistic missile. Once it's burnt out its fuel, it just carries on under the influence of gravity. From that point on, the only important part is the warhead in the very tip of the nose.

Detonating by the mid section of a Scud means you're just shredding empty fuel tanks. And the huge speeds involved might actually make Gulf War Patriot missiles detonate against the rear of the Scud and just shred the non-functioning engine, or even detonate behind the Scud entirely.

dm22• 12 years ago

I think it means that this was the first conflict that it had in against Scuds. There was a software update made a few days before this incident that was in response to another Patriot/Scud engagement. The big issue with this time was that the missile didn't even fire.

http://embeddedgurus.com/barr-code/2014/03/lethal-software-d...

It's interesting that there were 6 software updates to the Patriot missile batteries during Desert Storm, several of them were specifically to fix the clock errors.

It would be interesting to see the data for the misses and the hits compared to how long each system had been online for and what version of the software it was using.

Today's patriots are better (PAC-3, PAC-4 in testing) but still have several issues. I remember reading somewhere where a pilot got so mad that he was constantly being locked on by friendly Patriot batteries that he shot a missile at a friendly battery. http://books.google.com/books?id=7cU8Kpzu9osC&pg=PA339&lpg=P...

These seem to be mostly RADAR issues, I can't imagine how difficult it would be to program a missile to intercept another missile, there's hundreds of variables involved. I'd be willing to bet that these days they can run them through a computer simulator and detect most of these issues.

Unrelated to the missile but relating to the original topic: http://www.wired.com/2012/11/google-spanner-time/

chubot2• 12 years ago

I think the question is: when does it matter? What applications rely on an accurate clock? I guess it would matter if you are running some algorithm on logs that uses time as an input. Interested to hear any other examples.

DJB has a clockspeed package that addresses this problem. Anyone used it?

http://thedjbway.b0llix.net/clockspeed.html

It seems interesting but I haven't (yet) run into the need for super accurate clocks (or perhaps I am not running enough machines).

jfoutz• 12 years ago

spanner? https://en.wikipedia.org/wiki/Spanner_(database)

dsymonds• 12 years ago

Several security protocols (e.g. Kerberos) depend on systems having a somewhat reliable clock. They'll issue some sort of authentication token on system A that a user is meant to present to system B; the token has some expiry time, and the security of the system depends on systems A and B having comparable clocks.

tmarthal• 12 years ago

I know that offhand gaming and financial services rely on very accurate clocks at the application level. And you can't necessarily rely on the clients to tell you what their time is, as it can be spoofed.

Arelius2• 12 years ago

Gaming is actually a really bad example since as you say, you can't really on the clients telling you a time as it may be spoofed. Thus the clients times are always ignored, and thus time sync is irrelevant.

I'll say that in about 10 years of working on multi-player games, I've never once put a wall, or other absolute time on the wire for any purpose than just printing server time to the player for informational purposes.

MichaelApproved• 12 years ago

Communicating with other servers is a good example.

I need to have an accurate clock to within a few seconds when I sign Amazon S3 files. I have code that gives access to restricted files on S3. To give access, the customer makes a request from my server, I authenticate, sign a new URL and redirect them to the signed S3 URL.

For security reasons, the URL gets a signature that expires after a few seconds. The expiration date is based on X seconds from my servers clock. If my servers time differs from S3 server time then, according to S3, the link could be expired before it's even created.

In theory, a redirect takes a second or less to complete so you'd think an expiration that's 5 seconds in the future would be enough. The reality is, over time, the server clock can differ as much as 1 minute or so from S3's clock. So, I just set the expiration to be 120 seconds into the future and call it a day.

throw_away• 12 years ago

If ultrashort expiry time really mattered to you and ntpd not reliably working, you could always make a request of S3 and use the response's version of the date (which presumably is the version of time that really matters, as far as S3 is concerned).

jrockway• 12 years ago

I've used clockspeed. tlsdate + clockspeed turns out to be pretty reliable, and a lot less code than ntpd.

keeperofdakeys2• 12 years ago

As soon as you have any kind of highly-distributed system, accurate clocks are usually very important. When events can start on one machine, and end on another, it's important to have some kind of consistency (knowing that one event occurred before another). Probably the most famous paper on this is Lamport's "Time, clocks, and the ordering of events in a distributed system" - http://dl.acm.org/citation.cfm?id=359563.

But to give a practical example. On something like facebook, if you receive two comments on a post within a millisecond, how do you determine the order that they appear in the list? Each request may hit a different datacentre in the world, yet somehow the machines determine the ordering, and every request sees that ordering from then on. You can't just have a single server, because that couldn't handle the load for the entire world. These kinds of things require very strict temporal ordering, which is why time is such an important thing. This leads to the CAP theorem, and distributed systems in general.

chubot2• 12 years ago

So that's exactly why you don't depend on hardware clocks -- you use software techniques when your algorithm depends on some notion of time.

The point of Lamport's paper is that time in distributed systems is a partial ordering, not a total ordering. If you are taking values from hardware clocks, you are using a total ordering.

Dylan16807• 12 years ago

Using a highly accurate clock to solve ordering like that is punting on the actual problem. It's possible for requests to different servers to happen at exactly the same time. You need a mechanism to handle that. And any mechanism that can handle that can also handle clocks being a couple hundred milliseconds out of sync.

At a certain level of precision you have to accept that data only travels at the speed of light and there is no way to have a global ordering that shows data consistently and immediately. So you can use an easy non-immediate method like a vector clock.

chetanahuja• 12 years ago

"As soon as you have any kind of highly-distributed system, accurate clocks are usually..."

Let me stop you right there. If you have a highly distributed system, it's design CAN NOT rely on perfectly synchronized clocks among it's components. That's actually the point of the Lamport paper you cited as well as the OP. I actually started writing a detailed rehash of the Lamport's paper but I realize I'd do a poor job of it at best. Instead, I think it's best to just point to excellent and highly readable entire paper here: http://research.microsoft.com/en-us/um/people/lamport/pubs/t...

ajanuary• 12 years ago

For a bunch of distributed systems nodes need to be roughly matching real time, but also within some tollerance of each other; it's simplest to make everything as accurate to real time as possible.

I work in broadcast automation. It's a distributed system and the clocks need to be within a certain tolerance of each other. Historically that's done by having a dedicated piece of timekeeping hardware that dictates the station clock, but recently we've been experimenting with using NTP.

We have one node (A) where you author time-sensitive events, and another node (B) that polls for pages of those events, loops through them and takes some action when they should occur. Without both nodes agreeing somewhat on the time you'll end up missing events.

michaelt• 12 years ago

Clock years out of sync: All SSL certificates appear invalid.

Clock minutes out of sync: "Here's a temporary security token, I don't think it's expired" "That credential has expired" "This never came up in testing, I think I'll crash"

Clock seconds out of sync: Log files on different servers can't be merged reliably. Different requests went to different load-balanced servers and now you can't tell what order the logged events happened in.

spydum2• 12 years ago

And that last one has had many people chasing ghosts for years. I can also attest to time sync killing a number of SAML assertions and their window of validation.

superpatosainz2• 12 years ago

IRC: everything is timestamped to the subsecond. Any drifting causes channel takeovers and "random" nick kills, even links fail. Also, since a message from a server is broadcasted across the entire network, one misconfigured server can cause chaos in your hubs or completely unrelated leafs.

Or at least that's how I remember my experience managing an unstable (as in ddosed, every irc service in active development, a few in-house bots, etc) IRC network and having read the IRC and IRCv3 specs and unreal's and TS6 server message specs. If I'm wrong I'm sure someone here could correct me.

fanf2• 12 years ago

Your comment is very surprising. I don't see how clock drift can cause the problems you describe. There is no timestamping in the IRC protocol.

You can ask for a server's local time, but that is just for information.

There is a recently-added timestamp extension, but that seems to be to correct for delays so clients can record logs correctly. It is not for keeping the IRC network together.

sophacles• 12 years ago

Power grids require highly accurate, distributed clocks.

Well they don't strictly require them, but having that capability increases efficiency, robustness and safety.

They need to be accurate because electricity is fast - so a lot of protection schemes need to have a decision made across very long distances (which breaker, at which point on a 100 mile long transmission line should be fliped to isolate faults or stop "bad things" propagation) with minimal impact to the system as a whole.

They also need to be accurate because a good measure of frequency and phasor at geographically disparate points allows system tuning. A tiny change - e.g. speed up one of the 100s of generators could result in a much smoother, overall more-efficient transmission system. Similarly, adjustments to capacitor banks and transformers can provide slight phase adjustments for optimal power flow. But these resources are hundreds of miles apart - e.g the western half of the US and Canada (and parts of Mexico) (except for texas) is one big system.

Being a large, physical system, with feedback, and also being electricity, you get some very interesting effects to try and reason about. There can be very slow (relatively) oscillations in the system, but at the same time you have many, many high frequency noise signals mixed in to the elecrical signal. To find trends and patterns for the slow oscillations you need a very high sample rate to determine what noise to ignore. And these samples need to be time aligned, because even a perfect wave will have different phases at different points along it's propagation.

Anyway - that's an example I'm familiar with. I hope it made sense, seeing as I may not have had quite enough coffee yet.

rayiner• 12 years ago

Wireless networking protocols may very precise time synchronization at the PHY/MAC layer. Generally, wireless technologies like WiMAX or LTE divide time into frames (e.g. LTE is 10 ms). Then depending on whether you're frequency duplexing (FDD) or TDD (time duplexing), the frames are divided into subframes with defined meaning. If different radios on the network don't agree as to frame start/end times, interference occurs.

Here's an interesting paper describing the software implementation of certain features that are usually handled by the hardware (well, by software running on the baseband processor): http://people.freebsd.org/~sam/FreeBSD_TDMA-20090921.pdf.

jchonphoenix• 12 years ago

Um... In large scale systems... vector clocks? Or am I not reading this correctly (I skimmed)

MrBuddyCasino• 12 years ago

Not sure why downvoted, vector clocks are an interesting alternative. Its just that they won't fix your log timestamps or kerberos clock skew errors.

kijin2• 12 years ago

In the age of mobile phones that can get the time directly from GPS satellites with sub-100ns accuracy, the isolated hardware clock of a typical server strikes me as an arcane method of keeping time.

Is there an expansion card or USB device that can get the time from GPS and feed it to something like ntpd (and adjust for leap seconds from data downloded from other networks)? Or would that be impractical because of all the walls and metal cages that surround the typical server? What about an antenna on the roof of the datacenter that supplies GPS time to every server in the building?

rachelbythebay• 12 years ago

Datacenter GPS-backed time sources exist. You can get little boxes which sit in a rack and hook to an antenna and feed nice time to the rest of the place. Those other machines pick it up with ntpd.

The problem is when your system clock is so far out of whack (and/or unpredictable) that even ntpd won't help you. Get enough supposedly-identical machines running and this will probably bite you too.

kastnerkyle3• 12 years ago

This is 100% true. If you have ever watched ntpd ignore a stratum 0 clock (and every other time source) because some stratum 6 device reports the wrong time, you know the pain. Yes, this isn't supposed to happen, and yes it has happened to me many times.

feld2• 12 years ago

well, you can override ntpd to always sync no matter how far out of whack but it's not recommended

tinker panic 0

x1798DE2• 12 years ago

In the age of mobile phones that can get the time directly from GPS satellites with sub-100ns accuracy, the isolated hardware clock of a typical server strikes me as an arcane method of keeping time.

Genuinely curious, is it actually true that you can get sub-100ns time accuracy on a mobile phone? Is this because you have a direct connection with the satellite, which is calculating where you are? I would have assumed that the uncertainty on the latency of any such connection would be higher than 100 ns.

adrianb• 12 years ago

From https://en.wikipedia.org/wiki/GPS#Timekeeping :

GPS time is theoretically accurate to about 14 nanoseconds. However, most receivers lose accuracy in the interpretation of the signals and are only accurate to 100 nanoseconds.

kijin2• 12 years ago

GPS satellites don't do any calculating for you, and there is no "direct connection" between the satellite and your phone. The GPS receiver in your phone is purely passive. It uses latency (usually a few dozen milliseconds) to calculate its distance from each satellite.

As long as you have access to at least 4 satellites, triangulation (quadri-angulation?) allows you to determine your precise location in all 4 dimensions, three of space and one of time.

pjc504• 12 years ago

Yeah, GPS doesn't really work indoors. You can certainly get such devices, or even do it with consumer GPS, but then you have to account for the latency of USB.

cnvogel• 12 years ago

While most smartphones, theoretically, could get their time from GPS... do any use this method? As far as I know, my Android handsets get their time via NTP.

Ironically, often a misconfigured NTP server screws up GPS fixes (it somehow seems to be necessary for assisted GPS, which relies on the cooperation of the mobile network or Internet to faster acquire a GPS fix).

http://stackoverflow.com/questions/8308412/gps-how-ntp-time-...

sp3322• 12 years ago

I think cell phones get their time from the tower, since they have to sync clocks with the tower to talk to it anyway, and the towers get time from GPS.

ChuckMcM3• 12 years ago

For a long time synchronized clocks was the networking code's equivalent joke to Regular Expressions ("Now you've got two problems ...") That said, having implemented a 100ns accurate clock using the PPS on a $30 GPS module (Adafruit) and a Beaglebone Black (multiple) these days there are some interesting alternatives. It would be interesting if the Open Compute folks would specify a 1PPS GPIO input as part of the spec, you could easily deliver rack level 1PPS accurate signals throughout the data center pretty cost effectively (a few wires, a transmitter, a TDR tool. Not that it will ever be a 'solved' problem but it certainly makes some transactional protocols safer.

rdtsc4• 12 years ago

I've used PTP with special timestamping hardware (network cards) and could synchronize machines on a LAN pretty well. Now that doesn't guarantee global sync just sync between machines.

GPS NTP server can be obtained for as low as $600 or so. So those are becoming not as exotic now.

nick_riviera2• 12 years ago

We had radio time receivers (Rugby MSF, now Anthorn MSF) for a long time in the UK. Just plug them into an RS232 hole and have a site-wide master NTP server on that box, fire up the daemon and you're done.

I think the last one we paid for was in the region of $250 as the hardware is very simple.

http://www.npl.co.uk/science-technology/time-frequency/produ...

GPS is an interesting prospect but they are pretty HF which is a PITA if you have a unit inside a rack inside a machine room inside a building. MSF is 60KHz so is ground wave propagation so it had great coverage.

ChuckMcM3• 12 years ago

So FWIW my beagle bone is a "GPS NTP server" and it was built out of the following:

1) BeagleBone $55 (http://www.adafruit.com/product/1876)

2) Proto cape $10 (http://www.adafruit.com/products/572)

3) Power supply $8 (http://www.adafruit.com/products/276)

4) Enclosure $7 (http://www.makershed.com/products/beaglebone-black-enclosure)

5) GPS board $40 w/ PPS (http://www.adafruit.com/products/746)

6) Ext Antenna ($13) (http://www.adafruit.com/products/960)

7) SMA to Antenna ($4) (http://www.adafruit.com/products/851)

So what $137 + tax and shipping, and the Robert Nelson Ubuntu pack with Openntpd running on it. The trick though in data centers is always getting visibility to GPS satellites. I wonder when the data center guys will figure the can add a 'cross connect' to the GPS on the roof for $75/month or something :-)

sp3322• 12 years ago

"A man with one watch always knows what time it is. A man with two watches is never quite sure..."

rdtsc4• 12 years ago

You can buy GPS receivers and install antennas. Those are pretty good but expensive.

For S2, it can be configured to never jump. Instead you can just the time manually (run the equivalent of 'ntpdate <server>') at boot perhaps. Then don't start your software until ntp has been synchronized. Then maintain a watchdog that will alert you if you drop to a too low of a stratum.

Don't have an answer for S4 and S5. But I have heard that chrony is a good replacement for ntp (same network protocol but different discipline, faster conversion).

http://chrony.tuxfamily.org/

For a whole new other thing there is PTP (https://en.wikipedia.org/wiki/Precision_Time_Protocol). This is good if you want to synchronize machines to each other to withing a couple of hundred micros (maybe more with special network cards and switches).

kastnerkyle3• 12 years ago

The question is - how often do you reboot? For long running high uptime applications "ntpdate at boot" doesn't cut it... and the other solutions are nasty. In my experience, ntpd will ignore clocks almost at a whim even if they are stratum 0! Some stratum 6 clock decides to report a different time, and suddenly all your other sources are ignored until an ntpd restart. If your clock drifted too much... even ntpd restart with options to "force the time" can fail if the clock is too far out of whack. The you are back to forcing ntpdate - hope none of your processes are time dependent when system time jumps by 3 or 4 seconds.

Half of the GPS PCI devices require you to hack ntpd directly to get anything working, and the Ethernet sources have a strange tendency to go belly up after a year or 2 of running on non-120V sources. Not to mention that if you run the antenna cable too long, you might not notice that your GPS amplitudes are a little low until the next cloudy day. Or if someone adds a new hunk of metal on your roof, or blocks line of sight to some portion of the sky...

Long story short, time is a nightmare and I am glad other people think so too.

onli• 12 years ago

That is the nice thing about chrony. It tries to synchronize clocks even when offline, it works not only on a reboot. If it works as advertized (and I tried it some years ago at home and it worked there, for what it's worth), it should fix all of the issues described in the article.

rdtsc4• 12 years ago

That is why I mentioned to put a watchdog on it. At some point if you ntp server can't be reached, and you have clouds, and don't want to invest in an expensive atomic or maser based clock, you just don't get nice time sync.

bradknowles4• 12 years ago

With appropriate configuration, you can get ntpd to settle down and consider itself to be fully synchronized in a matter of seconds after startup.

You just have to configure an adequate number of upstream time servers, add judicious use of iburst, and you should be good to go.

On my laptop, I usually get good timesync within ten to fifteen seconds of startup.

lamontcg2• 12 years ago

Run a cronjob every hour which:

a. ensures ntpd is running and starts it if its not.

b. checks `nptq -rv` and looks for the self-reported state of ntp and bounces if it is not sync'd.

c. 'manually' checks ntpdate for the drift against a known stratum 1 and bounces the daemon if the drift is out of bounds (something like 50ms is a pretty good cutoff since ntpd normally does much better than that).

You can run this out of chef/puppet/whatever but it needs to run with a frequency of about an hour (faster than that and ntpd often won't settle well, and slower than that and you can be needlessly waiting too long to fix problems). Running it as a cronjob uncouples it from how often your config management runs.

This will catch all kinds of issues -- crashing ntpds, ntdps that randomly lose sync, kernel problems that keep them from keeping sync, ntpds that lie about their sync status, etc.

I set it up to e-mail me as a form of monitoring (with heavy-duty procmail filters).

Had 30,000 servers with this script running against it. I think 6 of them had shitty clocks that were getting reset every single hour, but were still keeping time correctly to under a second. Every couple days there would be a server that'd flip out and send a few e-mails until it quieted down and synch'd. At one point we had a shipment of several hundred new servers that all had issues with kernel drivers and the cronjob was bouncing ntpd constantly on them until a kernel upgrade fixed the problem.

I also used to bounce ntpd once a night, but had to take that out because it would cause non-monotonic slew in the clock which timers around service calls would turn up as negative seconds that due to unsigned int conversion would turn into 4 billion second p100 times.

You do want to be careful about things like network outages, if you can't ping your upstream stratum1/2 its probably better to leave things as it is. And again, you really want to limit the frequency that you bounce it at, and you want to be aware of servers where it gets into a state where you're bouncing ntpd constantly.

easytiger6• 12 years ago

This is a terrible idea when dealing with an application other than "users checking the time on their system clock".

Performing hard resets of the clock continually is a catastrophic infrastructure failure. A high performance 24/7 application that relies on sub millisecond let alone sub microsecond accuracy across multiple physical systems will be utterly destroyed by this approach.

pjc504• 12 years ago

If the clock is drifting substantially then your high-precision application will already be in serious trouble.

jacquesm2• 12 years ago

Not just that, monotonicity could be impaired.

lamontcg2• 12 years ago

Reading comprehension...

I had 30,000 servers, all but 6 of them were not getting hard resets. 6 of them were getting hard resets every hour because they were sick.

Occasionally a ntpd on a server would go a little crazy and it would get a few hard resets. This script got it back into spec so that it didn't need hard resets.

I mentioned specifically that you can't send hard resets to ntpd more than about once an hour or it won't have time to settle and you'll wind up in a situation where you are constantly sending hard resets to it (which is bad).

I took out the nightly forced hard resets of ntpd specifically because it affected monotonic time.

If you look at the algorithm I wound up with ntpd will only be reset if:

a. it is already crashed and not running

b. it is self-reporting that it is out of synch

c. is is more than 50ms out of synch with an upstream time server

In all of those cases, you have already lost the battle to get sub-microsecond accuracy.

easytiger6• 12 years ago

Err no mention of PTP at all?

There are hardly any oscillators which can be plugged into a host which have any decent holdover. One should never expect them to run in reasonable time. Standard TCXO prob losses 100 ms a day. Rubidium loses 0.01 ms a day. Time providers include the perceived accuracy of the clock in the protocol so that the client can make a decision about which time source to use.

Relying on the oscillator is never done in time critical applications except in emergency states where GPS is out of service.

Of course a lot of people run relative clock domains, which is just a pretty bad idea for most applications.

socialist_coder• 12 years ago

I have a problem in my AWS Windows cluster with the clocks as well. We have a mobile game with a tournaments feature, and tournaments can be as short as 5 minutes. So, it's important that all the clients get the same "end time".

Normally, at least 10% of the VMs are between 5 and 20 seconds off from the average.

By default, the VMs are syncing their time to an NTP server, but it just doesn't seem to be enough. I haven't changed any settings as of yet.

What are some of the things I could do on Windows box to sync up the clocks in my cluster? Scheduled shell scripts to force more clock syncs? Or maybe something on the application level?

bradknowles4• 12 years ago

First off, Windows isn't the greatest client for trying to get good timesync. The way the OS usually handles interrupts isn't sufficiently precise and accurate. That said, it's not uncommon for real-hardware Windows clients to get decent timesync down into the double-digit millisecond range -- usually about ~40-50ms or so.

Second, you're trying to run them in a virtualization environment, and that plays holy hell with whatever good timesync you might otherwise have been able to get. And AWS is probably about as bad as you could reasonably get in this regard -- VMWare has had a long time to get to where they are with regards to timesync for client OSes, and that's not particularly good.

So, your best bet may be to run NTP clients on the Windows machines, but to set them up to run at an artificially high rate of updates. Or maybe even run ntpdate periodically against good real-hardware time servers that are available to you.

You're starting off with both hands tied behind your back, and that's a hard place to come back from.

kiram9• 12 years ago

There is a standard to keep time on LANs now: IEEE-1588 which is built into a lot of new network cards. It enables the NIC to time stamp incoming packets which can be used to achieve sub microsecond timing.

Normally it can be implemented with a single GPS receiver on the network that then feeds all servers/devices. It is starting to become very common in broadcast applications as a reference. Furthermore most manufactures already offer some application notes/libraries to make use of it. And linux offers PTPd which implements the standard.

http://ptpd.sourceforge.net/

http://en.wikipedia.org/wiki/Precision_Time_Protocol

http://www.nist.gov/el/isd/ieee/ieee1588.cfm

bradknowles4• 12 years ago

PTP is a good choice for synchronizing clocks on a per-segment basis, but it doesn't work across a WAN and there are lots and lots of devices out there that don't support it.

NTP is good for doing time sync across the WAN, but to get the most out of it for certain applications you need to hook it up to reference clocks.

So, NTP can use PTP as a reference clock, and Bob's your uncle.

feld2• 12 years ago

FreeBSD timecounter in detail:

http://phk.freebsd.dk/pubs/timecounter.pdf

One second per second is harder than it sounds

Related Stories