GPS error caused '12 hours of problems' for companies

http://www.bbc.com/news/technology-35491962

by majke • 10 years ago

96 38 10 years ago

fudged71• 10 years ago

Funny, I just saw this post yesterday by DARPA about preventing GPS issues: http://gazette.com/defense-department-researchers-push-plan-...

skykooler• 10 years ago

Interesting. Would the 13-microsecond discrepancy mean that GPS positions were over two miles off during those 12 hours?

bjcubsfan2• 10 years ago

No, the discrepancy was only on a timing message for determining UTC. This is not used for determining position.

cnvogel2• 10 years ago

Specifically, one or few of the sattelites sent a wrong value "A0" in its "Page 18 of subframe 4" message. Page 37, §2.4.5.5 in http://www.gps.gov/technical/ps/1995-SPS-signal-specificatio...

https://www.febo.com/pipermail/time-nuts/2016-January/095692...

ars• 10 years ago

Probably not if they were all off by the same amount.

Position is based on relative difference between time signals received from different satellites, not absolute.

bjcubsfan2• 10 years ago

This error only occurred on a certain revision of GPS satellites, not all of them. It would have been a problem, except luckily the error was only on a message for determining UTC. The error did not affect the message or clock that are used in determining position.

spencerhakim• 10 years ago

It's fascinating that an amount of time small enough to be completely meaningless to most of us can have such a big impact.

madaxe_again3• 10 years ago

I'm curious as to why they're using atomic clocks in space for a high resolution time source, rather than an atomic clock on earth... Latency?

fennecfoxen• 10 years ago

At a guess, they need a very small number of clocks (for uniformity) and they use the existing clocks in space because that signal distribution was more effective than new terrestrial infrastructure.

jewel• 10 years ago

I think it's a matter of price. GPS receivers are cheap and widely supported by timekeeping software. The cheapest atomic clock I can find is $1,500, but it's a hard thing to search for since the term is used to refer to radio-synchronized clocks.

If you did have an atomic clock in every datacenter, you'd still need to make sure that they didn't develop a problem that caused them to drift apart, and also you'd need to make sure that they were in sync with each other in the first place.

ploxiln4• 10 years ago

it would be a real problem if they drifted at all ... that's kinda the point of atomic clocks, no?

madaxe_again3• 10 years ago

$1500 is nothing if you're a telco. If cost were the reason they wouldn't blink at having five caesium fountains in each location - but as another observed it's likely about synchronicity - relativity is a problem if you're looking to have two clocks on opposite sides of the planet in perfect sync.

varjag• 10 years ago

GPS reference clocks are nanosecond resolution. Synchronizing from Earth would be challenging due to varying transmission distances and relativistic effects.

chiph2• 10 years ago

A ground-based transmitter could be destroyed in an attack, so having the clocks on orbit meant they were protected and would still be operational for a retaliatory ICBM launch.

Edit: So far as latency - the on-board clocks get resynced today every now and again against a ground-based reference time as they pass overhead of a USAF transmitter (primary + backup). If they used only moderately-accurate clocks as a cost-saving measure (or were at the limit of late 1970's technology) drift would be more severe and the tail-end of each orbit would be less correct than at the start of the orbit. So additional ground stations would be required in parts of the planet not always so friendly to the US, and they would have to coordinate their time sources, allowing for the light-speed delay travelling around the earth. Much more expensive/harder solution from an operational aspect.

Edit Edit: Don't forget that the GPS system was designed around military requirements. All the satellites (even today) have nuclear weapon detonation detectors on board and are a primary input to NORAD.

The 2nd SOPS has three missions: global navigation, time transfer and nuclear detection.

http://www.schriever.af.mil/GPS/

madaxe_again3• 10 years ago

Totally get it in the military context, and why it's useful in the context of a missile lobbing holiday, but was rather more curious as to why it'd be used as a time source for synchronising packets in commercial applications as the article describes.

Based on what others have said and my own inkling I think it's about synchronisation at a fine timescale without having to trouble yourself with bothersome shit like relativity, and leaving that particular pain in the ass to the US military. That's what I meant by latency, but phone keyboard, walking, lazy!

elfchief4• 10 years ago

The problem with atomic clocks in general is that they have very good short term stability, but not-so-great long-term stability. So they are very good at measuring the amount of time that has passed between two events, but not as great for telling you exactly what time an event happened.

The other problem is that there's not really such thing as a single, true time. Precision timekeeping is one of those places where relativity matters -- differences in altitude and the geology of the earth under your feet at a particular point change how fast any particular clock 'ticks'.

So to accommodate these differences, if you want events to be synchronized across multiple locations over a long period, you both need a single agreed-upon "current time", and some way to get information about that time information to the sites that need it.

You could surely do this with directly connected wires that were very carefully measured, but that gets expensive very quickly. The GPS system already requires super-precise time just to function, so it makes for a great distribution mechanism of "what time is it" information, where the people running the GPS system are bothered with all the details of figuring out and standardizing a single time, and you 'just' have to listen.

The GPS folks also keep track of things like how fast the GPS standard clock is drifting from the UTC standard clock (since no two clocks will agree over the long term, period). And this is actually where the problem happened -- Two of the parameters broadcast (A0 and A1) were incorrect, and it's these parameters that are used to specify the current time difference between the GPS master clock and the UTC master clock (the GPS master clock is disciplined to be as close to UTC as possible, but is always off by a minute amount, and that's what A0/A1 represent).

Pretty much it boils down to, much as it is in distributed systems, "there is no now". Even with maximum care and the highest quality clocks, there's really not a definition of 'now' that's consistent across more than a single location. GPS is currently the closest thing we have to a globally-distributed definition of 'now'.

eru• 10 years ago

Google is also working on a globally distributed `now': http://research.google.com/archive/spanner.html

Of course, they make use of GPS, but also add their own atomic clocks.

planteen• 10 years ago

Yep and "simultaneous" is not simple either as Einstein figured out in 1905: "So we see that we cannot attach any absolute signification to the concept of simultaneity, but that two events which, viewed from a system of co-ordinates, are simultaneous, can no longer be looked upon as simultaneous events when envisaged from a system which is in motion relatively to that system."

https://www.fourmilab.ch/etexts/einstein/specrel/www/

cnvogel2• 10 years ago

    > atomic clocks in general is that they have very good
    > short term stability, but not-so-great long-term
    > stability

Well... The normal number to look at is the "Allan Variance". Over a timescale of days (100'000 seconds) a good rubidium will achieve about 10^-12, which is 10ns. A hydrogen maser will have 10^-13 or 10^-14 over even longer timescales. This is pretty great long-term stability in my book. A good, but undisciplined, quartz oscillator will have 10^-12 over timescales of hours.

http://ivs.nict.go.jp/mirror/publications/gm2002/takahei/img... http://www.ke5fx.com/rb.htm http://www.thinksrs.com/assets/instr/PRS10/PRS10diag2LG.gif

By definition, anything steered by GPS will have infinite precision for infinitely long time-scales, as GPS time and UTC time ultimately are steered such that they are within a certain tolerance of each other.

http://www.leapsecond.com/pages/tbolt-tc/

And UTC time is the pretty elaborate average of ensembles of atomic clocks, hence atomic clocks produce the international reference time there is.

You can, of course, improve the long-term stability of any clock by locking it to a GPS (or other means of time-signal distribution). And for a good Caseium that might only require you to look at the phase, and slightly nudge a potentiometer (magnetic correction field adjustment) on the front by a tiny amount every week. But atomic clocks in general are pretty darn impressive.

UPDATE: This presentaiton has all the interesting plots combined, it seems: http://www.hipster.net/ShadNygren_Atomic_Clocks_for_Amateur_... (pages 45 and following).

pjc50• 10 years ago

In particular, DAB depends on exact time synchronisation to avoid interference. Every transmitter in the country has to broadcast the same OFDM symbol at the same time.

https://en.wikipedia.org/wiki/Single-frequency_network

The kit is supposed to accomodate loss of GPS, but evidently doesn't handle error so well: http://www.microsemi.com/products/timing-synchronization-sys...

elfchief4• 10 years ago

It's not that surprising that it has a problem with error, because the reality is, there's no correct solution for dealing with this kind of error.

The first question, of course, is "how do you determine when there's an error?" Half the GPS constellation was broadcasting the data, and for almost all purposes it was 'correct' data, other than being wrong. Perhaps ground stations could determine that they're getting different data from different satellites (if they have some good ones in view), or maybe they could just decide that the new parameters differed enough from the old ones to be implausible, but neither of these is a cut and dry "I definitely have an error" ... but let's say you have a foolproof way to determine that A0/A1 are incorrect. Great. THEN what do you do?

You could just decide to use the previous value for that parameter... which is great for a single clock, but what about clocks elsewhere that have a different value (the one being broadcast by the rest of the satellites) or an out of date value (say, a clock that was powered off while traveling in a broadcast van)? Barring external communication, you could have an arbitrary number of different interpretations of the current time -- a different one for each ground station!

You could also just accept that the new parameters are correct, but how do you then get your clock to the new correct time, without having a second (or other time period) that's significantly longer or shorter than a second? There's a lot of gear that expects a second to be a specific length, which is not gonna be happy if the length changes. (Think of playing back an audio file that's exactly 30 minutes long, and needs to start and stop at specific times, down to the bit. You then configure your system to send one bit every (1/bitrate) seconds exactly. How does that system recover when 13µs of bits go missing? How do you coordinate that behavior over multiple disconnected systems around the world which may have gotten the same information at different times (or not at all)?

From reading the article, it sounds like what happened was that the units didn't fail because of the GPS issue, but that they start throwing alarms (probably while going into a mode where they ran off only their internal (disciplined) oscillator. This seems like absolutely the right thing to do -- "uh, boss, things look really not-right here, and I can't fix it without breaking something, you really need to come deal with this yourself." (Note the comments about 'thousands of system warnings' but none about 'cellphones broke.') This is, from what I can see, really the only reasonable behavior these devices could have.

(disclaimer: I don't have any direct data on what broke vs. what complained loudly and woke people up. I'd love to have some solid, not-parsed-by-the-tech-media data on this topic, if anyone has any.)

contingencies• 10 years ago

I don't know anything about the specific event.

However, regarding your question of how to programmatically handle partial failure scenarios, I would reduce the trust in individual components (eg. satellites, receivers, receiver firmware releases, etc.) via redundancy and a voting-type mechanism to evaluate individual component outputs and their changes over time versus probable reality. Such a system would possibly include multiple offline clocks to match prior signals and thus maintain increased fault/drift-tolerance.

basicplus2• 10 years ago

more like.. what secret military mission was taking place during this time...

chippy• 10 years ago

or space craft, satellite launch from an "unfriendly" state...

jrockway• 10 years ago

This would have no effect on military uses. It was a static piece of data that was misconfigured on one satellite, not the actual time code on any satellite.

Disrupting GPS is less useful for military goals these days, since there are independent positioning constellations, like GLONASS.

jevinskie• 10 years ago

This is the press release that I received: https://ghostbin.com/paste/hdj3j

Johnny_Brahms• 10 years ago

"After the initial jolt, one of the first effects you'd notice would be that your GPS would stop working. The satellites would stay in roughly the same orbits, but the delicate timing that the GPS system is based on would be completely ruined within hours"

https://what-if.xkcd.com/67/

GPS error caused '12 hours of problems' for companies

Related Stories