EU's GPS satellites have been down for four days in mysterious outage

https://www.zdnet.com/article/european-gps-satellites-have-been-down-for-four-days-in-mysterious-outage/

by new_guy • 7 years ago

218 121 7 years ago

saagarjha2• 7 years ago

In a statement published after this article's publication, the GSA blamed the Galileo outage on "a technical incident related to its ground infrastructure."

I wonder what this could mean?

rando444• 7 years ago

Since there is nothing available but speculation, I'm going to put my money on a network issue.

dcomp• 7 years ago

My 2p: I think an atomic clock has failed

wolrah2• 7 years ago

Insert "it's always DNS" meme here...

mtmail• 7 years ago

copying from https://news.ycombinator.com/item?id=20427544

Guenter Hein, Professor Emeritus of Excellence at the University FAF Munich told us, “As far as I know, it is a problem of the PTF [Precise Timing Facility] in Italy – time has an impact on the whole constellation!”

saagarjha2• 7 years ago

But the satellites should be carrying their own clocks?

pgt• 7 years ago

Not trying to be funny here, but I would not be surprised if an automated Windows update borked something. Recently heard about an Antarctic base that lost heating for 36 hears after a Windows Update triggered a several GB download over the base's modem downlink.

wrkronmiller• 7 years ago

Can you provide a link to that Antarctic incident? I'm curious how a network outage would affect a system as critical as heating...

eightysixfour2• 7 years ago

The heater required an active internet connection and available bandwidth?

privong• 7 years ago

The heater required an active internet connection and available bandwidth?

My guess is that the heater was being run by the machine that required the update and the update download process meant the machine was not available for usual operation. But I don't really know.

martin_a3• 7 years ago

Sounds like a pretty normal device to me in the times of IoT...

Fordec• 7 years ago

It was actually just a gaming PC with a Nvidia GTX480 belonging to a scientist perpetually left turned on

Uplink• 7 years ago

The closest thing I can find to this story on Google is this: https://www.newscientist.com/article/dn26019-what-to-do-when...

It doesn't involve Windows as far as I can tell - they don't say what the cause of the problem was.

donarb• 7 years ago

This press release from the British Antarctic Survey says about that incident that a coolant leak caused the generator to overheat and shutdown.

https://www.bas.ac.uk/media-post/update-power-down-at-britis...

adminu• 7 years ago

"hears" is actually a great mixture of hours and years, that leaves you wondering ;) I hope he ment hours, though

shareIdeas2• 7 years ago

Are there windows operating systems in the satalites?

The only thing those satalites need to do is, accept signal, do math, and transmit signal.

I don't see why you would use Windows for this.

Sanzig• 7 years ago

The satellites themselves are most likely running some sort of proprietary or semi-proprietary RTOS for their C&DH computer (eg: VxWorks, QNX, Integrity, RTEMS...).

However, it's more than likely that there's a bunch of Windows machines on the ground segment side of the operation, since a lot of satellite fleet management software is Windows based.

wolrah2• 7 years ago

Not trying to be funny here, but I would not be surprised if an automated Windows update borked something.

If someone is running even important tasks that must not be interrupted, much less critical infrastructure like this, on consumer versions of Windows they're absolute idiots and should never be trusted with those sorts of decisions again.

If one insists on using Windows for uninterruptible tasks they should be using a server edition.

That said I'd always recommend

Recently heard about an Antarctic base that lost heating for 36 hears after a Windows Update triggered a several GB download over the base's modem downlink.

That seems implausible, or at least to have been miscommunicated. The download and install phases are separate, so nothing would have gone down until the download completed. If the heating system required constant communication and went down due to not enough bandwidth being left over that's obviously a very poor design in general, especially in Antarctica. Even if that were somehow the case, it wouldn't take 36 hours for their network team to block/throttle the update traffic to let it work again.

organsnyder• 7 years ago

If one insists on using Windows for uninterruptible tasks they should be using a server edition.

The server edition can also be configured to perform automatic updates.

Someone12343• 7 years ago

Recently heard about an Antarctic base that lost heating for 36 hears after a Windows Update triggered a several GB download over the base's modem downlink.

I've been researching this. There was an outage. It wasn't caused by Windows. What is your source for this claim?

retSava2• 7 years ago

Curious and not refuting - what was the cause of the outage?

stupidcar• 7 years ago

Given that the project is funded by €10 billion of EU taxes, I hope we'll get transparency sooner rather than later on exactly what's gone wrong, regardless of who it might embarrass.

the_mitsuhiko• 7 years ago

Considering they are running this as an early preview I'm not sure if we are going to get much in terms of public response. Maybe now that it's a long outage journalists will be poking enough.

Pretty sure universities and other institutions would be able to inquire though.

rtkwe2• 7 years ago

Like so? [0] Seems like they're still fixing it and it was in a pilot phase anyways.

[0] https://www.gsa.europa.eu/newsroom/news/update-availability-...

lallysingh• 7 years ago

"Galileo provides ‘initial services’ since December 2016. During this initial "pilot" phase preceding the ‘full operational services’ phase, Galileo signals are used in combination with other satellite navigation systems, which allows for the detection of technical issues before the system becomes fully operational."

aw3c2• 7 years ago

No, that gives almost zero information as to the known cause:

A technical incident in the Galileo ground infrastructure is affecting the functioning of the Galileo system, as a result of which there is a temporary interruption of some of the Galileo initial services.
The cause of the technical incident is identified and recovery actions are implemented to ensure that the nominal service is resumed as soon as possible while safeguarding quality of the services.

mtgx• 7 years ago

If it costs that much you'd think they'd also offer encrypted GPS, too. They'll regret it eventually anyway.

Dont_panic• 7 years ago

Encrypted GPS do yu even make sense, what benefit an encrypted GPS will provide. Eu will know where were you when you requested something and even if your data was encrypted and i don't think you can intercept gps coordinates and you have to intercept at least three for it to be useful.

detaro2• 7 years ago

offer encrypted GPS, too

That's the plan (or at least something explicitly reserved as an option), yes.

dwild• 7 years ago

encrypted GPS

Encrypting what?

Except if you want people not to be able to use the GPS signal, which sure make sense in military context (and even then, it's pretty meaningless, except if you change the key pretty regularly and you have a safe way to propagate it that won't be possible for enemies to get their hands on).

You know right that GPS signal is a one way signal? The receiver doesn't transmit anything, he just receive and use the timing and its data with some fancy math to be able find your position.

t3rabytes• 7 years ago

It's a system/service that's still in beta, I'd expect some occasional issues until it's deemed "production-ready".

jhayward2• 7 years ago

Everything on orbit and on the ground is production product.

When you are several billion euros in, have 24 satellites on orbit, and have been broadcasting with the 'Valid' bit set for 2 years you aren't really in 'beta' as we use it.

They have been in production mode in all but name. Given the life/safety implications, there are some rather rigid qualification checks that have to be done to be 'fully certified'. That's what's been going on.

benj1113• 7 years ago

So what would break if GPS went down for 4 days?

Sat nav obviously, Amazon package tracking, Grindr? (I assume that falls back to cell tower triangulation???).

I guess theres some more serious things, any examples?

systemtest• 7 years ago

Shutting down GPS would be a good way to avert the upcoming attack on Area 51.

benj1113• 7 years ago

I'm guessing this is a reference that's going over my head?

galangalalgol3• 7 years ago

this will be a fun thread! high frequency trading relies on gps disciplined osscilators for the precise timing needed (ntp isn't good enough, ptp isn't even good enough, though White rabbit should be). so hft would cease or go wonky

trucking companies rely on it to enforce rest periods. so more truck accidents?

turn by turn direction on phones would go out. wi fi is enough for nav, unless your phone and the wifi are on different ntp networks maybe I had a phone with broken gps and I didn't really notice.

a3n• 7 years ago

American trucker here. Occasionally I will lose the GPS signal in my truck, but my hours of service still counts down. The electronic logs are kept onboard, and transmitted at some point to headquarters.

I suppose the Qualcomm unit might eventually be off by a second or three. But I don't know if its clock is updated by GPS, phone or wifi. I'm guessing phone.

We're also required to carry paper log sheets in case of Qualcomm outage. So then it would be paper, watch and honesty. The honesty can be sanity checked by speed an distance.

rtkwe2• 7 years ago

Datacenters use GPS as a timekeeping source to keep synchronization across multiple locations more closely (or did who knows what Google actually uses, I could believe they wait till they move to a better solution to release their whitepapers).

Angostura• 7 years ago

I had a phone with broken gps and I didn't really notice.

I certainly noticed in the UK on an iPhone 6 where the GPS chip was borked. You would drive for 10 minutes and then you would get an updated location. This was in outer London.

alecmg• 7 years ago

Some devices supposedly can fall back to Galileo (now also down, nobody noticed), russian GLONASS and chinese... BeiDou.

How could I test that?

_pmf_• 7 years ago

If you have 60000 EUR+: https://www.rohde-schwarz.com/de/produkt/smbv100a-produkt-st...

Something similar is used to test German Toll Collect On-Board Units (OBUs) before refurbishment.

M2Ys4U• 7 years ago

Galileo (now also down, nobody noticed)

Galileo hasn't reached full operating capability yet, so that's probably why few people have noticed (as why would you depend on an incomplete system?)

Cynddl• 7 years ago

Not sure why you were downvoted. Our infrastructure worldwide relies a lot on location data, incl. GPS data.

There is actually an article from the New Yorker answering your question: much less traffic, all planes grounded, no train, etc. [1]

The U.S. Department of Homeland Security classifies sixteen infrastructure sectors—including dams, agriculture, health care, emergency services, and information technology—as critical, and therefore particularly vulnerable to sabotage. All but three require G.P.S. for essential functions.

[1] https://www.newyorker.com/tech/annals-of-technology/what-wou...

dmitrygr• 7 years ago

At least the plane stuff is wrong. Planes do not need to be grounded and will not be, but the capacity of the system will be reduced as we once again crowd into the rigidly-defined airways between the VORs. Approaches will use ILS as many do already and it will be all ok. Airplanes were ok for decades before GPS and even now can function without it.

davidf5602• 7 years ago

All modern digital radio networks (cellular, police/fire, etc.) rely on GPS signals to discipline high-stability frequency oscillators. If you look at any cellular antenna you can always see the white mushroom-shaped GPS antennas, usually lower on the tower or sometimes just sticking up off the equipment building. Those oscillators are used as references to generate the transmit/receive frequencies and also for very tight timing between separate transmit locations (many digital networks actually broadcast the same signal on the same frequency from multiple places at the same time).

Without GPS, these reference oscillators can free-run for a while but they will eventually drift out of tolerance.

Basically, without GPS, most/all cellular/digital radio networks will fail in some manner after a while. Precise failure modes will depend on a lot of things including exactly what is wrong with GPS and the type of emission, how the base station software is written, etc.

tzs2• 7 years ago

Do they discipline by essentially being a full GPS receiver that decodes the timing information from multiple satellites, derives the time, and then adjusts the local oscillator so it is running at the right rate based on that time?

Or do they do something simpler, and just derive a frequency reference from the frequency of the carrier signals and sync the local to some fixed ratio of the carrier frequency?

The former would be more accurate, but would break if something went wrong with the data. The latter would probably not be as accurate, due to Doppler effect (I'm getting +/-0.00041% shift from Doppler effect), but would keep working as long as the carrier is being broadcast. You could probably correct for a good part of the Doppler effect just from the last known orbital parameters of the satellite--even if the data updates have been broken for a long time that should work.

lxgr• 7 years ago

This is true for S-CDMA networks (a.k.a. CDMA2000, primarily used in the US), but definitely not for the internationally more common asynchronous CDMA based UMTS (a.k.a. 3G or WCDMA) systems.

I'm not too sure about LTE, but some initial googling suggests that while it does need a precision frequency and time source, there are ways of deriving that from the network, which makes sense if you think about indoor base stations that often don't have a clear view of the sky.

ppod2• 7 years ago

They put the Precise Timing Facility in Italy?

herpes• 7 years ago

What's wrong with Italy?

outside1234• 7 years ago

Nothing runs on time in Italy. Someone on this thread is going to say that is racism, but its backed up by a lot of data.

rbrtdrmpc-2• 7 years ago

A little rain today, despite that things are pretty good.

toyg2• 7 years ago

Casual racism, because fuck yeah.

Italy has a pretty big space-related industrial sector, and excellent academic standards in some fields. It might not produce geniuses with the same frequency of the past (Galvani, Volta, Marconi, Fermi and so on) but it's not some random backwater.

martin_a3• 7 years ago

but it's not some random backwater.

And nobody really thinks that.

My god, what a time to be alive, when you can't make simple jokes anymore, because somebody somewhere always is offended and thinks there must be some racist agenda in place.

Even as a German with a total lack of humour (and an Italian motorcycle with strange quirks in the garage) I found this funny. Funnier than the Holocaust at least.

acollins1331• 7 years ago

Italian doesn't count as a race

hyayz• 7 years ago

The word "racism" does not mean what you think it means.

ppod2• 7 years ago

I see the German Humour Facility has joined us.

tmalsburg2• 7 years ago

Please no Youtube-style comments here.

dang• 7 years ago

Please don't post flamebait to HN, and certainly not nationalistic flamebait.

https://news.ycombinator.com/newsguidelines.html

dfeojm-zlib2• 7 years ago

Dang, you're always on top of HN, figuratively and literally. Thanks, bro for defending the community. :PrayerHands: :PeaceSymbol:

supermatt3• 7 years ago

I had someone come and measure boundaries for some land on Friday, supposedly with centimetre precision. Does this mean that the markers they installed are likely in the wrong place?

module0000• 7 years ago

It depends... were there any markers already present? In a past life I worked as a surveyor, and if a marker was present(it looks like a silver-dollar on the ground, with a spike holding it in place, these can last 100+ years), we would use that to measure from. If, however, they set up what looked like a tripod with a oval-shaped dish on top, that's a GPS receiver, and could be affected by this outage.

Link: here are some survey markers for sale, linking it so you can see a picture of what they commonly look like: https://www.berntsen.com/Surveying/Survey-Markers/Aluminum-S...

supermatt3• 7 years ago

No - they placed new markers (orange disks on long spikes) driven flush into the ground, but they were walking around using GPS to do this (with supposedly centimetre precision, which I understand is what GNSS is for). This is in Eastern EU.

tinco2• 7 years ago

No, those markers work by listening to the GPS signals for an extended period of time and maybe PPK or perhaps an alternative network that's not satellite based.

supermatt3• 7 years ago

So it would be regular GPS assisted with some basestation at a known location to boost accuracy, as opposed to GNSS (Galileo)?

michaelt3• 7 years ago

I'm more familiar with the older US GPS system than I am with Galileo, but I can make an informed guess.

GPS satellites broadcast "ephemerides" letting receivers work out where the satellite is - and some other details like clock error and predicted effects of the ionosphere.

You can make a rough prediction of these weeks in advance, but not precisely enough to meet GPS's precision requirements. Up-to-date short term predictions are needed to make things precise enough, so a ground-based 'ground segment' or 'control segment' broadcasts updates to these every 30 minutes or so.

The 'ground segment' is comprised of ~20 monitoring stations (at known locations, measuring the errors on signals from satellites), one 'master control station' that aggregates this data and decides what update to send next, 4 ground antennas that can send that update to the satellite no matter what side of the globe it happens to be on, and a backup MCS.

This is what is meant by "ground infrastructure".

The design of GPS is informed by its cold-war-era-military origins, so there's a backup master control station, and the satellites have an 'Autonav mode' letting them operate for 60 days in even if the entire ground segment has been destroyed.

Assuming the Galileo system has a broadly similar design, you can imagine the criteria for a four-day outage:

* Failure of both the master control station and the backup, in a way that takes more than 4 days to recover from. Maybe their backup control station wasn't fully commissioned yet, wasn't fully redundant, or some IT problem hit both sites at once?

* Discovery of some fault severe enough they felt they had to shut down until it was fixed, and the fix couldn't be rushed out.

* Either satellites have no autonav mode, or it somehow failed (such as due to bad data from the ground segment), or it's imprecise enough they decided pushing people onto GPS was a more responsible option.

makomk2• 7 years ago

The rumour I saw was that the entire ground segment is dependent on a single Precise Timing Facility in Italy, which went down and took everything with it. Not sure what the holdover capability is like without the ground stations, but I'd venture a guess it's not great,

michaelt3• 7 years ago

According to a report from 2009 [1] "The Galileo ground segment has two GCCs, each with a Precise Timing Facility (PTF) providing a very stable reference GST to all elements in the GCC. The two PTFs are developed by separate companies; this diversification is applied to reduce the possibility of common failures in this important element."

I suppose it's possible the second GCC hadn't been fully commissioned? Or the switchover failed in some very hard to recover way?

[1] https://reports.nlr.nl/xmlui/bitstream/handle/10921/210/TP-2...

retSava2• 7 years ago

Since TFA reports two sats being in `TESTING` mode (whatever that is), could this be that they have a fix that they are testing on those two satellites before deploying to the others? Ie the problem is on the satellite and not ground-based?

edit: no, the below link states clearly this is grounds-based trouble. > technical incident related to its ground infrastructure

dfeojm-zlib2• 7 years ago

This is a good reminder to buy and use satnav modules supporting multiple positioning systems and technologies: most of GLONASS, GPS, BDS, Galileo, IRNSS, SkyHook, etc.

Disclaimer: Trimble Ltd. alum

EU's GPS satellites have been down for four days in mysterious outage

Related Stories