How the Singapore Circle Line rogue train was caught with data

https://blog.data.gov.sg/how-we-caught-the-circle-line-rogue-train-with-data-79405c86ab6a#.qjsiufjb7

by sohkamyung • 10 years ago

921 110 10 years ago

blog.data.gov.sg

johnm1019• 10 years ago

TLDR; There was a single train emitting both correct and erroneous signals. This caused trains in the vicinity to lose data link which triggered an emergency brake safety feature. The article doesn't make clear what medium the trains communicate on which was affected (e.g. rail, wireless, powerline). They have not yet determined what caused the faulty hardware (or software???) on the faulty train to enter this state.

4ndr3vv• 10 years ago

Think you're looking at this story the wrong way; Its not about the fault but the bug hunting process.

I propose:

TL:DR - They had a strange hard to identify bug, but used a limited data set and interesting techniques to quickly find the esoteric cause of the problem.

sweetjesus• 10 years ago

imho his TL;DR is better than yours, but your improvements could be made to his. In my education we was taught that in science and business, the abstracts should tell the whole point of the story; the "spoiler" should not be saved till the end even though that makes for more fun; this is to allow a busy person to get the takeaway quickly and assess if they need to read all the detail. Whether anybody agrees with that or not, it's been burned into my way of looking at the world, so even when reading this article, all the way through I was gritting my teeth thinking "I wish I had some clue as to what I'm reading about."

so to add in your point, to his TL;DR I would put in "the intial dataset included only the trains that had suffered the fault, but as the fault was caused by a functioning train, a more comprehensive dataset was necessary to find the problem; had it been provided initially, less detective work might have been necessary"

xobs• 10 years ago

The trains communicate wirelessly. The system is entirely underground, do theoretically interference shouldn't have been an issue.

They shut off the mobile networks for a while to try and isolate the issue. It's interesting to read what was going on, though.

noisy_boy• 10 years ago

I'm not sure if you meant just the communication system was underground but some sections of the routes go overground (e.g. a large section of East West line is overground).

koliber• 10 years ago

When I interview developers, I like to ask them about their favorite bug. I want to hear a story. Most people have a good one. It's usually intermittent, hard to track down, and takes a long time to solve. It becomes a developer's nemesis, taking on an identity. The relief, when it gets solved, is huge. Good developers tend to talk about such bugs with passion.

This story is a beautiful example of such a story. Great read, great bug, great analysis!

usrusr2• 10 years ago

Tough one. I know exactly what you mean, I know I have been "there" many times, but I surely would not be able to retrieve any of those stories from memory even if my life depended on it.

We seem to condition ourselves to quickly forget the excitement because even the most epic bughunt will tend make you lonely if accidentally used for conversation in polite society.

toxik• 10 years ago

I think this goes for a lot of niche hobbies; a bike nerd might not be able to casually converse about that time his gears stopped working properly in the middle of a race, but to another bike nerd it would be a hilarious story! The same for us programmers, it's hard to relate to for non-programmers. Perhaps also why it's important that your friends share your interests.

voidifremoved• 10 years ago

One of my personal favourites.

Years ago I worked on a stock control system for a popular clothes retailer. Where previously store managers had to spend hours on the phone ringing the depots to see what was in stock and reordering popular lines, now they could fill in orders an they would be emailed to the depots in machine readable format, with the order balanced across multiple depots depending on availability of stock.

After the system had been in operation for a year or so, managers started to complain that some orders weren’t turning up. After a bit of digging it only seemed to affect one particular depot, but I couldn’t initially see a common factor between the orders. It had nothing to do with the size of the mail, or the originating store. I could tell from the logs the emails were being formatted and supposedly sent to the depot, yet they never translated into orders. The system would silently fail after we’d handed the email to the depot with no obvious reason why.

So I wrote a quick script to collect all the orders for a day from all the stores and compile a list of all product lines that were present in the orders that went missing, but not present in the orders that succeeded, just to see if there was a pattern. This turned up a few dozen items and after a quick visual scan of the product descriptions one stood out slightly, so I phoned the depot on a hunch.

The depot had an anti-spam filter on incoming emails that silently dropped messages with pornographic words in the body or any attachments. The retailer had just received a new seasonal product line, which was selling well, and the product description was “bondage tights”.

After getting the depot to whitelist emails from our system, orders for this product line started working.

epalmer• 10 years ago

Smiling at that one.

neotek• 10 years ago

We ran into a similar but much easier to identify problem with PayPal - any products we sold containing the words "Cuba" or "Havana" in their title or metadata, regardless of the context, would cause PayPal to reject the entire payment with a meaningless error and no indication of what had caused the problem.

PayPal refused to admit to us that this was the cause and insisted they couldn't shed any further light on the issue despite the fact we could trigger it 100% of the time.

In the last few weeks it's started randomly choking on the word "Pharaoh", God knows why, and God knows why it only does so about half the time, but I'm not going to bother asking PayPal about it.

smoyer• 10 years ago

Our neighbor, who started as an elementary school teacher but moved into elementary computer education (a roving teacher with an Apple computer lab cart) had a similar problem. Her title had been changed to something like Computer Education Specialist and her email would be blocked when sending to certain other government entities. When the school district upgraded their spam filter, her emails started to be blocked within the district as well. It turns out her title was being identified as Computer Education Spe CIALIS t.

SixSigma2• 10 years ago

My first 386 computer. Built in the computer shop I worked in.

It wouldn't boot if you tightened the screws on the case.

Solution: Don't tighten the screws on the case.

OJFord2• 10 years ago

My brother had exactly the same issue with his modern PC.

Solution: Move the HDD such that tightening the screws doesn't bend the SATA cable out of making good contact!

frandroid• 10 years ago

These stories are better if they contain an explanation for the problem.

ChuckMcM3• 10 years ago

Early development of a numerical solution for roots using Newton-Raphson in Fortran IV on a DEC 10. My code was correct, I walked through it manually many times and I had printed out many variables and yet it never found the roots.

In one of my subroutines I changed the value of a passed in variable. In my code there was a point where I called that subroutine with the value '0' for the variable.

The Fortran compiler was pass-by-reference, so the compiler accommodated me by creating a reference to the constant 0 and passing it to my code. In my subroutine I changed the value of that passed in 'variable'. Since the compiler was using that "constant" elsewhere any conditional that looked for 0 would fail after that point since the value of 0 had changed.

My professor and I only found it after we printed out the machine code generated by the compiler.

nitrogen• 10 years ago

I guess that might explain the "Have you ever changed the value of 4 accidentally" question from the hacker quiz I found so fascinating as a kid.

PhasmaFelis3• 10 years ago

Jesus jumped-up Christ on a snowmobile.

How does that get through testing? "Passing a number to a subroutine can result in the value of the number being changed"?

thisisquitecool• 10 years ago

Wow!!! Super impressive.

flashman• 10 years ago

This is going straight into my favourite bug-hunting stories, along with the 500-Mile Email: https://www.ibiblio.org/harris/500milemail.html

sdoering• 10 years ago

Thanks. Didn't know this one. Great read and highly entertaining.

hood_syntax• 10 years ago

A true classic, I never get tired of reading that one.

acobster• 10 years ago

Thanks for the recommendation! Great read.

qznc2• 10 years ago

Have fun: http://beza1e1.tuxen.de/lore/

erl• 10 years ago

Now that is a great collection, thank you!

One thing that I think would make the collection even better is spoiler free titles to maximize the suspense.

For example, the screen saver story lost much excitement when I knew the cause from the title.

overcast• 10 years ago

Oh man, as a systems guy, this warms my cold winter morning.

OJFord2• 10 years ago

"My printer doesn't work on Tuesdays" is another good one.

ethbro• 10 years ago

My favorite debugging challenges are the "Pfft, computers simply don't work that way" -> user reproduces symptoms in an inarguable manner -> "Hmm..." -> "Well, apparently when thirteen levels of stacked prisms are arranged in the right way, computers do work that way. Fixed (on Tuesdays)."

Edit: Or, you know, hardware failures that nobody expects anymore.

deugtniet3• 10 years ago

Not trying to be too negative here, but why wasn't a broad association analysis done on this dataset first? The number of outages correlated with train activity should have popped up like a sore thumb, no? Hindsight is always twenty-twenty, but I feel the author just wanted to show of his cool visualization knowledge, while the same could have been achieved with a much more boring association table.

EDIT: I am corrected by "Rifu". Activity data of all traincars was not available in the first instance, so a broad association analysis would probably have yielded nothing. The type of analysis the authors use seems warranted. Although I'd still recommend doing broad association analysis on any dataset, before moving into complicated visualization techniques.

folli• 10 years ago

I'm not sure if I understand correctly. I assume that the trains are in service the majority of the day. Wouldn't that make it very hard to correlate activity of a single train to the outages?

deugtniet3• 10 years ago

Not really, I believe the article states that 95% of the outages could be attributed to the single faulty train. That's a really big effect. Correlating traincar activity to outages will only NOT work when the traincar is active on the same track all the time, which it clearly isn't

Rifu• 10 years ago

As per TFA, they did say that the only data they had on hand were from the trains that broke down, so it would have been impossible to even detect the "rogue train" from a simple analysis of the data they had. Though it also said SMRT was slowly extracting the train logs from the incidents so I'm sure they would have gone with that if their preliminary analysis yielded nothing.

deugtniet3• 10 years ago

Right I overlooked this. That makes it more meaningful to do the analysis as in the article. I stand corrected.

jaclaz4• 10 years ago

I believe you are actualy "right", the article states "we were given ..." but WHY didn't they ASK for single train activities? These data is normally available in any railway, as it is used for several things, including programmed maintenance. In other words the Authors did IMHO a very interesting analysis of the data they were given, but - had they actually asked for the "right" kind of data - it would have been much simpler and faster. If you prefer they approached the problem from the viewpoint of data analysts (which is what they are, and evidently very good at it) but not from the viewpoint of an accident investigator.

kalleboo• 10 years ago

It's in the article. This all happened in one single day, while the train company was still compiling the extra data.

Hnrobert42• 10 years ago

Is there another name for broad association analysis? I have been unsuccessful searching for more information on it.

daneyh• 10 years ago

code snippets not displaying?

EDIT: Great article and good problem solving skills

reustle• 10 years ago

They're hosted by github, does the main site load for you?

adrianN• 10 years ago

Same for me.

foepys• 10 years ago

You need to accept cookies.

Terr_• 10 years ago

I hope they do a followup about what kind of problem they found in PV46.

I suspect that someone flipped the switch away from "More Magic" :)

http://catb.org/esr/jargon/html/magic-story.html

tankenmate• 10 years ago

I've been caught out by "ground" not really being ground before; two different sides of a whole floor computer / comms centre had different potentials. So most of the power was hooked up to ground on one side of the building and a small number of non computer / comms gear was hooked up to ground on the other side (think lights etc that were too difficult to rewire), but the other gear used wall sockets that were a different colour to make sure people didn't plug computers or network gear into them. This was all discovered after a temporary 20V spike difference in the different grounds trashed an expensive router. The electricians suspected that because we used a largish number of microwave links on the building the copper bars that run down the building to ground were acting like antennas and picking up the energy. So we defaulted all the sensitive gear to the "ground" that was least problematic. It was cheaper than re-doing all the ground circuits in the building with RF shielding. The joys of old buildings.

pizza• 10 years ago

This would make for a really great blog post! Could you write it up in greater detail?

verytrivial3• 10 years ago

A popular train line out of London has an electrical problem like this with one of the train models run upon it. Whenever these trains, travelling about about 100mph, cross from one power section to the next with the front the train entering one, and the rear leaving the other, there is some weird electrical event that causes a safety mechanism to violently retract the rear pantograph[1] (maybe more than one). This causes an almighty WHACK on the roof which scares the bejesus out of any visitors not used to impact noise. I've seen at least one coffee ejected from a cup onto someones trousers because of this. The train progresses unharmed.

[1] https://en.wikipedia.org/wiki/Pantograph_(transport)

a3n2• 10 years ago

It's in the press release linked from the article.

usrusr2• 10 years ago

It would also be interesting to see how the nature of the cause can explain the one statistical surprise that remains: PV46 is no outlier at all in figure 3, "frequency by train ID". According to the preliminary explanation, PV46 should either be so massively overrepresented there that the bughunt would be over before it really started (it is always close to itself), or it should only be affected by the estimated 5% of outages not related to PV46 which it clearly isn't.

nandhp• 10 years ago

Except that figure shows the frequency with which PV46 experienced an interruption, not the frequency with which it was near an interruption. As stated in the article, "We could not inspect the detailed train logs that day because SMRT needed more time to extract the data", implying that there was no position information for trains that were not actively experiencing a disruption.

Arguably, based on the preliminary explanation, it seems like PV46 might be over-represented (it had 4 interruptions) -- after all, it never travels in the opposite direction to itself.

waffle_ss• 10 years ago

According to the linked press release[1]:

... faulty train signalling hardware on PV46 was emitting erroneous signals in addition to the ones it was supposed to emit. These erroneous signals occasionally prevented trains travelling in the vicinity of PV46, including in the opposite direction, from properly communicating with the trackside signalling system. This loss of communications led to the activation of the trains’ emergency brakes.

[1] https://www.lta.gov.sg/apps/news/page.aspx?c=2&id=89defe38-4...

djsumdog• 10 years ago

Thanks! I was like, I got to the end of the article and they didn't tell us what the issue with the train was. :-P

gojomo• 10 years ago

Sounds like it may have been sending an (off-by-one, inverted, etc) signal indicating it's on a different track, or moving a different direction or speed, than it was in reality.

While the press release says these were "in addition to the ones it was supposed to emit", it's not hard to imagine that a similar error could happen where the wrong signals were sent in-lieu-of the intended signals, and thus leave trains rolling that should have detected an emergency-stop situation.

upofadown• 10 years ago

They might have problems reproducing such an intermittent problem without running the train. Often times such a problem clears up when someone physically messes with the hardware. Under such circumstances the problem would probably be solved by taking the circuit board that does the communications out and then carefully placing it in the garbage, possibly physically destroying it first.

Not all problems can be troubleshot and there are some things that remain mysteries forever.

a3n2• 10 years ago

Damn that was exciting.

From the bottom of the linked press release, which summarizes the overall investigation:

"In particular, I thank the engineers and data scientists from DSTA and GovTech respectively, without whom we would not have been able to theorise the possibility of a faulty train, and identify PV46."

A "steely eyed missile man" moment. https://en.wikipedia.org/wiki/John_Aaron

atmosx• 10 years ago

Nice write up.

As a side note, I find alarming the fact that the blog of the Data Science Division of a government (any gov) is hosted on a third party provider.

These guys look like they know what they're doing though, so I guess it was a weighted decision.

dnautics3• 10 years ago

do you find it alarming that governments (including the US) use official twitter and facebook accounts? Why should this be any different?

spraak• 10 years ago

This makes me excited about data science

yedpodtrzitko• 10 years ago

That's a really great analysis, almost feels like a detective story .)

OT: I am a huge fan of Python, so whenever I see Python helps to solve a thing I always say myself "yay Python!"

andrewvijay• 10 years ago

Wow what a digital Sherlock Holmes tale! Beautiful story. Incredibly hard if the data was not visualized that way. I learnt some good things. Thanks for sharing here!

hobaak• 10 years ago

Impressive work by Singapore government agency. I heard that they are as competent as private sector.

thex10• 10 years ago

Probably a lot easier to achieve as a unitary state!

anentropic• 10 years ago

we have private sector running trains here in the UK and they seem mostly incompetent

dnautics3• 10 years ago

trains in singapore themselves are private sector. SMRT is a publically traded company. If their trains keep exhibiting this problem, they run the risk of losing their contract to run the circle line to ComfortDelGro (or possibly an upstart competitor).

Probably the handoff to data.sg was probably done because it doesn't necessarily make sense for them to maintain a separate data science division within the company.

nl2• 10 years ago

I heard that they are as competent as private sector.

They are much, much better than that. They hire the best grads and pay lots, but expect a lot too:

One thing that stands out in Singapore is the quality of its civil service. Unlike the egalitarian Western public sector, Singapore follows an elitist model, paying those at the top $2m a year or more. It spots talented youngsters early, lures them with scholarships and keeps investing in them. People who don't make the grade are pushed out quickly.

http://www.economist.com/node/18359852

vkou• 10 years ago

Since moving to the US, my health insurance is handled by the private sector. Competent is not how I would describe it. (The more accurate term is 'Kafka-esque')

Back in Canada, on the other hand...

anigbrowl2• 10 years ago

The incompetence of government is a matter of religious faith among some people in the US, logic and empirical evidence be damned.

qq66• 10 years ago

The government is the highest-status and highest-paying employer in Singapore. The education systems are extremely strong, they send students all over the world for higher education, and those students usually return to Singapore to work for the government. The fact that their Prime Minister was the Senior Wrangler [1] at Trinity College demonstrates a lot about the overall way their government works.

[1] https://en.wikipedia.org/wiki/Senior_Wrangler_(University_of...

lyonlim• 10 years ago

Super cool to see how the team isolated the cause. I travel along the same route as well and this train caused half the circle line to be shut down during one morning.

For the next few days after, they shut down the telco signals in the train tunnels to determine if it caused the signal interference.

chaostheory• 10 years ago

Never heard of Jupyter Notebook until this story. It reminds me of Eve.

dnautics3• 10 years ago

Jupyter notebooks can run with other programming languages too. Julia, for example, is designed to interface with Jupyter for a more matlab/mathematica -ey experience.

chewxy• 10 years ago

You may have heard its previous name being used - iPython notebook. FWIW it took me a year plus to actually start using the name Jupyter. Wasn't until one of my students asked me why I kept saying "iPython" when the tab says "Jupyter"

UhUhUhUh• 10 years ago

The parallel with epidemiology is striking. And I keep on being amazed by the power of the visualization of data. The problem began to unravel when they "zoomed in" on a pattern that was not clearly visible initially.

ChuckMcM3• 10 years ago

ARGH! What hardware problems on train X can cause train Y to go into emergency braking mode? Curious minds want to know.

NamTaf3• 10 years ago

Wireless signalling transmissions, EM interference, at two guesses.

crocal• 10 years ago

Yep. Singapore Circle Line is a CBTC (Communication Based Train Control). Trains receive their movement authority through wireless transmission. If due to interferences the transmission is interrupted for more than some seconds, then movement authority expires and the on-board systems will bring the train to a stop through safe measures (typically, application of emergency brakes). It is quite common to have interference issues due to some geographical problem (e.g. incorrect installation of access point antenna) but much less common to have a jammer train. It is also quite impressive that the jamming would be long and wide enough to exhaust all the redundancies put into such system.

anigbrowl2• 10 years ago

Good work, though I can't help thinking that a defective train could just as easily have been caught through performing regular maintenance. Rolling stock and other physical things are subject to both manufacturing defects and wear.

I wonder why the rail managers allowed the defects to inconvenience passengers for so long instead of rethinking their maintenance procedures.

gridspy• 10 years ago

The defective train worked perfectly. It was the other trains that failed due to the radio emissions the rogue transmitted.

Not obvious.

jefurii• 10 years ago

Reminds me of a 1996 Argentine science fiction film called "Moebius" which involves a missing train, topology, the Dirty War, and of course Borges.

https://en.wikipedia.org/wiki/Moebius_%281996_film%29

Neliquat• 10 years ago

But what actually caused the interference? Never been blueballed so had by and article.

nicholasluimy• 10 years ago

faulty signalling hardware on the rogue train

http://www.straitstimes.com/singapore/transport/mystery-of-c...

grkvlt• 10 years ago

I was expecting something more like the 'A Subway Named Mobius' short story [1] - it's one of my favourite topology related bits of SF - OK, possibly the only topology related SF?

1. http://www.rioranchomathcamp.com/Topology/SubwayNamedMobius....

novaleaf• 10 years ago

I wonder if some "AI Gen 3.0" machine learning / cognitive machine system could have caught this, without human intervention?

I just yesterday attended a presentation by a ex NSA guy (who's of course pitching his company Adatos, so grain of salt) who claims you could feed the raw data in and the pattern will be found.

NamTaf3• 10 years ago

This is fantastic. I was in Singapore a couple of weeks ago and after mentioning that I'd narrowly missed out on getting an SMRT job earlier this year we got talking about the SMRT in general. He commented on this exact phenomenon of emergency braking for 'no reason', saying that the people of Singapore were losing faith in SMRT and their ability to keep up the reliability of the network after a spate of these issues due to 'interference'.

It's truly wonderful to see how they went about identifying the problem. This is fantastic work and the team deserves major kudos for nailing this.

mpweiher• 10 years ago

Dunno. About 2 paragraphs in, "oncoming trains" was the first thing that popped into my head.

How the Singapore Circle Line rogue train was caught with data

Related Stories