Due to failure in the IT system, it is not possible to run any trains today

https://www.ns.nl/en/travel-information/calamities/sunday-3-april-no-more-ns-trains.html

by ar0 • 4 years ago

651 355 4 years ago

samspenc• 4 years ago

I do hope they have a more detailed RCA at some point, right now this is the only helpful paragraph in the article about what happened:

The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.

qiskit• 4 years ago

I was expecting more as well. Not much information to glean from a barebones press release.

If I had to guess, it probably was an issue with scheduling/timetable software rather than anything to do with the trains, rails, etc. Nothing exceedingly seriously or difficult to correct.

XorNot• 4 years ago

Keep in mind that train schedules control what trains are on what tracks when, so while their are literally fail-safes in the system, this is not exactly far removed from being "safety critical".

gridspy3• 4 years ago

tl;dr : Restarting train schedule mid-day is too hard and unsafe. So we're sending them back to depot instead.

amelius• 4 years ago

Why "unsafe"? Isn't safety supposed to be handled at a different abstraction level (in hardware)?

coder5433• 4 years ago

Defense in depth? With this safety feature inoperable, the margin of safety is reduced.

Other systems that might normally provide safety critical redundancy could be providing the sole measure of safety, with no other redundancy available in case one of those fails.

“Unsafe” is always defined based on context.

spullara• 4 years ago

I think that trains can travel in both directions on the same track and the schedule keeps them from hitting each other head-on.

makeitdouble4• 4 years ago

Might not be your point, but I wouldn’t be surprised if having people stuck in wagons in the middle of a section for half an hour because of conflicting schedules would be seen as safety risks for the travellers.

jrockway2• 4 years ago

It sounds like it was a crew scheduling issue. So if you implement some manual replacement for the automated system, it's possible that train crews will "time out" before they reach the destination, stranding passengers in the middle of nowhere. Or, will make a tired crew operate the trains, and tired people aren't very good at safety sensitive tasks.

There was an incident a few years ago where a LIRR track crew member stepped in front of a train going 78mph and died.

https://www.ntsb.gov/investigations/AccidentReports/Reports/...

The employee that died had been on duty for 38 of the 50 hours leading up to the fatality. One of the NTSB's recommendations is to implement software that avoids worker fatigue:

"The FRA encourages the use of certified biomathematical models, such as the Fatigue Audit InterDyne Model and the Fatigue Avoidance Scheduling Tool (FAST) by railroads to help them develop work schedules for safety-sensitive employees that align with healthy work-rest scheduling practices; however, these safety measures do not apply to roadway workers. 14 The work schedules developed through biomathematical models avoid many pitfalls causing worker fatigue that arise from excessively long work hours, highly variable work shift times that disrupt human circadian rhythms, and infringement on sleep opportunity times."

TL;DR: it's not about the trains crashing into each other. It's about sleepy people making bad decisions that gets them or their colleagues killed.

3232• 4 years ago

Maybe it might cause a traffic jam - blocked forwards, blocked backwards.

user_7832• 4 years ago

I was in the station for a different reason when I heard the announcement. The people entering and hearing the announcement were unsurprisingly very crestfallen, but it appears that twitter helped organize a lot of travel, with people posting where they were/going and picking up other travellers.

Also probably worth noting is that trains here are used fairly frequently, however unfortunately this isn't the first time trains are stopping - for example snow is a common reason for delayed/reduced service. (Snow isn't very common here in the NLs)

mushyhammer3• 4 years ago

with people posting where they were/going and picking up other travellers.

That’s ludicrous, I wish people helping other people this way was more common.

wolverine8763• 4 years ago

Imagine if we did that all the time! An online ride board.

xiphias26• 4 years ago

That software bug made a lot of Hungarians miss elections today in Netherlands :(

csomar• 4 years ago

So you are saying it's a plot to change the election's outcome?

edit: It was a joke. :/

xiphias26• 4 years ago

I don't think so. Anyways the same dictatorship (Viktor Orban) got full control again of the country.

synicalx• 4 years ago

You must never joke on HN, this is a SERIOUS site where SERIOUS 10x people talk about SERIOUS things like.... uhhh.... playing Doom in Grafana.

orblivion• 4 years ago

For those outside of Europe, could you explain how this works? Are these Hungarian citizens living abroad who are able to somehow vote from polling places accessible by train within the Netherlands? Is this an EU organized thing?

xiphias26• 4 years ago

Only in Amsterdam inside Netherlands after waiting a long line. At the same time Hungarians in Romania could vote by mail and the government was using Fidesz to collect the votes and threw out votes of the opposition.

The soft dictatorship works very well, Hungary will go even more down in the future. I'll probably leave Hungary before Hungary leaves EU/NATO and gets into a war.

detaro• 4 years ago

They need to travel to the embassy to vote.

flak48• 4 years ago

Most countries (even non EU ones) allow its citizens residing in foreign countries to vote in national elections at its embassies.

So Hungarians residing in Netherlands could have been able to vote at Hungarian embassies/consulates in Netherlands (if they made it to the embassy).

miggol2• 4 years ago

My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience.

I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook outages last year. But outside of IT I don't think anyone really expects those over here. And there might be national security considerations preventing such disclosure until any chance of a repeat has been engineered away, at which point everyone will have forgotten.

tjr2252• 4 years ago

Physical infrastructure can be fragile/fail catastrophically, as well. I recall starting a job near Seattle, 40 miles from my home in Olympia, WA and a passenger train derailed onto the highway killing a few people and backing up traffic for days. That was not a pleasant first day!

https://en.wikipedia.org/wiki/2017_Washington_train_derailme...

Wowfunhappy• 4 years ago

I assume, however, that this didn't ground every single train in the state!

eepp• 4 years ago

Damn this made me read a lot about this specific derailment, but also about Puget Sound, Salish Sea, Positive train control and all this 220-MHz stuff.

Thanks!

gonzo41• 4 years ago

I bet their schedule once ran on a commodore 64, and then they upgraded to a 386, and then ignored the IT department for 30 years. Hence today, chickens are coming home to roost.

postingposts2• 4 years ago

My very good guess is that you had a database failure caused by a crypto virus. They will not wish to announce that this was the root cause until A. A plan of action to avoid a future infection can be enumerated. B. They can be certain they will restore the functionality of the trains.

The other part of my guess is that they have a SQL db which is stored on /likely/ a windows server instance. I’d even surmise that this instance may be hosted on Azure but that’s speculation, not a good guess.

lucb1e• 4 years ago

This is one of my leading theories as well, although I thought they usually hit on Friday night in order to catch admins off-guard and encrypt as much as possible before being stopped? "The IT failure occurred at the end of [Sunday] morning".

On Sunday there is also not as much pressure to get things running as there would be on Monday. Or perhaps this is the incentive to pay now and get things fixed on time for the work week? In that case Friday night would again have made more sense unless the attackers have some very specific insight into how much slower restoring without paying is.

My alternative theory is an expired certificate that makes some core systems just not talk to each other anymore. The announcements on the stations, for example, were also out, and they lost control of the mobile apps (couldn't make the apps show that trains didn't run, the in-app scheduler showed all was A-OK), and that sounds quite dissimilar from the core train operation service, making me think it's more of an infrastructure than a specific system's problem. On the other hand, once the app could be controlled again (assuming this singular underlying cause), you'd think they could then also start putting train service back in place and that didn't happen for hours still.

I can't really make the pieces fit together for any theory, so then presumably something multi-faceted (one thing tripping one or two other things so it escalated from restarting one component to not being able to restart the trains anymore the whole day).

skeeter2020• 4 years ago

> but that’s speculation, not a good guess.

Well, as long as you're clearly stating _this_ part of your otherwise validated and highly likely pre-post-mortem is speculation... </s>

slim• 4 years ago

My guess for the attacker : conti ransomware group. They've been pretty active since Ukraine war. Last weekend they targeted the central bank of Tunisia

jollybean• 4 years ago

What would lead us to be inclined to believe that this is a 'very good guess'?

gpvos7• 4 years ago

Do you have any way of backing up that guess related to NS, or are you just wildly speculating?

quartz2• 4 years ago

From the notice:

It affected the system that generates up-to-date schedules for trains and staff.

...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole.

I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system.

A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days.

Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables.

In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently.

Cool stuff.

Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.

[1] https://homepages.cwi.nl/~lex/files/Interfaces.pdf

[2] https://repub.eur.nl/pub/11701/ei200803.pdf

[3] https://werkenbijns.nl/werkgebieden/it/sap-specialist-bij-ns...

sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.

incompatible• 4 years ago

If they are scheduling days in advance, then why would a failure of the scheduling system cause an instant outage?

xyst• 4 years ago

I have seen “SAP” tossed around in this industry and still have no clue what they do to earn money. Are they the equivalence of “salesforce”?

samstave3• 4 years ago

I love deep diving on public info to determine the internals (even, not only SOPs, policies, but also *governing culture* can be easily gleaned from reading what is placed in public view.

For example, The Salvation Army is one of the biggest jokes in all of charitable history.

I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID)

TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force.

They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion.

They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant...

They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market.

Its a fn racket.

=---

The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..

MetaWhirledPeas• 4 years ago

everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.

I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife.

My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.

rgj4• 4 years ago

sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.

Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum.

Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work.

This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.

weq• 4 years ago

I worked in defence on systems that did similiar and i remmember the one my company inherited and wow and wow it was the most complex, nested peice of SQL stored procs i have ever witnessed. It did the job, but it took HOURS to run. We had to re-write it completely to make any changes. But thats what happens when you let companies like Accenture at a system of yours! You pay them the big bux to be sueable, and they milk you like a cow.

lultimouomo• 4 years ago

sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.

And we can assume that a position in the IT department just opened up...

sofixa5• 4 years ago

36hr week, 5 weeks vacation, pension

1 hour more than the standard French contract, and there are also far more holidays here..

free unlimited 2nd class + low cost 1st class train travel.

.. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.

tannhaeuser• 4 years ago

I have zero knowledge of how the Dutch railway scheduling system works, nor of the nature of this failure, but SAP has been blogging recently about using Prolog for combinatorical optimization tasks such as the one that may be in use here or in SAP's Advanced Planning [1]. There are also well-known Prolog proponents from NL.

[1] https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-pr...

miggol2• 4 years ago

Thank you for your informative reply! These are such good reads

totetsu• 4 years ago

If it happed on a work day I wonder if peoples pandemic logic would kick in and you would start hearing, "It's not worth closing down the whole economy just because of the risk of a few people dying in a train crash"

jokoon• 4 years ago

And there might be national security considerations

That especially. I wonder how expensive that thing is going to cost.

throw7• 4 years ago

Whatever happened, I actually kinda like the wording in this notice in that it did not have the word "glitch".

qwertox• 4 years ago

The short sentences were what got me.

spac• 4 years ago

I wonder if this is connected to a similar incident in Italy. https://news.yahoo.com/italys-state-railway-may-target-16341...

mushyhammer3• 4 years ago

Considering the state of Italian railways IT, I wonder if the attack was accomplished by entering admin/admin in some login box.

denysvitali• 4 years ago

I think it's more likely that they didn't have any password at all :)

JaimeThompson• 4 years ago

Maybe they outsourced running the rail system to the people that run the Texas power grid?

stuckindoors• 4 years ago

There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger.

If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.

wsc981• 4 years ago

If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.

It's been Russian hackers so many times last few years (Hunter Biden's laptop, Trump elections, etc...), it's a sure bet this time it's the same.

Nobody ever got fired for blaming Russians.

BlueTemplar• 4 years ago

Their bad reputation is their own damn fault.

See also : assassinations - maybe the death of the top polish government & military and Total's CEO in plane crashes were just freak accidents, but a nagging suspicion that Russia was involved will always remain...

selimthegrim• 4 years ago

Southwest has put something out about it so it's in the press/Twitter for sure.

rossdavidh• 4 years ago

As a developer, any day in which you don't see a national headline about how your f*#(up has just made a lot of other people sad and/or angry, is a day that could have been worse.

3232• 4 years ago

In 2019 there was a power blackout in London. When power was restored, a number of new trains refused to start without a technician physically coming over with a diagnostics laptop to them because power fluctuated outside allowed bounds or something before the blackout.

gridspy3• 4 years ago

It's likely the same power fault could occur in a single train in other circumstances. When an electric train cannot access the grid reliably it would become far more dangerous - no more regenerating brakes, no reliable acceleration ...

Better to force the operators to fix the boom arm / whatever other cause of power glitches with fault codes like this. Even if they false-trip when there are grid level outages.

Symbiote2• 4 years ago

No, it was considered a design fault with these particular trains, and fixed soon afterwards.

Being unable to accelerate or unable to use regenerative braking is not in any way a safety problem. Either situation can happen at any time. (Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply, and this was the normal method for decades before regenerative braking became common.)

mostertoaster• 4 years ago

Probably Putins fault.

Conspiracy theories are so fun.

ncmncm2• 4 years ago

It wasn't the IT failure that kept the trains from running. It was management failure, not ensuring they could run without the IT system.

lucasyvas• 4 years ago

Now we're getting somewhere!

Tepix2• 4 years ago

That sounds unrealistic given the size of the problem.

Aeolun2• 4 years ago

They did it before the whole thing was automated. Surely they can do it again.

j0057• 4 years ago

They do have some kind of crisis mode where trains run back and forth between three stations, but maybe they had no way to trigger even that.

system2• 4 years ago

Do you know anything about NL train management? Just curious, how do you even know this is a management fault.

regularfry5• 4 years ago

It's both. Even with a system you could run without it, having to fail over would still be a negative event.

chevman2• 4 years ago

It’s always DNS!

comprev2• 4 years ago

Does Not Schedule? :-)

chevman2• 4 years ago

Where is the DNS test environment??????????

supernova87a• 4 years ago

While having algorithms and computers dictating every aspect of train movements, crew scheduling, resource allocation is efficient and in a sense optimal, I think there's something compelling to be said about having a system that can survive with human-rememberable patterns when such an optimizer fails. That it can fall back on (or have at the core) simple patterns that people can still operate without algorithmic intervention. I.e. the algorithmic tuning enhances at the edges, and isn't at its core some tangled non-interpretable system that someone can't operate without computer assistance. The idea of a protected set of core routes and staffing patterns that operates without having to be computer-determined?

I used to be so impressed at European train stations that they had a single sheet that would give the entire day's arrivals and departures, down to which track. I assume that behind that were schedules that weren't so far from being hand-calculated with equivalent sheets that told people and equipment where to be at what times.

I think we lose something when we jump so far forward that you cannot fall back gracefully without a system completely breaking down.

pjmlp• 4 years ago

There is an Isac Asimov book about this.

A crew is stranded in space, because of a computer malfunction and no one on board remembers how they used to be programmed.

at-fates-hands• 4 years ago

Is that "The Last Question"?

https://en.wikipedia.org/wiki/The_Last_Question

KineticLensman• 4 years ago

In at least one Arthur C. Clarke short story the problem doesn’t arise because they still have sliderules

0xdeadb00f• 4 years ago

I was going to say it kinda reminds me of the AI(s) in Neuromancer. In that novel it felt like the AIs were 1000 steps ahead of humans and pretty much just using then as pawns, selecting them carefully from the population, knowing they could either abuse their conditions to control them, or just flat out blackmail them.

vages• 4 years ago

I work within the train system of a European country (with ticketing, so I’m on the top and not the bottom of things). From my experience with station and onboard crew, I am very certain that the schedules and patterns are still very much human rememberable, at least collectively: People have to show up for work, drive the train and many other things.

My guess is that the failure is related to signalling or some other knowledge that exists “between” the trains (and brains). Having systems that can negotiate right of way is necessary to increase the number of trains and the speed of each individual train beyond their pre-computer-era limits.

Having all trains stop for one day is probably a judgment call to lower the overall duration: Having all trains stop for a day is a pain in the neck, but you know when you’re done. If you could not wait, you could probably have (quite) a few trains run without problems. However, you would have to work for a very long time to get all the trains back to where they were to resume the regular time table. This would be the preferable option during a time of crisis.

Foobar8568• 4 years ago

Try to run the pricing system in Denmark if it hasn't been changed over the years.... It was "deem" "impossible" for a human to calculate the pricing within a reasonable amount of time (with all the associated bonus).

Do not forget that there might several companies running on the same track, cargoa and passenger trains etc.

Managing a busy track is hell.

t0mas882• 4 years ago

The Dutch railways said (in Dutch) themselves that it was a problem with the scheduling software which also has a role in safety. Apparently this is also used to route around for example a stopped train, so when it's not working it's not safe to operate anything but a handful of trains.

There was a lot of extra impact because this system also feeds the passenger information so their app, website and information screens at the stations were not working either leaving passengers in the dark.

mikebos3• 4 years ago

The problem was in the scheduling of personnel and trains. The train schedule in the Netherlands is not human memorable. This is something they tried to correct in the past but was met with protest from employees because then they would only have one route on a day which would be boring.

Aperocky• 4 years ago

Commercial Flights can still be flown manually, without many automated sensing system.

But if they don't work you don't take off, it's simply not worth the risk.

pmontra• 4 years ago

The train schedule as seen by passengers stays the same for years, often to the rail number in stations. What changes are the crews, wagons, locomotives, possibly routes on the railway (switches and reactions to delays and failures.) A friend of mine worked at an app used by a railway's personnel to know where to go next. The app was not particularly complicated but the backend, which has been running for ages, took a long time to run. I don't know the details.

lbriner• 4 years ago

You also have to ask at what cost to maintain the backup system and how easy it is to test the Disaster Recovery process. If once in a while, you have a few hours delay (or in this extreme case, a whole day) you might ask if it would be worth it, especially if you could only test it by adding random failures to the main system like Netflix do!

As another poster mentioned, they are many, many independent variables in a railway system. In the old days, people recovered from problems by gut feel and experience was OK although there is usually no "right" answer. The newer systems apply some statistical anlysis and try and make the best decision usually with humans doing the last part.

I'm not sure about "because safety" though since signalling systems protect trains from each other and they only have automation in the sense of first train that arrives gets it route sets, the system cannot make trains crash unless there is a critical bug.

djcapelis• 4 years ago

As a general response not specific to this system failure: Train systems are designed to fail safe when not in state and significant safety engineering is done to ensure that if a rail system remains in a known good state over multiple decades almost every potential threat to human life caused by various equipment failure possibilities is quantified and reduced to a level of risk that’s acceptable. (Zero typically, for any known non-adversarial risks.) Many of the pathways in the fault tree involve shutting something down if it’s out of the correct state. So you will see train networks fail safe rather than run in a regressed state since these vehicles with up to a thousand human lives on board drive themselves to some degree in every modern system and we tend to hold them with a level for safety incompatible with easily winging it when something is wrong. This absolutely makes the system more brittle and less resilient, but for the most part it’s the tradeoff we accept in most modern rail engineering projects which are designed with the goal of reducing human loss of live to zero over the entire span of the system’s operational lifetime.

That said, train systems themselves usually maintain a ton of active operational procedures that remain part of staff training which would theoretically allow them to be more resilient and these procedures are kept current and you will often see them used in emergencies. I.E. on automated systems, trackway signaling often exists in the “not lit” off state and if the automated train control system failures, is often capable of lighting up for human controlled movement (usually required to be done at reduced speeds) of the trains during irregular operations like bringing stuck trains to the end of their lines and letting the people in them out at the next station. And any trains that still have a human conductor in a cab (or a cab for a human conductor at all) have procedures for that human to operate the train directly, even on segments of track where automated train control otherwise control all train movements.

We could try and run these systems in these degraded states under these emergency procedures, but most modern systems have safety analysis and engineering which focus on bringing the system to a safe halt state while fixing the underlying issue and returning to full operational state rather than messing around with rolling along in degraded states with unknown but assumed to be increased potential for catastrophic engineering failures with outcomes including loss of life.

(In the US it is generally illegal for a human being to operate a train unaided by some form of train control above 79mph. Most other systems have similar rules. We no longer trust humans themselves to operate these vehicles just because so many human lives can be at stake when they fail.)

Source: I did a stint working on system security and a little bit of electromagnetic compatibility safety engineering for rail systems. It was enlightening to see how the capital E engineers I was working alongside handle these concepts and design risk out of modern systems. The bulk of my work was in the US but the parts that weren’t were on systems that spanned multiple continents and/or train cars which were distributed worldwide. (To my knowledge none of my work has ever related to system discussed here.)

laurent92• 4 years ago

I assume that behind that were schedules that weren't so far from being hand-calculated

It’s more than that: The trains are negotiated at political levels! The “8:18 to Marseilles” (fictitious example) could be a headline in the news if the region refuses to fund it, and the worker’s union may have striked to keep it, while inhabitants’ HOA has negociated with the city to keep it under 12€: They are the object of a convergence of fixed interests. They run for generations: the train I took as a kid is still arriving on the same track today (and in one stop, it stops at track E; tracks A-D were dismantled but never renumbered due to this legacy).

The good thing with rails is that they aren’t going away, and trains can be negotiated for decades, they’re far from being scheduled on-the-fly.

Agentlien4• 4 years ago

This reminds me of the story one of my university teachers used to introduce himself. He was the most brilliant educator I've had, really passionate about making sure people could actually follow and so humble. The course was called "Digital and Computer Technology" and it was a first year course teaching you to build a CPU starting from logic gates and then program it with assembly.

He said that when he had recently graduated and just moved from Norway to Sweden his first job here was as a technician for the local railway. One of his first assignments was a call where the computer controlling all the track switches had stopped working. Luckily there was a backup, but they needed him to fix it immediately.

He arrived, took a look at the computer and its backup running next to it. He started out by measuring voltages on both, comparing to see what the difference was. After a while some men in suits came in and asked how things were going. He said it was all good, they said it wasn't because now all trains had stopped. Apparently, he had short circuited the backup.

"And that's when I decided to go for a theoretical career!" My teacher happily concluded. The classroom was left in a stunned silence.

Saser• 4 years ago

Rolf Snedsböl? :-) Agreed that he was one of the best educators (if not _the_ best) I had

Agentlien4• 4 years ago

Yes, Rolf Snedsböl!

I absolutely loved his classes. I bumped into him a few years ago when I was holding a recruitment thing for Ghost Games and it happened to be just after his lecture. I told him how much I enjoyed his courses.

valvar3• 4 years ago

An absolute legend! My favourite of his jokes: "when I moved to Sweden from Norway, the average IQ went up in both countries".

joenathanone• 4 years ago

'Those who can, do; those who can't, teach'

-George Bernard Shaw

blowski• 4 years ago

TIL that quote is _actually_ from George Bernard Shaw. https://www.phrases.org.uk/meanings/176500.html

paranoidrobot• 4 years ago

We extended that at a technical college I went to for a while because so many of the teachers were downright incompetent.

"Those who can, do; those who can't, teach; those who can't teach work here."

Dyac• 4 years ago

There's a common extension to this about teaching colleges -

'Those who can, do; those who can't, teach; those who can't teach, teach teachers.'

acbart• 4 years ago

Maybe if we paid those who teach better, we'd attract the best of the best to teach. Instead we drive good people out by making the job unbearable.

ilaksh• 4 years ago

Or maybe he was just smart enough to go for a steady job with less stress and humble enough to make a self-deprecating joke.

bmitc2• 4 years ago

More and more we are seeing "the computer says no" in our lives. We've replaced human activity and decision making with computerized versions, which are nothing more than the automation of stored human knowledge and routines.

sofixa5• 4 years ago

A much more efficient automation.

For instance, running high speed rail without computer signalling would be a huge risk there's no sense in taking ( due to the speeds involved, by the time you see anything of issue like a train or damage in the tracks in front, it's too late to stop).

bmitc2• 4 years ago

I think that's a bit of a leap from what I was saying. I didn't imply that all software and computation is bad. Obviously, a human cannot live handle any control system that needs even remotely fast operation. The point is that due to scaling issues, we're removing humans from systems where humans excel or could serve as backups in the event of failure (and actually vice versa).

Scale creates problems that are partially solved via software systems, which only handle a subset of the existing variety and new variety introduced by scaling. This makes the systems leaky with regards to unhandled variety and incredibly rigid (i.e., not robust) against exceptions, exceptions that could often be readily be handled by humans. By software being a snapshot of some small group of humans' knowledge of a system, it is completely unable to respond to live operational exceptions. Stafford Beer's work is interesting background on this type of thing.

A much more efficient automation.

There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?".

burntoutfire2• 4 years ago

A couple years ago, politicians in Poland wanted to do a quick change to the country's universal pension system. The change had to be quick to score some kind of political win. They wrote the new law and went on with quickly pushing it through parliament. To their surprise, sometime along the legislative process , the contractor responsible for maintenance and changes in the pension system informed them that they can indeed implement the change - by June next year. The politicians learned that, in the mordern computerized world, just passing a new law does not change how the state bureucracy operates.

throwaway483752• 4 years ago

Could they not make it retroactive?

dt3ft• 4 years ago

There is a swiss company based in Zürich that develops this kind of software. I believe it was a c#, wpf application, the last time I spoke with them. The schedules that their software produces is calculated and set for several years ahead. That baffled me! The tracks and infrastructure is booked up so far ahead! I think they mentioned that the german railway uses their software, nut only the sbb.

vanviegen• 4 years ago

From what I understand, the failing system is a bit different, in that its primary purpose is to deal with outages and other unexpected events.

reilly3000• 4 years ago

I think it’s worth mentioning that while the IT system couldn’t coordinate the trains safely, it was a management decision to stop them from running. Coordination can happen myriad ways. For example, trains could operate according to their normal schedule, and if one needed to deviate they could radio in, and an operator could let downstream trains know to proceed accordingly. Maybe simple cell-phone GPS pings could suffice. Perhaps the state of the system was in such chaos that such orchestration wasn’t possible with the staff on hand. That said, a truly resilient system should have multiple coordination channels to fall back upon.

mnumber2• 4 years ago

Computers are great at ensuring your workforce cannot function without them. This is a small price to pay.

throwaway483752• 4 years ago

The cost of a single mishap is too high. It's just not worth it.

gpvos7• 4 years ago

The drivers and conductors would like to end up at home at the end of the day. Even if you just run the normal schedule, there will always be disruptions, and there needs to be a system to adjust the planning so people and things end up at the right place at the end of the day. That was the system that failed. Have you any idea how frequent the trains in the Netherlands are and how complex the schedule? "Simply" running the existing schedule is an absolute impossibility.

About the only thing they could maybe have done was to fall back on back-and-forth shuttle trains between major node stations. But that would only provide a very basic service, with incorrect train materiel being used for the normally expected number of passengers. It would be just enough to get passengers and personnel home. I would like to see something like a pen-and-paper schedule that they can fall back on, but they don't have it. (They did something like that at the first corona lockdowns, although with more planning involved.)

CaptainZapp2• 4 years ago

Reminds me of when the entire Swiss train network shut down in 2005[0]

This was, however, not due to a computer glitch, but problems in SBB's electrical grid, which lead to overload of some of the lines and then to a complete shutdown of the entire grid.

Warnings were displayed on control consoles, but were drowned in 1000s of other meaningless warnings (there's a lesson for an UI designer here).

I was in the ICE from Basel to Zurich at the time and after fuming for a few minutes I figured: "What can you do?" and moved to the bar wagon. There we had something of an impromptu party until there was no more power for the cash register and they stopped sales.

In the end it was a (relatively) funny adventure for most and a huge embarrassment for SBB. I don't think they like to be reminded of that faithful day.

[0] https://www.swissinfo.ch/eng/swiss-train-network-shuts-down/...

Scoundreller• 4 years ago

There we had something of an impromptu party until there was no more power for the cash register and they stopped sales.

Funny how the entire system has ground to a halt and the bartender is still worried that their cash register receipts and inventory won’t reconcile and shuts that part down too.

Tabular-Iceberg4• 4 years ago

I’m puzzled by the lack of resilience in retail sales. I thought “always be closing” was supposed to apply to all types of sales. A receipt pad and a credit card imprinter under the counter would be a very cheap backup. It would be slow, but at least they would be closing.

It can even be considered a national security issue in the case of food, medicines and other vital services.

Robin_Message• 4 years ago

Aside: in case that's an eggcorn and not a typo, the day would be "fateful", as in, something momentous happened, rather than "faithful", showing loyalty.

CaptainZapp2• 4 years ago

Thanks!

Not a native English speaker. So it's easy for me to confuse the terms. You're right, of course.

yummybear• 4 years ago

Failed OTA update?

j0e1• 4 years ago

What jumps out at me after reading this notice is how it completely lacks any apology. I am not prescribing what it should say but merely observing that having lived in North America for sometime now, it seems hard for me to imagine for such a notice to not sound profusely apologetic here.

Note: I do not consider words like 'unfortunately' and 'extremely unpleasant' apologetic. Also, I understand there are cultural differences between the Netherlands and the USA- no joke.

withinboredom3• 4 years ago

Why would they be sorry? They’re the ones who are missing millions? of Euros in revenue due to this issue while the passengers just had to take a bus or bike somewhere. They probably would tell you that if you had to bike, it’s healthier that way. A little bit of rain won’t make you melt.

mnumber2• 4 years ago

"Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." I think the writer is more pointing the finger at IT than feeling apologetic. Imagine getting called an enormous failure by your adjacent department, that writer wants you to know who screwed up instead of apologize for their screw up. Seems more useful than an apology in my opinion.

fwr• 4 years ago

This is very consistent with customer service in The Netherlands - nothing is ever the fault of service providers, and you manage to convince them otherwise - you'll get nothing more than "unfortunately (...) not possible"

MonaroVXR2• 4 years ago

Customer service is done by another comapny. Source: It's me this time, NS customers service errrr....We help, it's called Webhelp.

coolgoose2• 4 years ago

I don't mind the tone, I got tired of US style politically correctness just for the sake of saying it.

How would "We are trully sorry for this incident" make it any better ?

I do mind the fact that they don't pay the people their money back if they try to find alternatives to get to their destination. That's 100% unacceptable.

shankr• 4 years ago

When is being apologetic became politically correctness? Are we soon going to call every decent human like behavior in public space as being politically correct?

Vinnl• 4 years ago

I do mind the fact that they don't pay the people their money back if they try to find alternatives to get to their destination.

Is there any reason to assume that this is a fact? Usually in these situations, NS is pretty good about paying people back.

MonaroVXR2• 4 years ago

Customer service is done by another comapny. Source: It's me this time, NS customers service errrr....We help, it's called Webhelp.

CannisterFlux• 4 years ago

Sounds similar to a problem that happened a couple of weeks ago in Madrid, although that was not the whole network it pretty much shut down the trains for the day. That was an IT failure too. https://www.20minutos.es/noticia/4973632/0/graves-retrasos-e... (in Spanish)

tuukkah• 4 years ago

Wow, talk about a single point of failure when an issue in the systems of one station can stop all but one train line. Interestingly, it seems to be a station where tracks from six directions meet so a backup mode of trains simply passing through might be impossible: https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_...

comprev2• 4 years ago

I was walking up to Amsterdam Centraal when I noticed floods of people all exiting the building... within minutes an Uber was booked to Schiphol before prices rocketed due to demand.

AndyMcConachie2• 4 years ago

The fact that it happened on a Sunday suggests to me that this is the result of a service outage gone awry. Realize that the trains failing on a Sunday is really not a big deal. However, if the trains don't run for Monday morning rush hour that is a big deal. I bet they made a change Saturday night and something broke Sunday morning.

It's now Monday morning here and the trains are back to normal in time for rush hour. So it's not really a big deal.

regularfry5• 4 years ago

I have to say that I do find this framing which says "well, it didn't affect business, so no big deal" to be somewhat distasteful. It's completely dehumanising. Why run the trains at all at the weekend, if a weekend outage is no big deal?

tsimionescu3• 4 years ago

That's not the framing. For the majority of travelers, missing a weekend trip is an annoyance. In contrast, missing their work is a major headache, for them personally. It also cascades pretty badly if it happens on a lathe scale - meetings get pushed and overlap others, suplliers are not able to deliver, stalling production, financial operations get delayed, potentially affecting deadlines - it's a huge mess in everyone's life that won't just go away when trains come back. It's also as true for non-profits, state organizations and schools as it is for businesses.

lifty• 4 years ago

Most likely their Kubernetes cluster went down. (not a joke)

lqet• 4 years ago

Seems like trains are up and running again today as announced, with only little delays:

https://travic.app/?z=9&x=628581.0&y=6814491.0

woudsma• 4 years ago

I didn’t know about Travic, thanks!

qwezxcrty• 4 years ago

Based in my knowledge as a public transportation enthusiastist in China, I got really surprised to see a IT problem bring the entire system down. There should usually be some "more manual" backup mechanism that allows downgrade and operation of trains without computerized signaling system. By kinds of token signaling it will still be possible to run trains as long as the physical infrastructure is still there. The efficiency will be much lower and more prone to (potentialy deadly) errors, but no more trains today sounds unlikely.

apexalpha• 4 years ago

The Dutch rail network is top 3 busiest in the world. Many lines have 10+ trains in each direction, every hour.

The usage of certain lines at a certain time is calculated in periods of 6 seconds!

We're waaay beyond normal human coordination capacity.

NWoodsman2• 4 years ago

This post is a fascinating use of grammatic passive voice. The hallmark is gratuitous use of "being" verbs, i.e. "is,was". Passive voice is a way to avoid using active voice, which is an implicit way to avoid responsibility for an error.

"Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." // => "Due to the enormous impact of the failure in the IT system, NS cannot run trains today."

"Although the cause of the failure has now been resolved, the impact is considerable. To be able to start up reliably, systems must be updated and trains must be brought to the right place. That takes time. For our passengers, this is extremely unpleasant news." // => "Although we have fixed the cause of the failure, the failure impacted most of our customers. To fix this failure so that trains would start reliably, we needed to return all trains to a central depot, which took nearly 12 hours. We apologize for the extremely unpleasant interruption."

"The expectation is that tomorrow morning the normal timetable can largely be resumed. The night trains can still run." // => We expect to resume minor operation with night trains first and then major operations tomorrow morning.

"The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure." // => The IT failure occured at 11am in the scheduling system, a major requirement for safe operation; if train operations are interrupted at one location, the system normally reroutes other trains to prevent collisions. This system failed.

"The international trains are not affected by this failure. For information about the timetables of other transporters, passengers can consult the websites of these transporters." // => The failure did not affect international trains. If you have an inquiry about other transportation services, please consult those services.

"The journey planner is updated." // => We updated the journey planner accordingly to account for the interruptions.

// Generally the use of "passive voice" indicates poorly-taught grammar, however the use could also indicate desire to avoid responsibility.

NWoodsman2• 4 years ago

The "about us" section of the webpage :

"Who are we? NS is active in the public transportation sector. We encourage the use of public transportation and keep the Netherlands moving. Our travellers are our 1st, 2nd and 3rd priority in all of our activities, and we do our utmost to make their trip as pleasant and sustainable as possible from door to door."

This is also fascinating. I would expect to find "NS (Netherlands Service) provides rail services for the entire country". Instead they choose vague grammar, more the idea of NS rather than what NS actually is.

"I am a programmer" => "I encourage the use of computers to provide solutions for common business problems". Just a strange use of grammar.

ivraatiems• 4 years ago

I think this is a translation thing? I'm not sure they're doing it on purpose since a lot of it isn't proper idiomatic English.

rbreaves• 4 years ago

No - due to failure with management your IT failed.

Due to failure in the IT system, it is not possible to run any trains today

Related Stories