‘They Don’t Make These Anymore’: Maintaining the MBTA’s 100-Year-Old Signals

http://www.wbur.org/2016/03/07/mbta-signal-maintenance

by ilamont • 10 years ago

98 36 10 years ago

codys• 10 years ago

It’s not as simple as just changing the signal. In order to change it out and modernize, you need to update the entire system.

Does anyone have any idea why this is? What prevents new signals from being backwards compatible with the old system?

And if this were true, wouldn't all of the signals need to be the same hundred year old type? How would line extensions ever be able to be done in that case? Is the system segmented in some sense? If it is segmented, what prevents further segmentation to allow incremental updates of the signals?

mmoche• 10 years ago

I agree that that is a puzzling statement.

Perhaps the intent was to say something like "you can't reap marginal benefits from the replacement of a single signal since the whole system must be modernized to allow the advanced signalling afforded by new technology," or something.

VLM• 10 years ago

Its actually worse, in that replacing 1% of switches means the switch repair shop and field techs now have to learn 100% more and stock 100% varieties of spare parts etc. Eventually a breakeven point is found where the reduction in labor on the old switches exceeds the cost of being dual capable.

Its possible to imagine numbers where the cost of being dual capable is so high, the cheapest solution is some kind of forklift upgrade of the whole system. Which is kinda what they're doing.

A good hardware analogy is you have a company thats a windoze PC shop. Maybe even all the hardware comes from the same Dell contract, same OS image on every box, its pretty cheap. Surely OSX would improve any end user experience, but imagine the cost of instantly doubling the helpdesk workload to now handle two kinds of hardware and two OS images and two licensing agreements and generally two of everything.

Or a good software analogy is given a somewhat reliable mysql cluster, dropping postgresql on it and trying to use both at the same time is likely to result in some heartache.

bsder• 10 years ago

Does anyone have any idea why this is? What prevents new signals from being backwards compatible with the old system?

Presumably you can't use anything currently manufactured.

So, you'd have to commission a unique, backward compatible design. And test it. And qualify it.

And no company is going to do that without you basically paying them the same up-front amount as if you just contracted to replace all the signals.

kesselvon• 10 years ago

Because the old signals systems are analog or electromechanical, all modern signals systems are computerized. You'd essentially have to build something custom from scratch to do so, which would be way more expensive than simply replacing everything. Since both options sound unappetizing, they end up doing patch repairs like this to keep the system going, but that can only last so long before it just breaks down.

Eric_WVGG• 10 years ago

^ that

The Atlantic article linked above covers this in more detail, and is an excellent read all-around.

Symbiote3• 10 years ago

Further segmenting the existing system probably means a huge amount of changes to the existing system — more expensive than replacing the whole lot.

The old system will be based on detecting trains in "blocks", sections of track insulated from each other where the train's wheels make the circuit. I was about to say this would use relays, but Wikipedia says the first all-relay system was installed in 1929. So it's at least partly mechanical (levers moving in the way of other levers). The logic implemented in this will prevent the signalman from sending trains on a collision path, but only if drivers obey all red lights.

The new system will either be something similar, except implemented in electronics and stopping the train if the driver doesn't. But for a metro system, in order to get higher capacity, it will probably do away with the "blocks" altogether, instead ensuring that every train has a safe distance ahead of it. Maybe they will also automate the trains, so the train drives itself.

Either way, they'll probably upgrade the trains to support the new system, then install the new system in whatever sections allow it, then remove the old system once the new system is working correctly. I'd expect there's a huge amount of sunk cost once the project starts (design, modifications to trains, staff training etc) so it doesn't make any sense to stop halfway and keep the costs of both systems.

https://en.wikipedia.org/wiki/Interlocking

mike_hearn• 10 years ago

Actually the old mechanical systems can stop a train automatically too. They have levers that move out onto the tracks and hit brake activator panels on the trains themselves, which force the air brakes on if the train tries to go through the red signal.

The old mechanical systems aren't really unsafe. They're just inefficient, full of moving parts so they break all the time, and require the transit agencies to run their own workshops to maintain them because they're so obsolete.

The newer computerised systems tend to have no moving parts and can pack more trains onto the tracks.

EdHominem• 10 years ago

What prevents new signals from being backwards compatible with the old system?

Nothing, of course, despite the naysayers. You wouldn't, initially, try to replace - just augment; record and report the inputs and be a test-bed for the eventual digital replacement. In this case, because the originals are physical switches, it'd be more like a modern-machined duplicate of the original with a micro-controller stuck on the side.

The data it would send be immediately helpful for arrival timers and such.

floatrock2• 10 years ago

The Atlantic did a great longform piece last year all about NYC's (also past-useful-life) signaling infrastructure by asking a simple question: why is it so hard to build a train arrival countdown clock?

They go into a lot of detail on how the past-useful-live infrastructure works, modern replacement technologies, and even a race between infrastructure-giant Siemans and a bunch of hackers kludging together a countdown clock:

http://www.theatlantic.com/technology/archive/2015/11/why-do...

A more interesting read than this blog post.

----------

On a different note, these "they don't make these anymore" problems make me wonder if maybe we should be pushing government to look for open-source hardware for these types of infrastructure work.

When you have equipment lifetimes of (officially) 35 years (realistically up to 100 years), can you really rely on the same Sieman's division to still be in business supporting it? In this Boston story, I wonder how much of their repair workshop is reverse-engineering some hardware whos blueprints long ago decayed in some rusting factory.

Given the right open-source hardware, could you make "they don't make these anymore" obsolete because the technology has become accessible enough that you could build it yourself? (And in the process employing "New Economy" type high-tech jobs). More importantly, should a public entity push for making it's infrastructure publicly accessible? (Pragmatically, could you find someone to build it under those terms?)

HeyLaughingBoy• 10 years ago

That particular problem is easily solved by providing all needed documentation as part of the development contract. I'm a big fan of OSS & OSH, but it doesn't add value in this case.

I'm pretty sure the problem isn't knowing how the devices work, it's the limitations of the 100 year-old technology in the face of modern expectations.

rchowe• 10 years ago

For most of the (new) equipment procurement the MBTA does now, they actually own the specifications for the equipment, which means that if they want to make more, they put out an RFP for companies to manufacture new equipment based on their spec. The downside to this is that the T likes to over-customize their orders, which drives costs up compared to buying more standard "off-the-shelf" equipment, and if no responses to the RFP come back, they have to relax some of the requirements and put it out again.

a2tech• 10 years ago

While its true that they own the specification, in practice the large company (i.e. Siemens in this example) is the only one capable of producing the product at a non-prohibitive rate.

I suspect what they'll find when they finally rip out all this equipment is that the new system will be unrepairable without huge long term contracts with the original vendor.

These old mechanical systems have their own problems, but repairability is not one of them. Almost any competent machining shop could duplicate one of the switches in the article.

Symbiote3• 10 years ago

The Atlantic article was on HN:

https://news.ycombinator.com/item?id=10566776

102 points by jsomers 114 days ago | 21 comments

acomjean4• 10 years ago

live in MA and take the T (what we call the subway/bus service here). Public transit is a major problem in this city and surrounding areas. It what make this city OK and expensive vs being great and affordable.

While the transit system here needs upgrading badly, this piece is very political. The quoting of Charles Chieppo is a give away.

"For many systems, Chieppo said, the focus is on improving stations or opening new ones rather than fixing the core infrastructure." The T is working on the first expansion of the subway rail system since the 1980s. Its going into the neighboring city of Somerville and "surprise" its over budget. They might kill the expansion with the focus on "fixing the existing" as kind of justification, instead of doing both at once? They're also reducing late night service for the extra couple hours they tacked on to friday and saturday, while proposing a fair hike. Things cost money, but people don't like to pay more to get less.

(I'm actually usually on one of the MASCO "private" buses that supplement the T for its failure to provide adequate routing to the Longwood Medical area.)

roymurdock• 10 years ago

Green line expansion (two stops) initially budgeted at $954M in June 2010.

5 years later, costs estimated at $3.09B and project is terminated.

  A Reliable Project Budget is Defined as being
  within +10% and -5% of what it would Cost
  to Design and Construct the Project

Costs went up over 300%. The level of incompetence on display here is borderline criminal. Have a friend who works at one of the big 4 who is being called in to audit this failed project. Should be interesting to get an insider perspective on where taxpayer money was spent.

http://www.mbta.com/uploadedfiles/About_the_T/Board_Meetings...

sschueller• 10 years ago

3.09B!!! That is insane. I would think after the big dig disaster they would have learned their lesson.

In Zürich they just finished a huge expansion with a complex 10km tunnel und a 3rd underground 4 track station. Project cost was 2.0B up from 1.8B. At one point a drill got stuck and they had to abandon it.

[1] https://de.wikipedia.org/wiki/Durchmesserlinie_Altstetten–Zü...

acomjean4• 10 years ago

MBTA has huge project management problems to be sure, of course it should be looked into.

But its really the only subway the city has, plus I think they're required by consent decree to build sometimg (Part of the "big dig" project (putting highway in a tunnel), another fine example of poor project budget control in the commonwealth of Massachusetts)

Symmetry• 10 years ago

I honestly think that the $954M original price tag is a bigger problem than the 3X overrun.

radd9er• 10 years ago

I believe that the housing market in Medford and union Sq for years has been inflated on the speculation of the green line exrension. For years it was taken as a given. Has the Market declined since it came to light that the estimates were off, and the project cancelled ? I imagine a lot of people will lose money over this.

yummyfajitas2• 10 years ago

Wow, $3.09B for 4.3 miles of track and 4 stations? In possibly 7 years, if everything goes according to plan?

Elsewhere in the world, the Delhi Metro Phase 2 cost $2.9B and took 3 years. 77 miles of track, 85 stations, and countdown clocks. The Bangalore Metro (phase 1) is expected to be $2.5B (including a 25% cost overrun) for 26 miles and 40 stations. It also includes countdown clocks. Barcelona spent $200M for a 5.8km rail tunnel.

Where's your money going?

enumBoss2• 10 years ago

To be fair in regards to the late night service cut, it was only a test program when they rolled it out and supposedly ridership was declining since they introduced it. Plus, it was only available on the weekends (two days a week) anyway.

giarc• 10 years ago

I work in a 50 year old hospital. There was some work on the nurse call bell system and the maintenance staff had to scour eBay for old parts to fix the outdated system. I think procurement had a fit when they saw eBay receipts show up.

acomjean4• 10 years ago

I worked on a radar project with a similar problem. The target system was a computer called HP superdomes. HP discontinued those (the PA-RISC version anyway) . I think they bought a couple spares. Ebay was mentioned as a source for parts, which doesn't always give you a good feeling.

acveilleux• 10 years ago

In a previous job at a University, we had a couple SGI big irons from the 1999-2002 era. A 64-way Origin 3800 and a 16-way Onyx 3800. Along with a similar vintage SAN and some misc bits.

When I left in 2010, eBay was the source of something like one replacement HDD per 10-14 days for the SAN (73GB dual-port fibre channel, full-height...) and the two machines (made of modular "bricks") had been whittled down to a single franken-SGI of something like 20 CPUs. Some of the older processing workload still ran on there (i.e.: the student graduated ages ago...)

A few dual 5300-series (8 cores) Xeons had replaced the SGIs in most workloads and left the old 500Mhz R14000 far in the dust. Lots more RAM too. Let's not even get started on the advantage as far as HVAC and power went.

brk• 10 years ago

FWIW, the same thing goes on in Miami. I've been in the electronics shop for Miami-Dade Transit and there are always a bunch of electronic signs and signals of different vintages under repair.

na85• 10 years ago

A lesson in bad UX, or perhaps just bad editing: Nowhere in the article is the abbreviation MBTA expanded. I shouldn't have to guess at or Google for your article's main actor.

untog• 10 years ago

WBUR is Boston's NPR radio station. It's the 21st Century equivalent of a local news piece, while they could work harder to adapt it for a global audience it isn't really in their remit to do so.

Animats• 10 years ago

For a century, replacements were not much of a problem. Most of the components were either from Union Switch and Signal, or General Railway Signal, both in the US, and they still made them. Then US&S (which used to have www.switch.com [1]) was sold to Ansaldo and then to Hitachi, and General Railway Signal was acquired by Alstom. Alstom continues to make some GRS relays.[2] But not the whole line of them. Ansaldo offers a remanufacturing service for the older components.

The New York City system uses mostly General Railway Signal components, while Boston uses US&S. NYC had to rebuild old relays after the big flood a few years ago, because all available spare stocks, both theirs and the manufacturers', had been exhausted. But they were able to get new ones from Alstom over time. Boston seems to be having more of a parts problem.

These relay-based systems are designed to fail safe, in a very literal way. "Vital" relays (the ones in safety-critical circuits) open by gravity, and the open direction is always towards a red signal. Broken wires always force things towards a safer condition. This is not a property of computer-based systems. There are modern replacements, such as Ansaldo MicroLok II, which are programmable logic controllers which emulate relay systems.[5] These, unfortunately, can be Internet-connected.[6]

Here's a simulator for a GRS relay-based signaling system, down to the relay level.[7]

[1] https://web.archive.org/web/20080103132002/http://www.switch... [2] http://alstomsignalingsolutions.com/OurProducts/WaysideProdu... [3] http://www.rrsignalpix.com/grs_relays.html [4] http://www.ansaldo-sts.com/en/about-us/ansaldo-around-world/... [5] http://www.ansaldo-sts.com/sites/ansaldosts.message-asp.com/... [6] http://www.westermo.com/web/web_en_idc_com.nsf/AllDocuments/... [7] http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interloc...

‘They Don’t Make These Anymore’: Maintaining the MBTA’s 100-Year-Old Signals

Related Stories