United Airlines System-Wide Computer Problem

http://www.flyertalk.com/forum/united-airlines-mileageplus/1693331-shares-down.html

by SanjeevSharma • 11 years ago

92 116 11 years ago

danso• 11 years ago

Very little news about this on Google News, but heard over the local Chicago ABC affiliate that the FAA attributed this to an "automation error"

Edit: And its Twitter account has been relatively inactive, with more than 30 minutes since the last reply-to or general tweet...presumably a lot complaining tweets have come in in the last 30 minutes https://twitter.com/united/with_replies

lectrick3• 11 years ago

Just a minute ago they tweeted about it: https://twitter.com/united/status/618769524544942081

ceejayoz• 11 years ago

Took them until just now to go "oh, we should probably post something that isn't a @reply".

https://twitter.com/united/status/618777538865799168

caractacus• 11 years ago

Not a word about it on United's web site. Flight status page doesn't load correctly. "Today's Operations" gives an error message. United's Twitter is silent.

Meanwhile news articles and twitter complaints abound. http://mashable.com/2015/07/08/united-computer-problems-flig...

DangerousPie• 11 years ago

Official status page: http://www.fly.faa.gov/ois/jsp/summary_sys.jsp

Currently says:

    ATCSCC  ADVZY 027 DCC 07/08/15 UAL GROUND STOP REVISION
    DESTINATION AIRPORT: ALL AIRPORTS
    FACILITIES INCLUDED: ALL FACILITIES
    GROUND STOP PERIOD: 08/1200Z - 08/1315Z
    REASON: USER REQUEST AUTOMATION ISSUES

kieranelby• 11 years ago

Good to see the Sun Microsystems favicon on the FAA status page - not seen one of those for a while!

fnordfnordfnord3• 11 years ago

Last one I noticed was the FCC's comment system.

gesman• 11 years ago

Interesting bits:

"Departing DEN; taxied and then returned to gate. Pilot says nationwide failure of "three or four" computer systems. Only information from airport staff is that since the computers are down UAL can't book pax onto any other airline ..."

"Systemwide Ground Stop posted at FAA: Due to USER REQUEST DUE TO AUTOMATION ISSUES. UAL AND SUBS ONLY., departure traffic destined to ALL airport will not be allowed to depart until at or after 13:15 UTC."

scrumper2• 11 years ago

That's twice in a short time now - 'coincidence' on Ian Fleming's scale. A third time is enemy action. Airlines do seem like a pretty juicy target for cyber war operations - you can cause a gigantic amount of disruption with a successful attack on a single system.

eli• 11 years ago

That seems like quite a leap given how often complex computer systems fail without any malicious act.

mnw21cam2• 11 years ago

Indeed. Never attribute to malice that which is adequately explained by stupidity. https://en.wikipedia.org/wiki/Hanlon's_razor

dlgeek2• 11 years ago

Or, you know, the load is increasing over time, and the system is failing to scale to it, and so will fail more and more frequently as the load continues to increase?

joezydeco2• 11 years ago

The amount of daily UAL passenger seats doesn't change or grow radically over weeks or months. What is the scale-up here? Everyone using their phones to check rez/status/boarding passes?

scrumper2• 11 years ago

I don't see any convincing rebuttal. Why are airline systems not a juicy target? Why would a successful attack on the system not cause gigantic disruption? Why have these long-running, stable systems only recently begun failing so severely?

Yes, one is wise to attribute to incompetence over malice, but these systems are demonstrably not run by incompetents: they've been in operation for decades.

johnrydell• 11 years ago

Let's just be thankful that we haven't had a major air-based catastrophe due to an outage or hacking!

userbinator2• 11 years ago

Planes with pilots in them are mostly autonomous and can avoid each other as well as the ground.

https://en.wikipedia.org/wiki/Ground_proximity_warning_syste...

https://en.wikipedia.org/wiki/Traffic_Collision_Avoidance_Sy...

https://en.wikipedia.org/wiki/%C3%9Cberlingen_mid-air_collis...

chinathrow3• 11 years ago

Yes, mostly.

https://en.wikipedia.org/wiki/Vitaly_Kaloyev

JHof• 11 years ago

Automated, yes. Autonomous, no. The plane voices an alert, but the pilots still have to react to it.

lectrick3• 11 years ago

I have a nagging (intuition? feeling?) that software safety/reliability/security needs are going to explode soon (because unreliabilities multiply in non-resilient systems interacting with each other) and that these are simply foreshocks.

(yeah, I know security is already a huge deal, but as we come to trust software systems more and more, the safety/reliability factor will come more into play)

EDIT: This is also part of the reason I've been learning Elixir (http://elixir-lang.org/) since it's based on the highly-resilient Erlang and is designed to embrace failure. This was also informed by me reading Nassim Taleb's book "Antifragile" as well as "Thinking in Systems: A Primer" by the (late) Donella Meadows.

yAnonymous• 11 years ago

I doubt it. Nobody wants to pay for that.

DougWebb• 11 years ago

You might be right, unfortunately. If it's cheaper to buy insurance that will cover the (expected) losses caused by outages, most organizations will choose to do that instead of making the software more failure-resistant. The problem is that insurance only works well for isolated incidents, but a software failure can cause a cascading failure with a huge impact. Insurance companies generally aren't prepared for that and don't have the resources to pay out to everyone.

lectrick3• 11 years ago

I think we just need to build smarter. I've become disillusioned with the "runaway state" problem (as well as spaghetti-dependency problems) in OO languages which contributes to bugs and general nondeterministic behavior as well as making long term maintenance difficult, and at the same time I've become enamored of unit test suites and functional immutable languages like Elixir as well as static code analysis tools (I'm still coming around to Haskell-esque typing, but I generally think it's a good idea to write "potentially provably correct" code that has parts which provably have no side effects).

HeyLaughingBoy• 11 years ago

Nancy Leveson's been saying that for decades: http://www.amazon.com/Safeware-Computers-Nancy-G-Leveson/dp/...

donkeyd• 11 years ago

Schuberg Philis (https://www.schubergphilis.com/) in the Netherlands has been selling 100% functional up-time for a while now. They've set their entire business model and management structure up to support this.

Doesn't come cheap though.

Gravityloss2• 11 years ago

So obviously the "move fast and break things" philosophy is meant for some fun web applications. But what's the equivalent "modern best practice" for systems that are much more weighed towards stability and resilience as opposed to new features?

calinet62• 11 years ago

Systemic quality focus would be a good start. Deming-based management philosophy driving a systems-oriented model.

This actually applies to all businesses; speed improves, market fit improves (quality is just "what is good and valuable" after all), employee happiness improves (exponential gains from that alone). The effects are systemically positive.

But, no one cares, and the belief that we have to crack the whip to get people to make things faster, and we should reward the good people and punish the bad ones—will continue on as pure religious fallacy resulting in the failure or constant-operation-at-the-edge-of-failure of everything involving more than five people, probably until the end of the human race.

chinathrow3• 11 years ago

Nice reminder about a "glitch" happening in one of the datacenters use at the airline I used to work.

Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.

ianhawes• 11 years ago

I remember one of the LiveJournal outages (IIRC) was caused by a dude in the datacenter thinking that the emergency power off button opened one of the doors.

misterbwong• 11 years ago

Had a similar experience at an old company. One of the light switches in server room was placed next to a switch connected to the UPS of certain servers (I have no idea how/why this even came about).

Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.

js2• 11 years ago

At former managed-hosting startup, we would deploy a pair of EMC symmetrixes into our DC cage as storage for our DB tier, with volumes mirrored over the pair. While equipment was being moved into the cage one day, someone accidentally banged the power-switch protection plate on one of the EMCs. Fortunately the power switch had to be held depressed for several seconds to turn off the equipment. Unfortunately, the protection plate itself got jammed in against power switch keeping it depressed. Hoisted by its own petard, as it were.

Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh.

When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.

coob• 11 years ago

This is what happens when Homer Simpson works at your datacentre.

bitwize• 11 years ago

A data center where I work had a naked big red switch without a protection plate. Until I flagged up this lack and let the IT people know about it, it was a disaster waiting to happen.

There's a plate on the switch now, praise Eris.

geek_slop• 11 years ago

Similar situation here. A big red button on the wall in the basement data center - nobody knew what it was for and were afraid to turn it off. One day a night operator decided he was going to flip the switch and see what happened. It cut power to the entire datacenter, including the elevator systems. We spent a few days getting all the systems back online.

Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.

philipw• 11 years ago

I've seen many places where "the solution" to the unprotected EPO button was just a simple plastic cup!

TWAndrews• 11 years ago

I was able to get checked in via the United android App around 9am ET.

raus22• 11 years ago

Off-topic: Please use ISO 8601 format(YYYY-MM-DD) for dates in titles. The US date format hurts my poor logical soul.

https://en.wikipedia.org/wiki/ISO_8601

interdrift• 11 years ago

It's still month 7 so it's hard to be confused. PS. From Europe.

notNow• 11 years ago

But a month from now, some people will be confused for sure. So, it's more future-proof to settle for ISO or alphanumeric format to avoid confusion and miscommunication.

mnw21cam2• 11 years ago

Or just use words (8th July 2015). But yes, YYYY-MM-DD actually sorts correctly.

r00fus• 11 years ago

ISO all the way. Words have issues for different month languages (e.g. Auot vs. August), but otherwise is more elegant (no delimiter needed, e.g. 08JUL15)

anc84• 11 years ago

It's called "8. Juli 2015" where I live. Use numbers, please.

sk5t• 11 years ago

See also: https://xkcd.com/1179/

leopoldo• 11 years ago

I love how they write the dates when you hover over the image. XKCD.

grhmc2• 11 years ago

ISO8601? Easy. 2015-W283. Or maybe (?) 2015188. ISO8601 will hurt your logical soul, don't stare too deeply into its eyes.

krschultz• 11 years ago

Or you know, 2015-07-08, which is perfectly readable to me.

snarfy• 11 years ago

There's little endian, big endian, and the US date format, which I like to call middle endian.

Lancey• 11 years ago

Wouldn't the PC term be middle Native American?

chinathrow3• 11 years ago

Well, to attribute that to SHARES, we should go with big endian ;)

jsingleton• 11 years ago

For future proofing please use 5 digit years (YYYYY-MM-DD) e.g. 02015-07-08. :P

OedipusRex• 11 years ago

Stardate 1673.85

amelius• 11 years ago

Or use JSON: {"year": 2015, "month":7, "day":8}

juliangregorian2• 11 years ago

Except that month starts at 0.

VLM• 11 years ago

Are the airlines using systems that modern?

I was guessing something a bit more

01 OUTAGE-DATE-DATA.

     05  OUTAGE-DATE.

          10  OUTAGE-YEAR           PIC 9(04).

	  10  OUTAGE-MONTH          PIC 9(02).

	  10  OUTAGE-DAY            PIC 9(02).

     05  OUTAGE-TIME.

	  10  OUTAGE-HOURS          PIC 9(02).

	  10  OUTAGE-MINUTE         PIC 9(02).

	  10  OUTAGE-SECOND         PIC 9(02).

	  10  OUTAGE-MILLISECONDS   PIC 9(02).

     05  OUTAGE-DIFF-FROM-GMT       PIC S9(04).

MOVE FUNCTION CURRENT-DATE TO OUTAGE-DATE-DATA

I always kinda liked the meme from that language of "variable as picture". Calling something a picture has inherent and obvious implications WRT call by value vs call by name.

On the other hand intrinsic functions like FUNCTION CURRENT-DATE are not as logically amusing.

exelius• 11 years ago

We'll get around to it just as soon as we get around to using the metric system and providing universal health care.

drzaiusapelord3• 11 years ago

Its sad that this is the top comment. When I visit European sites I don't demand they change things just for me. Maybe be more of a gracious visitor?

briandear3• 11 years ago

Why is this being down voted? It is true. When British sites spell things with superfluous letters (colour, for instance,) should HN readers demand that they change it to the more efficient spelling? American English is spoken natively by more people in the world than British English, so therefore must we banish British English from these pages? Of course not. We could argue that the US has the largest number of visitors to HN than any other country. HN is an American site. YC is an American organization. Thus, those not comfortable with American conventions really ought to get over it.

The United States has the world's largest economy, according to the IMF, World Bank and UN. So obviously the US is doing something right. Perhaps the rest of the world ought to adopt American conventions. I say that in jest, but the point is that criticizing the American way of doing things is a popular sport, yet at the end of the day, the American system has resulted in an economic output greater than Germany, the UK, France and Italy combined. The EU has a per-capita GDP of $36,779 and the US is at $54,601. So apparently something is working with the American way of doing things. Just because "the rest of the world" does it doesn't make it better. This whole idea that the American way of doing things is somehow inferior is just as ridiculous as the British/French rivalry.

The fact that the parent comment is the top comment is ridiculous. It's a petty thing about which to complain and adds nothing to the discussion. In fact, the parent comment ought to be down voted for being absolutely irrelevant to the posting in question.

mark-r2• 11 years ago

I tried to get a former employer to switch to ISO format, since they had offices both in the US and Ireland. There was too much pushback so I didn't succeed, but I at least got them close: 2015-Jul-08.

peterjmag• 11 years ago

Ouch. And only a month after another major systems outage: http://www.wired.com/2015/06/united-flights-grounded-mysteri...

jonawesomegreen• 11 years ago

I bet this is an issue with an old mainframe used somewhere in the booking system, something that has worked well but is difficult to fix when things go wrong.

I think there is / will be a lot of money to be made trying to solve the problem of software security and reliability. This is obviously an extremely difficult problem, however the number of ancient systems that we currently have interconnected I think more large scale outages like this are inevitable.

caminante• 11 years ago

I submit there's more money to be [saved] from not replacing old systems.

Case in point, take this HackerNews fan favorite[1] about a school district using an "ancient system" and avoiding $M's in incremental costs.

[1] https://news.ycombinator.com/item?id=9705830

edit: As mentioned in [1], I assume that at least someone is aware of the cost-benefit of potential projects; in turn, I assume that someone would've pulled the trigger if the $'s make sense.

igrekel• 11 years ago

Truth is that most reservation systems are built on TPF and it isn't really easy to replace.

waqf• 11 years ago

just in case we have readers to whom TPF isn't a household name: https://en.wikipedia.org/wiki/Transaction_Processing_Facilit...

imgabe4• 11 years ago

United is a mess. I had the misfortune of flying with them a couple months ago. I ended up in a city a 2 hour drive from my actual destination and had to rent a car on my own to get to where I was going.

That was the worst of it, but almost every flight I saw on the way (both ones I was on and other flights at nearby gates) was delayed or overbooked or otherwise messed up in some way.

smackfu• 11 years ago

Was there a storm or something? Usually there is some motivating reason why flights are screwed up.

imgabe4• 11 years ago

Not that I know of. My particular flight was delayed because they had to replace a piece of the navigation equipment on the plane. This delay caused me to miss a connecting flight and that's how I ended up in an entirely different city.

boken• 11 years ago

If I ever listened to horror stories like these, there wouldn't be an airline left I would fly with. This is textbook; replace United with Delta, Southwest, Air Canada, etc., at will. The only company I've used that I haven't heard exactly this type of complaint about is Widerøe, a tiny regional line in Norway with a fleet of prop planes. This is unfortunate, as I live in Pennsylvania and get motion sickness on those little aircraft. And I imagine that if I spoke Norwegian—it seemed to me that while many Norwegians speak English, they don't do much complaining in it when the native language would do—I wouldn't even have Widerøe left.

imgabe4• 11 years ago

I've been flying 1-2 times a month for the past year or so, mostly on US Air or American (same thing now) and I never saw anything approaching the level of problems United had on one trip.

Obviously, with travel sometimes things go wrong, but the quantity and severity of things going wrong at United makes me think there's something off about the way they're running their airline. For instance, the two times this year that they've had to ground all flights.

driverdan• 11 years ago

Overall JetBlue and Virgin have been great. AA and US Air have been good. United is the worst.

anon8418• 11 years ago

It's also worth remembering that quality of service depends on a host of variables, including departure/arrival cities (major hubs are better than regional airports), routes, and times (morning flights tend to be less delayed, Thursday afternoon flying always a bit of a shitshow), etc.

I fly out of UA hubs frequently and have had nothing but excellent service from them this year (over 50 segments flown this year, ~60K miles).

Definitely helps to have status too...

Personally, I consider SW to the be shittiest of them all. I hate having to fight for a seat...

Lastly, use google.com/flights by far my favorite booking tool now.

vaadu• 11 years ago

How do you know it's a glitch(whatever that means) and not a big problem such as a data center outage or hacker caused?

oaktowner• 11 years ago

Yeah, "glitch" seems to imply something minor, while this seems anything but.

aaronkrolik• 11 years ago

Is anyone familiar with the UA tech stack? I'd be curious to see what they're running.

imroot• 11 years ago

They're running SHARES internally; most everything else is specific applications with hooks into shares for data processing and return.

coldcode• 11 years ago

Lol HP system. I used to work for SABRE, its far more dependable despite its antiquity. HP is a marginal player in this market which is dominated by SABRE and Amadeus.

jaybna• 11 years ago

I knew someone at United that once offered to give me a tour of one of their data centers: "It is like a computer museum - we have one of everything." Hard to imagine that they would have problems as a result. United is a really, really bad airline.

Sukotto• 11 years ago

What does "all airports" mean? Global or just US domestic?

fnordfnordfnord3• 11 years ago

US domestic would be my guess. ie: All airports under FAA authority.

kendallpark2• 11 years ago

Was reading this on HN and heard it on NPR simultaneously.

I have a sneaking suspicion that booking systems for most airlines run atop legacyware. It just seems like the type of thing that would've been put in place long ago and then be very expensive to migrate/updgrade.

joezydeco2• 11 years ago

Oh, it's not a suspicion.

http://www.pacbiztimes.com/2012/04/06/united-takes-a-step-ba...

kendallpark2• 11 years ago

The biggest problem, one that would drive any tech-savvy user crazy, is that United junked an award-winning, state-of-the-art reservation system and adopted the Continental Airlines model based on older technology known as System One.

Well, that answers that.

DannyBee• 11 years ago

Considering just yesterday their flight system didn't believe they flew from SFO for 2+ hours in the morning (I have screenshots), i'm not all that shocked.

carnesen• 11 years ago

I worked on United's computer systems for a year (never that one though), and so I get nervous when I see a headline like that. True story: one of their systems still runs on a mainframe that has 9 bits in a byte!

themartorana• 11 years ago

I did not know that was ever a thing. I knew that bit counts had been lower - 5 or 6 - but assumed once we hit 8, the whole power-of-2 thing was too comfortable in a binary system to ever be anything besides a power of 2 again - and sure enough, we get 16 bit, 32 bit, and 64 bit systems, and double-byte and triple-byte char sets, etc. Need more space? Take another byte.

Was the 9th bit special? Or just a standard bit in the byte?

Lx1oG-AWb6h_ZG0• 11 years ago

Someone probably read too much Iain M. Banks and decided to adopt Marain's base-9 system :)

If I'm not mistaken, some old consoles used a 9-bit RGB encoding, so this could theoretically help there. Minecraft also uses a 9-bit system for Redstone, afaik.

chiph• 11 years ago

The PowerPC based AS/400 systems were technically 65-bit machines, as the extra bit was a privilege flag to separate system code from user code. Hardware enforced security - one reason why it is such a stable machine.

Someone• 11 years ago

36 bit was popular for scientific computing because 35 gives you a sign bit and 10 decimal digits (yes, that's weird, but that's the argument I read everywhere, including at https://en.m.wikipedia.org/wiki/36-bit. 35 likely was skipped because it its only divisors are 5 and 7, limiting instructions to 7 bits even then was felt to be too restricting. For some architectures, the DoD had a say in this, too. See https://en.m.wikipedia.org/wiki/Unisys_2200_Series_system_ar...)

36 bits got us the 6-bit character (10 digits, 26 letters, and punctuation) with six characters in a word. Because of that, some OSes had six-character file names.

If you want to get upper- and lowercase, you need more than 6 bits. 9 is the smallest divisor larger than 6 of 36, so nine-bit characters made sense.

On such systems, file names could still use 6-bit characters, while applications used 9-bit ones. Also, some instructions could work on words, half words, quarter words, or sixth words.

wmf• 11 years ago

If hex hadn't been invented yet, 9 bits works better with octal. I don't know if that was a factor.

temuze• 11 years ago

Oh god stories like this scare me.

One of my friends used to intern for a very large company that maintained software for flight control towers. His entire summer was spent writing bash tests for these old Fortran apps that kept planes from running into each other. Most of the mainframes still had tape.

That's the code that keeps track of our planes.

Xylakant• 11 years ago

I must admit I don't take that as a negative thing. Code that's so old had a lot of time for corner cases to be ironed out. If it's maintained properly it's probably less buggy than any rewrite, even though fortran is less enticing than rust as a language.

StillBored• 11 years ago

Sabre, still runs all their reservations through ztpf... If you know anyone there ask them about how they get the data off the mainframe. The story I heard sounds like the process was designed by Rube Goldberg.

bst287• 11 years ago

Currently in the air on a United flight, EWR -> SFO. Took off at 7:30ish. No mention of this in the airport or on the plane. Yikes

pavel_lishin3• 11 years ago

"Automation error" rings a funny bell in my head, since I'm currently re-reading "A Fire Upon The Deep".

rwestergren• 11 years ago

Perhaps someone wasn't following the bug bounty rules?

United Airlines System-Wide Computer Problem

Related Stories