Blackout Bug: Boeing 737 cockpit screens go blank if lands on specific runways

https://www.theregister.co.uk/2020/01/08/boeing_737_ng_cockpit_screen_blank_bug/

by DemiGuru • 7 years ago

249 124 7 years ago

theregister.co.uk

fit2rule• 7 years ago

Oh, it is going to be fun to see them get re-certified.

Did someone forget to run the fuzzer, maybe?

mcv• 7 years ago

That is an awfully specific bug. I can see how it went undetected. But what could trigger this?

francisofascii• 7 years ago

From reading the article, it seems like it occurs when the runway is precisely running east to west and the airplane is coming in westward. And also the latitude/longitude is somehow involved. You can see one of the airports here. Notice how it is exactly horizontal, i.e parallel of latitude. https://www.google.com/maps/place/Wiley+Post-Will+Rogers+Mem...

Scoundreller3• 7 years ago

My thought is some combination of that and magnetic deviation.

Runway headings are always magnetic headings, so what the runway says and what GPS calculates (at the lowest levels) are two different things.

So maybe the issue only occurs at airports where you’re landing on 27 AND mag deviation is some specific value?

Would explain why it happens at only some US and some S. American airports:

https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/...

Someone• 7 years ago

The affected runways have fairly similar directions (between RW25 and RW29. Runway names follow compass directions. See https://en.wikipedia.org/wiki/Runway#Naming). I don’t know what the distribution of runways in the USA is, but that feels too similar to me to be a coincidence.

Even if it is, I think it wouldn’t be the whole story, as there must be zillions of such runways in the USA.

vidanay• 7 years ago

I got ya Bro.

https://trailsofwind.figures.cc/

pintxo2• 7 years ago

I guess the automated test suite just got larger. Testing all approaches on all runways on all airports worldwide...

rossdavidh• 7 years ago

I think you meant to say, "I guess the automated test suite should get larger". :) Otherwise, I am afraid your optimism might be misplaced.

pbhjpbhj• 7 years ago

Given these planes can take off and land on autopilot (I'm not sure that's true, just what I hear) then surely it's easy to run simulations of all those events; so if it was in scope for simulation it could be caught?

There's about 20k commercial airfields, if you ran 20mins of simulation for each one then you're done within the year. Presumably Boeing can run in parallel and above realtime.

Are their watchdog systems for everything, or do they primarily rely on pilots noticing problems?

Wild speculation: Perhaps it's a power issue, something about the conditions causes a power demand that exceeds specs (or makes an LED? voltage drop too low). What else links all the screens?

Piskvorrr4• 7 years ago

Actually, an autopilot landing requires CATIII ground equipment...but that could be easier to set up in a simulation.

But that is the nature of testing in multivariate conditions: e.g. if the bug triggers only when the heading is exactly 270.0000, I could see why a testing suite wouldn't catch it.

jojo20002• 7 years ago

In the automotive and aerospace world, of course tests in the real world cannot cover all cases (cost, time make it impossible).

Even most (not all, but almost none for meaningful applications) computer program cannot be proved to come to an halt [0], so complete testing is impossible by essence. We can only use more restrictive rules for programming but cannot formally guarantee anything.

As those systems are tied to the physical world, a whole lot of complexity is added by uncontrolled parameters.

Yet we love testing things. So a lot of techniques exist, such as SIL [1] and HIL [2].

So you could imagine using a real dashboard hooked up to a plane simulator. Which would enable testing the device in a wide array of conditions.

[0] https://en.wikipedia.org/wiki/Halting_problem [1] https://www.quora.com/What-is-Software-in-the-Loop-SIL [2] https://en.wikipedia.org/wiki/Hardware-in-the-loop_simulatio...

jcranmer• 7 years ago

Even most (not all, but almost none for meaningful applications) computer program cannot be proved to come to an halt

That is not true. If you can demonstrate that every loop or recursion is bounded, then the program must halt as a necessity. Loops of the form for (i = 0; i < N; i++) are trivially bounded, unless you're resetting i in the loop. If you have containers that are not being modified in the loop, the finiteness of your data structures is usually sufficient to form the lemma that the loop will terminate.

Recursive datastructures (such as cyclic graphs) are much more challenging to prove, and most challenging are fixed point algorithms (do { } while (changed);), as will be noted by them frequently being the causes of actual infinite loops in my experience. But if you had mandatory annotations to declare lemmas for termination, it is doable. With that feature, a programming environment that forbade you from writing code that couldn't be proven to terminate is probably sufficiently feasible to allow you to write large, complex applications.

mikl2• 7 years ago

I’m getting happier and happier that my usual airline (Swiss) has a mostly-Airbus fleet.

coredog64• 7 years ago

To quote Han Solo, “Don’t get cocky, kid”

Boeing and Airbus both source avionics from the same third parties. The 737NG uses Smiths, but everything else in the Boeing fleet (and all newer Airbus airframes that I’m aware of) come from Honeywell. While you can fault Boeing for not finding this, the vast majority of the fault lies with the OEM.

mikl2• 7 years ago

Even if this particular problem is not Boeing’s fault, but Smiths’, the whole 737MAX saga still puts Boeing in a pretty bad light – cutting corners on security, outsourcing their software to dubious vendors, etc.

mihaaly3• 7 years ago

Read again and get nervous. The article depicts a few Airbus mistakes.

I guess Boeing is just making more noise nowadays.

nradov• 7 years ago

That reminds me of the C-130 avionics failure when flying at negative altitude.

pintxo2• 7 years ago

Or the flight computer inverting the plane on flying over the equator (changing the sign of the latitude), luckily only in the simulator [1].

[1] https://catless.ncl.ac.uk/Risks/3.44.html

willvarfar• 7 years ago

Reminds me of a story about the Black Buck raids by Vulcan bombers during the Falklands Conflict.

Because they didn't have any maps of the southern hemisphere, they just turned a map of the Atlantic upside down.

Source: https://youtu.be/rpRtEABdxNc?t=359 (worth listening to the whole thing!)

alistairSH• 7 years ago

Why would they turn the map upside down? North is still north, even if you're below the equator.

rob74• 7 years ago

The lowest airfield in North America, for the record, is Furnace Creek Airport, Death Valley at minus 210 feet below sea level.

So, then I guess it's 210 ft _above_ sea level? (SCNR)

dsfyu404ed2• 7 years ago

They probably didn't mean to do the double negative.

I used to work at an airport that was under 20ft above sea level.

reaperducer• 7 years ago

Been there. Almost nobody uses it. But from what I read in the newspapers, the few times it does get used, it seems to have more than its share of crashes.

izzydata4• 7 years ago

Apparently another plane went down in Ukraine according to this article. That seems like a much bigger story. Maybe I happened to miss that one yesterday?

Edit: A Ukrainian airline in Iran.

DemiGuru• 7 years ago

It’s a different model it’s 737-800. Different plane model while still within the 737 family.

izzydata4• 7 years ago

When I say "another" I was referring to that previous incident where a passenger plane was shot down in Ukraine and not the Max plane problem. Any passenger plane crashing seems like a significant story. I hope we aren't numb to that already.

Kipters• 7 years ago

737-800 is part of the 737 NG family

zymhan• 7 years ago

Both the plane that crashed, and the model referenced in this article, are the same family of 737.

codazoda• 7 years ago

Thanks for mentioning it; I hadn't seen it elsewhere.

Looks like it's being blamed on a mechanical failure at the moment. Could be a big story.

volkl48• 7 years ago

A skeptic might suggest that a major plane crash (a very uncommon event), occurring in Iran at the moment of maximum military tension (after Iran had launched missiles at US bases and could be expecting a response), is a pretty unlikely coincidence more likely explained by an accidental shootdown or the like.

TomMckenny• 7 years ago

A Ukrainian airline but it occurred in Tehran, Iran a few hours after the missile attack. It was taking off, not landing.

artemonster• 7 years ago

Here was already a link on HN and a discussion, which just vanished for some reason (most likely removed by invisible admins). Hate this on HN.

salawat• 7 years ago

I believe dang pruned it due to not enough information being out in order to foster anything beyond the wildest of speculations. Such as people thinking it was another MAX 8 at some points.

While I don't agree with the course of action; I can see the temptation to doing so.

Often at least waiting til the CVR and black box data is found is necessary to really know for sure what ended up transpiring. Though there were apparently some mobile phone videos of things of dubious provenance/quality in the original threads that I was aware of and are still searchable. Of which there are at least 2 currently via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

spamizbad2• 7 years ago

A 737 malfunctioning and inadvertently setting off WW3 is very on-brand for Boeing right now.

briandear• 7 years ago

Except the airplane was riddled with holes indicative of air defense guns shooting the airplane down.

squeaky-clean• 7 years ago

It could be a big story, but unless it's a discussion on a technical failure, it doesn't really belong on HN. Ever since the last big election cycle people have been more strict about not allowing political and world-news posts that aren't related to tech or hacking culture.

izzydata4• 7 years ago

In the past big news often made its way to HN. I don't really care about trivial world news so I don't search tons of alternative news sources.

If a nuke was launched and exploded somewhere I'd still expect it to end up on HN despite not being tech news. So the requirement of significance for unrelated news to make it here has gone way up.

taylodl• 7 years ago

I'd like to learn how Boeing's software delivery practice has changed. Sounds like there may be some lessons learned we can all learn from.

Frost1x• 7 years ago

I would hazard a guess: their process became "agile."

eskaytwo• 7 years ago

You should clarify what you mean by “agile”. It means so many different things in different contexts.

A large part of this is likely the cost-driven pressure that lost the CEO his job. Agile can be abused certainly.

mihaaly3• 7 years ago

I'd like to learn how Boeing's software delivery practice has changed

To something like this perhaps?

- Make it cheap. Now!

cryptonector2• 7 years ago

Software ain't cheap.

robk• 7 years ago

Outsourced to India!

Seenso• 7 years ago

I'd like to learn how Boeing's software delivery practice has changed. Sounds like there may be some lessons learned we can all learn from.

Does Boeing build its own cockpit displays and avionics? I was under the impression that that stuff was developed by companies like Rockwell Collins and Honeywell.

jperras2• 7 years ago

Is someone dividing by zero somewhere?

"All six display units (DUs) blanked with a selected instrument approach to a runway with a 270-degree true heading, and all six DUs stayed blank until a different runway was selected."

cos(270°) = 0

gpm2• 7 years ago

But cos(90°) = 0 too, unless they thought to check for one case but they did check for the other, or something more complex is happening.

jperras2• 7 years ago

I mean, it's probably more complex than what I postulated.

It was just the first thing that came to mind, since 270° is zero in a few trigonometric functions/identities.

FriendlyNormie• 7 years ago

angle = arctan(dy/dx) where dx=0.

I guarantee this is the mistake they made.

In some languages you can use Math.atan2(dy,dx) to easily avoid this.

phkahler2• 7 years ago

> I guarantee this is the mistake they made.

No. East-West runways are very common.

If I may speculate... the blanking of 5 displays is probably not a bug. Maybe it's a (misguided) feature meant to keep people from landing at certain restricted airports and they coded the logic incorrectly?

dreamcompiler2• 7 years ago

In any code base an arctan function called with a single argument should be considered a bug until proven otherwise.

jffhn• 7 years ago

cos(270°) = 0 (parent)

I don't know of a language (but I don't know many) which trigonometric functions use degrees and not radians. In radians and finite precision, "cos(x) = 0" is false for all x (unless your cos is wrong), thanks to Pi being irrational (for example, cos(Pi_in_f64) is about 6.123233995736766E-17). You have "sin(0) = 0" though.

angle = arctan(dy/dx) where dx=0.

If dy != 0, dy/0.0 is +-Infinity (if following IEEE754), so angle is +-Pi/2.

Coordinates/physics is among the things I see developers get wrong the most (even excluding units issues), along with threading/locks and instances_ownership/pooling.

Data races and deadlocks issues can be rare, and leaks get fixed by a restart, but these coordinates issues can introduce constant errors and biases often overlooked by tests (in addition to almost systematically unhandled cornercases like these).

anilakar• 7 years ago

Arctan, yes; division by zero, no.

Boeing engineers probably were skilled enough to account for zero division errors in their software, considering that the software works for eastward-pointing reciprocals. It's the sign function returning an unexpected value for zero and causing problems down the pipeline.

JoeAltmaier• 7 years ago

Points out how you should use vector coordinates in navigation, instead of trig. All those pesky divides happen in one place, are obvious, and handling them is trivial.

malaya_zemlya• 7 years ago

That would also cause a divide by zero at a 90 degree heading

Piskvorrr4• 7 years ago

Which are the exact same runways when approached from the opposite end.

SamReidHughes• 7 years ago

Wind generally goes west to east though.

dreamcompiler2• 7 years ago

Likewise cos 90° (all runways designated 27 are also designated 9) but those haven't reported these problems, have they?

slim• 7 years ago

Next week un elreg : HN fixes 20 years old bug in Boeing 737

Scoundreller3• 7 years ago

Edit: the airports noted aren’t along the same magnetic declination, so maybe not that.

Given that this applies to a few US airports and some S. American ones, maybe it has to do with magnetic declination calculations gone awry.

Maybe only airports along one of these lines gets impacted:

https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/...

And magnetic declination changes all the time. Maybe different systems are using different sources of truth?

Maybe when they did initial testing, they included some airports at the extremes, but when the declinations got updated, they were no longer good test cases?

ravenstine• 7 years ago

Isn't this something that would be discovered through flight simulation? Does Boeing run their software and hardware through automated testing?

_bxg1• 7 years ago

I for one will never ride on a 737 again, even if they manage to get them back off the ground. For every problem that's come to light there are probably two more than haven't been discovered (or disclosed).

markus92• 7 years ago

This bug affects the Boeing 737 NG, the '97 model that everyone and their model still uses. The article is not about the grounded 737 MAX model.

johannes1234321• 7 years ago

Fact is: there are around 10,000 737s around, each if them starts and lands multiple times a day, by far mostly without problems. Risk is similar to you slipping while getting out of bed in the morning or being overrun while crossing a street. (Excluding Max versions, which are grounded)

ta9999991713• 7 years ago

Maybe if I have a heart attack or seizure during those activities...

I have 0 past history of slipping. Also, can't avoid walking.

I can avoid flying on something I know could be fucked.

mcsoft2• 7 years ago

While it is easy to blame Boeing these days, I suspect we have survivorship bias here. Dig into Airbus software and all sorts of similar scary bugs will pop up. For instance, one may remember inconsistencies between airspeed measurements that resulted in the loss of AF 447 on June 1, 2009. Overall, it's great that at least some airliner's closed software finally gets more eyes looking at it.

umvi• 7 years ago

I seem to recall Airbus having different engineering teams across EU using different versions of the CAD tool and then when they went to build the plane the wires were too short because one version of the CAD tool accounted for the radius of wires turning a corner while the other didn't.

https://www.nytimes.com/2006/12/11/business/worldbusiness/11...

zaphirplane• 7 years ago

Had only the version that didn’t account for turning a corner used, would things have worked out ? I don’t follow

kayfox• 7 years ago

Boeing had an approach to this issue: Your either running the correct version of the CAD software, or you can't check in designs. This also meant that every time it was updated there was a rush to get engineers who's update did not work back up and working. Fortunately those were a small minority and it was usually disk space issues.

t0mas882• 7 years ago

Frozen pitot tubes causing an airspeed disagree is something we train for. The issue there isn't software, it's mechanical, and what the Airbus did is exactly what it's supposed to do according to the manual.

The problem there was the junior co-pilot responding all wrong and the other pilot not noticing because Airbus sticks don't move together. The captain woke up, came to check and realized what was wrong but did so too late to fix it.

mcsoft2• 7 years ago

Agree, though the investigators also pointed to the lack of a clear display of the airspeed inconsistencies even though the computers had identified them. This sounds more like a UX problem rather than a bug, but still the software issue in the edge case.

codesforhugs• 7 years ago

I've wondered if someone in Airbus is currently investigating how they would fare under similar scrutiny, but I suspect the answer is no.

dsfyu404ed2• 7 years ago

If they're any think like every BigCo I've worked at then I'm sure the QA people have already raised hell and been ignored.

Glawen• 7 years ago

It is certain that under close scrutiny one can find bugs as well in Airbus airplanes. Bugs are inherent to software production, all processes are just there to minimizes the risks posed by critical bugs.

However, as airbus released the first fly by wire airliner (a320), they have more experience in software over boeing. Boeing was always afraid to give too much power to the software, a boeing plane has more mechanical backup than an airbus.

I sincerely hope that they are not fucking around by following the agile trend from the sw industry, and instead stay focused on old and proven engineering development practices.

I must say that this whole boeing show makes me scared to hop in a boeing, and is another reason why i will now seek flight in a a320 instead of a 737. The first reason is simply comfort, the 737 cabin size is really too small.

Edit : the a350 had a bug where the pilots needed to switch off the plane every 149h, and recently they discovered a bug in the a220 P&W engine (not airbus developped) leading to engine shuting down in flight. They also found a kind of mcas like bug the a321neo, but I think it happens in extreme manoeuver which can be only replicated in simulators

o-__-o2• 7 years ago

This anecdote was valid until the MAX:

A Boeing with an engine fire will pop up a dialog saying “flight computer detected a possible engine fire, would you like to shutdown?”. An Airbus with an engine fire will pop a dialog saying “flight computer detected an engine fire, shutting down engines”.

The joke being Boeing pilots have a chance of saving the plane in the event it’s not an engine fire, Airbus will just do what it thinks is best and let the pilot sort out the aftermath. MCAS changed all that

awinter-py• 7 years ago

The first F-22 they delivered to japan had serious trouble when it crossed the international date line

Sounds like nobody was hurt & they didn't lose any planes, but there was some period of time where the onboard computers were not good

https://www.defenseindustrydaily.com/f22-squadron-shot-down-...

Yet another reason to exercise low-level math functions with a wide range of inputs -- maybe not on every run, but at least in pre-release integration tests

There are only 4 billion floats

https://randomascii.wordpress.com/2014/01/27/theres-only-fou...

sudosysgen• 7 years ago

F-22 weren't exported, it was probably just at Okinawa

vmchale• 7 years ago

don't land on those runways!

Blackout Bug: Boeing 737 cockpit screens go blank if lands on specific runways

Related Stories