Triple-Triple Redundant 777 Primary Flight Computer (1996) [pdf]

http://www.citemaster.net/get/db3a81c6-548e-11e5-9d2e-00163e009cc7/R8.pdf

by aaronbrethorst • 11 years ago

46 19 11 years ago

donkeyd• 11 years ago

It was once explained to me that the development of the flight computer software is also done by 3 completely separate teams. Because of this, the chance that the same bug occurs in all 3 systems is incredibly low.

sitkack3• 11 years ago

Faults in human cognition aren't randomly distributed. While n versioning helps with these issues, it doesn't prevent them to the level one would assume. Nor does it correct for systemic flaws in the specification.

ucaetano3• 11 years ago

That's why I only use software coded by at least 3 different species, two of them being not-earth-based and one not-carbon-based :)

briansmith• 11 years ago

In Firefox 4, there was a need to implement the HKDF crypto algorithm in a variety of code bases: Java, Javascript, C using OpenSSL, C using NSS. One team coded three implementations. Later, another team coded an implementation. But, that last implementation didn't interop with the first three. Of course, the two teams disagreed about which was wrong, each believing that they were right and the other team was wrong.

Surely if three different implementations in three different languages agree, then the problem is with the fourth, disagreeing implementation! However, the second team annoyingly pointed out that they had automated tests that covered the official test vectors. Well, maybe the tests were coded wrongly too! Well, could the first team implement the tests that include the official test vectors? What a waste of time! There are automated tests that check that the first three implementations all calculate the same results for many random inputs!

It turns out that the very first implementation, in Javascript, had a bug, and that bug was transliterated into two other implementations. But since the second team hated reading Javascript code, and because that team consisted 100% of annoying (but good-looking) pedants, they implemented the algorithm by reading the spec. instead of by copying the first implementation, and so they got it right.

Incidentally, the bug was that the key and value in an HMAC calculation were swapped. HKDF uses HMAC in a counterintuitive way, where the key isn't (necessarily) a secret, but the value is.

samlittlewood• 11 years ago

From a quick scan - each system also has a different sort of processor: AMD 29K, Motorola 68040, and Intel 80486

DiabloD3• 11 years ago

Now that is impressive.

yuvadam• 11 years ago

Possibly in order to use three different architectures to avoid any nasty arch-specific bugs

spectralblu• 11 years ago

I believe these are requirements, that each of the separate teams working on these projects have to use wholly different suppliers to also minimize the chance that the same external factors negatively influence the system.

ejdyksen• 11 years ago

Also, different compilers:

Each PFC channel contains three dissimilar processor lanes, and software from Ada source code using three different Ada compilers to provide triple dissimilarity.

johnm1019• 11 years ago

This is a great reminder that some things are, in fact, complicated. Furthermore, smart people, good process, and hard work can conquer this.

Lately I've been flying a lot and even now, after hundreds of flights, each time the plane takes off I think about how amazing it is that this whole air-travel system works so reliably and safely. Kudos to Boeing and Airbus engineers.

I've worked in nuclear energy, surgical medical imaging devices, and in the auto industry - and it really gets me in a tizzy when a fresh "agile MBA manager" expects a "quick hack" solution to some of these problems. There just isn't always one when quality/reliability are requirements, not nice-to-haves.

ucaetano3• 11 years ago

Nothing like a good "engineers are gods and MBAs are dumb" post to start the day.

Sounds like you company is clearly hiring the wrong MBAs, maybe the problem is your company :)

peterwwillis2• 11 years ago

Sounds like you haven't worked in nuclear energy, surgical medical imaging devices or the auto industry.

Agile is designed to get a product to a customer, rapidly and adaptively. But this is mostly useful for software that stands alone; you can't always rapidly adapt microcontrollers, bone drill burrs, heavy turbines and optical sensors. Even if you could make a 'quick hack' to one, there's 50 standards to reassess, and if one fails, you have to go back and re-do your change.

The OP is right. There are problems that require longer term, complicated solutions that have to be designed first, approved second, and implemented third. An agile MBA is not dumb; they just don't all work on the same problems.

sitkack3• 11 years ago

This field is old and interesting

http://www.inf.pucrs.br/~zorzo/cs/n-versionprogramming.pdf

https://en.wikipedia.org/wiki/N-version_programming

This is an excellent overview http://www.adelard.com/papers/divchap.pdf in regards to https://appsrv.cse.cuhk.edu.hk/~lyu/book/sft/

rplst8• 11 years ago

I worked some of the follow on projects to the 777 that involved the next generation 737. A lot of people may not realize this, but much of the software and avionics that goes into Boeing aircraft is designed, built, and manufactured by other smaller third parties. The amount of work that goes into those embedded systems is truly impressive. Additionally, the rigor with which it is built is exemplary. Software standards for commercial aviation are extremely strict, disallowing the use of dynamic memory allocation, pointer math, and other typical low-level language "features" or "shortcuts". There are also strict code coverage test requirements, demanding that primary partitions of the flight avionics have their code tested in what is known as MCDC [1] Additionally, I/O code that uses low level data buses must be tested for bit independence. The process is extremely rigorous.

[1] https://en.wikipedia.org/wiki/Modified_condition/decision_co...

peterwwillis2• 11 years ago

Can you talk about the roles of people who enforce the standards? Is there just a typical QC role like in the medical device industry, or are there extra roles to verify more specialized standards, in addition to QA personnel?

toomuchtodo• 11 years ago

Sounds very similar to the process required by the team who wrote software for the space shuttle. Very impressive.

Triple-Triple Redundant 777 Primary Flight Computer (1996) [pdf]

Related Stories