How to Swiftly Destroy a $370 Million Dollar Rocket with Overflow "Protection"

http://blog.metaobject.com/2014/06/how-to-swiftly-destroy-370-million.html

by mpweiher • 12 years ago

9 19 12 years ago

blog.metaobject.com

dalke4• 12 years ago

I don't think the author has read the Ariane crash report, which says:

Although the source of the Operand Error has been identified, this in itself did not cause the mission to fail. The specification of the exception-handling mechanism also contributed to the failure. In the event of any kind of exception, the system specification stated that: the failure should be indicated on the databus, the failure context should be stored in an EEPROM memory (which was recovered and read out for Ariane 501), and finally, the SRI processor should be shut down.
It was the decision to cease the processor operation which finally proved fatal. Restart is not feasible since attitude is too difficult to re-calculate after a processor shutdown; therefore the Inertial Reference System becomes useless. The reason behind this drastic action lies in the culture within the Ariane programme of only addressing random hardware failures. From this point of view exception - or error - handling mechanisms are designed for a random hardware failure which can quite rationally be handled by a backup system.
... An underlying theme in the development of Ariane 5 is the bias towards the mitigation of random failure. The supplier of the SRI was only following the specification given to it, which stipulated that in the event of any detected exception the processor was to be stopped. The exception which occurred was not due to random failure but a design error. The exception was detected, but inappropriately handled because the view had been taken that software should be considered correct until it is shown to be at fault. The Board has reason to believe that this view is also accepted in other areas of Ariane 5 software design. The Board is in favour of the opposite view, that software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.

There were also questions about why the code was even still in operation. The functionality was useful under Ariane 4 but not Ariane 5, the code "serves no purpose" after launch, analysis was done to check some variables for overflow but not all variables, and the component's functional requirements weren't updated to reflect the new launch trajectory data.

To say it's only because of an exception during type conversion ignores the whole, very important context, which says that the exception wasn't the real problem.

mpweiher6• 12 years ago

Yes, the author did read the report, did the esteemed critic?

"It was the decision to cease the processor operation which finally proved fatal."

Which is exactly what the author argues against: don't terminate the process on integer overflow.

zvrba4• 12 years ago

Don't terminate a mission-critical process. Garden variety software: yes, PLEASE, DO kill it.

You don't seem to realize that many of SW practices that are adequate for "normal" software, are NOT applicable to mission-critical, fault-tolerant scenarios.

dalke4• 12 years ago

"The decision to cease the processor" was an explicit design decision by humans, who assumed that the analysis of the software was sufficient to be able to assume that an exception was caused by a hardware error instead of a software one. This was valid ... for Ariane IV. "Don't terminate the process on integer overflow" is not allowed by the specification.

It's incorrect to attribute the problem to an overflow exception. Rather, it's equally correct to say that it was a design fault (because that functionality simply wasn't needed for Ariane V) or an analysis fault (because the overflow could have been detected that way) or a testing fault (because testing with simulated Ariane V inputs would have revealed the fault).

If there was a random hardware failure in SR-2 then a shutdown, to only use SR-1 would have been fine. But SR-1 was subject to the same failure condition, because it used the same code. So it's equally valid to say that the failure was in not using two independent implementations, as a mitigation for simple coding errors.

Hence the report says the problem was "due to specification and design errors in the software of the inertial reference system", and does not place the blame only on the triggering event. It's also why "confining exceptions to tasks" is only half of one of the 14 recommendations.

vor_2• 12 years ago

You, Marcel Weiher, are the author.

zvrba4• 12 years ago

The author argues against crashing on integer overflow by drawing an example from a totally different domain for which Swift is not at all intended.

Integer overflow has been a source of numerous exploits, and I'd prefer the software on my desktop machine / phone / tablet to crash instead of giving unlimited access to the intruder.

The story changes radically if we start talking about mission critical systems, like car control.

mpweiher6• 12 years ago

Looking at https://www.owasp.org/index.php/Integer_overflow the consequences of integer overflow by itself either crashes/infinite loops. Crashing on overflow is not an improvement in these cases.

To get into exploitable access, you need buffer overflows, which can and should be protected against separately.

Once you protect against buffer overflows, crashing on integer overflows gets you exactly nothing. (Might be OK as an optional, assert-like debugging tool).

zvrba4• 12 years ago

To get into exploitable access, you need buffer overflows, which can and should be protected against separately.

Or you can use overflow to make the program to allocate too small buffer thus obtaining the buffer overflow you need to proceed further.

deliminator• 12 years ago

Overflow protection is fail-fast behaviour, similar to array bounds checking. It allows you to more quickly find the source of errors. I believe everybody agrees it's a good thing, at least in the array bounds checking case.

Optimization flags in general should not change visible program behavior, except for performance.

The only behavior you would be able to observe before, that you wouldn't after disabling overflow behaviour, is a crash, and you would have fixed that anyway as soon as you observed it.

Of course, if there is $370 Million on the line, maybe just disable it and hope for the best :-)

How to Swiftly Destroy a $370 Million Dollar Rocket with Overflow "Protection"

Related Stories