A Case Study of Toyota Unintended Acceleration and Software Safety

http://betterembsw.blogspot.com/2014/09/a-case-study-of-toyota-unintended.html?m=1

by mzehrer • 12 years ago

95 86 12 years ago

betterembsw.blogspot.com

This is an absolutely fascinating slide set. Thanks submitter very much for the link. I have written embedded C before, and the following facts just blow my mind:

1. The Throttle Angle function in the Toyota code had a McCabe Cyclomatic Complexity of 146 (over 50 is considered untestable according to slides) [slide 38]

2. The main throttle function was 1300 lines long, and had no directed tests. [slide 38]

3. I find the static analysis results quite alarming. [slide 37]

4. 80+% of variables were declared as global. [slide 40]

I find this to be a stunning lapse of quality, especially for a safety-critical system.

eropple4• 12 years ago

No ECC in RAM, either. Yiiiikes.

Edit: or peer review, or even a bug tracker. My coworkers working on web apps have a better SDLC than these jokers.

threeseed3• 12 years ago

They did have peer review just not on every module. This is very common even amongst companies who say they do peer review.

And curious about the lack of a bug tracking software. Many companies including the one I work for "technically" don't use them. I wouldn't call Rally or Trello a bug tracker but that do work just fine for that purpose.

I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them.

PhantomGremlin• 12 years ago

Maybe they had a "hardware guy" with little software training or experience write the code?

I've done a lot of both hardware and software, and I've seen a lot more bad software done by hardware engineers than I've seen bad hardware done by software engineers. The software guys usually know that they're out of their bailiwick when it comes to hardware design.

E.g. one of the worst in my experience was a 30,000 line shell script, few if any functions, used as part of our production flow. A simple refactoring could have cut it down at least 90%. Even worse, it was totally unsupported because the guy who wrote it was reassigned.

ScottBurson• 12 years ago

I think this is probably the essence of it. Hardware people see software as just hardware that can be easily reconfigured. That was a reasonable model back in the 8048 days, when one only had 1kB of ROM and 256B of RAM to work in. But these programs are obviously far larger, with orders of magnitude more data paths and astronomically more possible states. Different techniques are required. Someone from a hardware background isn't going to understand that.

magila• 12 years ago

I'm not going to defend the other points, but most variables being global is not at all unusual for code like this. These sorts of realtime embedded applications typically have no heap allocation, so the only way to define persistent storage is as statically allocated globals. Modularization in these systems is done by convention, e.g. by limiting which include files are visible to a compilation unit.

DangerousPie• 12 years ago

It's strange that the slides don't seem to mention this point at all, even though the author is presumably an expert in this field. Why would he make such a big point of this if it's commonly done in the field?

jacquesgt• 12 years ago

Non-stack variables can be static instead of global. Either file-scoped or function-scoped.

wyager• 12 years ago

80+% of variables were declared as global

This is actually important for safety-critical programming. It lessens the likelihood of running out of stack space, and it's useful for eliminating dynamic allocation.

Some safety-critical software has 100% global variables, without even using a stack.

jacquesm9• 12 years ago

I think this is possibly one area where functional programming could shine if the garbage collection issue could be dealt with once and for all (and that's a hard one). It's for practical reasons impossible to test something with that much state exhaustively, but once broken up into functional units you just might get there. Something along the lines of Erlang for embedded systems.

ufo2• 12 years ago

But even then, you want to declare those variables as static or otherwise find some way to reduce their scope. Most global variables in the Toyota code were visible and writable from the whole program.

This is not even counting all the variables declared with the wrong type or more than once, uninitialized variables, etc.

threeseed3• 12 years ago

Static analysis does not show you the complete picture. I have worked on countless horrifically complex, buggy and unmanageable J2EE applications that had perfect cyclomatic complexity et al scores. I have also written perfectly reviewed, manageable and well tested piece of code that didn't.

I very much question the experience of anyone who can refer to a codebase as "spaghetti code" without seeing it and relying solely on static analysis. Software is not that simple or transparent.

userbinator2• 12 years ago

I'm going to guess that the applications you refer to are "simple" at the level of individual functions, but that's only because the complexity has been spread out so much that it becomes difficult to understand the whole. That's exactly why I think cyclomatic complexity and related metrics are of little benefit, and may even be harmful - "refactoring" to reduce "point complexity" can result in increasing the complexity of the whole.

I've always wondered what the cyclomatic complexity of TeX is. While Knuth is a bit of an "edge case", it would be fun to see what static analysers think of his code... complete with copious use of global variables, goto, and very, very long functions. Give someone who has never had any experience with TeX the results and ask them what they think the defect rate would be, then show them the fact that it's one of the most bug-free pieces of software ever written.

CountSessine• 12 years ago

I have also written perfectly reviewed, manageable and well tested piece of code that didn't.

Ok, so I'll bite. The fact that not all bad code scores poorly on cyclomatic complexity scores does not imply that code that scores poorly in cyclomatic complexity isn't bad code.

So you wrote a function with cyclomatic complexity greater than some accepted value (50?) and you claim that it's well tested. I'm skeptical. How many of those code paths did you follow in your tests? What was the ratio of lines of logic code to lines of test code for that function? Can you really say that your tests provided adequate coverage of this function?

I've fixed a lot of cowboy-code with horrible cyclomatic complexity and no state separation, and this report about the 1500-line throttle control function dipping into system-global pool of universally-accessible variables sent shivers down my spine.

cbd1984• 12 years ago

I thought the "unintended acceleration" was proven to have been due to older drivers who got confused?

matt4077• 12 years ago

Well, Toyota probably has the better PR team (and more of an interest in getting it's message out).

It appears as if neither theory has ever been proven. The slides argue that the standard of evidence in civil proceedings is simply "more likely than not" and not "beyond reasonable doubt".

Reading between the lines, I think this expert witness is arguing that anybody who writes code this terrible for a safety-critical application deserves to pay through the nose, irregardless of the actual chain of events.

ArkyBeagle• 12 years ago

The Micheal Barr Group was selected to analyze the Toyota controller code. They've filed online reports you can read.

bigtones• 12 years ago

The example he gave was a young police officer who was a very experienced driver, and had a passenger call 911 in a panic while he was unable to stop the car with his foot firmly planted on the brake - and it ended up killing all four occupants. That negates the 'old people can't drive properly' theory.

lnanek2• 12 years ago

Although in Toyota's defense the officer was too clueless to know he just had to hold the start button down for three seconds to turn off the ignition.

ubernostrum• 12 years ago

One of the interesting slides points out that a failsafe in the system wouldn't actually work if the driver was pumping the brakes. Drivers who are used to anti-lock braking systems -- i.e., younger drivers -- probably will not pump the brakes, but drivers who learned on cars which didn't have anti-lock brakes -- older drivers -- are probably more likely to do so.

Which could account for some of the alleged age discrepancies, beyond the incident involving the police officer, which cannot be blamed on age/confusion.

jacquesm9• 12 years ago

Is it known why he did not turn off the ignition?

FeeTinesAMady3• 12 years ago

The Audi 5000 unintended acceleration in the 80s was shown to be that.

bb_wordsmith6• 12 years ago

No it is not. But one physicist, Dr.Ron Belt, has published some papers that may explain the reason for the skew toward older drivers among SUA victims. Basically, he says that their driving habits tend to deplete batteries, and the resulting unstable electrical supply shared by loaded cars and their ECUs can tend to set off software glitches in poorly written code. The pedal confusion theory is often used as a defense by automakers but it has been discredited.

codys• 12 years ago

Anyone know what the things he "couldn't talk about" were? I was hoping to hear the open mic bit at the end for a hint but the video is cut off.

MBCook• 12 years ago

He mentioned he's still involved in legal proceedings and expects to be deposed in further trials. I imagine specific implementation details, things under NDA, etc. are what he's trying to avoid.

In work like this, you certainly wouldn't want to accidentally cause a negligent company to get off the hook because of an unimportant public statement.

veidr5• 12 years ago

I think this is one of the most interesting software stories in recent years, up there with Stuxnet.

The software development process seems so staggeringly, jaw-droppingly incompetent and negligent, and it now seems clear that software flaws really did kill people despite the heavy layer of spin that it was driver error, floor mats, etc.

I almost certainly couldn't buy a Toyota ever again knowing this. But it also really makes me wonder: how bad is the QA and testing for the software components of other carmakers' vehicles?

tobinfricke• 12 years ago

I almost certainly couldn't buy a Toyota ever again knowing this.

What makes you think any other car company is different/better?

veidr5• 12 years ago

That's what I mean; I don't necessarily think it is (which is scary).

Maybe we only got to know how horribly flawed the software situation is at Toyota because they had finally had enough people killed that they just couldn't keep it hidden any longer, and Audi and Honda are just as bad and just haven't yet had this kind of exposure event.

I would prefer to doubt that, but it does all kind of make me want to buy a restored 1978 Datsun with carburetors and mechanical everything.

mattmcknight2• 12 years ago

It doesn't seem at all clear to me that it was "spin" when there were cases of floor mats getting caught under the pedal, incidents of driver error, and physical problems.

It also is striking that the analysis could not provide an example of a single error condition that would cause the crash scenario. It's only a high level analysis.

veidr5• 12 years ago

It probably wasn't just spin -- Toyota probably didn't initially believe that they had a multiple-fatality software bug. When I first read some of the news stories about this, years ago, I also thought it sounded far-fetched that a bug that caused your car to accelerate to 100+ mph and crash could have made it all the way to production.

But it is important to note that the post we are all commenting on here isn't "the" analysis. This is just an academic case study of this now-infamous and interesting case, based on public information.

The closest thing that we have to "the" analysis on this (since we will never see Toyota's internal analyses) is, as tokenrove linked to elsewhere in this thread, the one Michael Barr did that was the main analysis[1][2][3] used in the court case against Toyota.

But there have been a lot of other interesting articles and posts on this case, other than just this one.

[1] testimony part 1: http://www.safetyresearch.net/Library/Koopman%2010-11-13%20a...

[2] testimony part 2: http://www.safetyresearch.net/Library/Koopman%2010-11-13%20p...

[3] slides: http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUB...

aceperry3• 12 years ago

Your comments about the software development process reminds me of the arguments from a few years ago, against open source software, like linux. One of the arguments for professionally made, closed source, proprietary software (like Microsoft windows), was that; programmers getting paid by a corporation would obviously develop better software than a bunch of unpaid, unprofessional, part-time hackers. Funny how much that argument has changed today.

jacquesm9• 12 years ago

I can see an excellent case here for open sourcing such code.

nitrogen• 12 years ago

I've always wanted to build an open source ECU, but haven't had the time.

erdle2• 12 years ago

Not sure if it's widely known how they were caught... but it was when they outsourced their translation of documents and a translator read documents and blew the whistle.

Which is why the top translation firms now offer... security!

veidr5• 12 years ago

Wow, do you have a link for that? I think this is a fascinating story in and of itself, but one of the tangential things that also interests me about it is that nobody I know in Japan (where I live) has ever even heard of this story.

erdle2• 12 years ago

A friend is a director at an interesting company that provides secure translations amongst other services to companies and has made bank since this happened. I guess it's a bit of a secret outside the industry.

bb_wordsmith6• 12 years ago

Here's a link to one key article: http://www.asbpe.org/wp-content/uploads/2014/07/Toyota-Cover...

In Japan, some journalists know, but media organizations won't publish the story.

I am the translator.

zaroth• 12 years ago

'How they were caught' you mean how we discovered Toyotas suffered from "UA"? The story claims it was the incident of the off-duty highway patrol officer on a call with 911 for 4 minutes trying to stop the car before it crashed and killed all 4 occupants which blew the lid on the story.

stretchwithme• 12 years ago

Which is why its good to know how to shift into neutral.

bb_wordsmith6• 12 years ago

"How they were caught" involved thousands of consumer complaints and lawsuits that continue up to the present day. Tens of thousands of unintended acceleration incidents also go unreported and unlitigated because the damage is not major enough to be worthwhile. The government auto safety regulators investigated and then closed the investigations many times due to clever manuevering by Toyota's lawyers and government-relations people. They also lied to Congress (which they have since admitted to when the DOJ investigated their cover up and fined them $1.2 billion. This story is far from over.

upofadown2• 12 years ago

If someone ever comes up with a practical method of creating reliable software then everyone would use it. You can always find a programmer who will claim that they would of done it better... For all we know whatever Toyota is doing is way better than what everyone else is doing and this is their only safety critical bug (assuming that the bug actually exists)...

In the end the software is either correct or it is not.

tokenrove• 12 years ago

The information that has been released so far, especially Michael Barr's comments on it, suggest that this is far from the only bug.

For example, http://www.edn.com/design/automotive/4423428/Toyota-s-killer... quotes Barr's claims: "Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality." "Toyota’s source code is defective and contains bugs, including bugs that can cause unintended acceleration (UA)."

I am a little appalled at the number of apologist comments on this story. There is mounting evidence that this wasn't a "it could happen to anyone" bug, but rather a serious violation of software engineering ethics. Code must not kill.

upofadown2• 12 years ago

...software engineering...

Some of us do not believe that such a thing exists in any meaningful form. In the past every approach to creating reliable software has failed to deliver. That makes it hard to believe that any particular approach is finally the answer.

I agree that the Toyota software is poorly written. That in no way means that conforming to a particular standard or method would of automatically produced software that was more reliable.

snowwrestler• 12 years ago

Also see this report from another experts involved in one of the suits:

http://www.safetyresearch.net/blog/articles/toyota-unintende...

And the slides (PDF):

http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUB...

tek-cyb-org• 12 years ago

This happened to my parents in a 2002 civic.

serf• 12 years ago

2002 civic used a mechanical throttle cable and mechanical transmission linkage. These Toyotas/Lexus(s) did not use any such mechanical links to the drive train, they were dependent on software wholly.

The engine management on your parents' Honda cannot adjust the throttle percentage or gearing, only ignition and valve timing along with air/fuel ratios and gear hold-out timings. This could result in a surge of power, or a stalled car, but nothing like the experience of wide open throttle.

The Toyota UA is thought to have been rooted mostly in the fact that they adopted fly-by-wire throttle schemes while failing to compensate for the much more major software/hardware/management risks that come with such systems.

phugoid2• 12 years ago

My only criticism is that he's comparing Toyota's code and processes to a published but unenforced standard and clearly they are nowhere near that standard. It would be more relevant to compare Toyota to other manufacturers.

It seems the legal argument against Toyota is that they were not following industry standards - but if no one else was, could you really call it an industry standard?

rwmj• 12 years ago

So if all car manufacturers are as bad as each other, that's OK?

phugoid2• 12 years ago

Not at all. But if it turned out that Toyota was already trying three times harder than everyone else to get it right, it would be harder to argue they didn't take "reasonable" precautions.

In the talk, he did mention that the government agency that certifies vehicles does only basic checks and does not enforce any standards on software. One could easily argue that they are partially to blame, as they're leaving it up to the manufacturers.

aristidb• 12 years ago

I'm not really a fan of MISRA-C or V process and stuff like that, but the quality of the source code and process is truly appalling if these slides are right... tens of thousands of global variables, no bug tracking and so on and so on.

I really hope they improved since then, for the sake of anybody driving a Toyota.

EliRivers• 12 years ago

I work with some people in the static analysis industry, and I'm told that Toyota did have static analysis tools that would have identified a number of these issues; as a particular example, recursive functions, which are verboten under the MISRA rules they should have been following.

I heard to a less reliable degree that the tools had been used, and results ignored.

A Case Study of Toyota Unintended Acceleration and Software Safety

Related Stories