Blame for 'switch from hell' falls heavily on one GM engineer

http://europe.autonews.com/article/20140605/COPY/306059903

by r0h1n • 12 years ago

58 63 12 years ago

europe.autonews.com

seren3• 12 years ago

While interesting, I find the narrative pinpointing the issue on a single engineer behavior a bit too convenient to be believable.

fleitz3• 12 years ago

Why? Have you never fixed a bug in your own code, or not followed every single one of your companies' procedures?

watwut• 12 years ago

Of course people do mistakes and fail to follow procedures. That is why the company should have processes to catch dangerous mistakes and omissions in place. If one low level guy can cause such huge problems, then there is something very wrong with the way whole process works.

seren3• 12 years ago

I am a SW engineer currently working in a regulated industry, and there is no procedure in place for me to fix something quickly behind everybody's back. The code should be at least reviewed, integrated and tested by 3 independent people. And the issue and the resolution action would have been reviewed/signed off at the system level as well. So it would be very unlikely that I could hide something for years without suspicion.

This is in a way cumbersome and annoying but it is understandable why it is put in place this way.

I used to work in consumer electronics, where I could commit whatever I wanted without any kind of supervision or checks, but this is generally not something you can do when it is a matter of life and death.

dccoolgai• 12 years ago

Yeah, my thoughts exactly... I wonder if the project managers who pressured him to cut corners to save time are also getting fired...

forgottenpass• 12 years ago

As an engineer that works on safety critical stuff I see it as GM finding a fall guy, and/or having laughable-if-it-wasn't-so-tragic controls in place.

Edit: The bbc report says 15 were fired, half were senior legal and engineering executives. To me the issue stops being DeGiorgio's fuckup, but how so many other folks were incentivized to roll with the fuckup. The fuckup is bad, but the the environment that permitted it and later didn't correct it is the real danger. But don't worry, GM assures us that the Valukas report doesn't find a conspiracy or cover-up.

jusben13695• 12 years ago

I think it can be both. It can be a single guy who is responsible for the switch being deliberately misleading through acts and omissions. Then it can be the chain of command above and around him as well. To have GM yesterday fire 15 people - including many senior execs - and then today accuse them of copping out and finding a fall guy and planting the media story doesn't make a lot of sense. It's possible one of the execs who was fired leaked this if it was in fact leaked/planted.

leoedin2• 12 years ago

Does anyone with experience in the automotive industry have any thoughts on this? What sort of culture would lead to the selection of an inappropriate part in the first place? Would it be cost driven? Does this idea that an individual engineer's component selection decision would play such a big part ring true with other commentators experiences?

walshemj• 12 years ago

Well when I did my mech eng college course one lecturer mentioned that a UK car manufacturer used the wrong type of component to fix the steering wheel to the shaft.

Resulting in it being possible for the steering wheel to detach! whilst driving.

serf• 12 years ago

It's likely just an error in the design process that turns into a problem that is cost driven, but from the perspective that the problem was found after tooling had been acquired and production started and would require a lengthy/costly recall procedure.

fleitz3• 12 years ago

Not an automotive engineer, however, it's the same culture that produces the plethora of bugs in software. It's a human factors issue.

It's no different than an engineer selection of a single line of code, if it's in the wrong place, crashy crashy.

Here's a much better article: http://www.seattlepi.com/news/us/article/Engineer-s-switch-f...

Perhaps the engineer was distracted by an underground boxing ring.

leoedin2• 12 years ago

For reference, I work in the space industry, where piles of paperwork are produced to justify any component selection decision. I can't imagine a situation where anything, even down to individual capacitors, would be selected without at least 2 people having some input to that selection. Our qualification processes are absolutely massive and take decades though - not particularly suitable for the automotive industry.

I don't think that this can be compared directly to the kind of adhoc software development that is routine in many places. Engineering processes have developed over decades to eliminate human error and very, very rarely allow single unjustified decisions made by a single person to propegate through to a finished product.

I'm surprised that in a world where individual car platforms cost billions of dollars to develop there was never a point at which the switch selection was challenged. I haven't read the full report though - perhaps it's not quite as the news article suggests. It would be interesting to hear the point of view of someone in the automotive industry (if any of them are HN readers?)

valarauca14• 12 years ago

I've actually lived with a DRE for several months, and he's a drinking buddy of mine (works at GM too). The first thing you need to understand is the pressure DRE's are under.

It sounds odd, but General Motors is a large and complex entity. Their release system is old and buggy, not to mention never completely overhauled, just constantly having more checks ticked on when ever a recall happens. This is actually true at Ford too, the Big 3 hold on to technology a lot longer then other companies. Ford's time clock system is still ran on emulated System360's using MTS a circa 1967 OS.

Lastly DRE's are more of a projects internal sales arm (they sell the product to management. A DRE's job is to also get the test results from the test engineer, get it to the management, take the managements complaints to the designers/hardware engineers, and relay the tests plans back to the test engineers.

Basically high technical middle management. Their responsibilities at some companies may involve actually writing test plans, sometimes not. I don't believe GM does directly but somebody under the DRE does this.

Ultimately the DRE is just the guy who gets all the forms to all the right people so he can push the plans into the system. And takes all the responsibility when a product fails. On top of that your normally managing 5-6 products concurrently, each taking 18-72 months to clear the system. Your work load is basically endless meetings 500+ email a day, and 100+ phone calls.

swimfar• 12 years ago

DRE=Design Responsible Engineer

TheSpiceIsLife4• 12 years ago

It sounds like the automotive manufacturing industry is ripe for disruption. Enter the likes of Telsa Motors. Being able to focus on two or three products, getting them right, being 'agile'. Of course, for the agile new company to survive there has to be old cumbersome companies for it to carve a market from. I have a hunch there will be others, probably 'cheap Chinese knock-offs'[1] that will outperform the incumbents on price and quality. Hopefully the whole industry benefits.

1. I use the term 'Chinese' in this phrase to represent any state actor who can rapidly deploy, or incentivise, manufacturing capability at a significant discount to their Western counterparts.

weland2• 12 years ago

Ford's time clock system is still ran on emulated System360's using MTS

Fuck! MTS? The Michigan Terminal System? I haven't heard that being mentioned since I was, literally, a kid. Even IBM considers that obsolete, and IIRC even development stopped about 15 years ago!

Yunk• 12 years ago

I have a friend who worked for TI's automotive. They sold it off as the entire sector is bellow TI's minimum profitability level. So cost is issue #1..

But the subtext in the article makes it pretty clear. They are tracking cost of design as accurately as unit costs.

It is largely an "in for a dime in for a dollar" scenario for each internal engineer when they approve a design direction that appears to cut a cost. The larger oversight is unaware of some of the hidden issues and each engineer is motivated to paper over them to some extent to protect their relative standing. The engineers who are completely transparent are completely unemployed.

Shivetya• 12 years ago

Well when you have too many conflicting responsibilities tied up in one person your bound to see something slip through.

Its not a culture problem, its a management problem. One person cannot be allowed to essentially design, development, qa, and support. If they are who knows what essential steps are not fully performed if at all? checks and balances fall by the wayside when your both.

One note I remember from some buddies who used to be in similar careers, part costs are calculated not by individual unit prices but the production run. Hence a change you say, that is only a few cents can be hundreds of thousands of dollars.

pjmorris• 12 years ago

"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars."

- Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1]

[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993

puppetmaster3• 12 years ago

Blame the engineer. Hmm. Lets see. Will I ever buy Government Motors in my life? That would be a no.

jusben13695• 12 years ago

Kind of disappointing to see "Government Motors" on Hacker News.

shiven2• 12 years ago

Giant Monoliths are (understandably) not looked upon kindly on HN. Move fast & Break things is the mantra here. ;-)

SilkRoadie• 12 years ago

It doesn't matter what the engineer did. The fact that the car company had so few controls that a single engineer could approve something faulty then change the specs of it in future models without anyone noticing is worrying to say the least.

It is a bit like banking and these "rogue traders" who spend a billion dollars. They are at fault certainly BUT more so it is the banks fault for having such lax measures that they had access to this much money.

oldmanjay2• 12 years ago

it definitely matters what the engineerr did. your point of view is equivalent to blaming the police for crime.

keyme• 12 years ago

More like blaming lack of policing for crime...

spindritf• 12 years ago

They had checks in place for errors, not for being deliberately misled by one of their own employees. If they had checks against that, we'd be reading a blog about procedures and corporate culture in GM suffocating engineering.

bilbo0s2• 12 years ago

Being a pretty big process guy... I have to agree with SilkRoadie here. I should preface this by saying that my experience comes from developing FDA regulated medical diagnostic software and not automobiles. So I realize there may be a significantly lower quality threshold that GM is obliged to meet. Having mentioned that, I can tell you that, at least in that medical diagnostic software startup, the idea that an engineer could design and implement ANYTHING and get it into the software build without QA noticing is somewhat fanciful. Someone from QA would have had to be in on it.

The process and company structure pretty much made it impossible for the the type of situation outlined in the article to occur.

Just as a 30,000 ft overview of the process... imagine a place where every little change required a minimum of 4 signatures, engineer, QA engineer, Head of QA/Reg, and CEO. This allows not only QA, but also the FDA to determine precisely who was responsible for each change. Imagine further that the signatures have to be on paper, and that giant mass of paper documentation has to be submitted to the FDA, along with mountains of other documentation as well as the software itself, for approval. (But everyone understands that it's also submitted so that the FDA knows who to throw in prison if serious bugs crop up.)

The situation outlined in the article should be STRUCTURALLY impossible as well. At the startup I worked at for instance, build engineering was under QA. This meant that a development engineer could not give the build team a build directive. Only the head of QA/Reg could do that. Also, the head of QA/Reg didn't report to the COO or CFO or Chief Counsel or anything like that, she reported directly to the CEO. As well, she was a frequent contributor at board meetings. There was just no way to pull an end around on her.

That's just a rudimentary outline of the controls we had in place at that startup. Believe me, it actually got a WHOLE lot more restrictive than what I've outlined, but the other stuff is not important to the point I'm trying to make.

And that point is this...

While I realize that for cars the safety standard is probably lower than it is for medical imaging and RTP packages, you would think GM would have those rudimentary controls in place at a minimum ???

benjamincburns2• 12 years ago

Thanks for saying this. I've worked for companies who didn't trust their engineers. It's terrible.

To anyone who read this article and thought what the GP thought, consider that the value of any "thought worker" is inherently tied to the amount of trust you're able to place in them. This is why as an engineer the more you get paid, the more accountability you have.

If you ever work for a place which suffers from institutionalized distrust for its employees you should run. Run very very quickly in the opposite direction.

keypusher• 12 years ago

If the engineer was the same one making those checks, then they effectively did not have checks. I can't believe the quality control was so lax that nobody else checked the final product against specs, but apparently nobody did.

calinet62• 12 years ago

Yes! Systems are the problem here. He did not make that decision in isolation—there was some pressure from the supply chain, his manager, or cost that he would have been punished for had he not made this decision. A fear or motivation existed which was the result of poor management practices or a fragile quality process at odds with many other variables.

No matter how you slice it, this is a management problem, and a cultural problem. It is not an individual responsibility to ensure quality. If it is left to an individual, then things like this happen out of sheer statistical probability. The only thing you can depend on is human fallibility, so your systems should be designed for it; they should expect it, more than anything.

We should all look at quality as a systems, management, and fundamentally cultural pursuit.

Sadly, the individual getting the brunt of all the blame here will only worsen the cultural and management factors that caused it in the first place. Future employees will fear individual retribution, and will make rash, illogical, uninformed decisions in the future out of fear. This, too, will not be their "fault" but the fault of the company, the management, and the systems around them.

Deming. Look him up. There's a reason this does not happen in Japanese car companies. There's a reason that when it does, individual employees are not blamed. The reason is critically important to the success of American manufacturing in the future.

specialp• 12 years ago

The fact remains though that if you are doing anything that could result in injury or death to third parties, you should never bow to management pressure.

I used to inspect amusement rides for safety when I was in college. Occasionally there would be a defect that would arise that would make me feel it was not as safe as I thought it could be. Managers and owners of course would not want their ride shut down on a busy summer day and would pressure me not to close it but I did not care. I knew if children were hurt or killed I would feel horrible, and be the guy taking the blame or criminal penalty for negligence.

So if you are an engineer for some safety critical system never bow in to management pressure as it will be viewed as 100% your fault when something goes wrong. It is not all this engineer's fault as people have pointed out, but when you put your name on the line approving something it is then your responsibility.

funinobu• 12 years ago

This news story talks about the proximate cause of the failure and how the cover-up started. But it doesn't say how the cover-up propagated. 14 other people were fired. But we don't know why or whether that was the extent of either covering up or incompetently failing to to uncover either the problem or the cover-up.

dankohn1• 12 years ago

I think modern testing culture and source repositories can help avoid this kind of negligence in software projects today. I'm hardly Panglossian enough to suggest that it's common, but if you have an engineering culture where specifications and especially regressions are encoded in tests, it starts becoming clear when your software project is or is not satisfying the requirements. Decisions like removing a test because it fails intermittently are documented by source control, and can be reviewed by peers and management.

xyzzy123• 12 years ago

I have to wonder how much the focus on the switch is papering over other problems. The fact that a non-essential mechanical component can cause accidents like this seems fragile.

When the key pops out, the engine stalls, power steering goes out and the airbags do not inflate. Where is the "defense in depth"? A robust system should really require two things to go wrong.

Is it that "key in ignition" is a critical part of the vehicle's state machine?

hga2• 12 years ago

Pretty much, I think.

Not being able to decisively turn off the engine has also killed people, remember the Toyota issue not long ago? We think that in general the drivers panicked and floored the accelerator thinking they were pressing the brakes, but the complexity/difference of turning off the engine with a button---as I recall it had to be held down for something more than a second---vs. turning a key all the way counterclockwise and ultimately pulling it out could have resolved the user errors short of fatal crashes.

Ditto the theory that some or many of these were caused by the accelerator getting stuck on a floor mat. Or an embedded computer and/or components attached to it failing in a bad way.

Compare to the big red buttons on some computers and hopefully in all big machine rooms: there are times, when lives are threatened, that you want an easy way to immediately turn everything off. Even if you add a molly-guard in front: https://en.wikipedia.org/wiki/Big_red_button#Molly-guard

deserted2• 12 years ago

If you're driving down the freeway, putting the gear selection lever to neutral would be a much better choice than killing the engine and taking out the key.

dmritard963• 12 years ago

Its interesting how much attention the general public has given to this yet for only a handful of deaths. For some perspective, 34,000 people died in the US during 2012 due to motor vehicles[1].

The big automakers are ultimately causing way more deaths by not adopting autonomous vehicle technology sooner if you ask me. And they will continue to do so because they realize that autonomous fleets make more sense than every consumer buying a car or two hence autonomous cars mean their business is in jeopardy/going to shrink.

We might also look into deaths associated with air quality/pollution and the number of deaths resulting from it. Its always going to be hard to pin down what number of deaths correlates to how many cars driving some number of miles, but I would put serious money on lower emission vehicles significantly reducing deaths resulting from air pollution. Detroit's reluctance to adopt cleaner technology and its obvious love affair with oil is killing many more than their incompetent switch DREs.

Ultimately, even the switch design is a similar phenomena. While many automakers started to get away from keyed ignition switches altogether, GM has continued to use a super cheap part that's obviously a design from the past. You can't tell me the radio on my grandmothers key ring cost versus their switch is worth 13 lives.

Certainly, don't mean to trivialize those that have been killed by this, just want to point out that Detroit is so slow to change that ultimately its costing us way more lives than just those killed by this switch problem.

1 http://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in...

chris_wot2• 12 years ago

I disagree. This is particularly interesting because there was an active coverup of the issue. This could have been resolved with almost NO deaths, had the actions of this one engineer not been taken.

dmritard963• 12 years ago

I don't mean to troll. I agree its interesting because there is a cover up and an engineer who could have saved 13 lives and didn't.

My main point is that big automakers are doing this same think, cover up and all on a grand scale but because you have to look at it from 10,000 feet to see it, it doesn't get the same type of criticism.

you said you disagree, there were like six points I tried to make, was there a particular set you disagreed with?

worklogin• 12 years ago

I will challenge you on one point: Keyless ignitions are kind of cool (I own a vehicle with one), but the mechanisms of enabling the two stages of aux power before "engine on" are cumbersome. The behavior of the button changes with whether you're in N or D, in addition to whether your foot is on the pedal or not.

Knowing what state the vehicle will be in with a keyed ignition is much easier.

Source: mid-twenties engineer. I can't imagine how grandma must feel trying to fiddle with 'radio on, windows rolling, engine off'.

dmritard963• 12 years ago

thats an interesting thought. My grandmother hasn't had issues lol but more relevantly, I think you are talking about the fact that without aux, power steering, power brakes, airbags, etc. won't work?

I can kinda see that, but seems like there are other solutions to that. Say I was driving down the road at 40, and my ignition indicated that the car was to be turned off, you would think some little state machine (or even stateless) would say, oh hey, we are actually moving....maybe don't turn of aux power?

Blame for 'switch from hell' falls heavily on one GM engineer

Related Stories