The 787's Problems Run Deeper Than Outsourcing

http://blogs.hbr.org/cs/2013/01/the_787s_problems_run_deeper_t.html

by hype7 • 13 years ago

92 53 13 years ago

blogs.hbr.org

from the linked boeing paper:

if outsourcing is to be employed, it is understood to be absolutely necessary that detailed parts and subassemblies be designed with that process in mind. this requires considerable additional upfront effort in planning to avoid the situation whereby major subassemblies do not fit together at final assembly, increasing the cost by orders of magnitude more than was saved by designing in isolation from the work allocation activities.

forgive typos - cut + paste from pdf not working.

there's more good stuff about the timescales over which outsourcing for low labour is appropriate.

https://s3.amazonaws.com/s3.documentcloud.org/documents/6974...

graeham• 13 years ago

Seems a bit pre-mature for the conclusions of this article. The article is motivated by the grounding of the plane due to battery issues, but its not yet clear what is causing these batteries problems. In otherwords, the article's argument would still be true if there were no battery issues, but I think readers are more interested in the context of these problems.

Generally I agree with the thesis of the article - aircraft are incredibly complex and require both modularization and top-level design management. And the article does make a good point that this kind of framework is needed for this scale of projects - whether module are worked on "insourced" or "outsourced". I think a potential issue with outsourcing is that geographic and organizational isolation probably leads itself to tougher integration.

willvarfar• 13 years ago

The article cites a long list of delays and problems, does it not?

joezydeco• 13 years ago

I get the feeling the market had forgotten about the delays until this current situation. Planes were shipping in a highly visible fashion.

cpleppert• 13 years ago

I'm not sure the author hasn't run too far with his thesis here. Every modern airliner is outsourced to a major and entire systems must be designed by the supplier that provides them. If the requirements are understood this usually isn't an issue.

Major components of an airliner are often designed and manufactured in geographically separate locations. In the case of Airbus each national entity has different processes in some cases and this has caused issues in the past. The 787 is no different in this respect where entire systems are manufactured and built by different entities and then transported for finally assembly. In the case of the 787, if the author is correct that modularization has caused the issues then one would expect that the largest issues would occur with the major elements such as wings as they are built entirely by a subcontractor in japan. Of course, there have been no such issues.

The electrical problems are probably driven more by the extreme demands placed on the power supply than anything else. It wasn't as if Boeing wasn't aware of the risks involved in choosing Li-ion batteries and was just blindsided by a supplier. It made the decision and then sold it to regulatory agencies. If it didn't understand how to make the batteries safe or it wasn't comfortable with the design Boeing should have rejected it.

Yes, we still don't know the root cause of the issues. But suddenly leaping from that to placing blame on McDonnell executives keen a little too keen on outsourcing just based on a hunch is a little strange to say the least.

acomjean• 13 years ago

I think the article gets it. I worked as a Boeing subcontractor on a radar system. Boeing really seemed to want to get into the government contract game, a different beast than commercial aviation. They were making more on defense contracts than commercial aviation (although that seems to have changed back the past couple years).

quote: The thing about these Government contracts is that they are paid as development proceeds. This is entirely different — and a lot less risk.

hammock• 13 years ago

I wonder about SpaceX. They are also trying to get in the defense contracts game (already have as of Dec 2012).

pyre• 13 years ago

They also have strong leadership. I have a feeling that as long as Elon Musk is at the helm, he won't let that kind of stuff seep into their operation. After he's gone? It could be anyone's guess. Pulling in defence contracts could affect the corporate culture long-term, but I have a feeling the Elon will try to keep it at bay while he's there. I mean, he has aspirations to be on a Mars mission himself. I can't imagine he would want the equipment to be skimped on.

jordanb• 13 years ago

Boeing is in the government contracting game. They're the second biggest military contractor after Lockheed.

Military contracts are low risk for the contractor because the government absorbs all the overruns, but relatively low profit as well since the contracts specify the amount of profit the company is allowed to earn.

Boeing's huge military contract business is good, steady money, but the commercial aviation business is where they take the lion's share of their profits.

That being said, it seems like both Airbus (EADS) and Boeing are profoundly dysfunctional organizations that are really incapable of doing anything without consuming enormous resources. I wonder if both of them have the "military contractor" mindset built into their corporate culture, and that's why their commercial projects are a serial set of disasters.

I think A&B are lucky to have each other as competitors. If a lean, hungry company ever entered the market it'd wipe the floor with both of them.

phillc73• 13 years ago

Stepping up into the biggest end of commercial aviation is a big, big ask.

There are already a number of aircraft manufacturers playing in the next tier - Bombardier (Dash 8), Embraer (EMB 120), Fokker (F50) and others. I can't really see them ever building something the size of a 787 or A380, but I concede it could be possible.

Looking at existing manufacturers, I think Antonov may stand a chance. They already have the AN-124 and AN-225, which might be engineered into passenger configuration.

With regards to new players, I was reading an article about the Indian company Mahindra yesterday[1]. They have ambitious aerospace plans, already some exposure to the sector through their purchase of GippsAero[2], obviously with much smaller aircraft, but also are located in a fast growing market. The article mentions Boeing's expectation of over 1,000 new commercial airliners needed in India by 2020. Maybe this is your new lean, hungry company.

[1] http://www.flightglobal.com/news/articles/indian-automobile-...

[2] http://www.gippsaero.com

hencq2• 13 years ago

As a counter point, outsourcing has served the car industry quite well. At one point in its lifetime Ford had it's own rubber tree farms. Nowadays car companies rely a lot on suppliers and heavily involve them in design as well. It's just very unlikely that one company has the skills and the focus to be good at everything.

That said (and the article makes this point as well) suppliers have to be closely involved for this model to work. When Toyota was consistently beating American car manufacturers in terms of quality one of the biggest differences was in how they managed their suppliers. American companies tended to have multiple suppliers for the same parts and determine (sub) contracts mainly on a cost basis. Toyota on the other hand had very long term relationships with its suppliers and worked much more intensely with them.

nickpinkston2• 13 years ago

I don't think this is a counterpoint. They said that it makes sense to be vertical early on. Look at the Ford Rouge Works where iron ore and rubber came in one side and Model A's came off the other.

Now, it's a very mature category where it's all about refinement, not major innovation - so it's good to develop an interchangeable supplier base because you increase competition between them and you don't need to do anything radical.

If you look at SpaceX for example, they make >90% of their parts because there's full-stack innovation going on constantly.

hencq2• 13 years ago

> Now, it's a very mature category where it's all about refinement, not major innovation

I disagree. You are completely right that in that situation it makes sense to develop an interchangeable supplier base. as you mention it allows you to increase competition and put pressure on them to reduce costs.

However, in a situation where you're still looking for innovation, like developing a new car (or plane) it makes sense to work with highly integrated suppliers. These won't be interchangeable and because you're very dependent on them, the opportunities for cost saving are probably smaller. On the other hand, because they have specialized knowledge and skills they can actually help you innovate much faster. At least that's the theory; you're right that SpaceX seems to focus on doing as much as possible in-house and it seems to work for them. A reason could be however, that the type of suppliers necessary to make this model work are just not there (yet) in their business.

I guess in practice it's about finding the right balance. Insisting on doing everything yourself can lead to bad cases of NIH syndrome, while outsourcing everything leads to a loss of control.

ww520• 13 years ago

I think the point in the article is that outsourcing your design of a brand new product early on is a bad idea. Once the design is done and the components have been standardized, outsourcing would cut cost. For cars, a lot of components have been standardized and don't change much over time. A tire still has rubber outside and steel rim inside.

michael_miller2• 13 years ago

I really wish people would stop speculating about the 787s problems. Yes, it has had serious issues. But why is the author qualified to talk about them? From his bio, it's not clear that he has any aviation experience, and he doesn't appear to have any inside sources.

The article is littered with phrases like "My hypothesis is that McDonnell's mindset from its defense work — minimizing the amount of capital put at risk during R&D — was applied to the 787" and "But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon". How can the author make such brazen claims without having talked to a Boeing engineer or visited the factory? I assume if he had done either, he would have noted it in the article.

Only one type of person is qualified to comment on the problems: a person working at Boeing, or someone talking to him/her. Otherwise, we're just playing the Apple rumor game, piecing together tidbits of supply chain data to come up with the conclusion we want to hear.

noiv• 13 years ago

I'd like to add a second type of persons having certified its airworthiness and as a third type everybody else wondering whether the tons of flying metal labeled aircraft will stay over their head.

caycep• 13 years ago

it's a bit of a gray line - it's being published on HBR which is a "Big Name Publication" to a certain extent. that being said, it IS a blog, and blogs are for random musings such as this...

michael_miller2• 13 years ago

"Big Name Publications" do not get a free pass for speculation, in print or in blog. They gain credibility, so they can say things like "From talking to senior Boeing engineers, we uncovered..." but making up hypotheses out of thin air is shoddy journalism that should be criticized, regardless of the source.

AJ007• 13 years ago

When I read enough poor quality articles from a publication, I never read anything from that publication again. I haven't read HBR in about 4 years.

primitur4• 13 years ago

Do you know what the word 'hypothesis' means? Its a theory, something to be tested - its no fact, no statement of truth, just pure conjecture that something has gone wrong with the Boeing group.

You don't get this far into the 787 project, to be defeated by a faulty battery design, unless something is seriously wrong with the engineering processes. Unless management, senior to engineering in the grand scheme of things, has screwed up somewhere. This much is clear.

That the battery catches fire in an operational plane -i.e. its no longer in the test phases, but actually certified and operational - yet none of the billion-dollar Boeing process caught this issue beforehand - is of EXTREME INTEREST to the tech community, and in that context there is absolutely no reason why the issue shouldn't be discussed, openly, as such.

Look: Someone really, really screwed up. Its a Billion-Dollar project, being crippled by a fundamental design flaw - this proves the value of the circumstances to the tech community, for which the motivation for discussing this incident is, mostly, derived from an interest in the process failure.

That the battery is flawed somehow, that the fuel control system is flawed, somehow, that the fire suppression systems don't work so great - this is not so great news, perhaps, technologically, unless we have access to the real data: but that the Process by which Boeing produced a crap expensive product, in spite of billions of dollars of investment, IS very interesting news.

akiselev• 13 years ago

Remember when Toshiba recalled a ton of their laptops for problems with the lithium batteries? They had tens if not hundreds of thousands of those batteries deployed before the problem became evident.

The Boeing batteries are significantly more expensive and higher capacity than laptop batteries which results in longer and more complicated testing procedures. Not to mention needing to test the many various load conditions on the battery. At the end of the day, there is only so much you can do to test a technology before deploying it. There is a chance that Boeing, it's contractors, or manufacturers were negligent in their testing especially after the initial delays of a $32 BILLION program. However, I'm guessing the battery problem is the result of standards not fully prepared to test the battery technology.

hype72• 13 years ago

I did talk to people at Boeing. (And yes, I'm the article's author).

redwood• 13 years ago

Odd thesis: essentially that not outsourcing, but outsourcing too early in design is the problem. Well I don't exactly think you can divorce the two...Once you make a choice to outsource larger and larger components, you're inherently losing control of the design and perhaps more importantly exact construction of those components. In so doing you open yourself to design risk. The way the author describes the primary risk of outsourcing as "business profit risk" is odd to me.

Still the article is ridiculous considering the 787's battery problem, if indeed limited only to the battery, seems like it will be a fairly simple one to fix, and an isolated incident.

I like how he lists three 787 problems, two of which are "smoke in cabin" and "fire" as if the latter two aren't likely the same root cause. Sure fuel leaking was an issue too. So we work around it.

This is typical of the internet era where we can nitpick everything and then suddenly a multi-decade project comes along and we don't know how to handle its optimization/fix process.

hype72• 13 years ago

You might think it's odd, but it's exactly what Boeing had done in the past.

And if you think the 787's problems are confined to its batteries, please read this: http://www.bbc.co.uk/news/business-21230940 "Keith Hayward, head of research at the Royal Aeronautical Society, said that if the issue is no longer about replacing a faulty battery, it raised the prospect of Boeing having to do a major re-design.

"I think people had their fingers crossed that it was a battery fault... it looks more systemic and serious to me. I suspect it could be difficult to identify the cause," he said."

(Disclosure: I'm the article's author)

Cheers

-- james

kdsudac• 13 years ago

Went to see one of the Boeing leaders talk specifically about the dreamliner's development a few years ago at a Georgia Tech symposium.

He made a point to mention how many different international suppliers were involved and how that was a huge challenge. e.g. They had to custom modify several 747s to increase the cargo hold to fit some of the sub-assemblies.

He gave the impression that part of why there were so many different sub-contractors is to appease foreign governments--especially since many airlines are nationalized. The corporate equivalent of pork-barrel spending.

robomartin8• 13 years ago

Oh, please folks, temper your enthusiasm to hang these guys. For all we know it might very well be grotesquely misplaced.

As I stated in another thread, I'd prefer to read commentary from someone who has actually had first-level access to the relevant design and test documentation. Everything else is nothing more than conjecture.

What are some of the failure modes of battery systems, including Li-based systems?

OK, here's an incomplete list that has nothing whatsoever to do with the 787 case because, well, no design and test data has been released. Again, this has nothing to do with the 787. Think of it as the start of a list of things to watch out for if you were going to set out to design a battery pack and related systems.

    Sourcing
        - Latent defects due to manufacturing QA issues
        - With Li-based cells such things as humidity content can
          be critical as they might lead to the generation of methane
          gas during the service life of the cell.

    Assembly
        - Handling and storage problems
        - Mechanical damage, from dropping a cell to any number of 
          imaginable issues.
        - Assembly defects that can lead to (I**2)*R heating issues

    Testing (both at by supplier and OEM)
        - Failure to detect defects (sometimes impossible if internal)
        - A faulty testing process that actually causes or accelerates damage
          to cells, perhaps only cells that have specific problems
        - Inadequate issue escalation workflow

    Electrical
        - (I**2)*R heating of faulty interconnects (inter-cell and elsewhere)
        - Component defects that can lead to (I**2)*R heating issues
        - Oxide formation that can lead to (I**2)*R heating issues
        - Interconnect crimp quality problems that can lead to (I**2)*R 
          heating issues
        - Interconnect crimp quality problems that can lead to other issues
        - Main pack electronics problems
            This can include chargers and monitoring electronics
            - Charge termination failure or inadequacy
            - Thermal sensing failure or inadequacy
            - Pressure sensing failure or inadequacy
            - Fluid sensing failure or inadequacy
            - Inadequate semiconductor rating (for example: operating temperature)
            - Inadequate power semiconductor heat dissipation
                This doesn't necessarily mean bath thermal design.  
                It could be caused by inadequate or faulty mounting of TIL 
                (thermal interface materials).
                It could also be caused by mechanical and environmental issues 
                affecting the TIL or the mating of the semiconductor to the heat
                dissipation mechanism.  For example, vibration.
            - Inadequate or faulty failover modes
            - Static damage to semiconductors during electronics assembly, testing 
              or general handing
            - PCB via or trace fractures, contamination or over-etching
            - BGA ball soldering problems 
                (mostly only visible via x-ray inspection)
            - Bad, cold, fractured or chemically compromised solder joints
            - Dendrite buildup in lead-free solder chemistry leading to shorts 
              in small pitch components
            - Unaccounted race conditions
                Example: pull-pull technologies without adequate dead-band
            - Thermally-induced failures due to changes in operating 
              characteristics of components as temperature fluctuates.  
                Example: The aforementioned push-pull topology where the deadband
                might have been adequate within a certain temperature range but 
                causing shot-through shorts outside of this range.
            - Unaccounted for effects of such environmental issues as humidity.
                Example: Many VCO chips do not deal with humidity very well,
                particularly if it can condensate.
            - Faulty or non-existing conformal coating
            - Inadequate static damage counter measures 
                (inputs, anything expose to handling)
        - Sub-pack electronics problems
            This is applicable if a large pack is made from a multitude of 
            smaller packs, each with it's own control and monitoring electronics.
            Sub-pack electronics can suffer from the same issues as the main pack
            electronics above.

    Software
        - Myriad of code-based issues 
           (lock-ups, endless loops, pointer problems, uncaught exceptions, etc.)
        - Software problems caused by hardware issues
           Example: Static damage to a chip causing a bit to unexpectedly flip.
           If not caught in software this can lead to catastrophic failure

    Mechanical
        - Faulty inspection and certification process
        - Inadequate design materials
        - Adequate materials but supplier quality issues resulted in unexpected 
          materials issues
        - Inadequate design tolerances 
        - Adequate design tolerances but faulty assembly, subcontractor or testing
          process caused them to be violated
        - Faulty fasteners
        - Faulty fastener application
        - Unexpected interference between components
        - Mechanical pinching or chafing of wires
        - Mechanical puncture damage to cells during installation or operation
        - Mechanical abrasion to protective cell wall or packaging
        - Insufficient mechanical rigidity
        - Insufficient venting mechanism
        - Insufficient containment or venting of gasses (methane), 
          flames or heat
        - Mechanical issues once pack is installed
            - Thermal
            - Mechanical (pinching, squeezing, twisting, cutting)
            - Resonance

    Thermal
        - Inadequate thermal management
        - Lack of consideration of thermal runaway problem
        - Lack of thermal isolation between individual cells or sub-packs
        - Insufficient thermal sensing
        - Insufficient thermal overload coping mechanism 
          (both mechanical and electronic)
        - A design that might cause severe differential heating of cells deep
          inside the pack
        - Inadequate electronic and mechanical fail-over measures in the case
          of thermal runaway
        - Inadequate thermal runaway detection
        - Cell supplier issues (quality control, testing, etc.)
        - Thermal issues once installed
            - Problematic heat exchange
            - Heat input from adjoining assemblies
            - Accidental plugging of filters, inlets, etc.

    Final release
        - Electrical testing
        - Mechanical testing
        - Environmental testing
        - Random destructive testing at the cell level

    Operational
        - User fails to follow published procedures
        - Alarms are ignored
        - Mechanical issues are ignored
        - Inadequate maintenance
        - Handling issues

Anyhow, this is not, by a long-shot, an exhaustive list. There's a lot I am ignorant about and, of course, that stuff is missing. Feel free to expand.

The point is that a proper investigation would use --at a bare minimum-- something like this and, more than likely, a book full of specs and procedures before reaching a verifiable and reproducible conclusion.

Maybe it's just me. I don't know. I just don't like it when people do some of what is being done to Boeing. Particularly when they don't have access to the reams of FAA, Boeing and airline data that is required to really understand the problem.

EDIT: Formatting

primitur4• 13 years ago

>>Maybe it's just me. I don't know.

From your comment history it sure seems like you have a vested interest in maintaining favorable PR for Boeing in this forum.

Why shouldn't the industry discuss this in open terms? The fact is we don't have access "to the relevant design and test documentation". Boeing and their partners won't share it. So, how can we discuss this without the data, then?

Conjecture and informed opinion is all we have, and it seems as if you have rather a bit of the latter .. so why not share a technical analysis of the circumstances based on your understanding rather than attempt an appeal at what could be construed, frankly, as a PR move?

What if, Boeing are simply mis-managing the project? It surely is not the first - nor the last - time that a major aerospace company was bit by that bug - in fact, for anyone that pays attention to the big, fat, military-industrial corn-fed cows of the last few decades, its rather the norm than the exception!!

Nothing says "management really, really screwed up" than a smoking pile of brand new tech, being doused with fire-retardant foam, while the world asks questions like "why is our technology failing us?" from afar .. lucky, we're not there yet with the 787 - so this is really the best time for us to be discussing these issues. Lucky no lives are lost .. yet.

robomartin8• 13 years ago

Nope. Never worked for Boeing. Never worked directly in aerospace. I did manufacture motor and battery controls that were used in drones and robots maybe twenty years ago mostly in academia. I have no way to know if the motor controls we sold to institutions such as MIT ended-up in Boeing product. I doubt it.

It's easy. Go to my profile on HN. Send me an email. I'll reply back with my LinkedIn profile. Then you can go figure out if I have anything to gain from my comments.

I'm just an engineer with a lot of experience making mistakes who hates it when people run their mouths without data. That's all. Things at this scale are never as simple as some journalist might make it seem. Most people don't have a clue as to how such massive machines as a 787 are designed, built and tested. Not a clue.

Many years ago I had a chance to meet and enjoy long talks with the engineer who had been in charge of the F18 project. It was a fascinating learning experience for me. If I remember correctly, he said he had some 3,000 engineers working for him. Now, how many people would imagine that 3,000 engineers had their hand on the F18 design simply by looking at it? Better yet, how many journalist? And, how many of these people could diagnose an F18 problem --even if qualified-- without access to real engineering and test data?

robomartin8• 13 years ago

> Conjecture and informed opinion is all we have, and it seems as if you have rather a bit of the latter .. so why not share a technical analysis of the circumstances based on your understanding rather than attempt an appeal at what could be construed, frankly, as a PR move?

Crap, I didn't catch that one.

Really? That's how you see what I wrote? Amazing.

Here you have an engineer (me) saying: "We don't know enough. We need facts before we can reach a conclusion. Please stop."

And your interpretation is that I am using conjecture for PR purposes?

Are you an engineer? If you are not, maybe you just don't understand my vantage point. And that's OK. Life goes on.

Here's an analogy, imperfect, but, what the hell:

A guy gets arrested and accused of rape.

You say: "He probably did it. Hang him!".

I say: "Let's wait and look at the evidence, which should ideally include DNA. Then we can look at the facts as they may be available to us and make a decision."

These are vastly different approaches to the problem.

Who would you want in the jury if YOU were the accused? Someone like me or someone like these clowns that are flapping their jaws without having access to the evidence?

ubernostrum• 13 years ago

Couple things.

One is that I'm actually thankful for robomartin's comments; they break up some of the more outlandish hero-worshipping of Elon Musk (i.e., http://news.ycombinator.com/item?id=5138430, which is frankly just a ridiculous thing to say).

Another is that this is not new. What is new about the 787 is the sheer amount of media attention, not the fact that there are serious post-service-entry problems. For example:

You mean, the way we all went crazy, posted huge numbers of threads and begged people who don't work on airliners for a living to come save Airbus in one fell swoop after the A380 exploding-engine accident (which, on investigation, turned out to not be a one-off problem, but a general flaw with the plane's engines)? The one where a smoking pile of brand-new tech had to be doused with fire-retardant foam?

This stuff happens. The other thing that's different about it now is that when it happens, people don't die by the hundreds, as they used to when a new airliner entered service and a problem suddenly became visible. What is not different is that it still gets solved by research and hard work, rather than by pop-entrepreneurs snarking away in major newspapers.

ScottBurson• 13 years ago

There have been a lot more problems with the 787 than the batteries, as the article describes. And you didn't address the central thesis of the article, which is that the root of the problems lies in how the project was managed.

robomartin8• 13 years ago

I did say the list was incomplete, didn't I?

Feel free to tackle project management issues of a hypothetical project if you feel it might add value.

jamesseattle• 13 years ago

I worked on two airplane programs at boeing. The first was a derivative of the 747, the 747-400. The main features of this version that differed from previous 747s were going from analog to digital avionics, a slightly longer body and new (larger) wings. The schedule from project launch to first delivery was about 4 years. We were two weeks late on first delivery.

I also worked on the 787. The difference between the 787 and every previous Boeing plane was almost everything. The schedule from project launch to first delivery was about 4 years. We were two years late.

Moral: don't bite off more than you can chew.

Osiris• 13 years ago

All this speculation about what's happening from people that aren't aware of the facts reminds me of something that happened to me the other day.

A top manager in my company came to give our office an all-hands meeting about management changes. During that, he claimed that my product (Online Calendar) doesn't support timezones and bashed it in front of the whole group.

After the meeting my manager and I confronted him and told him that the problem was with the iOS Calendar application since we're only providing data via CalDav. He was insistent that because we have control of the data that it is, in the end, our fault and a problem with our software.

My boss's boss gets involved and immediately recognized that the problem is that the phone had a setting enabled that removed automatic timezone switching for the Calendar app. A quick change to that setting and the iOS calendar was showing events in the current (Mountain) timezone rather than his home (Pacific) timezone.

So, even someone who's worked in tech for a long time and has a pretty good understanding of things like protocols can still have no idea what they are talking about without all the relevant information.

The 787's Problems Run Deeper Than Outsourcing

Related Stories