Personal observations on the reliability of the Shuttle (1986)

http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

by umanwizard • 12 years ago

135 79 12 years ago

science.ksc.nasa.gov

stiff• 12 years ago

The last sentence from this piece is just beautiful, it has become my personal motto:

For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.

It captures in a capsule form the reasons for a huge fraction of all the big engineering catastrophes, maybe even most of them. For everyone interested in similar case studies, and in reliability from a wide engineering perspective, I strongly recommend the book "Design Paradigms: Case Histories of Error and Judgment in Engineering" by Henry Petroski.

marze4• 12 years ago

Collectively, the public has a similar approach to nuclear power plants, which make vast tracts of land useless for agriculture if they melt down.

We've had three meltdowns in 40 years, about one per 13 years so far. Have we been lucky? Or have we been unlucky? Time will tell.

maxharris4• 12 years ago

We've had three meltdowns in 40 years, about one per 13 years so far.

This is not a fair comparison. The shuttle was a bad design, with a high failure rate, and it doesn't make any sense to lump it in with tried-and-true rockets, which have a far lower failure rate. Similarly, it doesn't make sense to lump the dangerous RBMK reactor design used at Chernobyl with the far safer reactors designed in the West.

How many people died in flight during the Apollo missions? The answer is the same number that died as a result of the events at Three-Mile Island and Fukushima: zero. Design matters, and if you're trying to be objective on this topic, you have to distinguish between good designs and bad ones.

Also, not all meltdowns are created equal, as your post suggests. If you look at Three Mile Island on Google Maps today, you'll see all sorts of arable cropland in active use all around the site.

EDIT: Why the downvote? Is there anything I've said that isn't factual?

EDIT 2: here's a link that explains the differences between the Soviet RBMK and the Western LWR designs: http://users.owt.com/smsrpm/Chernobyl/RBMKvsLWR.html

kabdib• 12 years ago

Many reactors are bad designs, dating from the 1960s.

Newer designs are much simpler and safer.

Also, coal plants kill a lot more people than nuke plants do. . . .

ggreer2• 12 years ago

You're worried about three meltdowns, but not the 2,000 nuclear weapons that have been detonated since the start of the atomic age? Compared to bomb tests, the fallout from nuclear power is a rounding error.

But let's review those three meltdowns. Three Mile Island didn't cause a single case of cancer. Chernobyl was a ridiculously unsafe design run with incredible incompetence. The WHO estimates it caused 40,000 cases of cancer and 4,000 deaths from those cancers. That may sound bad, but pollution from coal kills around 30,000 people each year just in the US. Finally, there's Fukushima. It's expected to cause 100-200 premature deaths from cancers. Remember, the cause of that meltdown was an earthquake-tsunami combo that killed almost 20,000 people.

If anything, shutting down nuclear power worsens public health. Demand for electricity isn't going to go down, so we end up burning coal, and coal plants are much worse than nuclear. From http://squid314.livejournal.com/292620.html:

According to the Clean Air Task Force, coal plants kill about thirty thousand people per year in the US through pollution (which causes respiratory disease). There are six hundred coal plants, so that's about 50 deaths per plant. These numbers are much higher - maybe even by an order of magnitude - in Chinese and third-world coal plants, which lack the US' stringent environmental restrictions.

Even in the worst-case scenario, nuclear power still does better than coal:

When you hit a nuclear plant with the fifth largest earthquake ever recorded, then immediately follow that with a twenty foot high tsunami, and then it explodes, it still kills fewer people than an average coal plant does every single year when everything goes perfectly.

There are lots of things that cause long-term harm to the environment: toxic chemicals, heavy metals, and yes, radioisotopes. If we want to enjoy a first-world quality of life, we have to accept some pollution. The least harmful energy source today is nuclear power.

logicallee• 12 years ago

That is ridiculous. Let's say DWave was even better in what it actually produced, but even worse in its PR: it creates a true quantum cumputer proof of concept.

But it is even worse at public relations and EVERYONE thinks that it is a scam with rigged demos. It has no credibility.

Now I ask you: in this thought experiment, is nature going to fund your quantum company, because you actually kicked nature's ass and proved a true quantum computer in concept?

No. You have to actually maintain real credibility, much as the space program did.

Nature can't be fooled, but Nature also doesn't fund shit. Whether the government, VC's, or the people, only people fund people.

Credibility is super important!

hobs• 12 years ago

I think you are not wrong for saying you need to convince the public for giant expenditures, but if all of those things are not doing as they pretend to be, people die and all credibility is lost anyway.

pjmorris• 12 years ago

This is an absolute classic of engineering literature. The last sentence, perhaps deservedly, gets most of the glory, but the whole piece should be under every engineer's and every manager's fingers.

I constantly see the dynamic observed in the first paragraph, and it would seem that the question "What is the cause of management's fantastic faith in the machinery?" is eternal.

curtis4• 12 years ago

I think management's big problem is that they are often confused by the difference between what they need and what they have. Of course I've spent a lot of my career working for venture capital-based startups where this problem might naturally be more prevalent.

smacktoward• 12 years ago

I would phrase it slightly differently: the problem is that when human beings know that they need something, and also know that they will not be allowed to get it, they are remarkably successful at convincing themselves that they don't really need it after all.

Take the infamous O-rings from the Shuttle's solid rocket boosters, for instance. As Feynman notes in this appendix, the appearance of erosion on the O-rings was an indication that their design was fundamentally flawed. So why did NASA's engineers twist themselves into pretzels to argue that everything was fine? The reason is because they knew that there was no chance that those boosters were going to be redesigned. The political will for that, in NASA and in Washington, just was not there. Even if the engineers threw down their tools and refused to launch any more Shuttles on the grounds that they were unsafe, all that would happen would be that the Powers That Be would come down on them like a ton of bricks and force them either to shut up and get back to work, or to get out of NASA and therefore wreck their careers.

When people are forced to choose between bad options, the easiest thing to do is usually nothing. So that's what NASA's engineers did. And since "I know this thing I work on is likely to kill people, and I'm not going to do anything about that" is the kind of self-knowledge that leads to cognitive dissonance (http://en.wikipedia.org/wiki/Cognitive_dissonance), they put together ad-hoc rationalizations to help them live with it. Rationalizations like the misuse of the term "safety factor" that Feynman flagged, for example; if you can twist that term so it fits the facts in front of you, you can convince yourself that the Shuttle isn't really unsafe. Which takes care of the cognitive dissonance.

You can see this cycle all the time in software development, too. How many systems have you worked on that weren't really secure, but you know that management doesn't have the stomach to accept the tradeoffs -- the extra cost, the reduced convenience -- it would take to make them secure? What happens in those cases? In the ones I've seen, the people involved just convince themselves that the system isn't insecure after all. "We've never been hacked; if the system was insecure, we would have been hacked; therefore, the system is not insecure." It's not great logic, but when you desperately want to believe something it doesn't have to be great logic to convince you. It just has to tell you what you already want to hear.

calinet6• 12 years ago

Management's main problem is that they believe the problem is one of people, and designing counter-productive incentive systems to try to get people to do better work, with raises and firings and carrot-and-stick motivation, and not, fundamentally, in designing systems for quality.

This may apply less to NASA, but it does apply to the business world, yet the result is the same. Failures and defects and poor quality are primarily the result of incomplete systematic control over the end product.

So in that sense, I saw this more as an astute diatribe on the management issues of 20th century America than an engineering piece. The engineering here is known quite well; any engineer can explain it thoroughly as Feynman has done.

The part that flies in the face of truth is how we manage people.

Start here for a paradigm shift: http://en.wikipedia.org/wiki/W._Edwards_Deming

dmfdmf3• 12 years ago

This and the Ariane 5 first test flight failure are a must read for any practicing engineer.

https://en.wikipedia.org/wiki/Ariane_5#Notable_launches

altero• 12 years ago

I think root problem is that shuttle was starting and landing with people. They should use it just for cargo, second rocket with Apollo should have transport people from/to orbit.

bdunbar2• 12 years ago

Wings were the real problem.

They stuck out where they could be struck by stuff. They precluded the capsule eject system. They cost Shuttle tons of fuel. For a system used in only one part it's flight regime.

Why _why_ did Shuttle have wings? The Air Force insisted. So they could launch Shuttle into polar orbit from Vandenberg. Shuttle would need the wings to come _back_ to Vandenberg.

Then the Air Force withdrew from the program. Too late to get rid of the terrible wings, however.

gaius• 12 years ago

Curious, how would it land anywhere without wings? Apart from splashing down.

curtis4• 12 years ago

It's a legitimate question to ask if the wings were worth the trouble. However, the real problem from a safety standpoint was that Shuttle was riding on the side of the stack rather than on top.

InclinedPlane2• 12 years ago

Interestingly, a thorough risk assessment of the Shuttle was done much later by NASA (near the end of the program) and it concluded that the risk of losing a Shuttle in the pre-Challenger era to be much higher than 1 in 100, closer to 1 in 10. Many people look at Challenger and Columbia as instances where the Shuttle program hit a patch of bad luck. In reality the Shuttle program has been extraordinarily lucky, there were many other close calls, some not well publicized, that came within a hair's breadth of causing loss of crew and vehicle (STS-1, STS-8, STS-9, STS-27 being examples of such). It was always a tricky bird to fly, and in the early days there were about half a dozen different things that could kill it outright with a shockingly high probability (not just the SRBs or foam/ice strikes on the TPS, also the APUs (which caught fire and exploded on one flight), the computer (which was completely inoperable just before landing on another flight), the SSMEs (which came close to causing loss of the orbiter once or twice), and other components). Over time some of the systems were improved to such a degree that they were no longer serious risks, but the whole system was so complicated and there were so many elements of risk that even at the end of the program many substantial risks still remained.

marze4• 12 years ago

Even with Feynman's carefully reasoned essay, the next shuttle disaster was a mirror of the first: chunks of foam falling of each flight, careful monitoring but no serious action until the foam resulted in a loss of a vehicle.

The first loss was after careful monitoring of near-burn throughs of the SRB o-rings on many flights, but no decisive action.

rbanffy2• 12 years ago

It really astounded me when I learned no shuttle was inspected for damage while in orbit until after the loss of the Columbia.

The shuttle was an experimental vehicle. It was their job to gather as much data as possible on it. With that, the foam problem would have become evident long before the deaths of the Columbia crew.

toufka• 12 years ago

As I recall, part of the rationale, was that even if there was an issue, there was literally zero way for it to be fixed. So now you spend the next 2-3 days in the shuttle, "doing your job" knowing that you will not return alive. With that in mind, the though initially was, "let's just not check".

tanzam757• 12 years ago

It really astounded me when I learned no shuttle was inspected for damage while in orbit until after the loss of the Columbia.

Atlantis was hit by tank insulation during launch of STS-27 (1988). While in space, the astronauts inspected the tiles using a camera mounted on the Canadarm. Hoot Gibson says that it "looked like it had been blasted by a shotgun." They were convinced that they were going to die on re-entry, and Gibson had even planned what he'd say to Houston in his final transmission before he died.

They got very, very lucky. There were 700 damaged tiles, but only one missing tile. And the one missing tile just happened to be over an antenna mounting bracket, which made the structure stronger than the rest of the wing. If the foam had hit just a couple of tiles over, then they would've died.

This was the second flight after the Challenger disaster. It might have ended the Shuttle program, right then and there.

treblig• 12 years ago

There were 135[1] Space Shuttle missions with 2 resulting in human casualties (Challenger and Columbia disasters).

Thus, a failure with loss of vehicle and of human life of 1.48 in 100.

The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from the working engineers, and the very low figures from management.

The reality was even more dangerous than the engineers had predicted, and far more dangerous than management had.

[1] http://en.wikipedia.org/wiki/List_of_space_shuttle_missions

jessriedel• 12 years ago

Reality was not seen to be more dangerous than the engineers estimated. It was very consistent with that estimate.

curtis4• 12 years ago

One of the major problems with the shuttle is that the design of the Space Shuttle meant that a loss of mission pretty much also meant the loss of the crew as well. A 1 in 70 accident rate is not nearly as big a deal if there's a 90%+ chance of the crew surviving the accident. (I just pulled the 90% number out of the air -- I've never seen any actual estimates for the effectiveness of launch escape systems in the event of accidents. In practice the success rate has been 1/1.)

fnordfnordfnord• 12 years ago

The engineers hadn't factored in management's contribution.

VLM• 12 years ago

I would suspect the engineering estimate had one or at most two sig figs, which isn't bad compared to results. I'm sure the management estimate had six sig figures of course.

kens2• 12 years ago

When NASA was comparing designs for their post-shuttle rocket recently, they literally used four significant figures in the risk estimates [1], and used this estimate to pick the design. It seems crazy to me to have four sig figs of reliability for designs that were basically at the PowerPoint stage.

NASA started with the requirement that their new rocket have less than 1 in 1000 odds of loss of crew (LOC). They concluded that using an existing Atlas V had LOC odds of 1 in 957 (unacceptable), while the paper design of putting a capsule on top of a Shuttle booster had LOC odds of 1 in 1918 (totally acceptable). They then quoted this 1,918 number in a lot of places to justify the program.

This rocket was the Ares-I [2], which turned into a fiasco and was canceled four years ago.

My conclusion is that NASA's current risk assessments are as bogus as the ones for the space shuttle. They start with an unrealistic goal (1 in 1000 risk), make totally unjustifiable estimates to meet the goal, and then make bad decisions based on these estimates. Coincidentally, the decisions based on these estimates line up with the politically-desirable outcome.

The 1 in 1918 risk assessment turned out to be totally wrong, of course. The Air Force pointed out that the launch escape system wouldn't work since burning fuel would melt the parachute and everyone would die. [3]

My personal view is that NASA needs to admit that rockets are dangerous and you probably can't get the risk below 1 in 100. Then NASA can focus on doing the best job they can. [4]

[1] See for example http://www.nasa.gov/pdf/140649main_ESAS_full.pdf figure 1-26 [2] http://en.wikipedia.org/wiki/Ares_I [3] http://archive.is/YD1sh [4] See "Safe is not an option" for discussion on how NASA's focus on safety is harming the space program: http://www.thespacereview.com/article/2435/1

kens2• 12 years ago

Related: Gregg Easterbrook's article "Beam Me Out Of This Death Trap, Scotty" [1] is long but remarkably prescient, having been written a year before the first shuttle flight. It goes into the dangers of the tiles, how the costs would spiral, the danger of relying on a single launch vehicle, the benefits of disposable rockets, and other warnings that ended up being right.

The article talks about how unlikely the shuttle was to achieve the expected 500 flights, and would more likely only have 200 flights. (Real number: 135)

Some of the quotes from the article are scary in retrospect:

Quote: "Here's the plan. Suppose one of the solid-fueled boosters fails. The plan is, you die."

Another quote: "When Columbia's tiles started popping off in a stiff breeze, it occurred to engineers that ice chunks from the tank would crash into the tiles during the sonic chaos of launch: Goodbye, Columbia."

Remember, this article is from 1980, before the shuttle launched.

[1] http://www.washingtonmonthly.com/features/2001/8004.easterbr...

guard-of-terra2• 12 years ago

Interesting read, makes be wonder how Buran ever managed to fly.

I mean, Soviet space program of the time had neither financing nor time not priority outlined in the article.

Going to hit Wikipedia.

rst• 12 years ago

Buran was carried piggy-back on an expendable rocket (Energia), with no attempt at reusable main engines, less aggressive engineering on the expendable main engines than the SSMEs, and no SRBs. (The equivalent function was performed by liquid-fueled strap-ons.) The Buran orbiter looked a lot like the Shuttle, but the systems as a whole were engineered rather differently.

tanzam757• 12 years ago

Not only did Gregg Easterbrook's article describe the dangers that would eventually lead to the demise of the Challenger and the Columbia, but he also mentions a third failure mode: the Space Shuttle Main Engines.

Could easily have failed catastrophically and taken out a third Shuttle. Instead, all the failures of the SSME turned out to be survivable. We just got really lucky.

From "Mr. Feynman Goes to Washington": http://calteches.library.caltech.edu/3570/

NASA was claiming that the engines were in the regular range of engineering, but they're not; the engines had many difficulties that the guys at JPL told me about. (I found out later that the people who worked on the engines always had their fingers crossed on each flight, and the moment they saw the Shuttle explode, they were all sure it was the engines. But of course, the TV replay showed a flame coming out of one of the solid rocket boosters.)

robryk3• 12 years ago

Gregg Easterbrook wrote the following about simulated shuttle landings:

They've never flounced like a twig on the crazy rapids of "bias"--the bland physics term for unexplained variations in the earth's gravitational and magnetic fields.

I can't make heads or tails out of it -- I can't find any reference to such a phenomenon called "bias" and I don't see how gravitational field variations (I assume he means the ones caused by uneven density) could have any effect at their minuscule amplitude. Is this a result of some misunderstanding or am I missing something?

001sky• 12 years ago

http://maps.ngdc.noaa.gov/viewers/historical_declination/

benmorris• 12 years ago

In case you missed it Discovery Channel aired a pretty interesting Challenger docu(drama?) in November. The show portrayed Dr. Feynman's path to reaching these conclusions. Pretty interesting and disturbing at the same time.

http://www.discovery.com/tv-shows/the-challenger-disaster/ch...

alexhutcheson• 12 years ago

For anyone else interested in how the organizational incentives and institutional culture at NASA helped to set the stage for the Challenger disaster, I highly recommend The Challenger Launch Decision[1] by Diane Vaughan.

From the New York Times review[2]:

In "The Challenger Launch Decision" Diane Vaughan, a sociologist at Boston College, takes up where the Rogers Commission and Claus Jensen leave off. She finds the traditional explanation of the accident -- "amorally calculating managers intentionally violating rules" -- to be profoundly unsatisfactory. Why, she asks, would they knowingly indulge such a risk when the future of the space program, to say nothing of the lives of the astronauts, hung in the balance? "It defied my understanding," she says.

[1] https://www.goodreads.com/book/show/995029.The_Challenger_La...

[2] http://www.nytimes.com/books/97/04/13/nnp/19074.html

Personal observations on the reliability of the Shuttle (1986)

Related Stories