FAA closes SpaceX Starship mishap investigation

https://apnews.com/article/spacex-starship-launch-accident-faa-a8d6cec63de579af4b6d5f040e51825d

by fnordpiglet • 3 years ago

164 184 3 years ago

apnews.com

fnordpiglet• 3 years ago

FAA Closes SpaceX Starship Mishap Investigation

https://www.faa.gov/newsroom/faa-closes-spacex-starship-mish...

There doesn’t seem to be any publicly available copy of the report itself, just this summary:

The final mishap investigation report cited a total of sixty-three (63) corrective actions for SpaceX to implement. These included actions to address redesigns of vehicle hardware to prevent leaks and fires, redesign of the launch pad to increase its robustness, incorporation of additional reviews in the design process, additional analysis and testing of safety critical systems and components including the Autonomous Flight Safety System (AFSS), and the application of additional change control practices.

cmsj• 3 years ago

Is it normal the the report would not be public? I'm not often an FAA report enjoyer, but I would quite like to read this one!

psunavy034• 3 years ago

The same technology that lets you put a payload in space can let you put a nuclear (or conventional) warhead over a city. The first space launchers were repurposed ICBMs, and so space tech falls under ITAR and other export controls.

Between that and the fact that the details of the failures and fixes would probably reveal a lot of SpaceX proprietary design info, it's totally unsurprising the full report is not publicly releasable.

light_hue_1• 3 years ago

There's no report better the FAA did not do the investigation. They asked SpaceX to self investigate and report what they found.

toomuchtodo• 3 years ago

If you file the FOIA request, I'll cover the Muckrock request fee ($5). Muckrock is useful because it creates a public trail of the request as well as government responses and responsive docs (if any), and the request link can be posted online for public consumption.

https://www.muckrock.com/

psunavy034• 3 years ago

Dollars to donuts developing the fixes was an ongoing process parallel to the mishap investigation, and most of the recommendations are already done.

We've already seen that the pad has been rebuilt with a water deluge.

Avshalom2• 3 years ago

maybe but the water deluge system seems to drain directly into protected US waters

https://blog.esghound.com/i/135551935/spacexs-permitting-woe...

russdill2• 3 years ago

This is this guy's opinion, it may or may not be a valid interpretation. SpaceX is by far the world's largest operator of space launch deluge systems and other than south texas operates 3 coastal pads, with dozens of launches per year...each.

In this case, a retention pond and collection system does exist. However, when the system is starting and stopping there is overspray outside of the launch area that doesn't get collected. This water is potable and ambient temperature though.

ars• 3 years ago

Is that a problem? It's a kerosene engine, there's nothing harmful in the exhaust.

And if there was something in the exhaust (there isn't) it goes in the air and lands on that area anyway.

phkahler• 3 years ago

If it weren't for the salt, I'd expect them be using locally sourced water. They are not deliberately adding any contaminants to the water, just heating it with flames from burning methane. It should be relatively harmless, but we'll see if there's a show stopper soon.

stevenjgarner2• 3 years ago

I would concur that corrective action has already taken place. Won't the most significant thing in this release be the requirement that SpaceX:

apply for a modified FAA license before launching again

... as SpaceX will now need to go through the wait period of a modified FAA license application?

stevenjgarner2• 3 years ago

"FAA says it may grant a launch license for SpaceX’s Starship by the end of October" - https://www.cnn.com/2023/09/14/world/spacex-starship-faa-lic...

russdill2• 3 years ago

The other large visible change is a purge system for the engine bays to prevent future ongoing fires.

wredue• 3 years ago

I mean. We will see.

Many of the issues should never have happened. On launch day I remember people commenting within seconds of ignition that there was obvious large debris flying everywhere.

The investigation would never have had to happen with better controls and not outright lying on their reports.

jackmott42• 3 years ago

which lie are you referring to specifically?

grecy6• 3 years ago

Elon has a tweet in which he specifically says "Not building a deluge system on the pad might be a mistake, and time will tell".

They're learning, they're taking risks and they're doing new things.

Obviously they're not going to get everything flawless the first time around, and they're going to iterate and improve.

2OEH8eoCRo0• 3 years ago

Eh, maybe the obvious issues but why would you spend money "fixing" guesses?

justrealist2• 3 years ago

The problem with the FAA is only 10% the content of the reports, and 90% waiting interminably months upon months for the reports. So if SpaceX has a list of concrete items to address, it's good news for SpaceX basically regardless of the contents of the report.

entropicgravity• 3 years ago

In this case SpaceX is not the only client the FAA is dealing with. Both the US military and NASA have a lot riding on the timely progress of Starship's shaking out the weak areas so Starship can become viable and reliable.

justrealist2• 3 years ago

The FAA doesn't answer to either of those agencies.

They have no motivation to factor NASA timelines into their prioritization unless the executive branch lights a fire under them, and the executive branch is deeply unfocused on technical details like this right now.

ChrisArchitect2• 3 years ago

[dupe]

ChrisArchitect2• 3 years ago

Some earlier posts and discussion:

https://news.ycombinator.com/item?id=37434066

https://news.ycombinator.com/item?id=37435158

insonable• 3 years ago

A friend in grad. school would put obvious glaring errors in his preliminary drafts of write-ups so the prof's would then be happy to correct these, feel they'd done their work, and move on, avoiding any actual substantive criticisms requiring actual work on his part to fix.

qup2• 3 years ago

That's pretty much a designer/web dev trope, turning in mistakes to the boss that are easy to fix so he can put his stamp on it.

sidewndr463• 3 years ago

it works the same way in construction as well. Before inspection of something like electrical, you go introduce an obvious error that is easy to observe, doesn't take long to fix, and doesn't require additional materials. The inspector can write a report indicating the work is substandard and later gets to write another report indicating the work is corrected and the sign off is complete. Everyone gets to go home happy at that point.

sneak2• 3 years ago

"just remove the duck"

https://rachelbythebay.com/w/2013/06/05/duck/

unethical_ban• 3 years ago

I learned this lesson dealing with auditors many times. These people whose job really should be collaborative in nature (internally) still feel the need to find something, because if they don't, then their bosses say "did you look hard enough?"

So yeah, sometimes as we presented our security policies or documentation, we would highlight an existing problem for which we had a solution or proper executive backing to justify, so that when it was called out, we would haggle and then say "okay, let's fix that". Everyone had done their job.

Off-topic: I was blocked from reading that site using Proton's TX datacenters, but it worked as soon as I switched to Sweden.

Stop blocking VPNs, Rachel.

rbirkby• 3 years ago

It’s known as Atwood’s Duck. https://blog.codinghorror.com/new-programming-jargon/

vidarh• 3 years ago

Hairy arms: https://www.npr.org/2014/11/17/364760847/whats-with-all-of-t...

Though I believe the technique has been given many other names as well.

martincmartin• 3 years ago

"When preparing a defense, the good Samurai doesn't leave any weakness. The great Samurai leaves one weakness, so he knows from which direction the enemy will attack."

I was given that advice when preparing for my Ph.D. defense. :)

ilamont• 3 years ago

The worst is when they can't find anything and make shit up.

Had this happen during an audit very early in my working life - "where did this excess income come from?" The books I gave them were clean. Fortunately the accountant made an obvious math error that was easy for me to spot, and the next round went smoothly.

Or was it an error? Hmmm ...

itsoktocry• 3 years ago

A friend in grad. school would put obvious glaring errors in his preliminary drafts of write-ups so the prof's would then be happy to correct these, feel they'd done their work, and move on, avoiding any actual substantive criticisms requiring actual work on his part to fix.

There are people that believe Elon can do no wrong and is always playing 8d chess.

tetrep• 3 years ago

Are you implying that SpaceX purposefully failed in order to give the FCC something to critique?

Gibbon1• 3 years ago

I saw an interview with a movie producer from the code era of movies. He said they dealt with the Hays Commission and the Catholic Legion of Decency by including a risque scene in the preliminary cuts they submitted. They censors would reject them, and they'd resubmit with the scenes cut.

lgats• 3 years ago

The FAA didn’t release the full mishap report, citing “proprietary data and U.S. Export Control information,” but it did specify that the closure of this investigation “does not signal an immediate resumption of Starship launches at Boca Chica.” https://techcrunch.com/2023/09/08/faa-recommends-63-correcti...

https://www.faa.gov/media/70901

MR4D• 3 years ago

“does not signal an immediate resumption of Starship launches at Boca Chica.”

The FAA was not moving forward with a launch license until it was closed. So this is a big deal, even if it takes some time to get the license.

rnk• 3 years ago

I've been trying to figure out if launching from boca chica is going to be a continuing thing for years, or will they transition to Florida when it stops blowing up so much and transitions into regular launches? I haven't found any details about the long term vision of the boca chica site. My speculation is they will need 3-5+ years of test launches from there as they test out their extensive technology road map before they reach some near operational level (landing starships, turning it around, being a fuel carrier, refueling, then the landing tests for the moon, and much longer term mars development). But when that happens, will they make commercial launches from there? There aren't that many places to do that in the US, Florida of course, what about the New Mexico space port, or Vandenburg? Hawaii, some southern US ocean site?

I've thought about buying (or maybe renting) one of the huge number of cheapo condos there so I could watch some of these launches. I could even imagine working from there remotely instead of sitting at home watching on the net

avmich2• 3 years ago

Unless some plans have drastically changed, the idea with Starship is to launch a lot from many locations on Earth.

Recent reports mention Gwynne Shotwell describing the factory being built at Starbase as capable of producing around 3 Starships per week. If such a factory works for a year (which doesn't look like a long time for a factory) that's ~150 Starships. 150 Starships each have a chance to launch about twice a year if the launches are daily.

Surely numbers should be modified, some of them don't look right. First, we are probably going to have (way) more than 150 Starships. Second, a Starship is likely going to fly (way) more than twice a year. It's not a Space Shuttle after all. Next, what about those plans - Moon flights, which need refueling on low Earth orbit; Mars flights which need the same; all other flights - like orbital stations, sending probes to planets, space telescopes; finally, business class-like suborbital hops for passengers on Earth?

All of that requires a lot of places to launch Starships. Unless some plans drastically scaled down, it seems like we'll have to see an industry for launchpads growing and prospering in the near future.

darklycan512• 3 years ago

And how would that work exactly? they had to redesign the entire launchpad just to hopefully be able to sustain starship.

Let's say you go down in london (originally they thought basically a generic spacesport would let you, which is no longer the case).

You'd need their entire boca chica infrastructure to fly that again...

Beached• 3 years ago

what is more likely to happen is that they have 15x Launches for one mission, due to all the refueling needs. all these launches happen in rapid iteration from one to 3 pads over 1, 2 or 3 days. the. everything sits silent for a few weeks to a month. repeat 10 to 20 times a year.

adolph• 3 years ago

The Florida Starship factories and launch sites are in progress but watchers (NasaSpaceFlight channel) have reported things going on hold recently as Boca Chica makes changes.

My presumption is BC will be R&D-ish and FL PRD-ish with Vandenburg coming online after FL. If I had to bet, I'd bet on FL being where the first crewed Starship launches but BC where that vehicle class is launched first.

As a result I think the tempo in BC will be more irregular than FL where two Falcon launches in a week aren't rare.

Geee• 3 years ago

Boca Chica is just for testing, afaik.

dotnet005• 3 years ago

Initially they seemed to aim to make Starbase a primary launch site on par with KSC. However, since the bureaucracy has been proving to be ridiculous (as is the case with most American megaprojects these days), they appear to have shifted towards treating it as the R&D location with KSC being the primary site for operational launches.

mikeyouse• 3 years ago

Some say "bureaucracy" some say "balance of priorities between the environment, local residents, and private enterprise". Elon's envisioned a dozen+ starship launches per day -- there are tens of thousands of people (and a wildlife refuge) near the Boca Chica site that rightly have a say in whether that's a good idea.

ramesh31• 3 years ago

The FAA said Friday, Sept. 8, that multiple problems led to the launch explosion on April 20 over the Gulf of Mexico, off the South Texas coast.

What are the odds Elon pushed them to make an unsafe launch to hit his stupid little 4/20 fixation? I'm guessing somewhere around 100%.

dotnet005• 3 years ago

This has been repeatedly peddled by ignorant folk who don't care about spaceflight beyond adding in unnecessary snark but it should be obvious to anyone who paid attention that it was sheer coincidence.

bryanlarsen3• 3 years ago

The initial launch attempt was 4/17.

chasd004• 3 years ago

what was unsafe about it? I think the biggest health hazard was the dust on South Padre which turned out to be just beach sand.

FireBeyond• 3 years ago

There was a little more than "dust". There are plenty of videos of concrete chunks of 5-20 lb in weight being thrown over a mile and raining down on the beach.

valine• 3 years ago

The launch was scheduled for 4/17, 4/20 was a backup window. If anything they delayed the launch to hit 4/20, not rushed it.

Dig1t• 3 years ago

Hopefully the turnaround is quick.

Starship is literally sitting on the pad waiting for the bureaucrats to give the okay to launch.

Presumably they have already addressed all the concerns in the doc that the article is talking about.

https://x.com/elonmusk/status/1699233677979390280?s=46

dpifke3• 3 years ago

SpaceX's response: https://www.spacex.com/updates/index.html

Sounds like they've already implemented most of the fixes.

justinclift2• 3 years ago

The latest update there is from May 10th, or has something been pulled?

bryanlarsen3• 3 years ago

Cutting and pasting in case you're right:

September 8, 2023

UPGRADES AHEAD OF STARSHIP’S SECOND FLIGHT TEST

The first flight test of a fully integrated Starship and Super Heavy was a critical step in advancing the capabilities of the most powerful launch system ever developed. Starship’s first flight test provided numerous lessons learned that are directly contributing to several upgrades being made to both the vehicle and ground infrastructure to improve the probability of success on future Starship flights. This rapid iterative development approach has been the basis for all of SpaceX’s major innovative advancements, including Falcon, Dragon, and Starlink. SpaceX has led the investigation efforts following the flight with oversight from the FAA and participation from NASA and the National Transportation and Safety Board.

Starship and Super Heavy successfully lifted off for the first time on April 20, 2023 at 8:33 a.m. CT (13:33:09 UTC) from the orbital launch pad at Starbase in Texas. Starship climbed to a maximum altitude of ~39 km (24 mi) over the Gulf of Mexico. During ascent, the vehicle sustained fires from leaking propellant in the aft end of the Super Heavy booster, which eventually severed connection with the vehicle’s primary flight computer. This led to a loss of communications to the majority of booster engines and, ultimately, control of the vehicle. SpaceX has since implemented leak mitigations and improved testing on both engine and booster hardware. As an additional corrective action, SpaceX has significantly expanded Super Heavy’s pre-existing fire suppression system in order to mitigate against future engine bay fires.

The Autonomous Flight Safety System (AFSS) automatically issued a destruct command, which fired all detonators as expected, after the vehicle deviated from the expected trajectory, lost altitude and began to tumble. After an unexpected delay following AFSS activation, Starship ultimately broke up 237.474 seconds after engine ignition. SpaceX has enhanced and requalified the AFSS to improve system reliability.

SpaceX is also implementing a full suite of system performance upgrades unrelated to any issues observed during the first flight test. For example, SpaceX has built and tested a hot-stage separation system, in which Starship’s second stage engines will ignite to push the ship away from the booster. Additionally, SpaceX has engineered a new electronic Thrust Vector Control (TVC) system for Super Heavy Raptor engines. Using fully electric motors, the new system has fewer potential points of failure and is significantly more energy efficient than traditional hydraulic systems.

SpaceX also made significant upgrades to the orbital launch mount and pad system in order to prevent a recurrence of the pad foundation failure observed during the first flight test. These upgrades include significant reinforcements to the pad foundation and the addition of a flame deflector, which SpaceX has successfully tested multiple times.

Testing development flight hardware in a flight environment is what enables our teams to quickly learn and execute design changes and hardware upgrades to improve the probability of success in the future. We learned a tremendous amount about the vehicle and ground systems during Starship’s first flight test. Recursive improvement is essential as we work to build a fully reusable launch system capable of carrying satellites, payloads, crew, and cargo to a variety of orbits and Earth, lunar, or Martian landing sites.

dpifke3• 3 years ago

I see:

September 8, 2023

UPGRADES AHEAD OF STARSHIP’S SECOND FLIGHT TEST

macinjosh• 3 years ago

This article was at least partly written by AI. The give away is that after every use of an imperial unit of distance a metric conversion comes right after in parens. This happens even after unspecific uses like:

    “… for miles (kilometers) around…”

That makes no sense and only a bot would do that.

danw19793• 3 years ago

My favourite fact about super heavy is that it generates about 100-130GW of power at launch, which is roughly twice the peak demand of the United Kingdom’s electricity grid.

resolutebat• 3 years ago

The UK grid is generating 26 gigawatts as I type this: https://grid.iamkate.com/

danw19793• 3 years ago

Corrected my mistake. Thanks !

stratom• 3 years ago

130kW can't be true, maybe a typo?

danw19793• 3 years ago

yes, I meant gigawatts !

GMoromisato• 3 years ago

A few observations (which many of you might already know):

1. SpaceX is the one who wrote the mishap report. The FAA approved it, meaning they accepted that it conformed to regulations. In particular, it's SpaceX who came up with the list of 63 corrective actions--not the FAA.

2. That means that SpaceX has had a chance to implement those corrective actions, so I expect many of them (like the Automated Flight Termination System) are already done.

3. The FAA letter says that SpaceX must implement all "corrective actions directly tied to public safety". That implies that any corrective actions /not/ tied to public safety can be implemented later.

All in all, I see this is good news for SpaceX and IFT-2.

https://www.faa.gov/media/70901

dmurray• 3 years ago

So, does this mean the FAA letter is the weakest it could possibly have been? Or was it plausible they could have said "we know you identified 63 corrective actions, but don't worry about that - so long as you do #3, #18, #31 and make a start on #40 you're good to go on the next launch"?

tonyhb• 3 years ago

To be fair this is the same as most regulatory bodies. You have specific things you must do for SOC2. There can also be known issues that you must document and, as long as they fall within acceptable risk tolerances and you have a plan to fix them within some defined time period, it can be acceptable (obvs, again, depending on the risks).

It's not perfect, but then again... nothing is. Personally I think that being honest about all risks, documenting them, and having a plan in place to fix things is a good place to be.

Of course, FAA, aviation, space, healthcare etc. should have much lower tolerance to risk, so you'd expect that the portion of these that can be ignored should be very low risk.

mlindner2• 3 years ago

I'm not sure characterizing it as "weak" would be accurate. Well functioning government bureaucracy acts in kind of a mechanical fashion with very strictly used language. It's more accurate to say that it contains neither positive language toward SpaceX nor negative language toward SpaceX. It simply _is_.

RecycledEle• 3 years ago

Look at the big concerns:

1. Inadequate launch pad let debris fly. - Fixed as proven by SpaceX's static fires.

2. Leaky fuel tank caused all kinds of problems. - Fixed by fixing the launch pad.

3. Faulty FTS - Obviously a huge priority at SpaceX, so probably fixed.

sbuttgereit3• 3 years ago

This. Thanks for actually... you know... reporting accurately on the subject. It's interesting to see how this is characterized in the press by professional journalists.

Bloomberg: "SpaceX Ordered by FAA to Make Fixes After Starship Blowup : -- FAA blames ‘multiple root causes’ for flawed Starship launch -- SpaceX must apply for, receive FAA license modification"

CNBC: "FAA orders Musk’s SpaceX to take 63 corrective actions on Starship, keeps rocket grounded"

NYT: "F.A.A. Spells Out Needed Fixes for SpaceX’s Starship Rocket"

But then there's the FAA: "Following the launch, the FAA, consistent with its statutory authority [...], required SpaceX to conduct a mishap investigation in accordance with its approved mishap plan under FAA oversight.

[...]

The FAA has been provided with sufficient information and accepts the root causes and corrective actions described in the mishap report. Consequently, the FAA considers the mishap investigation that SpaceX was required to complete to be concluded.

[...]

Launch license VOL 23-129 for Starship authorized SpaceX to conduct one launch. SpaceX is required to apply for a modification to the VOL 23-129 license to allow for subsequent launches. When SpaceX applies for this modification, it will need to demonstrate compliance with 450.173(f) by evidencing the implementation of corrective actions adopted in response to its April 20, 2023 mishap."

As far as I can tell, these journalists looked at the FAA press release, and didn't take the extra 5 minutes to read the linked letter to SpaceX, or the extra hour or so to read the link to how the FAA handles mishap investigations. Add in some pre-existing feelings about Musk or his endeavors and.... BEHOLD! World Class Reporting! And this is the sort of thing that any number of governments are trying to force funding for?

Reading the FAA letter, it looks like nothing more than the routine and procedural review a regulatory agency does when processing paperwork. And everybody probably knew this mishap investigation process would taking place even before the launch just because of the likelihood of the launch actually going to plan.

grecy6• 3 years ago

There's clearly A LOT of money in play with regards to Tesla and SpaceX succeeding, and I think it has become extremely common to bash both companies at every possible opportunity.

The old money absolutely does not want these two companies to succeed.

refulgentis3• 3 years ago

This is BS, NYT spells this out exactly, and if I had to bet, based on tone, CNBC/Bloomberg shared similar info.

There's nothing mean or personal about Musk in the NYT either, I don't know what you possibly could have seen to imply that they're journalists publishing info punishing SpaceX and including info about Musk to denigrate him and that it's all so unfair because anyone would beg for SpaceX.

reaperman• 3 years ago

This probably doesn't add anything to the discussion but fun to note a name collision: Individuals with Disabilities Education Act (IDEA) [0]

0: https://en.wikipedia.org/wiki/Individuals_with_Disabilities_...

dpifke3• 3 years ago

The list of fixes: https://twitter.com/elonmusk/status/1700789411279966339

FAA closes SpaceX Starship mishap investigation

Related Stories