Alaska Airlines flight 1282 NTSB preliminary report [pdf]

https://www.ntsb.gov/investigations/Documents/DCA24MA063%20Preliminary%20report.pdf

by tomalpha • 2 years ago

266 257 2 years ago

ntsb.gov

mef• 2 years ago

anyone have a copy hosted? it's currently overloaded and not responding

hipadev23• 2 years ago

I believe this video mostly covers it: https://youtu.be/3m5qxZm_JqM

ilyagr• 2 years ago

This is old satire. It's good satire, but don't expect to find actual information here.

geoffeg• 2 years ago

https://aviationsourcenews.com/wp-content/uploads/2024/02/Re...

mcmatterson• 2 years ago

The fact that a critical piece of the evidence was cell phone photos sent between workers coordinating door re-assembly doesn't exactly instill a whole lot of confidence in their permit-to-work process. I didn't like it when it was medical teams doing shift handover via a Google Doc, and I don't like it when it's a matter of flight safety either. Or, as Homer might eruditely say: "guess I forgot to put the bolts back in" [1]

[1] (https://www.youtube.com/watch?v=IiNPLIauEig)

gowings97• 2 years ago

The data/photos should be in the ERP/MES.

ipython• 2 years ago

This is a puzzling attitude to me. Every time we technologists see a crappy proprietary solution being used for a problem, the first exclamation is, "why not use <commodity solution X>? That's so dumb, they spent $10k on that tool when they could have spent $100 on X!"

There must be a middle ground here- the paradox is that Google, Apple, etc have this ability to generate user friendly software and hardware at scale. But they aren't considered "battle proven". The expensive proprietary systems that are used instead tend to be hard to use and brittle, so what's the middle ground?

michael19992• 2 years ago

The issue here isn't using google chat, the accusation is that this was Spirit and Boeing conspiring to not record these in the proper work order system under the pretence that this work was being done by Spirit as-if-it-were pre-delivery.

Read https://www.airlinepilotforums.com/safety/146074-boeing-inte...

And then this from the doc: "The investigation continues to determine what manufacturing documents were used to authorize the opening and closing of the left MED plug during the rivet rework."

https://s3.documentcloud.org/documents/24410269/report_dca24...

imoverclocked4• 2 years ago

The investigation continues to determine what manufacturing documents were used to authorize the opening and closing of the left MED plug during the rivet rework.

I mean, there is already a ton of documentation and process surrounding the construction of an airplane. Adding more process doesn't safety make. Having a safety culture without the fear of retaliation, on the other hand, makes a world of difference.

TillE• 2 years ago

That line stood out to me, because it implies that no proper "manufacturing document" was used for the work. If that's true, that's very bad; unapproved maintenance procedures have been the cause of multiple crashes.

seo-speedwagon• 2 years ago

If the door was removed (which the NTSB report and the whistleblower post linked elsewhere around here say must have happened) there should be documentation for, at minimum, the removal and reattachment. If the door was not removed but was opened and closed, there should be documentation for both of those actions instead.

I don't know if this should be considered "adding more process" because it has been standard process for a very long time. All work done on an airplane is authorized, by someone, and after completion is recorded, by someone. Discrepancies and deviations from this standard operating procedure are a big deal.

spdustin2• 2 years ago

Also posted on DocumentCloud, since NTSB servers aren't responding (as of this comment)

https://s3.documentcloud.org/documents/24410269/report_dca24...

guardiangod• 2 years ago

[evidences] indicate that the four bolts that prevent upward movement of the MED plug were missing before the MED plug moved upward off the stop pads.

Photos from the interior repair that show the lack of bolts

Huh. Well that's conclusive.

ezfe2• 2 years ago

Photos may not have been after conclusion of repair so technically not conclusive. However it certainly lines up.

Rapzid4• 2 years ago

Not the photos but the fact that the bolts are missing and there are are no witness marks indicating they were present when the door ejected.

flaminHotSpeedo2• 2 years ago

The report makes it clear that the photo was taken after the rivet rework was complete, immediately before restoration of the interior

nostromo• 2 years ago

Overall, the observed damage patterns and absence of contact damage or deformation around holes associated with the vertical movement arrestor bolts and upper guide track bolts in the upper guide fittings, hinge fittings, and recovered aft lower hinge guide fitting indicate that the four bolts that prevent upward movement of the MED plug were missing before the MED plug moved upward off the stop pads.

Ooofff. No bolts at all! How did this pass Boeing QA?

krona• 2 years ago

How did this pass Boeing QA?

https://www.airlinepilotforums.com/safety/146074-boeing-inte...

jbverschoor3• 2 years ago

Just comment out the tests and they pass

function_seven• 2 years ago

Great link!

I like how that comment is from an anonymous source, but now that the NTSB preliminary report is out, it seems thoroughly corroborated to me. The dates of certain events and the reason for the door's removal—er, "opening"—both match the comment in your link.

Thanks.

koyote2• 2 years ago

So were the bolts missing because the Spirit team did not know they had to be put back (i.e. it was not recorded as a task that needed to be done) or was that simply just another mistake in the long line of mistakes they've made?

Reading through that post gave me nightmares of dealing with outsourcing software teams where you send them a small issue to fix and their fix breaks 3 existing items.

Rapzid4• 2 years ago

It's crazy their supplier is delivering so many must-fix defects they have to have a warranty team on site to help fix them. Crazier that Boeing just let that get more and more out of hand instead of making Spirit get their shit together on their end.

https://www.youtube.com/watch?v=xIAfCupuZ3w

Edit: Also that entire culture and dynamic between Boeing and Spirit on the production floor seems very toxic and driven by misaligned incentives. There should be zero place for bickering, aggressiveness, and finger pointing in that SAT channel. If something needs to be fixed Spirit needs to fix it with a smile. If Boeing and Spirit need to review who is responsible for what, and what needs fixing and doesn't, that should happen in review in a different setting. The production crews need to be able to execute on their processes and focus on the quality of the product without having timeline and budget concerns seeping into their day-to-day.

__loam2• 2 years ago

MBAs prioritizing the bottom line over engineering culture.

luxuryballs• 2 years ago

Also I’m kinda surprised an airline wouldn’t inspect a newly acquired plane before putting customers in it.

chx• 2 years ago

The checklist for that is very different from what the manufacturer would do. It would take a lot of time and therefore be very costly to verify every single bolt, rivet and weld was done correctly.

newZWhoDis• 2 years ago

Seems to solidly confirm the leak.

aftbit2• 2 years ago

The accident airplane was required to be equipped with a CVR that retained, at minimum, the last 2 hours of audio information, including flight crew communications and other sounds inside the cockpit.
The CVR was downloaded successfully; however, it was determined that the audio from the accident flight had been overwritten. The CVR circuit breaker had not been manually deactivated after the airplane landed following the accident in time to preserve the accident flight recording.

Classic. If they use CD quality audio at 1411kbps, they can store 2 hours of audio in about 1.2 GB. Given how cheap flash is these days, why not 20x that so that we don't have to rely on people pulling circuit breakers after accidents? If there's some concern about robustness and recertification, why not require all aircraft to carry two CVRs, one of the old "robust" style for kinetic accidents, and one that's less robust but has 20x the capacity, so we can record a full day after less violent accidents?

cjbprime5• 2 years ago

The largest US pilots union opposes it on pilot privacy grounds. (To be clear, I think having an expectation of vocal privacy while you are in charge of an airliner is absurd.)

CPLX• 2 years ago

Do you have a voice recording of you doing your entire job, every day of your life?

imoverclocked4• 2 years ago

Well, there is the theory and then there is the reality.

Theory: having less privacy makes things easier for accident investigators, post-mortem.

Reality: In this case, the pilots did their job and got the plane down safely despite rapid depressurization and literally having their headsets sucked off of their heads. It is extremely unlikely to be pilot-error that a door-plug ripped off the airframe at 16,000' or that investigators would learn anything significant from the process in the flight-deck before or after the incident. At least nothing that would root-cause this incident.

hencq• 2 years ago

I think there's some validity to the privacy concerns, but it seems those could be addressed with proper access controls and rules. The recordings should only really be listened to in the aftermath of an accident, in which case, as you say, the expectation of privacy should (in my opinion) take a backseat.

salad-tycoon• 2 years ago

My wife ridicules me because when we went out to eat, before a multitude of children , I would often say “nobody ever tipped me as a meat clerk when I was working in 45 degrees elbows deep throwing away and scraping rotting meat from the shelves and gutters and then serving ‘fresh shrimp ‘ and organic grass fed filet mignon” when I felt expected to tip 20% for an already over priced meal.

As my first boss, meat clerk young lady, told me “shit rolls down hill.” More powerful people tend to get shitted on less. It was a motivation to move up.

But I still think it’s shitting on people to expect or accept constant recording of everything mundane thing while awaiting the exceptional [screw up]. Pilots are more powerful than Amazon warehouse workers but recording every breath, every whisper, ever fart is undoubtedly shit in a warehouse or a cockpit or an operating room.

Then again, the only way I could accept it is if everyone is recorded all the time and it was all public or at least FOIA able for many people. Especially the government and universities and Wall Street other wise it’s just a way to control and hang things over peoples heads.

As to the tipping grumpiness I grew up partly in the 3rd world where tipping 50 cents was a great tip and I’m cheap and didn’t/don’t make tech bro money. I found the ultimate solution was to just not eat out so much except for truly special occasions. I’m sure there’s a lesson in there too somewhere.

zardo• 2 years ago

Privacy from an NTSB accident investigation is absurd. Privacy from your boss snooping you is reasonable.

SilasX• 2 years ago

Reminds me of an exchange from Stranger Things (S4E3):

School counselor: Max, I'm… I'm sorry, I… I really can't discuss this. You wouldn't want me talking to any other students about you, right?

Max: If I were dead and it would help catch the killer, then yeah, I most definitely would.

https://subslikescript.com/series/Stranger_Things-4574334/se...

p_l10• 2 years ago

Make it so they don't have reasonable suspicion CVR data won't be abused by the company, and you might get somewhere. For example criminal consequences for misuse that hit C-level and possible leakers.

Sincerely, someone who had death threats partially thanks to manipulated audio record that was done in good faith during investigation, which was leaked and edited later by third party who gained access to it 5 years later.

samstave• 2 years ago

Define. What EXACTLY does a pilot need privacy for?

dmitrygr2• 2 years ago

  > Given how cheap flash is these days

How cheap is flash that will survive a sudden stop from 400mph to 0 mph in no seconds flat, will survive a post-crash fire, and/or submersion for years in salt water?

Flash data retention at high temps is TERRIBLE (and gets worse for MLC/TLC/etc), see any flash datasheet. It is NOT nearly as simple a problem as you might think.

Yes, it is a solvable problem, but please do not dismiss it so outright as "trivial"

N19PEDL2• 2 years ago

Read it more carefully.

ammar2• 2 years ago

This isn't a technical limitation though, the European standard for airplanes newer than 2021 is in fact 25 hours [1].

[1] https://mentourpilot.com/who-doesnt-want-25-hour-cockpit-voi...

margalabargala2• 2 years ago

I don't think the problem you're describing is actually a problem.

Exposure to super-high temps occurs in a small set of circumstances, all of which overlap with the destruction of the recording device and the cessation of incoming data. So we only need the same 1.2GB (or whatever) of high-temperature-tolerant storage.

The 25 hour storage can be on normal flash, as if we're more than 2 hours past the incident and data is continuing to come in, then the incident of interest did not destroy the airplane, and the flash will have remained within its normal operating parameters.

antisthenes• 2 years ago

What you described is not a data retention problem at all.

It's a material science problem, and other forms of media are affected by high temperatures and physical deformation just as much as flash if not more.

randerson• 2 years ago

I often wonder why we still rely so heavily on local storage when in-flight Internet exists. Flight data could be streamed in real time to the cloud for redundancy.

bronco210162• 2 years ago

What’s missing from this accident investigation without the recording?

blantonl2• 2 years ago

ability to somehow claim pilot error. That's what.

sigwinch283• 2 years ago

The rule (edit: in Europe) is now 25 hours for aircraft over a certain weight, though it is not (currently) retroactively applied to existing equipment.

https://www.federalregister.gov/documents/2023/12/04/2023-26...

cjbprime5• 2 years ago

That document is an in-progress proposal to amend a rule, no? I think there was strong opposition to this rule before this accident flight, and the blowback from the missing data here might be strong enough to be able to get it passed anyway.

gruez• 2 years ago

ACTION:
Notice of proposed rulemaking (NPRM).
[...]
DATES:
Send comments on or before February 2, 2024.

Seems like it's a proposal, and not actually enacted yet?

hinkley• 2 years ago

The piece of hardware that was chosen for the avionics-adjacent software I was working on was chosen before any software was written, which was 3 years before the plane was 'supposed' to fly, and 5 years before anyone sane expected it to be in service.

Irritatingly, they didn't even pick the top-of-the-line machine from the vendor at that time. They picked a middling one. And then put an LTS OS version on it that didn't fully support the motherboard chipset. I spent way, way too much time an energy trying to get the software to run on the sort of timescales necessary. It took me months to get anyone to let me talk to the vendor in order to sort out the fact that the storage was being run in legacy PATA mode, reducing our IO throughput by an order of magnitude and the application throughput by about a third.

Ten minutes on the phone and I got them to agree to give us a patch that aliased the chipset to one it was backward compatible with, that was actually supported by the OS. But they really wanted us to take the never version of the OS that didn't have this problem.

That's not even the most hard-ware crippled I'd ever been, but it was top three.

WalterBright2• 2 years ago

They should also have a video recorder on a 2 hour loop. Many difficult investigations would have been easy if the investigators could see what the instruments were showing and what the crew was doing. And even, who exactly was in the pilot's seat!

jbverschoor3• 2 years ago

In a revision to the Flight Crew Operations Manual, issued on January 15, 2024, Boeing confirmed that the door functioned as designed.

Smells like CISCO

aftbit2• 2 years ago

Depressurization happened around 17:12:33 PST but the aircraft continued to climb until 17:13:41 PST, and the autopilot was configured for 10k ft at 17:13:56 PST. Why did it take the pilots a full minute to begin an emergency descent after the failure? I would expect that the nature of the accident would be clear nearly immediately, at least in the need to descend the aircraft.

throwworhtthrow• 2 years ago

You can't leave your assigned altitude/trajectory without coordinating with ATC. Otherwise you may collide with another plane, which would make a bad situation worse.

mazugrin2• 2 years ago

Sure you can, and they likely did start descending before contacting ATC. But before they did any of that, they had to spend some time donning their oxygen masks and doing a few other "memory items" before then descending.

lisper• 2 years ago

In an emergency you can do anything you think is necessary to address it. Source: I’m a private pilot.

luizfzs• 2 years ago

Also, running checklists for specific types of emergencies.

16bytes• 2 years ago

Aviate comes before navigate and communicate. The pilot in charge is ultimately responsible for the safety of the aircraft, not ATC.

bronco210162• 2 years ago

Step one is to put on the oxygen mask and establish communications. After the startle factor, the masks being put on, then declaring an emergency, a minute really isn’t that long.

sigwinch283• 2 years ago

More altitude means more time to work the problem.

blantonl2• 2 years ago

I would expect that the nature of the accident would be clear nearly immediately

Not really. The cockpit door was blown open, and the pilot's headsets were blown off. It was a pretty chaotic event, and when you are flying an airplane, you definitely don't want to figuratively "jerk the wheel" - you remain calm and start running checklists.

rootusrootus2• 2 years ago

A minute is a long time when you're sitting at your computer. But after the sudden depressurization, I imagine the pilot is focused first on making sure he has complete control of the airplane, assessing the situation, running checklists. Besides, 10K is just barely above the normal pressurization altitude anyway, it doesn't pose an immediate risk to the passengers that justifies just nosediving towards the ground. Especially given how much air traffic is at lower altitudes that close to PDX.

Edit: Re-reading, it was more like 16K feet when it popped, 10K is what ATC assigned them when requested. Still low enough not to be a critical emergency. Some people absolutely will get altitude sickness at that level, but it's likely to be mild. Many people climb mountains much taller.

SoftTalker• 2 years ago

First and most importantly the pilots have to get their own oxygen masks on, if they delay this at all they will become hypoxic, unable to think clearly, then pass out, and then it's over for all on board.

bagels2• 2 years ago

Clearance + they were probably putting on their masks, and other tasks.

engcoach• 2 years ago

Fast hands in the cockpit are scary. Pilots take their time in emergencies because rushing will take your birthday away

michaelt3• 2 years ago

According to the plane's "memory items" [1] in response to a cabin altitude warning or rapid depressurization, pilots must:

OXYGEN MASKS - DON

OXYGEN REGULATORS - Set to 100%

CREW COMMUNICATIONS - ESTABLISH

PRESSURIZATION MODE SELECTOR - MAN AC/MAN

OUTFLOW VALVE SWITCH - CLOSE

Hold in CLOSE until outflow Valve indicates fully closed

If Pressurization is Not Controllable

PASSENGER SIGNS - ON

PASSENGER OXYGEN SWITCH - ON

EMERGENCY DESCENT - ANNOUNCE

The pilot flying will advise the cabin crew, on PA system, of impending rapid descent. The pilot monitoring will advise ATC and obtain area altimeter setting.

PASSENGERS SIGN - ON

DESCENT - INITIATE

I do giggle a little at the thought of a door flying off, the air rushing out of the cabin, and the pilots responding by switching the seatbelt light on.

The plane was only at 16,000 feet when it lost its door and according to [2] you've got 20-30 minutes of 'useful consciousness' at such an altitude, even without your oxygen mask on. So there was no need for an abrupt dive.

[1] https://www.theairlinepilots.com/forumarchive/b737/b737memor... [2] https://skybrary.aero/articles/time-useful-consciousness

bombcar• 2 years ago

You follow the procedure because in an emergency you don't know what is going wrong.

Better to climb for a bit more as you get your oxygen mask on than to try to descend immediately and make some problem worse.

We know it was a door plug blowing out, but in the past it has been entire major sections of the airframe ripping off, in which case sudden extra stresses are not what you want.

ogurechny• 2 years ago

Let's assume that they did change the course when they were sure they can do that.

Pilots can't look at the rear view mirror, and see the whole plane. Accident reports on engine malfunctions routinely mention that someone had to check their appearance through the passenger window, and relay that to the pilots. In case of a blast so severe that the door flies off, it is safer to assume the worst. Say, that part of the plane disintegrated because of sudden collision. In such conditions, indications on what works and what doesn't are probably really messy and unreliable, and there can be not enough means to control the plane properly. Lower the nose too much, and you might not be able to pull it up any more.

Pilots probably did checklists with one eye on the instruments to check that they were not losing speed, that angles were correct, autopilot inputs resulted in stable flying, and so on, and deduced that everything still worked. By that time, they were probably informed that the plane was seemingly intact, although with a hole in its side.

NordSteve• 2 years ago

Let's imagine you're the pilot, and you're super busy with an emergency. You also know that there are mountains to the east of the airport. You also have ATC on the radio and they know about all of the meaningful obstacles in your area. Asking for lower in this situation (rather than using your emergency authority) is exactly what you want to do.

Looking at the track, they descend to 10000' until they start their downwind to base turn. Once they start that turn, they get a lower altitude (looks like 7000') until they are established on final and can fly an approach.

dzdt• 2 years ago

The report seems to mesh with and confirm many details of the anonymous insider account at https://leehamnews.com/2024/01/15/unplanned-removal-installa.... The bolts were not reinstalled following work on the plug rivets/seal. The official system doesn't record that work was done requiring the bolts to be removed.

lambda• 2 years ago

Yep, biggest new thing here is discussion of witness marks showing no evidence of the bolts, and photo before the rivet repair showing that at least two of the bolts were present, and photo after the rivet repair during installation of the insulation showing at least 3 bolts missing.

So this all just serves to confirm that report, and what people suspected for a while; the bolts were just missing, removed for removing the door plug for the rivet rework and never reinstalled.

albert_e• 2 years ago

removed for removing the door plug

there was apparently two (identical) procedures -- one for "removing" the door plug and one for "opening" the door plug -- but only one of them actually called for paper-work and quality inspection after it is done.

It seems they decided to "open" the door plug instead of "remove" it to work on the sealing issue? -- and thereefore documentary record of the work performed and inspection thereafter is lacking.

It would be useful IMO to explore what led the team to prefer "opening" versus "removing" -- and if the subtle difference in documentation requirement was a consideration in preferring one over the other -- and if that points to deeper pervasive culture issues that leans towards less work rather than give safety and quality the primary importance.

okdood64• 2 years ago

Can someone explain to me why this door plug wasn't actually a plug that physically stays sealed from cabin pressure? That seems like a sensible failsafe?

fredoralive2• 2 years ago

Probably because it reuses parts of the design of the emergency door so it can be swung out of the way for maintenance. The emergency door uses this complex designs of stop pads and guide rails etc to have “plug” characteristics (which did hold out for 3 months without being properly secured) because a true plug door would have to swing inwards, which isn’t an ideal property for an emergency escape, and I’d guess it might need more room reserved internally.

j-a-a-p2• 2 years ago

Good point. Boeing (737) has the door open to the outside. The Airbus (320) the door opens to the inside.

I can remember when I was small my Mom mentioned that doors of buildings in the US are mandatory to open to the outside, a rule that does not exist in Europe AFAIK. So there you have it, you have a significant higher chance of crashing in a Boeing, but when it happens, your can leave the plane 2 seconds faster.

belltaco2• 2 years ago

The CVR was downloaded successfully; however, it was determined that the audio from the accident flight had been overwritten. The CVR circuit breaker had not been manually deactivated after the airplane landed following the accident in time to preserve the accident flight recording

In addition to local storage, why isn't the audio(along with location, altitude and some sensor information) also streamed using something like Starlink or Inmarsat to a secure location where you can store more data for cheaper and with more redundancy?

sv123• 2 years ago

$$$

skywhopper• 2 years ago

This is an old system that works well and reliably for pretty much every incident. I’m not aware of another case of this sort of thing (relevant flight recorder data being overwritten) happening in recent years anyway. If you spend time constantly upgrading systems like this you’re asking for a higher failure rate, for very little gain.

That said, there’s a standard and reliable 25-hour flight voice recorder that solves this problem. But it’s only used outside the US. That’s a regulatory inertia situation and I suspect this incident will speed changes in this area.

However, finally, and particularly in relation to your proposal of streaming cockpit voice recordings to some cloud server. There is some resistance to this (and to longer recordings in general) from air crew on privacy grounds. The privacy issue is less about how much personal info is revealed in a crash situation and more about how easy it would be for a bad actor in management —or whatever operations group runs the audio storage—to listen in on conversations. And you can be sure this would happen if something like your system were implemented without the appropriate regulatory controls (and tbh even with them it would probably still happen).

flutas2• 2 years ago

I’m not aware of another case of this sort of thing (relevant flight recorder data being overwritten) happening in recent years anyway.

Got me curious how often this happened.

Last example I can find of a CVR being overwritten and not just exploded/missing was in 2018 for an engine fire, similar to this where the flight had to emergency land shortly after take-off. Before that...well a lot of complete failures ("not operative at time of flight") but not many like this scenario.

https://en.wikipedia.org/wiki/List_of_unrecovered_and_unusab...

Kaxo• 2 years ago

In 2018 NTSB issued a report (https://www.ntsb.gov/investigations/AccidentReports/Reports/...) listing 17 incidents where CVR was lost due to the recording not being turned off after the incident.

And it also lists 17 more incidents where something happened in a flight and it took more than 2 hours to land so data from the incident was lost.

fredoralive2• 2 years ago

The current 2 hour limit (which is now 25 hours in Europe) is a legacy of privacy concerns. If pilots are concerned that their bosses would make a habit of yanking longer CVR units to micromanage what goes on in the cockpit (or using events several hours before an incident to somehow push blame onto the pilot for an it), they’d love the idea of it being beamed to a remote location. Yes, I’m sure there could be complicated byzantine cryptographic scheme that would theoretically solve it, but not sure they’d trust it.

There’s also bandwidth and satellite coverage not being magic of course.

outworlder2• 2 years ago

Starlink is a consumer system. Won't happen without a specialized product. Inmarsat is expensive. And we are talking about streaming audio from all planes currently in flight.

belltaco2• 2 years ago

Starlink is a consumer system

Not true.

https://arstechnica.com/information-technology/2022/10/starl...

https://cleantechnica.com/2022/07/06/fcc-approves-starlink-f...

spdustin2• 2 years ago

Looks like the anonymous whistleblower on Airline Pilot Central Forums [0] was legit.

[0] https://www.airlinepilotforums.com/safety/146074-boeing-inte...

yborg• 2 years ago

I'm guessing twice as much corporate effort is currently being deployed to find this guy than is being applied to fixing Boeing's quality system.

gtmitchell• 2 years ago

A very thorough preliminary report. I've worked for a long time in quality systems, and this is a perfect example of a systemic failure. They've got work being handed off between Boeing employees and 3rd party contractors with insufficient controls in place to verify that very basic tasks are being performed.

I'd be curious to know how many non-conformances they typically see during assembly of a plane and whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them. I'm guessing that the production personnel are under tremendous time constraints and are constantly pressure the quality assurance people to sign off on whatever paperwork is holding up the line, no matter the safety implications.

Also, I think a lot of middle and upper level management needs to lose their jobs over this. I hope this mess ends up in textbooks and gets beaten into the head of every MBA student in the country.

lp4vn2• 2 years ago

I'd be curious to know [...] whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them

If management in the aerospace industry works like management in the software industry, then I guess they are pushing for results as agressively as possible without much concern about safety or anything else.

cezart• 2 years ago

At a company I worked in, we had a joke about this: "Good thing we don't build nuclear reactors".

In some software projects the level of rush, and the fact that bugs sometimes would leak into production was kinda horrifying. It would've been way more so, if it would've been the kind of project that could kill people in case of failure. Like it happened in Chernobyl with nuclear reactors, or at Boeing with planes.

I can't really imagine what these engineers feel when they rush this kind of work knowing what's at stake.

trompetenaccoun• 2 years ago

That's not how it traditionally worked in aerospace. Commercial airliners are such a safe mode of transportation because from start to finish there are (or used to be?) very high safety standards all around. I can't speak for what's going on with Boeing currently though. Things certainly seem to have deteriorated.

shadowgovt• 2 years ago

Rumor has it the controls are there, but subvert-able.

Apparently, there are two ticketing systems (one for "history of plane," Boeing internal, and one for "day-to-day onsite work," visible by contractors and Boeing management). The work to fix the rivets was logged in the day-to-day, but management and the onsite staff managed to convince themselves that merely opening the plug to fix the vacuum-seal trim did not constitute "removing" the plug, and since there was only an entry in the history-of-plane log for removing, not opening, they didn't log it there (when the intent was "there's no entry for 'just opening' because there's no such thing as 'just opening', breaching the pressure vessel at all constitutes 'removal of plug'").

The final inspection that should have caught the error would have been triggered by the update in the history-of-plane ticketing queue.

(And as for 'how many non-conformances,' the same source claims that Spirit is one of the few subcontractors with on-site staff at the factory because their parent company delivers such consistently shoddy out-of-compliance product that they are continuously doing final warrenty-work onsite. So maybe "fire that vendor" should be on the docket too).

ethbr13• 2 years ago

You might want to cite your source on this, which I'm guessing is the purported insider speaking about same?

From the portions of the report that hint at corroboration,

Documents and photos show that to perform the replacement of the damaged rivets, access to the rivets required opening the left MED plug (see figure 15). To open the MED plug, the two vertical movement arrestor bolts and two upper guide track bolts had to be removed.
Records show the rivets were replaced per engineering requirements on Non-Conformance (NC) Order 145-8987-RSHK-1296-002NC completed on September 19, 2023, by Spirit AeroSystems personnel. Photo documentation obtained from Boeing shows evidence of the left-hand MED plug closed with no retention hardware (bolts) in the three visible locations (the aft upper guide track is covered with insulation and cannot be seen in the photo)

onetimeuse923042• 2 years ago

I'd be curious to know how many non-conformances they typically see during assembly of a plane (...)

Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible.

For example, better quality control usually results in an increase of number of defects, at least temporarily. But that just because large portion of these defects were undetected before.

So... you are looking at a number that you have nothing to compare to that also depends on how closely the process is monitored and also depends a lot on the definition of what is non-conformance.

It is like trying to give an answer to "what is the length of Britain's coastline?" Everybody knows that you can get whatever answer you want depending on how long the ruler is.

outworlder2• 2 years ago

Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible.

That, by itself, should be the kind of thing that should shutdown a company permanently. Remember, this is the aviation industry, where they track the mine where the ore from the bolt was mined, and who tightened the bolt up and with which torque wrench.

javawizard• 2 years ago

This sort of thing always reminds me of this Dilbert comic: https://twitter.com/k8em0/status/1078824013843984384

mihaaly5• 2 years ago

I have tendency to hope for unrealistic things too. Not a reliable trait of mine, no. But in my clear moments I am afraid that much sooner will come the world peace and union of all nations and religions than unimaginative but determined bean counters learn from millions of catastrophes of the past and future to come giving up pushing their core value and first rule of 'take more, give less' into the infinity and beyond. And giving up their personal wealth with it.

zmgsabst3• 2 years ago

I think the hope is people of good conscience stop being political cowards.

If you believe in quality engineering, but refuse to engage in the political and business dimensions of an enterprise to fight for that view, then you are just virtue signaling — since you’re refusing to engage with the tools needed to make it happen.

michaelt3• 2 years ago

I'd be curious to know how many non-conformances they typically see during assembly of a plane

Well, the report says "During the build process, one quality notification (QN NW0002407062) was noted indicating the seal flushness was out of tolerance by 0.01 inches.

So I'd say they've had about 2,407,062 quality issues :)

Arrath• 2 years ago

Then its revealed "Well the numbering schema started at QN AXX... and we looped through the alphabet a few times.."

wmidwestranger3• 2 years ago

Also, I think a lot of middle and upper level management needs to lose their jobs over this.

Given that if a worker doing the work raised this problem or took initiative to resolve it, they'ld probably be punished; I completely agree.

Reminds me so much of how, once-upon-a-time, there seemed to be actual engineering management in a cooperatively adversarial relationship with business managers but not anymore. Now any sort of engineering in business seems to be completely business managed and business minded. I'm sure it's great for profits while it lasts but I haven't observed engineering becoming better and I suspect business is suffering by overextending itself too, I just don't have any solid observations.

(Well, maybe one, my brother does warehouse / logistics management and says, despite there being every reason in the world, he has never seen the accounting software and the inventory software successfully and productively linked. So, big opportunity there for a serious player but maybe not the profitable compared to the issue?)

wolpoli• 2 years ago

Once upon a time, QAs and developers also had a bit of cooperatively adversarial relationship as well. It's no more now as teams are restructured by business, so they cooperate.

hipsterstal1n• 2 years ago

I was speaking with some friends at a Christmas party who work for the Navy - they’ve taken deliveries of planes from Boeing with the same sort of issues that start in the factory. They even went as far as to say the whole lot of planes should’ve been rejected but weren’t. Multiple things not built to spec.

The planes they worked on did not share an assembly line with the 737 but another Boeing model…

fransje263• 2 years ago

If I recall properly, sometime after the MAX crash saga, the Air Force simply refused to take delivery of their 767s because of unacceptable build issues, including foreign debris (aluminium rests and shavings) and forgotten tools flying around.

I didn't follow how that issue evolved.

fransje263• 2 years ago

According to the "insider source", 392 non-conforming defects in the fuselage door installation in the last 365 calendar days.

As a result, this check job that should find minimal defects has in the past 365 calendar days recorded 392 nonconforming findings on 737 mid fuselage door installations (so both actual doors for the high density configs, and plugs like the one that blew out). That is a hideously high and very alarming number, and if our quality system on 737 was healthy, it would have stopped the line and driven the issue back to supplier after the first few instances.

Source:

https://leehamnews.com/2024/01/15/unplanned-removal-installa...

TurkishPoptart2• 2 years ago

"....after the left mid exit door (MED) plug departed the airplane leading to a rapid decompression"

Lol, they said the door plug "departed" instead of "blew the f* off"

ultimoo• 2 years ago

I look forward to reading a report from NTSB's internet outage.

Hansenq• 2 years ago

If people are looking for additional in-depth reading on how this happened, The Air Current did a great write-up on this systemic mistake using internal Boeing sources a month ago that the NTSB report fully supports: https://theaircurrent.com/aviation-safety/127-days-the-anato...

hn83058232• 2 years ago

I wonder how close the door plug was to hitting the tailplane or vertical stabilizer/rudder?

reddit_clone• 2 years ago

Considering all those scary scenarios, what happened was probably the most favourable outcome. It could have been a major disaster hundred different ways..

mikeyouse• 2 years ago

It sure could have -- the plane was still climbing which puts that plug door almost directly in line with the horizontal stabilizers;

https://i.cbc.ca/1.7077373.1704733027!/fileImage/httpImage/g...

WatchDog• 2 years ago

The flight crew reported that the cockpit door had opened during the depressurization event. In a revision to the Flight Crew Operations Manual, issued on January 15, 2024, Boeing confirmed that the door functioned as designed.

Interesting for terrorists. Cause a rapid decompression, and get easy access to the cockpit.

ledauphin• 2 years ago

yeah, this is a significant hole in the locked cockpit door plan.

bradfa• 2 years ago

Causing rapid decompression is quite hard. Opening a normal door is very difficult during flight except at very low altitudes.

mlindner2• 2 years ago

If they cause a rapid decompression they incapacitate themselves and won't be able to use the cockpit.

Also how do you cause a rapid decompression without a gun of some kind?

mihaaly5• 2 years ago

They may go then and blow the whole thing up good insted of such half measures. Sufficiently rapid decompressions may not that reliable and predictable to carry out so I can control the plane afterward easy kind of feats.

mihaaly5• 2 years ago

What is the analogy of leaving out all bolts from that door?

'Forgetting' to put in any of the screws holding a gas tank in place in a car?

'Missing' all welds in one of a skyscraper's lower columns?

An 'oversight' of providing rendundant instruments in an airplane with natural tendency to stall?

What a hopeless shitshow is going on there behind the company gates that these kind of things can happen in succession?

A duck forgot how to swimm, an eagle forgot how to fly, Boieing forgot how to build airplanes?

addaon• 2 years ago

'Missing' all welds in one of a skyscraper's lower columns?

Or just throwing the rebar in at random maybe? [0]

[0] https://www.gr-us.com/%E2%80%9Chorror-at-the-harmon%E2%80%9D...

mihaaly5• 2 years ago

Oh Jesus! I seriously thought my hypothetical example is extreme, but apparently not!

sofixa3• 2 years ago

According to an article in the Las Vegas Sun, “Some of the steel, known as rebar, was so badly positioned that it stuck out of the concrete floor and was sawed off to conceal the mistake.” In addition, “Harmon workers reportedly moved rebar without first getting an OK from the structural engineer . . . .which is a major no-no in the construction chain of command.

WTF! How was there no punishment for everyone involved in this?

johnflan• 2 years ago

You gotta wonder how the technician explained away the four leftover bolts after

userbinator• 2 years ago

A lot of comments here are going on about process, as if humans are mindless and otherwise perfectly controllable robots...

I'm going to be contrarian and say that this is exactly the sort of thing that happens when you train humans to be robots: They lose all signs of common sense and critical thinking, and what's worse is that on top of that, they'll still have their inherent imperfection. Normally the former would counteract the latter, but not if you only make them rigidly follow some process all the time. They stop thinking about what they're doing. They stop paying attention to all the other things in their environment they would've noticed, and even if they do, they won't question it because they'll just assume someone else also following a rigid process will take care of it. They won't think "this door plug should've been bolted in place now that the work that needed it opened is done, but where are the bolts?"

I'm not saying to throw out all the process and make them figure everything out, but I think there has to be a balance, similar to how overautomation and reliance on that has also lead to avoidable incidents in aviation.

projektfu2• 2 years ago

Potentially the result when you rob workers of the right to pride in workmanship. The most common complaint from old Boeing people who have left is that after the merger the McDonnell-Douglas people took over and the company switched from pride in engineering and quality of workmanship to cost cutting and bean counting. Also, shortly after the merger the corporate HQ moved, reflecting the priorities of the CEO. It has since moved again, apparently to be better for lobbying.

p_l10• 2 years ago

The original move was also related to lobbying potential.

But the CEO who started the moves etc. was a Boeing lifer.

p_l10• 2 years ago

Nope.

The process was there so that the people would know there was work being done on the doors despite not being there for it. If you see an unfinished work from a previous shift, it does not mean you can start messing with it - there might be context you do not know.

Which is why such things are supposed to be noted in appropriate ways. Similarly why aviation has so many procedures everywhere - because we know and understand that sometimes you miss things. For any human reason, not just mismanagement. The process is a way to have reliable place to double check with.

This is different from over reliance on automation, which is arguably less of an issue of automation itself (it's just more visible in such areas) as much as getting out of training because you do not encounter certain things so often. 96 people died because in a stream of many deviations, among other things, the crew never trained how to do IFR landing without ILS, autopilot or no autopilot.

The process is the part that says "yeah, I haven't done this in a long time, I need to train, here is documentation that provides we need to do it and can't delay".

Similarly CMES is supposed to track "work was done on this part of the ticket, now different work needs to be done, do not assume it will be done by other teams"

dreamcompiler• 2 years ago

Summary: Fuselage was delivered to Boeing with some damaged rivets near the door plug. They had to remove the door plug to fix the rivets. Then they reattached the door plug but forgot to reattach the 4 bolts that would keep it in place. Possibly because of a shift change at the plant.

There was noticeable damage to the door plug's mechanical fittings from the violence of it being blown out of the plane. But the holes where the bolts belonged were pristine. That would not have been true if the holes had had bolts in them.

flaminHotSpeedo2• 2 years ago

The accident airplane was required to be equipped with a CVR that retained, at minimum, the last 2 hours of audio information, including flight crew communications and other sounds inside the cockpit ... The CVR was downloaded successfully; however, it was determined that the audio from the accident flight had been overwritten. The CVR circuit breaker had not been manually deactivated after the airplane landed following the accident in time to preserve the accident flight recording

How the fuck is this still a problem on brand new aircraft?

TurkishPoptart2• 2 years ago

"....after the left mid exit door (MED) plug departed the airplane leading to a rapid decompression"

Lol, they said the door plug "departed" instead of "blew the f** off"

Alaska Airlines flight 1282 NTSB preliminary report [pdf]

Related Stories