Flying the Unflyable Plane: The near crash of Air Astana flight 1388

https://admiralcloudberg.medium.com/flying-the-unflyable-the-near-crash-of-air-astana-flight-1388-88878e2eb3c4

by mhandley • 4 years ago

105 51 4 years ago

admiralcloudberg.medium.com

Even though the author is critical of the GPIAAF's report, it's amazing to see how thoroughly incidents like this one are investigated - With a real focus on process improvement. Frustrating that the obvious mechanical fix for this (have different shaped connectors for each aerilon cable) wasn't adopted by Embraer, but still a great reminder of how we can do incident handling better in software dev

unsafecast• 4 years ago

Even something as simple as coloring them like RCA connectors would've worked. It's insane how lightly this was treated in designing it.

dingaling• 4 years ago

Investigate for cause, not blame

mannykannot7• 4 years ago

I find Embraer's rationalization for this (and, presumably, its acceptance by the relevant regulatory agencies) to be particularly galling. Even if it was true at the time that no procedure called for both cables to be disconnected at the same time, did they have any equivalent proof that no such procedure might be introduced later? Did they have any equivalent proof that no-one would misunderstand the instructions, or that it could go undetected if that happened? Could they be sure that this constraint, or its relevance, would not be overlooked at some point in the future? Well, we know they did not. Nobody addressed the question, "what could possibly go wrong?" with sufficient rigor.

MichaelZuo• 4 years ago

It's an extraordinary achievement by the pilots to keep going under the immense pressure of a 2 hour impromptu rollercoaster ride in the clouds with 6 lives on the line.

elromulous2• 4 years ago

I don't usually prefer a video over text. But if anyone is interested in a video about this, Mentour Pilot has an excellent one[1], including a bunch of details about how they ended up in this crazy situation, and how they figured out what's going on.

It's truly remarkable that they managed to land that plane, against all odds.

[1]https://youtu.be/5ywaMkMTwWk

coldcode• 4 years ago

Mentor Pilot's videos are top notch. Watching this one made me dizzy I can't imagine debugging a system while enduring 4+ Gs.

ilyt6• 4 years ago

I'd imagine it would be something akin trying to debug servers when riding rollercoaster

w10-1• 4 years ago

What reminds me of computer systems is how the spoilers partly compensated for the opposite role, the computer modes (direct vs. assisted) lead to different behaviors, the test display (green == good?) was mostly self-referential and did not indicate what they thought it should, etc.

In short, the mental model required to understand the plane and its indicia of state was way too complex. These guys literally had to read technical manuals and follow (run-book) procedures step-by-step while the plane was cavorting.

Dynamic stability is the tendency of a system to restore stability, given external forces. There should be a term for dynamic-breaking stability, the ability to restore stability when broken, while also maintaining dynamic and static stability.

This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.

ilyt6• 4 years ago

I watched a lot of accident videos and that kinda is the common trend. Many of them were caused not only by actual failure but by pilots not understanding what the plane was telling them, or plane sending mixed signals (whether actually or by pilot's misunderstandings)

I remember one accident where part of the cause was that both pilots were trying to yank the plane in different direction, and the system just averaged out both inputs and as it was fly by wire and there was no feedback and "too much to do", pilots didn't notice. That's the reason why many planes have pilot's controls connected with eachother - so one can feel what the other one is trying to do.

This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.

I have noticed the interesting trend when it comes to automating and "cattlefication" of everything - the less something breaks the less everyone involved understands it. Because it's automated and because nobody needs to even read anything to deploy it (just add a line in Puppet manifest and you're done) when it does break not even the author has a clue why. Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.

Ma8ee• 4 years ago

Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.

That is one of my pet peeves with modern development. A lot of things are way too easy to get started with, while very few people understands what is going on. And then when things doesn't work so well, there are even more tools and frameworks that can be deployed without understanding, but that in some ways mitigate the problems. But now it's even more complex, and suddenly there's a team of 20 people maintaining an application that doesn't do much more than moving some files between servers, reading them and saving records in a database. Sorry, not a database, five different types of databases that no one in any of the teams have any deeper understanding of.

mrlonglong• 4 years ago

I'm not sure why a visual check procedure was never done which would have shown the issue as part of checks after maintenance was carried out.

azalemeth• 4 years ago

Not an embraer pilot, but I suspect because (like many larger aircraft) you can't physically see the ailerons from the cockpit – that is why the big jets measure the deflection and (should) show it to you.

Part of me wonders if also as long as you see the ailerons _move_ you assume they've moved in the right direction. There have been similarly tragic rigging accidents in GA aircraft as well – and 38 incidences of aileron mis-connection in gliders between 1974 and 2014 reported in the UK. [1]

[1] https://members.gliding.co.uk/wp-content/uploads/sites/3/201...

deeblering42• 4 years ago

I would think one of the pilots would stand behind the plane and verify control surfaces via radio, but I guess that's too time consuming or error prone?

Works fine for cars though. Drive through lube shops and states with safety inspections will perform indicator checks this way. If you push the left blinker and the right light starts flashing you've got a problem...

Hopefully by now the cockpit has access to external cameras to visualize what's happening behind them in real time

ledauphin• 4 years ago

as a pilot, I have to really force myself to 'see' that the aileron is moving the correct direction. Not because it's hard, but because my lizard brain wants so badly to skip over the thing that it has seen so many times that it's no longer interesting.

mannykannot7• 4 years ago

I have not seen a glider where it is physically possible to cross-connect the controls simply through the process of de-rigging and rigging (removing and replacing the wings and stabilizer, which is often done for transport or storage), though I imagine there were examples in the past. I have, however, seen ones where the controls can be left unconnected or even insecurely connected (though such designs now seem deprecated), in which case they may move in the correct direction on the ground, but fail to do so under load (and may flutter). On account of these possibilities, the visual inspection and pull/twist test of the connections, together with positive control checks (where someone holds the surface to provide resistance), as mentioned in the article you link to, should be SOP.

twelve40• 4 years ago

Fortunately, however, that’s something only pilots will have to worry about, because the first flight after heavy maintenance is never conducted with passengers on board

whew : ) but yeah, a pretty epic save

chopin• 4 years ago

What I am curious about whether it would have been possible to deactivate the spoilers, at least in roll-assistance? From the MH-370 discussions here I had the impression that the pilot can shut-off every system.

mannykannot7• 4 years ago

I don't think the article was completely clear about this, but I was left with the impression that the solution was to always turn the yoke far enough that the spoilers overrode the ailerons. The alternative - to use reversed controls - is so against one's experience that it may not be feasible even if you know what is going on.

Are there any home-computer flight simulators (or their controllers) that can be hacked to simulate reversed ailerons?

Gordonjcp2• 4 years ago

You could probably do it in FlightGear, and modify the aircraft to model reversed ailerons but correct spoilers.

I wonder if you could do it by dicking around with the joystick inputs, giving "reversed" controls over the first little bit, then a dead spot, then correct inputs once you've moved more than a certain amount?

stavros2• 4 years ago

I don't know about simulators, but I can tell you that, out of the ten or so times I reversed the ailerons by mistake on my RC planes, zero landed without crashing.

Yes, it took me ten crashes before checking that the control surfaces move the right way before flight. Now I run a battery of tests, each one inspired by multiple crashes.

CrazyStat• 4 years ago

They eventually landed (after two failed attempts) by not touching the ailerons at all and using only the rudder.

blueflow• 4 years ago

Better source: http://avherald.com/h?article=4c04438e&opt=4096

JasonFruit• 4 years ago

The obvious solution — using connectors that cannot be fastened to the wrong cable — reminds me of Bob Hoover, an absolutely stunning test and display pilot, who had a forced landing after his fuel tank was filled with jet fuel rather than standard avgas. He devised a bell-shaped nozzle that is now universally used to dispense jet fuel, and cannot be inserted into a standard fuel filling neck.

In this case, though, better design, better tooling, and better training are probably the best fix; there are too many variations in stick-to-aileron connection for a single standard to be accepted.

tetris11• 4 years ago

I've been looking for flight path data for this flight but didn't realise that there is a severe shortage of accessible archived flight path data...

mgsouth2• 4 years ago

If you just want to see a plot, watch the Mentour Pilot video (link posted in a comment above.)

deeblering42• 4 years ago

It seems color coded cables would be a nearly zero cost solution. I wonder why identical steel cables were preferred, and how this wasn't an issue in manufacturing?

ilyt6• 4 years ago

Well, one less part on parts list assuming they are same length

and how this wasn't an issue in manufacturing?

Manufacturing does same exact thing every single time in order and then someone checks it.

But this was something staff don't do often.

elfchief• 4 years ago

I suspect having a color coding material/technique that can apply to random bits of random plane parts, that won't get rubbed off / broken off / obscured by dirt/grease/grime/etc, consistently throughout the plane, is probably harder than it appears at first glance.

hyperific• 4 years ago

This is like the unrideable bicycle on steroids... Massive respect to the pilots who managed to figure it out AND land the plane.

BWStearns2• 4 years ago

When I first started flying GA planes I did the control check and assumed that seeing the ailerons move was sufficient to check that item off. It never occurred to me that you _could_ reassemble the plane in such a way that the ailerons would be inverted. Some older pilot mentioned his friend wrecking a Mooney on takeoff that way (luckily walking out of the wreck) and it scared the hell out of me.

Hats off to the Air Astana guys. That's some solid debugging work and very impressive ability to keep calm and focused under pressure.

HeyLaughingBoy2• 4 years ago

In my private pilot training, I was taught to move the wheel both ways and visually check that the ailerons moved in the expected directions. Even if the connections hadn't been changed recently, they could still jam or a cable could have broken.

asciimike• 4 years ago

"Free, clear, and correct: check"

ilyt6• 4 years ago

I guess that goes to show that no matter the warnings and training, if something can be put in reverse, it eventually will be. Reminds me a bit of Proton crash where accelerometer (which even had arrow marking "up") was mounted in reverse and the rocket wanted to accelerate in direction of the ground.

I bet designers of software were like "there is no need to check whether moving it in direction actually moves it in that direction, we have thousand procedures to make sure hardware is done right". Defense in depth really is the best approach, not just strictly for security

bewaretheirs• 4 years ago

The 2013 Proton crash was more similar to the Embraer incident than you might think. Human pilots can sometimes step outside the box and re-learn how to fly but automated guidance systems aren't that flexible.

The Proton crash involved angular velocity sensors -- rotational accelerometers -- installed upside down. It's not that it wanted to fly into the ground, it was that it couldn't fly straight because (from the guidance computer's perspective) it was responding backwards to the behavior commanded by the guidance system. And rockets just don't fly sideways very well.

Watch the direction of thrust from the engines (the Proton, like most rockets, steers by thrust vectoring) in this slowed-down video of the incident:

https://www.youtube.com/watch?v=vqW0LEcTAYg

wazoox• 4 years ago

They were fortunate enough that cables on both sides were inverted; therefore the ailerons "merely" worked in reverse. If they had reversed the cables on one side and not the other, that would have been even tougher to sort out and maybe impossible.

stavros2• 4 years ago

Yep, that would have been a second elevator. There would be completely no way to roll the plane in that case, though I wonder if the plane has a natural tendency to fly level anyway.

mannykannot7• 4 years ago

Not so much an elevator, given their position on the trailing edge of the wing, but more like a flap with limited deflection both up and down. Like flaps in general, this would have some effect on pitch, but I doubt it would be enough to make pitch control impossible (or nearly so).

My guess is that in this scenario, small angles of yoke deflection would have little effect, and the pilot's almost instinctual response would be to increase that deflection, quickly reaching the point where the spoilers would kick in, giving the sort of roll response the pilot was seeking (together with some unexpected yaw that I suspect also happened in the actual case). It would be a problem, but not quite so counterintuitive and baffling.

skykooler• 4 years ago

In that case, the spoilers would still deploy when large control inputs were given, so roll control would still be present albeit with a very wide "dead zone".

ttcbj• 4 years ago

Thanks to the mhandley for posting this. I have now read a bunch of these posts (on 'admiralcloudberg') and they are absolutely fascinating.

The ones I find most interesting are those where I remember the media coverage at the time. So often, the post has fascinating details which emerged later. For example:

https://admiralcloudberg.medium.com/days-of-our-discontent-t...

I remember this crash of a Jamaica-bound plane after 9/11, and I remember the press reporting initially that the tail had been repaired at the factory, and then a few months later that the NTSB had issued a directive that pilots shouldn't respond to wake turbulence too aggressively.

However, the cloudberg post has fascinating details about how the particular characteristics of a training scenario for the A300 had led many pilots to conclude that the proper response to wake turbulence was massive inputs to the controls. It would have been easy to conclude the pilot who made them was just incompetent. But in fact, evidence was he was very competent, and that he had learned what the training taught him quite well. The trainers did not understand what the implementation of their simulation was teaching until after the accident was investigated.

Also, the industry did not fully understand what tests for structural integrity of the tail really said about its capacity to withstand inputs at lower speeds until after this crash. The details are just fascinating.

Or this one:

https://admiralcloudberg.medium.com/lost-souls-of-grammatiko...

About a plane where everyone is unconscious and it circles until it runs out of gas. I remember the coverage at the time, which presented it as a mystery. It turns out it wasn't such a mystery, the compression system on the plane was left in an incorrect state by maintenance, and the pilots repeatedly failed to recognize it. The details behind those short statements are fascinating.

Anyway, thanks for posting this.

Flying the Unflyable Plane: The near crash of Air Astana flight 1388

Related Stories