Boeing 737 Max Crashes: Sensors Vulnerable to Failure

https://www.bloomberg.com/news/articles/2019-04-11/sensors-linked-to-737-crashes-vulnerable-to-failure-data-show

by bushido • 7 years ago

102 167 7 years ago

jngreenlee• 7 years ago

Realize some pilots lurk here...was the angle of attack sensor ever associated with an automatic system prior to MCAS?

Seems like previous errors would have been less impactful due to human observation and synthesis with other instruments, window observation, etc.

If so, it shows how GIGO works in automatic systems...this should have been scrutinized as AOA sensor was not above the 1:1,000,000 threshold for single-input systems w/o failover.

dreamcompiler3• 7 years ago

was the angle of attack sensor ever associated with an automatic system prior to MCAS?

Yes [0]. The stick shaker and stick pusher. But those are overridable by pilot muscle power. With the MCAS, there can be situations where that isn't the case, and we have two crashes as a (probable) result.

[0] https://philip.greenspun.com/blog/2018/11/11/boeing-737-cras...

Xixi• 7 years ago

Not a pilot, but angle of attack sensors are used with flight envelope protection systems in Airbus airliners (and probably newer Boeing planes too, though I'm not sure).

Take for instance this particular incident on an A321 aircraft [1]: "two angle of attack sensors having frozen in their positions during climb at an angle, that caused the fly by wire protection to assume the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated forcing the aircraft to pitch down, which could not be corrected even by full back stick input. The crew eventually disconnected the related Air Data Units and was able to recover the aircraft.".

A321 have three AoA sensors, so if two freeze in the same wrong position, they outvote the good sensor...

[1] https://avherald.com/h?article=47d74074

EDIT: incomplete citation

howard941• 7 years ago

There are at least two A320-family AoA<>automation incidents listed here: https://en.wikipedia.org/wiki/List_of_accidents_and_incident...

lutorm12• 7 years ago

Specifically:

"On 5 November 2014, Lufthansa Flight 1829, an Airbus A321 was flying from Bilbao to Munich when the aircraft, while on autopilot, lowered the nose into a descent reaching 4000 fpm. The uncommanded pitch-down was caused by two angle of attack sensors that were jammed in their positions, causing the fly by wire protection to believe the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated, forcing the aircraft to pitch down, which could not be corrected even by full stick input. The crew disconnected the related Air Data Units and were able to recover the aircraft."

PaulHoule• 7 years ago

Airbus learned this the hard way in this incident:

https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T

which I think had to do with New Zealand regulators following up on

https://en.wikipedia.org/wiki/Qantas_Flight_72

A group of New Zealand pilots and regulators pressured a German pilot to test the envelope protection systems during a simulated emergency landing (so well simulated that the pilot did not seem to know where he was going ahead of time)

The AoA vanes were out of commission with this aircraft, the flight protection system malfunctioned and the plane flew into the ground.

That is, the danger got caught on a test flight and only 7 were killed. Compare that to the problem happening on not one but two revenue flights.

Had the FAA required that pilots ride out an MCAS failure in a simulator they probably would have learned that pilots could not ride out an MCAS failure before passengers were put at risk.

prolepunk• 7 years ago

AoA vanes not just failed, they failed because of incorrect maintenance procedure.

One of the contributing causes was incorrect maintenance procedures which allowed water to enter the angle of attack (AOA) sensors. During fuselage rinsing with water before painting, three days before the flight, the AOA sensors were unprotected. As specified in the Structure Repair Manual by Airbus, it is mandatory to fit a protection device on AOA sensors before these tasks.

In this case several failures all happening at the same time cause the crash: software, pilots following protocols and maintenance.

dreamcompiler3• 7 years ago

This thing is a small fin on a swivel. It's a wind vane about 6 inches long, sticking out into a 500 mph airstream. It's a heavy-duty piece of metal, but that swivel joint makes it vulnerable to bird strikes, freezing, etc.

dboreham• 7 years ago

Hmm...if it works that way it seems quite easy to determine if it is working -- if it isn't swiveling then it is probably broken. Perhaps also you could make an active version of this sensor where there's a stepper that pushes it a bit either side of the current neutral position, then measure resistance to that push with a strain gauge. It might be possible to thereby know that you're pushing against flowing air, not bird guts jammed in the mechanism, for example.

dreamcompiler3• 7 years ago

Probably could be done but it would require significant reengineering because it introduces new failure modes. Best approach is what everybody else does: Assume the AoA sensor can fail and design the software accordingly. Boeing didn't do that and the results were fatal.

cesarb• 7 years ago

if it isn't swiveling then it is probably broken

According to https://www.satcom.guru/2019/04/what-happened-on-et302.html it was swiveling freely because it was broken.

mansoor_• 7 years ago

Everything is vunerable to failure

AceyMan2• 7 years ago

I wonder if it's feasible to build a guard to cover the AoA sensor to mitigate the ability of foreign objects damaging the vane.

Clearly it'd need to be very transparent to airflow—something like the face guards of an NFL player—a heavy gauge titanium cage or something like that.

Btw, the Wright Bros knew the importance of AoA and included one on the Wright Flyer. Implementation? — a length of straight stick parallel to the wing chord with a bit of ribbon affixed to the end :-D (KISS at work!)

subb• 7 years ago

I wonder if they could take the mesure from the underside of the plane (90 degrees difference from the front)? It might help for bird strike, not much for ice.

CamperBob22• 7 years ago

I wonder if a small array of microphones couldn't be used to accomplish the same thing by cross-correlating the sound made by airflow hitting adjacent surfaces at different angles.

Seems like a bad idea to use mechanical vanes for this, especially if you're going to bet the whole farm on a single instrument as Boeing seems to have done.

lawlessone• 7 years ago

omething like the face guards of an NFL player

it would probably just get blocked then by bits of bird.

VBprogrammer9• 7 years ago

AoA sensors are sensitive to very minor deviations in the surface they are mounted to, there are aircraft which don't allow any minor damage in that area because it can cause misreadings.

HeyLaughingBoy3• 7 years ago

You seem to have a pretty high level of knowledge of avionics. Do I remember correctly that you worked in the field? If so, do you have any idea of what the allowable range of an AoA sensor might be?

i.e., on some systems I've worked on, we had a range that a sensor was expected to be in (normal, just do regular control loop), an outer range that was unlikely (stop reporting for a while/hold control output constant to see if it gets better or worse before warning operator), and a narrower range that was flat out impossible (notify operator of sensor failure/put system into a graceful shutdown).

I assume that the engineers designing this system had something similar, but given that MCAS activated with an AoA indication of 80 degrees (IIRC) and that's essentially impossible on a 737 (or at least would correlate with a hell of a lot of other warnings), was there anything they could do to warn the operator (pilot) that there was likely a bad sensor input?

HeyLaughingBoy3• 7 years ago

My general practice is to assume that the engineers in a highly regulated field are competent and have good reasons for the things they do/don't do. If there isn't a guard, then it's likely that having one would cause more problems that leaving it off or it would be ineffective. e.g., hitting a bird at 100+ kts would damage both the guard and the sensor anyway, rendering the guard useless.

peteradio11• 7 years ago

Engineers can have their hands tied as well, limiting their options to doing the job or resignation. Perhaps putting a guard on the vane would have only been necessary if it was considered an intolerable failure and being flagged as an intolerable failure would have required a recertification of the plane or something. Just a guess.

noir_lord2• 7 years ago

Given they test aircraft canopies by firing frozen (after defrosting) chickens at them it's usually a safe assumption.

Air travel is very safe because of literal generations of engineers working hard to make it safe, in this case the failure is one of procedure and people been people (cutting close to the wind on what was allowed).

Things usually tighten up after this kind of thing and then gradually loosen until the next disaster that could have been preventable.

laydn• 7 years ago

I was thinking whether they can "hide" an extra AoA sensor under the hood for emergencies. Just like a landing gear can be put down or retracted, an AoA sensor can be kept inside the fuselage, and brought out only when necessary. Maybe this one sensor will have better lifetime/reliability since it is not exposed to outside conditions.

jpm_sd2• 7 years ago

I am surprised that there is no ultrasonic method of measuring AoA.

CamperBob22• 7 years ago

Exactly, using mechanical vanes for this sounds pretty stupid, given what can be done with acoustical signal processing. It must be a more difficult problem than it seems.

HeyLaughingBoy3• 7 years ago

Not necessarily. It could be that after doing the safety analysis, the mechanical method is less likely to fail.

Remember that there are usually many ways to skin a cat. Often the one chosen hits the sweet spot of reliability, manufacturability, cost (both initial purchase and operational), maintainability (e.g., how easy is it to test/inspect), etc.

When you have decades of experience with something and it works well enough, there's generally no reason to improve it when your engineering resources can be focused on more pressing problems.

I was a team lead on a new system that was based on the IBM ISA bus in 2013. The reasons were simply that it worked well enough and we had tons of experience with it and could reuse a lot of custom-built hardware and code. The only downside (and reason we switched), was that it was getting harder to find blade PCs that still had an ISA bus and the system was expected to be in the field for 15+ years.

newsoul2019• 7 years ago

One thing I haven't seen fully explained ( maybe it was but I missed it ). Were these vanes damaged on both of these downed flights? Why didn't all the other 737 MAX 8's crash around the world, Was it something about the flight profile or the weather? Can the plane be flown as-is with a sufficiently conservative flight profile?

VBprogrammer9• 7 years ago

The speculation on the Ethiopian flight was a bird strike. The data was normal until a few seconds after take off.

Another issue could be the failure mode, if it failed at or near 0 AoA it wouldn't cause too many safety of flight issues on a normal flight.

linuxftw2• 7 years ago

Well, unless you need the MCAS to kick in, and the plane plummets to the earth in a hard stall because it didn't.

drawkbox3• 7 years ago

The AoA sensor that MCAS uses, as well as the nose pitch monitoring adjustment due to the larger more powerful LEAP engines that causes a pitch up on the nose, is external and in the front [1][2].

Stall sensors are common, but the MAX is the only 737 to have a blade-like sensor on the exterior of the plane that can automatically pitch the plane down when it detects stall conditions -- without input any input from the crew [2]

Not only is it susceptible to birds but it could be sabotaged as well from the location of it, if I were investigating it I'd check into maintenance/storage/possible sabotage just in case.

Since the nose pitch monitoring system and MCAS use the AoA single sensor it is a single point of failure, it also seems like a security issue.

About the single sensor and single point of failure[2]:

Boeing's 737 MAX, the model being flown by Indonesia's Lion Air that crashed last October killing all 189 on board, had been criticized for relying on a single sensor to detect the aircraft's angle of attack, according to today's Wall Street Journal. A preliminary investigation by Indonesia's National Transportation Safety Committee (NTSC) blamed the stall prevention system (known as Maneuvering Characteristics Augmentation System, or MCAS) which relied on a the single sensor that provided erroneous data, causing the MCAS to misfire, leading to a series of events that put the plane into a nosedive.
After the October crash, Boeing began updating the plane's software so multiple sensors would provide data to the stall prevention system, the company said in a statement on Monday.

About the external AoA sensor [1][2]:

The 737 MAX with its larger engines mounted even more forward needed a separate sensor to sense stall, the nose up attitude of a plane that turns off aerodynamic lift over the wings. Stall sensors are common, but the MAX is the only 737 to have a blade-like sensor on the exterior of the plane that can automatically pitch the plane down when it detects stall conditions -- without input any input from the crew --according to Seattle Times veteran aerospace reporter, Dominic Gates, who is close to Renton, Washington where the 737 MAX is assembled.
According to the preliminary report from Indonesia's NTSC and in a Boeing service bulletin, Lion Air flight 610 was providing erroneous data from a angle of attack (AOA) sensor. It appears the crew may have been fighting against: bad sensor data, a multitude of warnings and an automatic flight control system that seemed to be malfunctioning by automatically pushing the nose of the plane down repeatedly during the 11-minute flight. Ground control was contacted numerous times by the crew, who were repeatedly requesting permission to return to the airport.

[1] https://i.imgur.com/XTrf6EX.jpg

[2] https://www.engineering.com/ElectronicsDesign/ElectronicsDes...

cjbprime2• 7 years ago

There just aren't that many MAX planes. Only went into production in 2017, around 350 delivered to airlines. The vanes were presumably damaged (either mechanically or electrically) on both downed planes but not others.

stevelambert3• 7 years ago

On the lion air the sensor was non-operable from the start of the flight; so yes on both flights the AoA sensor was in a non-flightworthy condition.

One point of clarification: on the 737 and most other airplanes the airspeed is unreliable if the AoA sensor becomes damaged. This occurred on both the ET302 and the lion air flight. In such a state, it is up to the pilot to place the plane in safe flying state that does not depend on access to the airspeed. The pilot should by memory, ideally, set the thrust and nose up attitude of the plane along with other control surfaces. Failure to do this is what indirectly caused the AF447 disaster and it seems the the crew didn't execute this procedure on the ET302 flight. At the very least, attempting to engage the autopilot and disengaging flaps were contrary to standard procedure. Not executing this procedure dramatically increases the workload on the pilot and increases the risk that either airspeed wall outside of the safe operating range of the aircraft or the pilot will have to take drastic action to maintain airspeed in a safe range. Both are dangerous and even an experienced pilot can fail to take appropriate action especially in an emergency with the flight crew attending to other anomalies.

With that caveat stated, there is nothing inherently unsafe about flying with the combination of unreliable airspeed and an uncommanded MCAS activation in theory. The previous flight of the lion air flight was able to counteract 20+ instances of MCAS activation by simply trimming the aircraft using manual trim before they tried disengaging electrical power to the stabilizer motors. After disengaging trim the pilots felt comfortable continuing the flight and adjusting trim using the manual cockpit trim wheels. With all that being said, no aviation regulator would certify a plane that couldn't fly safely with an unreliable airspeed and manual trim turned off.

I think the question about why the 737 MAX's haven't crashed more is two fold. First, a birdstrike that takes out an AoA sensor is very rare. In that sense, Boeing got very unlucky in that they had a birdstrike that knocked out the specific AoA sensor tied to MCAS a month before they planned on rolling out the software update. Additionally, flight sensors are usually maintained very well as they are a critical item. You would never expect standard maintenance procedure not to fix an unreliable airspeed. When it has happened before though, the outcome can be tragic just as it was on the lion air flights regardless of any issues with MCAS.

heisenbit• 7 years ago

The sensor was never designed for what it was used. It was designed to provide input to a visual display for the pilot. The pilot always had a lot of other inputs. The reliability requirements were totally different to what was needed to directly control an airplane in a critical situation.

The software design compounded the problem with some bizare design decisions.

This was a system design failure. High level reliability requirements were not properly communicated to the subsystems. Subsystem failure was not properly analyzed. Human Machine interface considerations were ignored.

Considering the rush job and shortcuts Boeing made for this "feature" a valid consideration by a regulator would be to question their quality system. And that should trigger a systematic approach to reviewing all other requirements and design work going into the 737MAX.

gordon_freeman• 7 years ago

So if the sensors have a higher rate of failure then it looks like changing MCAS software to rely on both sensors may not be enough as there is a higher probability of both sensors malfunctioning and sending bad data to MCAS at the same moment. In that case, I am not sure how would MCAS behave.

stackola• 7 years ago

If these sensors have a common failure mode (eg. always showing lowest possible value after blowing up), this could be bad. If they just produce junk, it's unlikely they would agree, and MCAS would be turned off.

cjbprime2• 7 years ago

The common failure mode would be e.g. freezing, perhaps in identical positions, while on the ground before takeoff. You can't easily detect it before takeoff because the sensor requires airflow.

tzs2• 7 years ago

Do any planes do cross-sensor sanity checks?

For example, if the AoA sensors say the nose is too high and the flight control software trims down to compensate and then it doesn't see the AoA sensors report lower pitch, couldn't it check the Attitude Indicator (AI)?

Angle of attack and attitude aren't necessarily the same, because the wind vector is not necessarily parallel to the ground, but AoA changes and AI changes should be well correlated most of the time. If trim changes are showing up in the AI data but not the AoA data, then you've probably got malfunctioning AoA sensors.

VBprogrammer9• 7 years ago

The other part of Boeings fix is to prevent MCAS from being reactivated repeatedly. If I remember correctly they are not allowing it to reactivate until the AoA returns to normal.

I'm hopeful that they have analysed what happens if the AoA returns more or less random data.

zepearl5• 7 years ago

...they are not allowing it to reactivate until the AoA returns to normal.

Puah, but what is "normal"? This is probably kind of tricky to define if you cannot potentially rely on the sensor(s). Additionally I understand that the auto-trim of MCAS was created to actually react to "abnormal" situations (nose too high), which brings us here into a conflict :)

rasz• 7 years ago

Good thing its easy to manually trim then https://vimeo.com/329558134

salawat2• 7 years ago

This is not even remotely surprising, and by all standards should be taken into account in your implementation of dependent systems' gradual degradation fault modes.

markphip• 7 years ago

Shouldn’t it be easy to detect a stuck sensor and then stop using its data unless it recovers?

shadykiller• 7 years ago

Shouldn't the headline say "Boeing 737 Max" ? There's a big difference compared to regular 737s

oliveshell• 7 years ago

I think it’s well-enough implied by ‘crashes’. Regular 737s haven’t been crashing recently.

Someone12342• 7 years ago

Yes it should, and that's actually the original title of the article:

Boeing 737 Max Crashes: Sensors Vulnerable to Failure

Mods should likely change it.

sctb• 7 years ago

Updated. Thanks!

ansible• 7 years ago

I'm wondering now if MCAS should be an active system. Perhaps it should instead just warn the pilots that it thinks a stall is imminent instead, like when you get the stick shaking.

jhayward2• 7 years ago

As I understand it, that would require specific training for pilots of the MAX 8, and (not sure) perhaps a new type certification.

Those are expensive and it was a specific goal to avoid them.

magnamerc• 7 years ago

Having to get a new type certificate would likely kill the MAX. I work in aircraft certification, and I know it would be a very long and slow process on the order of years.

Someone12342• 7 years ago

They cannot do that, because then the Max loses its shared type rating due to the larger engines. MCAS exists to give the aircraft more similar flight characteristics to its predecessors.

tzs2• 7 years ago

MCAS apparently exists for more than that. A few days ago someone posted a link and some quotes from the FAA regulations on pitch stability. They require that the control force must be positive up to and throughout a stall.

Without MCAS there would be a point on that MAX where you no longer needed positive control force to continue pitching up. Without MCAS or some other system to automatically counter that, so that the pilot needs positive control force to continue pitching up, the plane could not be certified even if they did go for a new type rating.

Lind5• 7 years ago

Sadly, another "the sensor did it" scenario. https://semiengineering.com/dirty-data-is-the-sensor-malfunc... The tragic example of the Lion Air crash of a new Boeing MAX 8 aircraft, which on Oct. 29 killed all aboard, may be heading toward “the sensor did it” category. The black box recovered from the flight showed inconsistent data from one of the two angle-of-attack (AOA) sensors. With one half of the data apparently incorrect, it was enough to trigger this plane’s anti-stall system into a nose down action, which the pilots wrestled all the way into the Java sea.

052• 7 years ago

MCAS is not an ‘anti-stall system’, it’s a cheap type certification compliance hack

Doxin• 7 years ago

It’s a cheap type certification compliance hack that prevents stalls.

digikata3• 7 years ago

All equipment on the plane has a potential for failure... the question is what systematic design and pilot training is in place to take action in the face of failures.

iscrewyou• 7 years ago

I don’t think it’s fair to put pilot training and systematic design right next to each other.

If the design in itself, and the subsequent factors of safety are at fault, pilots can’t recover from that fast enough. Why put them in a dangerous situation and expect them to recover. How long could this go on even if we didn’t have these two plane crashes. The pilots/airlines could be at fault for not training too much but the bigger issue is the system in the question.

It’s like saying the drivers don’t know how to drive because the manufacturer didnt put it front and center on the driver’s manual that the car suddenly swerves right. As ic drivers should always have their hands tight on the wheels and be ready for situations like that. It’s not fair.

digikata3• 7 years ago

The pilot training is part of the overall systematic design, neither the equipment, nor the pilots exist alone.

peteradio11• 7 years ago

What? Does the degree of failure not matter at all? That makes no sense. The question is whether all these parts are actually meeting the specified failure tolerance rate.

digikata3• 7 years ago

It matters, but if the rate of failure is high enough to be expected to be encountered within an all-fleet, constant flight environment for the lifetime of the aircraft, then you plan for failures as well as having the pilot know what to do when there is a failure.

In air sensors like this, their failures are higher than the rate at which you can ignore. They fail on their own, they get bumped and bent on the ground, they ice over in the air, or have objects striking them.

duxup• 7 years ago

There certainly are examples of sensors failing or providing conflicting data, and pilots using prescribed backup systems to rely on in order to get home safe.

velox_io3• 7 years ago

I'm just amazed by how poorly this was implemented. Software engineers expect hardware to fail. This wasn't just the work of a newb (or I hope it wasn't). This was planned; both the requirements and the failure modes. Then implemented by another team, and the quality verified by another team, even a 3rd party. There wasn't even a cutoff if it started operating outside it's expected range, surely one of those people would have noticed the shortfalls. This software should have the same level of engineering as other systems/ parts of the plane.

Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup!

Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent.

This software wasn't even a crucial feature. It was only needed when high power is applied to the engines (like during takeoff) to stop the nose feeling too light by existing 737 pilots. This wasn't a jet-fighter, which would become unstable without the aid of computers. It's a shame 338 died because of what is effectively a 'schoolboy error'.

js2• 7 years ago

Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup!

Pitch != AoA.

https://aviation.stackexchange.com/questions/2287/can-the-pi...

http://www.aerospaceweb.org/question/aerodynamics/q0165.shtm...

yutyut• 7 years ago

Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup!

You would need a sensor to measure relative wind. Angle of attack is the angle between the wing and the relative wind.

flyinglizard3• 7 years ago

A six axis sensor could probably measure AoA in a pinch. It’s just like how a stability control system works in a car (which is measuring slip angle, which is the lateral “angle of attack” between tires and road). If you detect upwards rotation but no matching acceleration forces, you’re detaching from the air stream.

Obviously this system will drift over time, but it could more than adequate to detect a failed AoA sensor and revert to the second one.

LgWoodenBadger• 7 years ago

Is it really necessary to measure the relative wind or is that just how it's always been done?

What would be wrong with using an accelerometer to measure/calculate velocity with respect to pitch give you a better reading of AoA? In conjunction with a second velocity source to provide redundancy?

peteradio11• 7 years ago

This is a weather vane which needs to be exposed to the elements, its not something that can be inside the aircraft. It needs to be exposed to super harsh conditions to get its reading.

lutorm12• 7 years ago

Except you can estimate the Aoa pretty well just using airspeed and load factor.

inflatableDodo2• 7 years ago

Here's a version of how it occured;

"Instead, engineers were able to add just a few inches to the front landing gear and shift the engines farther forward on the wing. The engines fit, but the Max sat at a slightly uneven angle when parked.

While that design solved one problem, it created another. The larger size and new location of the engines gave the Max the tendency to tilt up during certain flight maneuvers, potentially to a dangerous angle.

To compensate, Boeing engineers created the automated anti-stall system, called MCAS, that pushed the jet’s nose down if it was lifting too high. The software was intended to operate in the background so that the Max flew just like its predecessor. Boeing didn’t mention the system in its training materials for the Max.

Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right.

Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes."

https://www.nytimes.com/2019/04/08/business/boeing-737-max-....

Datenstrom• 7 years ago

Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common.

I worked on the C-2 Greyhound aircraft in a past life. Part of an upgrade they got, only because it was an absolute requirement to fly in European airspace, was a magnetometer. The documentation on replacing and testing the new sensor, of which there was only one, was non-existent. The documentation said that the sensor could not and would never fail.

While in another country one failed. In disbelief my Chief had me read out every wire in the system even though I knew it was the sensor immediately. The bird was down for weeks until we got a new one because supply didn't exist and I had to make up a suitable plan for testing the replacement on the fly.

I wonder what kind of practices lead to a "this system can never fail" mentality. It should be assumed everything can and will fail.

breakingcups• 7 years ago

"Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right."

Shouldn't this have been more of a red flag not to rely on a single sensor when human lives are literally at stake?

052• 7 years ago

MCAS is not an anti-stall system, it’s a way to change stick forces by trimming the horizontal stabilizer - the surface with most control authority. There’s no evidence it has ever prevented a single stall or even activated properly, but there’s evidence it has helped crash two planes.

Obi_Juan_Kenobi• 7 years ago

GPS info might be something you can use for sanity-checking input, but AoA can only be reliably read from an actual instrument. Air moves, you can't rely on the plane's motion.

Also, MCAS isn't active with flaps extended, so "like during takeoff" isn't correct. MCAS won't be active during takeoff and landing because it's not designed to be active at those times.

velox_io3• 7 years ago

That's what I was getting at. If the nose is at an extreme angle (or reported to be) and the plane isn't climbing/ or descending - there's something wrong!

Just after takeoff while the plane is still climbing. I'm not sure that helps matters much, as the plane is off the ground and lore's the pilots into a false sense of security as plane's working ok..

Here's the bulletin Boeing issued after the first crash. It's hidden away in the bottom left and not highlighted, so it's easy to miss: "runaway stabilizer". Unfortunately, the brief is more focussed on assuring how safe the planes are, and what counter-measures to take in the event. Rather than what a runaway event looks like. https://theaircurrent.com/aviation-safety/boeing-nearing-737...

asynchronous13• 7 years ago

This software wasn't even a critical feature.

There are two crashes directly related to this feature. So, it doesn't make sense to claim it wasn't critical.

In Boeing's own hazard analysis, they rated the feature "Hazardous", which is the second most critical rating. Obviously, it should have been rated "Catastrophic". But even with their own policy, anything rated hazardous is not allowed to have single-sensor failure. By their own policy, this should have had at least two sensors. There were multiple failures that let this slip through to production.

velox_io3• 7 years ago

Sorry, I meant to write crucial (similar word, quite a different meaning..)

I believe the reason for MCAS was because the had to lower and move the engines forward. So at lower power, the forward engines make the plane nose-heavy. When the engines are near full power, the lower engines push the nose up. Compounding the difference in handling compared to the older models.

Here's the safety bulletin Boeing issued after the first crash. Even when you know what you're looking for it's hard to spot: "runaway stabilizer". It does seem they downplayed how serious the failure is, but they couldn't know how common it could be. That raises another question: Other pilots must have experienced the runaway event, but managed to gain control. What happened to those reports?

There will be lessons learnt from these crashes, in a number of fields.

[https://theaircurrent.com/aviation-safety/boeing-nearing-737...]

paulmd• 7 years ago

Ah yes, the classic "software engineer reveals one easy trick to solve a complex problem domain" post. Maybe they just needed a Raspberry Pi and a sixaxis sensor board, has anybody thought of that!?

joezydeco• 7 years ago

Phil Greenspun thinks it’s solvable in 10 bytes of object code! Why would you waste $35?

danaliv2• 7 years ago

Unless you’ve launched your phone into the air in a fit of rage, it doesn’t have an angle of attack. :)

Angle of attack is the angle between the wing and the oncoming air.

You might be wondering how that’s different from how high the nose is, and you’d be justified in wondering that. Every student pilot in the world gets these two very different things mixed up! But they are very different.

Next time you’re near an airport, watch the planes as they come in for landing. Try to get a vantage point from the side if you can. You’ll notice two things: (1) the airplane is descending, and (2) the nose is flat or pointed just slightly up.

It should be clear that in this situation, the angle between the oncoming air and the wing — the angle of attack — and the angle of the nose are different. Because the airplane is descending, the angle of attack is greater than the angle of the nose.

monksy• 7 years ago

Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent.

I think this is a good point on why Agile is so dangerous to the users. Agile allows for engineering to be micromanaged and due diligence from engineers to be overlooked. (It's not something the business people care about) Don't deliver without the safety feature? Well, you're underperforming... they're going to be let go pretty quickly.

bookofjoe• 7 years ago

The elephant in the room, which I've not yet seen alluded to: Boeing wants at all costs (which will be very high) to avoid having pilots be required to requalify with new MCAS software in flight simulators.

lurker0094• 7 years ago

So, 140 failures in 30 years of aviation... ~4 failures per year FOR ALL FLIGHTS IN A YEAR. That's unreliable?

peteradio11• 7 years ago

140 incidents. Untold number of averted failures.

cmurf7• 7 years ago

Cessna 150/152/172 stall warning "horn" works by negative pressure. https://www.youtube.com/watch?v=Q1JRTCBkWKQ

And then the Cessna 182 has an aerodynamic switch that activates an electric "horn". And in either case they can get jammed up with bugs. But we do a lot of training with slow flight, approach to stall, full stall, and recoveries including recovery from the ensuing dive and secondary stalls.

ggm4• 7 years ago

Root cause analysis: excessive pressure to keep type rating caused lax engineering to be considered acceptable.

Fix:let the FAA do it's job independently

mjevans2• 7 years ago

Fix: Make Engineers actually liable for the correct classification of systems and require signed justification for NOT defaulting to the highest possible standard.

ggm4• 7 years ago

...which demands a fully independent FAA. Not the one we seem to have which appears on the face of it, compromised by the consequences of embedded self assessment and 'same type' rules. Why was it in the national interest to have Boeing get 'same type' ? Because of the cost of competing with airbus.

The regulator was hamstrung in the interests of 'efficiency'

(I'm not an American and this is an argument i read which resonates. I'd welcome good rebuttal)

rusticpenn• 7 years ago

This also causes a lot of over engineering. ( Which maybe good for Boeing right about now)

zepearl5• 7 years ago

On a technical level: personally I think that the root cause could be a lack of "integration" of the system (MCAS) with the rest.

Meaning that e.g.

1) if the pilots' yokes/sticks are being pulled and

2) if the altitude based on gps and/or air pressure and/or radar is decreasing and

3) if the speed registered by the airspeed probe is high respectively increasing and

4) if the fuel being used by the engines to keep or even increase the speed is very low and

5) if the G-force registered by <some sensor, if it exists?> is lower than usual (neutral or even negative?)

6) etc... (anything else excluding AoA?)

...then the airplane is mooost probably "going down", even if the AoA says the opposite.

As the MCAS auto-trim has apparently a big potential impact on how the airplane flies (surfaces on the tail are big and their angle can move a lot), it should therefore not rely on just 1 (or even 2 duplicate) lonely sensor(s) but on multiple types that provide multiple point-of-views of the aircraft's situation to then abstract/determine/confirm the current high-level situation (in this case "is the aircraft going up or down?") and only after such result to react/take countermeasures to save the aircraft respectively help the pilots and/or lower costs for the flight.

If this is not totally wrong, is this (the total integration of all sensors) how Airbus did it with their super-automated systems like e.g. their side-sticks/joysticks just telling the airplane what pilots "want"? (meaning, did they start already from scratch creating a totally integrated system, if they did create such an integrated system?) (not sure about how automated the new Boeing airplanes are, so I did not mention them...)

bbojan4• 7 years ago

Angle of attack is not related to any of the items that you list. You could be approaching stall even while your altitude is increasing (2), while you are at a high speed (3), at less than 1 G (5) etc.

I'm not sure you quite understand what a stall is. A great resource I can recommend is a book called See How it Flies (http://www.av8n.com/how/).

ggm4• 7 years ago

You miss my point. Root Cause typically does what you are doing, focusses on technology and in-the-moment behaviours and remediation.

I wanted to go above this to a higher layer: Had the regulatory environment the FAA has mandate to operate run better, then Boeing could not have asserted "same type" for such a huge change to the flight dynamics of the aircraft moving its CG with the new engine.

The "root cause" is not technological, its the compliance processes, and the desperate desire to artificially re-create a "level playing field" between Boeing and Airbus

DuskStar• 7 years ago

Funny thing is, that sounds a lot like a certain Air France flight. Besides the airspeed probe, though, since it was inoperative. (And the AOA sensors agreed) But in that situation, pushing the nose down would have been the correct decision.

RespectForTruth• 7 years ago

You hit the nail on the head with this.

cmurf7• 7 years ago

Too much is being made about the sensors. From my pilot's perspective it's far worse that:

a.) MCAS can mistrim the airplane faster than pilots can even recognize what's going on.

b.) Boeing and FAA directives still don't account for a.), and therefore they're still giving pilot's inadequate advice. And it's sufficiently bad that I think it's intentional because the advice is consistent with prior Boeing 737 models: runaway stabilizer. If they came up with a better checklist to run that only applies to the MAX, that is suggestive that at least difference training is needed, and might even suggest a different type certificate is needed.

Much of this is discussed here which I found to be decent: https://www.satcom.guru/2019/04/what-happened-on-et302.html

incredibly it just dumps the stab and leaves it to the pilot to get it back.

Electric trim must be used to neutralize control column pitch forces before cutout!

ngcc_hk• 7 years ago

3 is the minimum for any critical system.

Boeing 737 Max Crashes: Sensors Vulnerable to Failure

Related Stories