Air traffic system failure caused by computer memory shortage

http://www.reuters.com/article/2014/05/12/us-airtraffic-bug-exclusive-idUSBREA4B02320140512

by hudibras • 12 years ago

65 44 12 years ago

damian2000• 12 years ago

Sounds more like a bug rather than a lack of memory to me ...

The flight plan did not contain an altitude for the flight, one of the sources said. While a controller entered the usual altitude for a U-2 plane - about 60,000 feet - the system began to consider all altitudes between ground level and infinity.

The conflict generated error messages and caused the system to begin cycling through restarts.

hudibras• 12 years ago

They didn't have enough memory to cope with an infinite number of inputs.

Obviously a hardware problem...

gonzo• 12 years ago

is there a plane that flies at infinite altitude?

Is there an aircraft of concern to the FAA that flies above FL1000? One or two historic flights have been that high, so raise the ceiling to 150,000'

still not infinite.

venomsnake2• 12 years ago

Also how dare they not to have cpu that can run infinite cycle in under 2 seconds.

MattHeard• 12 years ago

It sounds like the altitude existence assumption caused cascading errors which in turn exhausted the system memory.

ars2• 12 years ago

That's not what the article said. They said that since it didn't know the true altitude the computer assumed the plane could be at every altitude, and checked every single possible one.

This was made worse by a flight plan that had the plane going in and out of the area, so there were a tremendous number of permutations of possible flight paths.

The computer attempted to allocate memory to calculate every one of them and ran out of memory.

vince_refiti• 12 years ago

This scenario wasn't unit/functional tested then.

tomp2• 12 years ago

  unsigned short int altitude;

ghshephard• 12 years ago

The flight was at 60,000' - You would have to look far and wide for a platform in which "short int" wasn't two bytes.

For example, Here is what the compiler for the 8-bit ATMEL Atmega 328 in an arduino does (on a platform with 32KB for program text and 2KB for RAM.)

  sizeof(bool) == 1
  sizeof(short) == 2
  sizeof(int) == 2
  sizeof(long) == 4
  sizeof(long long) == 8
  sizeof(void *) == 2

ape4• 12 years ago

Maybe it attempted to allocate a huge structure for each foot of altitude.

kijin• 12 years ago

said Dan Kaminsky, co-founder of the White Ops security firm and an expert in attacks based on over-filling areas of computer memory.

That sounds more like buffer overrun than OOM. One is a Heartbleed-class vulnerability, the other is a pretty straightforward DoS situation. But of course why would we expect Reuters to be able to distinguish between the two...

andreasvc7• 12 years ago

That's irrelevant to the rest of the article, it's only how his expertise is characterized.

awjr• 12 years ago

Having worked on ATC, this really does sound like a software bug. Fixing such a bug on a delivered system would require an immense amount of costly testing. Adding memory is the cheap option.

The other thing to consider is that some of these systems are rather old and could easily have less than a gig of ram.

andreasvc7• 12 years ago

That sounds very worrying. The fact that this issue occurs shows that they didn't prepare for OOM conditions, and another issue could easily cause a different OOM condition they didn't prepare for. It's not inconceivable that there could be an issue which gobbles up an amount of memory you throw at it. It is a tragic irony that the fact that testing is costly may be the very reason this condition was not tested for.

andreasvc7• 12 years ago

I can no longer edit. "an amount" should be "any amount".

lyndonh• 12 years ago

Adding memory is the cheap option

Only in the sense of having a broken system, putting in place some procedural constraints and then praying that everyone follows the rules.

As you say, it's a software bug. Until it's been 100% characterised and tested this leaves a huge question mark hanging over the system. Adding memory is a band aid solution.

huhtenberg2• 12 years ago

I would guess that it was some form of an numeric overflow, signed/unsigned issue or out of bound array access, because the only abnormal thing here is U2's altitude.

andreasvc7• 12 years ago

Have you read the article? The cause is explained quite explicitly, and it was the very absence of the altitude not any of the things you mention.

huhtenberg2• 12 years ago

very absence of the altitude

And? Consider what this means exactly and how come the system saw the plane but didn't record the altitude.

sehugg• 12 years ago

This is part of a system the FAA has been trying to replace (unsuccessfully) since at least 1990: http://www.ainonline.com/aviation-news/aviation-internationa...

(The AAS often appears in software engineering classes as an example of a classic "train wreck" project)

jmnicolas2• 12 years ago

What's the point of flying the most stealthy plane on earth if you have to register a flight plan ?

I thought such a plane would be 100% autonomous and be able to "see" and avoid other planes.

ars2• 12 years ago

flying the most stealthy plane

A U2 is not at all stealthy. You must be thinking of some other plane.

if you have to register a flight plan

It's not a military operation in a foreign country, it's a routine flight in the home country.

I thought such a plane would be 100% autonomous

The U2 was made long before such computers existed. They do have drones now.

and be able to "see" and avoid other planes.

It wasn't the plane with the problem, it was ground control.

This must be a world record: A faulty assumption in every single sentence!

jmnicolas2• 12 years ago

I effectively mistook the U2 and B2.

For 100% autonomous flight, I wasn't thinking no pilot, but about being able to navigate without any assistance from other radars

joezydeco• 12 years ago

"It's not a military operation in a foreign country, it's a routine flight in the home country"

Is it worth asking, at all, why we're flying a military reconnaissance aircraft back-and-forth over Los Angeles?

andreasvc7• 12 years ago

It's not the most stealthy plane on earth, in fact it has been shot down over the Soviet Union and China in historical cases, because of being tracked by radar. Moreover it was flying over American airspace, it makes sense to register it.

khuey2• 12 years ago

The second U2 was shot down by Cuba, not China.

venomsnake2• 12 years ago

A good idea when you consider that it could be mistaken for some bad guys plane, shot down and probably cause WWIII (not that long shot - the US and USSR were at a few times very close to annihilation due to glitches in their own software)

prof_hobart• 12 years ago

If you don't register military flights on civil ATC systems, you run the risk of this -http://en.wikipedia.org/wiki/Hughes_Airwest_Flight_706

strictfp• 12 years ago

Lack of constraint leads to infinite search space. Sounds like constraint programming.

atlantic• 12 years ago

So much talk about Snowden, and then a newspaper reveals how to trigger a software bug than can shut down the entire air traffic control system for a large segment of the US. This is a genuinely damaging revelation, even if some critical details are lacking. Curious that the government is not up in arms about it.

colanderman• 12 years ago

A perfect example of why embedded or safety-critical systems should never dynamically allocate memory from a common pool.

Such systems should always preallocate fixed-size heapsfor each subsystem, each of which can then dynamically parcel out pieces of their own heaps if necessary. This helps limits the damage of any memory leaks in the system.

Air traffic system failure caused by computer memory shortage

Related Stories