The Ugly History of Tool Development at the FAA (2002)

http://www.baselinemag.com/networking/The-Ugly-History-of-Tool-Development-at-the-FAA/

by angersock • 12 years ago

24 14 12 years ago

shalmanese• 12 years ago

Just FYI, this article is from 2002.

m_mueller• 12 years ago

Oh cool, I'm sure the article is not really relevant anymore today since our profession must have collectively learned how to do large projects. Oh wait[1].

[1] http://www.theguardian.com/world/2013/oct/25/obama-appoints-...

pcl• 12 years ago

Given that the FAA just had problems in the last week with what seems to be a software upgrade, the temporal context is particularly relevant.

pmorici• 12 years ago

So long as there are no consequences for the people and businesses involved in these kinds of failures this stuff will continue to happen and nothing will ever change.

From the perceptive of the companies involved in this it sounds like it was a resounding success to the tune of several billion in revenue.

lanewinfield• 12 years ago

I'm curious to see what kind of technology the FAA would establish to manage flights if it was founded today.

snom380• 12 years ago

Most likely it would have been called "ObamaAir" by critics and would have forced a government shutdown before a flawed compromise would be implemented. :)

More seriously, a next generation system that they are looking to implement in the future will involve aircraft automatically "negotiating" routes and deviations between themselves and digitizing ATC communications, which would reduce the need for air traffic controllers. However, getting there is a long term project because there are so many things that can go wrong and need to be accounted for.

EliRivers• 12 years ago

The airlines would lobby against gubbermint interference in their flights and insist that in a free market they were best placed to manage their own airspace :)

zyxley• 12 years ago

This gives me major flashbacks to reading through the case study of London Ambulance Service's tech upgrade failure in 1991: http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase....

A lot of major points are shared:

- A vendor with no previous experience in the field

- Massive underestimation of project difficulty by the vendor

- No real oversight from the contracting organization

- A system meant to replace every preexisting process in one fell swoop

- Opposition from unions uncomfortable with replacing every preexisting process in one fell swoop

RogerL• 12 years ago

A few points - I used Ada in the late 80's and early 90's, and also had plenty of interactions with the FAA.

The paragraphs about OO and Ada did not ring 100% true to me. OO was not added to Ada until 1995; The project was cancelled in 1994. I just pulled Grady Booch's "Software Engineering with Ada" C 1987, and it does have 8 pages on OO, but all it is talking about is packaging and encapsulation - the word "object" is not really defined. I.e, if your plane has a throttle, you'll write a 'throttle' package (e.g. module).

On the other hand, he does talk about IBM's use of Ada. They used a subset of Ada as a design language. These were the days of full on waterfall. And, the book's conclusion is quite giddy, prophesying that Ada will become the langue of choice, and extolled for enabling safe, maintainable code. Well, naturally, you can create horrible code in any language, and bork up any project with bad planning and execution. I recall working for a boss (until he was fired) that fully drank the koolaid - I'll spare you the number of steps you were supposed to, and the number of different CASE tools we were planning to use - years of work before the first line of code would be written, and then of course you would do "bottom up" development, coding each function, on and on, until you had a 'big bang' integration. Fortunately upper management saw how flawed that was, and we were able to do a much more sane process described as "design a little, code a little, test a little. Design a little, code a little, test a little".

It is easy to point fingers at companies, and to be sure they are culpable, but the government forces much of this on the companies. I recall government reviews where we were held to the exact language of the contract despite us pointing out why X was a bad idea, wouldn't work, and so on, and you would just be treated like a criminal.

On a different (quite small) project, I was forced to emulate all kinds of buggy UI behavior in an upgrade - I wrote a bunch of code to simulate bugs, because otherwise, well, you aren't meeting the contract, and dialog A used to do X, why is it now doing Y? Trying to explain why you might want to fix bugs (and yes, we tested and proved it was correct) just got you nowhere. So we spent a bunch of money writing deliberate bugs in a very acrimonious atmosphere where any attempt to reason or change things when you found a flaw in the spec was just treated as trying to rip off the government. There's a reason defense contractors charge a lot - there is no end to the amount of illogical things you will be expected to do on your own dime.

To be fair, it is not always like that. I was able to run quite a few projects, and at the kickoff meeting I would say "we can do this in one of 2 ways, and you get to choose. One, we can follow every 'shall' ('shall' expresses a deliverable requirement in contracts) to the 'T', and any deviation will get charged to you. Or, we can meet regularly, more than called for in the contract, you can observe progress and see the project at any time (typically contracts only gave the government a few opportunity to see progress during a project), and we will discuss and solve problems as they come up. You'll have to trust me, but you'll have a lot more insight, and equally I'll have to trust you not to skewer me with non-performance over some 'shall' we agreed to not address". Almost always they went with the second option, and we did some type of XP/iterative development. But these were small, couple people efforts; that would probably never fly with 100+ people teams.

username42• 12 years ago

At that time, the main idea of OO was binding data and operations. Ada 83 does not provide polymorphism, nor inheritance, but provides many levels of encapsulation: package, generic package, generic types. Since Ada 95, child packages and tagged types (object with polymorphism and inheritance) have been added.

Even in Ada 2005, the general advice given to Ada developers is to use objects only when they need polymorphism. Ada is a marvellous language that is far richer than just objects and classes.

mikestew• 12 years ago

Ah, those heady days of the 80s and 90s, where CASE tools and object-oriented programming were going to save the world. Hell, with CASE tools we wouldn't even need programmers anymore, folks will just enter the requirements and compilable code will come out the other end!

The world of computing machinery was rife with that bullshit around that time, more so than before because most folks had never used a computer to do something useful and up to that time computers were mostly ginormous calculators or data sorters. I think the industry today has finally found a bit of skepticism (and cynicism) thanks to such fiascos as the FAA's project and because $100,000 CASE tools proved to be largely a waste of money. Object-oriented programming eventually found its place in the world, but I think it, too, was largely oversold in the late 80s and early 90s.

So then we were left in an in-between time where we were beginning to discover the magic that computers can work, but had not yet developed the necessary level of skepticism needed to sniff out the snake oil.

The article in one sense lays the blame at the feet of old-school Big Bang development. We, the readers, of course are wondering "did you not read The Mythical Man Month?" But it's my recollection that the snake oil, umm, I mean "tools" of the time were said to save us from such pitfalls. But many forgot to read the chapter on "There is No Silver Bullet" (or maybe that was a later paper, not in the book).

Though the dollar amount is huge, it's still the same old story that plays out to this day: magical tools (replace CASE with MongoDB, node, or whatever) that may have a use but don't pack the punch the vendor says it does, vendors that don't want to say "no, that's not practical" and ever-changing requirements from customers. Some 20, 30 years later we don't seem to have collectively learned a whole lot.

cheepin• 12 years ago

OO is still oversold in my opinion. Functional programming may be an up and coming star, but OO dominates industry still.

phaedrus• 12 years ago

The problem with CASE tools is that any system composed of "(CONFIGURATION+CASE_TOOL) => (GENERATED_CODE+COMPILER) => STATIC_PROGRAM" could be replaced with just "DYNAMIC_PROGRAM + CONFIGURATION". The generation of the source code and then compiling it is an unnecessary middle step.

tim333• 12 years ago

One participant says, "It may have been the greatest failure in the history of organized work."

Which may have been true in 2002. We in the UK managed to do rather well with the NHS IT project which started in that year and has gone through about £11.4bn (~$19bn) vs the FAA's $2.6bn. (http://www.telegraph.co.uk/health/healthnews/8780566/Disastr...)

The Ugly History of Tool Development at the FAA (2002)

Related Stories