Lockheed Reassigns Workers to Fix F-35 Software

http://www.military.com/daily-news/2013/06/19/lockheed-reassigns-workers-to-fix-f35-software.html

by olympus • 13 years ago

37 50 13 years ago

macmac2• 13 years ago

Not only have they added 200 engineers from other projects which will now suffer delays and quality issues, both also "They're running 24-7, what we call lights out." which will ensure massive burn out. Brilliant...

joezydeco• 13 years ago

Why is it called "lights out" if the offices are occupied with coders non-stop?

thezilch• 13 years ago

Coder caves; no lights, especially not natural; time of day or passing is not in your peripheral.

sliverstorm2• 13 years ago

If you time it right, you don't have to hit burnout. In my view, "timing it right" basically means completing the project before burnout. A successful completion followed by a little R&R goes an incredibly long way to combating burnout.

macmac2• 13 years ago

And I suppose they will use their awesome estimation powers that have led them into this mess, to time it right...

sp332• 13 years ago

The mythical man-month in action!

AnIrishDuck• 13 years ago

Adding manpower to a late software project makes it later.

- Frederick P. Brooks, Jr. "The Mythical Man-Month".

jere• 13 years ago

Nine women can't make a baby in one month.

bediger40002• 13 years ago

Any one who's worked at Martin Marietta in the past (now part of Lockheed) will recognize this. Martin had a long, long history of doing this sort of thing for projects in trouble.

When they get past this (and they will, they have the DoD over a barrel, they just have to not gloat too publicly) the fallout to the F-35 will last as long as F-35s are in operation. In the early 90s, Martin people hand-assembled Titan rocket guidance and control software because some time in the distant past, the assembler had some bugs. The F-35 software group will live with whatever horrific "best practices" the current group of reassigned workers come up with.

masklinn• 13 years ago

In the early 90s, Martin people hand-assembled Titan rocket guidance and control software

What the fuck?

bediger40002• 13 years ago

To be completely truthful, they compared binary dumps of hand-assembled, and machine-assembled software, to ensure that everything was exactly the same. But yes, they hand-assembled some weird instruction set. I can't recall asking if they assembled to ASCII hexadecimal word (24-bit) representations, or to something else.

I believe this was for "Commerical Titan", a short-lived, Titan 34D variant. Four flights, one of which left Intelsat VI in the wrong orbit. That wasn't software, but plugging the wrong cable in.

SkyMarshal• 13 years ago

I'm hoping this will help justify limiting purchases of the F-35 and replacing it sooner with cheaper, more expendable, but roughly equally effective UCAVs and better missiles.

jfb6• 13 years ago

Common sense couldn't kill the utterly superfluous F-22; the F-35 has way more stakeholders, so don't count on it going away any time soon.

LowKarmaAccount• 13 years ago

The F-35 requires more than 8 million lines of code, compared with about 2 million for the F-16 and less than 1 million for other fourth-generation fighter aircraft

The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.

The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.

hosay123• 13 years ago

a system that is designed to never crash

I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).

choice of using C++

Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?

olympus2• 13 years ago

system that is designed to never crash

Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."

Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

iyulaev• 13 years ago

sometimes do so automatically without physically crashing the jet

I hope by "sometimes" you mean "almost every time!"

outworlder2• 13 years ago

Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

If that's the case, why not use something like Erlang?

jfb6• 13 years ago

I think the difference in LOC count is likely due to the much more complicated avionics in the F-35 than the technology stack, as compared with older aircraft.

cobrausn2• 13 years ago

The F-35 has multiple variants, one of which is a STOVL (Short Takeoff and Vertical Landing) aircraft with a vertical flight system. The amount of code necessary to deal with that is likely a large part of the difference in LOC, not to mention the more complicated systems for weapons, radar, communications, etc...

In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.

MichaelGG• 13 years ago

expanded their pool of potential programmers

Is that true? I'd think that the main requirements would be experience, intelligence, security clearance, probably familiarity with avionics.

Any good programmer worth working on such a project should be able to switch languages.

I'd hope it's not like some corporate web site that just needs the programming equivalent of warm bodies.

johnchristopher• 13 years ago

Why did they switch from ADA to c++ ?

smackfu• 13 years ago

Maybe all the ADA programmers retired.

jarrett4• 13 years ago

For anyone who's unfamiliar with the term "Mythical Man Month," it's the title of a book which argues, amongst other things, that assigning more engineers to a late software project will make it even later.

The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.

Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.

I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.

I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.

outworlder2• 13 years ago

I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway?

Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).

john_b• 13 years ago

Frequently aerospace companies will have a limited number of licenses for various pieces of software. C++ as a language obviously doesn't have this limitation, but a limited number of licenses for various custom test suites and third party software (e.g. Matlab) is frequently a bottleneck. If this is a bottleneck, it makes sense to have these licenses in use 24/7.

Purchasing more licenses is always an option, but usually not a quick one.

DanBC• 13 years ago

This website has some information about serious bugs. There are a variety of languages being used, by different industries, with either expensive war machines or expensive civil aeroplanes full of passengers or impossible to service space-craft.

(http://www5.in.tum.de/~huckle/bugse.html)

jongraehl• 13 years ago

"Accountability". A conspicuously heroic effort (likely counterproductive long-run), because their +$6 billion drip is threatened by lawmakers who don't think they're taking this whole thing seriously.

For the sake of whatever competent programmers were already working on the system, I hope this is mostly PR noise.

kevincennis2• 13 years ago

"There’s nothing wrong with flying before we buy. In fact, most of us test drive cars before we [buy]"

Except that if we custom-order a car, the manufacturer isn't legally prohibited from selling it to someone else if we end up refusing to buy it.

davorak• 13 years ago

There analogy is not good, but I would also assume that lockheed won a competitive bidding process to get the contract as well. They promised to build it at a certain price and in a certain time.

kevincennis2• 13 years ago

Yup, Lockheed's X-35 beat out a (horrendously ugly) Boeing airplane to win the contract, but my understanding was that the initial orders were cost-plus. That's pretty typical for large procurements of advanced aircraft. It's hard to know long it will take and how much it will cost to build something that bleeding-edge.

stcredzero• 13 years ago

That plane is not a replacement for the A-10, unless you kid yourself. The loiter time is atrocious.

nether• 13 years ago

Because loitering and enduring battle damage make so much sense in an era of countermeasure-immune SAMs and huge directed-shrapnel warheads. Spitting DU rounds onto a city block also isn't a great idea in today's conflicts. The A-10 is obsolete, the tankbuster role is obsolete, the F-35 isn't perfect but is far more appropriate for today's conflicts.

kbolino• 13 years ago

Yet we fight a lot of "obsolete" enemies. The effectiveness of the F-35 won't be known until it is battle-tested. Before then, it's all rainbows and unicorns.

harshreality• 13 years ago

Drones are far more appropriate for today's conflicts.

vaadu• 13 years ago

The program should be cancelled.

It's way over budget, not as stealthy as originally claimed and completely unnecessary. The DOD should be putting this money and effort into UAVs.

I also have to wonder how quickly the Chinese will get the software updates.

Lockheed Reassigns Workers to Fix F-35 Software

Related Stories