Why US government IT fails so hard, so often

http://arstechnica.com/information-technology/2013/10/why-us-government-it-fails-so-hard-so-often/

by Jtsummers • 13 years ago

57 62 13 years ago

I just got through speaking in DC about Agile, startups, and functional programming, mostly to an audience that is intimately familiar with government work. It's really bad when you try to be diplomatic about how bad things are fucked up and get open laughter from the crowd. There's no BSing people who live this everyday.

Talking with folks afterwards, I heard about systems that I could have rewritten in the late 90s in a few months that are still in the process of being rewritten, 15 years later. After literally hundreds of people have worked on them costing the taxpayer millions of dollars. And these are not unique or oddball edge cases. Stuff like that is standard operating procedure. I can't impress on the HN audience enough how terrible federal IT is overall. (Aside from some really great outliers). There's a vast and increasingly large gap between political rhetoric and the reality on the ground. This has been true for at least the last 20 years.

I take issue with some of the details the supporting argument makes here, but that's mostly a quibble. The author is definitely headed in the right direction.

I'm a veteran of several federal IT programs, and a real veteran. I have seen some really good federal teams, but never a good federal team of teams. It all falls apart once you get past a few people and start throwing some kind of structure and management in place. I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Think big tools, big management, lots of stupid measurements, and no accountability. All with the word "Agile" stamped on the front of it.

...The bottom line is that federal IT programs' success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to...

Yep.

Jtsummers6• 13 years ago

"Agile" when federalized will probably turn out like "Lean". Lean in manufacturing and similar processes, when done right, is a good thing. However, the government version of it is almost a complete perversion of the intent. Instead of bottom up, employee-driven changes, it's top down based strictly on metrics and the changes typically effected so that whoever's currently in charge can say, "I made <process> 10% more efficient by reducing the number of wasted <things>." Employees resist the changed processes/taskings because they fear a loss of employment, and are kept out of the loop when the changes are being devised. Federalized Agile would likely be similar.

DanielBMarkham4• 13 years ago

Yep.

We saw this same thing with source control. Who doesn't like source control? But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in.

This created such a mess for developers that it wasn't unusual for one check in to take hours to accomplish. You'd read a book while the system tried to accept your code changes.

Agile is all about failing fast and learning quickly. Most contractual work for the government does not include the possibility of failure. At all. In fact, the way the contracts are written is such that the contractor takes on all responsibility for everything, even things there's no way they could control. In response to this idiotic situation, contractors start bidding at crazy low rates on these things, say 40 bucks an hour, then actually make their money on changes. So it's always in the contractors interest to capture and execute changes, while it's the government is trying to get the world delivered to them on a platter for five bucks. People use the term "perverse incentives", but this is like that taken to the next level. It's a multi-layered thing. It's total systemic failure. We'd call it systemic collapse, but the money keeps flowing in anyway, so that term isn't applicable, although it certainly feels like it.

This kind of distrust doesn't make for anything but a living hell for everybody, government employees and contractors alike. So, with tons of rules and structures in place, the way things really work is through relationships. People who are easy to get along with and helpful get more government work. Both sides of the fence can't control the process, and nobody rocks the boat and everybody gets a piece of the pie. You end up from the government side with some sort of informal and unspoken version of I don't care if you make a lot of money, just make sure me and my friends look good Also it'd be nice to pick up a little lucrative consulting work once I've put in my 20 and can start double-dipping.

Your average developer has absolutely no idea about this. If they've worked in IT in federal work, they probably have horror stories, but the more you see the worse it gets. Oddly enough, in the best cases you end up with people who really like working together, are motivated, but spend most of their time fighting a Sisyphean fight against the idiocy of the system.

dragonwriter• 13 years ago

I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use.

Agile can't be mandated; you can mandate methodologies (e.g., Scrum), but by the very act of mandating a methodology you are not agile. Agile is a culture, and it is a culture of flexibility in methodology with focus on the goal.

And its a culture that is very much contrary to the entrenched culture of most large bureaucracies, government or private (and, in government, often in tension with binding mandates where the authority to make changes to the mandate is very distant from the locus at which one would normally expect efforts to adopt development approaches.)

jfb• 13 years ago

It all falls apart once you get past a few people and start throwing some kind of structure and management in place.

That's hardly confined to Federal IT projects, however.

kilroy123• 13 years ago

Isn't this how any government project works? IT, construction, etc. Sees like the government is terrible at building anything, especially when contractors are used to do the job.

Probably why the government is so big and expense. I can only imagine what that $600+ billion gets us for the military. -_-

http://en.wikipedia.org/wiki/Military_budget_of_the_United_S...

iskander2• 13 years ago

Sees like the government is terrible at building anything, especially when contractors are used to do the job.

Strange to contrast that with a tremendously active and efficient era of public works from the 1890s through the 1950s. Just thinking of New York City, I suspect that the total cost of all the bridges, tunnels and highways (adjusted for inflation) might add up to a lot less than the 800 billion USD that disappeared into our recent stimulus bill. What changed?

edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally construct, which now would be somewhere between 300 million to 400 million USD (http://historical.whatitcosts.com/facts-brooklyn-bridge.htm). Even though these are rough figures and inflation over large time horizons isn't a perfect measure, I'm still astounded. The government's totally broken healthcare website cost more to build than the Brooklyn Bridge.

Jtsummers6• 13 years ago

The relationship between government and business, the much more short term approach to making money for businesses and individuals, a loss of the concept of "public good" or "common good".

Spooky233• 13 years ago

Lots of things changed, especially unions and labor costs. The guy who digs a subway or water tunnel in NYC today probably makes $250k, retires at 55, and makes $150k for the rest of his life.

The guys who built the Brooklyn bridge were day laborers who made nothing and died young.

nostromo• 13 years ago

Many people don't know that the New York Subway was built and run by private companies until 1940.

http://en.wikipedia.org/wiki/Interborough_Rapid_Transit_Comp...

http://en.wikipedia.org/wiki/Brooklyn%E2%80%93Manhattan_Tran...

Busses and ferries were private too:

http://en.wikipedia.org/wiki/Fifth_Avenue_Coach_Company

digikata2• 13 years ago

Personally, I suspect that modern management practices for large scale projects are more inefficient than ad-hoc practices of the day.

digikata2• 13 years ago

The inefficiency is primarily due to the scale and customization of the projects, not that it's government per se. In any project of sufficient scale, 1) you very likely use contractors at some level 2) you will have to invest into some sort of oversight infrastructure 3) with that much money invested, you want a strongly guaranteed outcome, so you'll err on the side of reliability over cost efficiency (or pay for the cost of failure and rework in subsets of the project).

I agree though that the mix of in-house expertise vs outside contractor effort does play a big part in the outcome. Look at Boeing's 777 vs 787 aircraft development for different sides of cost to use internal resources, vs leaning more heavily on external resources. The 787 with it's heavy use of subcontractors has had a much more difficult introduction.

The government in general has been upping the contractor content in a lot of projects with the belief that more "competitive private contractor" content automatically means better efficiency. But, in truth, it's no silver bullet, often increasing costs if it's blindly applied - especially if you fall below some threshold of in-house expert judgement.

polemic2• 13 years ago

I disagree that "government projects" are uniformly terribly executed between industries.

I can't vouch for the American case, but New Zealand has a similar history of disastrous government-run IT projects - from a terrible teacher payroll system[1] to a police information system[2]. The IT community hear is desperately trying to stop the same mistake with a massive tax system overhaul[3].

Nevertheless, civil infrastructure projects appear to be very well executed. Recent major roading engineering projects like the Victoria motorway tunnel in Auckland were completed on time and under budget, for example. There has been an active discussion in this country about why the IT industry, in particular as manifested as major systems projects executed by foreign software companies, are such consistent clusterfucks.

It's a big discussion and I won't go into many of the points raised here, but the issues is, IMHO, very much a "software industry" one, and not a "government" one.

1. http://en.wikipedia.org/wiki/Novopay 2. http://en.wikipedia.org/wiki/INCIS 3. http://www.stuff.co.nz/business/industries/9261063/Row-deepe...

eigenvector• 13 years ago

Most first-world governments have a century or more of experience in proper administration and project management of construction contracts. Designing, estimating, awarding and administering nine or ten figure construction projects is extremely complex but by and large, we know how to do it properly.

Construction management is a serious discipline that is studied by civil engineers. It helps that construction methods don't change very often. Software technologies can become obsolete in a matter of months.

I wouldn't start calling IT procurement a failure compared to civil infrastructure until software engineering has matured to the point that civil engineering has.

jacques_chester2• 13 years ago

Non-software projects can go horribly badly too.

I live in Perth, Australia. Almost every civil project there is blowing out like a balloon. Opera houses, hospitals, fancy developments. All going over budget and schedule.

I was in Toronto a few days ago. What's on the cover of the newspaper? A civil project. A subway station running way over cost and time.

Industrial and civil megaprojects routinely go way off course. Bad project management is universal, good project management is hard, any project of any serious size will require owner-commissioner organisations to not be useless idiots.

acdha4• 13 years ago

You're repeating Tyler Cowen's Libertarian Vice:

http://marginalrevolution.com/marginalrevolution/2006/08/the...

Large projects fail or run over budget because of organizational failures: poor goals, lack of accountability, internal gridlock, etc. Nothing about that is specific to government – and if we wanted to change that in the United States we could reach the level of, say, Scandinavia if we were willing to make significant cultural changes.

tjaerv2• 13 years ago

Surprised they didn't mention the all-time fiasco of the FBI's disastrous $170M Virtual Case File project: http://en.wikipedia.org/wiki/Virtual_Case_File

DanielBMarkham4• 13 years ago

That's just small potatoes. The really big stuff is almost invisible. That's because the liability is too large, but every now and then you can see how things are going. Check out the IRS modernization efforts. (The quote and link are probably 5-10 years old. Haven't heard much about it one way or the other lately)

...The IRS has failed twice to modernize its system, in the 1970’s when taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten years and $2 billion on the project and had little to show for it. The current project launched in 1999 with a budget of $8 billion. The project is late and over-budget; problems have been well publicized after the release of the IRS Oversight Board investigation...

http://www.kellogg.northwestern.edu/student/courses/tech914/...

AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with little to nothing to show for it.

I used to joke around and say heck, they could have paid me 2 Billion and I would have also provided them nothing, but look at the money they would have saved!

tjaerv2• 13 years ago

Thanks for that, I had no idea.

cjoh• 13 years ago

This is all right, but a little vague. Here are some more specific problems: http://blog.dobt.co/post/63655420372/how-healthcare-gov-went...

GVIrish4• 13 years ago

This is an excellent analysis. Particularly the point about setting the go-live date long before the scope of the project was understood, not to mention the fact that there's no way they could have had an open bidding process AND meet the deadline. In some ways it's a miracle this went live on-time, flaws and all.

However, I suspect the true extent of the flaws with Healthcare.gov and the exchanges is not understood yet.

If anything good comes out of this, I hope it's that the government finally starts paying attention to the woeful state of government IT projects.

uptown• 13 years ago

Yet if you look at what's been revealed about the NSA's capabilities, they're excelling in some areas of technological execution.

EpicEng• 13 years ago

A new version of Windows Server doesn't address the problem of loading 64 JE files and 11 CSS files on every page. There is some programming ineptitude at play as well, though I realize that the article was focusing specifically on the new ObamaCare site.

acdha4• 13 years ago

That's also a minor side issue: it'll cost you a sub-1-second page load time but it doesn't cause the kinds of problems people are talking about where the backend components are overloaded to the point of failure.

The size of healthcare.gov is actually well under the average:

http://www.webpagetest.org/result/131010_ZG_X5C/1/details/

(see http://www.webperformancetoday.com/2013/06/05/web-page-growt...)

For comparison, look at WashingtonPost.com: it uses considerably more data and more total requests and yet the page loads much faster:

http://www.webpagetest.org/result/131010_GG_XDK/

If you look carefully, though, you'll see a) that they aren't setting Cache-Control headers and are setting ETags (both performance killers) and b) there's this weird staggered response behaviour which indicates something is seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so presumably there's some interaction with the browser going on as well:

http://www.webpagetest.org/result/131010_ZW_XD6/1/details/

mmagin• 13 years ago

At an individual level: How many programmers who are good enough to get a job that pays well in the private sector want to take a job (either in the government or for its contractor) with all that bureaucratic overhead?

vonmoltke• 13 years ago

More than I think most people here realize. Some people find particular, unique problems fascinating. Others have a sense of duty that outweighs material or personal concerns. Others just don't know better.

ihsw• 13 years ago

Large organizations need homogeneous computer systems, so it stands to reason that centralized configuration management would be a perfect fit.

So why hasn't the likes of puppet and chef caught on? Genuinely curious.

jl6• 13 years ago

Large orgs want guarantees and certainty, and so typically look for software support from other large organizations. Easier to hold IBM to account than a small independent vendor who may have gone out of business by the time you need them. Yes, this is an expensive approach.

Who is selling support for Puppet and Chef?

archangel_one• 13 years ago

As per the article, they're very (very very) slowly migrating from Windows XP to Windows 7. I suspect Puppet and Chef are not even on their radar at this point.

eitally• 13 years ago

The reason it fails is because of articles like this disincentivizing the talent they need from being interested in working for either a) .gov/.mil or b) government contractors. Additionally, the contracting process being divorced from the project execution weighs heavily.

Jtsummers6• 13 years ago

Articles like this don't disincentivize a lot of people, and burying it would just let them be lured in and find out the hard way how bad it is. Articles like this air a problem that needs to be corrected.

What disincentivizes talented (specifically young talent) people is the ageism inherent in the system and its perverse method of rewarding people (specifically talking civil service here, contracting's another beast). Unlike the ageism you hear about in the occasional article about Silicon Valley, its the young that have a hard time in civil service. Positions aren't available because too many people are waiting to retire (for various reasons). When they do get in there are few leadership roles open for advancement (see previous sentence). Their pay is strictly time and not merit based (well, not 100%, but NSPS is gone and only a handful of places use its equivalent of merit based pay increase). Career fields are disincentivized, for instance engineers are treated better than computer scientists and mathematicians even in offices where the latter make more sense (bonuses, faster promotion schedules for trainees, pay differential for the degree). Lack of proper training programs (this is office by office, so not sure how universal the problem is, but it's pretty shabby in a lot of places). People are expected to be hired in and, within months, become experts on arcane and outdated systems, with no support because the actual expert has just retired (that's whey there was an opening). People leave quickly, similar to the Teach for America article earlier in the day, because they burn out or realize there's no where to go, neither up nor down, just steady.

On the other hand, in some regions older, talented folks can get a good deal. They can come in as a high grade GS employee with a middling step, automatically enrolled in a decent 401k (5% matching), good insurance (less critical with Obamacare?), and, after a couple years, effective tenure so they have stability which they might value for their family.

richieb• 13 years ago

Not much stability with the current Congress.

GVIrish4• 13 years ago

This article is only touching on a small portion of why government IT is terrible.

People like to blame contractors but I think a large part of the problem is that the federal employees who are responsible for these big projects are rarely held accountable for failure. In fact, the lack of accountability for federal employees is why so much work was farmed out to contractors in the first place.

Then there's the fact that a lot of the decisions in government IT are made by people who have little understanding of how successful software gets built. People don't understand the importance of defining the scope of the project properly. Or you get people that change requirements willy nilly. Or you find people that are just trying to protect their little fiefdom so you end up with wasteful duplication of effort or withholding of vital information.

The Ars article indirectly touches on the fiefdom issue. When you have dysfunctional IT departments or applications, often times instead of fixing the dysfunction, someone just creates their own little separate IT effort to get around it. So the VIP's might get something functional while the peons get classic ASP & Access.

What I find very interesting about the Healthcare.gov problem is that this may be the first time that a major federal IT project failure has major political consequences for elected officials. If Healthcare.gov ends up being a crash and burn failure, it could result in Republicans gaining enough power in the next election to overturn the whole ACA.

RougeFemme3• 13 years ago

I don't think lack of accountability is what pushed the work out to contractors. And if that is the case, does that mean the accountability "stick" got pushed out to the contractors. I think work got pushed out to contractors from a desire to out-source and because contractors, for various reasons, generally do a much better job of staying abreast of best practices than the government employees. And I say this as someone who has worked on both sides of the fence.

I agree that requirements are a big issue. In addition to the requirements issues you mention, there is the problem with the "business line owners" attempting to unnecessarily dictate implementation details, poorly defining the functional requirements in the first place and ignoring interfacing systems.

GVIrish4• 13 years ago

Where lack of accountability comes is the idea that contractors can be fired if they don't do a good job, whereas it is much more difficult to fire a full-time employee. So rather than solving that problem, most of the government's software development is farmed out to contractors. The other reasoning for contractors is that contractors don't get retirement benefits that FTE's do, so in theory that saves the govt money but in practice it probably doesn't, given that contractor bill rates are usually 200-300% or in excess of what FTE salaries are.

Not to say that FTE's are great or anything, contractors do tend to me more up-to-date on technology but I think that is a result of the government employee hiring practices and compensation scale.

Yeah I've seen a lot of "business line owner" issues like the ones you've mentioned. Another common one is that software is often built to mirror dysfunctional and wasteful business processes. So you either get software that is needlessly complex, or you build the software then have to re-do it because people realize how bad the underlying business process was.

djhworld• 13 years ago

Here in the UK most government contracts just go to the big enterprise shops (think Accenture, IBM, Logica, Cap Gemini, ATOS etc) who manage to negotiate fairly hefty long term deals and charge a healthy commission for taking them on and running them for years.

This is why you run into legacy codebases that are 10-20 years old and will keep on going.

specialist• 13 years ago

I've done IT at both megacorps and govt. Frankly, I don't see a difference. If anything, govt work has greater transparency and accountability.

Contributing factors are, but not limited to:

- scale

- complexity

- legacy / installed base

- laws, regulations, rules

- organizational processes (workflows)

- quality of developers

- resources

- politics and agendas

Just trying to support all the entrenched chaotic mutant use cases and edge cases will wreck any project plan.

I much prefer working at younger, smaller companies. There's fewer roadblocks. But the healthcare and other benefits working for government rock, so here I am.

Why US government IT fails so hard, so often

Related Stories