A $300M IT flop

http://fcw.com/articles/2014/07/24/it-flop-at-ssa.aspx

by jsvine • 12 years ago

71 67 12 years ago

jsvine• 12 years ago

Throughout the past six years, the project has been stuck in the beta phase. According to McKinsey, "for past 5 years, Release 1.0 is consistently projected to be 24-32 months away."

adwf• 12 years ago

It's still got nothing on the NHS £10bn IT flop...

I always wonder when I hear about these projects, where exactly has the money gone? It can't possibly be in hardware, that would make some of the biggest data centres in the world just for the NHS. I can't even see how it can be manpower, how many IT people can you get for £10bn? And surely if it were corruption, someone would notice so much money being squirreled away...

There's just no logical explanation behind these things.

arethuza• 12 years ago

I can recommend the book "Plundering the Public Sector" by David Craig that explains how projects like the NHS IT fiasco happened:

http://www.amazon.co.uk/Plundering-Public-Sector-Letting-Con...

[NB Note recommended for UK taxpayers who have high blood pressure and/or easily upset]

martiuk• 12 years ago

gov.uk which had a great success, was done by hiring their own people to work on it. I don't see why anyone would think outsourcing is good for public projects.

beat2• 12 years ago

I worked on a high profile government project that had five different vendors involved. FIVE. The inter-organizational management problems were astonishing. I'm pretty sure that by the time I left, managers actually outnumbered technical staff.

The biggest problem, though, was the lack of pushback. The government officials would ask for the moon, and the vendors would promise to deliver it. There was almost no one saying NO - the word that project most desperately needed to hear. Features outweighed core infrastructure reliability, so configuration management was a nightmare.

TheOtherHobbes• 12 years ago

It's very good for the companies that benefit from the outsourcing.

tootie3• 12 years ago

Because ramping up a huge staff to build a new project generally requires keeping them forever. It's much easier to request a budget for a project then it is to try to expand headcount of well-protected government jobs.

sudont• 12 years ago

Management. Business Analysts. "Metawork" in the words of a coworker who used to architect BPM implementations at the parent corporation of my job. Getting consensus from possibly hundreds of stakeholders.

Generally, the disconnect is that management is under the assumption that you can manage your way to success, so they add more overhead to watch and prod at the proceedings. Development is almost always an afterthought in most business projects. The first thing they do? Get consensus, which is generally everyone putting in ideas so that they seem involved—see Atwood's Duck[1]. After you have a ton of often contradictory ideas, you need to have them prioritized. And since nobody managing the project is technical, they need to hold countless "prioritization meetings with stakeholders and implementers." Often times the meetings will grind to a standstill while process analysts and engineers argue over the wording of things. Usually the analyst will tell the engineer that the business has standardized on a set of terms, of which are not the trade names for things. Maybe everyone uses a set of terms which aren't the "correct" wording.

After things are prioritized based on the wrong criteria, they're then pushed off to an engineering group who are tasked with implementing things exactly to spec. Since the definitions are incorrect, they need to constantly circle back and check with the stakeholders to double check. Occasionally checking in will involve a "subject matter expert" who is on vacation, and the only one allowed to answer the question. (Thus stalling that item and anything depending on it.) Others will often say "it's above my paygrade" in an attempt to avoid culpability.

After the project has run long and only has a majority of the features complete, people are reassigned to other projects, and the app is sent to QA to hand-test features. Sometimes it will involve the "stakeholders" sitting in a room for days or weeks, manually clicking their way through the interface. Maybe the project is declared a boondoggle, deleted, and replaced with someone else's pet project.

So yes, not logical.

1: http://blog.codinghorror.com/new-programming-jargon/

filearts• 12 years ago

It is a bit scary that an accurate description of reality could easily have been sourced from Dilbert.

The tough reality is that certain projects have a scope such that a small team could never reasonably be expected to capture the requirements of the project. Also, remember that large enterprise projects are generally built by people who are not users themselves. This is where the technology crowd often draws false conclusions, reflecting on projects of their own, where they have intimate knowledge of the problem space. There are likely very few forward-thinking engineers who also have visibility into the intricacies of the various SS systems. This is where the layers get piled on and the efficiency vanishes.

joncrocks• 12 years ago

This is true. And it doesn't just apply to governments, but any organisation that is large.

Once a organisation gets to a sufficient size, organising and allocating resources and communication starts to eat up a disproportionate amount of time, if the organisation is left to drift into whatever shape it likes.

VLM• 12 years ago

"I can't even see how it can be manpower"

From personal experience in software development, dilution of responsibility can be incredibly expensive.

Scenario 1: I wonder about a minor detail, call up a random user, we hash it out in 5 minutes. Cost is about 10 minutes of total labor, figure maybe $10

Scenario 2: I have to organize a meeting with the exec stakeholder, the dept manager, and the entire dept staff. Which due to scheduling conflicts is in two weeks. Finally the meeting arrives and due to bike shedding we spend 2 hours talking about everything other than the topic until its hashed out. So the cost is two weeks of my salary doing nothing but surfing HN, lets say $3K, plus 2 hrs of mgmt for exec at $100/hr and the manager at $75 thats $350, plus double that time for them to make the "command decision" and later inform me, so thats $700 of management, plus the ten person working team has to burn an extra 2 hrs of overtime each at lets say $25/hr and 10 people that's $750 so we'll call it $4450.

So figuring out the default value, if any, for a blank of a form can cost anywhere from ten bucks at a personally responsible organization up to about five grand at a fully diluted responsibility organization.

Its obviously orders of magnitude more if you involve consultants, people flying in from out of state, etc.

ww520• 12 years ago

The hourly rates are kind of low across the board. May be doubling it or tripling it?

jeremymcanally• 12 years ago

You'd have leaner, smarter firms that could actually deliver something getting involved if the bidding process wasn't so obtuse and opaque.

When I was consulting, I responded to a few RFP's but they were always either (a) obviously tailored to the capabilities of a single firm so there was no way to win unless you were them or (b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time. Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow. We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.

Government contracting in IT, from my point of view, is nearly 99% about "playing the game" and about 1% actually delivering something. Especially since it seems that even if you fail spectacularly, you still get paid (possibly paid even more to fix your stupidity).

dm23• 12 years ago

That's exactly right. The staff organizing the project for the government gets advice (bad advice) on the technologies that it should use from approved development firms then make using that technology a strict requirement. These firms are some of the few that have developers that use that technology and win the bid, everybody except the hired firm loses.

Any small competent development team can built almost anything imaginable with 6 years of time, proper funding, and without unnecessary restrictions.

$300 million to develop anything is insane. I can't imagine what happens when the check is cut, "let's hire 100 programmers, 10 managers, 5 accountants, 10 support staff and they'll figure it out."

Maybe there needs to be some kind of "insurance" company that oversees the project and has to pay back a portion of the money if the project fails. I'm not sure if that's reasonable, just an idea. If the project starts going off-track then they start letting people go and find competent replacements.

Plan projects out, have milestones, goals, backup plans. If they spend 6 years and $300 million dollars on a computer system and fail then they were just winging it from the beginning. The managers responsible should be identified and go on a blacklist. Right now it's either "oh well, hopefully we'll do better next time".

untog• 12 years ago

Startups certainly don't help themselves in this regard, though. The sheer number that shut down, get acquired etc. means that I don't totally blame government bodies that choose large, established companies.

danielweber2• 12 years ago

I've frequently encountered it as "let's spend a lot of money on making sure we don't waste any money."

beat2• 12 years ago

Exactly. My interpretation of it is "Appearance of risk is more important than actual risk". Letting a small team run loose with unproven technologies appears risky. Using big names with expensive "enterprise solutions" appears low-risk. But really, those expensive tools and complex processes create whole new categories of risk.

Glide• 12 years ago

That's brilliant.

How much staff do they need to make sure everything is okay? Oh you have a database? Well you need a DBA on this. Security? Who is making sure the application is secure? Where is the security officer? Etc. etc.

Many of the boxes can only be checked by having more people.

cafard• 12 years ago

My impression working for a government contractor years ago was that the government had great systems for getting the best price on typewriters, and that these systems more or less guaranteed the purchase of obsolescent computer technology.

mseebach• 12 years ago

The way the UK government is starting to do things is to insource project management and then run tenders for smaller components, explicitly shaped so smaller firms have a chance of competing. This seems like a sane approach.

ProAm• 12 years ago

is nearly 99% about "playing the game" and about 1% actually delivering something. Especially since it seems that even if you fail spectacularly, you still get paid (possibly paid even more to fix your stupidity).

This is true for almost all business, especially consulting work.

pc864• 12 years ago

I worked for a company who provided a SaaS to a few private entities but also some very large state agencies, including an entire branch of the government. I remember sitting in a meeting with my boss and the owner of the company, while we sat on the phone with someone from the procurement department of another agency while they crafted the bid to only allow us to compete. It was disgusting and they did nothing to hide it. The state employee (who knew the company owner socially) even mentioned the competition by name, along the lines of "if we require ______ then ______ won't be able to bid, right?"

justinpaulson• 12 years ago

So you immediately reported this to other government officials?

beering• 12 years ago

Competitors can bid, but will have generally worse answers for the specially crafted requirements. The competitors can either say "no, we don't do that" and hope that the committee doesn't mind, say "yes, we could do that" and see if the agency will pay extra for that special req, or say "yes, we do that" and build out that feature if they win the bid.

rockdoe2• 12 years ago

Now consider the situation that the government official knows your company can deliver on time and in budget, with little fuss, and your competitors are known to underbid, over-promise, deliver very late or never and discuss endlessly over contractual fine language when they start failing.

It's absolutely disgusting that he'd craft his bid to the supplier he knows isn't taking shit, right?

bmelton• 12 years ago

government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.

To be fair, what that really means is that the contractors that they want to win already know the requirements, and that's the closest way they can get to impartially telling you to fuck off.

Generally, that response connotes that the bid is tailored to contractors already doing work with the agency in question, who are not only intimately aware of the request for proposal, but likely had a hand in its creation.

tootie3• 12 years ago

This happens in the private sector too. Sometimes consultants are hired to write the RFPs which, suprise surprise, they tend to win.

incision2• 12 years ago

'(a) obviously tailored to the capabilities of a single firm so there was no way to win unless you were them'

For what it's worth, this isn't necessarily corruption at work.

As I've related on HN before, the procurement arms of these agencies are often woefully unequipped for handling anything IT related. It's typically a process made for sourcing completely interchangeable, relatively simple supplies or services seeing terribly complex software or systems dumped into them - not 'shoe-horned' as that would imply an attempt at fitting - simply dumped so that requirements spill on the floor and everything gets jumbled up onto the existing conventions.

Given that, providing an agnostic SOW is a great way to end up with stuck with a lowball bid which makes the most self-serving, word-splitting assumptions possible to stick you with something utterly apart from what you need.

'(b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time.'

Related to what I said above - these requirements are not as hard and fast as they might seem. Writing a defensively worded response, getting your foot in the door to talk with the people who actual request and implement is not a terrible way to start doing real work.

'Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.'

Right.

It's not unlike any sort of courtship, business or otherwise. You can't just ask outright, you have to make friends and hear the information gush over lunch or volunteer some no-fee consulting and take notes.

Big vendors and VARs have people who do this nearly full-time so it's certainly much harder for a small company, but not impossible.

'We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.'

Something to be aware of, when dealing with a Lockheed terms like 'adjustments, feature additions' are euphemisms for 'one million dollars'. If you're pitching these possibilities and expecting/willing to make them to earn the business then it might pay to be explicit about it.

incision2• 12 years ago

'The project was conceived to replace 54 separate components used in the SSA's disability determination system.'

This is the bit that jumps out at me as clear indicator of the sort dysfunction that dooms projects like this more so than anything else.

It's indicative of a the classic exploding scope and fragmented responsibilities leading to an overall lack of accountability that unfortunately characterizes IT in and around Government.

Gershman seems to agree.

"If there's not strong leadership, problems arise with coordinating work between contractors, and there's no one person looking at the system as a single entity -- everyone is making sure their piece is done, and fulfilling their responsibility up to a certain point,"

However, I'm not wholly keen on the the start-ups as a solution direction taken later. It's not that it's wrong, just beside the point unless the start-up savior is specifically targetting culture and process change as opposed to technical implementations.

The 54-armed kraken of a defined by committee project will drown and devour a fleet of confused start-ups lost in the mist of *BE requirements and contract mods just as readily as the Lockheed leviathan.

greggersh2• 12 years ago

What I meant by startups is that in that world, there's market forces driving you to do things better, faster and cheaper. Not necc. that we need startups to solve the IT problems of government (although that could help), but that we need to create an environment where there's market forces pushing government contractors to do things for less people/less money and better results (which are all very, very possible).

rayiner• 12 years ago

Interesting quote from the article:

'You have companies like Lockheed Martin -- they're not web development companies, they don't do work in the private sector, they're not building start-ups,' Gershman said. 'There's very little in the market that forces them to adopt new technologies.'

All of these are true, but I think this glosses over some fundamental structural problems with government contracting. The SSA has almost 60 million beneficiaries, and deals with distributing an entitlement that is extremely sensitive and protected by layers of legal process. 8,000 Social Security checks get lost,[1] and its not a blog post, its national news.

Moreover, a lot of lean and agile processes just aren't consistent with political reality. There's no minimal viable product, there's no invite-only beta testing, there's no iteration, there's no A/B testing, etc. Technical decisions ("let's just roll out to northern Alabama first") become divisive political ones. Purchasing decisions ("we've had good luck with this vendor before") become the basis for accusations of favoritism and corruption.

Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you. This creates a perverse incentive to drag a project along so failure happens over years if not decades.

[1] https://medium.com/@jan.curn/how-bug-in-dropbox-permanently-...

[2] Most big IT projects fail: http://www.computerworld.com/s/article/9243396/Healthcare.go.... Most startups fail. Most internal prototypes never amount to anything. This is just how engineering works.

tootie3• 12 years ago

Later on the article describes the project as being "adrift" with ill-defined requirements and scope. That will kill absolutely any project and is a completely separate problem from technology. Lockheed failed at the engagement level to define scope. Their development team didn't have a prayer.

phaus2• 12 years ago

Pretty much all government IT projects have ill-defined requirements because GS-15s and Field Grade Military Officers are like little kids. They change their minds frequently and they are so used to having their way that they simply won't take "no" for an answer. You can't explain to them that it will ruin a project because the overwhelming majority of them go through life thinking that they know everyone else's job better than they do. Because of this a typical government IT project will keep having requirements added until the last day of the contract.

You are right, the development team never had a real chance at achieving success.

enraged_camel• 12 years ago

>Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you.

This is not unreasonable. After all, projects in the private sector are funded by private funds. If you fail, you only waste your own money. Whereas public projects are funded by the taxpayer. When they fail, what's really happening is that taxpayer money is going into the pockets of some private company, but there's only a failed project to show for it. So yeah, of course the backlash is going to be much stronger.

timdierks• 12 years ago

In the private sector, investors provide capital that companies (and employees thereof) use to pursue projects. When the company fails, the employees are generally still paid (while some companies have employees who participate in profits / share risk with investors, it's rare for it to be a dominant form of compensation). Thus, the situation is more similar than you describe: if the projects fail, the investor/owner has paid out a lot of money to employees and vendors and is left with nothing.

The difference is that smart investors/owners are aware that the possibility of failure is inextricably tied to any attempt to excel, and allows people to try things that might fail in the pursuit of success.

In our current political climate (and probably in other climates as well), governments are dumber, and tend to drive their staff to manage towards over-safety and covering of asses rather than towards best outcomes.

greggersh2• 12 years ago

All of what you say is correct; but I think the answer is to look for ways to change the status quo. There are ways to do that, it just takes engagement and creative thinking.

jinushaun• 12 years ago

Can't you build an MVP with dummy data?

discardorama• 12 years ago

The sad thing is not that the government pissed away $300M on this project; but that the prime, LMFS, will continue to get away with it because there are no penalties!

But the system is rigged that way: if you do decide to hold them accountable, you'll find that they have documented every little specification change that the SSA put in after the project was awarded. So now the blame comes back to the SSA. Both sides will point fingers at each other; the consultants on this project will laugh (on their way to the bank), and things will remain the same.

In other words: the system of contracting in place today is broken. From labyrinthine requirements to bid for one, to obtuse specifications to lack of oversight; it's one huge mess.

And the system is corrupt to boot. Primes like LMFS, etc. employ former employees of these agencies (and vice versa), so it's very hard for an outsider to break in.

easytiger• 12 years ago

Pennies in an ocean:

http://www.theguardian.com/commentisfree/2011/aug/03/nhs-dat...

...final cost to be as high as £20bn, indicating a cost overrun of 440% to 770%

Xophmeister• 12 years ago

Rather offtopic, but does anyone else find the dollar glyph in the header font jarring?

quux• 12 years ago

Now that you mention it, yeah it is... It looks like a lot of the symbols and punctuation characters in Garamond Pro are slanted with no logical pattern or reason. Would drive me nuts if I were using it.

http://i.imgur.com/Cbd74Gq.png

bane• 12 years ago

For more like this

https://en.wikipedia.org/wiki/Virtual_Case_File

http://www.reuters.com/article/2012/03/14/us-manhattan-saic-...

mathattack• 12 years ago

"For past 5 years, Release 1.0 is consistently projected to be 24-32 months away."

I have worked on project like this. Usually it's the sign of management failure (poor communication, poorly defined requirements, bad governance, inability to retain knowledgable employees) rather than technical feasibility. It's not like the software is pushing technical boundaries.

"Echoing McKinsey, Gershman said that without a strong leadership element, IT projects of this magnitude are almost doomed to fail from the start."

This is a big problem. Because employees know these problems will fail, the smartest avoid them like the plague.

remon• 12 years ago

It's amazing to me how consistently government IT projects fail and how widespread the problem is. A simple google teaches us that this has been and still is a consistent problem for quite a lot of countries. Some with an incredibly strong tech sector (the US is obviously a prime example). Surely there must be a way to organize/orchestrate government IT projects in a way that improves delivery and risk management. And...300 million USD?

al2o3cr• 12 years ago

Scratch out "government" above - the overall failure rate of IT projects in general is crazy. The difference is that private project failures don't get news coverage (barring utter disasters that bankrupt the company).

ryandvm• 12 years ago

This is business as usual in government contracting. The goal is not really to deliver working software - instead, it's multiple companies working together to figure out how they can maximally extract income from an entity with unfathomably deep pockets. It's layer upon layer of red tape sprinkled with the occasional sociopath just to spice things up.

The filthiest part is that nobody but the taxpayer is interested in things being more efficient. The government agencies don't really care because there is no real financial accountability. Besides, if they use up all their budgets, they just get to ask for more next year. And contractors generally have their profits limited to a percentage of the project cost, so their goal is to balloon the cost as large as possible.

For several years I worked on a very large government project. When you're sitting in a two-hour, 100 person, sprint review every two weeks and everyone is fucking off on their phones the whole time, it literally makes you question your sanity.

steven2012• 12 years ago

Feeding from the government trough is where people REALLY get rich. This is why Palantir is pretty successful, because getting in bed with the government means being able to feed off our tax money much easier at elevated prices.

duaneb2• 12 years ago

This is why Palantir is pretty successful

Ahhhh

81024602• 12 years ago

I have actually worked on the DCPS product as a former LM employee, so here's what I saw: there are two sides to the problem. The biggest problem is the number of stakeholders. Currently, each state has their own system to process disability cases, and each system has its own way of doing things. SSA & Lockheed are trying to consolidate these 50+ systems into one national system. Every state this product is rolled out in creates some new challenge to make it work the way the state's SSA office expects. Hopefully this new exec will have the power to make some decisions and say "no" once in awhile.

Lockheed has their own set of problems, mostly hinging on the contract they've signed with SSA. They are forced to hire a certain number of subs per their contract - hiring a percentage of subs is not unusual, but the level in this contract is very high. As such, the subs know they can force feed trash into LM. The resumes coming from the subs were complete bullshit. Things like "10 years of experience administering X", but then in the interview, you find out they know next to nothing. Especially bad with databases when you even try to ask them how to write a join. One meeting we had while I was still there was a "skills assessment" - from 1 to 10, how well do you know X? There was no negative to this meeting, we just wanted to know who knew what. The number of "10s" in that meeting was beyond belief.

So sure, Lockheed may not have a lot of experience here, but that problem is exacerbated by the stakeholders and the contractors they are forced to hire.

nazgul• 12 years ago

Why are the forced to hire subcontractors, and a certain # of them? Do you know what the thinking is behind that?

81024602• 12 years ago

The overall idea behind being "forced" to hire subs is that it helps promote smaller, local businesses. In the past this worked out very well, because the subs brought in some very knowledgeable people. I believe the old number was 25% of the people, and you could pull from anywhere. If you had a 20 person team, you could have 15 LM, 2 from Company X, and 3 from Company Y.

On DCPS, the problem is that LM is forced to hire 45% (FORTY-FIVE) of the subs through one specific subcontractor. That company had a previous contract with SSA, and SSA liked them. Since the sub essentially lost their work with SSA to LM, and they know LM was contractually obligated to hire through them, they stuffed us with a ton of below average folks.

On the face of it, I agree with the idea of a large contractor having to sub out some of the work to the smaller players; it's just that LM overplayed their hand trying to get this contract, and got screwed for it. But that's another story on how LM management is trying to make revenue numbers for bonuses...

androtheos• 12 years ago

No wonder taxes are so unbelievably high. The FBI did the same thing a few years ago. http://www.cnn.com/2005/US/02/03/fbi.computers/

duaneb2• 12 years ago

Yes, it's not at all programs like the absolutely-crucial F-35 program costing $399B up front.

junto• 12 years ago

McKinsey and Co probably charged the government a couple of million dollars to get some of its junior partners to write that report which eventually a senior partner signed off before golf and took the credit.

Says the cynic in me...

A $300M IT flop

Related Stories