List of failed and over-budget custom software projects

https://en.wikipedia.org/wiki/List_of_failed_and_overbudget_custom_software_projects

by monort • 10 years ago

154 111 10 years ago

en.wikipedia.org

jamessb• 10 years ago

IEEE Spectrum had a great series of interactive information graphics on 'Lessons from a decade of IT failures' [0].

In particular, "The Staggering Impact of IT Systems Gone Wrong" [1] and "Monuments to failure" [2] are good lists of failures, many of which are not included on Wikipedia's list.

[0] http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-...

[1] http://spectrum.ieee.org/static/the-staggering-impact-of-it-...

[2] http://spectrum.ieee.org/static/monuments-to-failure

thrownaway2424• 10 years ago

I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead?

For my money, a real failure is like the giant BART fiasco in the 90s where they spent 80 million dollars on a new train control system that simply never existed, at all. They had to sue General Electric to get their money back.

AnimalMuppet• 10 years ago

I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead?

If it didn't work? Hopefully not.

But military stuff is a bit different. Military systems tend to try to push the state of the art as far as possible, for as much edge in combat as possible. That makes for much higher chances of failure.

cthulhujr• 10 years ago

This article contains one or more incomplete lists which may never be able to satisfy particular standards for completeness

Well isn't that ironic.

SmellyGeekBoy• 10 years ago

Don't you think?

kozukumi2• 10 years ago

The modern NHS system in the UK was the biggest fuck up I have ever seen. I worked with people who also worked on the project. They are seriously amazing people but apparently all the layers of management, red tape and politics was, according to my friends, the worst they have ever experienced. Two friends were so traumatised by the experience they ended up taking 2 years off work to travel/recover.

The sad thing is it shouldn't have been such a difficult thing to develop. Complex sure but no more complex than many other similar systems. It was destroyed by too much meddling from the government adding, removing and changing the spec every other day and many, many managers just finding non-sense jobs that were totally pointless.

It is shocking it went so over budget and was allowed to continue doing so for so, so long.

teh_klev• 10 years ago

Which NHS body in the UK are you talking about? There are four separate healthcare services in the UK which operate and are funded separately:

NHS Scotland

NHS England

NHS Wales

HSC Northern Ireland

noir_lord• 10 years ago

NHS England - The National Programme for IT.

It is as far as I've seen the 'greatest' fuckup in UK development history, 12.something billion down the drain, massive overspending, they basically made every mistake you could make short of setting the office on fire.

kozukumi2• 10 years ago

I do not know how the funding was split but NHS England was the primary however I would assume all NHS bodies put something into the pot for funding?

tyingq• 10 years ago

Of course, there's a level of disclosure for Government run projects that means their blunders get exposed more easily.

I'm quite sure there are some spectacular failures in the corporate world that won't make the list.

pascalmemories2• 10 years ago

Sadly, no, they are not more easily exposed. Most are quietly swept under the carpet and no-one (other than those working on it) ever know it what a crazy mess it was. It gets classed as something else, is re-tagged as a proof of concept (so was never supposed to deliver anything other than improve understanding of an issue) etc.

The ones you hear about are the ones the government ministers foolishly chose to boast about in some answer in parliament or announce in some TV interview. Normally that's when consultants pounce on it and offer to help make it succeed [i.e. vampire all the money away in insane fees]. A minister announcing it is practically the death-knell of a project.

The ones which succeed generally keep publicity and announcements to an absolute minimum and keep tight control. That's way more difficult than you imagine on a government project as civil servants go hunting for projects to buddy up to in order to pad their promotion cv. If your project budget is more than the last one they screwed, they're all over it so they can boast about the incremental increase in projects they contributed to - a major promotion point (as no-one checks if the project succeeded, the promotion is hinged on how much public money it pissed away).

I was ecstatic to leave UK Gov projects behind.

DubiousPusher• 10 years ago

There's also an asymmetry in the way we think about public and private projects.

When a piece of private software flops, it still exists as a product and however bad, often offsets some of its cost in sales.

Since most public software is offered for free, no cost is visibly recouped.

Healthcare.gov is actually a great example. It was a colossal failure at first. A quantity easily measured in dollars spent. However, it has largely functioned in the years since but measuring its positive economic impact is hard because the service is delivered free. None if that money has been reclaimed but it certainly has had some positive economic value.

bdavisx• 10 years ago

No, they don't always flop and still exist privately - sometimes they are bad enough that they don't ever see the light of day anywhere - even internally to the company.

mrweasel2• 10 years ago

I should sit down and add the failed IT projects of the Danish government, but I don't have that kind of time. Our government is notorious for initiating software projects that ends in almost predictable failure.

The latest project to fail was a system for the Danish Tax Office. It was suppose to automatically collect debt owed by citizens and companies.It's not entirely certain how that was suppose to work and it has been described as a utopia project with an unprecedented expectation of what kind of problem computers are capable of handling. The specification where 9000 pages long, tax law not included, and some how failing to supply a sufficient amount of actual requirements. It failed, of cause, costing almost $150.000.000 (DKK 1.000.000.0000), not including the 78 Billion Danish Kroner the tax office haven't been able to collect due the system not being delivered on time and generally not working.

busterarm2• 10 years ago

Doesn't the government there (and most companies for that matter) contract IBM for big software projects?

You know the old adage "nobody gets fired for buying IBM".

I have a friend that's partner at a consultancy in Copenhagen and he says IBM is often their competition.

mrweasel2• 10 years ago

Mostly CSC actually. They have contract for systems that should have been part of licensing rounds, which would have allowed other to take over support, but no other where ever allowed to bid. The reason: CSC documented the systems so poorly that no one else will be able to handle the support.

The government knows that it's illegal, yet CSCs contracts have automatically been renewed, multiple times.

But yes, IBM have a large number of contracts as well. CSC just seems to be behind the worst projects. I suspect they bid for project and just focus on delivering exactly what's in the contract, knowing that it will never work. Either that or they are even more incompetent that anyone could imagine.

mixmastamyk2• 10 years ago

Why not add one, the biggest or most egregious.

lordnacho• 10 years ago

It has a lot to do with nobody being responsible. Everything goes through layers of management, none of whom are technical. You then have the top tax office guy going free because, you know, blame travels down.

What's sad is there's lots of smart coders in Denmark, a small team of which could easily do ANY of the many failed projects.

But because you need a certain certification by the bureaucracy, what tends to happen is one of the major players gets the mega contract and splits it with some of their friends, who split their piece with their mates, and so on until you have an organisational ball of spaghetti. And remember, the code will end up looking like the organisation.

orf2• 10 years ago

Can someone explain to me how the fuck my government can spend 15.8bn[1] on a software project? We could build 5 Queen Elizabeth class aircraft carriers for that. Five! For the cost of a bloated Java app!

Seriously, someone fill me in please. How can a software project possibly ever cost more than a billion pounds? You could employ 20,000 software engineers full time for £50K ($70k) a year for that much, and I expect the number of engineers working on Universal Credit is much lower than that and paid less.

1. https://en.wikipedia.org/wiki/Universal_Credit#Costs

detaro• 10 years ago

They don't employ the developers. The development is done by companies that might pay the developers that much, but bill the client more per hour and bill the client more time. And of course you can't forget project managers, architects and other consultants that are more expensive. And have a lot of meetings. And hardware, and oracle databases/SAP licenses, and support contracts for those, and audits, and extra charges for change requests, and travel budgets, and...

Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it.

And that's just development. Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system.

organsnyder2• 10 years ago

Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system.

Yup. My employer is undertaking a massive IT project (many many many millions of USD), and it's my understanding that training accounts for over half of the budget.

orf2• 10 years ago

Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it.

It doesn't mean you should pay, or plan, to spend 20,000 man hours in meetings and devote huge chunks of your budget to expensive licenses for something that could take 200 man hours, especially when it's taxpayers money.

Sure there are unavoidable overheads but why does our government keep splashing billions on them and failing, with little accountability?

Training is going to be expensive, sure, but perhaps plow millions of pounds into that after the program is finished. Maybe don't use the same massive, costly consultancies that fail to deliver? The Obamacare website was a great example of this IMO. Costly consultancy failed to deliver, small agile teams brought in to save the day - if only a bit too late.

sliverstorm• 10 years ago

And of course you can't forget project managers, architects...

Which, IMO, are extremely important investments in any 20,000-developer undertaking. Valve Software is famous for having a relatively flat hierarchy, but they also have ~330 employees.

Avshalom6• 10 years ago

According to Wikipedia in 2015 Facebook took in ~18billion and ~3.5 was profit which means it costs 14.5 billion dollars just to maintain Facebook, for a year.

tehbeard• 10 years ago

1. Facebook is a global company, not a national project.

2. Advertising/marketing budget is in there somewhere.

3. So are acquisitions (Oculus etc.).

4. Operational costs for their datacentres, ops staff etc.

morninj• 10 years ago

An enormous part of that $14.5b funds growth, not maintenance.

gaius2• 10 years ago

A normal programmer will be billed out at £500-1000/day by a big consulting firm.

By the way you cost your employer a LOT more than your salary. There's taxes, rent on the building, utilities, equipment and supplies, support staff, overhead staff, yadda yadda.

jandrese2• 10 years ago

Legacy costs are why. You think "oh, it's just a big record store with some privacy controls, no problem!", but pretty soon you're trying to integrate some COBOL based system hosted on an IBM 360 to load data that is stored in a poorly specified format (mostly just text transcriptions of whatever idiosyncratic method that practitioner used--everybody did something a little different) on the order of a hundreds of millions of records. You may literally have to hire and army of data entry specialists to retype every single record in your new format, millions and millions of them. And you can't outsource that part due to regulations, they have to be housed in a secure facility that you need to build and get accredited (by who? It's unclear, you need to figure it out yourself. Oh, and if you guess wrong you're going to prison.).

And this is before the lawyers get involved with their reams and reams of regulations that all make your job more difficult. If you think you could just dump the database and load it on the new system think again, that's an unacceptable data risk, you need to broker all of the data through yet another system.

huuu• 10 years ago

Risk is also what raises prices:

  - We are going to fine you if you don't deliver on time
  - Ok, but then we have to tripple our price

flatM• 10 years ago

One word: rent-seeking

atom-morgan• 10 years ago

During my senior year of college I discovered, while working on my senior project, that the head of my state's financial department had a lot of outsourced software projects. Who builds those software projects? A contracting company. Who owns that contracting company? The head of the state financial department.

jonesb62• 10 years ago

That.. deserves its own hacker news post.

easytiger• 10 years ago

Assholes like this:

Richard Granger the former Director General of IT for the NHS, took up his post in October 2002, before which he was a partner at Deloitte Consulting, responsible for procurement and delivery of a number of large scale IT programmes, including the Congestion Charging Scheme for London. In October 2006, he was suggested by The Sunday Times to be the highest paid civil servant, on a basic of £280,000 per year

Deloitte are a cancer on society and business. Presumably he learned how to ruin companies there/

OhHeyItsE• 10 years ago

Would have been much easier to create a list of projects that were completed within budget (assuming that's ever actually happened).

softawre• 10 years ago

Hey, I wrote a little shell script yesterday that I had budgeted 30 minutes for and had it done in 20.

But outside of that one thing.. lol

rdudekul• 10 years ago

I was trying to look for patterns in these projects. Looks like most of them were outsourced and most of them were US/UK Government projects. It would be interesting to see patterns in very successful US/UK Government projects.

My personal experience has been that it almost always boils down to upper management/leadership issues, that result in hiring weak project managers which in turn leads to talent drain on the implementation side.

gaius2• 10 years ago

What it boils down to is no-one is spending their own money, they have a bottomless pit from the poor taxpayer.

panglott• 10 years ago

Why are these all in the UK or US? Better transparency?

slantyyz• 10 years ago

> Why are these all in the UK or US? Better transparency?

Maybe because that particular Wikipedia page's editors are from the UK or US? [edit: just speculating, don't know this for a fact]

If I recall correctly, the Gun Registry [1] in Canada was a huge, expensive debacle, with a lot of the costs coming from the underestimation of the software development.

[1] https://en.wikipedia.org/wiki/Canadian_Firearms_Registry

jordache• 10 years ago

when they list healthcare.gov has costed 1.5bn, does that include setting up the back office and staffing, or simply the project that responsible for putting the website?

strommen• 10 years ago

In 2006 I worked briefly on an air-traffic control system (ERAM) that was eventually delivered 5 years late and $400M over budget. I was a small cog in a giant government-contract-software machine, but it was easy to see the project was going to be a disaster.

There are some pretty basic strategies to make building "large" software easier. All of them were attempted, but in an entirely broken way:

- Break the architecture down into smaller components that can be tested independently.

- Break the functionality down into smaller features that can be released independently.

- Develop automated tests at the system level (i.e. integration tests, not unit tests).

The software took 2-4 minutes to launch in development, and very few of us knew how to meaningfully test it. This meant that any change that compiled (and passed code review) would ship.

Oh yeah - and the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.

dotnetchris2• 10 years ago

the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.

What could go wrong

Someone1234• 10 years ago

"We should have just used XML for everything!" - said every Java developer ever.

"We should just have used binary for everything!" - said every C/C++ developer ever.

"We should just have used Json for everything!" - said everyone currently.

duaneb• 10 years ago

I mean, you could argue the network stack is a "binary structure in shared memory to communicate across processes". Really it depends on how it's done: the interface is everything.

jimjimjim• 10 years ago

There is nothing wrong with binary protocols.

Sharing c structs across languages is brave but it is doable as long as padding/offsets/lengths are expertly crafted and very strictly controlled.

but on the other hand sharing pointers would be madness.

joelhaus• 10 years ago

Clay Johnson had some relevant comments during the flawed healthcare.gov launch. Many critiquing the procurement process itself. As you stated, the process generally favors the "government-contract-software machine":

http://blog.dobt.co/2013/10/07/the-healthcare-gov-fiasco/

jayvanguard• 10 years ago

I'm surprised the outsourcing companies aren't listed on this. They are often huge contributors to the overruns.

amist• 10 years ago

As a side note, I cannot understand why these articles are written in a table format. Where there are cells with such a long description, bullets would be much suitable.

organsnyder2• 10 years ago

Agreed on the formatting, but the sorting functionality can be handy.

Grue3• 10 years ago

...in UK and US. Apparently other countries never had any of these.

edgyswingset2• 10 years ago

I remember, when as a resident of Oregon, I was wondering why we couldn't sign up with the national healthcare system. I then noticed that the Cover Oregon system was being built by Oracle.

Suffice to say, I wasn't the least bit surprised that it was a complete and total catastrophe, and even less surprised that Oracle tried to sue Oregon for attempting to use the software. Nothing like LDD - Lawyer Driven Development.

Yhippa• 10 years ago

I remember this being quite the story. Anybody know what came of it? Still in litigation?

schmichael• 10 years ago

Still in litigation. OR has spent more than $6.6m in legal fees fighting Oracle.

phonon• 10 years ago

Still in litigation?

Yup.

agumonkey• 10 years ago

for attempting to use the software

I don't understand that

edgyswingset2• 10 years ago

Info here[0].

[0] http://www.law360.com/articles/578344/oracle-adds-ip-claim-a...

api• 10 years ago

I see a lot of billions, with a b, and then I look at what small teams do on bootstrapping or little seed rounds. The mind boggles.

jandrese2• 10 years ago

A huge, huge, cost driver in many of the applications is the fact that they have to integrate with the old systems in numerous ways.

A startup could probably build the basic version of many of these apps no problem, but that's only the tip of the iceberg. When you have to then import millions of customer records, which were not stored in any particular format (they're idiosyncratic with each doctor/office), and then integrate with the COBOL based system vendor 1 uses, and the IBM Mainframe that vendor 2 uses, oh, and those systems were never intended to interconnect with anything and the vendors are years behind schedule as well, and all while you have lawyers breathing down your neck with a 28 inch thick binder with all of the regulations you need to follow written in impenetrable legalese you can see how it can cost billions of dollars.

Your startup guys heads would explode if you asked them to write a formal specification for an interconnection system between their elegant web frameworks and a big blob binary COBOL based record system on an IBM 360 with only hardcopy documentation written by a vendor that went out of business in the 80s. And yes, healthcare systems still run in environments like that for the reasons mentioned above: making a new system is too hard/expensive.

Oh, and the vendor that had the lowest bid for the contract is planning to outsource to some country that doesn't speak English and is 12 hours offset from your local time.

jonesb62• 10 years ago

It still seems like an excuse, "it's hard". The cost of "hard" shouldn't be $15B.

ergothus• 10 years ago

The primary problem with lists like this is that they don't compare to successes.

For example:

* I've been on projects where we're rewritten applications from the ground up that were totally successful and we were better able to build in the future because we had a clean setup.

* I've seen government IT shops hire contractors to do a large project and ended up with something better than the internal devs would have done.

Both of these are situations where the opposite case (time-consuming and expensive failure) gets a lot of attention. (and, in my experience, failure is more common). Show me one of those failures and my instinct is to say "Of COURSE that was a failure! It was obviously going to fail!". But, in reality, not always.

Rather than building unhelpful conventional wisdom, we should be trying to understand the distinctions between the failures and the successes.

pessimizer• 10 years ago

SAP doesn't even get a mention?

Waste Management: http://www.computerworld.com/article/2517917/enterprise-appl...

Levi's: http://www.theregister.co.uk/2008/07/10/levis_erp_costs/

mixmastamyk2• 10 years ago

It's wikipedia, add it. ;)

mschuster91• 10 years ago

The problem with SAP isn't the software, it's "consultants" writing purposefully bad code - the customer can't assess the delivered quality and the consultant will keep his "ongoing maintenance" fee.

(Okay, yes, the software itself is a problem too but it's not the major problem)

pkorzeniewski• 10 years ago

When it comes to government projects, I always wonder how many people involved treat it only as an opportunity to make as much money as possible without being caught - after several years you just sweep the whole thing under carpet so nobody remembers how much money has been wasted and you're set.

zhte415• 10 years ago

This is true not only of government projects, but of all projects.

Anyone in charge of a budget is potentially corrupt. I'd argue that it is more common in corporates than in government, because the personal damage that could be inflicted is less. A company doesn't want reputation damaged by being involved in fraudulent bidding (so will sweep under the carpet readily), and as long as the job gets done, nobody cares (unless internal politics get stirred). The employee may be fired, but they can go somewhere else. And all is off the public record.

TheAndruu• 10 years ago

This is by no means a conclusive list.

codingdave• 10 years ago

That list is much short than the headline would imply. I was afraid of a list of thousand of projects, with a final footnote of "Every Enterprise software application ever written."

timwaagh• 10 years ago

this is why haskell has support for infinite lists

frik• 10 years ago

Now we need a list of overbudget customized COTS software projects.

With all these companies running on Excel macros, years old SAP R/2 or 3 (just two common examples) and so on, the list will be extensive.

(COTS = commercial off-the-shelf)

rbanffy• 10 years ago

That'd need a whole Wikipedia just for it.

tyre• 10 years ago

96% of government IT procurements over $10m are deemed failures.[1]

Absolutely shocking how ineffective the system of contractors and RFPs is. Part of it is a mindset, part of it is just needing better options.

[1] https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-l...

mcguire• 10 years ago

These lists seem a little short. What about:

* The Distributed Computing Environment (or at least the OS/2 portion of it, anyway), https://en.wikipedia.org/wiki/Distributed_Computing_Environm...

* Taligent, https://en.wikipedia.org/wiki/Taligent

* IBM Workplace OS, https://en.wikipedia.org/wiki/IBM_Workplace_OS

And that's just a few from off my personal resume.

mrslave• 10 years ago

I will document my bias before reading the article in the hope that I am proven wrong.

I estimate that most of these have occurred in apparatus of the state, including organizations that superficially are not but for all practical purposes (funding, license, regulation) are.

Wise man once say: there's nothing easier in this world than spending someone else's money.

peterburkimsher• 10 years ago

Did nobody mention Xanadu?

natetan• 10 years ago

Great idea.

The UK tried a NOMS (offender management) system and that was an expensive disaster too..

Would be nice to put down the contracting companies who were ripping of the Govt along with the civil servants who over saw the massive waste of tax payers money/time.

ap22213• 10 years ago

Note: very few, if any, of these examples failed because someone on the team didn't pass a coding test. Next time you interview someone remember: bad code could bring your project down, but there are 9 other factors that are more likely to bring your project down.

repeek• 10 years ago

For those interested in this topic, I highly recommend episode 181 of the Omega Tau podcast, "Why Megaprojects Fail (and what to do about it)" [1].

It's a fascinating listen into the psychology behind gigantic failures.

[1] http://omegataupodcast.net/2015/09/181-why-megaprojects-fail...

tetraverse• 10 years ago

"Sabina SEIFERT's Project: Software Project Failures"

http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj12...

stretchwithme• 10 years ago

The Software Engineering Radio podcast did an episode on this recently:

http://www.se-radio.net/2016/02/se-radio-epislode-250-jurgen...

boxfire• 10 years ago

List of failed and over-budget custom software projects

Related Stories