IBM to pay more than $30m in compensation for Australian census fail

http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-in-compensation-for-census-fail/8057240

by timv • 10 years ago

87 42 10 years ago

abc.net.au

cyberferret5• 10 years ago

That's a pretty hefty fine, but I wonder what the original billings were for developing the system in the first place?

I've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)

I guess we can be thankful that is was 'merely' a census and not something like electronic voting!

timv2• 10 years ago

I believe the original contract with IBM was only in the order of $10m

See: http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...

freehunter• 10 years ago

That's actually insane. I work for a different part of the company, sure, but I'm on an $11m USD project right now, as just one of my ~5 projects.

For as much as I've heard about this census issue, I thought it was a bigger (monetary) deal.

unsignedqword• 10 years ago

In the US, Census data is used for redistricting purposes, which CAN influence election outcomes in the future - so the implications of a census may be wider than you think. I don't know exactly the way it works in Australia, but some googling has led me to believe they use a system that is similar.

https://en.wikipedia.org/wiki/Redistribution_(Australia)

cyberferret5• 10 years ago

Absolutely, I didn't mean to downplay the importance of a national census. It is just that the results are used for longer term project planning and policy making, so the effects aren't as immediate as a whole government inadvertently being kicked out or brought into power as a result of an election day debacle.

Haydos585x2• 10 years ago

Another major problem with the census was the Australian Bureau of Statistics holding onto the data and changing their privacy policies which meant that in an act of protest a ton of people entered false information or ommitted details. A really poor census overall.

rodgerd• 10 years ago

Australia - or more specifically - already have a known-vulnerable electronic voting system they used in an election, so that horse has already bolted.

fowl2• 10 years ago

All our votes are carried out on paper. If whatever scheme they're using for collation is compromised/suspect a recount could be done entirely by hand.

shakna3• 10 years ago

I'm not sure it is a hefty fine for the situation. Its simply paying the remaining costs for IBM's failures.

I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.

"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]

Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]

And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.

As to their reputation, IBM doesn't have a brilliant name already:

* Failure to meet deadlines [4]

* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.

* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].

Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.

[0] http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...

[1] http://www.abs.gov.au/websitedbs/censushome.nsf/home/privacy

[2] http://www.news.com.au/technology/online/security/ibm-faces-...

[3] http://www.businessinsider.com.au/ibm-surprises-with-q2-2016...

[4] http://www.abc.net.au/news/2016-09-27/ibm-unlikely-to-hit-cu...

[5] http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...

[6] https://en.wikipedia.org/wiki/IBM_7030_Stretch

marktangotango• 10 years ago

Stories of IBM versus state or national governments come up fairly regularly. This one is unusual that IBM was actually found guilty/culpable/whatever. Usually they weasel out on some contractual loop hole.

cyberferret5• 10 years ago

Availability and load balancing aside, did any other Australian citizen (who actually managed to complete the census at all) notice that the UX was quite ordinary? Reminded me of a web form circa 1990.

For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.

Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.

jestar_jokin• 10 years ago

Except you may have step-children, adopted children, foster children, children in the house not part of your family (friends of your child, children of your friends), etc. Keep in mind that the census needs to include everyone who was in the house at the time it was filled out, not everyone who is part of your regular abode/family situation.

You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)

Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.

(Having said all that, I haven't seen the online census.)

shakna3• 10 years ago

UX wise, a radio button that hides that section labelled "same as John Doe" might have given that flexibility without the pain of rewriting a dozen things again.

cyberferret5• 10 years ago

Sure. I did consider the scenario if the kids had a friend sleeping over that night or something, or if the respondents had adoptive or foster children etc., but I think when I traced back the line of questioning in my case there seemed to be a logical thread link there that could/should have negated asking for the same information again.

I am relying on my memory here from few months ago, so I could be wrong, but I did feel at several points during the session that quite a few questions could quite possibly be chained together in better fashion. Perhaps not so much a failing of the actual site UX designer, but the census analysts who designed the questions in the first place?

PhilWright• 10 years ago

I thought the presentation was very good. I was pleasantly surprised that the design was professional and modern looking.

It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.

ACow_Adonis4• 10 years ago

See my other comment further down, but I just thought of another issue again:

Design consideration has to be given to how/what people will be using to access said questions.

If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.

beedogs• 10 years ago

We requested paper forms long before census night, because we were pretty sure this was going to happen. Didn't put our names or any other personally identifying info on them, either. (Never received a fine.)

jacques_chester3• 10 years ago

The Prime Minister is correct in saying that it was a predictable debacle.

Because it's bloody hard to DDoS paper forms. Widely distributed atoms are curiously disinterested in IP packets.

stirlo• 10 years ago

Before census day they were predicting two thirds of the 10 million Australian households to complete the census online. They stated their site had been tested with half a million concurrent users and was able to sustain double that.

Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.

What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.

ACow_Adonis4• 10 years ago

I noticed the exact same thing. Read about their load testing in the paper, did a quick back of the envelope, and concluded before the census that they hadn't texted big enough.

Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...

6nf• 10 years ago

There likely was no DDoS. This was an early guess based on the symptoms at the time, but after investigating IBM never claimed DDoS in the days following census night.

candiodari• 10 years ago

I often wonder why companies and governments even trust large organisations like IBM over individuals they've interviewed themselves. I mean, it's not like IBM's consultant's competence is any secret. IBM is also trying to make it worse, or at least, not making the competence factor any priority. Rather, it's cost over everything:

http://www.businessinsider.com.au/ibm-layoffs-1-month-severa...

I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).

ACow_Adonis4• 10 years ago

Such individuals are usually unable to detect real signal in the thing they're hiring for, so interviewing in and of itself doesn't provide them any qualitative information.

They also know the people judging them are in the same boat.

Thus they fall back to the only thing understandable to themselves and people who will judge them: big brand and prestige/shininess. If the project succeeds, you take credit. If it fails, you can't be blamed for "buying the biggest and the best" (even if the biggest and the best are just a bunch of stary-eyed ignorant grads with a salesman/project manager at the head).

/that's the good version. The bad version points out how many of these firms gets business through networks, personal connections and "you scratch my back I'll scratch yours" type dealings. //it's not a rant if it's true.

Freak_NL• 10 years ago

Liability. If the project fails, it is not your fault.

I cynically suspect that whoever drafts these contracts might even believe that by agreeing on an excessively expensive failure clause, failure will thus not occur — after all, it is legally covered.

We see this in The Netherlands as well (and I suspect it is a global phenomenon). Government needs software, big-name consultancy firm gets the job, builds stuff that won't work or barely functions, project implodes or is delivered in a half-baked state. Repeat ad nauseam. These companies (Cap Gemini comes to mind) thrive by these projects, even though most seem to fail or end up underwhelming. There appears to be some degree of corruption there on the side of these contractors — investigative journalism has uncovered several questionable practices.

Some part of my wonders if having a well-funded government department dedicated to these projects that creates and maintains the (free!) software needed is not a more tenable (and cheaper) approach. On the other hand, large government-driven software projects may just be impossible to deliver at this point; we simply lack the know-how to prevent the inevitable implosion of such complex projects.

bitmapbrother• 10 years ago

My only disappointment is that they didn't have to pay more for their incompetence. IBM has been on a mission to replace their workforce with cheap labour from India for quite a while. I'm glad the work they deliver to their clients is finally being exposed for what it is - junk code with no quality control.

dghughes• 10 years ago

Just wait until IBM's Phoenix Pay system and the scandal here in Canada comes to a head. With 300,000 civil servants waiting six months to be paid and it's still not fixed.

There must be a big penalty looming for such a massive level of incompetence.

Phoenix seems like a rather awkward name for an application.

http://www.nytimes.com/2016/11/18/world/canada/government-wo...

Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.

My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.

nihonde• 10 years ago

As someone who has negotiated similar contracts with providers like IBM, I'm willing to bet that someone on the customer side gave into the pressure to rush the lawyers while IBM did a nice job of generating loophole-filled statements of work, and I wouldn't hold my breath for that penalty.

adambrenecki• 10 years ago

Oh God, that sounds an awful lot like two more Australian systems: the Queensland Health payroll debacle, where a bunch of health workers were paid incorrectly or not at all. Or the National Disability Insurance Scheme debacle, where healthcare providers haven't been paid, resulting in small business owners having to take out huge amounts of personal debt just to pay their employees, and sole traders unable to provide services their clients need to live day to day while still keeping a roof over their own heads. (They're both IBM projects too.)

shakna3• 10 years ago

Do you have a link for IBM's mishandling of the NDIS? Interested, but can't find a link.

justinsaccount• 10 years ago

I wonder what the infrastructure provided by IBM consisted of.

Based on non-cloud setups like http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...

A rack or two would have been able to handle the load.

will_hughes• 10 years ago

You have to understand that what StackOverflow gets away with in terms of hardware is not at all reflective of what other shops will be able to do.

SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).

Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it. The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.

suprjami2• 10 years ago

I recall someone on Reddit detailed the infrastructure, I cannot find the post again now sorry.

However it was all non-cloud self-hosted hardware.

sqldba• 10 years ago

Ummm the article isn't very clear.

It says the damage was estimated at 30 million. It says IBM paid something. It didn't explicitly say IBM paid $30 million.

SyneRyder• 10 years ago

First line of the article:

"Computer giant IBM will pay more than $30 million in compensation for its role in the bungled census"

IBM to pay more than $30m in compensation for Australian census fail

Related Stories