Healthcare.gov crashing because backend doomed in the requirements stage

http://www.forbes.com/sites/anthonykosner/2013/10/21/obamacares-website-is-crashing-because-backend-was-doomed-in-the-requirements-stage/

by blurpin • 13 years ago

45 54 13 years ago

forbes.com

wheaties• 13 years ago

I can't wait to see what the report of it didn't work actually says. It's quite possible that one of the services that the website hit couldn't handle the traffic which in turn caused a bottleneck in other systems.

mrharrison• 13 years ago

I know the people who worked on this. This is not what happened. Just yet another propaganda piece by Forbes.

pfraze• 13 years ago

Can you elaborate on what did happen?

briandear• 13 years ago

Well something happened. You might want to tell those guys that 500 million lines of code is a bit ridiculous.

dodyg• 13 years ago

You know what's cool? 1 billion lines of code.

twistedpair3• 13 years ago

I wonder what those LoC were for? XML parsing remote API calls between various disparate systems? Sounds like it's well beyond operation system complexity. Linux kernel is currently 15M LoC.

Then again, kernels and operating systems are quite logical and laid out by architects. Systems like these are rats nests defined by bureaucrats and integration nightmares.

darkarmani• 13 years ago

the interface kept asking me to reenter the information for our dependents. Farther on in the process it doubled the number of our children (to 6!) Even though it listed our children’s names twice (identically) and even though these clones had identical SS numbers, the validation software was not smart enough to realize that the duplicate records were describing the same people!

This didn't happen? Are you suggesting he made this whole story up?

sp332• 13 years ago

The article, as the headline states, is about poor requirements dooming the project.

GVIrish3• 13 years ago

Can they elaborate on what actually did happen? If I were your friends I would be trying to contact news agencies with corrections if at all possible because this is shaping up to be the highest profile government IT failure in US history.

sunraa• 13 years ago

Per the NYT: "But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process."

Source: http://www.nytimes.com/2013/10/13/us/politics/from-the-start... (October 12)

And from today (Oct 21): http://www.nytimes.com/2013/10/21/us/insurance-site-seen-nee...

In a previous life I worked on similar software development projects at the US State level. This project was several orders of magnitude larger in scope and in complexity than what I worked on. Assuming a generous start date for Spring - say March 1 - this boils down to ~7 months to design and develop to the Oct 1 hard deadline. Nearly impossible in my opinion.

I'm not discounting that there couldn't be issues else where in whole process. These would include: communication overhead between all the various agencies and contractors, integration nightmares, and not enough skilled folks at Medicare to manage all of this. But having the requirements delivered several months before roll-out seems to me a recipe for disaster.

Retric• 13 years ago

You don't need 100% final requirements to start building the system. If nothing else start working though the interfaces with other systems, and try and get as close as possible to a working prototype. Granted funding is often an issue, but even a tiny team given a few months head start can get a lot done.

rco8786• 13 years ago

I don't know if I would use the word propaganda...but speculation definitely fits the bill.

gry• 13 years ago

Thoughtback [https://thoughtback.com/] sent me a pertinent reminder this morning. No affiliation, just a great tool.

    Organizational metrics, which are not related to the code, can predict software
    failure-proneness with a precision and recall of 85 percent. This is a significantly
    higher precision than traditional metrics such as churn, complexity, or coverage 
    that have been used until now to predict failure-proneness. [1]

[1] The Influence of Organizational Structure On Software Quality: An Empirical Case Study

http://research.microsoft.com/apps/pubs/default.aspx?id=7053...

edit: formatting

joshdick• 13 years ago

"If you have four groups working on a compiler, you'll get a 4-pass compiler" -- esr

gesman• 13 years ago

Could anyone share which platform / framework they used?

Iftheshoefits• 13 years ago

There probably isn't any one "platform/framework" they used. This beast touches too many systems, many of which are 'legacy' for the phrase "platform/framework" to really apply.

If it's anything like what I saw at the one (and only) government contractor I ever worked for, the site is a frankensteinian nightmare of COTS ('Commercial Off The Shelf') crap (which federal contracts encourage, when they don't outright require it) and sweatshop-style integration development run by contractors who hire anybody with a heartbeat and a handful of competent people, and pay them all peanuts while billing $200/hr in labor to the government.

jonknee• 13 years ago

I'm sure tons. The web site's code is pretty straight forward, it's that interacting with so many other parties that is not (besides anything having to do with actual insurance, Healthcare.gov interacts with the IRS, social security, homeland security, Equifax, etc etc).

I'm not sure what the solution is since these checks are not optional. It would be a huge political scandal if just one illegal immigrant was able to buy insurance using Healthcare.gov and there are surely many groups trying every day to make sure this happens.

Is there any large benefits system that works well? All that I have seen from MegaCorps are also awful and they don't even have all the complex stuff to make sure you qualify (since you work at the company, you qualify).

babs474• 13 years ago

mnsure.org was throwing springframework exceptions the day it launched

uxp• 13 years ago

healthcare.gov itself was a Ruby Jekyll static site. It was on GitHub[0] for a while, but mysteriously disappeared. Here[1] is a clone of it, and what seems to be the currently agreed upon canonical version.

My state appears to be using Joomla based on the meta-generator tag in it's head tag. http://www.avenueh.com/

[0] https://github.com/CMSgov/HealthCare.gov-Open-Source-Release

[1] https://github.com/blencorp/HealthCare.gov-Open-Source-Relea...

powertower3• 13 years ago

In my case, even the most expensive plan for our family in the State of Maine, where we live, is half the cost of what we are currently paying, exclusive of any subsidies.

The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-...

chimeracoder• 13 years ago

The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-....

It still boggles my mind that people are surprised by this. Mathematically, including people with pre-existing conditions will mean either that those people are required to pay for the (expected, not actual) costs of their own care, or that other people will be required to pay for those costs[0]. This effect is much larger than the effect of mandating insurance for younger, healthier people who otherwise elected out of getting insurance[1].

Note: this is not a normative statement about whether this is a "good" outcome. This is just a a mathematical observation about the way insurance works, from a mathematical and economic perspective.

[0] Subsidies come from taxpayer money, so they have no net effect on this accounting - it changes how the money changes hands, but the money still has to come from somewhere.

[1] I'm waving my hands around the math here, but if people are interested I can dig up the more detailed blog post draft that I did a while back explaining this in full detail.

lukasb• 13 years ago

The effect that you're ignoring is that insurance premiums have a hefty amount of margin built into them, and the ACA produces a collective bargaining effect that gives insurers an incentive to offer a competitive price on the exchange.

powertower3• 13 years ago

It still boggles my mind that people are surprised by this...

I was surprised because I was told that this new coverage affordability came from getting insurance companies greed in check.

I didn't know that insurance companies were actually lobying for Obamacare, not against it.

http://www.policymic.com/articles/62541/here-s-the-proof-oba...

joshuahedlund• 13 years ago

It still boggles my mind that people are surprised by this.

I agree with your mathematical observation, but maybe part of the surprise that such math was often underplayed or outright ignored in the claims that premiums would be going down, ex. [0][1])

[0] http://www.washingtonpost.com/blogs/fact-checker/post/presid... [1] http://www.nytimes.com/2013/07/17/health/health-plan-cost-fo...

001sky• 13 years ago

It still boggles my mind that people are surprised by this. Mathematically, ...

Its an interesting social phenomenon, that large-scale opaque repricing of a commodity is met with opportunistic ~scalping. This happened accross europe during the conversion to the Euro. US consumers are probably rightly suprised.

Despite the expectations of economists that the euro change over would have no effect on prices, European consumers perceived the opposite. To shed some light on this puzzle, we develop a model of imperfect information in which cheaper goods experience higher price growth after the changeover. Retailers, aware of consumers’ difficulties in adopting the new currency, use currency changeovers to increase profits by in creasing prices. The lower the price transparency after the changeover, the higher the euro–related inflation. Using data on inflation (Eurostat) and price levels (Economist Intelligence Unit), we show that although the euro changeover did not significantly increase inflation, it nevertheless had distortionary effect on prices. {etc}

http://www.kellogg.northwestern.edu/faculty/dziuda/personal/...

jamroom2• 13 years ago

Just want to add another data point, that here in Washington State, a comprehensive plan for my family is going to be around $600 less per month under Obamacare than what I can get right now. This is without any subsidies of any kind, so while I know it is just 1 extra data point, I consider myself a "typical case" (3 dependents, 1 pre-existing condition).

I can say that the Manhattan institute's numbers (what the Forbes article is based on) for Washington appear to be incorrect (based on just my experience though), as they list a premium for a 40 year old man to be currently $181. I've been buying individual health care for myself and family for the last 8 years, and maybe back in 2006 it was about that price. A "comprehensive" plan in Washington would easily set me back ~$1300 per month for my family of 5 - under Obamacare the same (or even sightly better) plan would be around $750. I currently only carry a "catastrophic" coverage plan, and even with a $7500 deductible it's still $600 per month.

So for me I'm looking forward to Obamacare.

powertower3• 13 years ago

The problem here is that for some people's coverage to be lowered, impoverished people to be covered, and pre-existing conditions paid for, a bunch of young and healthy individuals (without families) are now required to sign up for insurance. And the cost of that insurance for those healthy individuals is now 2-4x more expensive than it would have been before Obamacare took place (obviously the money has to come from somewhere to balance it all out). *This is of course for the lower and mid level coverage plans, the higher end stuff reduces that margin between the pre and post levels - but not for everyone, and in some cases people are going to pay 15-20% of their TOTAL income for worse plans. And that's going to enrage those people.

joering2• 13 years ago

Just want to add another data point, that here in Washington State, a comprehensive plan for my family is going to be around $600 less per month under Obamacare than what I can get right now.

I get your point and I congratulate you. However, please don't be prematurely happy, "because $600 less" is not what will fix you -- the coverage itself will.

Let's hope that when you need that coverage, you won't be treated the same way as ObamaCare website right now is. You see, before you could complain to your health plan, make a big stink out of it, even sue them, because there were private entity. When to GOV takes over, all you can is pray. Go ahead make a big stink that their system is broken, but don't be surprised to get heavily audited, for example.

My point is that I would prefer to pay even $6000 more knowing I will get the quality of coverage I pay for. I would not want to pay less to be hit with bureaucracy, abuse, lack of transparency and fairness the way government will serve me like they did with their website.

lightstrike722• 13 years ago

Adding another anecdotal Washington state data point, my private individual health insurance plan (Regence Evolve HSA) with a premium of $149/month will no longer be available come January. The plan most similar to my current one is $252/month (Regence Direct Silver), which is a pretty massive increase. As a healthy, single 25-year old though, I knew some version of this was coming.

SeanLuke• 13 years ago

Citing Avik Roy does not help your claim. See http://www.washingtonpost.com/blogs/wonkblog/wp/2013/06/01/t...

mbesto• 13 years ago

If there is anything that can come of good of my taxes (a ridiculous amount of it in fact) is that we should educate people on the difficulties of software development. This would make a fantastic case study on what I presume are the issues at hand:

- Maintaining complex code (+1mil LOC)

- Classic "waterfall" approaches to software development (and the inherent problem with trying to blueprint requirements up front)

- The mythical man month[1]

- Change management

Funny thing is...these are all human problems.

[1] http://en.wikipedia.org/wiki/The_Mythical_Man-Month

drakaal• 13 years ago

I didn't see anything that meant it was doomed from the requirements stage.

This was more of an Obama love fest and defense of ObamaCare than a technical article describing who the backend was doomed.

For what this cost to build it could have been much better, much simpler, and handled the scale gracefully.

Yes privacy concerns keep it off the elastic cloud. (not a point mentioned in the article)

Yes authenticated users limit caching and make scaling harder. (not really mentioned)

But when you are talking about everyone in the US, you only have so many combinations. Income in something like 32 ranges, number of kids 1 through 12 (greater than 12 they are all free) a bunch of ages.

We just saw a post that only 15 searches are new to google each day, so the number of combinations is much less than search which does lots of caching.

The requirements were messed up, the implementation was. Clearly not built by people who had ever done "Facebook scale" stuff.

GVIrish3• 13 years ago

Well, some reports are that the requirements were changing until 1 week before launch, and that development didn't start until March of this year due to requirements delays. Just those two factors alone would doom just about any project of this scale.

coldcode• 13 years ago

The primary contractor has had plenty of large gov projects including some for Medicare and SS. Of course so have all large consulting firms and they fail often as well. What the issue is isn't clear from random musings. Unless someone who worked on the project at a high enough level can explain all we have is guesswork.

twistedpair3• 13 years ago

They screwed the pooch, clearly. However I wouldn't assume that the insurance landscape is so simple. You're talking about an insurance industry and legal system that has had a fifty year head start making the process of insurance buying as complicated and obfuscated as possible.

chucknelson• 13 years ago

Wow, this particular excerpt from the article is stretching things a bit too thin:

"Clearly there are problems with the government’s IT procurement processes, but if the same level of dysfunction had been present in the software behind Obama’s political campaign (as it was, apparently, behind Romney’s) the President would have never been reelected."

Really? I doubt the campaign websites were the deciding factor.

Anyway, a lot of speculation in this article...not very insightful or informative.

ebiester• 13 years ago

The software in question was the software used to determine the best media buys, the volunteer organization software, and similar that was known to be part of his advantage.

nightpool• 13 years ago

See http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864) for comparison.

skwirl• 13 years ago

They aren't talking about the campaign websites. Both campaigns had complex internal software systems to help target field worker campaign activities, get out the vote efforts, monitor and respond to turnout data on election day, etc.

According to various reports, the Obama campaign was hugely successful in this endeavor [1] while the Romney campaign's election day voter turnout system was a disaster [2].

[1] http://www.drdobbs.com/architecture-and-design/software-deve...

[2] http://www.businessinsider.com/romney-project-orca-disaster-...

twistedpair3• 13 years ago

But that is a very agile, very incentive aligned project. You don't need to negotiate compromises and tip toe around various laws to run some stats for your media predictive analytics. Your vendors want you to continue in the campaign, so they still have contracts. Everyone is running for the same goal line.

Health care and insurance entitlement programs are clearly orders of magnitude more quadmired before a single line of code is written.

Plasmoid3• 13 years ago

Why government software projects go wrong [http://www.mrobinson.ca/2012/09/why-government-software-proj...]

ibid - Part 2 [http://www.mrobinson.ca/2013/08/government-projects-going-wr...]

teyc• 13 years ago

I bet you half the problem is the software being specified in a synchronous and not asynchronous manner.

If a flow chart is expressed synchronously, it immediately bottlenecks around the slow backend systems. e.g. check the applicant has 4 kids, and if that checks out, present them with these options.

I attribute some of these problems to inadequate advances in today's mainstream programming languages.

For instance, if futures/promises were common parlance in programming, some of these bottlenecks would have been identified earlier.

I once read an interesting essay on what future programming languages could work on. One of the interesting ideas is a language that allows programmers to implement undo/redo easily. Recently there was a blog post about fuzzy computing, where a function is able to return approximate values when it is busy. The Knight Capital incident teaches us that if we architected our monitoring systems the same was as intrusion systems are done, we could have stopped activity when suspicious incidents occur outside the normal operating range. There was another incident where a Japanese trader accidentally keyed in the wrong number and promptly lost his employers a lot of money. Languages that incorporate ideas like validation and expected values as first class constructs might be able to help here. The Adriane reminded us how badly humans do when reasoning about systems where multiple agents perform the same task.

Healthcare.gov crashing because backend doomed in the requirements stage

Related Stories