The Obamacare software mess

http://www.economist.com/news/united-states/21587816-signing-up-obamacare-still-ridiculously-hard-obamacare-software-mess

by csa • 13 years ago

62 47 13 years ago

economist.com

pasbesoin• 13 years ago

This situation has reminded me of the Perl Success Stories booklet that O'Reilly used to spread around. And how, for example, the Bank of Scotland or some such institution ended up keeping the Perl prototype for production, it worked so well.

I can help thinking that a few good Perl programmers, and a few high-level bureaucrats to force open the maws of the respective agencies, could have had this sorted in much shorter, and more cost-effective, order.

Also, as I commented elsewhere before, first make it work. Then make it pretty.

I guess I'm being a bit snarky. But I'm also being more than a bit serious.

dangero• 13 years ago

The goal of a contractor is to extract as much money as possible from their client, not necissarily to do efficient quality work. In some cases it makes sense for a contractor to do quality efficient work, meaning that is the best path to profits, and sometimes it is not.

shitgoose• 13 years ago

If contractor is IBM or Kumaran, then yes, their only goal is to extract profit. They are not concerned about product they make it look like they are building. If contractor is Joe Blow Perl coder, then most likely his goal will be to deliver a good system he can take pride in.

jsumrall• 13 years ago

I read that CGI Federal developed the site, but did they get the contract to maintain it? I can imagine that, compared to a non-gov't contract, they have to go through a lot of red tape to fix the issues and build up capacity.

Maxious• 13 years ago

The contract has an estimated total contract value of US$93.7 million over a two-year base with three, one-year option periods. The US$55.7 million base contract has been awarded.

http://www.cgi.com/en/CGI-selected-build-US-wide-competitive...

nudetayne• 13 years ago

The WA state one is full of XSS and SQL injection vulnerabilities. It's quite beautiful.

dangrossman• 13 years ago

They're making progress at least. When the site first opened I couldn't get in, but I had no trouble today. It works just like eHealthInsurance.com -- the site I used to buy my own insurance 3 years ago.

http://i.imgur.com/GwXTkdc.png

tunesmith• 13 years ago

Sort of short on details or perspective.

I do remember reading articles six months ago or so that the federal government was disappointed that they would be managing the exchanges for so many of the states. They were originally hoping that they'd only be managing the exchanges for a handful of states, rather than the 36 they're having to shoulder the load for. I can see how that level of political uncertainty adds an extra challenge for estimating load.

marknutter• 13 years ago

Ah, and here I thought $600,000,000 was too much for a website.

TazeTSchnitzel• 13 years ago

I think it's mostly red states that aren't running their own ones.

hga6• 13 years ago

Idaho is the only solidly Red state that's doing its own.

A number of Blue or Purple states are also punting to the federal exchange, or doing a "partnership" where the federal exchange is doing the heavy lifting:

New Hampshire, New Mexico for the moment for individuals, Utah permanently for individuals, Arkansas, Iowa, Michigan, Illinois.

In many of these cases of partnership, whatever the governor of whatever party wanted, the legislature didn't buy off. New York is the only case I know of where the governor told the legislature to jump in a lake and set up one for the state through an executive order using the temporary money the Feds are donating early on.

And there's Missouri, who's people outlawed a state run exchange, and New Jersey's governor vetoed 2 bills to set up one.

shavenwarthog2• 13 years ago

On NPR the other day they said solidly red Kentucky is running their own site, and is doing really well. It sounded like they were more specific in what they wanted, and perhaps had reduced functionality thus complexity. Either way, kudos!

I would have thought California (note: I live in CA) would have had a good healthcare site, considering all the techies, but I guess in this case that didn't work out.

map of state/fed/joint-run sites: http://www.dailykos.com/story/2013/10/03/1243855/-Kentucky-s...

jzelinskie• 13 years ago

I had someone visit one of my classes at PSU and describe the backend system as it is being implemented in Pennsylvania. The system that was described acts as one big middle man and interops multiple legacy systems mostly written in COBOL. I was told they have the time constraint of 7 seconds to process approvals, which to me seems like a lot, but I'm sure the older systems are probably the real constraint. If these legacy systems supported modern APIs and data transfer formats, this could probably all fit cleanly into any web framework and you wouldn't need a backend dedicated to doing tricky data cleansing. I hope the plan is to use this system as a shim while they improve the legacy systems.

XorNot• 13 years ago

Those systems are legacy systems for a reason. They're never being upgraded.

Also, before anyone gets all "hurr durr government!" on this - this is the exact same situation at any major bank or company founded before about 1990.

bsullivan01• 13 years ago

Good point, before judging the price or quality we have to know what the Government asked them to do and the legacy systems they had to deal with.

triton• 13 years ago

Spot on this is normal. I work in the financial services sector and stuff still arrives on dialup and ftp from mainframes regularly in all sorts of odd formats. SOAP is what is in the executive bathrooms and JSON is the cleaner's name and that is it :)

bryogenic• 13 years ago

Here is a great Planet Money podcast about just that:

Why does it take so long to send money from one bank account to another?

http://www.npr.org/blogs/money/2013/10/04/229224964/episode-...

gigq• 13 years ago

From what I can tell it's not just legacy systems that are the problem. I just tried the system for the first time today and there are multiple form issues (errors like "Error: id" with no further explanation) and broken links (a link to https://www.healthcare.gov/help/how-do-i-choose-marketplace-... for more info results in a 404).

curlyquote2• 13 years ago

When will we finally hold contractors that build shit products for the public accountable for wasting a huge amount of tax dollars?

DanielBMarkham3• 13 years ago

They are being held accountable. The press prints a story, you get all angry, hopefully you vote somebody into office, then they create new rules, and in ten or twenty years we can start making generalizations about how well those rules are working (or not).

How else would you do it? Public executions of any public-funded program that looks bad? We have an accountability system -- it just might not be designed to optimize the things we would like for it to optimize.

curlyquote2• 13 years ago

Why not have the federal government have an actual organization of software developers (with possibly a standards committee) that do it themselves instead of contracting it out to private companies?

Something like a NASA for non-space software development?

Daniel_Newby• 13 years ago

When agency heads stop being appointed as favors by beauty pagent winners.

Encosia• 13 years ago

Officials say that too many people tried to log on at once; within three days, 8.6m had visited healthcare.gov.

That is an order of magnitude less traffic than I expected. 33 visits per second isn't child's play, but my WordPress blog hosted on a 1GB Linode VPS has easily handled bursts of that kind of traffic in the past.

That's totally an apples and oranges comparison, of course, but it's more and more difficult to understand how this project has failed so miserably. I half bought the excuse that it was one of the highest traffic site launches in history, but apparently that was more speculation than truth.

jasonlotito• 13 years ago

Except your making the classic mistake of assuming that it's 33 visits per second because you are looking at the 8.6m visitors happening evenly over the course of 3 days. You're also making the mistake of confusing the number of visitors with the number of hits on the site. 20 requests to the server for information is still one visit. And yes, 33 hits per second on a static site, cached at various points is far, far easier to develop.

but it's more and more difficult to understand how this project has failed so miserably.

So, imagine someone estimated 8.6m visitors over three days, and decided to load test the site for 33 visits per second... =)

Then it becomes really easy to see why it could fall over.

Disclaimer: I've been doing load testing the last few weeks, and have been beating up on other peoples server side applications. It's a lot of fun when it's not your code getting hammered. =)

jacques_chester2• 13 years ago

Agreed. Capacity planning is not about the average case; it's about the 99th percentile case and whether you can afford it.

jonknee• 13 years ago

Your WordPress blog doesn't interact with hundreds of APIs with varying quality and reliability. Healthcare.gov does and while many departments it interacts with are all furlough.

jpollock• 13 years ago

In phone networks, the difference between peak and average load is typically at least one order of magnitude higher. Every day there is a difference of 4x between min and max, with the peak being sustained for multiple hours.

That's why telco systems are sized as "BHCA" (busy hour call attempts) and "CAPS" (call attempts per second), which are peak rates, not average rates. Well, they are both averages, but not over the whole day or year!

Then, as others have pointed out, there's a difference between a static site and a non-static one. This is mostly determined by I/O. A single disk can do between 100-150 IO operations per second (IOPS). Reading a row from a database will take several IOPS. In the systems I work with, we have to keep the entire database in memory to meet our performance numbers, and even then we are still O(database reads).

Doing a SOAP request to a legacy platform that was designed to only support batch requests? Lots of opportunity for slowness. Particularly when you start to run into TCP timers controlling port reuse.

Doesn't excuse the 404's though. That stuff is easy to test automatically.

DanielBMarkham3• 13 years ago

I work with a lot of technology teams, in a lot of different situations. I'm also a programmer.

I think the tendency in these Monday-morning quarterback sessions is to blame the people or the tech. If they only used X,Y, and Z! people will say, or Why can't they find qualified people who can do this, it's not that hard!

When that fails, we start blaming groups of people. Those damn contractors. Always screwing us over! or Dang government bureaucrats! Lazy bums!

These patterns of analysis are almost always mistaken. You don't drop a billion bucks and get screwed because you hired dumb people, because you're lazy, or because they decided to use WhizBang 1.0 instead of 2.0 You get screwed because you have smart, motivated people working inside of a system of delivery which is hosed. Incentives are wrong, feedback loops are loose (or non-existent), and so on.

So while this is a nice political topic, and sure to get everyone's emotions roiling, be aware that we see this same stuff all of the time. (I would make the case that government is far worse, for exactly the reasons I mentioned above, but I won't because it's only going to push many of your buttons).

It's not the simple stuff that kills projects. It's certainly not tech. It's particular configurations of people that choose bad tech, or that refuse to change bad tech once it's identified. Most all of technology development problems boil down to social problems, not the other stuff. If you've got a good social structure with ten qualified people, you can rule the world. If you have a bad one with a thousand geniuses, you'll be lucky to get your shoes tied in the morning.

jacques_chester2• 13 years ago

Like Weinberg said in Secrets of Consulting:

   No matter how it looks at first, it's always a people problem.

pingswept2• 13 years ago

The evidence in this article is:

  1. Jason Lahoz couldn't sign up.
  2. @Ginnyproffitt was frustrated.

I tried signing up a couple of days ago. The site loaded quickly and worked fine. The only quirk I noticed was that I received two registration confirmation emails instead of one.

We should not be so quick to jump on the government without evidence that there is a problem, especially when you can just go look at the site yourself and see if it works.

pjmorris• 13 years ago

I tried signing up on Thursday. I filled out the account information, submitted, the site spun for a few seconds and replied 'Important: Your account couldnt be created at this time. The system is unavailable.' (Note no apostrophe in couldn't, verbatim from site.) Twenty minutes later I received an email 'Your Marketplace account has been created.', with a link to verify my email address. I read the email an hour later, clicked the link, and was informed that the link had expired and that I would need to re-enter my account information.

I realize that the plural of anecdote is not data, but there's got to be some multiplier of plural that is data.

pingswept2• 13 years ago

Agreed, but I think it's a pretty big multiplier.

pbreit• 13 years ago

I think a lot of this "gov is incompetent" is overblown. Complex systems like this rarely work exactly right on launch (and rarely launch on time). People have another 2.5 months to register before any coverage actually kicks in. This whole episode was just a layup for right wingers to torch government.

simonholroyd• 13 years ago

Not sure if this is what you meant, but I don't think it's fair to call The Economist "right wingers" in the American sense, and probably not in the general sense either.

http://www.economist.com/blogs/economist-explains/2013/09/ec...

And on universal health care they're generally on the left.

http://www.economist.com/node/15720396

dm2• 13 years ago

I'm not a right-winger but am disappointed that the healthcare.gov site isn't working at all.

It's 2013 and there are tons of technologies that allow this type of system to be set up fast and scalable. The fact that I can't even login (with a registration I made a month ago) and that it doesn't even give me an error message, and that support (phone and chat) just say "it's experiencing problems right now" is inexcusable when you consider that they had half a billion dollars to work with.

Is a secure registration and login system that at least gets people started on the site too much to ask?

hga6• 13 years ago

"Is a secure registration and login system that at least gets people started on the site too much to ask?"

Evidently so, even given 3 and a half years after passage of the bill that's this Administration's signature accomplishment to date. While it's hardly unknown in other sectors, I've read one of the problems was HHS/Sebelius making important decisions way late in the process, and I think I recall the usual required changes as well.

But unless the signup process hooks into outside live databases for verification or what have you there's as you note no excuse, especially since disallowing window shopping is one of the biggest modern UI hindrances, something pointed out to the people doing this (some information gathering is needed to offer the right plans, but others and I have compared it to the medicare.gov Part D sub-site, which collects that but doesn't require giving all your information prior to you officially selecting a plan, after which the site informs the chosen provider).

Anyway, I am a "right-winger" who's upfront about wanting to ... eliminate many parts of government, and just from memory I can point out several massive and notorious IT procurement failures that would kill or cripple most private concerns. E.g. last time I checked (which is some years ago):

The FAA is still using their 1940s (sic) techniques, and every attempt to really modernize the initial system architecture that automated it has failed. (They famously hired a couple of retired IBM engineers to make microcode patches for some of their old machines for Y2K). This one has really serious economic consequences.

Every comprehensive attempt to upgrade the IRS following their very first system has failed. They're still using code for the 1959 IBM 1401 (https://en.wikipedia.org/wiki/IBM_1401) in their front end processes.

The FBI is legendary for their failures to ditch paper. I remember at least one failed attempt pre-9/11, there's one acknowledged failure after it, and I don't know if they're really using the Sentinel system as they claim as of 2012, after 7 (!) years of troubled development.

These are the really big ones that come to mind. Which of course is not to say the government is always incompetent. The Census Bureau was famously cutting edge, using punched cards in 1890 (sic), buying a UNIVAC I in 1951 (sic), and when I talked to some of their people in the mid-90s I got a sense of serious competence.

As noted in previous discussions, the same contractor operations CMS.gov and Medicare.gov, and I can attest that the latter site, while a bit clunky, works just fine. CMS's backends seem to work as well.

Off the top of my head, the CDC and FDA do well in getting out timely info. The DoJ's statistics sections are good, the NIH's sites are stellar. Heck, the BATF has a nice web page where you can type in some of the numbers of an FFL (firearms dealer, gunsmith, etc.) and confirm it is legitimate and still valid. Etc.

smartician• 13 years ago

I was surprised that they had anything ready on October 1st. I expected at least a six month delay. Not because of hurr durr government can't do anything right, but that's the nature of large scale IT projects. They're extremely hard to manage, especially when there are a lot of third party systems involved.

shadesandcolour• 13 years ago

I wonder if this would still be the case if fewer states had punted the work to the federal government. The fact is that the federal government is running 36 exchanges was never the plan.

The Obamacare software mess

Related Stories