Health Exchange Tech Problems Point To A Thornier Issue

http://www.npr.org/blogs/alltechconsidered/2013/10/08/230424841/health-exchange-tech-problems-point-to-a-thornier-issue

by jessaustin • 13 years ago

68 83 13 years ago

npr.org

niels_olson2• 13 years ago

I work at a DoD hospital. Trying to do some research. I have been told

1) Python is not allowed

2) Open source software is insecure

3 Perl is allowed

4) there is a list of approved software.

5) I can't see it

6) Enthought is on at least one version of the list

7) I can verify there are several python projects on forge.mil

jessaustin2• 13 years ago

Are these rules in force on any host connected to the network? Do they have different networks with differing rules? How are the rules enforced? (That is, I can imagine switches and firewalls enforcing network rules, but "don't install python"?)

kreitje• 13 years ago

They do have different networks with different rules. When I was on the Michigan DOD network, we had the Microsoft Document Image Viewer. When I was deployed, the DOD network there did not have the Microsoft Document Image Viewer on the "approved" list and therefore could not be installed. It was such a pain working with TIFFs without it.

staunch2• 13 years ago

#3 solves all your problems then :-)

niels_olson2• 13 years ago

Are you serious? Is there a perl implementation of python? Did I mention I am having to formally present to the executive steering committee to get permission to run perl?

frozenport4• 13 years ago

1. Maybe they know of vulnerabilities that you don't. Remember the bug in Debian where cryptography was being seeded by the process id?

2. They might have put a lot of time into ensuring that one particular product was 100% secure and trust only that one. This is often the case for hardware, where vetting for security compliance can be difficult.

jaggederest• 13 years ago

Ha, sounds a lot like most big companies. Why get and train competent staff to build a software development process when you can pay 10x as much to a vendor for software that doesn't really work.

maaku• 13 years ago

Because in many cases hiring competent staff (aka civil servants) requires an act of congress.

vinhboy• 13 years ago

I have never worked on a high traffic site before, so I am just mostly talking out of my ass. But when I looked at the html source for the coveredca website, it had a lot of of basic problems: there were 3 DOCTYPES on one page, developer comments were still visible (you can actually see the names of all the developers who worked on the project), Java error pages were visible, they disabled the back button completely by enforcing javascript refreshing, they did not minify any of their assets (which would also remove the developer comments) AND they were actually loading many copies of the same assets multiple times because, like I said earlier, their site had 3-DOCTYPES in one html page.

As someone who has never worked on a huge site, I don't want to naively say that it could have been done better. But I feel like those are basic issues that could have easily saved them a lot of bandwidth.

Also, it seems to me that if they had so many issues on the front-end, it's a hint that maybe they also had many similar issues in the backend as well.

ericd4• 13 years ago

Yeah, the CA version (https://www.coveredca.com/) had so many visual glitches and various problems (404s and 500s), I threw up in my mouth a little when I found out that Accenture charged the state over 180M to build it and another 174M to operate it for four years. I don't know how that's possible.

scarmig2• 13 years ago

Are you serious? Christ.

akjetma2• 13 years ago

Do you have a source for those numbers? I need to vomit, but I can't without a source.

edit: Nevermind: http://newsroom.accenture.com/news/accenture-chosen-to-imple...

Vomit everywhere.

frozenport4• 13 years ago

I had a job interview with Accenture that had 2 parts a dinner and a technical chat.

1. At the dinner their representative spoke about how life at Accenture was a breeze and how as a undergraduate he spent most of this time drinking. He drank on the weekdays but his favorite time to drink was on Sundays. He said he used to do some SQL but now he had moved onto the business side of things. One of the candidates told us how his hobby of skydiving let him see the big picture and another told us that his desi background would enabled to come up culture specific features for software services. I probed this question further, and he said that in India showing your muscles was considered hubris, while in America it made you look strong. On his facebook page he wouldn't show his bare chest... Nobody at the table `looked` like they could do any programming work.

2. During the technical interview I was asked what `clubs` I was in. I told them I wasn't in many clubs and instead spent my time doing research work (which by that time had yield publications) and administrating websites. I told them my hobby was programming. I wanted to tell them how I dealt with EC2 sight outages, worked with clients and their clients to write WordPress plugins. I wanted to talk about my experience engineering our production and outsourcing workflow. I wanted to summarize the techniques I learned overseeing our production staff. They didn't comprehend and asked he what `clubs` I was in.

I was scheduled to have 3 interviews, I walked out after the first. Don't work for Accenture, they are a joke because their employees are a joke, which is much in agreement with this article.

They also have 275,000 employees.

nostromo2• 13 years ago

The WA version (https://wahealthplanfinder.org/) doesn't redirect you to www, so if you click that link you get a big red SSL warning.

This caused a number of people to worry that this was one of those scam exchanges people had been warned about.

Total cost: $54.8 million

smsm424• 13 years ago

Wow! Given how awful it was on Oct 1 and that it was broken under the load of less than 10 req/s [1], it is outrageous how much it costed. It is probably standard for the government contracts everywhere, but when it is where I know how badly it is done and how much it costs, it's just doubly disgusting.

[1] http://www.latimes.com/business/money/la-fi-mo-california-he...

BTW it looks like Accenture is not the only one. This page: https://www.coveredca.com/shopandcompare/#calculator

has this in the source: Produced for Covered California and the State of California by Kiefer Consulting, Inc. www.kieferconsulting.com

(and then for some reason Apache license - copy-paste from somewhere?)

So 180M might be just one part of it, the whole sum may be much bigger.

JangoSteve• 13 years ago

Interestingly enough, healthcare.gov itself is open source and uses Jekyll for the static site. It seems to be decently done. But the marketplace itself I guess is a whole different story.

https://github.com/CMSgov/healthcare.gov

swamp40• 13 years ago

This article on the Healthcare.gov authors (Development Seed) from back in June makes me so mad I could spit.

http://www.theatlantic.com/technology/archive/2013/06/health...

Especially these paragraphs:

"When we worked with the World Bank, they chose a plan from Rackspace for 16 servers," said Gundersen. "That added tens of thousands of dollars, with a huge hosting bill every month."

HHS had similar strategic plans for the new site, at least at first. "They were planning 32 servers, between staging, production and disaster recovery, with application servers for different environments," said Cole. "You're just talking about content. There just needs to be one server. We're going to have two, with one for backup. That's a deduction of 30 servers."

ars2• 13 years ago

"that contractors built a system that could handle 50,000 people using the site at once"

There are over 100 million families in the US and they build for 50,000???

Is that number at least 50,000 pages / second? (Which would be reasonable.)

smartician• 13 years ago

I think 50,000 concurrent users is actually a fairly reasonable number, given that not all US households are in the market for health insurance on the exchange, and that not everyone will be using the site at the same time.

ars2• 13 years ago

Yes, 50,000 pages per second is quite reasonable. But I suspect that's not what the number means here.

warfangle• 13 years ago

Concurrent, perhaps. But a thing like this? You're going to see spikes of many, many times more than that opening [insert time period]. See also: the Slashdot Effect, the Digg Effect, et cetera.

dangrossman3• 13 years ago

Even if they could theoretically make a site that can handle every person in the country using it at exactly the same time, it'd still be bottlenecked by the much older systems it relies on at the IRS, SS, VA, DOD, DHS, etc. As the article alluded to, the Healthcare.gov site checks your family size, AGI, social security benefit status, citizenship status, incarceration status, veteran status and enrollment in other government health programs during the signup process to determine what you're eligible for.

cantankerous• 13 years ago

This. The article sorta pines in the end for smaller "more agile devs", but I'm not sure a small shop could just come in and bust this out. There is SO MANY legacy systems that need to interface with this beast. I wouldn't put it past the designers to have been rate-limited at 50,000 whatevers by one of the vendors of the other systems that they have to talk to. Computers from the IRS have been hanging in there sometimes as far back as 40 years ago. I'm sure systems at many other of the lettered agencies are hobbling along just as well.

I totally agree that the government needs to become more lucrative to smaller more agile groups. I think it could ultimately get better products at lower prices if it opens up bidding. That said, the government has massive needs and massive legacy systems that many small shops may have a hard time reconciling. There's always ways to theorize about how the government could standardize and modernize its disparate stuff, but that requires strong leadership, a good direction, and money. None of these things appear to be in abundance right now. It's really unfortunate.

justinsb4• 13 years ago

I posted this in another comment, but do you have any ideas as to why that information has to be verified in real time?

smsm424• 13 years ago

I tried to check prices on CA site on Oct 1. I never tried to sign up, so the system actually didn't have to go to IRS, DOD or DHS - I just wanted to see the plans offered. In fact, it had to do no more than look up a few tables and do a couple of basic calculations. Still, the site was as broken as one could be - about 75% of requests just timed out. The rest were presented in broken design and half-loaded pages. Navigation was completely broken. I spent couple of hours and many tries and retries until I could actually see how much an insurance plan would cost me. And LA times reports[1] the site has had 645K hits for the whole day. DOD, SS and VA is not the problem here - the site just had a crappy implementation.

[1] http://www.latimes.com/business/money/la-fi-mo-california-he...

InclinedPlane2• 13 years ago

Compare this with the medicaid "part D" government website launched in 2006. It was designed around an anticipated load of 20,000 simultaneous users so they scaled it to be able to serve 150k users. Anyone who builds websites in 2013 without understanding the spiky nature of traffic or the necessity to build in capacity overhead is incompetent, end of story.

The amount of money spent on building healthcare.gov (many millions) just makes that incompetence even more infuriating.

dangrossman3• 13 years ago

Being handed impossible constraints (serve 250k simultaneous users, verify them all with the IRS, but the IRS service you have to talk to can only handle 1000 simultaneous requests) isn't incompetence on the developer's part, and we don't know whether something like that isn't what's happening. The team that built this site (which has had no unintentional downtime, latency or server errors -- whatever code they wrote hasn't fallen over, it's just stranded most people on a "please wait" page when they try to register) has no power over systems at all the other agencies they have to work with.

PaulHoule• 13 years ago

More generally the problem is a lack of standards in technology management.

Malpractice is endemic in the IT space and nobody, either technologists or managers, are held accountable.

For instance, the issues @vinhboy talks about are "bad smells" that reek of unprofessionalism -- like the primary keys that aren't unique, the predictable race conditions, and all of the problems that recur again and again.

drewpc• 13 years ago

The article mentioned only one side of the problem: the acquisition process. The government must follow the Federal Acquisition Regulations (FAR - http://www.acquisition.gov/far/) when contracting purchases of equipment and/or services. The FAR is not agile and is so strict that you CANNOT already know what you want, you must make the contract as generic as possible. Especially on large contracts (over $5M). For example, you can't say that you need laptops with Intel Core i5 processors because that would unfairly exclude other processor vendors from competing.

The other side of the problem, arguably the bigger side, is that most of the people in the acquisition process do not know how to write a contract that is flexible and really gets the government what it needs. For such a large project as the health care website, the people at Health and Human Services most certainly did not know how to articulate what they wanted. They could've hired a company to help them determine their requirements and write a proposal, but that's a significant (and long) undertaking itself.

In short, government systems will never be user friendly, scalable, or secure because we don't know how to acquire such a system.

MichaelGG• 13 years ago

I thought you can get around excluding vendors just by specifying generic requirements that only one vendor happens to fulfill. Like "must have inter-processor bandwidth of at least Xgbps" or whatever generic-yet-specific thing you can find.

interstitial• 13 years ago

I actually know someone who has worked for the VA in medical coding contracts. He still has Cobol programmers on his speed dial because some of the live code is still written in it.

trimbo• 13 years ago

They might have COBOL around, lots of banks and government agencies do. But the VA's critical system, VistA, is written in MUMPS[1].

It's NoSQL!

[1] http://en.wikipedia.org/wiki/MUMPS

justinsb4• 13 years ago

Apparently integrating with "legacy" systems was a big bottleneck (e.g. to verify your SSN). I don't understand why this verification had to be done "live" though; the obvious alternative would be to collect all the data, verify it asynchronously/overnight, and then flag up those people that had issues. You enter your birthday, SSN, etc and then the 1% of people that got their details wrong get an email the next day saying "we weren't able to verify your details, please log in and fix it".

jes• 13 years ago

To what degree is the sheer complexity of the underlying government legislation responsible for cost, delays and complexity in the IT systems government pays to have built?

No wonder we are going broke. It's nothing but special cases, all the way down.

siliconc0w• 13 years ago

Just saying there were like 20 articles about the all star obama campaign tech team that built a super scalable cloud infrastructure and ab-tested, email spammed, and narwhal'ed their way to victory. Couldn't get any of them for the actual governing?

integraton• 13 years ago

FTA:

"One might look at this and go: Why can't we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?" Johnson says. "The problem is that the way that federal contracting works is so burdensome that the only people who get contracts like this are experts at lobbying and experts at regulations that require you to get these sorts of contracts. And they're not experts at doing the job of building these websites."

TWAndrews• 13 years ago

No, quite literally not. The campaign team is a private organization, not subject to rules for hiring federal employees or spending tax money.

This is precisely the sort of shit that make people believe that the government is incompetent. It really is for certain undertakings that require some degree of operational flexibility.

smsm424• 13 years ago

Wait, isn't the government the one who makes the rules? So if they made the rules that makes them incapable of doing things that they are supposed to do, while the people, technologies and ability to do the same thing is readily available and the same thing can be and is done by others - isn't that a very definition of incompetence? Not doing what is perfectly doable because of self-inflicted inhibitions?

It doesn't "make people believe", it is incompetent. And, unfortunately, the very capable and competent people are working hard at making it worse by electing exactly the people who would perpetuate and expand the problem.

scarmig2• 13 years ago

Two points:

1) Our government has a huge set of rules about procurement. Who's allowed to get their bid approved, the right kinds of projects, who's allowed to manage them, who's allowed to set project requirements (different than the former), who's allowed to pay them, etc. In theory this is all really smart and awesome: you don't get someone giving their brother-in-law a swell multi-million dollar contract, physically impaired people can access your site, etc.*

2) Governments and campaigns face different incentives. Campaigns don't have many cross cutting concerns: the goal is to get a candidate elected. Sure, some high powered consultants are simultaneously angling for a new job after, but IT and most staff are relatively immune from that. Your job is to get something done, and you want to get somethng done. Government, though, has no such incentives: you can do shitty work and not face consequences, and the mission is inherently less aspirational and more shoveling shit. Much of the time government bureaucracies also double as a social welfare program, or a private welfare program if a well-connected corporation has a well-placed surrogate legislator or bureaucrat.

*Reality, needless to say, can turn out a bit different.

Spooky232• 13 years ago

Blaming IT is an easy scapegoat. The notion that the Feds and various states should be operating marketplaces in the first place is absurd on its face. California and New York aren't subject to federal procurement guidelines, and their systems are basically non functional as well.

Government is successful with complex public facing IT in areas where the processes are well understood. The DMV, tax collection, and unemployment are good examples in most states. Medicare billing is a success story in the federal space. Most other things are not very successful unless they are limited in scope.

The whole system is a screwed up farce. "Obamacare" is a flawed system to begin with, to the point that nobody really understood what was needed for months. So the Feds and states were forced to roll with some highly generic requirements and specifics were made up as they went along. That scenario is a dream for Accenture (Cali) or CSC (ny) to churn hundreds of millions of dollars.

ajross• 13 years ago

Can you elaborate on how selling an insurance policy is not "well understood" or how it's inherently more complicated than administering the tax code (!) or running medicare? I don't follow that at all. Insurance has been sold successfully for well over a century. Hell, governments have been running and regulating markets for all sorts of things (many of them much more complicated than a handful of insurance plan choices!) for the whole of modern history.

That doesn't mean that this site isn't a mess, but I don't see where the "screwed up farce" bit comes from. I rather suspect it's driven more by your choice of political tribe than any real intuition about what the government is capable of.

threeseed• 13 years ago

Hilarious considering that Medicare had far more serious problems throughout its inception and you consider the billing component to be successful.

Like all website launch problems in the months and years from now people are going to forget they ever happened. And instead will find having an independent marketplace to be incredibly useful. The states that have exchanges have already seen insurance premiums drop significantly.

itbeho• 13 years ago

I keep reading anecdotal stories on the Internet about premiums dropping significantly but I, some in my family and a friend have recently experienced the opposite.

smsm424• 13 years ago

I just compared premiums for essentially the same plans (3-person family, no subsidies, California) on http://www.ehealthinsurance.com/ (old insurance, 2013 plans) and on http://coveredca.com/ (new ACA plans). The former are roughly 2x cheaper that the latter for comparable plans (cheapest one ~350 vs ~750). I also read other reports [1] of costs increasing. Could I see which reports of premiums dropping significantly - e.g. in CA - you are referring to?

[1] http://www.mercurynews.com/nation-world/ci_24248486/obamacar...

Spooky232• 13 years ago

Huh? Medicare billing is a successful program that works efficiently and well. Have their been issues with the Medicare program over the last 50 years? Yes. Most of the serious ones are policy problems.

downandout• 13 years ago

The government IT procurement process puts people that don't know what they are doing in charge of multi-million dollar budgets. The primary determinant of whether or not a company gets hired is its ability to navigate the procurement process - not their ability to actually do the job. In fact, many companies specialize in winning government contracts and then subcontract the work out to others.

It's a sad, broken system that will almost certainly never change - in part because we don't vote for the people in charge of these things. It really doesn't matter who is in the White House or in control of Congress. The fiefdoms within the government that control these issues have maintained a culture of mediocrity for decades, and it will likely remain that way.

dnautics• 13 years ago

how do you propose reconfiguring the procurement process? Remember, it's got to be fair. You have to figure out how to charge a fair price. Everything has to be documented, so that it can be audited by the bureaucrat in a few months and given a second rubber stamp. Also we have to score the service providers, don't forget to ask them if they are minority-, veteran-, or disabled- owned, because each of those gives them +20 points.

lisper• 13 years ago

it's got to be fair

Why? It's not fair now.

dragonwriter• 13 years ago

how do you propose reconfiguring the procurement process?

For most of the problem areas, the answer isn't to mend it, but to end it. Stop contracting out core functions.

nhebb• 13 years ago

I believe the exchanges are run by the states. Since there is overlap - a federal law being implemented at the state level - the procurement process may have had to follow federal guidelines, but I wouldn't be surprised if this were a case of 50 states with 50 different procurement processes.

nutate• 13 years ago

Nearly all of the GOP states refused to play ball and are just defaulting to the federally made site.

I have a feeling one state's site will be the best and will perhaps be mirrored to other states within the coming decade. This is clearly not an overnight problem, but in the end, once each user gets their insurance they should never need to visit again unless they want to switch.

balloot• 13 years ago

It should be noted that healthcare.gov might be the highest traffic website launch in the history of the internet. With millions of visitors on day one, I certainly can't think of one that would even compare.

They are dealing with scaling issues that are almost completely unprecedented. Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up.

Given this, I can't help but think that 90% of the scorn heaped upon healthcare.gov is politically motivated. When Twitter spent YEARS going down on a regular basis, or Simcity just completely shut down on launch day (with a fraction of the users of Obamacare), nobody demanded the companies that running them be shut down. Shit happens when you build a website that gets incredible amounts of traffic. It will work out just fine.

thwarted• 13 years ago

Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up.

And how much those months or years of experience were leveraged in figuring out the touchy parts of the infrastructure? This shouldn't have been the first rodeo for anyone involved. There are numerous blog posts, books, videos, etc with ideas, what works, and what doesn't, for running websites of all sizes. Many of them are posted and debated on this very site.

akjetma2• 13 years ago

I don't think what happened with SimCity supports your argument. People were outraged about it for weeks.

pandaman• 13 years ago

Simcity is exchanging orders of magnitude more data between each client and the server with much stricter latency requirements. Same applies to Twitter (though the data density is less than in a game there are more connections and the volume of data is higher). I don't see how a website that serves static (in the sense they do not change over time, like twitter feeds or game sessions) pages is comparable to either.

wdr1• 13 years ago

It should be noted that healthcare.gov might be the highest traffic website launch in the history of the internet. With millions of visitors on day one, I certainly can't think of one that would even compare.

I can think of two that I've worked on personally:

- Yahoo!'s tribute site for the first anniversary of 9/11 - Ticketing for the 2008 Beijing Olympics

Yes, in both cases, the companies had experience & infrastructure for building high-scale website, but that should have been a requirement for the entities building healthcare.gov as well.

In both cases I've cited there were a lot of significant changes to the technical stack (e.g., it was first time we used PHP for a large scale project at Yahoo!).

TL;DR: This isn't really excusable.

joshuaheard• 13 years ago

Start at 1:00

http://www.nbc.com/saturday-night-live/video/weekend-update-...

robomartin• 13 years ago

The real scary part is that we entrust these people with our most private information as well as decision making matrices that might very well determine what, when and if you receive the appropriate care.

To me these incidents, the broken procurement systems, the mediocrity (or worst) that permeates government and many other factors only serve to reinforce the idea that we need massively less government reach into our lives, not more. Frankly, the combination of healthcare and the IRS scares the living daylights out of me.

rurounijones• 13 years ago

Good luck changing the procurement process. The UK Gov has been trying to get more SMBs involved in UK Gov procurement[1] in hardware and software is failing pretty miserably at the moment.

[1] http://www.informationweek.com/smb/services/uk-pushes-govt-i...

[2] Nuts, cannot find the link. Was an article on theregister.co.uk for what it is worth

nostromo2• 13 years ago

The jaw-dropper here is how much this website cost: $634,320,919.

http://www.digitaltrends.com/opinion/obamacare-healthcare-go...

With all the state exchanges included, the websites for this program must be over a billion dollars spent. Amazing.

ericd4• 13 years ago

Yeah, CA alone brings it almost to $1B (another $330M). It's pretty unbelievable how wasteful the procurement process apparently is.

perfunctory• 13 years ago

In order for the exchange marketplace software to check on which coverage and subsidies you're eligible for, it has to ping requests to the ... Homeland Security data

Really?

tantalor• 13 years ago

This is the same company that build MapBox?

quannah• 13 years ago

No. But the company that built the working sections of healthcare.gov - that is, everything except /marketplace* - and open-sourced it (https://github.com/CMSgov/healthcare.gov) is Development Seed. They are indeed the makers of MapBox.

philipashlock• 13 years ago

For those wanting deeper technical analysis around the problems and building off of some of the arguments Clay made about procurement, check out https://news.ycombinator.com/item?id=6518739

pasbesoin• 13 years ago

Make it work. Then make it pretty.

You missed a step.

Health Exchange Tech Problems Point To A Thornier Issue

Related Stories