Could A Tech Giant Build A Better Health Exchange? Maybe Not

http://www.npr.org/blogs/alltechconsidered/2013/12/02/248225524/could-a-tech-giant-build-a-better-health-exchange-maybe-not

by daigoba66 • 13 years ago

40 50 13 years ago

npr.org

saosebastiao• 13 years ago

40 million? Thats just for one core of the database! Oops, we forgot to tell you about that? Gee, I'm really sorry. Have I told you about our Spatial Extensions?

jsmeaton• 13 years ago

The Oracle licensing model drives me crazy.

andrewflnr2• 13 years ago

I think the lesson to take from this whole fiasco is not that anything the government does is bound to fail, but that the redundancy offered by multiple competing entities trying to do the same thing is valuable. Everyone here knows software projects have a high failure rate, especially those with complicated specifications. Trusting something as important as a national health care system to the success of one project seems insane to me.

dreamdu5t• 13 years ago

I don't buy it. This contractor got paid. I don't see how a business would pay a contractor for not delivering, and if they did deliver and the product was crap, they would directly suffer as a result.

The only reason the government has to change healthcare.gov is because it's an issue that republicans have put on TV to use against democrats. That's it. If it sucked, but wasn't in the news cycle (like everything else), then the government wouldn't do a damn thing about it.

Tloewald• 13 years ago

So, do you have any experiemce of consulting work (I mean the big firms like McKinsey and Accenture)? Most consulting work comprises being paid handsomely for failing to deliver. Sometimes they get paid to destroy companies. (I think McKinsey has run some countries into the ground.)

andrewflnr2• 13 years ago

I'm confused about whether/how you disagree with me. In case it wasn't clear, I'm generally in favor of private enterprise being in charge of things; I'm trying to rein in my "government is stoopid!" instincts a bit :).

Generally, it seems stupidity of an organization scales with the number of people therein, private or not. Even corporations let horrible products persist if they don't hear enough whining. Government just has some extra features that make it bad at projects like this.

yogo• 13 years ago

The way I see it is that that parties contracted to build these systems are not invested in its success whatsoever. Payment is guaranteed and you can keep on billing. It's not a trivial software system but it's a lot different when the livelihood of your company depends on it. It's a valuable lesson for the states and government but I'm skeptical as to whether anything would change.

sseveran• 13 years ago

In fact the economic incentive is often to never really be done. I saw this first hand at a couple of government agencies.

ibsathish• 13 years ago

Give it to a dying start-up. They'll certainly pull it off with cutting edge technology & tools with great eye for UX, which usually giants fail to do.

This will give them a runway till take-off.

tibbon• 13 years ago

I'm honestly not sure why a better health exchange couldn't be built by practically any team. I've kept hearing on NPR that one of the big problems is that they 'needed to make it talk to X number of other government/insurance service'. Have these people never heard of REST or JSON? The problems that the health exchange is facing seem to be largely solved problems and not ourside the realm of reasonable scalability. I'm good, but not amazing at development, and I think I could make Postgres do that is needed for this within a week or two.

acchow• 13 years ago

Have these people never heard of REST or JSON?

I think the difficulty is really in schema matching/mapping.

hackinthebochs• 13 years ago

Right. The systems healthcare.gov has to interface with are established and NOT subject to change. It is up to the new system to be interoperable with whats already there. And this is where this get hard: you have to be able to speak with a myriad of different systems, translate the data to and from the schema of each target system, and do it all securely and robustly. This is all much more complicated than most of the armchair critics realize.

eranation2• 13 years ago

They probably heard of it, but they are probably using SOAP webservices or Java EE MQs

jrmcauliffe2• 13 years ago

Or whatever came 10 years before that. FTP? Batch tape?

patio11• 13 years ago

Man I hate it when HN makes me feel like a grizzled veteran of the technology industry, since I'm barely 30, but I suppose having ever worked with a legacy system qualifies me to comment here:

Some challenges with "Just use REST/JSON and then Bob's your uncle:"

1) You will probably not have the source code of the system you are integrating with, which is likely a mainframe system written in COBOL.

2) The engineers who built that system are retired or dead.

3) The company which they worked for is no longer operating / was sold 15 years ago to IBM and then reverse merged into a different unit 7 years ago / lost all of its records in a fire / etc.

4) Your sole documentation is the dogeared paper copy describing file formats for their sole input method, CSV files, which was current as of its publication date, before you were born.

5) Your sole documentation incorporates by reference four other documents which, ahem, yeah.

6) You have a working test suite which accurately models the production system in all respects. It is the production system. Please do not run the equivalent of Accounts.all.delete when cleaning up from your test runs. (The system won't stop you from trying to do that. No, really.)

7) Praise God, there is actually a bridge between the mainframe system and the Internet, so at least you don't have to reverse engineer wire protocols. It was coded in 1996 by IBM's crack team of integration engineers. This will give you an excellent opportunity to brush up on your Java 3.4.1. It speaks REST/JSON, as long as as you spell REST/JSON "XML". Good news, though: your time spent learning the CSV file format won't be wasted, since the XML is just a straight mapping to it. Except for the four bugs in the mapping, which are dutifully recorded in a database in Hyderabad whose existence will be exposed to you 8 months into the project.

Welcome to legacy integration! We hope you enjoy your stay.

incision• 13 years ago

Anecdotally, what's described here would actually be unrealistically optimistic given the context of Government where I have entirely too much experience.

First, you can forget about having documentation of any kind or anything as recent as XML.

Also...

8) Nearly everyone you need to interface with on the customer side is apathetic at best. More likely, they are plain adversarial for fear that the work you're doing will eliminate their jobs. A single person can plop themselves in your critical path and refuse to cooperate - millions of dollars or public safety be dammed - there is nothing you can do other than work around them.

9) You will need to coordinate with half a dozen other subs under the contract prime. Some of those subs will be unassailable due to pre-existing relationships. Others will depend on talent with a 10 hour time difference. One will suddenly be in charge of "testing" your code in order to generate more billable hours. You will be in a constant battle to avoid being used and abused by the prime to make up for the deficiencies of other subs.

reuven• 13 years ago

This is totally right.

My first job out of college involved connecting to various pieces of hospital equipment. The equipment connected to our gateway (the project I was assigned to work on) via RS-232 cables, communicating using a standard protocol known as HL7.

Standard cables, standard protocol -- what could possibly go wrong? The answer: Everything. Everyone interpreted RS-232 in whatever they wanted. Everyone ignored the parts of HL7 they disliked, and added things that were "obviously" important. Getting decent documentation was nearly impossible.

The bottom line is that software development is relatively easy when you're in a vacuum. The moment that you need to integrate with someone else's system, things get complex and difficult. And when you have to deal with several vendors, each of which has implemented a superset of a subset of the standard, things get even crazier.

When I was editing the student newspaper in college, our editing/layout computer system stopped talking to our typesetting system. The vendors blamed each other for failing to do the right thing. We had technicians come in from both companies, and they basically yelled at each other, saying that he was not implementing the standards correctly.

So yes, the Obamacare Web site will go down in software history as a cautionary tale. But if they still haven't gotten the back end to communicate with the individual insurance vendors, then I fear that the debugging process is far from complete. And claiming that interoperability is a Small Matter of Programming is easy to say when you haven't actually needed to do it.

FireBeyond• 13 years ago

Precisely!

I once worked for a company that was all "cutting edge". We built Digital Video On-Demand systems for airlines, hotels. Python, REST, JSON, media streaming. Gorgeous things. We were invited to respond to a RFP for the transit system of the city I lived in, for platform displays. Bear in mind, these systems had been in place since the seventies, and consisted of two CRTs side by side, and you could watch the image drawing line by line down the two screens like a GIF downloading over a 300bps modem (it would take up to 15 seconds to draw a screen).

Alas, this would have been so much nicer.

We prototyped a HD flatscreen system, all sorts of niceties, news, temperature, more detailed train, trip, other information... then we had to integrate things.

We -eventually- dug up a line printed (literally on that blue and white tractor dot matrix printer paper) "spec" for the message format for the PDP system hooked into their network.

Prior to that we were pulling and reverse engineering a raw serial on the wire binary protocol with messages. Oh, did I mention there was a primitive packet system written for this that would preface messages with a destination, so they could be multiplexed - i.e. you could get "Train Destination (for train 1), first stop, second stop", then "fifth stop (for train 2 on a completely different display, to be ignored)" and then back to your messages.

When was this? 2007.

sanderjd• 13 years ago

This may be the most hubristic comment I've ever read. No, you could not "make Postgres do what is needed for this within a week or two" just because you've heard of REST and JSON.

etrevino• 13 years ago

I'm not sure how this is significantly different from the Federal government's approach. Both hired Big Data. There was little to no accountability during the process. In the end, neither product was delivered because the consequences of nondelivery weren't, well, consequential.

eranation2• 13 years ago

These things drive me crazy, 40 million $? I'm sure half of the people here in HN if not more, would do it twice as good, twice as fast, for a fraction of the price.

On the other hand, I've worked with tough clients that want a solution "for everything" and force the vendor into a rigid waterfall / BDUF but in the same time keep changing the requirements as they go.

But come on, for 41 million dollars you can develop it as static pages, that get submitted to mechanical turk and will have change to operate this for years... I don't get the math.

I'm in the wrong business, I should start applying for government contracts.

cheez• 13 years ago

When I was in school, I used to troll around government offices looking for easy contracts my friends and I could actually deliver on. What we discovered is that the amount the government pays is not for the work, but for the pain you endure in dealing with the government.

comicjk• 13 years ago

Ironically, most of this pain was designed to prevent government waste.

nhebb• 13 years ago

Two issues came up during congressional testimony that I haven't seen discussed are (1) the constantly evolving requirements and (2) the lack of security. I don't know whether the Oregon site has security issues, but the evolving requirements would likely affect both the federal and state sites. According to the testimony I heard, the 2400 page law resulted in tens of thousands of pages of regulations, which were still being codified up until September of this year and impacted site requirements. If this is true then it's little wonder the project failed.

BTW, as an Oregonian myself, I find it irritating that the site doesn't work but we are still being subjected to the truly awful Cover Oregon ads (see http://www.youtube.com/watch?v=Xv2UUcXCo9g).

olefoo3• 13 years ago

I don't know I thought the one ad with the musician from Bandon, was kind of catchy; but it did seem like a poor use of public funds.

The musician is Laura Gibson http://www.youtube.com/watch?v=Xv2UUcXCo9g and the song is very hopeful.

But, buying all that airtime and all those billboards... http://www.wweek.com/portland/article-21410-live_long_and_pr...

Definitely showed the risk of a marketing campaign too far in front of the product. Not to mention that a marketing campaign for something with so much pent up demand is kind of goofy.

olefoo3• 13 years ago

As an Oregonian, this whole episode irritates me to no end; the fact that I'm shoving an application into this meatgrinder is just insult piled onto injury.

State and local governments should learn from this and other episodes ( our failed DMV upgrade from a few years back for instance ); that an RFP and a procurement process that eats 5% of the budget before anything actually happens is bound to betray the public trust.

While I'd like to think that there was an easy solution; say appointing an 'implementation czar' and requiring open source solutions be the default. I don't fool myself as to the likelihood of that working any better.

Governments are human nature magnified and focused and as such they have all the human frailties; not to mention that everyone involved is pushing and pulling in a different direction.

That said; I do believe that all government IT projects should be open sourcing everything they do or have done for them ( not the data, the functionality ) so that it is open to public review and provides the possibility for public improvement.

At the very least if that $43 Million had been spent on something that left a github repository behind; Oregonians would have something to show for that money...

cgs• 13 years ago

Speaking of Oregon, I went to find the online application today and was disappointed (but not surprised) to find out it can only be submitted using IE. Tens of millions of dollars spent and I can't even use a modern browser.

olefoo3• 13 years ago

Yeah, that was pretty special. It's because it's a fillable PDF...

pcurve• 13 years ago

Anyone here working for Fortune 500 company's IT department knows that hiring companies like IBM global service, accenture, capgemini to do large IT projects is like rolling dice on craps table. Large private corporation's procurement process isn't any better. Anybody here ever had to deal with PEGA systems? :shudders:

rtx• 13 years ago

I left my job because of them.

ChuckMcM• 13 years ago

I swear I am astonished that they didn't just tell Zynga to build it, call it 'Healthville' or something, I am sure my Facebook feed would fill up with "Jimmy Can't Sign up for Colon insurance unless he can get 8 friends under 40 to sign up with him, click here to sign up for Jimmy!"

fletchowns• 13 years ago

I know you're just joking, but it's kind of spine tingling to think about what Zynga might do with personal health data.

Glyptodon• 13 years ago

I'm not real thrilled with their use of Oracle as an example... If you've used Oracle you've probably done it wrong is kind of a mantra for me, anyway.

sliverstorm• 13 years ago

That seems like a rather conceited thing to say.

malandrew2• 13 years ago

Not really. Have you ever worked on projects outsourced to Oracle, IBM or similar large company?

The project managers are total PMP dopes that know fuck all about software engineering. People get promoted to those positions because they are political. Not because they know what they are doing. Back in my big corp life from 5-6 years ago, I witnessed several multi-million dollar train wrecks. They were the epitome of sunk costs fallacies.

Oracle and IBM both have great developers working on products that are core technology that both those companies sell, but their software engineering consulting and project management is atrocious. There is a reason why companies like Thoughtworks exist. TW and similar firms come in to clean up the messes started by the Oracles and IBMs of the world.

chaz• 13 years ago

This was never a technology problem -- it was a business problem. Extremely complicated industry dealing with legacy infrastructure, data, and processes with sprawling, out of control requirements.

As context, take a look at Alabama's RFP for their health exchange. [1] It's 286 pages long. A version of it was issued just a month earlier. And this is just the RFP -- I assume the product spec was longer, much more detailed, with changes issued right into summer 2013 or later.

[1] http://www.aldoi.gov/PDF/Consumers/FINAL-Alabama-HIX-RFP-v47...

malandrew2• 13 years ago

I simply don't know why we don't handle this like we do tech startups. Start a small $15 million dollar fund with $1 million for each startup and a project whose scope is a very small starting point for the long term project. Give all 15 startups 6 months to a 1 year to develop the first version. Whittle it down to like 3-4 startups and grow the scope for the second round of funding. Give each like $3-4 million. Whittle it down to two or so startups.

Anything is better than this big design up front with a million must have requirements.

I also don't understand why every state has their own portal. There should be maybe at most half a dozen different teams/designs/systems. We could expand on the startup funding idea by having each state act like a venture capitalist. They hear pitches and invest in the teams and designs that are most promising each round.

All code from all participating startups should be some open source license, preferably a Public License, but a BSD-style would work too. This allows the creation of an ecosystem where all are allowed to borrow from one another.

rmc• 13 years ago

a project whose scope is a very small starting point

I can see the headlines and complaints now. People will complain that you're forgetting about $EDGE_CASE, like pensioners who worked abroa/new unmarried mothers/mothers who divorced recently/young children with $RARE_DISABILITY/etc./etc.

No matter what you do, so photographic sympathy story will have been "delibrately left behind". No politican wants to sign off on that.

You can't win.

MetaCosm• 13 years ago

Some people are working on it, but startups are DOA for government work because they can't survive the absolutely grueling proposal process.

It isn't about the tech at all, it is about getting the contracts. To pick those 15 companies, you would still need a proposal system, a validation system, an auditing system. Because the scope would be small and doable, and 15 independent contracts going out -- tens of thousands would flood in with proposals. Now you created a brand new problem, now you can't even START the contract, because picking winners has become to onerous.

The reason government gets so bogged down is an attempt at fairness and legal constraints in how they work -- the problems are actually really complex, even for stuff that seems simple on the surface.

mumbi• 13 years ago

why would anyone give 40 million to Oracle for a web development contract?

jrmcauliffe2• 13 years ago

I would assume because the sales team has told them they already have a 'best of breed' solution in 'your vertical market'. Oracle then bills the client for them to write ('customise') said solution, then adds it their product portfolio.

moron4hire• 13 years ago

I read this and I hear every experience I've ever had on government contracts: a bunch of entrenched liberal arts majors who-know-what-these-computer-things-are-about on the governments' side thinking they can throw money and jargon on literal paper at a bunch of fresh-out-of-college interns hired by the company owned by the government's project manager's college roommate, getting pissed when the programmers don't understand their acronyms and 20 year old paper business process with steps labeled "call Jim in accounting".

saalweachter• 13 years ago

I wonder how the math works out.

If you assume that each dev has a salary of $100k, they're probably costing the company $200k for benefits, office space, etc. so $40 mill pays for at most 200 years of dev time.

Add in managers and a profit margin and $40 mill probably buys you a team of fifty devs for two years.

That doesn't sound too absurd. They may have failed, but $40 million seems like the right ballpark for a healthcare exchange.

__chrismc• 13 years ago

Don't forget to account for licensing (after all, this is Oracle), hardware/hosting, software 'maintenance' fees, etc... I'd have thought a significant part of the $40m figure to be in these types of cost alone.

nostrademons• 13 years ago

I think the root cause of both this and the healthcare.gov failure is Gall's Law:

http://en.wikipedia.org/wiki/Gall's_law

The problem with dealing with the government is that they have a long list of requirements that they absolutely must have. As a result, the programmers will work to satisfy all of those requirements, and will let the unwritten requirements (like, oh, "must not fall over when more than a dozen simultaneous users connect") lapse. Frequently with software you have to trade off requirements against each other; when you go to implement it you realize that what the customer actually wanted is self-contradictory. The essence of good product management is being able to make these trade-offs smartly.

One thing that all the big companies realize is that to build a large working system, you have to start with a small working system, and then evolve it so it keeps working at all points. Google doesn't dictate exactly how the product is going to turn out at the end - it sets a general direction and product statement, and lots of individual product managers or tech leads then work on figuring out the details.

danenania• 13 years ago

This makes a lot of sense. As a consultant, I consider the most valuable and important service I offer to be analyzing, simplifying, and focusing a project's goals and requirements. If this part of the process goes well, it's usually smooth sailing on the design and implementation. If it gets impeded by obstinance, scope creep, or indecision, the project is probably doomed from the start.

Being handed a pile of requirements that are sloppy and contradictory and then being told they are non-negotiable is kind of like being asked to build a submarine out of swiss cheese. No amount of money or expertise can make that work.

dethstar• 13 years ago

I dont really understand why you guys dont get mad at your taxes beung thrown like that. Last time something like this was to happen in mexico this happened: http://techcrunch.com/2013/04/14/bringing-down-the-mexican-m...

Could A Tech Giant Build A Better Health Exchange? Maybe Not

Related Stories