Programming Language Breakdown for the HealthCare.gov Website

http://www.randalolson.com/2014/05/22/programming-language-breakdown-for-the-healthcare-gov-website/

by yawz • 12 years ago

93 59 12 years ago

dang3• 12 years ago

Url changed from http://www.randalolson.com/2014/05/22/programming-language-b..., which points to this.

rflrob• 12 years ago

I would argue that the graphical breakdown on the original link is more helpful to grok the breakdown of languages than the original reddit post.

dang3• 12 years ago

Ok, we changed it back.

rhiever• 12 years ago

Yeah, why was the URL changed?

dang3• 12 years ago

When a post's content is taken from another source, we switch the url to that source. I didn't realize that the graphic didn't come from there as well, so we changed the url back when a user pointed this out.

It's still borderline, though. Making an infographic is a pretty dubious claim to being an original source, and posts that consist of nothing but an image are usually pretty unsubstantive.

outside1234• 12 years ago

I still don't understand how they have that much code. 3.5M lines is still crazy - what exactly is all that goop doing?

aaronem2• 12 years ago

Destroying souls.

EpicEng• 12 years ago

Does that count auto generated code? I agree that 3.5m loc seems absurd (I work on a complex medical device which has about 1m loc), but to be fair, they do have to talk to a ton of different and incompatible data sources.

pdubs• 12 years ago

Most if it is in Java, which is pretty verbose in comparison to many languages.

wes-exp• 12 years ago

If Java were actually succinct, it wouldn't be Enterprise™

_random_• 12 years ago

AbstractSingletonProxyFactoryBean and friends I guess.

EricDeb• 12 years ago

Wouldn't this type of verbosity just increase the length of lines, not the line count?

watwut2• 12 years ago

Maybe they counted all dependencies or libraries. If you add all that to any project, size will grow ridiculously big. I suspect so because the coding part of the project too only half a year or so, so they could not possibly write that many line by hand.

Cowen• 12 years ago

Maybe they counted all dependencies or libraries.

This is really the only way I can imagine they got to that number.

Spring, a beastly enterprise Java codebase if ever there was one, is still "only" 460K lines of Java code according to cloc.

r00fus• 12 years ago

I imagine that 90% of that is dealing with the all of the various insurance providers and plans across the states, i.e., glue code/integration logic.

awj• 12 years ago

Tying together fifty states worth of healthcare systems and laws to provide some kind of reasonable abstract interface. On top of systems that may or may not have been designed with this form of use in mind.

Go look up some of your own state's healthcare laws to see how bizarre they can get. Then think about how each state has organically arrived at their own set of bizarre and nebulous laws. Then imagine what the infrastructure built around those laws - adapted over time as they changed and ignored as much as possible - looks like.

People surprised at this number are judging a vast and complicated problem by the (relatively) simple and pretty face plastered on top of it. I'm actually kind of impressed that the number isn't higher.

huherto• 12 years ago

Lack of code reuse. (other than copy and paste) So they probably do the same thing over and over in different parts of the application.

Reusing code is not trivial, the design has to be good, the code needs to be easy to find and call. It becomes harder as the number of people increases.

Phlarp• 12 years ago

What could you possibly need that many individual yet mostly similar languages to accomplish? Even if we grouped things like XSD, XSLT into the XML category or Groovy and JSP into the Java category it feels silly to need this many languages in a single app.

I can only imagine what kind of things all the long tail files are doing. A smattering of Perl and Python, then a lonely VB and ASP.NET page?

pnathan• 12 years ago

I've used cloc a fair bit... cloc is a bit sketchy in recognizing code occasionally. I wouldn't trust the "tail" of its output, it's frequently wrong or misleading down there.

yen223• 12 years ago

That probably explains the 5 lines of Python there.

glassx• 12 years ago

Yep. In the same Reddit thread, someone ran cloc on the Linux Kernel, and it even reported one ASP.NET file in there too (just like it did with the Healthcare.gov source):

http://www.reddit.com/r/dataisbeautiful/comments/265yns/mill...

at-fates-hands• 12 years ago

This was also my first impression. Clearly this was not a well thought out project from any perspective.

I will double clutch on my rant about government inefficiencies for now, but man alive what a mess.

mhurron• 12 years ago

Sounds like lots of small groups doing their own thing without working together.

troebr• 12 years ago

I don't understand, are you saying that there are too many languages? I find it very reasonable: - css, js, html are unavoidable for a web app - sql is as standard as it gets for interacting with your database - shell is also widely used as a glue + mini tools/scripts - maven, xml, xsd, xslt are pretty much all xml, and good luck writing a modern java app without touching it - java is the main language in there.

If anything I'm surprised at the small number of languages.

It would be way bigger for the project I'm currently working on at work: we have python for scripting and integration tests, ruby for chef deployment, java as the main language, some shell tools, js/html/css/xml/yml, some legacy groovy, etc..

It's very common to have little scripts and tools hacked together that are rarely used (or even one-offs) making it to the main codebase.

As for the number of lines of code, I don't know how they count libraries and dependencies, but with maven it's very common to be pulling half the universe's libraries through transitive dependencies for parts of the libs that are not even used. It doesn't mean your code is wrong.

It feels like people are just trying to find reasons to criticize that project.

ineedtosleep• 12 years ago

With all the healthcare.gov hate, I've been wondering why there's barely any scrutiny towards coveredca.com -- at least I haven't seen any.

us0r2• 12 years ago

CoveredCA is a true government offshore success story.

A $360M mess. Powering that beast?

1,714 cores, 9,394GB of RAM, 392,327GB of storage. Every single piece of software is Oracle with the exception of Adobe LiveCycle.

protomyth• 12 years ago

Only thing I've seen on CoveredCA is the folks that cannot find doctors after buying the insurance. I get the feeling the tech might be good enough that the social issues are taking center stage.

brikis98• 12 years ago

I just tried to go through the coveredca.com application process and it was a nightmare. It took me 5 hours, spread over two days, 3 tech support calls (I never call tech support!), and 3 different web browsers to get through it.

1. Fill out the very first part of the application - just basic name and address - click "continue". Error page. WAT.

2. Notice they have an outline view of the application. Use that to get to get past the error page. Now it's asking for my "Certificate of Citizenship" number. WAT. No form I have ever filled out - tax returns, financial aid forms, employment forms, DMV forms - nothing has ever asked me for this. Spend a day searching for it.

3. Finally find the number, fill it in, click "continue". Error page. WAT.

4. Use the outline again to get to the income section. It lists me twice, as if there are two separate brikis98's in my household. It even asks me "how is brikis98 related to brikis98"? WAT.

5. Call to tech support #1. After 45 minutes of messing with the application, clearing cookies, logging out/in, we couldn't figure out how to remove the extra "me". There is no way to completely restart an application. Solution from tech support: sign up for a completely new account - new user name, password, etc - but don't enter your SSN too early or it'll detect the duplicate account. WAT.

6. Sign up for a new account. Start filling out the application from scratch. Error page. WAT.

7. Use the outline trick again. Go to income section. No duplicate "me" this time - yay! However, now it asks for "monthly income". I just left my job, so my monthly income is zero... but I obviously had a monthly income the first 5 months of this year. Call to tech support #2. They tell me to take all the money I've made this year, divide it by 12, and put that as a monthly income. WAT.

8. Finish filling out the application. Get to a page that has a button to "See Eligible Plans". Click the button. Error page. WAT. There is no outline view to get past this one.

9. Call to tech support #3. After half an hour more, the suggestion is to try another browser. I switch to Firefox. It tells me coveredca.com does not have a valid SSL cert. WAT.

10. Switch to Safari. No complaints about the cert. Log in, click the "See Eligible Plans" button. The page loads - yay! However, the UI is broken, with strange scrollbar-in-a-scrollbar setup that makes it very tough to browse the plans. Download PDFs of each one for easier reading.

After all that, I found that the plans were all way overpriced for the comically bad benefits they offered.

ericd• 12 years ago

I ended up signing up for one, and I've been pretty happy with it, not too expensive, and decent when I've tested it. I also experienced a bunch of errors on the way to getting there.

The worst part, though, was after going through the 500 error gauntlet, hearing the person in charge of CoveredCA go on NPR and just pat himself all over the back for how great a job they had done, and much better the result was than Healthcare.gov. After spending only a few hundred million to create that hellish mess. One of the few times I've ever yelled at my radio.

aaronem2• 12 years ago

I've recently been solicited by a couple of recruiters, on behalf of contractors who've got some of that work. One of them in particular amused me by trying to hard-sell a six-month contract job with a firm which organized a "career fair" an hour south of here on two days' notice. When I showed disinterest, he "sweetened the pot" with an offer of a $25 gift card. That was when I told him not to bother contacting me again. (It's not even so much that he tried to bribe me, as that he tried to bribe me so low. I mean, really -- $25? What kind of naïf does this jackass usually deal with?)

If that's the at all representative of recruitment efforts around healthcare.gov, then I'm not sure exactly what it says about the quality of the staff working on the project, but I can't imagine it saying anything good; the best of them are probably quite bright, painfully inexperienced, and not yet equipped to understand how awful this project is going to look good on their résumés -- and I'd tend to suspect that the best of them are a small fraction of the whole. Between that, and the inevitable bloat and quality issues of any federal technology project with this many fingers in the pie, I'm really not surprised the whole thing has turned out to be such a rolling catastrophe.

us0r2• 12 years ago

Based on the last notice they sent out - around 400 people are working on this mess. I got a feeling they are more concerned with just filling the positions then hiring (and paying) experienced developers.

watwut2• 12 years ago

Based on what I seen in hacker news few month ago, a guy trying to hire people on healthcare.gov described it somewhat like this: we are saving the project and are looking for people. We hire them for few months stints (3-6 months), they work super hard that time and then leave. He mentioned occasional sleeping under desk, it did not sounded like a joke.

If true, they have perpetual death march with constantly changing workforce and everyone being essentially newbies on the project. How many experienced people can you get with that sell?

draker• 12 years ago

I've got to work on my submission titles.

Healthcare.gov developer provides estimate in response to "500M lines of code" 5 hours ago

The chart from the submitted blog post does provide a nice visual perspective of the breakdown that you don't get from the source alone.

http://www.randalolson.com/wp-content/uploads/healthcare-gov...

turar• 12 years ago

So the user-facing portion of healthcare.gov is written with JSF then?

dpeck• 12 years ago

So I have a theory:

http://www.ohloh.net/p/openjdk/analyses/latest/languages_sum...

The numbers even lineup fairly well in the 2010-2012 time frame which is around when the project would have started.

LVB• 12 years ago

At least this pile of code turned into something, unlike my local Cover Oregon which has been killed following a $250 million outlay and still no end in sight [0].

Funny that oft-mentioned Oracle used their work with Oregon to tout how their enterprise prowess "plays an essential role in establishing Oregon as a leader in healthcare reform", but the link to the source piece is now dead [1]. I did, however, save a copy of this PR fluff when it came out in 2012, just to see how it would all pan out... [2].

[0] http://www.latimes.com/nation/nationnow/la-na-us-attorneys-o...

[1] http://www.oracle.com/technetwork/articles/entarch/oeea-home...

[2] https://www.dropbox.com/s/jnt15vqdc15aopj/state-of-oregon-18...

dmix• 12 years ago

Almost as bad as Ontario's eHealth scandal:

the government spent five years and $647 million on the forerunner of eHealth Ontario: the Smart Systems for Health Agency, which used 15 per cent of its $225-million annual budget on consultants despite employing 166 people with annual salaries exceeding $100,000, before the project was shut down and restarted as eHealth Ontario.

https://en.wikipedia.org/wiki/EHealth_Ontario

Nothing like the massive bureaucracy of government healthcare combined with government run software.

pessimizer• 12 years ago

You don't have to be a government to spend half a billion on a system that never works.

vhost-• 12 years ago

Cover Oregon was a disaster. That work should have been done by an agency here in the state. I don't understand the logic of government sometimes.

selimthegrim• 12 years ago

I once went out on a Tinder date with one of the IT managers for Cover Oregon (in December/January). She was a music major at Denison and apparently Oracle was vacuuming up everyone with a pulse and a BS in DC and the East Coast to manage their contractors. I guess she was doing pretty well too since she could afford to go to Sun Valley once a month from Portland. What I wonder is why couldn't Oracle find managers with more experience?

esbranson2• 12 years ago

Are you a Oregon elector? If so, what was your logic? Did you not read the procurement notice? Did you agree with it? If you didn't agree with it, did you lodge a complaint? Or were you too lazy? Were you too lazy to even read the proposal? To read and analyze the resulting award? To read others' analyses? And if you were too lazy, how can you now complain without being a hypocrite? If you're not an Oregon elector, have you done any of these things in your state? Because I don't think an executive would be able to complain about the functioning of a corporation if they don't bother to show up at board meetings or even read the meeting minutes.

dkersten• 12 years ago

But it has all those smiling people in the pictures! What could possibly go wrong...

bjohnso5• 12 years ago

In what way is Maven a programming language? Is it simply to disambiguate other XML configuration from the build declarations?

rnd33• 12 years ago

How is 50k lines of build declarations even possible?

elwell• 12 years ago

Brute force as opposed to a few moments of contemplation for a reasonably elegant solution

fournm• 12 years ago

That's not actually surprising--it's really easy to have a 1500+ line long POM with how verbose it is. Depending on how many maven modules there are, I could see 50k pretty easily.

ChrisArchitect• 12 years ago

wasn't this floating around 6 months ago? do we really need to discuss this again?

gulbrandr• 12 years ago

At the recent dotScale conference in Paris, Robert Kennedy explained how he managed to fix the site.

See: https://twitter.com/search?q=%23dotScale%20Robert%20Kennedy

jaunkst• 12 years ago

Seems like a result of alot engineers and an inhuman level of understanding the systems architecture. Reminds me of the Lsd usage for IBM. There must be an outrageous number of bugs and exploits in the code. Google Chrome is something like 0.06 per thousand lines of code.

joshwa• 12 years ago

from http://www.enterprisetech.com/2014/04/04/hyperscale-lessons-...:

“The way that a lot of the code was developed for the application was using automated modeling tools” he says. “They configured their modelers to [generate middle-tier Java objects] in a three-tier style architecture. Essentially, they created a layer of abstraction that didn’t need to be there.”

dukerutledge• 12 years ago

Well there's 2.4 million lines of Java that will have to be maintained for the next 100 years.

Programming Language Breakdown for the HealthCare.gov Website

Related Stories