CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

https://www.technologyreview.com/2021/01/30/1017086/cdc-44-million-vaccine-data-vams-problems/

by donsupreme • 5 years ago

1167 619 5 years ago

Why was Deloitte awarded the project on a no-bid basis? The contracts claim the company was the only “responsible source”[1] to build the tool.

[1] https://www.acquisition.gov/far/6.302-1

stefan_4• 5 years ago

Deloitte is a whole lot better manipulating and sweet talking the government than it is at building anything, that is for sure.

Surely their track record should see them permanently disqualified?

tidepod124• 5 years ago

I'm sure if you actually looked at their full track record, the vast vast majority of projects probably result in "mediocre but workable" systems, you just don't hear about them because that's a boring headline. Deloitte is a $40 billion company, it's not like every single one of their projects is a complete failure.

jbnorth• 5 years ago

I think that usually comes down to the contract being so specifically tailored for a company that they're the only one who could possibly bid and meet the specs for it.

et-al2• 5 years ago

Usually government agencies write the contracts in a way that prevents other parties from bidding to streamline the procurement process.

Now what I don't understand, is why wasn't the contract written so the state could refuse broken software, or to get a partial refund if it's doesn't meet the specifications.

Someone got wined and dined pretty well. And government needs better lawyers when dealing with private contractors.

BadInformatics2• 5 years ago

In my limited experience and anecdata, governments have been so trigger happy in getting rid of and/or marginalizing technical staff that there are few qualified individuals in the procurement office who can properly draft RFPs and evaluate proposals to keep bidders accountable. What you're left with is mostly (M)BAs and admin types who can hammer out high-level Word docs of specs but not much else. In that environment, it's not surprising that Deloitte, McKessen, IBM and co (or as I like to call them, the Procurement Vampires) can lowball and BS themselves into these projects. If you don't know anything, better the devil you know, right ?

ffhhj• 5 years ago

And government needs better lawyers when dealing with private contractors.

The bureaucracy is expanding to meet the needs of the expanding bureaucracy.

FireBeyond3• 5 years ago

Which is horseshit (though we know that). I know of another, far smaller, leaner consulting firm that has handled for at least half a dozen states a contact tracing ecosystem tying into DOH, and utilizing the mobile APIs.

snypher• 5 years ago

"because configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform. Therefore, no other contractor has rights or access to leverage the system to carry out O&M activities"

mensetmanusman• 5 years ago

Why reinvent google forms?

presentation4• 5 years ago

You can say that about any app though, everything kinda can be boiled down to a spreadsheet lol. This does probably have complex requirements, not saying it’s not egregious but this comment is facetious.

endisneigh• 5 years ago

It’s also amazing to me how much a custom skin of Django Admin costs.

Has there been any research into what needs to happen to align government administrators incentives with the population? So much waste - why isn’t the government forcing Deloitte to fix?

I have no familiarity with government contracts, but as far as I know with Big Tech contracts they’re definitely written to favor the Big Tech company in these types of scenarios. The government should take a page from that playbook.

throwaway36992• 5 years ago

Cutting taxes to half might force these guys to become more competitive. I don't see any incentives for the government to give up (I'm sure this benefits their friends at these firms) until forced to make hard decisions.

adultSwim• 5 years ago

Is that a lot for a website?

klyrs• 5 years ago

I made about a dozen websites in the late 90s and early aughts... most I ever made at it was 50k/y. OTOH, they worked and continued to work long past my involvement. Maybe it's a fair price for a shoddy website?

hogFeast• 5 years ago

With these projects, there are Veblen effects. No-one wants the $50k/year consultant who quietly cranks out good work...they want the $300k/year consultants whose work may or may not be good but they can spin, make things easy with your boss, etc.

...stuff like this drives me mad. You see it everywhere. I worked in financial services, and (on my weaker days) it was enraging to see someone pay three or four times the price to get a product that was not only bad, it was literally worthless...life is weird.

cm2187• 5 years ago

Modern websites are designed to blow up as soon as there isn't an active development team anymore. The docker image needs to be rebuilt from source, the cloud APIs get deprecated, etc.

didibus4• 5 years ago

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad.

I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

BoorishBears• 5 years ago

I have no doubt the government got scammed, but often these estimates provision for things like ensuring phone support is available any time the website is up for accessibility requirements.

That's also why you see some government sites operate on a 9-5 basis

buzzerbetrayed• 5 years ago

That's also why you see some government sites operate on a 9-5 basis

I’ve always wondered why this was the case. But surely inconveniencing the majority instead of making it more convenient for the minority is a step in the wrong direction. Whatever. Off topic...

dataflow• 5 years ago

I just don't understand these software deals, the price is so high.

Don't contractors do everything they can to milk government contracts as much as possible? I'm pretty sure I've even seen people provide advice on how to do this effectively right here on HN. So I can definitely understand how it happens...

dmix• 5 years ago

The difference is they aren’t paying the engineers 300k they are paying an endless amount of administrator executives with “industry experience” aka 20yrs emailing spreadsheets between other managers at IBM.

I realized this once when I started attending (Canadian) gov ‘innovation’ events and ‘startup’ funding initiatives.

There was hardly anyone I’d consider an engineer that is benefiting from these systems. It’s a giant well organized grift for washed up “industry” execs and others hangers on with friends in the right places.

And I don’t mean to be ageist saying that. That would be over simplifying, but there is a inherent preference to those with safe names in their resume, regardless of relevancy. But there’s plenty of spaces in between for youthful people who understand how these meat factories work. Especially ones who understand their value, usually the unnamed ones working on the edges.

But by the time it comes to actually making the meat most of the money has already been allocated to the aforementioned people and they rush to find outsource the “real” work to some overpaid consulting company who are also experts at fleecing these gov contracts with endless meetings and perfecting the right checkmarks get checked.

I remember going to a Zuckerberg talk around 2010 at YC school at Stanford and he mentioned how he was hiring engineers even for the typical ‘executive’ jobs and I remember feeling like that’s the difference between startup tech companies and these failed hundred million dollar imitations.

It all comes down to the people and which ones are getting the power. Basically which side of the “iron rule” they come from https://www.jerrypournelle.com/reports/jerryp/iron.html

walrus012• 5 years ago

Or you could hire 73 engineers each paid 300k for a year, and have 22 million left over for hosting/operations/overhead costs... Absolutely ridiculous.

Raidion• 5 years ago

You get those 73 engineers in a room, then what? What do they write? You get 15 engineers, 3 DBAs, 5 sys admins + their managers. While that's spinning up, you get a bunch of business analyists to write the requirements. A first version gets written, it doesn't do what people want and the data it works from is really crappy, so you have to write a bunch of custom edge logic to deal with the crappy data. Repeat several times, including massive changes in scope.

Like is there bloat on these projects? 100%, and a lot of it, but saying "just get 73 solid engineers in a room" doesn't really get you very far when the requirements are really really unknown.

FireBeyond3• 5 years ago

You forgot annual bonuses for partners.

xyzzy123• 5 years ago

I don't think you're really paying for the development per se.

Most of the effort is actually managing the customer. That is, their contracting requirements, the whole requirements process, procurement hoops, change process, optics management, cost of "sales", reporting requirements, various additional compliance requirements...

fatbird2• 5 years ago

This is a large part of the truth. Crank out a booking website? Give me a couple weeks. Deal with your byzantine organization, your incoherent standards, your security department coming out of the woodwork with list of requirements, your torturous system for provisioning government services like VMs in secure data centres... and then pile on that integration with a variety of legacy systems across another slew of organizations that bring all their own baggage to the process.

I would blame government, but I'm simultaneously experiencing all these issues with a Fortune 100 company. They're almost indistinguishable in the degree to which their cumulative organizational headwind is a hurricane I'm supposed to fly into. It's simply hard past a certain scale for purely people and procedure reasons.

hntrader• 5 years ago

The Obamacare website cost $2 billion according to a Bloomberg piece in 2014. Don't ask me how that's even possible.

pxeboot• 5 years ago

And don't forget the State of Oregon paid an additional $240 million for their own version of it [1].

[1] https://www.koin.com/news/oregon-oracle-settle-cover-oregon-...

rtx• 5 years ago

It's possible if some money is flowing back to the people approving the deal.

seshagiric2• 5 years ago

Was it built by Deloitte too?

canofbars2• 5 years ago

Because governments outsource everything possible and end up with no persistent talent that really understands whats going on so you end up spending all the money on interfacing with contractors.

tzs• 5 years ago

My recollection of prior HN discussions is that a large part of it was that it involved interfacing to systems in a large number of agencies or departments. Most of those systems were old systems, independently developed systems, that had not been designed to have APIs for other agencies to use (or had not been designed to have APIs at all).

I think it might also have had to interface to state systems in the states that were not running their own ACA exchanges, and to the systems of numerous insurance companies.

tidepod124• 5 years ago

I've no love for Deloitte, but the one benefit they do have here is the ability to mobilize quickly on something like this. It's simply impossible to "hire 146 engineers" within any reasonable time frame, let alone get them all organized in a way that you can manage them effectively (and don't forget the project managers/scrum masters/etc that would need to be hired too). Deloitte, OTOH, has an entire existing business model based on taking vague business requirements and having huge teams of project managers/developers/consultants/advisors/etc ready to go at nearly a moments notice.

$44 million is an egregious number and I don't know how it's that high, but I'm betting at least part of it has to do with the fact that this was a website with an accelerated timeline, which probably necessitated a premium.

1986• 5 years ago

Yeah, but if it got done in time but works so poorly that no one is using it, does that model actually work for anything other than being able to say "we got it done"?

maxerickson• 5 years ago

Can't we throw away the information implied by the contract that resulted in something worthless?

(I'm not taking the dismissive 'this is easy' side of the argument here, but the project failed, the lessons learned need to start there, not with the terms of the contract)

300bps• 5 years ago

That’s not my experience.

I was hired for a particular engagement with Microsoft years ago by a similar firm. Microsoft needed x, the firm had nobody so they put out an urgent call and passed me off like I was one of their long-time employees.

I think it’s safe to say nobody has a bench of 146 people ready to go at a moment’s notice.

AshWolfy2• 5 years ago

This should have been started at the beginning of the pandemic, if this was a rush job then that raises further incompetence questions

zucker42• 5 years ago

but the one benefit they do have here is the ability to mobilize quickly on something like this

Isn't the whole article about how they failed to mobilize quickly?

skybrian• 5 years ago

This might be true for your average project, but the pandemic is not average; it’s very high-profile. Put out the word that the government needs help and you would have Googlers taking a leave of absence, like they did to fix the Obamacare website.

There are lots of great people who would rather be doing something meaningful than whatever they’re currently doing and governments need to do better at tapping into this.

It’s a different problem, but for a more recent example, look at how fast vaccinateca.com got going.

catlifeonmars2• 5 years ago

It might sound crazy, but a previous team I worked on would build solutions in this exact space (public sector) in 10 weeks using one PM and 3-5 engineers. And we would do it for free.

lumost• 5 years ago

As software goes, it's pretty likely that if you hire 146 engineers to build a website that could be built and supported with a team of 8.. you are almost certain to get a buggy mess that takes 146 engineers to maintain and support. After all, the architecture will need to accommodate 18 8 person teams, provide scope for ~20 Staff engineers along with a full layer of middle management.

I wouldn't be surprised if these proposals take into account headcount as a positive for both quality and job creation purposes.

mdorazio• 5 years ago

These contracts are generally bloated, but if you've never seen how a $10M+ contract for things like this works you're probably thinking about it wrong. The actual development piece is a pretty small portion of the overall work. The bulk of it is figuring out all the requirements, wrangling legacy systems and data into some kind of shape that can either be interfaced with or imported, documentation, support (including phone), client hand-holding, etc. Yes it's a lot of overhead, but this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks.

stefan_4• 5 years ago

When did we have the last nationwide vaccination campaign, whats the "legacy"? Pretty sure that was when IBM still made tabulators.

I think you will find that if only there was someone in charge to put their foot down, you can do without a whole lot of these things. But that's part of the problem of course: the government contracting out is just a bunch of clueless people making up a wish list with a partner on the other side whose only inclination is to say yes to everything.

j4510• 5 years ago

Sounds like a lot of $6-700/hr management consultants that don't know much about tech but are happy to channel their inner Steve Jobs to try and architect something.

The 22 year olds might not be able to do it (I beg to differ), but can management consultants/accountants who barely took typing in school who have Excel as the height of their digital literacy be expected to have all the answers?

By the time this gets to technical folks, 70-80% of the budget has been burned.

Edit: Beware Management Consultants and Accountants encroaching into "IT". Their worlds are drying up because tech folks can learn business easier than business folks can learn tech.

Edit 2: removed age reference.

nataz2• 5 years ago

This guy gets it. Development is literally the simplest part of the problem set.

The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience.

Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.

Covzire• 5 years ago

I worked at some government contractor a few years ago and one of the tasks I was given was to oversee the implementation of a ticketing system for a client. The basic requirements were for something slightly better than using an excel spreadsheet with basic search functionality. After a preliminary sales call with one well known vendor, they told us the next steps to implementing this system was to bring a consultant on-site to assess all the ancillary requirements, at a cost of nearly $100k. I had a little laugh about it with some colleges and my manager and I kind of wept a little inside, because I doubt this is a rare occurrence and it only takes 1 incompetent manager to create a massive amount of useless revenue for some vendor.

abhgh• 5 years ago

For many corporate client system developments this is my experience too. Producing the actual LOCs is not the primary cost; it's requirement gathering, figuring out how to plug in the new component into an existing system etc. As an example of the first, you might be digitizing a cross-department workflow. Your job now basically becomes to interview stakeholders from each dept and have them commit to a single online workflow. An example of the second is you might want postgre for your backend, but the client might want a single stop db on whatever they're using today.

Then there are challenges around integrating with existing APIs, workflows etc.

Let's not forget feature creep. Sometimes the client might not be able to visualize the final UX in its entirety (often simply because multiple people are involved),and therefore, they might only begin realizing its impact as and when parts of the new system becomes available.

Essentially, you are not paid to build an isolated system, but to understand well and play nicely with the larger context - human and technological - within which your system must live.

brokenkebab• 5 years ago

From the article it seems that they don't provide tech support neither to doctors, nor patients. And wrangling legacy systems in the field still may be worked out for much less money.

The real problem is that requirements are written in such a way that very few (in this particular case just ONE) companies can satisfy, so essentially there's no competition - exactly a kind of situation which lets to pump up prices.

And if you asking yourself are those requirements really benefit users, or specifically created for money to go to, you know, "right hands" - you're probably asking the right question.

jariel2• 5 years ago

No, this is not cynical enough.

Even the part you mentioned: 'wrangling data for legacy systems' does not cost that much, it's still a small piece of the pie.

The 'bulk of it' goes to sales, BD, marketing, profits, layers of excess management, specialist consultants, accountants and lawyers.

This contract is an example of 'modern world graft'.

Healtchare.gov cost $1-2 Billion and it was also a mess until Google Engineers came in to save it.

Software is not like Construction where the costs can at least be well estimated and you don't need geniuses.

It's frankly very depressing and this kind of graft and stupidity is all over the system.

There are actually not that many people who are both: 1) technically/project competent enough to figure it out and 2) have enough gravitas on the business side to make proper bids. Partly that's the fault of government.

To be fair, it's also bad buyers who change requirements all over the place and throw projects in dispensary often for what are ultimately political reasons.

It's a real failure of leadership.

blackrock• 5 years ago

It sounds like they needed a senior business systems analyst to map out the functional requirements specifications document.

Usually, though not always, this helps to give a road map for the organization to follow.

Bugs will arise of course, but usually, it becomes localized, and can be addressed individually.

But for large systems, with lots of moving parts and data integrations with other systems, then you normally need a good, near-air-tight specifications document.

bitexploder2• 5 years ago

And then supporting it. For a decade. 44m might be bloated, but, it goes quick and it is work no one really wants to do due to how hairy and legacy it can get.

didibus4• 5 years ago

Yes it's a lot of overhead, but this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks.

I get your point, but also yes it is. The issue is superficial, it's like when I try to Zoom with my mom, it'll take 10 times longer as I need to explain and bring her along every single step of the way. This is the same for these type of contracts. A lot of the slowdowns come from people just not letting development take its course as it should.

Just look at the article, the states ended up choosing to use a bunch of SaaS startup not even designed for this use case in its place, like eventbrite.

And look at the issues:

“It won’t work on Internet Explorer; it only works in Chrome. The ‘Next’ button is all the way down and to the right, so if you’re on a cell phone, you literally can’t see it,” says Rowe.

That's not a project management issue, that's a full on development issue, it doesn't even work on mobile in 2021 !!

robomartin• 5 years ago

Not to mention the terms in these kinds of contracts. The only way to deal with them is to charge enough to be protected. I have sold product (hardware) to places like the Navy Surface Warfare Center and White Sands Missile Test Range. If you are not careful you can come out losing in a big way.

For example, for each day/week/month you deliver past the estimated delivery date they discount 5% from your invoice. Sound like great motivation to deliver on time, right? Well, no, it's a guaranteed 100% (or whatever you can get away with) markup beyond what you would have charged because if anything goes wrong you'll go broke.

That said, this issue of super-expensive government websites done wrong feels all too common. You have to wonder how much cash we burn repeating the same mistakes.

bloqs• 5 years ago

Ouch my imposter syndrome

brabel2• 5 years ago

Bullshit. The broken system Delloit developed is being replaced with Eventbrite in certain states, pen and paper in others. All the users needed was a simple scheduling system which shouldn't cost anywhere near 40M to develop no matter how you spin it. Is Eventbrite works for this, why did no one actually think to just use that (or the many similar systems created for this kind of thing)? Pretty sure Deloit itself just wrapped something like Eventbrite around a skin UI and delivered that.

The only answer I can come up with, even after trying to be generous, is corruption. The fact that there was no bidding should be enough for that contract to be highly suspicious. People in the top are pocketing most of the 40M and everyone involved knows it.

getlawgdon• 5 years ago

Still not a $44mm problem.

Aeolun10• 5 years ago

this isn't a run of the mill green-field SaaS app a few 22 year olds can crank out in a few weeks

It’s probably exactly that. I always strongly wonder if you wouldn’t get a better result if you gave a pair of 22 year olds 1M and a year to develop and figure out all the same things.

NearAP5• 5 years ago

This is not a 'software engineering' thing. If you look at it from a software development POV, then the issue is more of not having 'product managers' whose job it is to - understand the requirements, write the specs, basically champion the POV of the end user (after actually identifying the correct end user). Examples

a) The article talked about the software not working on Internet Explorer (IE). Older people still use IE. Software Engs with fewer years of experience tend to optimize for FF, Chrome and latest version of IE.

b) The article talked about 'wrong' placement of buttons (probably the software is not responsive or not optimized for phones or older phones)

c) Input fields seem to be rigid (older people will make mistakes)

d) Once you're registered, you can't switch from patient to staff. This should have been designed like a 2-sided marketplace with the ability of one person to sometimes play the 2 roles

If you don't have the right spec/requirements, the output of the engineers will still be bad

bbeesley• 5 years ago

Internet Explorer has less than a 2% market share in the US, so under normal circumstances (given the sacrifices you’d have to make everywhere else) not bothering to support is is often a sensible decision. When it’s an access to healthcare issue though you can’t really exclude 2% of users.

That said, building something that works well on mobile (re your point b) but also works on internet explorer is a pretty tough ask. No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.

Idk, points a through c are basically already solved problems. I’d be disappointed to see problems like that if I was spending $20,000 let alone millions. Issue d though does feel like the sort of thing that you didn’t know was on the horizon up front, would be a pretty major rebuild to add in later on.

jijji• 5 years ago

alot of these issues can be solved by pushing code to UAT or QA environments and having a bug tracker that allows everyone to submit their bugs. It seems like either none of this happened, or it was rolled out too quickly.

didibus4• 5 years ago

I agree with you on all those listed points, but I don't know if it's my experience as a software engineer, but all those things you point out in general are something I find developers are cogniciant of, but often the processes in place doesn't allow them to do what's best, instead they are forced to always seek approval and follow stringent specs and requirements even if they don't make sense.

If you let qualified engineers drive those instead, in my experience, the end result is always better.

db48x• 5 years ago

Exactly. Civil Engineering does this better, I think. In a civil engineering project such as a bridge, the requirements may be written down by the customer, but the engineering design of the bridge is entirely the responsibility of a single architect. You can pretty much bet that the city council who bought the bridge doesn't know a thing about physics or engineering, so if the bridge falls down it has to be the architect's fault (assuming that it was built according to the plan). This gives the architect the power to say "no" to the customer.

vbordo• 5 years ago

Agreed. You can also argue that inflated contract cost in a statement of work leads to artificial labor bloat on the project. This benefits the consulting firm while disadvantaging the client. The unnecessary labor can create collaboration friction and lead to technical decisions like adopting microservices despite not being necessary to fulfill the technical requirements of the project. This kind of complexity doesn't guarantee a higher quality product is delivered to the client. Instead, it's used to justify follow-on work, further extracting dollars from the client.

It's possible there's context we don't have that exponentially increases the complexity of their tech requirements and justifies the price here but I'm skeptical. Looks more like misaligned economic incentives commonly at play in engagements between consulting firms and the gov.

walrus012• 5 years ago

If you want to see another ultimate example of this sort of contracting gone awry, google "canada federal payroll phoenix system"

https://www.google.com/search?client=firefox-b-d&q=canada+fe...

example news URL: https://www.itworldcanada.com/article/phoenix-payroll-system...

amscanne2• 5 years ago

Seems like it’s a few hundred million. IMHO Canada has no better example of idiotic spending than eHealth in Ontario. Literally 1 billion spent, and nothing to show for it. (Though they now have something, after effectively restarting.)

exabrial• 5 years ago

This is way the government works and why taxation is the fuel that feeds the fire.

Trust me, this money did not go to waste. Someone got paid, someone has a new yacht.

stefan_4• 5 years ago

Apparently this is the system:

https://vams.cdc.gov/vaccineportal/s/login/

It didn't load earlier (infinite redirect), right now its a Salesforce error page, of course. Salesforce and Deloitte, a match made in heaven.

sk5t• 5 years ago

It's long been said: if you want good software, get a bunch of accountants--and folks who want to take marching orders from accountants--to build it.

jimsmart• 5 years ago

It's a bit like some of the costs that have been quoted for some of the covid track and trace apps.

Whilst I do realise some of these quoted costs involve more than just the development of the app itself, some of the numbers are utterly ludicrous IMO as a software developer. e.g. Germany's app apparently cost approx €20 million.

https://www.dw.com/en/germany-launches-best-coronavirus-trac...

ilaksh• 5 years ago

Many years ago early in my career I worked at an an engineering firm as a third-party contractor doing programming for a large municipal blackwater system.

Despite the fact that I was working on critical code implementing core requirements of the system, I was only paid $20/hr. I believe the firm was charged $40-60.

Long before I started working in the product and before any working software was deployed, the majority of the budget was spent. I believe they had at least three civil engineers, and a hands-off software architect who I only saw once, developing a document with hundreds of pages of studies and requirements.

I remember one severe issue we had was that I was not allowed to test on the real database or one of a similar size. So a main query was tested the day it was deployed (on some extremely expensive Oracle server which was very outdated). And unfortunately my query did not perform adequately on the larger dataset. But the senior programmer decided not to tell me or to try to fix it. He just truncated my query with LIMIT and said nothing. This resulted in the query breaking and showing an error, and totally incomplete and invalid data. It is amazing it actually continued at all from that point but unfortunately it did. I only found out about it several weeks later when the project manager decided to take me on a visit. Strangely, no one ever mentioned the issue, nor addressed my immediate suggestion that the software could not operate without the problem being fixed. I assume the other programmer just said "well that guy is inexperienced" and they had other projects to work on and no budget left so they just left it for quite awhile. Anyway I could not believe that so I left shortly after.

bawolff• 5 years ago

I mean, given the results, it seems pretty clear they didn't know what to ask for, how to specify requirements, how to verify they got what they asked for, etc. That sort of thing doesn't get easier if you're hiring your own engineers.

bluedino2• 5 years ago

> At 44 million, you can hire 146 engineers each paid 300k for a full year.

It’s Deloitte, they probably hired 100 developers for 80k and 50 project managers for 300k

aweiland5• 5 years ago

Both numbers are too high.

FireBeyond3• 5 years ago

Deloitte. 60k, 150k. $13.5MM. That leaves $30MM for partner bonuses.

mikeho1999• 5 years ago

Full disclosure: I have had the distinct "privilege" on having worked with all aspects of large consulting firms, from working at Cap Gemini, to being a sub for a smaller company on a large gov't contract with Lockheed Martin, to actually competing (and winning) against the large companies with my own, small consulting company startup.

In short, the above comment is spot on. Companies like Deloitte, Cap Gemini, PwC, etc. all make their margins purely on their reputation. Just like back in the 80s when the phrase "nobody ever got fired for buying IBM" became popular, I think from the mid-90s all the way through to even 2010s there has been a similar sentiment with large, multinational consulting companies.

These consulting companies know they have a strong reputation for large scale, enterprise software projects, and so they will milk margins by charging high rates, upselling all sorts of additional features, and fulfilling projects with the lowest cost resources, whether it'd be outsourcing overseas, utilizing H1-B workers, or bringing in recent college grads. (Accenture is probably one of the most notorious for this -- not sure if they still do this now, but back in the early 2000s, they were known for bringing in college grads from all sorts of usually non-STEM majors, and they would put them through a 2-week bootcamp (er "Core Analyst Training") out in St. Charles, where more focus was on the partying and hooking up than anything else, but then shortly afterword, they'd be put on projects where they'd be expected to do technical design and even coding, while being billed out at $180+/hour)

Now, in all fairness, for every highly publicized, disastrous rollout that one of these companies gets notoriety for, there are actually still dozens of much less public, relatively successful implementations that gets deployed without any issue or fanfare.

But despite this, the costs for even the successful projects are still ridiculously high. As the main reason for their huge margins are primarily in their reputations, then as their reputations continue to erode from things like this news piece, clients will start to wonder if the large costs can continue to be justified.

And I'm beginning to see this sentiment directly -- my small dozen-person consulting company has not only been able to compete directly against these larger companies, but we have started winning contracts against them as well. For the latest contract we won, we bid at about 20% of what one of the larger companies ended up coming in at -- they actually thought they were a shoe-in because they were the incumbent, having built out several other projects already for the client. But we ended up being selected not just because of our pricing, but because of our overall proposal, favorable terms and expertise on the domain.

What's even more surprising is that even though we offered "bargain basement" pricing relative to theirs, it will still end up being the most profitable project we have had to-date -- it just makes it crazy to think what kind of profits/margins that they can make.

smabie2• 5 years ago

I have no experience in this field, but do you think it's more likely they are making crazy profits or that they are just really incompetent? probably a little bit (or a lot) of both I'd imagine.

angry_octet2• 5 years ago

I endorse everything you've said! Also, big consultancies have high costs (hello Booz! great lunch btw) like trade shows, political donations, lobbyists, ex-Agency/Military high muckety mucks on the payroll, etc. At least 50% of the project billings are set aside for this. Projects still need a 20% profit margin too. That might mean they're using budget staff; sometimes this means global software centres. If you think DBAs and coders in Mumbai or Warsaw might have a hard time understanding the organisational and political geography of all the States you'd be right.

Zenst2• 5 years ago

At 44 million, you can hire 146 engineers each paid 300k for a full year

Maybe they did and the too many chefs to cooks ratio played out, seen and experienced that.

nojito2• 5 years ago

Liability and integration causes bloated costs.

The bigger issue is that it’s a no bid contract. However there’s no real process for verifying individuals who bid in a timely way.

2OEH8eoCRo02• 5 years ago

I work in defense and I always say the govt pays $100 for $1 of actual work. It makes me pretty angry at times as a taxpayer.

The worst part is that we know government software contracts are done completely wrong but the government doesn't want to change. Too much money to be made.

known• 5 years ago

This issue/corruption is addressed in https://yts.mx/movies/a-good-american-2015

tyingq2• 5 years ago

Deloitte billable rates for the higher end folks are $350-700/hour. Scapegoat as a service isn't cheap.

scythe• 5 years ago

At 44 million, you can hire 146 engineers each paid 300k for a full year.

You can, but the government has to jump through a lot of hoops. Even government contractors have to jump through a lot of hoops. Sometimes there are regulations affecting the hiring process even for government contractors, e.g. drug tests, citizenship requirements, background checks, location requirements, etc. Every relevant law named after a sympathetic child adds $5B to development costs. And so forth.

option• 5 years ago

this just smells corruption. Plain and simple, alas happens not only in “3rd world countries”

tootie2• 5 years ago

Aside from all the other answers, it would cost you millions to hire and onboard 146 qualified engineers. A place like Deloitte can drop them on your doorstep with a week's notice. They also have all the product design, strategy, and organization layers required to stand up the equivalent of mid-size enterprise from scratch really quickly and then tear it all back down a year later. I've worked in this business for smaller agencies and we could absolutely work miracles for stodgy organizations. My biggest account probably took in $20M all told and I'd say we earned every penny and launched a really successful product really quickly. Failures can happen too and they're usually a lot more complicated than just incompetent developers. It's just as likely the client is unable to articulate need accurately and corners get cut.

ryanSrich• 5 years ago

These deals are mostly to help out friends and line pockets. Building something that works doesn’t even matter. Government contracting is one of the biggest scams happening in broad daylight.

jmeister• 5 years ago

They needed to hire somebody better. Palantir would probably have gotten it done for much lesser, faster.

adamjb• 5 years ago

Divide that number by 3-5, for starters. Charge out rates are multiples of salaries.

thegeekpirate• 5 years ago

FWIW, feel free to take a look at this contract from the BC government for a centralized education website (http://docs.openinfo.gov.bc.ca/D29078314A_Response_Package_E...). This is what it looks like (https://i.imgur.com/fmUVYhG.png), and for comparison, here is the old website (https://i.imgur.com/IW5J9gS.png).

This is projected to cost 95 million dollars over 12 years.

Yep.

jiggawatts• 5 years ago

Let me try to explain using analogies.

Organisations are like people. Their culture is like someone's personality. Dealing with easygoing people is much smoother than paranoid delusional people with OCD. In the same way, dealing with government agencies is difficult because their corporate culture -- their personality -- is not what most people would categorise as normal.

These deals aren't normal deals. They're the equivalent of specialist treatment for someone with mental problems.

Take for example an organisation like SpaceX: They simply do things. If some manager goes to Elon with a rational but inherently risky proposal, he'll weigh the risk and then more than likely give the goahead. Other companies have a lower risk appetite, but they will take occasional risks. Sometimes huge ones.

Large federal government organisations are the diametric opposite of this kind of agile, low-overhead, no-bullshit private company. Their appetite for risk is precisely zero. That's difficult for people to wrap their head around. It's like a superconductor. It's not just a "really good conductor". It's not a "little better than copper". No, it's something fundamentally different. This kind of zero tolerance squeezes out entire lines of thought, just like a superconductor expels a magnetic field.

Notice how in this article the blame is on Deloitte and the "problem" is that there was a failure. This isn't a fault, this is the system working precisely as designed. Nobody will get fired for picking Deloitte, their purpose all along was to take the blame for the low-low price of a mere $44M so that a government employee making $100K won't have to have a couple of uncomfortable meetings.

The problem isn't that there was a failure, that happens all the time with software projects. The problem is that risk in government agencies is managed by people with essentially infinite source of free money that they can draw down on in order to protect themselves.

If a magic genie gave you a million dollars every week to spend on your own safety with no strings attached, wouldn't you? Would you say "no"? Really?

timmy-turner• 5 years ago

Reading "Let me try to explain using analogies.", I was worried someone would come up with another car anology.

Seeing orgs as people with characters, oh my, my eyes are opened now. Sounds weird but it makes sense the few times that I have worked on contracted projects.

For me, an added complexity in gauging the other side of the contract is that their personality changes depending on the actual people that are involved. When some higher-ups join the project it in later stages, the personality may turns even more mental than it already is.

codpiece2• 5 years ago

I noticed a Salesforce blue cloud favicon when I interacted with the system.

tyingq2• 5 years ago

Yep...

"The vaccine management solution is a mobile-ready application that is built on our existing GovConnect Salesforce platform."

https://www2.deloitte.com/us/en/pages/public-sector/solution...

juskrey• 5 years ago

From time to time I am trying to crawl government procurement portals for projects, and dropping the idea after a couple of hours again and again: only proposal, initial negotiation and documentation will probably require several people full time, this is even before any project movements.. Mind that, this it's the system where you're bidding for months for a project, and even if you have won and was good at delivery, you will be doing all the rounds for the next one again. So governments, sort of meaning to drive the price down via tenders, in reality are paying for all the losing bids too, and paying a lot.

Cd00d• 5 years ago

Your point stands, but a nitpick - don't forget overhead. You should roughly double engineer salary to cover overhead, so it's 146 engs at 150k (still good), or 75 engs at 300.

Either way, the deal should be possible.

tvon• 5 years ago

It is not a tech problem, it is a people problem.

ddingus2• 5 years ago

And the government can own the code, publish, and as the site finishes up, that team can proceed to apply all they learned and built to other pressing tasks.

A decade of that, and a whole lot of government would be awesome.

Of course, these arguments have been made many times. Money in politics prevents it.

I have presented them. I have watched large donations and a couple nice dinners make the whole idea go away.

WalterBright• 5 years ago

I once hired a company to do some structural repairs. The company usually did contract work for the government. Some weeks went by. I was billed a progress payment of $5000, but the company hadn't actually accomplished anything - just reviews, several site visits, meetings, evaluations, memos, emails and constant reassignments of who was working on it.

I fired them and refused to pay their bill (they hadn't even produced a plan for how they were going to proceed). Hired another outfit, who did the entire job front to back for $5000, and got it done in a couple weeks.

bredren• 5 years ago

This is the same old argument used against Burning Man org’s ticketing system.

3327• 5 years ago

Deloitte should be investigated for defrauding tax payers

vncecartersknee• 5 years ago

I used to have a few friends who worked at deloitte as analysts I'd often catch up with them for lunch or after work drinks and I can guarantee you this was most likely made by a bunch of coked out 22 year olds working until 3am every night for a couple of months

It reminded me of a university style group project, constantly astonished that they’d turn over a few 10's of thousands of lines of python code that (usually) some government department was paying millions for. Not to smear my friends who worked there who must have to work incredibly hard given the conditions and the stress they were under, I always suspected their code must have been absolutely riddled with bugs, and I shuddered to think what it must have been like for the next poor grads who had to work on it after them.

And it was always some project where I was amazed there wasn't already some mature software out there that accomplished what they were doing anyway.

But anyway they seemed to have fun so whatever

bhouston• 5 years ago

I believe Deloitte just PMs the project and farms it out to a sub contractor. This is standard operating procedure.

I understand the Deloitte prefers heavily documented and waterfall development with overseas developers doing the work. Basically top heavy and expensive management and cheap dev resources.

CraigJPerry• 5 years ago

Those would be public employees, the argument is you’d be crowding out private enterprise. Government should be small goes the argument.

KorematsuFred• 5 years ago

Oh my sweet summer child, thou has not seen the internal working of government contracts.

In order to qualify to even bid on a contract there is a pretty high entry barrier. The amount of time it will take you to cross that barrier would be more than amount of time it will take you to start a real business. Bidding is another expensive process which costs money. If you bid on 10 contracts in a year and win one the cost of bidding on rest 9 contracts must be covered to the won contract.

Then comes the compliance costs of various government regulations. In a startup world you can take up accessibility as a P2 item but in government contracts it is a P0. So there are all these complexities involved which bloat the contract. Government has passed probably thousand regulations on whom you can hire and how you work, where the data is stored and what kind of background verification you have done on your employees.

Of course if you and I were to develop a similar website we could do it for $4M and spare some cash to buy couple of Ferraris.

PS. There is also bribes. Of course these are never called so but you have to spend good amount of time golfing with right people, throwing parties, offering front row seats to NBA finals and those sort of things to get your files moving.

that_guy_iain2• 5 years ago

My first IT job was with a hosting company that noticed how much the goverment contracts were bloated. They then spent ages trying to become an offical supplier so they could undercut companies charging so much more. Literally, one of the issues they had was that so many people were curious as to how the company could make a profit since the prices were so much lower than all the other contractors like Serco.

We also ended up being in contact with Serco about a tender they had to redo one of the goverment websites. We ended up not getting any work from that because Serco was under pressure to stop outsourcing all their contracts.

Goverment contracts just send lots of money to companies who already have lots of money.

barrkel• 5 years ago

Serco

atlgator5• 5 years ago

The 44M is most likely over 5 years. Knowing Deloitte generally comes in on the high side of pricing I'd say it's around a 37 person team, cross functional including PMs, BAs, developers, testers, devops, Section 508 compliance, IA, user support.

atlgator5• 5 years ago

It's Time and Materials/staff aug contracting - meaning Deloitte isn't selling an end product, they are selling staff on a per hour basis. As with other types of contracting, the hourly rate has a huge markup on it to cover indirect costs as well as profit. They make a lot less profit than you think though. Most of it is indirect costs.

lmilcin3• 5 years ago

I have worked with Deloitte on two projects (on clients' side, I would never willingly work for Deloitte). Generally expect them to do absolute minimum necessary to meet the contract. The code was obfuscated to ensure support contract. The people employed to write it were not interested in the quality in the slightest but were interested a lot in specifics of the contract.

sillysaurusx4• 5 years ago

Lol.

How was the code obfuscated?

That brings back memories of php obfuscator. What a strange world that was. I think there was even a deobfuscator that tried to rename the variables sensibly, or something.

lmilcin3• 5 years ago

Well, you can make simple functionalities so baroque that you will not be able to work with them without some kind of notes. It is not easy to define in contract what is sane code.

ntsplnkv27• 5 years ago

They're getting paid for a deliverable. Why do you expect free work?

imranq• 5 years ago

No good engineer I've worked with only did the bare minimum. Usually it was 2x or 3x beyond that e.g. thinking about the customer, understanding the right technologies, designing for scale, etc.

alasdair_2• 5 years ago

They're getting paid for a deliverable. Why do you expect free work?

And this is how we get the 737 Max.

eCa• 5 years ago

I have only worked directly with an external consultancy company once, and the problem (the way I see it) is that the incentives are mis-aligned.

A good internal developer will notice that requirements are missing, or could be improved, and has the incentive to bring it up with the project manager and get it fixed.

That incentive does not necessarily exist for a consultancy developer, because as you say that means their employer loses money.

My conclusion: Use consultans for very well defined and testable problems that don’t require too much business-specific domain knowledge.

lmilcin3• 5 years ago

Imagine you are signing a contract with a car shop to add a spoiler to your car.

The car shop spends months of your time obsessing about the dimensions of the spoiler. Then the car is out of service for as much time and you start wondering what they have been doing.

Then the car arrives and there is a bunch of paint missing around the spoiler, the spoiler is completely unpainted an it falls off when you accelerate to 100mph.

So you ask, "WTF?" and they tell you that none of these things were specified in the contract.

That's more or less how it worked.

I have been contracted for a number of projects and I always assume some level of standard that my client would want me to meet.

For example, around 2003 I have been contracted by a publishing house to restore couple of their servers. Their employees made a mess they were not able to get out of, completely mangled Linux distributions, took out drives out of RAID5 and did not know what to do with them, etc. Drives had files with only copies of texts of books they had so it was extremely valuable to them.

So obviously I did what ANY sane person would do:

- I have discussed with responsible employees what exactly happened,

- We went out on a little trip to buy backup drives

- We made backups of every single drive involved in the mess (I keep saying we because I involved the guys in the recovery so they know how it works)

- We put together the drives and they worked

- We fixed OS-es which they mangled

- We tested that everything works

- We have discussed why they need some procedures and especially backup procedures

- I have showed them how to construct couple of these procedures so that they know how to do that.

I didn't have to do all that. I was already being paid handsomely for the recovery and would not suffer (other than lost pay) if I failed recovery. But I also recognize how important this data was for the company and so I took necessary precautions proportional to how important this.

thrwn_frthr_awy• 5 years ago

I too worked with Deloitte. The goal was to get the project signed-off on as quickly as possible because once that is done they get to bill time for maintenance/bugs. QA did not exists.

brundolf6• 5 years ago

Serious question: why doesn't the government have its own internal software development department at this point? One that's familiar with all the regulatory and bureaoucratic hurdles and the legacy systems, but doesn't have the incentive to grift. For the amount of money they spend on these contracts surely they could afford to pay competitive salaries?

et-al2• 5 years ago

There's 18F for the US government, but unfortunately they're limited by government salary caps and other policies.

offtop54• 5 years ago

For various reasons this wouldn't work, you have the various bands of government salary

https://federalnewsnetwork.com/pay/2020/01/more-top-career-e...

But for a GS-15, step 10 in Washington, his or her salary should total roughly $185,509 in 2020. But due to the federal pay ceiling, that same GS-15, step 10 in Washington will make the limit of $170,800 this year.

And that's the absolute Max, after you go to the trouble of getting a security clearance, which you just don't need in the private sector.

Even government contracting involves so much red tape, more innovative engineers would probably just rather not do it.

clintonb• 5 years ago

It does: https://www.usds.gov/. It's my understanding that USDS also bids on projects, but isn't necessarily the go-to team for such work.

ergocoder• 5 years ago

Generally, it's much better for the government to outsource. More competency and reduce corruption.

I came from a country where government tries to do all these things in house. It was bad. Paying premium isn't a big problem.

JohnJamesRambo• 5 years ago

I'm reminded of a guy that used to go on Leo Laporte's podcast and had a program or joined a program where Silicon Valley tech people join government entities to make their tech and websites better. I'm sure they took massive paycuts to do this but they felt it was their duty, like military service. This guy and others like him are true heroes and patriots to me.

gunsch3• 5 years ago

As others have mentioned already, 18F and USDS are in this space. They do some work, but their scope and manpower is limited, and they try to multiply their impact through contracting. Part of that is helping federal agencies write better contracts to be able to source the kind of contractors they want --- ones who think in terms of product development, user research ("build with, not for"), and are using modern software development practices. Bringing in contractors helps them multiply their impact, especially with how difficult hiring people into government is. (and a lot of people in tech simply didn't want to work directly under the Trump administration, which --- as liberal as the tech field tends to lean, seems to be a factor on hiring at 18F and USDS).

I work at one of the contractors in the space (Ad Hoc [1]) and get to work with people from USDS every day. Like others have commented here, the government salary ranges are a problem for them to hire enough. Our last COO had been an administrator at OPM, and would say things like "on the government side, I couldn't ever hire the kinds of software developers that we're able to here on the contractor side."

[1] https://adhoc.team/ -- we're hiring, full-time remote!

test6554• 5 years ago

"One that's familiar with all the regulatory and bureaoucratic hurdles and the legacy systems"

How much does it cost to get a qualified person to put up with all that though?

throwhtrhwthrow• 5 years ago

throwaway for obvious reasons. I've worked with Deloitte, PwC, KPMG, and other big consultancies for a diverse range of reasons (regulation audits, security reviews, staff augmentations, etc.) and the value you get for the money you're paying is pretty insane. I'd say the value you're getting is probably 20% of the price you're paying, and I'm being optimistic. You can get good value if you know the people you're hiring, if you scope your project well, and if you have clear expectations. But let me tell you, nobody knows how to do this, unless they've been on the other side.

Now, you don't always have a choice, the expertise is sometimes there, or you need one of the big four to do a prior audit before a real audit from another big four. But be prepared to have a team of OK people who will spend a lot of time producing slides and organize unproductive meetings with lots of other people (including yours) to give out a false sense of "we're making progress here". You'll have to do the PM'ing and be extremely strict with what you want from them, and you'll have to be ready to collaborate heavily. What I mean by this is: don't wait for them to ask questions, you'll have to do most of the work of onboarding them, almost like they're your coworker except that they don't have connections with the folks they need to work with, they don't have access to internal stuff, they don't know how to get there.

aasasd4• 5 years ago

Sounds like one should split the system into small parts, order each one as a separate contract and basically micromanage the contractor into delivering the actual thing.

madaxe_again• 5 years ago

The people doing the development were likely not developers. A particularly common path is studying a random subject at university, doing a law conversion course, failing to get a pupillage or a position with a commercial solicitors, and going to Deloitte as a trainee consultant.

There, they put you through a six week course in which you become a developer, and you’re then hired out for £800 a day as a technical expert.

Source: literally half the people I went to university with.

Fond memories of sitting at the kitchen table and trying to coach my flat mate through unfucking the air ground traffic control system that they were writing for BAA in VB5, in 2005. No idea if it ever went into production.

StrauXX• 5 years ago

In Austria (at least in the province I am living in) the provincial government does have its own software development departmant. I am currently working with them (for free :)) on a software project of theirs for a class.

bitcharmer3• 5 years ago

I worked on huge projects in telco where big consultancies where a common thing. We had a joke about Accenture and their tactics. Upon being questioned about the quality of the software they delivered they would reply:

"There was no requirement that the whole thing should work. We can put in a change request and bill separately".

Also - the "no one has ever been fired for hiring IBM" syndrome among the higher management.

leephillips• 5 years ago

I could have made a broken website for less than half of that.

EduardLev• 5 years ago

Good article.

I thought this line from the article was interesting:

Online sign-ups are especially challenging for older people, perhaps the worst group to beta-test a new system.

I think I understand the author's intent when saying "worst group", but in this case I can't imagine a better group. They're literally the first people who will need to be using this system! If you beta-test your product with the people who will be able to use it despite all of its flaws, is that really testing?

Anyway yeah, just upset at this comment given that I have family members struggling with all this vaccine appointment software.

presentation4• 5 years ago

Worst in the sense that older people often are unfamiliar with computers at all - perhaps that group should have robust options that don’t come in the form of a web app if many have never even used a computer before.

asdf333• 5 years ago

what an embarrassment.

bitcharmer3• 5 years ago

Sadly this is the norm among big consultancies.

alsoc• 5 years ago

If we draw UML diagrams the software will be better, oh wait before we draw the diagrams we can make a project plan and present how long each stage will take and how many resources are needed to draw those diagrams. Ok that comes out to about $44M

johnthuss• 5 years ago

My favorite part was this quote: “It won’t work on Internet Explorer; it only works in Chrome.” As if that is a serious problem.

hitpointdrew• 5 years ago

It is if your target audience is 70+ year olds.

presentation4• 5 years ago

It is if you’re working in government or other fields like medicine, where a ton of both legacy and new software is still being written with IE as the target, that have prevented people from moving off it.

Pasorrijer• 5 years ago

Also a problem if you're working with airlines, medium to large companies over 15 years old, the government, etc. Etc. Comments like these are why the people who have to interface with actual customers get so mad with development teams.

userbinator2• 5 years ago

It is a very serious problem when you are the Government and you need your site to work for everyone.

Also, fuck Google's monopoly.

(I'm a long-time IE user.)

notapenny• 5 years ago

According to the login page for the system it works in FF, Safari and Edge as well. I suppose "only Chrome" is more accurately "if it works in Chrome, it'll run in everything but IE".

Still makes me wonder why though, most projects I've worked on have have to support IE, and though it annoys developers because they cannot use the hotHewThingN api, mostly its fine.

yalogin• 5 years ago

This kind of looting is common place and is built into the federal contracting situation. After this 48 million or so they will get more to ix the bugs and maintain it. The maintenance is the most lucrative aspect of the contract.

Unfortunately this corrupt behavior is deeply rooted and condoned and expected and there doesn't seem to be much motivation to come up with a solution.

Wonder why the government doesn't just staff up a tech wing themselves instead of outsourcing. There is so much to do and its much more efficient to just create an tech department that serves all branches of government.

antipaul2• 5 years ago

But then that tech department would become government itself, and would that be better?

Maybe, maybe not.

Though rare, there _are_ private success stories, such as healthcare.gov 2.0 [1] and Palantir, though the latter gets in hot water for some tangential reasons. But maybe figure out how these happened, and scale that.

[1] https://www.theatlantic.com/technology/archive/2015/07/the-s...

estaseuropano3• 5 years ago

But healthcare.gov was a private sector disaster and then fixed through a gov-internal startup-like tech team.

oivey2• 5 years ago

But you see, something something, core business government isn’t tech, something something, private contractors are more efficient.

NearAP5• 5 years ago

After reading the article, I think it boils down to

a) Poor/lack of understanding of who the actual users are e.g. these are systems that will be used by both the young and old people, especially older people in the early stages since they are the ones at the front of the vaccine line; worst case, the system should have been optimized for them.

b) bullet a then leads to poor requirements e.g. given that most libraries are shut down (lots of elderly people get help with online stuff at community centers/libraries), requirements should have included less rigidity (on errors) or a module that is fun and friendly that walks people through it, system should be designed with 'plain' mobile first in mind, and account for a person being both a patient and an admin, etc

presentation4• 5 years ago

It’s hard to know what actually went wrong without being in the project itself - there’s probably organizational issues within Deloitte and bad incentives around the entire procurement process, but hard to know from the outside just seeing that it’s a disaster.

mikesabbagh• 5 years ago

Companies like deloitte sees the CDC as a big cow that needs milking. They will never finish the job. It is like going to the hospital and tell the doctor I have a great insurance. The Doctors will do every possible procedure. It is better to check. CDC needs to employ developers and do the job internally.

corytheboyd3• 5 years ago

Hey CDC, I’ll build your website for half that price. By myself. And it will work. Just uh, hit me up haha

It’s insane how far those unfamiliar with software will take the mythical man month. Nonono throwing money and people at your problem is absolutely not guaranteed to solve it better.

My offer is of course a joke, well unless they want to talk specifics and I agree it’s reasonable to take on myself. Or just plain reasonable.

xyst2• 5 years ago

are you also going to answer the phone calls and e-mails of thousands (potentially millions) of technology illiterate users?

corytheboyd3• 5 years ago

I mean with $22M on the line I’m willing to at least investigate a means of doing so.

xyst2• 5 years ago

should have hired the bois at palantir to build the system

(I am invested in PLTR)

singingfish• 5 years ago

OK, so the fundamental problem with health IT is a greater problem in the commercial software industry as a whole. The buyer tends not to be the person who uses the software, and the more bureaucratic/ technocratic the process you use to get to the end user the more likely you are to not meet their needs. Put crudely and inaccurately, the procurement team end up going with the consultants who do a good game of golf.

Add government and a big consulting firm like Deloitte in health IT, and you'll be ready for a world of pain. Here [0] is a good paper on the topic.

[0] https://pubmed.ncbi.nlm.nih.gov/22188347/

anewlanguage• 5 years ago

Can anyone actually load the website (https://vams.cdc.gov)? I get a "too many redirects" error, which is quite a bit worse than "the buttons are poorly placed".

8192kjshad09-• 5 years ago

It seems to redirect to https://vams.cdc.gov/vaccineportal/ which redirects to https://vams.cdc.gov/vaccineportal/s/ which redirects back to https://vams.cdc.gov/vaccineportal/ for an infinite redirect loop.

Wonder if anyone will pay me $44M to set up a redirect loop?

billions3• 5 years ago

The last time the US government made something to be proud of was 50 years ago with the Apollo program. Counter examples welcome!

ceejayoz2• 5 years ago

The Hubble Space Telescope? (and the other "Great Observatories" satellites)

billions3• 5 years ago

Hubble was funded in the 1970s with a proposed launch in 1983, but the project was beset by technical delays, budget problems, and the 1986 Challenger disaster. It was finally launched by Space Shuttle Discovery in 1990, but its main mirror had been ground incorrectly, resulting in spherical aberration that compromised the telescope's capabilities. The optics were corrected to their intended quality by a servicing mission in 1993.

Meh.

damon_c2• 5 years ago

TCP/IP is pretty awesome. How about that?

https://en.wikipedia.org/wiki/Internet_protocol_suite

To be fair... it's also from about 50 years ago. Hmm...

damon_c2• 5 years ago

Ugh... The GPS project was started by the U.S. Department of Defense in 1973...

hbcondo714• 5 years ago

Reminds me of the discussion here a couple years ago when Accenture was sued over their website redesign for Hertz rental cars:

https://news.ycombinator.com/item?id=19737070

nickthemagicman• 5 years ago

This makes me so upset.

I'm currently also at a federal research institution thats millions of dollars over budget and years behind on a software project, and there is NO PRESSURE on the management because federal grant money printer go brrrr.

If time is money, this is peoples lives. Who holds these people accountable?

lxe• 5 years ago

Why hasn't any politician taken up a stance against bloated contracts like these? I think it's a cause that lots of folks could get behind.

aweiland5• 5 years ago

Because the companies that get these contracts pay for the politician's war chest.

0xcde4c3db• 5 years ago

Not to mention employ the politician's constituents.

I don't know if it's ever been conclusively proven, but it's been alleged for decades that the major defense contractors deliberately structure their big projects into subcontracts that can employ people in as many Congressional districts as possible. There are allegedly people in 300+ districts working on the F-35 program, for example.

zaroth• 5 years ago

Deloitte is the only contractor that can meet the project requirements, because configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform. Therefore, no other contractor has rights or access to leverage the system to carry out O&M activities, such as the administration of the system, change and defect management and end user support. As the only contractor with the GovConnect proprietary platform, and the need to stand up O&M support for VAMS while concurrently developing future releases, Deloitte is the only contractor that can achieve the O&M scope and contract requirements.”

Can’t make this stuff up.

https://beta.sam.gov/opp/2f5fc512fdde4e22860832337aac420f/vi...

satya71• 5 years ago

Holy cow! Talk about vendor lock-in! Even AWS is easier to get out of.

faramarz• 5 years ago

Wow! the next article should be about who signed that contract!

animex2• 5 years ago

US Digital Service Assemble!

JoBrad• 5 years ago

Exactly. We need someone to “Amazon” the shit out of our governmental computing systems (e.g. APIs everywhere), especially those designed to interact with other systems. They should be required to use an accessible API with published schemas and documentation. I’m surprised someone hasn’t identified this as a stability and security risk for our government.

mgh22• 5 years ago

Isn't Palantir doing something similar?

MikeDelta• 5 years ago

Yes they are from what I understood. Clever strategy if you are dealing with a client who doesn't see further than the next move.

aasasd4• 5 years ago

Reads like a copy of Russian government contract practices. Put up a contest, list requirements in such a way that only one company would satisfy them without going bust or mad, award the contract to that company. ‘Make the thing using this and that technologies’ does appear regularly.

smsm425• 5 years ago

Russia at some point instituted a law that all contracts have to be published in a centralized database. So, to award contracts to specific companies, an array of techniques was developed to prevent anyone from using that database to bid on those contracts. The names of the goods are misspelled, spelled using a mix of Cyrillic and Latin alphabets, the items are required to be of specific model with tiny modification from common one that only one (pre-arranged) provider happens to have, unusual names are used for goods, etc. etc. It's a whole branch of art now.

pessimizer• 5 years ago

It's absurdly common and has nothing to do with Russia other than plenty of Russians also do it. The reason you pay more to get a political job than political jobs pay is because you get control of contracts. Contracts get you rich.

edit: Speaking from direct American government experience.

brtkdotse2• 5 years ago

There was some hubbub in Sweden a while back about the armed forces putting up a _really_ specific job ad, like it was tailored to one person.

Well, it turned out it was, because the person posting the add forgot to take out the phrase “Make sure Björn meets these requirements” in the middle of the ad.

tw20212021• 5 years ago

Not Russia but eastern Europe. A long time ago I was working at this company 'subscribed' to government contracts. There was a new project for the army/nato and my colleague was writing our bid offer. Right next to him sat an army officer whispering him exactly what to write down to win the bid. Needless to say all public contracts where won by a small group of companies in the same manner. But sometimes the bribes were even higher, and companies hq-ed in central and western Europe, like IBM and the likes, would win the contracts and then sub-contract my company to do the work.

philjohn• 5 years ago

This is endemic in tender processes the world over.

I used to work in a role where I had to respond to tenders for the software I led, and it was obvious that if the requirements list wasn’t written by our competitor, the institution had decided they wanted their software and listed every feature they knew only that company offered.

komuher• 5 years ago

Sounds like poland and hungary today :)

ThinkBeat• 5 years ago

This is quite common in the US.

For a while my entire job ended up writing proposals and all the related nonsense for federal an state government contracts. I did not like that one bit.

However spending week after week trying to divine the meaning behind phrases and terms.

It shortly became apparent that some parts of the government would consistently have requirements only applicable to one vendor.

That saved a lot of time since I could just skip them whenever they came out.

These types of «arrangements” often go all the way to congress promised pork for donors and secure job for voters.

If I have given the impression I was doing all the work myself that is not intended. We had a big team of all sorts of disciplines

user3939382• 5 years ago

Reminds me of an RFP for the Pennsylvania Land Trust Assocation I spent $10k preparing a bid for when I was young and naïve before I understood that RFPs were often a sham to give auditors the illusion of due diligence and that it would have been impossible for us to win the contract.

eplanit• 5 years ago

It reminds me of how during the Iraq war (fiasco), Halliburton, partly owned by dick Cheney, was awarded a huge "no bid" contract (no competitor by design, no bid required) for logistical support in Iraq.

No corruption to see here folks, move along.

mgkimsal2• 5 years ago

That's nothing to do with Russia. I had a client make an RFP based on my company's service specs. The entire intent (stated behind the scenes) was to have us get the work. We'd already done other work - they wanted to keep us involved for the next phase of work. We lost to someone local. They charged basically the same as we did, but they were local so didn't have to include travel costs, so they 'won'. The client lost out because there was no consideration for continuation of knowledge/experience - we'd done an initial training course for them - our teams knew each other and had done some informal prototyping already while on site. But... "the other bid was lower".

ineedasername2• 5 years ago

It's not that it reads like a Russian government contract, it reads a bit like any government contract.

The problem here isn't even necessarily with the government: Had they gone with anyone else at a sane price tag, Deloitte would have sued that company to oblivion and the Government as well, probably for all sorts of IP issues.

bsaul• 5 years ago

Russian government certainly doesn’t have the monopoly of that kind of practice. It’s a standard practice in wordlbank projects as well : wordlbank finances the development of a system in a foreign country, and the projects requirements are written in such a way that only US companies can answer.

enigma20• 5 years ago

This also happens a lot in some European countries, where governments/municipalities buy/lease cars for. They specify features list in the way, that only one car model form particular manufacturer matches the requirements. Pure corruption.

smsm425• 5 years ago

Somebody had written this with exact purpose to give this contract to Deloitte. I would be very surprised if there wasn't bribery involved.

8bitsrule2• 5 years ago

Geez, I thought the whole idea is that 'government is the problem. We need to let the private sector handle all these things.'

Making the CDC look bad appears to have been part of that program last year. Shame the price has been so high.

closeparen2• 5 years ago

For your next laptop, car, etc. try doing what you think the government ought to do here. Write down the requirements, give them to a neutral third party who doesn’t doesn’t use computers/drive, and ask them to figure out the cheapest thing that meets the requirements. Commit yourself to buying only that. See how it goes for you.

StavrosK• 5 years ago

I don't understand the analogy. Why would I give the requirements for making a website to someone who doesn't make websites? The government should do whatever anyone who wants a website does, hire someone to make a website. What's the issue?

kuschku• 5 years ago

But that's how I procure anything and everything. I go to geizhals.eu, select the category, select my requirements based on comparative testing from test.de, sort by cheapest total, and click buy.

Every single piece of electronics I own was bought that way.

How else would you buy stuff without being influenced by some subconscious bias caused by some stupid ad?

thw0rted• 5 years ago

This is a common trap that people fall in while trying to understand the problem. It's not that they chose the "wrong software", it's that they tried to scope every aspect of custom-designed software ahead of time.

Imagine if you had to describe in exact detail every possible aspect of a mission to Mars in 1940. You are guaranteed to wind up causing the death of everybody involved, because there is an enormous amount of research standing between where you are now, and having enough information to create a workable design.

The real problem is that the nature of the contracting system doesn't allow for open-ended / minimum-viable projects with iterative development, you have to do your best to write the requirements then throw it over the wall and hope for the best. Of course you're going to wind up with a mess.

rishabhd• 5 years ago

Hate to say this, but having worked in similar industries, RFPs are sometimes deliberately skewed/ created in a manner that make it hard (or downright impossible) for others to participate.

bbarn• 5 years ago

A lot of the time, that's ok. A few people in the know want to avoid a lengthy and wasteful evaluation process for the product or company they already feel confident will meet their needs. Or worse, they're afraid someone might underbid, people without context will pick the lowest cost option, and the product or service will end up being a disaster.

The problem comes in when things like this happen - that a company forces itself to be the only option through unnecessary infrastructure and proprietary software in the public space.

notagoodidea• 5 years ago

That is the easiest a company can leverage an invitation to tender ad infinitum. I believe that going for the cheapest is always the wrong way to conducting those projects. On the company side, bargain you first contract, install your proprietary program, lock everyone up in it and higher your fees when you have leverage on the next requirements.

Government, public offers, invitation to tender must not only play by the cheapest but also by the long term impact that changing contractor every 4, 5 years has on their own management and system. This is getting as absurd and stupid. Force contract to use standard/boring technology and leverage a large market of companies for support. Being the cheapest never was a fair decision maker.

Aeolun10• 5 years ago

Deloitte is the only contractor that can achieve the O&M scope and contract requirements

Clearly not. Unless... the requirements were garbage to begin with?

snarf21• 5 years ago

This is a major problem with using contractors. They bill by the hour. Their continued employment is not dependent on doing a good job. They half ass everything because it is about billing and being done. Also, doing a bad job means "overruns" for more free money. The government needs a Department of Technology that runs all these systems for all national systems that are needed. This isn't like building a combat fighter, it would be straightforward to be build a team that can build a scalable website with today's technology.

info7814• 5 years ago

Good people want to work at places with stock options and bonuses for good performance, the government can not offer that. Government is more suited for people interested in management, not pure technical people

wp3816402• 5 years ago

The problem is because of government pay scales they can’t fill those digital agency roles with the right people

Within a year or two the good staff end up working at the consulting shops and influence their former employers

GizmoSwan5• 5 years ago

The problem is that any success in delivering past contracts does not guarantee success in future in software technology where a typical software technology becomes obsolete every 3 years.

In software world things change so fast that someone with 6 years of experience developing software will not have enough experience with newest tools, languages, methods, etc. Newer graduates with experience is newest applications in their resume get hired and they sometimes lack real development experience.

Building and delivering software is not like building a road or sidewalk where technology does not change by much every year or decade. They still construct sidewalks like they did say 100 years ago.

So major software contractor's success is pretty much random depending on who gets hired and how good the chosen packages are for the given application and how experienced are the staff in putting it together and having good test specs.

Another problem in software world is that software developers move to new IPOs and so the staffs are ever more transient. Government jobs with complex hiring practices are actually becoming unattractive to developers.

mgkimsal2• 5 years ago

The problem is that any success in delivering past contracts does not guarantee success in future in software technology where a typical software technology becomes obsolete every 3 years.

They don't even seem to be saying "because Deloitte was successful in the past". It's literally saying "because they're the only ones who have access to their proprietary system that we already locked ourselves in too.... they get the business".

"...so the staffs are ever more transient". Yeah, there's some idea that "company X" does ABC. The people at company X do ABC, but you are at the whim of institutional knowledge and tribal info systems within that company. The superstars who made ABC work 4 years ago may all be gone, and the current people are trying to make a name for themselves by making 'The New Process' so any success from ABC isn't at all guaranteed to translate to your new project.

martincolorado• 5 years ago

This right here “configuration of the VAMS application is occurring using Deloitte’s propriety GovConnect platform.”

Because it is so difficult to launch a project and provision the requisite infrastructure in the federal government due to various IT regs and laws and subsequently obtain an Authorization To Operate (ATO) in a timely fashion an existing platform is routinely sole sourced especially when speed is necessary.

Why? This is both legit and utter bs. Legit: the fed gov actually attempts to protect a persons’ private data. BS: after the drain of the acquisition process (led by non-technical folks), ATO, slide decks to more non-technical management (Deloitte and their peers crush this), the competent folks find other jobs either in or outside gov that free their creativity.

Positive: groups like USDS are reducing the drain, but once an exec gets burned they go back to checkbox management and shun the learning organization culture that has made tech the anti-thesis of this type of acquisition in other sectors.

*edited misspelling to show gov tries to protect not product privacy data

silexia• 5 years ago

Government will always be very inefficient. Government certainly has some strong benefits, but you should avoid using government wherever it is not absolutely necessary. Private industry would have done a far better job of handling the vaccine roll out.

ed25519FUUU• 5 years ago

Corporations love regulations. Let’s remember this when we support things like net neutrality. They always try and sneak regulatory burden in any of these bills.

Trias11• 5 years ago

Yup. Allocate budget. Hire few $10/hr offsore guys to craft some frontend in a few months. Transfer money. Deliver kickbaks. Shutdown project. Rinse and repeat.

billions3• 5 years ago

When doing a home remodel and the contractor breaks something they either return the money or fix it. Blame the CDC for not vetting the work and enforcing contracts.

tormeh• 5 years ago

Germany also has its own controversy about corona software. The government wants to standardize on SORMAS, but many local governments don't want to switch.

I expected a dump, but to be honest, SORMAS looks pretty good. It's already in use by Nigeria and Ghana, and it's on Github. It's Java EE with Postgres on Ubuntu, so safe and sane tech choices. They even have a play version: https://sormas.helmholtz-hzi.de/sormas-ui/login that also looks ok. I'm thoroughly impressed.

ex-deloitte• 5 years ago

Any insiders know the technology behind VAMS? I wonder if it is built on their ‘exalink’ platform?

I’m no longer with Deloitte, but I saw that they recently sold off their life sciences department to their only competitor- Model N [1]. The Exalink platform was born out of work being done for the life sciences department. I thought it was an interesting development that Deloitte sold them off since when I was still there it was being peddled as this great in-house cloud-agnostic system that could do whatever you wanted it to do (spoiler: it couldn’t). Anyways, maybe it is completely unrelated?

1: https://www.nasdaq.com/articles/model-n-modn-acquires-deloit...

schnable• 5 years ago

“The health-care software industry is enormous, and it exists largely because it’s privatized, it’s not standardized,” says Stone. “There are a lot of free-market inefficiencies. And the country doesn’t have a public health infrastructure, so there isn’t any real drive to fix it.”
“You think about the industries that have been transformed by technology—someone said, How do we get a pizza to your house faster? That’s a competitive advantage,” he says. “That has not happened in American health care.”

The pizza industry is not a exactly standardized industry with public infrastructure either... the failure of VAMS (and healthcare.gov before it, and the unemployment systems, etc) shows that single government solution is not a fix for broken procurement and project management.

estaseuropano3• 5 years ago

Healthcare.gov was a private sector disaster that then was fixed through a government-internal 'startup'.

starik36• 5 years ago

The ridiculous expense of software in the health system is endemic. It’s a shame it’s been allowed to go on for so long.

A write up on why it’s so expensive from a standpoint of a developer. https://maxwelljordan.medium.com/why-healthcare-in-us-is-so-...

bit_logic• 5 years ago

Does anyone know if this was yet another React website disaster? This is exactly the kind of website that should be built with reliable technologies. You don't need fancy animations or whatever junk the SPA style sites have. Just old style server side frameworks, minimal Javascript, standards supported HTML and CSS. The article mentioned Internet Explorer as an issue? Cross browser support is no issue if you stick to the old standard HTML and CSS and minimize use of Javascript. Mobile support? That's just loading the right CSS library. The goal here isn't "engagement" and all the other shit SPA style sites optimize for with their heavy JS tracking and advertisement code that makes the laptop fan run. The goal here is a simple reliable CRUD website for scheduling appointments, the previous generation of web development tech stack is perfect for this.

duxup• 5 years ago

Usually these big debacles have more to do with the back end, integration of legacy systems, and etc than front end frameworks / choices.

userbinator2• 5 years ago

No kidding --- the fact that they mentioned it only worked in Chrome is a sad commentary of the state of the Internet and what Google's effective monopoly has lead to.

info7814• 5 years ago

We had to live with sites than only ran in IE for years.

GizmoSwan5• 5 years ago

Not surprising to me. It was probably contracted out to multiple subcontractors with heavy top down design and review process. Each module is reviewed but the reviews can not locate the bugs.

Sometimes they hire junior programmers who would have the latest stuff in their resumes but the kids are into copying and pasting stuff from internet not knowing exactly what the code does and they don't have enough experience producing clean and bug free code themselves. Other times the top/down design produces code that lacks elegance and resembles what truck drivers would produce with too many people working on it and no one responsible for the whole thing.

Have seen that before. Not sure about this case.

amluto2• 5 years ago

I find myself wondering why much technology is needed. If I were running a mass vaccination campaign, I would approach it like this:

First, I would create extremely fine grained but easy to check eligibility tiers. Going by date of birth would be the most basic solution. At any given time, vaccinations are open to people born on or before a given date. [0]

Then, I would set up real mass vaccination sites. None of this almost-all-vaccines-come-from-big-medical providers nonsense we have in most of California. The CA scheme is fundamentally inequitable -- I got an email from a medical system that told me that they would email me when it was my turn if I had seen a primary care provider in the last three years or had two specialist appointments in the last three years. This fundamentally discriminates against people without adequate insurance who haven't seen a doctor in a major medical system recently.

Mass vaccination sites would be open to anyone with a date of birth <= one threshold on a walk/drive-in basis with no appointment and with a different threshold for appointments. Optionally there would be another threshold for people who want to hang around and see if they can get an extra shot at the end of the day. I would adjust the thresholds as needed to keep occupancy at the level I want it. This requires minimal IT infrastructure.

I would let major clinics administer vaccines with whatever appointment system they want, with some rules. First, the clinics may only vaccinate people who have been eligible at the mass vaccination sites for some amount of time, say two weeks. So if you want your chance first, you go to a mass vaccination site -- clinics are just for convenience. Second, the clinics only get more vaccines when they are administering an adequate amount of the vaccines they have already received.

I would track people who have been vaccinated with a database with no integrity constraints whatsoever. Each site submits a trivial machine readable form with a list of people they have vaccinated. This list gets imported to a database. Dealing with all the inevitable issues is not permitted to block actually giving vaccines. [1]

Finally, everyone who gets their first dose gets a printed (or hand-written!) card with the date of their next vaccine and a reference number that can be used to look up the card in the database. This reference number is (date, site, line at site, sequence number) -- these are supposed to match up with the end-of-day logs, but they require no central coordination. [2]

If you have a card for a second shot, it is valid at any site once it's the right time.

Damnit, this whole scheme could be done entirely without computers, and that's how it should be. It doesn't even need two-way communication -- there could literally be a section of the newspaper saying "if you were born before m/d/y, come to Dodger Stadium or Levi Field or ... and get your vaccine, no appointment needed. If you were born before m2/d2/y2, you may call this number of visit this website to make an appointment." If the whole IT infrastructure goes down, then it will be annoying but everything will continue to work.

[0] There are many possible improvements, but they should be kept simple. For example, people born on or before a given date or people with a doctor's note explaining that they have some risk factor.

[1] Pay some contract development firm to develop a nice app that scans an ID card, does all the data entry, and prints out a receipt. The app has one primary requirement: it does not print a receipt until the server acknowledges receipt of the request and commits the transaction. But make sure that, even if the app fails completely, everyone at the clinic knows how to use a piece of paper and a pen to do the same job. Or a laptop running Excel or LibreOffice -- there is no shortage of laptops around. Or a Google Sheet, etc.

[2] Of course you can cheat, but most people won't. And the logs will mostly reveal the cheaters.

antipaul2• 5 years ago

The small problem with your kind of approach is that it would make the army of administrators and contractors like “Deloitte.” quite redundant, and then they’d actually have to do non-Bullshit Jobs!

amluto2• 5 years ago

Nah, they can just do nothing at all or bid on other unrelated contracts.

Sometimes people get so caught up with new technology that they forgot about old technology, and technology predates computers. Believe it or not, it wasn’t that long ago that you could go to stores, buy things, eat at restaurants, and even fill up your fuel tank without directly interacting with any form of computer. Heck, it hasn’t been that long since you could fly on Southwest Airlines with little reusable plastic boarding passes that the agents would hang out in exchange for a paper ticket. These systems were imperfect, but they worked and were extremely reliable.

j4510• 5 years ago

This is what happens when accounting firms, or worse management consulting firms trying to be in technology.

When those groups talk about innovation and disruption, I often run.

The next time your government awards a tech project to tech illiterate smooth talkers, keep this in the back pocket.

jnathsf• 5 years ago

Deloitte is using salesforce for the underlying technology of the system. So almost all of this work is configuration and maybe some customization. We worked with another state that almost selected Deloitte but they bombed the no cost bake off and ultimately chose another vendor which completed 80% of the work during the bake off. This approach lowers risk dramatically and you’re able to set an aggressive and a more accurate launch date.

I hope we see more govts adopting bake-offs for buying IT.

tootie2• 5 years ago

They probably got sold on it being "off the shelf" and just a matter of customization and not coding. But the minute they run into edge cases (of which there are likely hundreds given the nature of the product) they just end up stuck.

ianhawes• 5 years ago

"COTS software" (commercial off-the-shelf) is the biggest meme in government contracting.

I recall reading a 300+ page PDF RFP for a DMV information management system. The state wanted to convert their legacy VB5(!) system to a modern solution and preferred a COTS solution. I am fairly certain there are no COTS software solutions for a statewide DMV that also meets the legislative requirements of the state.

jbm• 5 years ago

One of the vendors I had the "pleasure" of working for re-implemented Salesforces' approval system in APEX due to an obscenely minor requirement that should have been pushed back on.

It's completely believable that Deloitte might have built a handful of boondoggles like that and dressed it up as a "Salesforce" project.

pickpuck• 5 years ago

These are mostly front-end issues.

Software is often judged based on these UI/UX nuances, but the complexity of building a competent front-end system remains an undervalued part of the process.

The front-end team is usually (1) brought in too late in the game to provide effective feedback and (2) not given enough time to build.

When there's not front-end representation at early design reviews or at initial discussions on how APIs should work, then it's easy to forget things like form error handling.

Designs don't account for all possible states, services/APIs are in flux, until the very end.

Everything else is late, and then all of it has to get slapped together with duct tape and shipped in a few weeks. Performance and browser support be damned.

justinzollars• 5 years ago

abandoned due to bugs

That can be translated into a new corrupt administration is replacing the old corrupt administration and will spend an additional 44 million on a new website built by their own politically connected friends.

Ericson2314• 5 years ago

I would check out https://pedestrianobservations.com/ for information on other but analogous cost disease.

Basically, no, it's not just a meme. American managerial culture (or even Anglosphere at large, as alleged there) really does suck. And no amount of crafty mechanism design gets around the fact that least exploitable system seems to be one where the civil service isn't maimed and the work is done in-house.

Basically government should fund some free software for various general purposes (or just do a decent UBI), and then any "consulting" should basically be public sector and free software sector exchanging best practices.

The higher end academic consulting (ARPA) is legit[1] and should still exist, but it shouldn't be defense specific, and like drug bounties basically should exist to smooth over the "research...development...product" gap --- development is severely sort-changed current since outside of DARPA and ARPA-E so much research that doesn't contribute to someone's rent-seeking agenda just lies wasted.

[1]: I suppose should say I once was an intern at https://galois.com/

MangoCoffee• 5 years ago

this happened to smaller/mid size company as well.

a long time ago i work for a mid size company with a small division that sell specialized software build with Delphi.

its a desktop app. when i got hired by the company, they outsourced it to Capgemini. after 3 years, they still can't get the app convert to .Net. (that's the goal) by the time i got out the company. they changed to another more local company instead of big outsource firm like Capgemini. they still haven't got it all work out and the new company they hired throw away all the code done by Capgemini. the whole project was estimate to be 2 million by Capgemini.

S_A_P• 5 years ago

In my field (ETRM) I’ve worked with Deloitte on a couple of occasions. I have seen this sort of nonsense on more than one occasion.

A specific example that comes to mind was a project a few years back where Deloitte stuck 4 junior consultants along with 5 senior consultants and spent a couple million dollars for 12 months and were unable to produce even one business scenario in the system of choice. They did however produce reams of documents showing how it should work. The software vendor was called back onsite and I happened to be in the right place at the right time. A team of 3 of us ended up doing full implementation in 5 months. That included about 20 scenarios from front to back office along with foreign business.

moron4hire• 5 years ago

The problem with Deloitte is that it's massive and organized as a partnership. A massive company requires several layers of management to be able to organize, and a partnership expects billable hours out of everyone. So you have this massive, sickly, distended torso of middle-management whose only purpose in life is to recieve decrees from on high and push the thumb down on the people doing the actual work. And the only way they keep their jobs is to figure out how to stick their hours on the client's bill.

So you quickly see large numbers of leads, directors, managing directors, managing partners, and partners show up into meetings with 1 or 2 SMEs to make a show of "managing the project", whose only purpose in life is to suck the oxygen out of the room.

And I mean literally suck the oxygen out of a room. The meeting rooms are tiny--cuz real estate isn't billable!--so when you get more people than chairs in a room like that, the CO2 levels literally start to put you to sleep.

I spent a year and a half at Deloitte, building Augmented Reality projects. Every single project I worked, I was only brought in during the last 10% of the project time. The time was always not enough to do a good job. The one project that I cared about--incidentally also the only one that got any traction with any actual clients--got canned during a restructuring from working within Deloitte's "government" division to it's "industry" division. Except we always worked blended across industries and the app was targeted at industry, not gov. To this day, I really think the entire point was to just waste money, and my app getting traction edged too dangerously close to one day having to show efficacy.

I'm now the head of VR (yeah, I never actually wanted to do AR, and didn't even get hired for it, but that's where Deloitte stuck me) at a small company that does foreign language instruction for gov. A year and a half in and we've got an exciting project that has tested well with some limited number of students and is soon deploying company-wide. Barring some natural disaster, I'm extremely happy to never change jobs again.

alexpetralia• 5 years ago

At some point, someone should compile a bullet point list of all these consulting train wrecks on a simple website, like www.consultingfails.com.

Then, whenever you hear another multi-million dollar catastrophe, add it to the list. The brand damage will accrue - as it should - over time.

MattGaiser• 5 years ago

IBM has a long list. Doesn't matter.

seshagiric2• 5 years ago

Consulting can be a boon or a curse.

This case it was a government client, and one could argue there is no singular point of authority/ responsibility that may have caught on to the troubles early on. However I have also it happen in private companies where 350-500$/hr 'consultants't do not provide anything more than code lifted from code samples or community tech forums.

GizmoSwan5• 5 years ago

They should have funded it using Gamestop stock. Fraction of fraction of its one day gain would have paid for it.

victor106• 5 years ago

This makes you realize the state of IT in 2021.

Humanity can make an mRNA vaccine in 6 months. That's literally (almost) instructing your cells to make a particular protein.

But one of the largest consulting company in the world cannot develop a freaking website.

codpiece2• 5 years ago

Maybe it's me, but I was quite happy with my VAMS experience. CDC notified me by email that I was eligible. My data was pre-populated. Signed up for an appointment quickly and got messages reminding me of appointment.

Considering that this came out of nowhere, I was impressed how quickly and easily I got a vaccine. Big thanks to the Army who ran the vaccination site.

rainyMammoth• 5 years ago

I have worked with Deloitte type consultant in the past. It was exactly what I was expecting. A couple 20-something dressed up in suits, really good at talking and bullshitting high level visions.

Very little technical knowledge overall. It was a disaster.

throwarayes• 5 years ago

Having seen Federal Govt contracting it’s easy to see how this works. And it’s a 2 way street of dysfunction:

- Many fed agencies have little tech savvy or ability to evaluate solutions

- Feds can have complex, mandated processes for software procurement, development and deployment that must be followed. This drives up the cost and limits the companies that choose to bid.

- There’s not much of a “get sh*t done” attitude. There’s little consequence for the individuals that managed a failure on both sides. Usually “oh well, another day at the office!” is the norm for many federal agencies/contractors.

- buzzword chasing tech ignorant administrators that consulting firms readily take advantage of - “yes! But does it have machine learning!?”

- there’s many fiefs at these agencies that must be navigated: “Oh! Wow - you spent 6 months building and it’s time to deploy? The security team really hates CYZ tech and we’ll have to escalate this to the director”

- so many g-d stakeholder to please and woo: from managers to administrators to political appointees and congresspeople. The relationship cost can be staggering.

These stereotypes aren’t always the case - some govt groups can actually execute really well. But many of these points dominate sadly.

joncrane• 5 years ago

Wow - you spent 6 months building and it’s time to deploy? The security team really hates CYZ tech and we’ll have to escalate this to the director

This right here! The "cyber" folks always throw a wrench in the works at the last moment. They rule by FUD and also have literally ZERO idea how stuff works. There's an old saying about teaching, but these days, and especially in government contracting, it applies to "cybersecurity."

"Those who can, do. Those who can't go into security."

kangarooglacier• 5 years ago

It's easy to bash poor work by Deloitte (rightfully so), but this reflects decades of poor decisions by the Federal government. The number of Federal employees has remained constant for 30 years so that politicians can claim the Federal government isn't expanding; but the budget has ballooned. The balance all goes to contractors. They need Deloitte because they have lost the capability to do anything themselves.

To make a sweeping generalization which doesn't give enough credit to some amazing groups, the Federal government does not have people who are capable of managing complex IT projects. The government has outsourced so much work that it no longer has personnel who are capable of knowing what is going on. Federal workers are competent in managing contracts, but not actual work. So when you arrive at a new and complex project (like widely distributing a vaccine for an unforeseen global pandemic), they don't know what to do. How do you write that in a contract and enforce it?

That doesn't even begin to get in to the requirements around what it takes to launch a production IT system while meeting government cyber security requirements/processes. A lot of comments here are lamenting how easy this is. Let me set it straight: you cannot launch a random Django/Rails MVP in the government. They don’t operate that way, for reasons good and bad. The fact that Deloitte had something in production in under a year, no matter how buggy, is a minor miracle.

mherrmann• 5 years ago

I just experienced something similar with Accenture. They were hired by the state of Bavaria in Germany to build an appointment management system for Covid vaccines. They didn't get it done in time and then it sucked so badly that the doctors didn't want to use it.

They now use my tiny little business's [1] solution instead. I've been on the market since 2014 and they found me through my web site. They called on a Friday "we need the system on Monday". Signed the contract in the evening, set everything up over the weekend, and now they're loving it. Sometimes, I think these big organizations are just so overloaded with bureaucracy and politics that a little guy like me can compete on pure product quality.

[1] https://www.indiehackers.com/interview/terminerinnerung-56e9...

eigenvector2• 5 years ago

The Canadian government paid Deloitte $16M for exactly the same thing, with a deadline of January 31 to get it running.

https://www.ctvnews.ca/politics/ottawa-gives-accounting-firm...

One of the factors cited in awarding the contract was Deloitte's "experience" doing the same thing in the USA.

http://www.canadian-accountant.com/content/business/why-delo...

pts_• 5 years ago

Knew a guy who joined Deloitte. Was called star performer when he was focused on beating others down instead of the work at hand. You hire careerists, you get wall street brokers for every job, including existentially challenging ones.

odyssey7• 5 years ago

A $44M scapegoat service?

throw7• 5 years ago

"It won’t work on Internet Explorer; it only works in Chrome."

Sometimes I wonder if flat earthers are right... we indeed live in clown world. A sucker born every minute.

jandrewrogers• 5 years ago

I’ve done a lot of work for multiple national governments on multiple projects. There are two facts that can be true at the same time:

- The government often grotesquely overpays for mediocre product delivered by vendors without holding them accountable in a meaningful way

- The government process drives up costs to an insane level independent of implementor good faith efforts and strongly selects for companies good at the process rather than execution

These are two different problems, and both of them manifest with any large contract. I would make the observation that it is easier to hide mediocre product and execution when you can hide behind a grotesquely inefficient and mediocre government process.

What transpired here should never happen. But I know why and how it happened.

pulse7• 5 years ago

IBM was always good at understanding these "government processes", while Apple always struggled with them... and kept up with what they understand very good: consumer buying/selection processes... :-)

jpxk• 5 years ago

I just started reading 'Skin in the Game' by Nassim Taleb. These kind of overpriced consulting contracts, where nobody in particular bears the downside risk of incompetence, fit into his framework well. Hiring Deloitte et al usually amounts to outsourcing responsibility, except the people who do the work also bear little or no long-term responsibility if things go bad. A 3-person startup, on the other hand, suffers from severe downside (at least in a business sense) if they mess something like this up.

Raymondfx• 5 years ago

Agile vs Waterfall Score 10 - 0

paulie_a• 5 years ago

Why does the government contract with deloitte? They seem to suck at everything they do

jspaetzel2• 5 years ago

The worst part of all of this is that the CDC didn't already have this type of tool in place, years ago, before a pandemic even hit. This is the type of basic thing they should have already in place without any new spending.

It doesn't even need to sit around unused. They should be using it every year for flu shot distribution or coordinating other types of vaccines which are normally in abundant supply.

thomasfl• 5 years ago

25 years ago, websites like this was seen as something of a novelty. They were made just as much as an experiment as something necessary. As a young web developer at that time, I often had to do all the programming, ux design and graphic design myself. There wasn`t any budget for more people. Full stack development wasn`t a thing yet. I still get amazed by how well staffed and funded web development projects have become. Web development has become more complicated too. More animations, accessability and higher security all adds up to making the whole project being more time consuming.

j_barbossa• 5 years ago

The contracts claim the company was the only “responsible source” to build the tool.

Looks like the CEO of Deloitte and the director of CDC played a few rounds of golf together before...

peter_retief• 5 years ago

I am very cynically about large sw development projects. They are very political, people who do well are "well connected" The client wants to mitigate the risk so falls into the trap of legal and technology word salad served up by the vendor. The actual work is often done by junior developers who dont have the experience to spot potential failures or often too scared to point out system flaws. One could write a book on the failures of these big projects, would make depressing reading though.

mns06• 5 years ago

Does anyone know what the system architecture actually is?

habosa• 5 years ago

Deloitte's habits of stealing tens of millions of dollars to deliver broken technology to the government are so deeply ingrained they couldn't even stop themselves from taking this critical assignment. People are going to die because of this, there's no way around that.

cblconfederate• 5 years ago

I will install wordpress for them for $43M

varispeed• 5 years ago

These big contracting companies ganged up in the UK and persuaded the government to change the law so that small independent contractors are no longer competitive (for example they essentially cannot deduct any business costs and have to draw entire contract revenue as salary, that means no funding of other business avenues) and at the current rates such contractors have to pay over 50% in tax, whereas companies like Deloitte not only don't have such costs, they can also hire such contractors as employees without rights. It's a scandal that is going to blow up later this year when the law goes live in April. Similar measures were already introduced in public sector, which caused mass exodus of small firms.

mkl95• 5 years ago

Deloitte is well known among devs in my country for the poor working conditions of their employees (including unpaid overtime and low pay) and the overpriced contracts they sign with public bodies. They do things like charging €2M to build a broken CRUD app written by devs fresh out of uni. They then follow up with a "maintenance contract" that consists of not doing much to fix it at all, and replacing the people who are not fresh out of uni anymore with cheaper devs.

p0d• 5 years ago

Ok government, give Deloitte the contract in a crisis. Pay them a year after they deliver. They are not short of money. Penalise them for not delivering.

Or alternatively spend a few billion making the government more business savy. It will probably save you money.

Then again maybe not. That will be another few billion going to Deloitte.

rmason• 5 years ago

If I was to guess the CDC wrote poor specs. Probably never talked at length to a variety of different county health departments to understand their needs. Never had a persona written so they didn't understand the need for IE or that people might be using it on a mobile phone.

Deloitte just coded the spec, cashed their check and walked away. Deloitte probably outsourced development overseas.

I know some smaller government contractors in the D.C area who would have done it for a third as much and done some of the CDC's work for them when they saw that it wasn't done.

bitcharmer3• 5 years ago

No. Big consultancies are hired to - among other things - capture the requirements (simply because many of their clients don't know how to do it properly). But they intentionally do it in a form that lets them over-charge and under-deliver. Seen it too many times.

AdrianB1• 5 years ago

I have 2 close friends that work or were working there. One was an application key user in a different company, he was hired as a deployment manager for that app for more than double the pay; I am happy for him, but he knows he is not a manager.

The other one was hired on a position, asked to do something completely different he was not qualified for, the work hours were around 70-80 per week (no overtime paid), zero management implication so he left after a few months. They went in panic mode, offered him double the pay and less work hours, he refused to stay there so they become aggressive.

They use to pay above the market for whatever they can hire, but the chaos inside is making any competent people leave asap, so they pay a lot to retain whatever they can, not the experts. It is just a bubble with nothing than hot air inside.

codecamper• 5 years ago

I am working on something now. Almost ready. I wrote the very first Taxi app on the iPhone ages ago. And Scanner911.

may4m• 5 years ago

Deloitte was part of the destruction of South Africa's state energy company

zxcvbn4038• 5 years ago

The classic “Must have caught ten pound trout in Colorado river during August 1984” method.

Deloitte’s name seems to come up a lot when there is a multi-dozen million dollar web site that is falling over. I wonder if it was a former Deloitte person who wrote the Parler backend because it seems to be the same sort of massive footprint that falls over when the wind blows type setup.

achow• 5 years ago

‘We are very sorry’: Deloitte apologizes to RI about UHIP (Unified Health Infrastructure Project)

https://www.wpri.com/target-12/we-are-very-sorry-deloitte-ap...

dalbasal2• 5 years ago

Government contractor cynicism is one of the saddest games plaguing the modern world. It literally guarantees the inability of governments to have decent, working technology.

IDK what the structural solution to this is, but it is horrible that Deloitte will walk away from this unscathed. In fact, there doesn't seem to be any financial incentive for them to deliver usable software.

Imagine screwing over a client that does this much business with you.

christophilus• 5 years ago

The government really needs to start insourcing its IT. I worked briefly as a contractor, caught a glimpse of how the sausage was made, and left within a month or two. I’ll never do it again. I can’t imagine insourcing producing worse outcomes, and the end result would probably be less costly, even if it was still low quality.

consultSKI• 5 years ago

It’s called Payola.

cromwellian• 5 years ago

Beltway Bandits win again. Literally the worse kind of grift and project paralysis. I worked on government contracts in the 90s, and the process is so full of grifters and people over billing, companies selling software licenses far above market rate with supposed levels of support and reliability that are actually no better, or worse, that I wouldn’t recommend anyone work in that sector.

Oh and all of this talk about 95% of the work being understanding the problem or integrating with legacy government systems, gimme a break. Every time I’ve seen someone purport to integrate with existing government systems it’s always been a buggy disaster and hideous from the user perspective.

This is the same kind of rot that infects a lot of enterprise software. Addiction to hyper complexity, too many layers, too many points of failure.

Just how complicated does a vaccine management system need to be? If you’re adding more and more requirements, it’s not software that’s the problem, it’s the requirements that the vaccine distribution is being made more complicated than it is needs to be.

I think at this point, a random distribution would be more effective than the complicated rollout they’re trying.

js2• 5 years ago

I thought we have 18F and USDS to avoid this sort of situation:

https://www.washingtonpost.com/news/the-switch/wp/2014/08/11...

Various discussions over the years:

https://hn.algolia.com/?q=healthcare.gov

wheaties• 5 years ago

"it only works in Chrome"

Thanks, I hate it. I use Firefox. This 50% of my time on the internet.

info7814• 5 years ago

These big companies always have a lot of overhead. The middle layers of management are useless. Often the government will try to work with smaller companies that are under 50 people. However, if these small companies land good contracts the big companies will buy them up.

sfc32• 5 years ago

What's it written in?

1cvmask2• 5 years ago

Reminds me of when Hertz sued Accenture for a website that failed after spending $32 million on it:

https://www.theregister.com/2019/04/23/hertz_accenture_lawsu...

temporallobe• 5 years ago

This is not atypical for US government projects. In 2012 I was brought in to “rescue” the UI for a huge multi-contractor system that was supposed to replace an aging legacy system written 40+ years ago.

I worked with the architect tirelessly for almost 3 years, trying to implement various changes but at its core the new system was highly flawed and suffered from various team changes and reboots. The UI/UX “expert” was not a developer and had very little technical background, and was therefore unfamiliar with basic HTML and CSS concepts. For some reason they let end-users dictate style and behavior aspects of the application. For example, the legacy system consisted of green-screen terminals and all the associated keystrokes (think ancient POS systems you still see in some retail stores). The vocal minority of these end-users demanded that we replicate this interface exactly instead of building it from the ground up as a modern best-practices UI using the latest web idioms and paradigms.

It gets worse. At some point someone important said Javascript is evil, and we were to avoid using it as much as possible, despite the application using several embedded and third-party Javascript libraries.

I ended up leaving that project for a slightly less insane one, but the pay was good. Anyway, I heard about a year later that the entire project (a 10-year effort that cost tens of millions of dollars) was abandoned. They continued using their greenscreens but I heard they hired a company to build an interface that reads these screens and presents a modern HTML facade over there screens that operates them kinda like a proxy.

You can’t make this stuff up.

curtis3389• 5 years ago

I tend to side with that vocal minority on UI design.

I remember watching my friend who was a manager at a Domino's back in the day place an order for a pizza with their old POS system that was driven by key strokes, and it was ridiculously fast and easy. A pizza was a list of characters that described it.

Fast forward a decade, and I was tapping away at a touch-screen POS system for a smaller pizza chain, and it was woefully slow in comparison for even the most basic pizzas, and more error prone because Anchovies and Artichokes were right beside each other.

If you ever have a user-base where time=money, they will want hotkeys for everything.

nojvek• 5 years ago

It baffles me how bad non-SaaS software tends to be. Software written for non-tech folks. We have a 100 shiny JIRA clones but some niche software for nurses has awful and slow UX.

Everytime I’ve gone to a doctor, they’re using a different kind of software portal. Many claim its an expensive software the hospital bought but doesn’t do what they want so they write things on paper instead.

This is one of my fears as a dev. I have dev problems that I know how to fix but I’m blind to the universe of other problems that I could potentially fix and deliver huge value. I’d really love to learn more about other people’s niche problems that can be solved by software.

diogenescynic• 5 years ago

Deloitte is a horrible consulting company. They designed an awful financial system implementation at the company I worked at and we just kept giving them more money and they kept bringing in more entry-level consultants. They even billed us multiple times for work we did ourselves... I thought it was also shady that one of our department heads was married to the Deloitte partner who led that practice area.

martin-t• 5 years ago

Incompetence is a source of extreme inefficiency in modern society. People making decisions don't understand the things they're deciding. This is broken. And there's nobody sane in a position where they can say "STOP, this is dumb" and demote people to the level of their competence or fire them.

A harmless example is movie / tv show plots not making sense - there's nobody willing to call out bullshit like obvious plot holes. This is how GoT went from a masterpiece of scheming characters driven by their motivation, making decisions based on information they had and dealing with the consequences to "she kinda forgot".

A dangerous example is the government being full of people who are too incompetent, stupid or useless to get a real job. Sometimes the number of them in the system is so large that competent people can't get anything done to fix things.

We need to do something to fix this. Positions of decision making power should require an IQ and/or knowledge test.

fasteddie31003• 5 years ago

The levels of incompetence in government is embarrassing. The government definitely does not attract the best and brightest. However, they control ~30% of the US GDP. I need to make a website that list all the government fails. Looks like http://usagovernmentfails.com/ is available.

tekkk• 5 years ago

I agree with your point but i would also want to point out this has been a problem since the dawn of man. Governments and organizations have been plagued by unending bureucracy and corruption that grows worse the longer it has been in existence. Humans have a tendency to protect their belongings or ego more than altruistically help others so it makes sense.

Democracy has, as a general idea, been a way to solve this problem by creating a counterbalance that should prevent important decisionmakers from being overly selfish and at least show they are being effective and altruistic. But that system isnt really applicable for smale scale power dynamics. Unless efficiency, or in some cases having just common sense, is not monitored by anyone aka. there are no consequences for making poor decisions sooner or later no one will pay attention and they will be doing whatever they _think_ is best (or easiest) but without reality checks.

But how to evaluate performance when dealing with abstract things? At least in the case of GoT the ratings did deteriorate so one could point out a problem in the system (albeit way too late, IMO s6 was when it should have been fixed). In other cases no one can see the damage they are doing (especially if they have never been really doing anything) so how to even tell it needs to be fixed? I think best cure to this problem is a great culture where being efficient is praised and rewarded. But changing already terrible culture, well, that's tough...

(Interesting film on this matter is Ikiru by Kurosawa)

pixel_tracing• 5 years ago

So can people sue Deloitte or the government for their tax money back?

TruffleLabs• 5 years ago

Sadly, the Mac SE pictured in the article is being drug through all this ;)

daanlo• 5 years ago

Imho this isn‘t all about the procurement process being broken. There must be something additinal broken that caused deloitte to fail at delivering this project.

fireeyed• 5 years ago

“Govconnect powered by Salesforce”. How many licenses did the Federal government buy from Salesforce either through lipstick on a pig resellers like Deloitte or directly. You scratch my back, I will scratch yours.

wiseleo• 5 years ago

A large portion of legacy systems integration can be solved by asking the question “which data must be updated in real-time and which can wait for batch processing?”

Screen scrape data from legacy terminal apps, export into SQL middleware, have the new app interact with the SQL middleware via modern APIs. All legacy data gets batch updated back into legacy systems, all new data gets stored and accessed in middleware. Eventually, the middleware becomes the primary data source and the entity can save on the electric bill.

menomatter• 5 years ago

I see this in big companies as well. I work for big pharma and 20-40 web projects are not unheard of. The dysfunction is not one sided. There are way too many people involved from the Big company that make it impossible to move fast allow engineers to do what's right. Instead it quickly turns in the Phinoix Project

wakahiu• 5 years ago

Well if it helps anyone:

https://timelyhealth.co/

CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

Related Stories