NYC mayor demands $600M refund on software project

http://www.itworld.com/software/179037/nyc-mayor-demands-600m-refund-software-project

by abennett • 15 years ago

102 73 15 years ago

bluedanieru2• 15 years ago

Michael Bloomberg might have a point, but he really isn't in a strong position to complain about shitty overpriced software. Glass houses and all that.

DevX101• 15 years ago

He's in the perfect position to complain.

The role of Bloomberg the businessman is independent of Bloomberg the public servant. He can sell overpriced software as a businessman all he wants, but once he becomes mayor his responsibility is as a steward of the public's money and to combat ridiculous overpriced projects such as mentioned here.

hvs2• 15 years ago

I don't think his complaint is so much about shitty software as it is about criminal activities relating to its creation. Carl Bell (the executive in charge of the project) plead guilty to approving charges from a contractor for years when he wasn't even on the project. I think Bloomberg has a strong argument that there was fraud going on here.

tkahn62• 15 years ago

What position are you in to declare the Bloomberg terminal 'shitty overpriced software'?

mdda• 15 years ago

Just so people can put this in perspective, a Bloomberg terminal license costs approx $1750 per month, with a two year contract. The software basically gives users a fairly raw interface to financial data, analytics and an email/IM system.

In a sales call, the Bloomberg sales reps will demonstrate endless functionality of the terminal : there's a huge amount there. The dirty secret is, though, that most people use only 4 different functions (different people, different functions, of course). The big switching cost is that the email/IM is entirely proprietary : Everyone has to have a BBg terminal, because everyone else has one.

The software is probably as well written as you would expect from a company in a monopoly position. One thing in their favour : If they fundamentally changed the way the 'MSG' button worked, 50% of their users would complain.

bluedanieru2• 15 years ago

I use it and know how much it costs?

wglb• 15 years ago

The point of a bloomberg terminal is not the software, it is the data. The monthly fee is for a one-user license of the data that they provide, and it is quite vast.

byrneseyeview• 15 years ago

If you're talking cost versus revenue derived, Bloomberg's software may be some of the cheapest ever produced.

kenjackson2• 15 years ago

CityTime was launched in 2003 at a budget of $63 million, but costs swelled dramatically as the project stumbled along for nearly a decade.

From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.

jrwoodruff• 15 years ago

In these types of projects it seems like no one wants to point the finger. Once costs have doubled, or tripled, everyone is afraid of taking the blame for wasting all that money. So they keep 'working with' the vendor.

Clearly fraud and kickbacks kept this stumbling along much longer than a normal failing IT project would. I'm not sure why or how an expense like that on an overage like that on a major payroll system would not be on the mayor's daily agenda. The right people must have been getting those kickbacks.

bmelton4• 15 years ago

I've been on the wrong end of a couple of projects like this, and in my experience (not necessarily indicative of this case) the customer can be VERY at fault.

I've been dragged into 'feature acceptance meetings' that literally dragged out for weeks when testing of a new chat component devolved into a days-long discussion on whether the graphic in the header (provided by the customer) was appropriate. The unspoken issue proved to be that they didn't want us to use 'barren trees' in the picture of their building. We switched the photo used from one taken in winter to one taken in summer. This took something like 400 man-hours to resolve, after which we achieved signoff, but at the expense of having met budget.

Another instance on the same product was feature creep to the extreme. We had originally won a firm-fixed contract, meaning we would build products a, b and c for x dollars, and if it cost us more, it came out of our pocket. If it cost us less, we pocketed the profits. The customer decided, on day 1, that they actually wanted a lot more features than were described in the project scope defined, and that we won a legally binding contract to perform. Out of accomodation, we changed their product to a time-and-materials contract, meaning we had a fixed markup on labor and a minimal markup on hardware / tangibles. This then involved them dragging the project out for years longer than anticipated and signing invoices all the way, then griping at us on the budget near project completion and asking for a bunch of free labor (which is illegal under their new contract). The problem in that case was that the project management and senior execs changed repeatedly. We were at first working with their Office of Science and Technology, then with Information Services, then with Public Affairs. Each of these department changes involved swapping our equipment out so that we could have appropriately charged-back equipment to the appropriate department, which meant weeks of non-work. Each department had a different spin on what they wanted out of the project. We spent 8 months and thousands of man-hours implementing a very elegant services oriented architecture that could surface all of their legacy systems to a single web portal. Then we changed departments, and they literally told us to turn that feature off, and instead focus on a visual redesign. We implemented a single sign on service that we integrated with our product, but could be extended to work with literally every system in their enterprise. It was abandoned due to a department change.

During the first phase of the project, we'd worked with the customer to build out their data center. They wanted a new data center because the division that owned the project was going to be adding a bunch of other programs to integrate with what we were building. They spent over a year on construction, converting a service garage into a data center. They bought millions of dollars worth of EMC storage hardware and Cisco routers. Then they realized that they didn't have any budget planned for out-year maintenance, so they moved the venue. That was somewhere in the realm of $50 million dollars that was literally just thrown away, as the last I heard, they were re-converting the data center back into a garage.

In short, government contractors catch a lot of flack, because they control the visibility of these things. The contractors often can't speak to details of these projects due to security clearances or NDAs, and at the end of the day, the government is always happy to blame the vendor. It's well practiced, and designed with a variety of scapegoat clauses built in.

protagonist_h2• 15 years ago

what? it took 400 man hours to replace a logo?

zaidf2• 15 years ago

Wow, sounds a lot like our university's tech overhaul, especially the single signon stuff.

josephcooney• 15 years ago

This plus 10. The government projects I've worked on were all like this on a macro and micro level.

joe_the_user• 15 years ago

Great stories...

... it's just the article notes a subcontractor has already been convicted of fraud and kickbacks, so this might not have been a project quite like the projects you describe (though if this is a Terry Childs-type case, I'd love to hear about it).

bugsy2• 15 years ago

What you describe is not really the "wrong end", that is the standard way that projects are done by government.

cheez• 15 years ago

I think there were kickbacks.

bane3• 15 years ago

Actually the final bill is going to be north of $700m.

InclinedPlane2• 15 years ago

That's only about a 30% growth per year though, which may not seem so ridiculous on a year-over-year basis. I can see how such a mistake could be made, though it is pretty crazy.

bugsy2• 15 years ago

This is a common scenario. Government picks the lowest bidder, but the price can go up every time there is a change in specs. Then every government agency imaginable wants to decide what color to paint the shed and the whole thing bogs down, sometimes permanently. It's amazing that on this project they eventually managed to deliver a working system.

hvs2• 15 years ago

Some background on this:

Mark Mazer, CityTime consultant accused of scamming millions, has checkered past of investigations http://articles.nydailynews.com/2010-12-23/local/29441786_1_...

CityTime Crook Pleads Guilty As Defendant List Grows http://gothamist.com/2011/02/11/citytime_crook_pleads_guilty...

Mark Mazer Indictment [scribd] http://www.scribd.com/doc/58405089/Mazer-Mark-Et-Al-S2-Indic...

thematt• 15 years ago

Although New York has "received a working system that will advance our management ability ... because the project was apparently tainted by fraud and kickback schemes, the City must be made whole"

...

The city is also planning to launch a request for information (RFI) process in January in search of alternative timekeeping systems.

Wow. So they eventually got a working system. Now they're throwing it away and going to purchase a new one? Meanwhile turning 83 contractors into FTE's to maintain the existing system? This seems like complete craziness for something as commoditized as timekeeping software.

kmfrk• 15 years ago

This is fairly common; a decent, working solution already exists, but (the) government wants to do its own instead of adopting the existing system, and the bills begin piling up.

bradleyland• 15 years ago

It would seem that way, but this is municipal government, so there is likely local code (law) that imposes requirements that will boggle your mind and are likely unique to NYC. Take any trivial software task, target municipalities as your customer, and you'd better triple your budgets.

Municipal governments are a large portion of our business, so I'm intimately familiar with the challenges. What's unfortunate is that it's infrequently the fault of the individuals running the municipality. We meet a lot of really great people out in the field. It's just that they're doing their job with one hand tied behind their back. They have to operate within a framework that is (mis)guided by public interest.

kenjackson2• 15 years ago

Timekeeping systems are actually extremely complicated. EXTREMELY sophisticated. The level of detail it takes to get one right is painstaking, and it is key to a city's survival. It's complex enough that you wouldn't understand it, nor would any auditor.

For that reason, I just started a new startup company: TimeKeep NYC. We specialize in timekeeping software for NYC area cities. Looking for employees now, especially ones who aren't snitches. Based on past products in this market, I think we can do quite well.

jerf2• 15 years ago

To be fair, timekeeping software really isn't as trivial as you might like, at least assuming that we want to move beyond the raw recording of who worked when out into actual management of such things. Requirements come from federal, state, and local laws, as well as negotiations with all relevant unions. So even before you've gotten to the usual BS requirements laid on by management, you've already got a large pile of requirements that A: are legally mandated and B: written without so much as five minutes consultation with a software engineer.

But, yeah, on the other hand that's just the usual business software story. 10x budget overruns are probably not called for.

clistctrl• 15 years ago

I'm curious, what are some of the complexities involved in a timekeeping system?

billt• 15 years ago

They are more complex than this. Not just anyone can start a company to provide 99.9% reliable enterprise-level management of this problem. A recent study indicated that a major city would have to spend at least $100M to tackle this systemic issue. NYC's mistake was trying to cut corners with $60M.

bmelton4• 15 years ago

Have you already looked into DCAA compliance? If not, your system can't be used for any federal systems, and likely aren't available to be used by most state and municipal government agencies.

anon7865• 15 years ago

> especially ones who aren't snitches

The brazenness...only in NYC.

Also, if you ever get in the position of saying "X is extremely complicated and I can't explain it to you" it is because you stand to benefit by designing a system poorly, or you aren't bright enough or don't have the experience to realize that you can reframe the problem. Algebraic geometry is also complicated, but these days there are plenty of books that will explain it to you, and the tools (like sheaves) are well-established and standard.

Obfuscation is the MO of installed, configured enterprise software. If there were an incentive to create a simple, effective solution, then, lo and behold, one would be created. The incentive is almost never there.

Take Unix for example. It is simple and effective (and flimsy as all heck) but before it came along, obtuse operating systems were the norm. It is worth pointing out that A) Thompson and Ritchie are geniuses and B) there was no financial incentive.

mavelikara• 15 years ago

For those wondering how the project could have cost so much, here are some of the annual salaries that were paid out:

  Job Title                     Annual Salary
  -------------------------------------------
  Specialist, Level 4A          $676, 031
  Project Manager, Level 4      $661, 628
  Project Manager, Level 3      $632, 653
  Project Manager, Level 3D     $601, 286

From http://www.nydailynews.com/topics/Gerard+Denault/photos

calloc• 15 years ago

Those job titles are set by the government and specify what pay is going to be. My job title at my current company is Computer Analyst, yet I am a computer programmer. The gov't contract we have mandates that specific title.

ap22213• 15 years ago

Yeah, those look like outrageous salaries. But, to be fair, I doubt they are salaries. They're probably derived from hourly consultant rates of around $250 / hour.

emehrkay• 15 years ago

I could write one this weekend for only 3 million dollars. Somebody get me bloomberg's info

patio11• 15 years ago

Do payroll slips issued by the system to part-time workers have the option to translate the statement of worker's rights mandated by NY city law on the back of the slip into Mandarin Chinese? No? "Did you even read the bid document? It was right there on page 487 where it said 'The system must be fully compliant with NYC labor law.'"

There are literally a hundred thousand requirements like that one.

BadassFractal• 15 years ago

How complicated must that piece of software have been for it to cost that much? For that kind of money it better be at least as complex as any OS out there.

almightygod• 15 years ago

my guess is most of the time was spent in analysis and consensus building being that it was for government. The code is surely atrocious

InclinedPlane2• 15 years ago

I would wager good money that you could supply thedailywtf.com with entries every day for a month from the code base of this project, easily. When you do the math it works out to something like $100 million a year, which implies a fairly large development team (hundreds of people, at a bare minimum).

It's really easy to waste money on big development projects of this sort. The bigger the development team the harder it is to manage, the less well it is managed the less productive it is in aggregate, and thus the less productive it is per salary-dollar. And these are not small effects. A small team of very talented devs can quite easily compete head to head with a giant team of hundreds of devs, precisely because of these scaling and management problems.

scubzero• 15 years ago

Worked for subsidiary of SAIC contracting out to NYC government over a decade ago. Trust me when I say this is nothing out of the usual... project will go order of magnitude over budget constantly.

As for the requirements that drive these projects, typically they are written specifically to present the appearance of an open bidding process while allowing the departments to know ahead of time who will actually implement the contract because they are so specific in certain areas that only one contract can possibly fulfill them.

jarrett• 15 years ago

Those who say that the city's payroll needs are complex are entirely correct. A city government that size will inevitably have a mind-boggling number of variations in the business rules that the software must model.

Yet $600 million? I've built accounting software, and though it wasn't as complex as what NYC needs, I'm 90% sure $600 million is astronomically higher than what a reasonable price would be.

What about the original price of $63 million? That still seems awfully high to me, but I'm not sure. Does anyone have an opinion on that?

kingkilr• 15 years ago

I've never built a time keeping system, but this is how I put that number in perspective. For $63 million I can hire everyone who's ever been a Django core developer (people I've worked with, and who are excellent) for 5 years, at 500k a year... and there'd be money left over.

protagonist_h2• 15 years ago

Building a time-keeping system for large city like NYC is many,many times more work than building web-framework like Django. It's not a technically complicated problem, but it has a different kind of complexity. That complexity comes complex business requirements, which have to be analyzed and implemented piece-by-piece.

gamble• 15 years ago

My university spent $10 million updating their student information system. It's not hard to hit eight figures when you're paying an army of extremely unproductive consultants $500-$1000/hour.

$600 million is extortionate, of course - hence the demand for reparation. That's why bloated consulting companies like this are able to demand such high prices. Their size provides a target for the inevitable lawsuit if things go south.

n-ion• 15 years ago

Maybe some of you know, maybe some of you don't - but the real hero here in bringing these people to justice is Juan Gonzalez http://en.wikipedia.org/wiki/Juan_Gonzalez_(journalist) He did the reporting that led to the federal indictments. Went on to win the 2010 George Polk award for exposing them. http://www.democracynow.org/blog/2011/2/22/juan_gonzalez_win...

vaksel2• 15 years ago

that seems a little excessive even at 63 million...I'd bet if you gave a YC startup $1 million, they'd be able to do the project in 1 year.

brendoncrawford2• 15 years ago

See: http://news.ycombinator.com/item?id=2716446

vaksel2• 15 years ago

with 1 million, you can afford to pay a thousand bucks to have someone translate that for you

bane3• 15 years ago

Funny, I did part of my M.S. coursework in Enterprise Management using SAIC as a case study in Enterprise Management Failures.

peter_to• 15 years ago

In case anyone wants to read up on the CityTime project, the VillageVoice had a pretty lengthy spread on it in January.

http://blogs.villagevoice.com/runninscared/2011/01/this_week...

NYC mayor demands $600M refund on software project

Related Stories