Europe’s biggest city council faces £100M bill in Oracle ERP project disaster

https://www.theregister.com/2023/05/25/europes_biggest_council_faces_100m_erp_bill/

by Bender • 3 years ago

81 94 3 years ago

theregister.com

MayeulC• 3 years ago

Is that an example of the sunk cost fallacy, or corruption/lobbying?

rightbyte• 3 years ago

My take is that it boils down to insane requirements from the buyer and yes-men sellers.

The technical problem is wanting one end-all goto system instead of like 20 different utilities I guess.

magicalhippo• 3 years ago

I haven't been on projects of this size, but with many of our customers a big issue is that the decisions are taken by people sufficiently high up in the organization that they don't know what's really going on, and without including key resources from lower levels to provide that insight.

In my experience it only takes one or two levels to get to that point.

Regarding the insane requirements, it's also my experience that a lot of companies think they're snowflakes, thinking they do something really special, when they're really just doing what everyone else is. For historical reasons they ended up doing it this way instead of that way and now they'd rather have the world change around them than change their ways.

notdang• 3 years ago

They changed from SAP ERP to Oracle ERP. There were no 20 different utilities to begin with.

red-iron-pine6• 3 years ago

The technical problem is wanting one end-all goto system instead of like 20 different utilities I guess.

At some point these systems have to share info, dump info somewhere (data lake, backups, whatever), and output info.

Inevitably there will be one or two systems that connect them.

Plus the overhead of dealing with 20+ different vendors, systems, updates, vulnerabilities, interconnections, etc. is a nightmare. There is a point where a "single source of truth" is too much effort and too expensive, but a clusterufck of different systems is not a great solution.

daenney2• 3 years ago

¿Por qué no los dos?

peapicker2• 3 years ago

Implementation partner was EY (Ernst & Young). Taking a heavily customized SAP into anything else is going to be painful, and the wrong partner for implementation can really cost.

namdnay2• 3 years ago

In my experience, with these types of project, 20% of the error is on useless partners, 80% on useless clients, who have no knowledge of how their current system works, are not willing to compromise on any existing features (because it's always easier to say "no" than to try and convince others to say "yes"), and don't really care about the migration one way or the other, because they weren't really asked in the first place

If Apple had been in the B2B enterprise space, the iphone would be an exact clone of the blackberry

rqtwteye• 3 years ago

“ 80% on useless clients, who have no knowledge of how their current system works,”

I tend to think that’s true. I have been involved in some process improvements at my company and it was almost always that:

* Nobody really knows how things are working now

* A lot of people don’t want to adapt to new processes

* Execs give you a lot of verbal commitment but don’t do anything to really help

It’s really hard to implement a good software system within these constraints.

phkahler• 3 years ago

Does ERP really need customisation? That seems to be a big cost driver as well as causing lock-in.

namdnay2• 3 years ago

it's less "customisation" than "implementation". they're not changing core code in the system, they're just writing all the config and the workflow apps on top. this is the case for any large organisation, whatever the ERP. And even more the case for a local government that is trying to use a tool designed for industry

jrochkind15• 3 years ago

I think that's the right question that gets at the heart of the ERP mess, but it doens't have a great answer.

In theory, you should be able to have your software support the workflow that works for your specific organization, business, and context. Rather than having to change your workflow to what the machine demands as universal. The latter seems really inhumane.

In practice, it never seems to work out very well. Does _anyone_ like their ERP and it's implementation?

These products are designed around being customizable -- if they gave that up, they'd be much simpler products, which means they'd probably work a lot better (for what they are designed for, which may not be what you need, which is the problem), and be a lot cheaper (which is not necessarily in the interests of the companies that sell them -- the complexity is fine with them, as long as they can keep charging for it, including with their consultants).

It almost makes me wish yo ucould do things without software at all -- but that's a disaster too, this isn't 1970 and pretending it is doesn't work.

There's really no way out. We've built a society where you can't do this without software but the software that would be _good_ takes more skill and resources than society can possibly afford.

kodah• 3 years ago

The alternative is to build purpose built software, but that would require this municipality to pay SWEs (among other folks) directly. An interesting case study would be if it were actually cheaper in the end to just employ some software engineers and write something catered to themselves.

politician• 3 years ago

It really does. They are all custom tailored for each client. The customization includes data sources, processes, reports, and a range of other topics. It's difficult because the project is to replace a working system with another different working system with minimal downtime. Often, a large number of people need to be retrained on the replacement system -- a system that integrates with a labor force of 50,000 people is not uncommon.

stareatgoats• 3 years ago

Does ERP really need customisation?

Yes, for a number of reasons:

1. An ERP system consists of thousands of modules, out of which a smaller subset is needed by the customer. One typically pays by module so you want your number of modules to be as small as possible.

2. A customer this size has already implemented a number of business processes that they are not willing to sacrifice, even though an ERP provider will do their best to have them import "their" way of doing things.

3. Customers want to acquire an edge, not be on the same level playing field as the competition. "Kill you darlings" - eh, no.

4. If a municipality buys an ERP system tailored to manufacturing/trading, well then you have substantial customization ahead.

denton-scratch2• 3 years ago

not be on the same level playing field as the competition.

Birmingham City Council doesn't have competition.

FredPret• 3 years ago

Former ERP consultant here. There are infinite ways to do business: different perspectives to take on the same situation, different deal structures, different data sources, reporting, etc.

You can run your business / org exactly as the ERP designers envisioned, but that means you'll have to adapt to your ERP, which is wrong-headed and certainly much more expensive in a large org.

Even the most vanilla implementation for a small business will have a degree of customization.

red-iron-pine6• 3 years ago

You can run your business / org exactly as the ERP designers envisioned, but that means you'll have to adapt to your ERP, which is wrong-headed and certainly much more expensive in a large org.

Bingo. I'm not going to tell my CEO to change how they run the business just to conform to how the lowest-bidder ERP decides I should do things.

Unless it's ServiceNow, for some reason.

bluedino• 3 years ago

Worst case the client wants the new system to emulate the old system, so yea, you need a lot of customization.

orwin2• 3 years ago

Leaving Sap was a good idea, but i'm pretty sure replacing Sap by oracle ERP wasn't. Especially, it seems it's one to one in functionalities?

The better idea is to take a more modular ERP imho, and slowly transition.

stakhanov• 3 years ago

The better idea is to take a more modular ERP imho, and slowly transition.

...or just imagine what £100M in bounties to open source developers could accomplish. It would probably get their requirements met and, as a side-effect, contribute billions to the world economy by liberating many others from lock-in, besides just Birmingham City Council. As it stands, all they seem to have accomplished with this ludicrous amount of money is getting rid of one evil by inviting another.

mschild• 3 years ago

Even just normal competent salaried employees would've been an improvement.

With that amount you could've reasonably employed a 50 person department with roughly 100k cost of employment for 20 years.

justrealist2• 3 years ago

Open-source only helps organizations that take ownership of their own design and decision-making.

Contracting disasters are always because the customer is trying to outsource thinking. Usually it ends poorly. Either way, you don't solve the problem with a configurable OSS tool.

RobotToaster• 3 years ago

It does sound like a very "out of the frying pan" decision.

generic920342• 3 years ago

Leaving Sap was a good idea

From the article: "Standard SAP is the same, but BCC customized SAP to get it working really well and apart from some minor annoyances, SAP was a good product that should never have been ditched," the insider said.

orwin2• 3 years ago

I've worked with SAP in two companies: for the end user, it's slow, ugly, hard to use, and full of idiosyncrasy.

I've used two open source ERP, both as end users and as maintainer (granted, in smaller companies). Both were objectively better. Props to ERP5 and the composition it allows by the way, especially for the web pages that are easy to modify without prior knowledge (controllers and the database especially are a pain to work with however, but unless you want to create a new module yourself, you won't need it).

daenney2• 3 years ago

"The UK's biggest city council faces £100M bill in Oracle ERP project disaster". There's really no need to rope in the whole continent for a specific thing in a specific country. The fact that they're the biggest local authority by population in Europe is really not relevant to anything that's happening here.

KptMarchewa• 3 years ago

I don't understand how they are "biggest local authority by population in Europe". Surely Istanbul is larger.

pjc502• 3 years ago

Is Istanbul a single unitary local authority or done by boroughs/arrondisements?

justrealist2• 3 years ago

Europe is a state of mind.

fredoralive• 3 years ago

It's a bit of a weird semi-pedantic stats claim, they're basically counting "largest lowest level of government". Birmingham is a large unitary authority, there are no district or borough councils below it. Looking at Wikipedia Istanbul seems to be split into districts which have their own local governments (similar to London with its boroughs). So they're counting the 514,900 population of Başakşehir or 259,146 of Hackney vs the 1.1 million of Birmingham, rather than the total populations of Istanbul or Greater London.

pelorat• 3 years ago

No one living here considers Turkey to be part of Europe

rossdavidh2• 3 years ago

Well I suppose it's relevant in two ways:

1) this is a huge bill, but it is relevant that it is a bill for a large city council (as opposed to a village facing the same bill)

2) they are big enough they should not really be considered as too small to afford good counsel on how to do this sort of project properly

throwuwu2• 3 years ago

What does ERP mean in this context? Asking because it means something else that is making all this discussion of ERP projects and consultants sound quite funny.

candiodari4• 3 years ago

At one point I contributed to "Odoo" (OpenERP). It essentially means having lots of Applications work against a single database so that everything about the whole company uses this single database.

ERP vendors generally have a whole bunch of standard applications ready. Things like inventory, sending out stuff, payroll (for X number of countries), ...

throwuwu2• 3 years ago

Thanks! Still does nothing to break the association with the other definition for me though so I’ll add ERP vendors to the list of things to giggle at

denton-scratch2• 3 years ago

Things like inventory, sending out stuff, payroll (for X number of countries),

Long before I ever heard of ERP, I worked on the Burroughs accounting systems. They had inventory management, accounts payable, accounts receivable, payroll, subcontractor accounting etc., all feeding into the general ledger.

So is that really what ERP is? Is it what used to be called an accounting package?

alistairSH• 3 years ago

Enterprise Resource Planning.

Basically, the totality of "back-office" operations software. At minimum, it's usually accounting+finance, often including HR, sometimes also CRM. And in niche spaces, other stuff too (an ERP for higher education/colleges would also include student enrollment management, etc).

pjc502• 3 years ago

This happens a lot. Another way of looking at it is that this is a successful outcome for Oracle, because they've hugely increased their billable hours.

Councils are I think too small to access GDS? If they could I'm sure the superior GDS project management would mitigate this kind of disaster.

I remember when the University of Cambridge had a similar disaster and ultimately published an inquiry: https://www.admin.cam.ac.uk/reporter/2001-02/weekly/5861/1.h...

toyg5• 3 years ago

Councils are I think too small to access GDS?

Councils probably don't want to access GDS. Why lose all those sweet vendor kickbacks? Free ticket to this, free travel to that...

There is a lot of corruption in Britain at the lower level of government. It's all in plain sight, it's all legal, and because it benefits the same networks (someone would say "classes") that are structurally in power in Westminster, nothing gets done about it.

revelio3• 3 years ago

Birmingham council is run by Labour but Westminster is run by the Conservatives. Unless by classes you mean all politicians, it's not the case that it's all linked back to central government.

Besides, corruption isn't needed to explain this. ERP projects fail all the time, with SAP as well.

vladharbuz• 3 years ago

Could you say a bit more about how those running the councils benefit from these contracts? I don’t think I fully understood what you were saying about kickbacks.

_Wintermute• 3 years ago

Reminds me of the University of Edinburgh this year with their own Oracle disaster, seems like it was largely on time and budget, it just didn't work: https://www.theregister.com/2022/11/23/edinburgh_university_...

sega_sai• 3 years ago

It was on time, because they forced it on us before it could actually work. Regarding on budget that's not really true anymore as: "The institution has confirmed implementation partner Inoapps and software company Oracle will see the price of their contract rise from £25.4 million ($31.1 million) to £33.5 million ($41.12 million), owing to changes in requirements and additional work orders by the customer" But it's been a big debacle.

908B64B197• 3 years ago

This happens a lot. Another way of looking at it is that this is a successful outcome for Oracle, because they've hugely increased their billable hours.

Honestly, it's not that surprising.

Government salaries aren't competitive with the private sector, and especially not with FAANG. Same government workers are now tasked with identifying requirements and managing consultants. But are they even qualified to do that?

red-iron-pine6• 3 years ago

Same government workers are now tasked with identifying requirements and managing consultants

I've been in the private sector for the last 10 years and most of the folks I worked with sure aren't.

AngeloAnolin• 3 years ago

Not surprised.

Enterprise software is always *enter-pricey*.

Lots of $$$ poured into consultants and very little to the folks who actually do the work and implementation.

And of course, a project (or program in corporate speak) of this magnitude will have a lot of meetings involving a lot of people that usually ends up in very meaningless decisions.

revelio3• 3 years ago

Does anyone know why they decided to leave SAP?

pera• 3 years ago

Another one six months ago: University of Edinburgh orders investigation into Oracle finance disaster

https://news.ycombinator.com/item?id=33739657

AlbertCory• 3 years ago

The term for ginormous software packages like SAP and Oracle ERP is HostageWare:

Some people inside these customer organizations are fanatically committed to it, since what choice do they have? A disaster like this?

drumhead• 3 years ago

If it's not a fail with Oracle, it's SAP or dynamics or some other "ERP". Do these implementations ever work, and why do these big organisations keep going for them?surely they've seen the string of failures preceding them? I suspect it's ego, and the need to pad CVs that does it. And the unwillingness of the people who should know better to say no.

rossdavidh2• 3 years ago

They "work" just fine for Oracle, SAP, etc.

nradov3• 3 years ago

Most implementations do "work" to the extent that they are better than nothing. I mean what's the alternative? Running business processes on paper is no longer a practical option. And most enterprises lack the scale and competence that would be needed to build their own custom ERP in a cost effective way.

user568439• 3 years ago

Yes, they do work and sometimes pretty good. It mostly depends on the implementation team. SAP which is the one I know, has plenty of defects but also a lot of virtues. A good implementation and customizing team can make it run properly.

red-iron-pine6• 3 years ago

SAP S4/HANA was insanely expensive but generally worked, and on time.

Finding consultants who could help was like pulling teeth, though. The offshored Indian ones were terrible, and the European ones often didn't speak English well enough. Mexico City was had good offshoring options but couldn't supply enough bodies.

Never had a strictly Oracle project go well though. Exadata is a dirty word in these parts. Only thing that was more of a PITA was Azure...

taylodl2• 3 years ago

Overpromise, underdeliver, and overcharge. It's the Oracle Way.

In my experience, Oracle only ever built one product that was good, and you can replace it with PostgreSQL for significantly less money and gain greater flexibility to boot.

I think the source of Oracle's execution woes lies in part with their Fusion Middleware. It's garbage. Even though JEE has become somewhat outdated, it's still viable. That isn't the issue. The problem is the horrible memory management, continually stuck threads, and their poor messaging implementation. It makes solutions built on the platform prone to constant failure and requiring lots and lots of hardware - which of course increases licensing costs back to Oracle.

I've given up trying to figure out why people keep buying their stuff.

peapicker2• 3 years ago

EY was the implementation partner, Oracle the vendor. The article doesn't make that clear, but a linked one does. PWC and KPMG were brought in to try and fix after EY failed at additional cost. All three are known for putting brain or three and a bunch of new hires at top rates on projects. Oracle is a red herring here.

taylodl2• 3 years ago

Oracle is not a red herring. These are just Oracle's "implementation partners." They're all in on the take.

red-iron-pine6• 3 years ago

Oracle ERP project disaster

Redundant headline. Oracle & ERP almost always implies disaster.

Seriously, never again. Ditto for IBM, and a lesser extent, SAP. I wish Red Hat had a real competitor, cuz Canonical ain't there yet.

ripperdoc• 3 years ago

It seems to me that most city councils will operate similar services and therefore have similar needs from their software. Therefore it doesn't make sense for each council to buy and customize their own system. Even better, they could co-fund an open source tool that can be used by all. I realize it's a simplistic view but it seems like there is a lot of money to be saved.

richliss• 3 years ago

Birmingham Council has always been effective at wasting money.

Europe’s biggest city council faces £100M bill in Oracle ERP project disaster

Related Stories