The BBC's hi-tech failure: Don't Mention It

http://www.economist.com/blogs/blighty/2013/06/bbcs-hi-tech-failure

by chrbutler • 13 years ago

99 90 13 years ago

_pmf_• 13 years ago

I vaguely remember reading uplifting, grandiose articles and presentations about domain driven design at the BBC; search for "BBC domain driven design". From this article, I cannot deduce whether this failed CMS is the same system that has been glorified in these presentations or something else; maybe someone can shed some light on this.

Uchikoma• 13 years ago

First thought when I read this: Is this the system das was the base for all their DDD talks?

kryten• 13 years ago

It probably is. I've experienced the fact that DDD scales up only until the consultants have gone :)

Same as NserviceBus only works until you've paid for the training.

rplacd• 13 years ago

A side matter, but handing the BBC's governance over to Ofcom? They haven't held the Independent Television programme contractors to their remits since 2001 under the threat of any sanction - or, for that matter, put their contracts up to tender (not that it worked or ever will work if we want a One Nation Tory-style public broadcasting system again - and not that I remain suspicious of their inconsistent singling out of Thames Television) since 1993.

The BBC Trust, at the very least, has an image to mantain.

arethuza• 13 years ago

I did a quick check to see how Wikipedia defined a "One Nation Tory" - after all they have been pretty thin on the ground this last 30 years (especially here in Scotland!).

I'm somewhat amused to see that "One nation conservatism" appears to now be the policy of the Labour Party:

http://en.wikipedia.org/wiki/One-nation_conservatism

JamisonM2• 13 years ago

Real headline: Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way.

It seems to me that months or years of auditing and hearings and cries for heads to roll are probably wasted. Just like a private company a few mangers should probably get tossed or reassigned and the second version of the same project should get started to address the businesses needs the first one was supposed to.

macspoofing3• 13 years ago

Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way.

Actually large private enterprises don't usually make this kind of mistake. This is a typical big government project. Software development is hard. Building large, complex, poorly defined software systems is hard.

//EDIT

I should clarify that. I don't think a private enterprise would engage in a project that would have it build an internal system, at that scale ($100 million), having so little impact on it's primary revenue driver, over so many years!

Big enterprises fail with big software, that's a fact. But they wouldn't fail in this specific way. This is a quintessential big government software project failure.

Retric• 13 years ago

There are a lot of numbers tossed around without a lot of hard data backing them up, but it's generally accepted that over half of all large software projects 'fail' at both private companies and government institutions.

jerf• 13 years ago

"Actually large private enterprises don't usually make this kind of mistake."

Really? You sure about that? Because hard numbers are hard to come by, but I'm pretty sure they do with some frequency. It's notoriously common for large consultancies not working for the government to have the same outcome too.

It's possible private enterprise makes this mistake at a different rate than the government, but I'd despair of trying to prove that claim, or even give it a solid definition. In general large software initiatives seem to fail, regardless of whose bright idea they are.

rikthevik• 13 years ago

Actually large private enterprises don't usually make this kind of mistake. This is a typical big government project.

Your last two points contradict your first two. There's nothing unique about a government's ability to make terrible decisions. I'd say one difference is that private enterprises typically go under if the mistake is big enough, and they have no great interest in telling the public about their mistakes.

andyjohnson02• 13 years ago

There's nothing special about private enterprises that makes them immune to risk. As others have said, many IT projects fail [1]. In the private sector this is just seen as a business risk, but when a public sector project fails then its "wasting taxpayers money".

For some reason the Economist article doesn't mention that the project was originally awarded to Siemens and Deloitte [2], but technical failures and cost overruns by these private companies caused the BBC to bring the project in-house. It could be argued that the BBC lacked the capability to deliver the project on its own, but thats also a mistake found in the private sector.

[1] http://www.computerworld.com/s/article/9116470/IT_s_biggest_...

[2] http://en.wikipedia.org/wiki/Digital_Media_Initiative#Initia...

protomyth2• 13 years ago

Actually large private enterprises don't usually make this kind of mistake.

I would say "Actually large private enterprises don't usually [admit to making] this kind of mistake." I swear the ERP and document management are the two areas where every enterprise tries to "standardize across organizational units" and fails badly[1].

I have come to believe that this type of software is hard because it imposes rules foreign to the entity on it and doesn't value a companies existing workflow[2], or it requires thousands of developer hours to adapt.

1) oh, they keep using the new system, but their customers feel it

2) often a very successful workflow for the entity and its customers since the entity can now afford a multi-million dollar software package

gtani• 13 years ago

    Actually large private enterprises don't usually make this kind of mistake.

counter-anecdotes, 24.4k hits

https://www.google.com/search?q=lawsuit+failed+erp+implement...

4M hits, but lots of dups:

https://www.google.com/search?q=lawsuit+failed+crm+implement...

JamisonM2• 13 years ago

Your edit: ..having so little impact on it's primary revenue driver..

Sorry, you are still wrong. The BBC is specifically in the business of video content and their failed system was a video content management system. You can not get any more core to a business than that! This is a quintessential big software project failure I do not see how it being in government has anything to do with it or how you have made any kind of case for that belief.

apalmer• 13 years ago

This is not actually true, I am not aware of much info on CMS roll outs, but large CRM and ERP software projects at private enterprises fail all the time.

This is very common, I think perhaps the hacker news crowd probably have more experience with real software behemoths like google, facebook, microsoft and/or scrappy smaller start ups. Perhaps in those realms this doesnt occur because software delivery is their bottom line. In run of the mill fortune 500 companies major software projects fail commonly, and CRM and ERP projects fail even more frequently and have price tags in the 10s of millions, and when they fail the company has no choice but to roll out a new one

stephenhuey• 13 years ago

Actually, there are lots of stories of millions and billions being blown!

http://www.insearchofstupidity.com/

I have this book, and I'm sure there are plenty more like it.

run4yourlives• 13 years ago

Actually large private enterprises don't usually make this kind of mistake.

Are you new or something? Do you know how many projects I've been brought in to rescue? Do you know how many times my advice to "Stop throwing good money after bad" has been ignored for one reason or another?

Big enterprises fail in exactly this kind of way. All. The. Time.

DamnYuppie• 13 years ago

Like many of the comments below I don't agree with this statement.

I have been at several mid to large sized private companies and have watched at least 4 $30mm+ projects go up in smoke over the years. Fortunately I have not been involved in any of them directly but I got a front row seat. Surprisingly more then half didn't result in extensive turnover of the key parties involved.

pessimizer2• 13 years ago

A productive use of this trollpost is to list failures of giant software initiatives by private companies. You could fill up a thread w/just SAP.

Levi-Strauss, SAP, and $100M: http://www.zdnet.com/blog/projectfailures/levi-strauss-sap-r...

"Levi’s is bucking the industry-wide trend by going ahead with ERP now. 'It’s expensive, high risk and requires a lot of cultural change, and unless you have a really compelling reason to do them you don’t [during a downturn],' says Paula Rosenblum, an AMR Research analyst."

Waste Management, SAP, and $100M: http://www.zdnet.com/blog/btl/promises-promises-a-look-at-wa...

JonnieCache2• 13 years ago

Note that the content in this CMS is hours and hours of raw video footage rather than the usual snippets of text.

true_religion• 13 years ago

Duely noted, but I don't think it ought to make a difference in the price tag unless the CMS was doing conversion and distribution.

untog• 13 years ago

...which it was. One of the big features was the ability to edit video in a lower bitrate (i.e. faster to work with) then send it off to a server to process on the full HD video. Great idea, but apparently it was a total mess to work with.

archivator• 13 years ago

IIRC, their CMS was supposed to be the Holy Grail of video management. Centralized and decentralized storage, processing farms, search-by-image, advanced tagging, etc. Sounded quite impressive but from what I understand, there are now commercial solutions that have some of those feature sets. The way they explain it, when they started no one had these features.

rurounijones• 13 years ago

hours? This is the BBC. It is probably months, possibly years worth of material.

Suitov• 13 years ago

Very definitely years. It wasn't just for finished programmes (i.e. ~ number-of-channels*24 hours of footage/day): it was for all rushes. All the footage that was filmed in the course of making the final programme.

We're talking stupid quantities of data here and I've never seen a detailed answer to how it was all supposed to be stored.

JonnieCache2• 13 years ago

Hours and hours and then more hours until you get to years.

amalag• 13 years ago

In the US I think noone even blinks an eye at $150 million anymore

dopamean• 13 years ago

It's funny you mention that. My first thought was, "wow, that's a lot of money." But then I thought, "but how much is it really?" I actually have no idea how much it should cost for them to do what they were trying to do though I suspect it is probably a fraction of that $150MM.

mikeash• 13 years ago

For a more concrete perspective, if you assume that a good developer costs about $150,000/year, then this is one hundred years of development time for a team of 10 good developers, which is quite a lot.

I don't know just what this thing was supposed to do, so I don't know what the scope is.

sopooneo• 13 years ago

I expect that the lack of a clearly defined "what they were trying to do" was a large part of the problem.

Aqueous• 13 years ago

Article doesn't actually tell us what it is, beyond a traditional CMS.

taylorbuley• 13 years ago

More important, what is in it. A large difficulty with content management systems are the data types; pairing down or normalizing metadata while trying to map old to new.

kailuowang• 13 years ago

$150 million roughly equals to 1000 developer years.

justinph• 13 years ago

From what I understand (and I work in the public media sphere), this was a monster CMS project. A way to integrate audio, video, web content, reporters notes, etc. Is it any wonder a project like that failed?

_f75i• 13 years ago

As a big fan of the Economist, I'm surprised they didn't catch the typo in the name of the project they're reporting on. "Digitial Media Initiative (DMI)" Especially since a rudimentary spell-checker would have caught it.

shitgoose2• 13 years ago

I guess Economist's CMS is not as sophisticated, as one at BBC

squidi• 13 years ago

Previous discussion, including info from someone close to the project, here: https://news.ycombinator.com/item?id=5762116

DrinkWater• 13 years ago

I had the chance to see some of the work Adobe is planning to do with their CMS (some actual demos from the "Adobe Labs"), and i cant deny the resemblance to BBCs case. I am wondering whether Adobe was involved...

astrodust• 13 years ago

The same organization that couldn't be arsed to host some of their historical web properties due to "high costs": http://853blog.com/2011/01/25/pulling-the-plug-on-the-bbcs-i...

MrDOS• 13 years ago

It strikes me that the BBC is hilariously bad at archival. They scrapped dozens of Doctor Who episodes in the name of saving space (http://en.wikipedia.org/wiki/Doctor_Who_missing_episodes).

objclxt• 13 years ago

The BBC is, today, very good at archiving programming material. All output is kept for at least five years, and the overwhelming majority is kept permanently (exceptions are things like long-running quiz shows, where samples of material are kept post five years, although any disposed material is offered to the BFI and other third parties before being junked).

Doctor Who wasn't scrapped in the name of saving space, it was scrapped because nobody thought it was worth keeping. The shows couldn't be repeated (because at the time, Equity had negotiated limits on repeat rights), and quad tape was very expensive and easily re-used.

More importantly, nobody really thought that Quad was an appropriate archiving format. Programs would be tele-recorded onto 16mm for overseas sale and archive. This is not, in of itself, a crazy idea - try finding a quad tape player today vs a 16mm projector. The problem at the time was there was no mandate for archival, so the engineering department were junking the tapes, and the commercial department (who were selling Doctor Who on to other broadcasters) weren't retaining the 16mm reels.

Even in the 1970s, many people were still not convinced of the value of archiving TV output. It's not totally dissimilar to the early days of the web - it was seen as ephemeral. This was happening at pretty much every broadcaster around the world - Wikipedia has a pretty big list of lost TV programs, most of which were wiped for space.

KaiserPro• 13 years ago

In the name of saving cost.

Tapes were (and still are) really expensive. at the time the BBC were really strapped for cash (as ITV were not long after.)

leejoramo2• 13 years ago

Not just the BBC. Everyone did that.

Read about how Groucho Marx's grandson saved "You Bet Your Life" from NBC's trashing.

Even into the 1980's Ted Turner made a large part of his fortune by buying the back catalog of movie studios who saw almost no value in their archives. Turner saw the value of playing these old films on cable and video tape sales.

EDIT: forgot this link

http://boingboing.net/2013/03/12/the-day-my-grandfather-grou...

pessimizer2• 13 years ago

That was common.

https://en.wikipedia.org/wiki/DuMont_Television_Network#DuMo...

"By the early 1970s, their vast library of 35mm and 16mm kinescopes eventually wound up in the hands of ABC, who reportedly disposed of all of them in New York's East River to make room for more recent-vintage videotapes in a warehouse."

dmragone• 13 years ago

I'm surprised more public projects don't run as bounty-style "build x and get $1 million" type systems. Rather than picking who's going to do what (or worse, building all in-house), having many different small times competing for one pot seems like it might be a better way to get a good starting foundation. Or perhaps that's a crazy idea for something as big as this.

jiggy2011• 13 years ago

Do you think any decent consultancy is going to be willing to invest millions of £ into building a system where they will most likely be able to recover none of it?

OTOH , more uk government code is being released as open source. So in theory any developer could study the source and offer to fix bugs/add features and give a more accurate estimate. This might increase competition.

k2enemy• 13 years ago

It might not lead to savings though. Suppose that the normal contract price is $1,000,000. If you have a bunch of small firms try to build the system, each on only gets the reward with probability p = Prob(they succeed and they get picked out of all the successful firms). A firm would only participate if the reward were greater than $1,000,000/p.

jasondemeuse• 13 years ago

Because the best aren't going to put good work into a project that will probably just get thrown away.

If you entered a bounty project with two hundred other entries (all of which have systems similar to yours), would you put hundreds of hours and all of your effort into the project in spite of the fact that there's only a 0.5% chance you'll get paid? Not if you want to stay in business.

The size of the project is barely even relevant, take a look at bounty programs that are up now for areas like design (99designs, etc) and how they are chastised for pumping out bad products from designers who are only putting a fraction of effort into the projects.

This type of system sounds great for those looking for cheap labor, but in practice it falls apart.

Finster2• 13 years ago

That's spec work.

tomjen3• 13 years ago

Yes, that seems quite obvious, which is why it puzzles me that you took the time to post it.

jcromartie• 13 years ago

Who has time to build a $1M system and only potentially get paid for it? I know the basement Lisp hacker[1] probably thinks he could cobble it together in a weekend, but the reality is that these are huge systems. How could you get a team of people to build a system with a $1M budget without knowing if they'd ever get paid? How would the bounty be awarded? How would they know the winner produced software that is up to snuff? Would there be multiple parallel betas? Would this system encourage people to cut corners? Would the person choosing the winner be able to just give their buddy the prize and turn down everybody else?

[1] at this very moment I'm in my basement, hacking on some Lisp

tehwalrus2• 13 years ago

Nowhere near as much as the £12Bn the NHS spent on an impossible moving-spec IT system that they also scrapped(1). Oh wait, it's a tenth of the amount.

(1) http://en.wikipedia.org/wiki/NHS_Connecting_for_Health

tubs• 13 years ago

A tenth of 12B would be 1.2B. This is a hundredth.

tehwalrus2• 13 years ago

Apologies, physicists are often off by an order of magnitude ;)

lsd5you• 13 years ago

This is widely reported size of the waste, but as I understand it, because the program was cancelled 'only' £2bn was wasted.

rch• 13 years ago

The US has contributed some nice failures as well. For example, caBIG ($300M+):

https://www.google.com/search?q=cabig+failure

thisone2• 13 years ago

Saying the program was scrapped is a bit off. It makes it sound like nothing has come out of the project at all.

Choose and Book is widely in use, the Spine exists, as well as the N3 network.

Data format and transfer standards are coming into play now. So instead of one system to rule them all, it's working towards interop. Which probably should have been the overall goal to begin with.

People like to think of the NHS as a monolith, but it's really hundreds of tiny fiefdoms.

hahainternet• 13 years ago

Tiny fiefdoms all on 172.16

They could easily have picked ipv6 and done it the smart way. Oh how things could have been.

arctangent• 13 years ago

Relevant recent news:

http://www.nao.org.uk/report/review-of-the-final-benefits-st...

jamespo2• 13 years ago

When I was at BBC Online ~10 years back they wasted a couple of million on a content management system through a "leading" consultancy that never made it to production.

Finster2• 13 years ago

I've personally worked at 3 large organizations that have tried to formulate and develop an all-encompassing content management system. One of those organizations is a broadcaster that was also trying to build a digital content management system.

Every single one of those projects has been a total boondoggle. Always over budget. Always changing scope. Always missing every single milestone and deadline.

Now, I know that this is not a statistically significant sample size, but this sort of thing seems to be more common that I initially thought. It's like the worst-case scenario of design-by-committee.

podperson• 13 years ago

The only reason failed content management system isn't an oxymoron is that content management system is three words.

caycep• 13 years ago

Sounds like a typical hospital EMR/EPIC/Cerner/Meditech deployment!

handelaar• 13 years ago

Was wondering if anybody here would notice that the "CMS" being described here was a project to handle every single second of radio and TV footage from ingestion through recoding through editing to broadcast, and then beyond that to transcoding everything for whatever devices want to watch or hear it once it's finished.

People here hear "CMS" and think we're talking about text.

Incidentally the parts of this boondoggle which actually exist are being used as components in the BBC. The headline last week was about the decision to terminate that particular project because it had got far, far out of its depth. Also to put the highly-charming person responsible for it on gardening leave indefinitely pending the investigation.

It's a massive cockup, but it has had some useable outcomes, and it is a product class that doesn't exist outside the world of the Beeb, or other national/international broadcasters. There's nothing off the shelf to buy here. In the 90s this sort of thing wouldn't have got cancelled, so I'm seeing this as a sort of progress.

ttrreeww• 13 years ago

Now they didn't. They "transfered" $152 million of public money into private hands.

It's a success. Plus, they get to do it again.

clientbiller• 13 years ago

They were trying to build netflix?

bifrost• 13 years ago

I'd say "well this is what happens when government runs something technical" but I also watched similar stuff happen at MSN when I was there. Not to say that the Beeb is like MSN, but I do know that there were a lot of dim bulbs to go around...

pero• 13 years ago

Related further reading for trainwreck geeks:

EDS (now HP) infamously botched a 1-year ~50million-turned-4 year-multi-hundred million dollar CRM project for BskyB a decade ago, and eventually lost a years-long lawsuit.

http://www.linklaters.com/Publications/Publication1403Newsle...

The BBC's hi-tech failure: Don't Mention It

Related Stories