As Digg Struggles, VP Of Engineering Is Shown The Door

http://techcrunch.com/2010/09/07/digg-struggles-vp-engineering-door/

by icey • 16 years ago

143 117 16 years ago

techcrunch.com

rbranson• 16 years ago

Queue NoACID haters declaring this the final nail in the coffin.

dangrossman• 16 years ago

This isn't Digg.

heffay2• 16 years ago

I'm really interested to see what comes of this and what went wrong. It sounds like (from reading his blog) they were making a lot of customizations to Cassandra?

pquerna• 16 years ago

They have made many feature additions to Cassandra.

They haven't said much about the details of why they are having trouble.

It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest.

ergo98• 16 years ago

"It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest"

Digg made a big deal about their move to Cassandra (just as Digg's move to Cassandra was used to legitimize Cassandra, and by correlation NoSQL, among a wide range of zealots), going back over a year.

The thing about talking big like that is that it often comes back to bite you in the ass if things don't go well.

If Digg quietly released a new version that worked more reliably and provided a better experience, they would have been in a perfect position to pontificate on technology.

madridorama• 16 years ago

Digg turned into nothing but a paid distribution and SEO engine anyway

blasdel• 16 years ago

Digg was something else before?

josefresco• 16 years ago

Yeah originally it was Slashdot but better, more current and user powered (which was a new concept in the mid 2000's). What it turned into was a traffic generator for a small group of power users who fed the community the lowest common denominator of content.

agentultra• 16 years ago

This man's reputation is on the line. I hope they release more details on what exactly it is that is causing the problem. As it stands he appears to be an unfortunate scapegoat.

heffay2• 16 years ago

Hopefully he creates a blog post explaining the disaster. He seems to regularly update that site

mikeryan3• 16 years ago

I actually wondered if they were running into issues with Cassandra. I'm not a NoSQL hater - but its still pretty bleeding edge and it always seemed like making it your core db was super risky.

arg - really want more insight, maybe Quinn will elaborate now that he's gone.

narrator• 16 years ago

Reddit runs on Cassandra + PostgreSQL. They use Cassandra as a key/value store and not as the primary database though.

cookiecaper2• 16 years ago

They are planning to transition to Cassandra as the primary database. PostgreSQL is not used in a relational way on reddit -- it is used as a makeshift k-v store. You can take a look at this yourself as reddit is fully open-source: http://github.com/reddit .

zzzeek• 16 years ago

> They are planning to transition to Cassandra as the primary database.

do you have a source for that ?

> it is used as a makeshift k-v store

That is very true, they should switch. It just seemed like they had no plan to do so.

pavs6• 16 years ago

According to reddit admins some of the recent downtimes in the last few months were due to cassandra and they were having some bad performance and stability issues. I can look it up if anyone is interested but it will be a bit of a work, since they stated it (several times) in the comment section and perhaps once on a blog post too.

ketralnis4• 16 years ago

> They use Cassandra as a key/value store and not as the primary database though

These are orthogonal.

btipling• 16 years ago

For some reason this reminds me of the "no one has ever been fired for choosing IBM." slogan. I guess maybe there is some truth to it. Seems like new tech needs to start slow on big places, only startups really have the freedom to risk it all.

aaronblohowiak2• 16 years ago

your post implies digg isn't a startup.

soofaloofa• 16 years ago

Digg isn't a startup. It started in 2004 and has been mature for about 5 years. They are a business now. And a big one at that.

cookiecaper2• 16 years ago

Is there any reason that Cassandra is the focus of this article? It is really silly and irresponsible to peg a nascent project like that without any reasoning or sources. I'm sure something changed besides just a Cassandra rollout, and wasn't Digg using it on v3 too?

I think Cassandra is pretty well tested. There have been lots of super-large-scale deployments. It just seems lame to blame it on that, but I guess maybe their anonymous sources inside Digg revealed it? But then we'd hope they'd know if the problem was with the datastore or the implementation.

riffraff2• 16 years ago

as far as I remember: yes they were using it on v3 too, for some things

wsongk• 16 years ago

About a year ago, they were using it to store a list of friends of viewing user who dugg the viewing news.

original blog post: http://about.digg.com/blog/looking-future-cassandra

Treads on HN: http://news.ycombinator.com/item?id=813528

sliverstorm2• 16 years ago

Placing all the blame on Cassandra helps Digg justify letting go the VP, since he was allegedly the guy who pushed it.

TheCondor2• 16 years ago

So what's the digg story here?

I get that some VP suggested a new buzzwordy technology, they gave him enough rope to hang himself and he did and left the company with a broken pile of crap. It could happen, if you have a healthy company you give trust to people. That it got this far doesn't speak well of the rest of management. It doesn't speak really well of the rest of the team either. Shouldn't there have been some circuit breakers or something?

Digg isn't a poor, bring-your-own-laptop startup. They've got resources, they've had substantial investment. They can afford to build and test software and I know of no real marketing reason they had to push something untested out. Rose could go out and say it wasn't done, it's going to take more time.

How does Rose keep his job? Wasn't he this VP's boss?

And it's single technology? That couldn't have been vetted and tested independently of all of digg? Really? And MongoDB, or hadoop, or one of the dozens of other nosqls wouldn't work, either? You do what you have to do and there is never a 'truth' with VPs and CxOs are canned, but it all doesn't float with me, just looks like another over valued and under talented company, got lucky and the blind squirrel found the nut, there isn't gonna be a second act.

Maybe it just seems low brow to me, name and finger the guy, blame the opensource tool you use, never explain or elaborate why you launched anyways when you were fixing bugs in the tool at the 11th hour.

jshen3• 16 years ago

Reminds me of the way NASA blamed lisp back in the day

epoxy• 16 years ago

Part of the reason that Cassandra may be the focus is that Kevin Rose places much of the stability problems on Cassandra's shoulders. Two or three minutes into the most recent diggnation he talks about Cassandra and describes Cassandra as "very beta-stage software" and says how days before the launch at least some of their focus was on fixing "Cassandra problems" rather than issues with Digg v4.

http://techcrunch.com/2010/09/07/kevin-rose-responds-to-digg...

brown9-2• 16 years ago

If true, it's a lame excuse: blame should be placed on the people that decided to use the beta software to power their very-important-to-their-paycheck website

mikeryan3• 16 years ago

So if you watch the Revision3 video where they talk about the stability issues Kevin throws Cassandra out there nothing really specific but when discussing downtime its the only technical factor he mentions.

He calls it "still beta software" and states they were fixing bugs in Cassandra during the days leading up to the release of v4.

hga• 16 years ago

In reply to you and epoxy, sounds like a gross failure of engineering management. If you're having problems of this sort of foundational nature a few days before the planned release, it strongly suggests you should delay and figure out what's generally gone wrong with the project.

houseabsolute• 16 years ago

I wonder what issues they are running into that a dark launch would not have found. As I have discovered painfully in my own projects, making big changes without a rollback plan is usually a bad idea, and it sounds like this is no exception.

gphil• 16 years ago

Yeah, it sounds like the launch process is more at fault than the technology for this if there's really no way to rollback to stability.

pclark• 16 years ago

Unless Digg cannot roll out new features due to Cassandra (eg: a) Cassandra woes taking all dev team time, or b) tech limitations prevent features) it seems highly unlikely that cassandra is the reason why Digg 4 is grating with users initially.

It's a change of features/product rather than technology that is the problem.

enjo• 16 years ago

I think the folks at Digg where prepared for the heartburn associated with the content changes. They weren't ready for some very serious issues with the implementation. Digg has definitely seen a lot of downtime over the last week or two.

mikeryan3• 16 years ago

Digg has had some significant downtime during this transition (Can't blame Cassandra - but can't clear it either). Issues with product features are one thing - but nothing probably got more people jumping to a different ship (reddit) then not being able to find a good lolcat when you're bored, and your favorite news aggregator is down. Hmm - I wonder if fark has seen the same uptick as reddit has.

parfe• 16 years ago

I have to say I like the new Digg. Sad this guy is getting his career stomped on over it. Lot of people complain that the new site lets big sites submit their own content automatically, but how was it any different with a middle-man user submitting it himself?

gamble2• 16 years ago

The real problem is that v4 removed the illusion that average users had an influence on the rankings, without altering the fact that they don't.

hop• 16 years ago

Yeah, I may be blind, but how do you even get to the upcoming stories?

amix• 16 years ago

Like Joel Spolsky said it: "They followed the single worst strategic mistake that any software company can make: They decided to rewrite everything from scratch" - http://www.joelonsoftware.com/articles/fog0000000069.html

I think rewriting from scratch is the core of their problem and not really Cassandra. Gradually going over to Cassandra would have been a much better idea.

gaiusparx• 16 years ago

The other problem is they have been hyping V4 for a long time by making beta testing invite only and hard to get. It would have been better if they can have a public V4 beta running in parallel with V3 since it is a rewrite.

dawie• 16 years ago

Even better: Have a link that says: "Try the new Beta" at the top like gmail and google docs, with a "Go back to Old Version".

Implementing this should not be to hard and it allows them to get some feedback.

blantonl• 16 years ago

you can't do a parallel beta test of two functionally different sites. You would end up with two separate data stores, workflows, and ultimately a confused user base.

Never mind the fact that the user base is confused by the rollout.

And never mind the fact that Google has forever damaged the term "Beta" in the minds of general Web consumers.

plnewman• 16 years ago

Actually if you google "digg cassandra" you'll get a number of blog posts dating back to 2009, describing different things they use Cassandra for.

binspace• 16 years ago

Yep, big bang rewrites are always scary. Doing something more incremental would have been safer.

One way to "safely" do a big-bang change would have been to use both architectures in parallel and sync back and forth. That way, if the new architecture fails, you still have the old one.

Of course, that is lots of effort, and restricts new features (which may be a good thing anyway).

squidsoup2• 16 years ago

I've heard this time and time again, but what if your application is a genuine ball of mud? Would you really not advocate a rewrite for an unmaintainable spaghetti classic asp app still in production today?

dasil0032• 16 years ago

I'd say it depends on if your engineering talent is vastly superior today than what it was when it was written. Even then it's so risky. It's probably always better to do it incrementally even if it takes twice as long to do so, because you can maintain working software and fix bugs as you go.

The approach I would take is to get the minimum set of engineers who know the most about each major aspect of the code, and put their heads together on what the ideal architecture would be. But rather than building it from scratch, figure out how to implement just one of those pieces now. That way you can decrease entropy in the codebase piecemeal without chucking out all the code at once, which is no doubt full of forgotten assumptions that no one will remember until it's too late.

MartinCron7• 16 years ago

I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system.

Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events.

A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame that so many people think it's a good idea, even in 2010.

_delirium• 16 years ago

Microsoft could be one example of a successful one. Their flagship Windows codebase that ran the 3.1->95->98->ME line was basically ditched with the rewrite-from-scratch NT (famously done by an ex-VMS team), which later had some APIs ported to it to make Windows 2000 and especially XP be close to drop-in replacements for the old line, while not really sharing much code. I think in retrospect that was probably a good idea: the NT rewrite put the codebase on much better footing than the aging, incrementally updated classic Windows codebase had been.

Solaris is another example of a rewrite that seems to have worked, though the rewrite did derive from a different set of existing code, not a total from-scratch job. But the classic SunOS 1.x.-4.x codebase was ditched, and SunOS 5.x / "Solaris 2" replaced it.

scorpion032• 16 years ago

but what if your application is a genuine ball of mud?

Hello, Flash? (via Gruber)

Yaa101• 16 years ago

No, what they done wrong is not to make a proper 2 way conversion tool for their database backup. Starting from scratch is no problem if you got your core data covered, then you can always revert back.

Don't ever develop yourself into a one way street!!

mmaunder• 16 years ago

This is my all time favorite Spolsky blog entry. This lesson has saved me a few times. The broader lesson is that new and shiny in software isn't better and is often worse.

mcantelon• 16 years ago

Reddit went through similar issues a few months back (downtime, slowness, etc.), but they overcame these issues without turfing people. My guess is Digg pushed the engineering VP out to make the investors happy rather than to actually move forward.

pavs6• 16 years ago

How does it make investors happy? Why would investors care what kind of database you use?

fookyong• 16 years ago

it's an attempt to control the PR message.

by doing this, they are laying the blame on one person so the media can stop hating Digg and start hating the ex-VP of Engineering who "killed Digg".

of course, whether he was actually responsible in some way is something we may never know. for all we know he may have been completely against releasing v4 but was vetoed by Rose et al. or on the other hand he may have overpromised and under-delivered, putting the company in jeopardy, in which case he deserves to be let go.

it's all speculation until we hear an official comment from either side.

code_duck2• 16 years ago

The typical investor doesn't take the time to really understand the companies or technical issues, unfortunately. They see 'there was a problem, management fired the guy who was at fault' and are happy.

megaman821• 16 years ago

This article is a fine example of the poor logic that pisses me off nearly every day. Just because Digg v4, which heavily uses Cassandra, can't be used as a Cassandra success story does not automatically mean you can assume the opposite, that it is a Cassandra failure story. There is no indication why Digg has been down so often and there is really no conclusions to draw as of yet on the technology they use.

fleitz• 16 years ago

The article basically comes right out and says it. You just need to read between the lines a little bit.

From the article: Quinn was the main champion of moving over to Cassandra, say our sources. Now the site is taking a huge hit, at least in the short term, because of that decision and/or how it was implemented, and Quinn is paying for it with his job.

It's always a toss up between whether it was implemented correctly or not. The correct course of action of course would have been to slowly move the site over to the new technology piece by piece rather than a wholesale switchover. The risk is in the migration strategy not the technology picked. They could have been equally stupid switching over to a new architecture with mysql.

jshen3• 16 years ago

I'm not sure I'd use tech crunch as a reliable source.

zephjc• 16 years ago

Digg v4 has added back the features I liked after taking them away - mostly the Upcoming section, which tends to have interesting things that never make it to the front page, and the setting that allows you to default to Top Mews, instead of things I have already read ('My News').

For most users, change in features is what drove them away, not necessarily the spotty QOS (though that was pretty bad for a while).

mminolt• 16 years ago

Ok Digg is having a hard time. I feel like I witness a strange blood thirsty tone in some of the comments though. I wish Digg the best and I hope they can prove all the skeptics wrong. It's going to be tough though!

nl2• 16 years ago

Digg's put out a lot of information about their Cassandra implementation, eg https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf (from November 2009).

Back then they seemed to have a rather sensible migration strategy (ie, basically running the new Cassandra back-end in parallel with the MySQL backend).

It seems to me that it was the v4 upgrade that broke, not Cassandra alone. It's possible their frustrations with Cassandra were more long term, and the fact the v4 upgrade didn't go well was the last straw.

For example, Digg has done a lot of work on Cassandra internals and tools. If you are using a new, open source product you kind of expect that, but it's possible the expense of that didn't seem like good value once v4 started to get into trouble,

c00p3r4• 16 years ago

Let me guess - the problem is about the difference between theory and practice.

In theory, Java is great and Cassandra is great. In practice - Java under a heavy load is a disaster, because it was never designed for it, and Cassandra is a just a hype and propaganda.

Face the reality - it doesn't work in production as it supposed to - as a primary storage engine.

People at the Digg aren't amateur idiots, so I think they do everything as it described in docs, but the damn thing just doesn't work.

nl2• 16 years ago

> In practice - Java under a heavy load is a disaster

Google's heavy use of serverside Java would indicate otherwise.

> because it was never designed for it

Yes it was. Java has a lot of problems, but one thing that isn't a problem is heavy load.

> Cassandra is a just a hype and propaganda

Facebook seems to be using it ok.

c00p3r4• 16 years ago

Google probably use server side Java without a really heavy load, and just add more servers to keep the load low.

System which was designed for being isolated from an OS (leave alone hardware) will have a bottleneck exactly in this level.

Facebook doesn't use it as a primary storage with a high load.

Dervish• 16 years ago

Digg 4.x is like Star Wars Galaxies NGE.

As Digg Struggles, VP Of Engineering Is Shown The Door

Related Stories