No one ever got fired for using Hadoop on a cluster (2012) [pdf]

http://research.microsoft.com/pubs/163083/hotcbp12%20final.pdf

by CHY872 • 11 years ago

37 14 11 years ago

research.microsoft.com

shmerl• 11 years ago

There are other systems which don't limit you to mad reduce patterns. For example HPCC.

threeseed• 11 years ago

Modern day Hadoop doesn't limit you either.

YARN containers are pretty generic and I have used them to run all sorts of things: Kafka, ElasticSearch etc.

krick• 11 years ago

mad reduce patterns

I wonder if it's a typo…

wmf• 11 years ago

More recent related project where a laptop outperforms clusters: https://www.usenix.org/conference/hotos15/workshop-program/p...

caminante2• 11 years ago

That's awesome.

On a meta-note, I felt like the abstract wasn't very clear and I had to skim the rest of the doc to figure out what was going on.

Smerity• 11 years ago

His blog posts are easier to follow and aren't forced to be in the academic format :)

http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...

istvan__3• 11 years ago

Actually I know several projects bursted into flames because of Hadoop. There is a notion among some developers of Hadoop solving all of the problems and this is the only solution they consider (literary). Needless to say that real time systems with tight SLA cannot really use Hadoop, and if you are trying to sell it for this use case, you might get fired. :)

sixdimensional2• 11 years ago

I concur and I have seen the same, especially for folks with complex projects, tight deadlines and budgets, and older distros of Hadoop that were not as stable and/or production ready.

In a rush to get to the next big thing, many have taken a lot of risk with Hadoop. Some with the resources to really support/make Hadoop their own have had a lot of success. But I think there are fewer who have success stories.

istvan__3• 11 years ago

On the other hand, Hadoop is great for analytics. If it is down there is no customer facing outage. Sometimes the failover works and only few things lost, you are back in ~15 minutes if you want to check the fs image consistency. For internal teams this is acceptable SLA. If you don't care about consistency than you can probably do this faster.

I just don't see a customer facing SLA sensitive workload fitting this well. At least I would not sleep well being oncall... :)

reality_czech• 11 years ago

This is just a bunch of FUD (fear uncertainly and doubt) that Microsoft spread back in 2012 when Hadoop didn't have Windows support. Get in our time machine and go to 2015, and Hadoop has Windows support. Suddenly Microsoft doesn't seem to mind it. Gee, I wonder why?

Hacker News is ridiculously susceptible to propaganda (MongoDB is WEBSCALE! JS framework of the month is the bee's knees!). Please think for yourself and don't read transparent propaganda that is years old. 3 years is an eternity in the big data world.

pav7en• 11 years ago

Seems like a valid point of view. Not sure why it got downvoted.

Though there are very big kinks to work out in getting Hadoop production ready.

So the thesis of the parent article is valid too and not just propaganda.

No one ever got fired for using Hadoop on a cluster (2012) [pdf]

Related Stories