How Hunch Built a Data-Crunching Monster

http://www.readwriteweb.com/hack/2010/12/how-hunch-built-a-data-crunchi.php

by tgriesser • 16 years ago

40 13 16 years ago

readwriteweb.com

abhikshah• 16 years ago

Interesting and very unexpected. Any other examples of companies going the non-COTS/Map-Reduce/Cloud way?

pashields• 16 years ago

"more conventional technologies like Hadoop"

It's remarkable how quickly distributed computing has come to the forefront of scaling. 3 years ago, before hadoop and horizontal scaling became huge buzz-ideas, this (buying comically large boxes and heavily optimizing your math libs) was the standard practice.

That being said, there is a reason why people have slowly but surely been moving towards distributed systems: If Hunch needs boxes that big to deal with 500k uniques/month, they are going to need to run some seriously insane hardware when they actually grow a large user base.

jbooth4• 16 years ago

Graph problems are really hard to scale out because any node could theoretically be connected to any other node. That means you can't just break the graph into 5 chunks because some edges will cross server boundaries.

The only way I've seen it done in Hadoop is to store a big file listing every edge as a pair and do N passes if you want to do a calculation with a distance of N. Not the most elegant approach but with enough horsepower it'll work.

zoudini• 16 years ago

In practice though, is the degree of connectedness really ever that high? I was under the impression that many real world phenomena (from social networks to biological interaction networks) exhibit small-world phenomena and you tend to get more well-connected subgraphs that are connected by important hub nodes. So it would seem to me like applying some careful heuristics (and keeping track of important bottlenecks/hubs) would still allow you to do computations in chunks.

Would love an expert on the subject to chime in.

pjscott2• 16 years ago

Google has published a very short paper about a program, called Pregel, which expresses graph algorithms using message passing between vertices. Apparently its scaling doesn't suck:

http://www.royans.net/arch/pregel-googles-other-data-process...

They do try to group related vertices as much as possible, but if edges span computers, that's okay too. I assume that it works best when the interconnect between computers is low-latency, though.

chunkbot2• 16 years ago

We've experimented with this, and we've found that with "social network" graphs (that is, scale-free networks whose degree distribution follows a power law) it's possible to determine "good" partitions on which to split the nodes. "Communication" (that is, algorithms that run across edges) within a partition is fast, and message-passing between partitions is minimized.

brisance• 16 years ago

That is darn impressive. What sort of network interface and protocol is it using?

sroussey• 16 years ago

I hope they are running with swap turned off...

How Hunch Built a Data-Crunching Monster

Related Stories