GraphLab: A New Parallel Framework for Machine Learning

http://www.graphlab.ml.cmu.edu/

by helwr • 16 years ago

22 3 16 years ago

graphlab.ml.cmu.edu

I read through the pagerank example, looks interesting. That said, with such a rich infrastructure built up around Hadoop (Cascading, Pig, Hive, Elastic MapReduce deployment, etc.) I am not sure how much sense it makes to consider another platform for horizontally scaling out calculations over large data sets.

akshayubhat• 16 years ago

GraphLab is intended for efficiently parallelizing Processing intensive Machine learning algorithms, this is different from "Bring Computation near data" philosophy of Hadoop/HDFS/Map-Reduce. Generally most of enterprise mapreduce application are primarily IO intensive, in that case it is important to have multiple machines accessing data from multiple disks.

GraphLab would be suitable for machines with multiple processors or the new cloud on chip machines which have ~40+ Cores.

Most of the Machine Learning algorithms tend to be processing intensive, and simple abstractions such as Map-Reduce dont help a lot, thus GraphLab has its own niche to serve.

npp• 16 years ago

I haven't read the GraphLab paper in detail, but note that Google itself developed a new framework called Pregel for doing graph-based data processing that doesn't fit MapReduce well. They have a paper on it at SIGMOD 2010. So there is definitely a need for things other than Hadoop/MR.

GraphLab: A New Parallel Framework for Machine Learning

Related Stories