Ask HN: To everybody who uses MapReduce: what problems do you solve?

https://news.ycombinator.com/item?id=6706545

by valevk • 13 years ago

153 120 13 years ago

news.ycombinator.com

roschdal• 13 years ago

"How can I make Google shareholders richer?"

PaulHoule• 13 years ago

MapReduce is great for ETL problems where there is a large mass of data and you want to filter and summarize it.

darkxanthos• 13 years ago

Yup. This. Once I filter down to the data I actually care about, I typically find I'm no longer anywhere near "Big Data" size.

ahulak• 13 years ago

I came here to say the same thing. We use to rip through anywhere from ~100M-3B rows of data every day.. updates to this dataset are time sensitive so we use it to finish the job as quickly as possible.

clubhi• 13 years ago

"How do I make all my projects take 10x longer"

alexrson• 13 years ago

How strongly does a RNA binding protein bind to each possible sequence of RNA?

nextos• 13 years ago

Interesting, I'm using it for protein localization prediction, PTMs, and NGS data.

rch3• 13 years ago

NGS phage display data here.

collyw10• 13 years ago

How big is your data?

Yours is the first example where I a decent knowledge of the field, so can understand the needs accurately. In most cases I see people using NoSQL in places where MySQL could handle it easily.

Maybe I am just too set in my relational database ways of thinking (having used them for 13 years), but there are few cases where I see NoSQL solutions being beneficial, other than for ease of use(most people are not running Facebook or Google).

I a bit sceptical in most cases (though I would certainly like to know where they are appropriate).

rch3• 13 years ago

First, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though.

The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for instance, into Postgres wouldn't be impossible, but I don't know of anything that's available now.

Edit: A FDW for PostgreSQL around HMMER just made my to do list.

sidcool• 13 years ago

A host of batch jobs that source data from up to 28 different systems and then apply business rules to extract a substrate of useful data.

iurisilvio• 13 years ago

How much can I save in my mobile phone and plan?

hurrycane• 13 years ago

Billion of metric values.

willvarfar2• 13 years ago

We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users.

But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes.

Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted.

Add: down voters can explain their problem with this data point?

srl• 13 years ago

This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving.

(Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)

res0nat0r• 13 years ago

not what problem you're actually solving.
targetting expressions over billions of time series events from users.

_random_2• 13 years ago

Hearing to what problems a technology failed to solve is usually even more interesting. It cuts the hype.

pwang• 13 years ago

run targetting expressions over billions of time series events from users.

benihana• 13 years ago

How can I make myself feel better about what I do by trying to diminish the work other people do using the same technologies?

bijanv• 13 years ago

Taking dumps of analytics logs and pulling out relevant info for our customers on app usage

sitkack2• 13 years ago

This is the `grep/awk` use case. The nice thing about streaming mr interface to hadoop (calling external programs) is that you can literally take your grep/awk workflow and move it to the cluster. Retaining line oriented records is a huge step in having a portable data processing workflow.

alecco9• 13 years ago

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder.

A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem.

Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera https://www.coursera.org/course/datasci

thrownaway2424• 13 years ago

I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total.

Right tool for the job and all that. If you need to process a 50PB input though, map/reduce is the way to go.

collyw10• 13 years ago

I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out).

Does anyone have a use case where data is on a single machine and map reduce is still relevant?

(I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).

twic• 13 years ago

Mao/reduce systems

Well, that certainly sounds like ...

puts on sunglasses

... a Great Leap Forward.

AsymetricCom3• 13 years ago

Moving Big Data around is hard.

I never had any issues with Hadoop. Took about 2 days for me to familiarize myself with it and adhoc a script to do the staging and setup the local functions processing the data.

I really would like to understand what you consider "hard" about Hadoop or managing a cluster. It's pretty straight forward idea, architecture is dead simple, requiring no specialized hardware at any level. Anyone who is familiar with linux CLI and running a dyanamic website should be able to grok it easily, imho.

Then again, I come from the /. crowd, so YC isn't really my kind of people, generally.

skrebbel• 13 years ago

Then again, I come from the /. crowd, so YC isn't really my kind of people, generally.

You sound like a snob.

alecco9• 13 years ago

Is this serious? Have you ported a program to Hadoop? Unles you use Pig or one of those helping layers it is quite hard for non-trivial problems. And those helping layers usually come with some overhead cost for non-trivial cases, too.

Edit: no downvote from me.

nl2• 13 years ago

Then again, I come from the /. crowd, so YC isn't really my kind of people, generally.

W T F does this even mean? I genuinely do not understand what point you are trying to make?

I have used Slashdot longer than you have (ok, possibly not.. but username registered in 1998 here...).

I find HN has generally much more experienced people on it, who understand more about the tradeoffs different solutions provide.

The old Slashdot hidden forums like wahiscool etc were good like this too, but I don't think they exist anymore do they?

petar• 13 years ago

Consider using the gocircuit.org

It was expressly designed to avoid non-essential technical problems that come in the way of cloud application developers, when they try to orchestrate algorithms across multiple machines.

Identical HBASE SQL queries and their respective counterparts implemented in the Circuit Language are: Approximately equally long in code, and orders of magnitude faster than HBASE. In both cases, the data is pulled out of the same HBASE cluster for fair comparison.

continuations• 13 years ago

What does Go Circuit do differently that makes it orders of magnitude faster than HBase SQL?

jfxberns3• 13 years ago

"A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem."

It's not that hard to program... it does take a shift in how you attack problems.

If your data set fits on a few SSDs. then you probably don't have a real big data problem.

"Moving Big Data around is hard. Managing is harder."

Moving big data around is hard--that's why you have hadoop--you send the compute to where he data is, thus requiring a new way of thinking about how you do computations.

"Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera https://www.coursera.org/course/datasci"

Data science does not solve the big data problem. Here's my favorite definition of a big data problem: "a big data problem is when the size of the data becomes part of the problem." You can't use traditional linear programming models to handle a true big data problem; you have to have some strategy to parallelize the compute. Hadoop is great for that.

"A large telco has a 600 node cluster of powerful hardware. They barely use it."

Sounds more like organizational issues, poo planning and execution than a criticism of Hadoop!

joeblau• 13 years ago

I was using it on a D3.JS chart to aggregate data flow though our custom real-time analytic pipeline.

shamsulbuddy• 13 years ago

Will u mind sharing some more info on this real time analytics using d3 as i am also planning to do something like tht..

jackgolding• 13 years ago

look up crossfilter.js

cstigler• 13 years ago

MongoDB: where you need Map/Reduce to do any aggregation at all.

tzury• 13 years ago

You may consider using Mongo's aggregation framework[1] instead which is zillion times faster[2].

[1] http://docs.mongodb.org/manual/aggregation/

[2] http://stackoverflow.com/questions/13908438/is-mongodb-aggre...

redwood• 13 years ago

Obviously, in today's theme, Facebook is not using mapreduce effectively to figure out who of our "friends" we actually care about :)

KyeRussell• 13 years ago

Could you be any more edgy?

jread• 13 years ago

Generating web traffic summaries from nginx logs for a CDN with 150 servers, 10-15 billion hits/day. Summaries then stored in MySQL/TokuDB.

zengr• 13 years ago

We use MR using Pig (data in cassandra/CFS) with a 6 node hadoop cluster to process timeseries data. The events contain user metrics like which view was tapped, user behavior, search result, clicks etc.

We process these events to use it downstream for our search relevancy, internal metrics, see top products.

We did this on mysql for a long time but things went really slow. We could have optimized mysql for performance but cassandra was an easier way to go about it and it works for us for now.

oacgnol• 13 years ago

Can you estimate how much faster your processing is now vs. before on MySQL? I find it interesting that your cluster is only 6 nodes - relatively small compared to what I've seen and from what I've read about. It'd be interesting to know the benefits of small-scale usage of big data tech.

alecco9• 13 years ago

Did you try a columnar database first? Vertica, InfinyDB, MonetDB... There are many.

lclarkmichalek3• 13 years ago

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over.

Before someone comes along and says "this isn't big data!", I know. It's medium data at best. However, we are bound by CPU in a big way, so between throwing more cores at the problem and rewriting everything we can in C, we think we can reduce processing times to an acceptable point (currently about ~4 mins, hoping to hit <30s).

baudehlo2• 13 years ago

This sounds like something you could just do in SQL and have it all done in milliseconds.

collyw10• 13 years ago

Most of the NoSQL cases I have heard seem to be that they could be done at least as well in SQL.

I have asked a lot of questions on this topic, and no one has yet convinced me (please do if you have a legitimate NoSQL case).

lclarkmichalek3• 13 years ago

For our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.

ameen• 13 years ago

Are you talking about match stastics or player stats? Dota 2 has a logging system that records various events? during a match.

prawks• 13 years ago

Replays contain more detailed data (I believe) like player positioning and actions. I think logging is pretty limited.

lclarkmichalek3• 13 years ago

Match stats atm, though we can get much everything. We can go far beyond the combat log, reproducing an entire minimap, along with things like hero position trails, map control, etc etc. Here's a screenshot of a minimap recreation we produced for the recent Fragbite Masters tournament: http://i.imgur.com/kuOvUZX.jpg. We have more complex tools such as XP and Gold source breakdowns over time (i.e. jungle creeps, lane creeps, kills, assists etc), along with other fun stuff still in the pipeline.

tumanian• 13 years ago

Wait, it takes more then a few minutes to spin up the hadoop mapreduce job. Are you using mapreduce for preprocessing or did you optimize hadoop for this scenario?

StavrosK• 13 years ago

Which service is this for? It sounds very interesting.

batbomb• 13 years ago

I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world.

If processing latencies don't matter much, it's an easier more flexible system to use.

evanb• 13 years ago

It is built into QDP++, which is the low-level code underlying lattice QCD code bases like Chroma and MILC, though!

Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.

virtuabhi3• 13 years ago

Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc.

I do hope physics people consider MR systems in addition to the existing ones they use.

apw• 13 years ago

MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC.

It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.

mileswu• 13 years ago

At least on the experimental LHC side, we process/analyse each event independently from every other event, so it's an embarrassingly parallel workload. All we do is split our input dataset up into N files, run N jobs, combine the N outputs.

Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so many users (several thousand physicists on LHC experiments) that's why we have hundreds of GRID sites across the world. The scheduler sends your jobs to sites where the data is located. The output can then be transferred back via various academic/research internet networks.

HEP also tends to invent many of its own 'large-scale computing' solutions. For example most sites tend to use Condor[1] as the batch system, dcache[2] as the distributed storage system, XRootD[3] as the file access protocol, GridFTP[4] as the file transfer protocol. I know there are some sites that use Lustre but it's pretty uncommon.

[1] http://research.cs.wisc.edu/htcondor/ [2] http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ [4] http://www.globus.org/toolkit/docs/latest-stable/gridftp/

memracom2• 13 years ago

I remember when HEP invented that WWW technology stuff which turned out to be rather popular outside of HEP as well.

gregoryw• 13 years ago

Validating performance testing simulations. Tie the inputs of the load generators to the outputs from the application server logs and verify the system works as designed at scale.

rubyfan• 13 years ago

We are using Map/Reduce to analyze raw XML as well as event activity streams, for example analyzing a collection of events and meta data to understand how discreet events relate to each other as well as patterns leading to certain outcomes. I am primarily using Ruby+Wukong via the Hadoop-Streaming interface as well as Hive to analyze output and for more normalized data problems.

The company is a large Fortune 500 P&C insurer and has a small (30 node) Cloudera 4 based cluster in heavy use by different R&D, analytic and technology groups within the company. Those other groups use a variety of toolsets in the environment, I know of Python, R, Java, Pig, Hive, Ruby in use as well as more traditional tools on the periphery in the BI and R&D spaces such as Microstrategy, Ab Initio, SAS, etc.

clockwork_189• 13 years ago

Data aggregation to calculate analytics.

Aqueous• 13 years ago

To most people who use MapReduce in a cluster: You probably don't need to use MapReduce. You are either vastly overstating the amount of data you are dealing with and the complexity of what you need to do with that data, or you are vastly understating the amount of computational power a single node actually has. Either way, see how fast you can do it on a single machine before trying to run it on a cluster.

sitkack2• 13 years ago

Image restoration, OCR, face detection, full text indexing. Mostly just a parallel job scheduler.

DenisM2• 13 years ago

It's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say incremental updates for arbitrarily defined indexes! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database engine. I found this to be pretty clever.

smoyer2• 13 years ago

I've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.

DenisM2• 13 years ago

Strangely, there aren't many discussions of Couch* family on hacker news. Do you know why would that be?

I'm thinking about basing a new product around couchbase lite, but lack of popular acceptance is one of the things holding me back.

platz• 13 years ago

Billions

btown• 13 years ago

This is a system in early development, but my research group is planning on using MapReduce for each iteration of a MCMC algorithm to infer latent characteristics for 70TB of astronomical images. Far too much to store on one node. Planning on using something like PySpark as the MapReduce framework.

bcoughlan• 13 years ago

A lot of people in this thread are saying that most data is not big enough for MapReduce. I use Hadoop on a single node for ~20GB of data because it is an excellent utility for sorting and grouping data, not because of its size.

What should I be using instead?

grogenaut• 13 years ago

Obviously you should throw out the solution that worked for you and start over. 20GB just isn't cool enough to use M/R.

alexatkeplar2• 13 years ago

We use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowplow events containing geo-IP information, referer attribution etc. We then load those events from S3 into Redshift and Postgres.

So we are solving the problem of processing raw user behavioural data at scale using MapReduce.

All of our MapReduce code is written in Scalding, which is a Scala DSL on top of Cascading, which is an ETL/query framework for Hadoop. You can check out our MapReduce code here:

https://github.com/snowplow/snowplow/tree/master/3-enrich/ha...

sandGorgon2• 13 years ago

thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce.

have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.

alexatkeplar2• 13 years ago

Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful:

- http://stackoverflow.com/questions/6206105/running-web-fetch...

- http://petewarden.com/2011/05/02/using-hadoop-with-external-...

I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here.

We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.

pjbrunet• 13 years ago

It would be nice if someone collected/counted the actual answers. I read the whole thread and "analyzing server logs" was the only answer. If you don't have funding or have dreams/plans, you're not really _using_ the technology.

Tossrock• 13 years ago

We use our own highly customized fork of Hadoop to generate traffic graphs [1] and demographic information for hundreds of thousands of sites from petabytes of data, as well as building predictive models that power targeted display advertising.

[1] https://www.quantcast.com/tumblr.com

mrgriscom• 13 years ago

Things that could be trivially done in SQL :(

espennilsen• 13 years ago

Saw this tweet today https://twitter.com/Andst7/status/399333514803294208/photo/1

conradev• 13 years ago

Airbnb uses distributed computing to analyze the huge amount of data that it generates every day[1]. The results are used for all sorts of things: assessing how Airbnb affects the local economy (for government relations), optimizing user growth, etc.

[1] http://nerds.airbnb.com/distributed-computing-at-airbnb/

nl2• 13 years ago

While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.

In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.

That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.

HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.

jfxberns3• 13 years ago

It still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.

kirang1989• 13 years ago

Just FYI: Even small problems can be solved using the concept of MapReduce. It is a concept and not an implementation that is tied to BigData and NoSQL. A simple example would be MergeSort. It uses the concept of MR to sort data.

estebanz01• 13 years ago

Process a huge amount of spanish text in paper surveys

mfeldman• 13 years ago

"Things that could be trivially done in SQL :(" We use HIVE over HDFS. Sure the type of things we are doing could have been done in SQL, if we had carefully curated our data set and chosen which parts to keep and which to throw away. Unfortunately, we are greedy in the storage phase. Hive allows us to save much more than we reasonably would need, which actually is great when we need to get it long after the insert was made.

Ask HN: To everybody who uses MapReduce: what problems do you solve?

Related Stories