Getting Creative with MapReduce

http://sritchie.github.com/2011/09/29/getting-creative-with-mapreduce.html

by sritchie • 15 years ago

58 7 15 years ago

sritchie.github.com

epenn• 15 years ago

The Cascalog abstraction layer fixes this issue by separating logic from data, allowing you to play creatively at massive scale.

I just checked out Casacalog and I like what I see, although I have yet to try it out myself. Does anyone know of something similar that would work with Scala as well?

mitultiwari2• 15 years ago

Nice.

Check out another similar clojure library called "MR-Kluj" that you can use to write Hadoop MapReduce jobs in Clojure: https://github.com/cheddar/mr-kluj

moultano2• 15 years ago

My approach to this is to take the complicated bits of the mapreduce and put them in a separate class. Then I do a combination of two things as appropriate:

1. Hook it up to a debug server that fetches from the same datastore as the mapreduce, then test it on some keys that I'm interested in.

2. Test it like any other class.

The only awkward part of this is abstracting out the output calls, which I usually do by passing in a "handle some data" callback that outputs in the mapreduce and dumps some pretty html in the debug server.

The great part about this is that if the mapreduce ends up being something important, you already have the tools to introspect its internals on data you are interested in.

sritchie2• 15 years ago

And if it ends up being un-important, what then? My contention here is that if you can't pursue small ideas that might end up being unimportant, you're limiting yourself creatively in a big way.

These methods exemplify the sort of complexity I was getting at in the blog post; how many tests do you end up writing if you need a full debug server?

Midje and Cascalog make it possible to do test-driven development at large scale. Is anyone else in MapReduce able to practice TDD? Really curious here.

moultano2• 15 years ago

I find writing a debug server easier than writing tests, but maybe that's just me. :) Especially for speculative things, the notion of whether the code is "correct" could change from moment to moment, so I find it much more useful to look at what it's doing than to test if what it's doing matches a spec that doesn't exist.

mitultiwari2• 15 years ago

I use Pig to write many of my Hadoop Mapreduce jobs and test my Pig script on a small dataset using Pig Grunt. Pig 0.9 has a lot more debug information as well.

Getting Creative with MapReduce

Related Stories