Mrjob: A Python 2.5+ package that helps you write and run Hadoop Streaming jobs

by ColinWright • 13 years ago

30 10 13 years ago

Oh hey, I work on this. It's come up on HN before, sometimes with misinformation attached. I'm happy to answer questions.

thauck• 13 years ago

Thanks for this working on this project. I've used it before with success.

I don't think python on hadoop is the answer for complex map/reduce jobs, but this project has fit an niche for me.

andrewguenther• 13 years ago

You may want to actually link to the project GitHub on your "Contribute" page

stevejohnson4• 13 years ago

Good point. https://github.com/Yelp/mrjob/pull/677

(merged.)

kevnin• 13 years ago

Does anyone know how this framework stacks up against Dumbo (Last.fm) and/or Pydoop? I've used Dumbo before and had great luck.

With Dumbo, the only problem is the lack of consolidated documentation. Much of the knowledge is lost in the maintainer's blog

stevejohnson4• 13 years ago

I don't know a thing about Pydoop, but here's what makes mrjob unique with particular respect to Dumbo:

First of all, mrjob has a lot of documentation. Some things are too hard to find in the haystack, but that's more a problem of organization than content. (Always looking for suggestions on how to improve that!) As for the rest...

mrjob tries to make all parts of Hadoop and Amazon Elastic MapReduce jobs easy to write, run, and test. It can run without Hadoop at all, it can use your own Hadoop cluster, or it will automatically handle all the pain of shipping a job to Amazon. mrjob gives you a single API to use across all those platforms and a simple way to switch between them via the command line. It helps you manage file dependencies and greps the myriad logs for errors.

Dumbo (and AFAICT Pydoop) handle the Python-Hadoop bridge, but don't do all that file management and config mishmash. They also don't provide a "Hadoop simulator" for your automated tests. If you want to use one of those tools, you need a Hadoop cluster to run your code, and they won't help you move dependencies and input files around.

The flip side to that is that mrjob doesn't give you the same level of access to Hadoop APIs that Dumbo and Pydoop do. It's simplified a great deal. But that hasn't stopped several companies, including Yelp, from using it for day-to-day heavy lifting. And in my opinion, the simplification is a good thing for most cases.

I should also mention that Dumbo can be faster if you use typedbytes. In theory mrjob could support typedbytes too, but we haven't gotten a patch we were happy with yet.

In the end, mrjob's goal is for these to all give you the same result:

  > python my_job.py -r local input.txt > output
  > python my_job.py -r hadoop input.txt > output  # assuming Hadoop configs
  > python my_job.py -r emr input.txt > output  # assuming AWS configs

mattj• 13 years ago

(original author of mrjob here)

Steve's post is 100% correct. I originally wrote mrjob as an internal tool at yelp out of my frustration with using dumbo for multi-step jobs. Specifically, I found myself writing the same incantation of "wrap a mapper / reducer function with an encoding scheme" over and over again. I tried to add protocol support into dumbo (so you could specify that your job reads json, uses pickle for intermediate data, and writes thrift), but I had a hard time working with the dumbo codebase (disclaimer: I haven't looked at it since, so it might be easy to do this now). I also wanted to represent mappers and reducers as python generators, which makes writing memory-performant steps natural (eg you normally want to rely on the shuffle / sort to perform the hard work of aggregating by key). Finally, I wanted my jobs to be easy to test both from unittest and from the command line - debugging hadoop streaming jobs is way more of a pain in the ass than it should be.

brendoncrawford• 13 years ago

MrJob is great. If you do not want to run full blown Hadoop and are not using Amazon, Gearman is also a great alternative.

miga• 13 years ago

Would be nice to have something multilanguage for users of minority languages (Erlang, Haskell, or Lisp.)

stevejohnson4• 13 years ago

It's actually pretty close. It's theoretically possible for mrjob to run a script written in any language as long as it supports a simple stdin/stdout protocol. We just don't have any users dedicated enough to implement that stuff.

Also, you can write a very small Python script that uses subprocess to call your script. That way you can still use mrjob to set up all your dependencies and handle AWS for you. If anyone is actually interested in that sort of thing, I'd be willing to write a tutorial.

Mrjob: A Python 2.5+ package that helps you write and run Hadoop Streaming jobs

Related Stories