Walkthrough of a MongoDB Map/Reduce Job

http://blog.fiesta.cc/post/10980328832/walkthrough-a-mongodb-map-reduce-job

by mdirolf • 15 years ago

37 4 15 years ago

blog.fiesta.cc

Hey Mike, do you think you could talk about how you test those js functions? Do you actually hardcode them as strings in python, or do you load them from a separate js file that you can develop / test separately. I've always been a little sketched out by the requirement that those MR jobs be written in js, would be curious to hear how you do it.

mdirolf• 15 years ago

That's a great question, and to be honest it's a bit of a pain point so I probably should've talked about it in the post.

When developing a new M/R job from scratch I start by mirroring (at least part of) the data to a local database. Then I can iterate locally on the M/R using print() and printjson() to debug the map() and reduce() functions - those will print directly to the database log.

I tend to just embed the map() & reduce() functions as Python strings like you see in the post. I'm confident that there are better ways to handle this, though. One approach that can be interesting is to do development from the shell, that way you can write and debug the map() & reduce() in an actual JS environment. Once you're happy with them you can just drop them in as strings with the rest of your application code. Would love to hear how other people are approaching this stuff, too.

All of that said, I expect that the tooling here will improve over time.

justinhj• 15 years ago

That's strange; this site is blocked at my office (we use Brightcloud.com) but it doesn't seem to have any history of problems.

dgottlieb• 15 years ago

We're using tumblr for our blog if that offers any insight. Otherwise I'm not sure what could be causing this.

Walkthrough of a MongoDB Map/Reduce Job

Related Stories