Bored in grad school? Learn Hadoop

http://commoncrawl.org/learn-hadoop-and-get-a-paper-published/

by Aloisius • 14 years ago

52 21 14 years ago

groundshop• 14 years ago

It's the start of the Summer, I'm in my 2nd year MS-CS. I began a simulation last night at 9PM that's still running (usually 14 hour turnaround). I just got in, sat at my desk and thought "I'm bored, guess I'll check HN" and THIS is what I see.

malloc47• 14 years ago

I've been authoring a post on solving interview questions in map/reduce between experimental runs myself, so I guess I already hit the "bored graduate student" point too...

oacgnol2• 14 years ago

I remember having to resist the urge to map/reduce everything in interview questions :P. Somehow, I don't think interviewers want to hear how you'd use a cannon to solve a tiny problem.

Bootvis• 14 years ago

Down for me, can someone paste the contents?

denzil_correa• 14 years ago

Down for me too. I can't see a cached version in Google too.

brucehart• 14 years ago

I was able to find a cached version here: http://webcache.googleusercontent.com/search?q=cache:http://...

migpwr• 14 years ago

I apologize for the off-topic comment but I am hoping a few of the folks on here who are familiar with Hadoop can help me with a small career decision.

I'm fortunate enough to be up for two systems positions with companies in my area, and one of them is part of a group maintaining a Hadoop cluster. I've never maintained Hadoop infrastructure before so I'm wondering if it's worth the "career capital" investment. There don't appear to be too many openings that I know of for Hadoop Sys admins, and while I don't expect to lose any skills working on it, I wonder if this platform is realistically expected to grow, and maybe become something I could maybe build a valuable niche set of skills for...

Any help would be appreciated...

oacgnol2• 14 years ago

I would say go for it. Hadoop is becoming more of a norm in industry and I can tell you from personal experience that having prior hands-on Hadoop work makes your resume pop a little more.

Even better is if you're able to muck around in core Hadoop versus abstracted management layers like Cloudera. The things you have to learn in maintaining a Hadoop cluster like cloud systems management, tuning, running jobs, etc. makes for valuable experience, even if niche.

throwaway19792• 14 years ago

I don't see exploding demand in this. However, the experience will likely translate over to other domains and make you better for it. I'm not a sys admin but setup a small hadoop cluster ... but wow ... talk about learning experiences!

daviddumenil• 14 years ago

Definitely the right direction:

http://www.itjobswatch.co.uk/jobs/uk/hadoop.do

http://www.pcworld.com/businesscenter/article/255142/idc_exp...

http://www.mckinsey.com/Insights/MGI/Research/Technology_and...

Although I'd second the point that getting your hands dirty with the Hadoop core is likely more valuable than straight sysadmin work.

cwhittle• 14 years ago

While learning Hadoop is a good idea, if you're bored in grad school, you're doing it wrong.

achompas• 14 years ago

Agreed 100%. Grad school is a time to explore all kinds of CS fields (or various topics within a field you're interested in). Right now I wish I had 28-hour days to mess around with GPU computing, compiler design, language theory, and random machine learning projects.

EDIT: its also worth pointing out that Hadoop is not the 'be all, end all' framework for large-scale data analysis. Many problems do not fit the map-reduce paradigm, and are better suited for other frameworks (looking into GraphLab once I get free time, for example).

srean• 14 years ago

Hahaha I hear you. "Me too"s are frowned upon here, but I could not resist. There are so many things that I want to learn that I wish sleep was just optional.

I know Hadoop is the poster child of all things good, but its API really makes me fall asleep. Not a big fan. Add to that the fact that Google's implementation is (or atleast used to be) 4~6 times faster for similar sized processing jobs and more fun to code in (the latter may be entirely subjective). The funny part is that Google's clusters then were made of weaker machines ! I dont know how it is now.

EDIT: It is indeed in C++ that alone cannot fully explain the discrepancy though. Java can be slower but shouldnt be that much slower. I am sure caching, memory footprint and as you said data access latencies and overall design plays a big role. If I remember correctly UIUC has an open source mapreduce framework written in C++ and they claim a similar speedup over hadoop.

scott_s• 14 years ago

You're right, but I imagine most grad students will inevitably go through a "doing it wrong" phase. Mine lasted close to a year.

LisaG2• 14 years ago

I am from Common Crawl. Apologies for the site being down! Too much traffic from HN :) We're working on getting it back up. The Google cache below has all the contents, so please refer to there for the moment. Here's the excerpted beginning..

Learn Hadoop and get a paper published

We’re looking for students who want to try out the Hadoop platform and get a technical report published. Hadoop’s version of MapReduce will undoubtedbly come in handy in your future research, and Hadoop is a fun platform to get to know. Common Crawl, a nonprofit organization with a mission to build and maintain an open crawl of the web that is accessible to everyone, has a huge repository of open data – about 5 billion web pages – and documentation to help you learn these too

http://webcache.googleusercontent.com/search?q=cache:http://...

RobAtticus• 14 years ago

Perhaps I'm just jaded from NSDI a few weeks ago, but I'm tired of papers about Hadoop and systems like it. If you're going to do work with Hadoop, at least start comparing what you're doing with other work that improves Hadoop. There's been a bunch of papers about improving the scheduling or the shuffle phase or whatever but they all compare to vanilla Hadoop and not each other.

LisaG2• 14 years ago

Site is back up. Thanks for your patience!

Bored in grad school? Learn Hadoop

Related Stories