Command-line tools can be faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

by wglb • 12 years ago

812 309 12 years ago

greenyoda• 12 years ago

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal.

    sleep 3 | echo "Hello world."

That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked.

    cat *.pgn | grep "Result" | sort | uniq -c

Does this have any advantage over the more straightforward verson below?

    grep -h "Result" *.pgn | sort | uniq -c

Either the cat process or the grep process is going to be waiting for disk I/Os to complete before any of the later processes have data to work on, so splitting it into two processes doesn't seem to buy you any additional concurrency. You would, however, be spending extra time in the kernel to execute the read() and write() system calls to do the interprocess communication on the pipe between cat and grep.

Also, the parallelism of a data processing pipeline is going to be constrained by the speed of the slowest process in it: all the processes after it are going to be idle while waiting for the slow process to produce output, and all the processes before it are going to be idle once the slow process has filled its pipe's input buffers. So if one of the processes in the pipeline takes 100 times as long as the other three, Amdahl's Law[1] suggests that you won't get a big win from breaking it up into multiple processes.

[1] https://en.wikipedia.org/wiki/Amdahl%27s_law

Edit: As someone pointed out, my example needed "grep -h". Fixed.

petemc_• 12 years ago

You could skip both cat and grep and do it all in awk. Also if speed was an issue you would want to make sure LANG=C is set for grep.

Edit: I see they did use awk later in article, I should really read all of things before commenting.

omoikane• 12 years ago

"grep <pattern> <files>" is not the same as "cat <files> | grep <pattern>", in that the former will prefix lines with filenames if there is more than one input file. What you want instead is "grep -h <pattern> <files>".

The advantage of using cat, therefore, is the few seconds of laziness saved in not reading the manual.

salgernon• 12 years ago

The advantage to using "cat foo | grep pattern" is that it is trivial to ^p and edit the pattern before adding the next pipeline sequence.

clarkm2• 12 years ago

You can make it even faster by using fgrep since you're not searching for a regex.

simpleigh• 12 years ago

The article addresses this:

The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing.

I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.

aadrake3• 12 years ago

Thank you for the comment. The fact that it can run independently as soon as it's forked was exactly the point I was making.

Also, the cat | grep pipeline is illustrative. I remove it at the end.

sgt1012• 12 years ago

on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.

simonster• 12 years ago

If you want disk parallelism, RAID 0 is probably easier than Hadoop.

cmurf• 12 years ago

That would depend on the data set and the strip size. Striping is good for streaming. Linear/concat of 2+ drives with XFS would be faster with a lot of files than end up in separate AG's on separate drives which can be accessed in parallel.

KaiserPro2• 12 years ago

Depends on the dick, depends on the storage.

HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.

In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.

In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

radoslawc• 12 years ago

Depends on the dick

not really, sorry I had to

back to the topic, HDFS is really somewhat waste of disk space, especially when used for something like munching logs

At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

depends, if you need monthly reports from logs, as long as you don't loose storage completely, then using even second hand hardware or decommissioned from prod is cheapest choice

JensRantil2• 12 years ago

Reminds me of "filemap" - a commandline-like map/reduce tool: https://github.com/mfisk/filemap

JensRantil2• 12 years ago

Related: sometimes I wish most of my data was in JSON streams so I could simply map and reduce the data using jq (http://stedolan.github.io/jq/), pipes and possibly filemap.

crcsmnky• 12 years ago

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with?

The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they can sound intelligent with a certain segment of the population.

That said, I like seeing how people put together their own CLI-based processing pipelines.

EpicEng2• 12 years ago

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

texthompson• 12 years ago

That's not even medium data. Most people probably would be surprised to find out that their data could be stored and processed on an iPhone, and that using heavier duty tools isn't necessary or worthwhile.

volker48• 12 years ago

If the data can fit on a thumb drive it's not big data.

underpants• 12 years ago

In a similar situation. In fact stupider. We have a 120Gb baseline of data inside a relational store. The vendor has a file stream option that allows blobs to be stored on disk instead of the transaction log and be pushed through the DB server rather than using our current CIFS DFS cluster. So lets stick our 950Gb static document load in there too (while i was on holiday typically) and off we go.

Do thing starts going like shit as expected now it has to stream all that through the DB bottleneck.

So what's the solution? Well we're in big data territory now apparently at 1.2TiB (comedically small data and almost entirely static data set) and have every vendor licking arse with the CEO and CTO to sell us Hadoop, more DB features and SAN kit.

We don't even need it for processing. Just a big CRUD system. Total Muppets.

dalke• 12 years ago

Indeed. For more real-world examples of people who thought that they had "big data", but didn't, see https://news.ycombinator.com/item?id=6398650 ("Don't use Hadoop - your data isn't that big"). The linked essay has:

They handed me a flash drive with all 600MB of their data on it (not a sample, everything). For reasons I can't understand, they were unhappy when my solution involved pandas.read_csv rather than Hadoop.

User w_t_payne commented:

I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth.
All of them wanted to feel like they were doing something special.

I think that last line is critical to understanding why a CIO might feel this way.

azernik• 12 years ago

Perhaps a good way of explaining it to management is, "If it fits on a smartphone SD card, it's tiny data. If it fits on a laptop hard drive, it's maybe medium data." I think at that point many of these conversations would end.

threeseed5• 12 years ago

Another explanation is that your CIO is not an idiot but rather they know about future projects that you don't. CIOs want to build capabilities (skills and technologies) not just one off implementations every time.

Not saying this is the case but CIO bashing is all too easy when you're an engineer.

asuffield• 12 years ago

I have had this exact conversation at various past employers, usually when they started talking about "big data" and hadoop/friends:

"How much data do you expect to have?"

"We don't know, but we want it to scale up to be able to cover the whole market."

"Okay, so let's make some massive overestimates about the size of the market and scope of the problem... and that works out to about 100Mb/sec. That's about the speed at which you can write data to two hard drives. This is small data even in the most absurdly extreme scaling that I can think of. Use postgres."

Even experienced people do not have meaningful intuitions about what things are big or small on modern hardware. Always work out the actual numbers. If you don't know what they are then work out an upper bound. Write all these numbers down and compare them to your measured growth rates. Make your plans based on data. Anything that you've read about in the news is rare or it wouldn't be news, so it is unlikely to be relevant to your problem space.

x0x03• 12 years ago

I've used hadoop at petabyte scale (2+pb input; 10+pb sorted for the job) for machine learning tasks. If you have such a thing on your resume, you will be inundated with employers who have "big data", and at least half will be under 50g with a good chunk of those under 10g. You'll also see multiple (shitty) 16 machine clusters, any of which -- for any task -- could be destroyed by code running on a single decent server with ssds. Let alone hadoop jobs running in emr, which is glacially slow (slow disk, slow network, slow everything.)

Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers. I imagine it's similar to early ejb coding.

IndianAstronaut• 12 years ago

Imho, hadoop is only for 100tb or more. Anything less can be easily handled by other tools.

sedachv2• 12 years ago

Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers.

It's comical how bad Hadoop is compared even to the CM Lisp described in Daniel Hillis' PhD dissertation. How do you devolve all the way from that down to "It's like map/reduce. You get one map and one reduce!"

jbergens4• 12 years ago

It sounds a little bit like BizTalk :-)

sputknick• 12 years ago

Lots of people think that is "big data". For most people if it's too big for an Excel spreadsheet, it's "big data" and the way you process big data is with Hadoop. Of course once you show them the billable hours difference between setting up a Hadoop cluster, and (in my case at least) using python libraries on a MBP, they change their minds real fast. Its just a matter of "big data" being a new thing, people will figure it out as time goes on and things settle down.

theVirginian• 12 years ago

"Although Tom was doing the project for fun, often people use Hadoop and other so-called Big Data (tm) tools for real-world processing and analysis jobs that can be done faster with simpler tools and different techniques."

I think the point the author is making is that although they knew from the start that Hadoop wasn't necessary for the job, many people probably don't.

mrweasel• 12 years ago

People love the idea of being big and having "big problems". Wanting to use Hadoop isn't that different from wanting to use all sorts of fancy "web-scale" databases.

Most of us don't have scaling issues or big data, but that sort of excludes us from using all the fancy new tools that we want to play with. I'm still convinced that most of the stuff I work on at work could be run on SQLite, with designed a bit more careful.

The truth is that most of us will never do anything that couldn't be solved with 10 year old technology. And honestly we should happy, there's a certain comfort in being able to use simple and generally understood tools.

danso• 12 years ago

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens...making it much easier to interactively iterate through a process.

That it also happens to very fast and powerful (when memory isn't a limiting factor) is nice icing on the cake. I moved over to doing much more on CLI after realizing that doing something as simple as "head -n 1 massive.csv" to inspect headers of corrupt multi-gb CSV files made my data-munging life substantially more enjoyable than opening them up in Sublime Text.

hanoz• 12 years ago

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Someone• 12 years ago

Running a shell in a GUI doesn't make it lose its "I am a CLI" property. That is a CLI.

hueving• 12 years ago

Do you not see the horrific syntax of what you just suggested as simple?

TylerE• 12 years ago

Or just use vim or any other editor smart enough not to try to slurp the whole file in one go.

alrs2• 12 years ago

The example was a multi-gigabyte CSV file. You just sucked the whole thing off the disk into RAM so that you could shave off the first line.

If you're unlucky, you started swapping out to disk about halfway through.

Spooky23• 12 years ago

A few years ago between projects, my coworkers cooked up some satirical amazing Web 2.0 data science tools. They used git, did a screencast and distributed it internally.

It was basically a few compiled perl scripts and some obfuscated shell scripts with a layer of glitz. People actually used it and LOVED it... It was supposedly better than the real tools some groups were using.

It was one of the more epic work trolls I've ever seen!

jefftk• 12 years ago

Maybe I'm misreading you, but it sounds like you're saying "my coworkers made something with a really great UI and people loved it!"

wallflower• 12 years ago

Awk and Sed aren't very accessible to most people who did not grow up learning those tools.

The whole point of tools built on top of Hadoop (Hive/Pig/HBase) is to make large scale data processing more accessible (by hiding the map-reduce as much as possible). Not everyone will want to write a Java map-reduce in Hadoop. However, many can write a HiveQL statement or Pig textual script. Amazon Redshift brings it even farther - they are a Postgres compatible database, meaning you can connect your Crystal Reports/Tableau data analysis tool to it, treating it like a traditional SQL database.

nfm• 12 years ago

I think the author's point was that the example in question was orders of magnitude smaller than "big data" and that it was more efficient to process it on a single machine, not that Hadoop and friends aren't easy to use.

NyxWulf• 12 years ago

This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon for my jobs to be much faster than the same jobs run on the cluster with Hive or some other framework. Much of the speed boils down to the optimizer and the planner.

Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.

azylman• 12 years ago

What toolset are you using that you can run both locally and on a Hadoop cluster?

arjie• 12 years ago

I believe he mentioned it. The Hadoop streaming mode.

mdaniel• 12 years ago

Almost all of them?

The vocabulary of the grandparent comment implies they are using hadoop's streaming mode, and thus one can use a map-reduce streaming abstraction such as MRJob or just plain stdin/stdout; both will work locally and in cluster mode.

Or, if static typing is more agreeable to your development process, running hadoop in "single machine cluster" mode is relatively painless. The same goes for other distributed processing frameworks like Spark.

exabrial• 12 years ago

tl;dr: You do not have a big data problem.

contingencies• 12 years ago

I would have said Right tool for the job.

cortesoft• 12 years ago

I am glad to know the 4 million requests per second I am processing isn't big data...

liotier• 12 years ago

'xargs -n' elicits fond memories of spawning large jobs to my Openmosix cluster ! I miss Openmosix.

pjmlp• 12 years ago

Oh! I lost track of it.

dkarapetyan• 12 years ago

Oh ya and it turns out when all is said and done the average data set for most hadoop jobs is no more than 20GB which can again fit comfortably on a modern desktop machine.

pkrumins• 12 years ago

The example in the article with cat, grep and awk:

    cat *.pgn | \
    grep "Result" | \
    awk '
     {
        split($0, a, "-");
        res = substr(a[1], length(a[1]), 1);
        if (res == 1) white++;
        if (res == 0) black++;
        if (res == 2) draw++;
      }
      END { print white+black+draw, white, black, draw }
    '

Can be written much more succinctly with just awk, and you don't even need to split the string or use substr:

    awk '
      /Result/ {
        if (/1\/2/) draw++;
        else if (/1-0/) white++;
        else if (/0-1/) black++;
      }
      END { print white+black+draw, white, black, draw }
    ' *.pgn

dice2• 12 years ago

Keep reading, he removes the cat and grep in the final solution.

omaranto• 12 years ago

Yes, but he still keeps the awkward Awk code with the substr and such. I haven't benchmarked, maybe that's faster than the pretty regex matches.

vander_elst• 12 years ago

until ~10 GB you'd better keep on going with single core machines, you'll see some improvementes with bigger sets > 100 GB

kylek• 12 years ago

I feel ag (silver surfer, a grep-ish alternative) should be mentioned (even though he dropped it in his final awk/mawk commands) as it tends to be much faster than grep, and considering he cites performance throughout.

ggreer2• 12 years ago

GitHub link for those who don't know about it: https://github.com/ggreer/the_silver_searcher/

I built ag for searching code. It can be (ab)used for other stuff, but the defaults are optimized for a developer searching a codebase. Also, when writing ag, I don't go out of my way to make sure behavior is correct on all platforms in all corner cases. Grep, on the other hand, has been around for decades. It probably handles cases I've never even thought of.

a3_nm• 12 years ago

I think it is unsafe to parallelize grep with xargs as in done in the article, because, beyond delivery order shuffling, the output of the parallel greps could get mixed up (the beginning of a line is by one grep and the end of a line is from a different grep, so, reading line by line afterwards, you get garbled lines).

See https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

haddr• 12 years ago

great article! PS. probably some hardcore unix guy would tell you that you are abusing cat. The first cat can be avoided, and you might gain even better performance. Also using gnu grep seems to be faster.

aadrake3• 12 years ago

Thank you for the compliment. Point taken on cat, but that's the way I like to introduce the process to people. I took cat and grep out at the end of the article anyway.

Swizec• 12 years ago

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's.

As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI.

It always comes a surprise when somebody suggests that there are professional developers out there who do not use predominantly CLI.

frozenport• 12 years ago

Indeed, there is a part of me that hurts when people mention teaching something that is a signifier of legitimacy. As if you could fake being a real programmer by listening to a lecture in CS 102!

mech4bg3• 12 years ago

This shouldn't be a surprise. Tons of development is done on Windows. Most game development, obviously Windows app development, .NET websites, etc.

There are command line tools there, but in my 10 years of being a Windows developer, GUI tools were more the norm.

There's a time and a place for both. Now developing predominantly under Linux, it amazes me how time consuming and clunky some tasks are on the command line compared to using a GUI (e.g. debugging, Visual Studio is just a fantastic IDE), but also how much faster and easier other tasks are with a CLI.

kyllo4• 12 years ago

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

prodigal_erik• 12 years ago

People are always surprised when I mention that the Microsoft devs I worked with had free access to the highest tiers of Visual Studio, yet what they actually worked in was vim and the internal fork of make. I don't know whether that's still true; it's been a decade now.

wglb4• 12 years ago

When you say "Microsoft Devs", do you mean MS employees?

If so I heard s similar tale back in the first days of .net, where most of the guys in one particular group were using emacs.

wrs• 12 years ago

When I was an MS dev (8 to 18 years ago), Visual Studio had nothing to do with "real work" on Windows systems programming.

Quite a few people used a rather obscure editor called Source Insight (http://www.sourceinsight.com/) because of its code-navigation abilities, which were similar to an IDE's but worked on huge codebases that would take hours to actually parse and analyze "properly". Sort of a supercharged ctags.

jmcphers• 12 years ago

It's still true (source: Microsoft engineer for 8 years, used vim and internal version of Unix tools).

A lot of this has to do with the sheer size of many of Microsoft products' codebases. Visual Studio just can't handle projects with millions of lines of code, whereas vim + ctags elegantly handles the fragments of projects you build locally.

UrMomReadsHN• 12 years ago

15 years ago when i first started programming i learned on the *nix CLI. I worked that way for years, thats just how it was done. Well, I started using an IDE in windows and now.... I really like it. I don't want to go back to the command line. That being said, I still use the Windows command line from time to time.

roel_v2• 12 years ago

Yeah me too. I do quite a bit of 'bit bigger than average' data processing, and I write the tools in cmd.exe or as command line C++ applications first - and then when they work and I need to run them more often than a few times, I made a small GUI in front of it so that I don't have to spend half an hour every time to set up the parameters right.

wglb4• 12 years ago

Every serious programmer I know, especially those who are about my age, lives their life in the CLI.

This is not even close to true. Of the developers that I know, many of them quite serious, live in the Windows world, and are very happy with for example VS or eclipse. This is very likely to be the case in most BFEs.

JabavuAdams4• 12 years ago

Every serious programmer I know

So, you know all the same programmers the parent poster knows? The parent explicitly stated that this was anecdotal, and from (not your) personal experience.

Blackthorn• 12 years ago

Can confirm, was there in high school at the same time and had the same experience :-)

I still use programming languages like AWK to this day (I even list it as a known language on my resume; someone commented on it once, it's always a +1 for the company if they do). Recently though I've been exposed to some stuff in the Windows world that makes me think that, sure, they evolved slower but they evolved in a really interesting direction and possibly pulled themselves out of the local maxima that is the Unix world. (I'm referring to PowerShell.)

It's a little weird to me when devs don't immediately go to the command line, given that's just how I learned, but you've gotta recognize that it's not a flaw that they don't. Everyone just learned a little differently.

zokier• 12 years ago

Author begins with fairly idiomatic shell pipeline, but in the search for performance the pipeline transforms to a awk script. Not that I have anything against awk, but I feel like that kinda runs against the premise of the article. The article ends up demonstrating the power of awk over pipelines of small utilities.

Another interesting note is that there is a possibility that the script as-is could mis-parse the data. The grep should use '^\[Result' instead of 'Result'. I think this demonstrates nicely the fragility of these sorts of ad-hoc parsers that are common in shell pipelines.

tracker1• 12 years ago

It probably depends on what you are trying to accomplish... I think a lot of us would reach for a scripting language to run through this (relatively small amount of data)... node.js does piped streams of input/output really well. And perl is the grand daddy of this type of input processing.

I wouldn't typically reach for a big data solution short of hundreds of gigs of data (which is borderline, but will only grow from there). I might even reach for something like ElasticSearch as an interim step, which will usually be enough.

If you can dedicate a VM in a cloud service to a single one-off task, that's probably a better option than creating a Hadoop cluster for most work loads.

zobzu3• 12 years ago

Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work...

The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur own financially).

I suspect its a general lack of understanding the way computers work (hardware, os ie system architecture) vs "why care it works and python/go/java/etc are easy for me i dont need to know what happens under the hood".

dice2• 12 years ago

In the final solution at the end of the article there are only two pipes:

1. A pipe to feed the file names into xargs for starting up parallel `mawk` processes.

2. A pipe to a final `mawk` process which aggregates the data from the parallel processes.

There's still some performance that could be gained by using a single processes with threads and shared memory, but this is pretty good for something that can be whipped together quickly.

zobzu3• 12 years ago

Yeah its not bad. In the final command, it is basically leveraging mawk for everything which works out well since there's fewer pipes.

But in this case its about replacing hadoop with mawk basically. Which is indeed a good point as well - and incidentally also confirms my own comment =)

gasping• 12 years ago

incl. an acid db.

Why would you want to use a database for this problem? The input data would take time to load into an ACID db and we're only interested in a single ternary value within that data. The output data is just a few lists of boolean values so it has no reason to be in a database either.

This is a textbook stream processing problem. Adding a database creates more complexity for literally no benefit assuming the requirements in the linked article were complete. I would be baffled to see a solution to this problem that was anything more than a stream processor, to say nothing of a database being involved.

_delirium3• 12 years ago

If it really is just a one-shot with one simple-ish filter, I agree. But I often find myself incrementally building shell-pipeline tangles that are sped up massively by being replaced with SQLite. Once your processing pipeline is making liberal use of the sort/grep/cut/tee/uniq/tac/awk/join/paste suite of tools, things get slow. The tangle of Unix tools effectively does repeated full-table scans without the benefit of indexes, and is especially bad if you have to re-sort the data at different stages of the pipeline, e.g. on different columns, or need to split and then re-join columns in different stages of the pipeline. In that kind of scenario a database (at least SQLite, haven't tried a more "heavyweight" database) ends up being a win even for stream-processing tasks. You pay for a load/index step up front, but you more than get it back if the pipeline is nontrivial.

zobzu3• 12 years ago

The interesting part is that its still faster, not that its the best-case solution. The main reason is that the data set fits in memory and is no slower to load (you need to read the data in all cases, duh. Both piped and db will read the data from disk exactly once in a sequential fashion).

There is no locking issue, and you can be smart in the filtering steps (most dbs do some of that automagically anyway). You don't have that level of control with the pipes, you are limited by the program's ability to process stdin, and additional locking.

This is exactly where knowing how things really work under the hood give you an advantage vs "but in theory..". You can reimplement a complete program, or even set of programs that will outperform the db abd the piped example. But will you? No, you want the best balance between fastest solution with the least amount of work.

sam_lowry_• 12 years ago

Next to using `xargs -P 8 -n 1` to parallellize jobs locally, take a look at paexec, GNU parallel replacement that just works.

See https://github.com/cheusov/paexec

pmoriarty• 12 years ago

What's the advantage of using paexec over GNU parallel?

ole_tange• 12 years ago

See comparison here: http://www.gnu.org/software/parallel/man.html#DIFFERENCES-BE...

smegel• 12 years ago

What about if you are processing 100 Petabytes? And you are comparing to a 1000-node Hadoop cluster with each node running 64 cores and 1TB of main memory?

ggreer2• 12 years ago

Right tool for the right job. 100 petabytes is 50,000,000 times larger than the data in the post. It's the difference between touching something within reach and flying around the world.[1]

1. Earth is 40 megameters in circumference. 40Mm / 50M = 0.8m

dundun2• 12 years ago

Then you're hardly using commodity hardware anymore. While jobs like that probably actually work on Hadoop, I'd imagine a problem like that might be better suited for specialized systems.

TallGuyShort2• 12 years ago

IME, most installations where Hadoop is "successfully" used it's running on pretty high-end machines. "Commodity hardware" really means standard hardware, not cheap hardware (as opposed to buying proprietary appliances and mainframes).

mabbo• 12 years ago

I had an intern over the summer, working on a basic A/B Testing framework for our application (a very simple industrial handscanner tool used inside warehouses by a few thousand employees).

When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't maintainable or reusable. shrug It happens. I had worse internships.

I put together some command line scripts to parse the files instead- grep, awk, sed, really basic stuff piped into each other and written to other files. They took 10 minutes or so to process, and provided reliable answers. The scripts were added as an appendix to the report I provided on the A/B test, and after formatting and explanations, took up a couple pages.

arthurcolle• 12 years ago

I used Hadoop a few times this semester for different classes and it seemed like the code was so easy to write and then because everything is either a Mapper or a Reducer, you just read enough of the docs to figure out what is intended to be done and then build on top of it, can I ask how it wasn't maintainable?

Just curious

m_mueller• 12 years ago

On a tangent, I'd be interested in how you format heavily piped bash code for documentation. Can comments be intersparsed there?

mappu• 12 years ago

Functions, mostly - the big `awk` command in the example goes into something like

    # @param $1 whatever
    chess_extract_scores() {
         awk blah blah blah
    }

and then your whole pipeline simplifies to

    cat foo | grep bar | chess_extract_scores

which is pretty readable. You can even do most of this in a live bash session with ^X ^E.

dsr_• 12 years ago

bash has functions; functions are just like commands. Write your comments in the functions, and your final line will be the pipeline of awesomeness:

   generate_data ()
       {
        # make it rain
       }

   process ()
       {
        # chunky
       }

   gather ()
       {
        # puree
       }
  
   generate_data | process | gather

dschiptsov2• 12 years ago

Everyone with basic knowledge of CS could realize that Hadoop is a waste.

Unfortunately, it isn't about efficiency at all. It just memeization. Bigdata? Hadoop! Runs everywhere. Same BS like Webscale? MongoDB! meme.

threeseed5• 12 years ago

Well sorry but you don't have a clue what you're talking about.

I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce.

There isn't anything on the market today that has been proven to work in environments like this and has the tooling to back it up. Unless you want to prove everyone e.g. Netflix, Linkedin, Spotify, Apple, Microsoft wrong ?

wglb4• 12 years ago

Well sorry but you don't have a clue what you're talking about.

From the Guidelines:

Be civil. Don't say things you wouldn't say in a face to face conversation.

When disagreeing, please reply to the argument instead of calling names.

dschiptsov2• 12 years ago

The point here is that the same functionality could be implemented way more efficiently even with shell scripts.

The idea of using standard UNIX tools for the showcase is good one. Basically, it tells you that a modern FS is very good at storing chunks of read-only data (one don't need Java for that) with efficient caching and in-kernel procedures. That using pthreads for jobs is a waste, because context-switching has its costs, etc.

To put it simple - by mere rewriting basic functionality in, say, Erlang, one could get orders of magnitude more efficient implementation.

The only selling point of Hadoop is that it exist (mature, stable, blah-blah). It also has one problem - Java. But as long as hardware is cheap and credit is easy - who cares?

gd1• 12 years ago

There isn't anything on the market today that has been proven to work in environments like this and has the tooling to back it up.

Here you go:

http://kx.com/

sukilot• 12 years ago

A few large organizations and microscpic scientists have big data. 99.9% of users do not.

MrBuddyCasino2• 12 years ago

To quote the memorable Ted Dziuba[0]:

"Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.

The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script."

[0] since his blog is gone: http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...

make3• 12 years ago

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

jessaustin• 12 years ago

Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.

untog• 12 years ago

You usually don't need that though, that's the point. If you're building an entire service around page crawling, sure. If you're doing a one-off task, don't bother.

MrBuddyCasino2• 12 years ago

That would be my first gut reaction too, but if it is as simple as downloading webpages, this is actually a really great solution. I suspect he used that when he built Milo, a now defunct startup sold to eBay where they had to update prices and inventory data regularly. A startup should make different choices than Google.

threeseed5• 12 years ago

Oh right the "cool kids" approach.

Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.

x0x03• 12 years ago

Sure -- you can level multiple complaints.

What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc

But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up emr, installing and configuring nutch, figuring out network issues around emr nodes talking to commodity internet, installing a persistent queue, performing remote debugging, building a task dag in either code or (god help you) oozie/xml, and on and on.

saidajigumi• 12 years ago

Dreadful for the long term.

Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipelines. Sure, there's a discipline to building pipelined tooling well, just as with any other kind of software. Your problems seem to stem from a lack of disciplined software engineering rather than the tools, or maybe just an environment that encouraged high technical debt.

The kicker? We were using pipeline-based tooling ... running on a Hadoop cluster. Honestly, I'm a bit surprised to see such an apparent mindshare split (judging by some recent HN posts) between performant single-system approaches and techniques used in-cluster. The point that "be sure your data is really, truly big data" is obviously well made, and still bears repetition. Yet the logical follow-on is that these technique are even more applicable to cluster usage. Why would anyone throw away multiple orders of magnitude performance going to a cluster-based approach?

AnthonyMouse2• 12 years ago

The Unix people have thought of these things. You can easily do them with command line tools.

Operational Supportability: How do you monitor the operation ?

Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget.

Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ?

wget has command line options to skip downloading files that already exist. Or you can use tail to skip the number of lines in the input file as there exist complete entries in the destination directory.

Maintainability: Can we run the same application on our desktop as on our production servers ?

I'm not sure how this is supposed to be an argument against using the standard utilities that are on everybody's machine already.

Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

Again, what? Extensibility is the wheelhouse of the thing you're complaining about.

gpapilion3• 12 years ago

I look at this article as a criticism of the hadoop being the wrong tool for small data sets.

This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change.

The fact that shell commands were used makes for an easy demo that might be hard to support, but if a solution were written using a traditional language with threading or IPC instead of relying on hadoop you should always be faster, since you don't incur the latency costs of the network.

Symbiote• 12 years ago

'pv' shows a progress bar, something like '&& touch $x.success' can help restart recovery.

I'd probably pick the shell approach for something I expect to be a one-off, but reconsider each time the task is repeated.

I printed http://xkcd.com/1205/ and stuck it to the wall. It's a useful reference when someone seems to be under or overengineering something.

coldtea• 12 years ago

Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

Yeah, and then they produce some over-engineered monstrocity, late, over-budget and barely able to run...

pacala3• 12 years ago

I fully agree with all the operational / restart / features comments. However, I've often been surprised on how a little thought / research can build all these requirements on top of off-the-shelf components. I also agree that it is likely that one will eventually outgrow wget, but, for example, one may run out of business / pivot before that.

JustSomeNobody5• 12 years ago

I hate developers who over engineer everything and then when it's time to perform some of that support and extensibility, they leave because maintenance is beneath them.

They put this behemoth together with a thousand moving parts and then walk away from it.

This, too, happens often.

na85• 12 years ago

I agree, it's much better to re-invent the wheel.

mwotton• 12 years ago

I've implemented this. It isn't too bad up to a certain point. You have to be a bit careful about your filesystem/layout of files, lots of filesystems don't particularly like it when you have a few hundred million files in one directory.

Kiro• 12 years ago

And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need to.

charltones• 12 years ago

There are command line tools available to help the transition from 'hack' one liner to a more maintainable / supportable solution. For instance drake (https://github.com/Factual/drake) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline.

The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning one liners into extensible shell scripts, using drake, using Gnu Parallel.

2ion2• 12 years ago

Just these pointed remarks:

- There are things like pv(1) which allow you to monitor pipes. Things like systemd open other interesting possibilities for implementing, grouping and monitoring your processes.

- Recovery could be implemented by keeping a logfile of completed steps like a list of completely processed files or moving processed files to elsewhere in the file system (could be done in memory only using ramfs or tmpfs). Of course, it depends on the case whether it's feasible or not.

- Extensibility: Scripts and configurations can be done in shell syntax. Hook systems and frameworks of varying complexity exist. I agree that doing extensibility in shell code is going to turn out to be hazardous when done without proper concept and understanding of the tool at hand.

debacle• 12 years ago

What kind of operational supportability do you need for a script that took a few hours to write and takes <5 minutes to run?

module0000• 12 years ago

We don't really "come up" with the xargs/wget approach. The approach is already there, waiting to be utilized by someone who understands the tools. The "cool kids" don't like(or are not able) to understand the tools.

The author (I think) is trying to point out that these problems are already solved, decades ago, with existing UNIX tools.

profquail• 12 years ago

archive.org has Ted's blog post, "Taco Bell Programming": https://web.archive.org/web/20101025124303/http://teddziuba....

lloeki4• 12 years ago

Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval).

zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps).

My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.

sebastianavina• 12 years ago

to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.

wmt• 12 years ago

Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.

You should always aim at working with people who are smarter or better than you. Unless they have stack ranking.

101914• 12 years ago

Are these "web pages" all on the same website?

If so, using wget is a poor solution. I have not used wget in over a decade but as I recall it does not do HTTP pipelining; I could be wrong on that - please correct me.

I do recall with certainty that when wget was first written and disseminated in the 1990's, "webmasters" wanted to ban it. httpd's were not as resilient then as they are today, nor was bandwidth and hardware as inexpensive.

HTTP pipelining is a smarter alternative than burdening the remote host with thousands of consecutive or simultaneous connections.

Depending on the remote host's httpd settings, HTTP pipelining usually lets you make 100 or maybe more requests using a single connection. It can be acomplished with only a simple tcpclient like the original nc and the shell.

In any event, the line about a "distributed crawler" is spot on. Never understimate the power of marketing to suspend common sense.

Also, I find that I can often speed my scripts up a little by using exec in shell pipelines, e.g., util1 |exec util2 or exec util1 |exec util2.

There are other, better approaches besides using the builtin exec, but I will leave those for another day.

juliangregorian2• 12 years ago

Having written distributed crawlers, saturating the network connection is quite easy to do and is the main reason for even distributing that type of work in the first place.

mercurial• 12 years ago

You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved.

1: http://doc.scrapy.org/en/0.24/intro/tutorial.html

skynetv2• 12 years ago

its a sensational headline ... the reality is someone applied a wrong tool and got bad results.

jacquesm2• 12 years ago

See this very good comment by Bane:

https://news.ycombinator.com/item?id=8902739

aadrake3• 12 years ago

Hi all, original author here.

Some have questioned why I would spend the time advocating against the use of Hadoop for such small data processing tasks as that's clearly not when it should be used anyway. Sadly, Big Data (tm) frameworks are often recommended, required, or used more often than they should be. I know to many of us it seems crazy, but it's true. The worst I've seen was Hadoop used for a processing task of less than 1MB. Seriously.

Also, much agreement with those saying there should be more education effort when it comes to teaching command line tools. O'Reilly even has a book out on the topic: http://shop.oreilly.com/product/0636920032823.do

Thank you for all the comments and support.

jeroenjanssens• 12 years ago

Author of Data Science at the Command Line here. Thanks for the nice blog post and for mentioning my book here. While we're talking about the subject of education, allow me to shamelessly promote a two-day workshop that I'll be giving next month in London: http://datascienceatthecommandline.com/#workshop

andyjpb• 12 years ago

This is a great article and a fun read. A friend sent it over to me and I wrote him some notes about my thoughts. Having since realised it's on HN, I thought I'd post them here as well.

Some of my wording is a bit terse; sorry! :-) The article is great and I really enjoyed it. He's certainly got the right solution for the particular task at hand (which I think is his entire point) but he's generally right for the wrong reasons so I pick a few holes in that: I'm not trying to be mean. :-)

----- Classic system sizing problem!

1.75GiB will fit in the page cache on any machine with >2GiB RAM.

One of the big problems is that people really don't know what to expect so they don't realise that their performance is orders of magnitude lower than it "should" be.

Part of this is because the numbers involved are really large: 1,879,048,192 bytes (1.7GiB) is an incomprehensibly large number. 2,600,000,000 times per second (2.6GHz) is an incomprehensibly large number of things that can be done in the blink of an eye.

...But if you divide them using simple units analysis; things per second divided by things gives you seconds: 1.383. That's assuming that you can process 1 byte per clock cycle which might be reasonable if the data is small and fits in cache. If we're going to be doing things in streaming mode then we'll be limited by memory bandwidth, not clock speed.

http://www.techspot.com/review/266-value-cpu-roundup/page5.h... reckons that the Intel Core 2 Duo E7500 @ 2.93GHz) has 7,634MiB/s of memory bandwidth for reads.

That's 8,004,829,184 bytes per second.

Which means we should be able to squeeze our data through the processor in...

bytes per second divided by bytes = seconds =>

>> 8004829184 / 1879048192.0 4.260044642857143

so less than 5 seconds.

We probably want to assume that there are other stalls and overheads, but a number between 20 and 60 seconds seems reasonable for that workload (he gets 12): the article says it's just a summary plus aggregate workload so we don't really need to allocate much in the way of arithmetic power.

As with most things in x86, memory bandwidth is usually the bottleneck. If you're not getting numbers with an order of magnitude or so of memory bandwidth then either you have a arithmetic workload (and you know it) or you have a crap tool.

Due to the memory fetch patterns and latencies on x86, it's often possible to reorder your data access to get a nominal arithmetic workload close to the memory bandwidth expected speed.

His analysis about the parallelisation of the workload due to shell commands is incorrect. The speedup comes from accessing the stuff straight from the page cache.

His analysis about loading the data into memory on Hadoop is incorrect. The slowdown in Hadoop probably comes from memory copying, allocation and GC involved in transforming the raw data from the page cache into object in the language that Hadoop is written in and then throwing them away again. That's just a guess because you want memory to fill up (to about 1.75GiB) so that you don't have to go to disk. That memory is held by the OS rather than the userland apps tho'.

His conclusion about how `sleep 3 | echo "Hello"` is done is incorrect. They're "done at the same time" because sleep closes stdout immediately rather than at the end of the three seconds. With a tool like uniq or sort it has to ingest all the data before it can begin because that's the nature of the algorithm. A tool like cat will give you line-by-line flow because it can but the pipeline is strictly serial in nature and (as with uniq or sort), might stall in certain places.

He claims that the processing is "non-IO-bound" but also encourages the user to clear the page cache. Clearing the page cache forces the workload to be IO bound by definition. The page cache is there to "hide" the IO bottlenecks where possible. If you're doing a few different calculations using a few different pipelines then you want the page cache to remain full as it will mean that the data doesn't have to be reread from disk for each pipeline invocation.

For example, when I ingest photos from my CF card, I use "cp" to get the data from the card to my local disk. The card is 8GiB. I have 16GiB of RAM. That cp usually reads ahead almost the whole card and then bottlenecks on the write part of getting it onto disk. That data then sits in RAM for as long as it can (until the memory is needed by something else) which is good because after the "cp" I invoke md5sum to calculate some checksums of all the files. This is CPU bound and runs way faster than it would if it was IO bound due to having to reread all that data from disk. (My arrangement is still suboptimal but this gives an example of how I can get advantages from the architecture without having to do early optimisations in my app: my ingest script is "fast enough" and I can just about afford to do the md5sum later because I can be almost certain it's going to use the exact same data that was read from the card rather than the copied data that is reread from disk and, theoretically, might read differently.)

He's firmly in the realm of "small data" by 4 or 5 base 10 orders of magnitude (at least) so he's nowhere close to getting a "scaling curve" that will tell him where best to optimise for the general case. When he starts getting to workloads 2 or 3 orders of magnitude bigger than what he's doing he might find that there are a certain class of optimisations that present themselves but that probably won't be the "big data" general case.

Having said that, this makes his approach entirely appropriate for the particular task at hand (which I think it his entire point).

Through his use of xargs he implies (but does not directly acknowledge) that he realises this is a so-called "embarrassingly parallel" problem. -----

wglb4• 12 years ago

A similar story: http://blogs.law.harvard.edu/philg/2009/05/18/ruby-on-rails-...: Tools used not quite the right way.

edit: with HN commentary: https://news.ycombinator.com/item?id=615587

raincom• 12 years ago

Hadoop is replacing many datawarehousing dbs like netezza, teradata, exadata. In the process, many datwarehousing developers have become hadoop developers, who write sql code; after all, hadoop got a sql interface via hive.

Informatica (another ETL tool) also provides another tool called powerexchange, which automatically generates MR code for hadoop.

Whenever you hear hadoop, first ask yourself whether it is another disguised datawarehousing stuff.

spydum• 12 years ago

Yes, this is very much happening -- mostly based on the insane pricing difference of supporting Hadoop clusters vs ntz or td infrastructure. Just following a simple 3year lifecycle of HW depreciation essentially boosts your performance for next to nothing. The same cannot be said of the big DWH vendors

uxcn• 12 years ago

This kind of approach can probably scale out pretty far before actually needing to resort to true distributed processing. Compression, simple transforms, R, etc... You can probably get away with even more by just using a networked filesystem and inotify.

sabalaba• 12 years ago

I've had the pleasure and displeasure of working with small datasets (~7.5GB of images) in shell. One often needs to send SIGINT to the shell when it starts to glob expand or tab complete a folder with millions of files. But besides minor issues like that, command line tools get the job done.

_delirium3• 12 years ago

Until semi-recently, millions of files in a directory would not only choke up the shell, but the filesystem too. ext4 is a huge improvement over ext3 in that regard; with 10m files in an ext3 directory you ended up with long hangs on various operations. And even with ext4, make sure not to NFS-export the volume that directory is on!

ajuc2• 12 years ago

I've encountered this (or similar) issue on production.

We had C++ system that wrote temporary files to /tmp when printing, /tmp was cleared on system startup, it worked ok for years, but the files accumulated. At some point it started to randomly throw file access errors when trying to create these temporary files. Not for each file - only for some of them.

Disk wasn't full, some files could be created in /tmp, others couldn't, it turned out after a few days of tracking it, that filesystem can be overwhelmed by too many similary named files in one directory - and it can't create file XXXX99999 even if there's is no such file in this directory, but it can create files like YYYYY99999 :)

I just love such bugs where your basic assumptions turn out to be wrong.

rkwasny• 12 years ago

Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data.

Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed.

Bottom line, the popular saying is true: "Hadoop is about writing crappy code and then running it on a massive scale."

treve• 12 years ago

What makes 2TB the cutoff?

wobbleblob• 12 years ago

I think the consensus is that as long as your data fits on a single (affordable) machine, 'big data' tools are probably not the best solution.

earino• 12 years ago

Dell sells a server with 6TB of ram (I believe.) I think the limit is way over 2TB. If you want to be able to query it quickly for analytical workloads, MPPs like Vertica scale up to 150+TB (at Facebook.) I honestly don't know what the scale is where you need Hadoop, but it's gotten to be a large number very quickly.

juliangregorian2• 12 years ago

They do, I checked. It comes in at a cool half million (Helloooo, investors!)

virmundi2• 12 years ago

My question is what do you mean by 2TB? At my current client, we have 5 TBs of data sitting (that's relatively recent). Before we had 2-ish. However, we had over 30 applications doing complex fraud calculations on that. "Moving data" (data being read and then worked) is about 40 TB daily. Even with SSD and 256 GB of RAM, a single machine would get overwhelmed on this.

If you're only working one app on less than 1 TB, maybe you don't need something as complex as Hadoop. But given that a cluster is easy to setup (I made a really simple NameNode + Two Data nodes in 45 minutes, going cold), it might not be a bad idea.

I'll take this further and say that some tools for Hadoop that are not from Apache are really nice to work with even in a for non-Hadoop work. For example, I've got to join several 1 GB files together to go from a relational, CSV model into a Document store model. Can I do this with command line tools? Maybe. Cascading makes this really easy. Each file family is a tap. I get tuple joins naturally. I wrote an ArangoDB tap to auto load into ArangoDB. It was fun, testable and easy. All of this runs sans-hadoop on my little MBP.

Fun fact about the Cascading tool set is that I can take my little app from my desktop and plop it onto a Hadoop cluster with little change (taps from local to hadoop). Will I do that in my present example? No. Can I think of places where that's really useful? Yes, daily 35 fraud models' regression tests executed with each build. That's somewhere around 500 full model executions over limited, but meaningful data. All easily done courtesy of a framework that targets Hadoop.

ronreiter• 12 years ago

Hadoop is highly inefficient when using default MapReduce configuration. And a single Macbook Pro machine is much stronger than 7 c1.medium instances.

Bottom line - run the same thing over Apache Tez with a cluster that has the same computational resources as your laptop, and I'm pretty sure you'll see the same results.

robbles• 12 years ago

Is there a cached version of the original article that's referenced in this anywhere? Site appears to be down.

notpeter• 12 years ago

This article echoes a talk Bryan Cantrill gave two years ago: https://youtu.be/S0mviKhVmBI

It's about how Joyent took the concept of a UNIX pipeline as a true powertool and built a distributed version atop an object filesystem with some little map/reduce syntactic sugar to replace Hadoop jobs with pipelines.

The Bryan Cantrill talk is definitely worth your time, but you can get an understanding of Manta with their 3m screencast: https://youtu.be/d2KQ2SQLQgg

cheng1• 12 years ago

I have developed a one-liner toolset for Hadoop (when I have to use it). It's fresh to see a ZFS alternate of the concept. Don't like the JavaScript choice though.

GUN parallel should be a widely adopted choice. Lightweight. Fast. Low cost. Extendable.

xer0x• 12 years ago

You can use command-line tools for Manta without touching any Javascript. That's probably the best way to go. Although I do like Javascript.

linuxhansl• 12 years ago

So don't use Hadoop to crunch data that fits on a memory stick, or that a single disk spindle can read in few seconds.

Why is this first on the HN front-page?

Reminds me of the C++ is better than Java, Go is better than C++, etc, pieces.

Yes, the right tool for the right job. That's what makes a good engineer.

Somebody who thinks there is _no_ valid use case for Hadoop is a fool. (The author did not say that, but many of the comments here seem to imply that view)

lucaspottersky• 12 years ago

Why is this first on the HN front-page?

Because controversial topics are always fun! d:-)

colin_mccabe• 12 years ago

About 5 years ago I worked at a company that took the "pile of shell scripts" approach to processing data. Our data was big enough and our algorithms computationally heavy enough that a single machine wasn't a good solution. So we had a bunch of little binaries that were glued together with sed, awk, perl, and pbsnodes.

It was horrible. It was tough to maintain-- we all know how hard to read even the best awk and perl are. It was difficult to optimize, and you always found yourself worrying about things like the maximum length of command lines, how to figure out what the "real" error was in a bash pipeline, and so on. When parts of the job failed, we had to manually figure out what parts of the job had failed, and re-run them. Then we had to copy the files over to the right place to create the full final output.

The company was a startup and the next VC milestone or pivot was always just around the corner. There was never any time to clean things up. A lot of the code had come out of early tech demos that management just asked us to "just scale up." But oops, you can't do that with a pile of shell scripts and custom C binaries. So the technical debt just kept piling up. I would advise anyone in this situation not to do this. Yeah, shell scripts are great for making rough guesses about things in a pile of data. They are great for ad hoc exploration on small data or on individual log files. But that's it. Do not check them into a source code repo and don't use them in production. The moment someone tries to check in a shell script longer than a page, you need to drop the hammer. Ask them to rewrite it in a language (and ideally, framework), that is maintainable in the long term.

Now I work on Hadoop, mostly on the storage side of things. Hadoop is many things-- a storage system, a set of computation frameworks that are robust against node failures, a Java API. But above all it's a framework for doing things in a standardized way so that you can understand what you've done 6 months from now. And you will be able to scale up by adding more nodes, when your data is 2x or 4x as big down the line. On average, the customers we work with are seeing their data grow by 2x every year.

I feel like people on Hacker News often don't have a clear picture of how people interact with Hadoop. Writing MapReduce jobs is very 2008. Nowadays, more than half of our users write SQL that gets processed by an execution engine such as Hive or Impala. Most users are not developers, they're analysts. If you have needs that go beyond SQL, you would use something like Spark, which has a great and very concise API based on functional programming. Reading about how clunky MR jobs is just feels to me like reading an article about how hard it is to make boot and root floppy disks for Linux. Nobody's done that in years.

weitzj• 12 years ago

There is also an interesting and fun talk to watch by John Graham Cumming from CloudFlare. http://www.youtube.com/watch?v=woCg2zaIVzQ using Go instead of xargs. Kind of fits into: "Using the right tool for the job". There is no Big Data involved but it shows a sweetspot where it might make sense(make it easier) to not use a shell script (i.e retries, network failure)

dundun2• 12 years ago

What is missed in the article and many of these comments is that Hadoop isn't always going the best tool for one job. It shines in its multitenancy- when many users are running many jobs-each developed in their favorite framework or language(bash/awk pipeline? No problem) running over datasets bigger than single machines can handle.

It also comes in handy when your dataset grows dramatically in size.

ricardobeat• 12 years ago

Don't shoot me, but out of curiosity I wrote the thing in javascript: https://gist.github.com/ricardobeat/ee2fb2a6d704205446b7

Results: 4.4GB[1] processed in 47 seconds. Around 96mb/s, can probably be made faster, and nodejs is not the best at munging data...

[1] 3201 files taken from http://github.com/rozim/ChessData

philgoetz• 12 years ago

First, you don't score points with me for saying not to use Hadoop when you don't need to use Hadoop.

Second, you don't get to pretend you invented shell scripting because you came up with a new name for it.

Third, there are very few cases if any where writing a shell script is better than writing a Perl script.

davecheney2• 12 years ago

1.75gb is not big data. It's not even small data.

steego• 12 years ago

It's not even small data.

We get it. Your data's pretty big.

davecheney2• 11 years ago

1.75gb is three cd roms. It's not big data.

1.75gb of ram is less than the virtual address space for 32 bit windows Xp, it's not big data.

Protip: if it fits on the computer on your desk, it's not big data.

decisiveness• 12 years ago

If bash is the shell (assuming recursive search is required), maybe it would be even faster to just do:

    shopt -s globstar
    mawk '/Result/ {
        game++
        split($0, a, "-")
        res = substr(a[1], length(a[1]), 1)
        if(res == 1)
            white++
        if(res == 0)
            black++
        if(res == 2)
            draw++
    } END {
        print game, white, black, draw
    }' **/*.pgn

?

nraynaud• 12 years ago

on a tangential note, sometimes I use a slower methods for UI reasons. For example avoiding blocking the UI, or allowing for canceling the computation, or displaying partial results during the computation (that last one might completely trash the cache).

sleepythread2• 12 years ago

One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size.

I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem.

Hadoop will be helpful in case if there are millions of games are played everyday and we need to update the statistics daily e.t.c. For this case, the given solution will hit bottleneck and there will be some optimisation/code change needed to keep running the code.

Hadoop and its ecosystem are not a silver bullet and hence should not be used for everything. The problem has to be a Big Data problem

yuanchuan2• 12 years ago

It is that buzz surrounding Hadoop that makes people misunderstood its use and capability. I have met non-technical analysts who want RDBMS performance on Hadoop. They expect seconds to minutes scale queries on hundreds of GB of data.

I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon?

Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describes their capabilities and use cases.

Ultimately, it is cost vs efficiency. Hadoop can solve all data problems. Likewise for RDBMS. This is an engineering tradeoff that people have to make.

sleepythread2• 12 years ago

I totally agree with you. Capability <strong>"LIKE"</strong> will drive Hadoop adoption, Hadoop should not be seen as replacement of R.D.B.M.S. These are two different tools for made for different purpose.

pacala3• 12 years ago

They expect seconds to minutes scale queries on hundreds of GB of data.

Use BigQuery from Google.

2ion2• 12 years ago

Data analysis using a shell can be amazingly productive. We also had a talk about this at TLUG (http://tlug.jp/wiki/Meetings:2014:05#Living_on_the_command_l...).

TeeWEE• 12 years ago

The gotcha here is: He is talking about 1.75GB of data. Off course you dont use hadoop for this. Hadoop is for BigData, not for a few gigs.

Use the right tool for the job. If you think you will scale to TeraByte size, dont start out with command line tools.

impostervt• 12 years ago

Any decent tutorials out there to get me up to speed on CL tools? I use grep and a few others regularly, but have avoided sed and awk as they seem difficult to jump into.

jodrellblank2• 12 years ago

A couple of days ago this clear, brief introduction to AWK was submitted: http://ferd.ca/awk-in-20-minutes.html

thehal84• 12 years ago

A similar article that articulated this well.

https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

khaki54• 12 years ago

If the entire dataset fits into memory on your laptop then it's not big data, and the only reason for using map reduce etc is to build experience with it or proof of concept for a larger dataset.

taltman1• 12 years ago

This is a great exercise of how to take a Unix command line and iteratively optimize it with advanced use of awk.

In that spirit, one can optimize the xargs mawk invocation by 1) Getting rid of string-manipulation function calls (which are slow in awk), 2) using regular expressions in the pattern expression (which allows awk to short-circuit the evaluation of lines), and 3) avoiding use of field variables like $1, and $2, which allows the mawk virtual machine to avoid implicit field splitting. A bonus is that you end up with an awk script which is more idiomatic:

  mawk '
  /^\[Result "1\/2-1\/2"\]/ { draw++ }
  /^\[Result "1-0"\]/ { white++ }
  /^\[Result "0-1"\]/ { black++ }

  END { print white, black, draw }'

Notice that I got rid of the printing out of the intermediate totals per file. Since we are only tabulating the final total, we can modify the 'reduce' mawk invocation to be as follows:

  mawk '
  {games += ($1+$2+$3); white += $1; black += $2; draw += $3}
  END { print games, white, black, draw }'

Making the bottle-neck data stream thinner always helps with overall throughput.

guardiangod• 12 years ago

Here is an analysis from a developer who looked at Hadoop- http://ossectools.blogspot.ca/2012/03/why-elsa-doesnt-use-ha...

(ELSA is a logger that claims to be able to handle 100000 entries/sec (!!))

When to Use Hadoop

This is a description of why Hadoop isn't always the right solution to Big Data problems, but that certainly doesn't mean that it's not a valuable project or that it isn't the best solution for a lot challenges. It's important to use the right tool for the job, and thinking critically about what features each tool provides is paramount to a project's success. In general, you should use Hadoop when:

    Data access patterns will be very basic but analytics will be very complicated.
    Your data needs absolutely guaranteed availability for both reading and writing.
    There are inadequate traditional database-oriented tools which currently exist for your problem.

Do not use Hadoop if:

    You're don't know exactly why you're using it.
    You want to maximize hardware efficiency.
    Your data fits on a single "beefy" server.
    You don't have full-time staff to dedicate to it.

The easiest alternative to using Hadoop for Big Data is to use multiple traditional databases and architect your read and write patterns such that the data in one database does not rely on the data in another. Once that is established, it is much easier than you'd think to write basic aggregation routines in languages you're already invested in and familiar with. This means you need to think very critically about your app architecture before you throw more hardware at it.

knodi123• 12 years ago

We have a proprietary algorithm for assigning foods a "suitability score" based on a user's personal health conditions and body data.

It used to be a fairly slow algorithm, so we ran it in a hadoop cluster and it cached the scores for every user vs. every food in a massive table on a distributed database.

Another developer, who is quite clever, rewrote our algorithm in C, and compiled it as a database function, which was about 100x faster. He also did some algebra work and found a way to change our calculations, yielding a measly 4-5x improvement.

It was so, so, so much faster that in one swoop we eliminated our entire Hadoop cluster, and the massive scores table, and were actually able sort your food search results by score, calculating scores on the fly.

saym• 12 years ago

May I ask: Who is we?

sgt1012• 12 years ago

There's a perception that hadoop commands are terribly complex. If you run

$spark-shell

you can execute (interactively)

val file = spark.textFile("hdfs://...") val errors = file.filter(line => line.contains("ERROR")) errors.count()

And wordcount a file - ok the wget is not there, but this is really not complex!

hawleyal• 12 years ago

Flamebait

ec664• 12 years ago

This is blown out of proportion... actually increase is probably a factor 10-20X, not 100s. The fact that EMR is used is a problem, provisioning, bootstrapping the cluster alone accounts for probably half the time.

The fact that shell commands were run repeatedly means that the data ends up in the OS buffer cache and basically in memory.

I'm not discounting that CLI is faster than Hadoop by an order of magnitude on small datasets. Nor will I dive into Hadoop vs CLI. The answer to all that IMO is that it depends. And in this case, it's not well warranted.

What I do take exception to is the Fox News style headlines that are disproportional to the truth. EMR != Hadoop.

Command-line tools can be faster than your Hadoop cluster

Related Stories