Search Big Data in Five Minutes with Pig, Wonderdog and ElasticSearch

http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/

by rjurney • 14 years ago

39 14 14 years ago

hortonworks.com

rjurney4• 14 years ago

I'd like to see more YC companies using Pig.

lazy_nerd• 14 years ago

Any comparisions between Pig, Hbase and Hive might help

virmundi• 14 years ago

See my problem with Pig is that it doesn't unit test well. It's easy, it's quick, it's fairly powerful, but unit testing is horrible. I spent 3 days extending PigUnit to make it easier to overload hard coded reference files in the script so you can provide mock test files. Even getting this to work, and having under 5 megs of reference file for the test, the test takes 3 minutes for 7 simple filters and joins and about 900 megs of RAM.

Compare this to Cascading which is far easier to unit test.

Also PIG is a hog when it comes to making jars. If you don't install pig on EVERY node in the cluster and rather provide it via a submitted uber jar, the jar is HUGE (50 or so megs). PIG's dependencies are ridiculous.

Again compare to Cascading who has a far smaller foot print.

I would like to a strong comparison of HBase. Pig is useful for one-off, ad-hoc stuff, but I don't think it's production ready.

rjurney4• 14 years ago

Lets be clear: Pig powers Yahoo and LinkedIn. Large enterprises. There's no question whether it is production ready.

squarecog3• 14 years ago

50 Megs? How in the world are you getting that? I just checked trunk, and it's 3.1. Granted, Cascading comes in at about half a meg for 2.0, but this 50 meg figure is just silly.

Our tests take seconds, and can run from eclipse (we use Pig's local mode).

You sank 3 days into modifying internals of PigUnit instead of just parametrizing your loader statements?

matan_a• 14 years ago

We've been using Pig extensively for our MR queries and we're very happy with it. While i haven't used Cascading, i do like the pipelining approach of Pig more than the SQL-influence method of Hive. It lends itself better for meeting our needs.

Testing is a good point. If you're running Pig through their APIs it is definitely easier to test than command-line running scripts. We've written test code that reads and runs pig scripts through the API using fixed sample data (stored in HDFS for easy access), read the results, and compare it to expected results (also stored in HDFS). Honestly, you don't need too much input data to prove the correctness of the query.

Remember that Pig also supports placeholders in your scripts so you that you can set them in run-time to define input/output paths, etc. This makes testing easier.

Dependencies can also be stored in HDFS which makes it simple to run your scripts w/o the need to distribute jars around.

ajays• 14 years ago

FTA: "Note however, the ElasticSearch UDF returns nearly instantly, while the FILTER is slooooooowwww… "

It's just 450MB of data (in Avro format). Why should the filter be "slooooooooowwww" ?

villagefool• 14 years ago

Agreed, the actuall time measurements are missing for the comparison.

rjurney4• 14 years ago

I'll add them.

big_data• 14 years ago

Pig may lack some syntactic sugar, but it really gets the job done! It's a great way to get MR jobs going in a hurry.

Search Big Data in Five Minutes with Pig, Wonderdog and ElasticSearch

Related Stories