Getting 100x better search performance on Riak

http://blog.clipboard.com/2012/03/18/0-Milking-Performance-From-Riak-Search

by gwf • 14 years ago

95 17 14 years ago

blog.clipboard.com

joshu2• 14 years ago

Hi Gary!

gwf5• 14 years ago

Hey Josh!

joshu2• 14 years ago

Man, it's downvote central over here.

gfodor• 14 years ago

The deficiency with AND queries due to term based partitioning reminds me of a hack I did a few years ago trying to build a basic IR system on simpledb:

http://codingthriller.blogspot.com/2008/04/simpledb-full-tex...

Of course, I was backed into this corner by the various constraints at the time, and this was just a fun hack. It sounds a bit precarious to consider using Riak for text search, at least compared to something like Solr, if it truly has issues with the basic bread and butter of search like sorting by recency and conjunctions. (of course I am only going on what is in this post, I have no experience with Riak)

gwf5• 14 years ago

It really isn't all that bad. When you compare what we had to do compared to keeping two big data systems in sync (instead of Riak, which is a breeze), I think we got off easy.

nosequel• 14 years ago

Showyou designed Mecha to run Solr with Riak, there are slides here http://www.slideshare.net/jmuellerleile/scaling-with-riak-at... and I noticed someone else posted the video of the talk.

pkulak• 14 years ago

Mecha is being built to directly solve these exact problems:

http://vimeo.com/30396564

tpsreport4• 14 years ago

What's the baseline? When the baseline is terrible, getting 100x performance is not a big deal.

No good engineer would cite "100X improvement" without quantifying the baseline level. This is PR, not engineering and not science.

gwf5• 14 years ago

The baseline was a substantial test corpus that we scaled several orders of magnitude over a series of runs, all meant to simulate typical clip size with typical word frequencies. The 10-100x gains had two contributions that each was in the 3-10x range. We tested it against the standard search which incorrectly performed pagination because of the misorder on sort versus slice. We also tested in on two types of map/reduce jobs that correctly implemented the sort and slice (and had been in production).

Ideally, we would have kept the data around to give a fuller a report. But the truth is we did this over 9 months ago and didn't save the data. After informally sharing the impact with a lot of people, we heard a lot of encouragement to share the techniques.

And you're right, this is not hard core science nor engineering. But it is a good tip, which you can take or leave.

tpsreport4• 14 years ago

I'm still not seeing you report something that says "an operation that took X ms now takes Y ms." Note the absolute numbers whose units are in seconds. That's what I'd like to see for a baseline.

And I won't even comment on the "we did this 9 months ago and did not save the data" part. That kind of stuff would not fly in medicine or science or most traditional engineering fields. Why should you be exempt?

nikcub• 14 years ago

interesting bug where a new user who is downvoted no longer shows up as green.

Getting 100x better search performance on Riak

Related Stories