Searching 20 GB/sec: Systems Engineering Before Algorithms (2014)

http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-engineering-before-algorithms/?1

by misframer • 11 years ago

105 28 11 years ago

blog.scalyr.com

kiallmacinnes• 11 years ago

The linked article has been posted before, I can't find the old HN thread.. But it was certainly worth a re-read :)

I wonder has scalyr reached their expected 100GB/s yet?

caust1c• 11 years ago

It was posted 17 hours ago by the same guy and a different url: https://news.ycombinator.com/item?id=9201444

misframer• 11 years ago

I was asked by HN to repost it.

swatow• 11 years ago

Judging from the comments, this article was written around May 8 2014. Can we get a (2014) in the title?

dang3• 11 years ago

Good catch. Added.

hglaser• 11 years ago

Great post.

BTW, this product (Scalyr) is a lifesaver. We (Periscope) are able to operate ~ a dozen heterogeneous servers with no FT DevOps largely because of Scalyr.

snewman2• 11 years ago

Hi! Great to see this pop back up on HN. I'm the author of the blog post (and Scalyr founder), happy to answer any questions.

Downthread, someone mentioned that they couldn't find the HN discussion from when this was originally posted; it's here:

https://news.ycombinator.com/item?id=7715025

imaginenore5• 11 years ago

I wonder why they chose Java for substring search. Why not C (strstr) or grep?

http://www.arstdesign.com/articles/fastsearch.html

snewman2• 11 years ago

We started with Java because the rest of our backend is written in Java. We stuck with it because it has not turned out to be much of a bottleneck in practice. This came as a surprise; I had expected that we'd need to rewrite the substring search loop in native code. We may still do so eventually, but we're able to get very high performance in pure Java. (1.25GB / second / core, as noted in the post.)

Edit to add: "good enough is not good enough" -- very well put! Yes, it would be good to move this loop to native code. However, with our current design, the overhead of calling out from Java to native code would probably outweigh any benefits. We're planning to move our core database storage to off-Java-heap memory buffers, at which point it will become much more feasible to call out to native code.

The inner loop speedup on this particular code will be more like 2x than 3x, and the speedup of the overall system will be quite a bit less than that. But your point is still correct.

imaginenore5• 11 years ago

Well, if you can triple your performance, you will require 1/3 of the servers, and your clients will enjoy faster responses. I thought that was the idea, good enough is not good enough.

Anyways, congrats on your success. I would love to read more about the business side - finding clients, profits, expenses, etc.

twotwotwo• 11 years ago

Lots of attention goes to OLTP-type loads for good reasons, but when you do design to just stream fast, some fun things happen:

You can use lots of relatively cheap spindles in parallel, and think of each one as (at least) 100MB/s of sequential read speed and a couple TB of space. You have fast compression available that can increase your effective bandwidth and make the effective cost of space cheaper.

You can draw on well-understood ways to search, sort, do hash- or sort-based joining and grouping, and so on.

Streaming doesn't need a big in-memory cache to avoid disk seeks, so you can use those gobs of RAM for other things--aggregating results or holding data to join against, say. (Of course, if you don't need the RAM, disk cache might still be useful for some access patterns.)

Besides log search, you see a stream-fast approach in analytics-focused DBs: BigQuery, Redshift, Vertica, and open-source ones--Facebook put up a good post about the work that led to the their Hive ORCFile design.

Some bioinformatics tools load a big hashtable into memory and, roughly, hash-join against a ton of raw data streamed from disk, then sometimes then repeat the process with another hashtable.

These are not at all original observations, but I managed to hear about these sorts of analytics and bioinformatics tools for a while before really getting how or why why they did things all that differently from a typical random-access-oriented database.

imaginenore5• 11 years ago

I have another idea for you guys. Instead of relying on expensive AWS SSD instances, why not switch to Hetzner, and keep everything in RAM?

128 GB RAM for $135/month:

https://www.hetzner.de/en/hosting/produkte_rootserver/px120

And you will have so much extra disk space, you can use it for backups. Or even resell it.

Your i2.4xlarge cost you $2,455/month.

noir_lord2• 11 years ago

Tried hetzner last year would never touch it with a bargepole.

imaginenore5• 11 years ago

What's wrong with them?

lostmsu• 11 years ago

Nice. But does not scale.

Searching 20 GB/sec: Systems Engineering Before Algorithms (2014)

Related Stories