jtai

Comparison to shared memory or mmap'd files -- by using files we let the filesystem manage the cache, so we don't have to be concerned with the data growing and causing memory pressure. We also don't have to worry about how to store the data in a format friendly to fast retrieval, since this is provided by sparkey.

Comparison to local dbs like cdb, sparkey, mapdb, gettext files etc. -- hammerspace is a gem that uses sparkey under the hood, so these solutions are more or less one and the same. The difference is that hammerspace exposes a ruby hash-like API to make integration with existing applications easier. It also provides concurrent writer support, which many local dbs don't do.

Whether it's a good idea or not is for you to judge -- we've open sourced the gem in hopes that it will be useful to someone, just as sparkey and gnista were useful to us.

Our community of translators update translations all the time. Updates are immediately reflected so that our translators have instant feedback, and we'd like our users to see the updated translations sooner than every time we deploy too. So that rules out a purely build-time solution for us.

We ended up using sparkey as the first backend for hammerspace, but hammerspace was written to support multiple backends. We benchmarked both cdb and sparkey, and their performance was very similar for our use case. At under 100mb of data, we weren't concerned about the 4G limitation or about the data not fitting in cache. I don't think sparkey has the 4G limitation, and someone has already forked cdb to support 64 bit offsets: https://github.com/pcarrier/cdb64

Good points!

We did initially try dedicating ruby processes to particular locales -- when we saw a big improvement we knew we were on the right track. Doing so permanently would be more difficult. To "follow the sun" we would need to shift capacity to Europe and Asia during some hours and back to the US in others.

Parallelizing memcache queries is difficult because we don't know ahead of time what translations will be required to render a page.

We /are/ only working with the most recently used strings. Strings not accessed in the last 4 days are not loaded.

I'm not sure what "denormalize locales" means exactly.

Sparkey is fast because the files end up in the filesystem cache and most of the work is done in C. Going through the dalli gem to grab the translations out of memcache causes a lot of temporary ruby objects to be created.

We did benchmark local memcache, but accessing the strings over the network (even locally) was much much slower than sparkey. Redis would likely have been similar to memcache.

The savings came not just from avoiding redundant GC over the same 80MB, but also from not going through the churn involved with loading that data over the network on process startup.

We have a large codebase that's written against MRI, so it's non-trivial to just switch to something else.