Rediscala: Non-blocking Redis driver for Scala

by etaty • 13 years ago

67 16 13 years ago

weego• 13 years ago

Interesting. It feels like it bleeds a bit too much of the underlying "Akka-ness" out into the actual everyday implementation of the library, which makes it feel slightly more boilerplatey than perhaps is needed.

I noted that Debasish Ghosh recently updated his redis client to have Akka as the underlying handler as well.

yummyfajitas• 13 years ago

Bleeding out the akka-ness is a feature, at least if you live in akkaland. For example:

    path("get_a_list" / PathElement) { externalKey =>
      get {
        complete {
          for {
            internalKey <- redis.get("internal_keys_from_external":externalKey)
            finalResult <- redis.smembers(internalKey)
          } yield (finalResult.toList)
        }
      }
    }

Spray already expects an akka future, as does everything else, so this library fits in perfectly.

shpxnvz• 13 years ago

And it appears to use futures that are part of Scala core (which Akka now uses as well) so I'm not sure how much this can be said to be Akka-ness anyway.

etaty5• 13 years ago

You are welcome to create a issue or a pull request.

Rediscala needs an akka actor System and sometime replies are of type ByteString.

weakwire2• 13 years ago

What the diff with https://github.com/debasishg/scala-redis ? Almost every scala redis driver supports futures and is easy to wrap it into futures. Also by making all redis call async we have noticed performance issues as redis serves only 1 req every time there was too much time spending in thread-join. We found that the best practice is to have non blocking requests (methods) and all the redis reads in a blocking way. The only way to gain speed it to have N redis CPU's with N threads but that's almost impossible.

etaty5• 13 years ago

The main difference are the performance and a better support of commands (almost all of them are available right now). Rediscala matches redis type answer to scala types.

Redis serves 1 req at a time, but you can write to Redis socket and read from Redis socket during that time.

Rediscala (the RedisClient) use 2 actors: The first one handle I/O (I/O bounds). That actor also merge redis requests in a buffer, so there is less messages (but bigger) going to the TcpWorker (akka worker handling a tcp socket). The second decode replies (CPU bounds)

weakwire2• 13 years ago

Really interesting. I'll try the Rediscala driver and let you know about performance then.

anorwell• 13 years ago

It looks like the asynchronous interface to debasishg/scala-redis is synchronous under the hood, using a pool of blocking connections. This means that there is no pipelining. Each of these blocking connections must wait for a request to travel over the wire, be handled by redis, and travel back before it can be reused. An asynchronous client allows full pipelining automatically, which is a much more efficient use of a single connection.

we have noticed performance issues as redis serves only 1 req every time there was too much time spending in thread-join

Performance should be better with an asynchronous client. When I've benchmarked e.g. jedis vs various async clients, that's been the case.

terhechte• 13 years ago

I'd love to see this setup in the web framework benchmark (maybe even with spray) to see just how fast it can be. I may give it a try if I find some free time.

Lazyshot• 13 years ago

I'm inclined to think that finagle-redis is a much less bulky (akka) library to choose.

https://github.com/twitter/finagle/tree/master/finagle-redis

kul_• 13 years ago

@etaty Thanks much for this effort! Looking forward to a user guide. Also did your redis-benchmark gets to 200k+, if not this will be one hell of performance mark. And even if it does, what the heck, its very hard to find use case which saturates this rate of inserts(except may be data generation frequency in cern reactor)

etaty5• 13 years ago

Yes, it does 200 000 requests/second

Benchmark : http://bit.ly/12QZsRs

Readme : https://github.com/etaty/rediscala#performance

ninjakeyboard• 13 years ago

Cool. There is actually another similar project from mDialog here: https://github.com/chrisdinn/brando

May be helpful to compare the underworkings to see different approaches.

etaty5• 13 years ago

Thanks. Rediscala doesn't have a shard manager yet. Brando version looks interesting.

felipehummel• 13 years ago

A few suggestions: you could remove this kind of idiom: .map( result = { ... }) In the blocking commands example the snippet resembles the Javascript callback hell when it could be simpler to read: https://gist.github.com/felipehummel/6187248

Also, even simplifying it, it still seems overcomplicated. The returning type: Future[Try[Option[(String, ByteString)]]] is too much. Can't you remove the Try or the Option? Futures can already hold either a Sucess or Failure, so isn't the Try redundant?

etaty5• 13 years ago

I removed `Try` in the version 1.1

Rediscala: Non-blocking Redis driver for Scala

Related Stories