How to take advantage of Redis just adding it to your stack

http://antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html

by antirez • 15 years ago

270 53 15 years ago

antirez.com

Joakal• 15 years ago

Would it help game servers to run it with Redis? Like say, a FPS or RTS server.

I'm not sure of the typical game server stack though.

getsat• 15 years ago

Those kind of game servers are typically a binary blob that you cannot configure externally at all. Your only configuration options are in-game settings like the map list, server name, player limit, etc.

reitzensteinm• 15 years ago

Usually the game server would keep everything in the game instance in memory, for an FPS or RTS. As great as Redis is, it's way more cumbersome than having everything in your data structures ready to go.

Redis is suitable for persistence, so where it would really be appropriate would be storing things like player profiles and other misc data that is not tied to a specific server. In fact, that's exactly what I'm using it for.

illumin82• 15 years ago

I realize that for most FPS or RTS titles, Redis might not be a perfect fit because the dataset is small for each map and the number of players never exceeds some preset limit like 64, but I think Redis might really shine as a means of storing persistent world data - for example MMOs, without having to shard heavily. Most modern MMOs use heavy sharding and can really only support a couple hundred players in the same area or shard at the same time. With Redis, you might be able to handle many thousands of players in the same area without sharding. You don't need sub-millisecond response time - as long as you can do most operations in a few milliseconds, things will be fine, since most players are on ~100 millisecond WAN connections anyway.

You might finally be able to break the model that current large games like WoW use - heavily sharded with all persistent data stored in Oracle. Redis should allow you to shard less and use SQL less, resulting in a much better experience, especially when parts of the virtual world get crowded.

mickeyben• 15 years ago

Very good article: there is some good exemples on how you could use Redis.

We just added it to our stack for caching and storing sessions.

It's blazing fast !

We're now trying to use it for different other purposes; autocompletion, counting and ab testing.

geuis2• 15 years ago

What are some inexpensive cloud options to run redis for large data sets up to say a gb or so?

moe2• 15 years ago

RAM is expensive in the cloud. So unless the cloud is a strict requirement it will normally be much cheaper to build out a box with 32G (or whatever fits your need) and run redis on that.

va_coder2• 15 years ago

Anybody know why RAM is so expensive in the cloud? I can go to BestBuy and get a 8GB server for 600 bucks these days.

jarin4• 15 years ago

"The Cloud" isn't always the right solution for everything. If you're looking to go inexpensive, you can put it on a 2 GB Linode VPS on a ramen budget.

code_duck4• 15 years ago

The confusing thing is some people refer to VPSs as 'cloud', which sort of makes sense when you compare them to EC2. EC2 seems more 'cloudy' because... um, we don't know what kind of hardware it's running on? I don't know. I was trying to explain what 'cloud' means to my tech-savvy, but non-industry brother and was puzzled by the issue of whether Linode is a 'cloud' service or not.

geuis2• 15 years ago

Thanks for the suggestion. I've been working on a project that does dozens of writes per second and on my local machine I used redis. The costs of running redis for this on EC2 is prohibitive. The costs at Linode for even a 1gb vps is quite reasonable and scaling that up to larger sizes isn't that much more.

randito• 15 years ago

In your first example, you use redis to cache the id's of the latests comments, with a fallback to SQL in order to populate the list. However, you still need to call the DB to load the comments. I don't see the gain here.

Yes, you've replaced a "select * from comments order by created_at limit 10" with a "select * from comments where id in (list_of_ids_from_redis)".

Wouldn't you cache the comment models in a top-10 list?

courtewing• 15 years ago

When dealing with a lot of data, the latter SQL query would potentially be significantly faster than the former. If your goal is simply to show a top-10, then caching the entire results would be a great idea, but if you're goal (like in the article) is to make retrieving the first 5000 comments quickly, then this implementation is pretty solid.

HarrisonFisk• 15 years ago

I don't think the latter SQL would be significantly faster assuming the appropriate index on created_at.

The database can read the last 10 via the index directly and they would all most likely be on the same index page. Assuming any sort of normal caching, this would be at most 1-3 random reads and most likely none since I presume that the created_at index is generally be written in ascending order.

Once that step is done, it is essentially identical to your IN statement you did.

antirez• 15 years ago

Often the DB will show unacceptable time to reply to the ORDER BY stuff but will fetch comments by single ID without problems. But when that second part is a problem as well it is a good idea to use Redis as a "vulgar" cache for items as well, so that the recent stuff are probably into a Redis hash and you can fetch everything with a single pipelined MULTI/EXEC call (and fetch all the items returned as nils from the DB).

rbranson2• 15 years ago

A composite index will work wonders for this situation. EXPLAIN is your friend.

bretthopper• 15 years ago

I've read about Redis before and heard how companies are using it, but never completely understood it's purpose. After reading this I can actually say I understand Redis now and how it's useful. Amazing that after hearing so much about it all it took was a relatively simple article.

jr299• 15 years ago

Agreed. This is a great article that finally connected the dots in my head.

I'd like to see some recommendations on similar articles for other related tech (couch, mongo, hadoop, etc) that did the same for others.

jarin4• 15 years ago

On a basic level, you can think of it as just a faster version of Memcached.

Digging a little deeper, you can think of it as a faster, extremely powerful version of Memcached.

eliben• 15 years ago

I'm not too familiar with memcached, but isn't one of Redis's strong points is the data structures it provides out of the box, and not simple lookups?

gregwebs• 15 years ago

Its not faster than Memcached. More likely it is slightly slower. The benchmark the author of Redis did is under dispute.

datadon• 15 years ago

These little fixes are how I got into Redis and a month or so later, it's a primary data store (with disk based fall back) and I find myself doing 99% of aggregation and temporary storage operations with it.

Really great tool for the belt.

rch• 15 years ago

Does anyone know if hdf5 would be an acceptable optional replacement for the current Redis disk format?

I have a console app that's backed by Redis (in much the same manner as described in this post), but I save my sessions to h5 when I switch between datasets. That means I need to combine the Redis data with my app data and export -- I do this using two separate h5 files, with with the appropriate links.

It would be nice (for me anyway) if I could do a Redis-native save, and move the resulting file. That would also improve my startup times when I reverse the process.

But, while h5 is nice for My data, I can't say it would be any good for generic Redis data...

thoughts?

jarin4• 15 years ago

Resque and redis-store are like auto-adds for almost any Rails project I work on these days.

Resque is for background jobs (with many add-ons for locking, scheduling, retries, etc.), and redis-store is a drop-in store for Rack::Session, Rack::Cache and Rails.cache. Easy and super fast.

bretthoerner• 15 years ago

We use a ton of Redis, but I think the main takeaway from this article applies to all "NoSQL databases".

The "movement" is about polyglot persistence and not leaving RDBMS completely. Pull pain points out into something that's a better fit. Rinse and repeat.

stephth2• 15 years ago

I'm a little concerned with the added complexity of decoupling the datastore into two different systems that are relied on for application logic. What are good strategies to maintain consistency between the two, in the event of a failure? In the first example, what if the SQL transaction succeeds but the redis one fails? Would you rollback the SQL transaction?

joevandyk• 15 years ago

Depends on your application.

If you're using ActiveRecord, if the save to redis fails, raise an exception and it'll rollback the transaction.

Note that you'll always want a way to resync the redis copy of the data with what's in your database.

tednaleid2• 15 years ago

In most systems, you're probably already doing it. Any ORM that has a level 1/2 cache built in (like Hibernate and EHCache/terracotta/etc or SQL caching in Rails) is storing the data in more than one place. If the data is in the cache (or if the cache fails for whatever reason), you're in the same boat you are now.

If you're just using redis for write-through caching/memoizatoin and there's some failure, you still have the answer in hand and can return it to the user, you just lose the benefit of the speedup from the cache.

edanm• 15 years ago

It's a big problem, at least for our application (chattybar.com - a chat plugin). We have 2 copies of every chat message sent, one in the DB (for persistency) and one in Redis for quick retrieval. Keeping them synced has a lot of edge cases, especially since multithreading means that adding things to the DB and then to Redis doesn't necessarily guarantee they'll end up in the same order.

va_coder2• 15 years ago

I'm looking forward to the book

pg_bot• 15 years ago

This is a great example of how to promote adoption of a new technology. More companies should pay attention to how their product can be used rather than what their product is.

true_religion• 15 years ago

What I get from this is that Redis is so powerful that its best to not use it as a simple read-cache where the database is still the cannonical source.

Its better to use it as the write-cache for complex datasets with the database being the backup.

Periodic• 15 years ago

Right. I believe we already have plenty of read-caches out there. Our web servers can implement them, we have Varnish and MemCache and ~100 others.

What Redis appears to do great is that it is much more than just a key-value store with CRUD operations. You can model sets, lists, queues, counters, and do complex operations on these in-memory values.

It looks to me like it is best thought of as providing an interface to data-structures that are well optimized and persist across requests.

ww520• 15 years ago

Anyway to use Redis in App Engine setting? Or does Google have similar service in App Engine?

StavrosK• 15 years ago

No and no. They only offer Memcache.

bravery• 15 years ago

"just adding it to your stack" is the best conclusion that i have seen so far. Great article!

How to take advantage of Redis just adding it to your stack

Related Stories