Simple Twitter clone using only the Redis key-value store as db and PHP

http://code.google.com/p/redis/wiki/TwitterAlikeExample

by paraschopra • 17 years ago

51 15 17 years ago

code.google.com

Redis actually looks pretty good alternative to traditional RDBMS. This tutorial really shattered quite a few myths about key-value databases' real world application. Anybody having experience with Redis? Anybody knows practical implementations of key-value dbs on a large scale?

davidw• 17 years ago

antirez, the author, occasionally hangs out here, and I know that he's using it in production.

christonog• 17 years ago

I actually first heard about Redis from a talk at the latest Ruby meetup in our area. The main things I took away from the talk is that it's really fast and is a pretty good way to manage sessions.

You can find more information at the Redis Google Code page:

http://code.google.com/p/redis/

antirez5• 17 years ago

Hello, there are guys using Redis in production with 350 million keys for server (64 GB ram for server). Also there are several other startups using Redis in production. Of course I'm using it in production myself.

If you search 'redis' in search.twitter.com you can see a lot of guys hacking with Redis in paywork environments. We are ready to release Redis 1.0 stable, it'a a matter of one month now. Then we'll have two development trees, one stable and one instable.

To deliver a rock solid product is one of my top commitment with Redis.

pipagiorgos• 17 years ago

Question: (newbie here) if we assume that with 350 million keys for the 64GB ram server we have reached the peak of it's capacity, if we double the number of keys, do we have to get new hardware, like a second server or double the ram?

Or is it something simpler? Like storing the current keys on disk (somehow) and keep going with the server as it is.

To rephrase my question: keeping all in ram... is "all" like ALL ??? Would this solution work for the "real" twitter, with the billion twits, if 350 million keys were created every hour?

dagheti• 17 years ago

The issue with key-value store isn't that you can't create an application, its that the tools to control data integrity and access your data in unexpected ways are missing or will be alot of work to create. A simple example is that two way key-value pairs for the UID and username in this article. Now any update to one value (or is it key?) must update the other.

If your main concern is scalability, then Redis and the like can look very attractive, but we should be clear on what is being given up: data coherence and query flexibility. That may be fine on your app, or even many apps, but its important to point out what is missing.

antirez5• 17 years ago

I agree in general, SQL is like magic (this is why it can't scale). You write abstract tables and then automatically you can do incredible queries against it, and this is very cool indeed.

But there are some kind of problems that Redis can fix in a very natural way. For example the idea of PUSH and LRANGE solves the get-last-N-items problem in a trivial way compared to MySQL.

Also Set intersection is able to solve tagging easily compared to the complex queries needed in a SQL db.

So SQL is very flexible, but strangely enough some trivial data access pattern is hard to model and slow even if it is as direct as take the last N items from a list.

pierrefar2• 17 years ago

Is it true that Redis requires that the whole dataset fit in RAM?

I've seen that mentioned a few times and if so, it's a serious consideration/fact to know about when deploying apps.

antirez5• 17 years ago

Absolutely true. This is why Redis can do things that with an on-disk storage engine are almost impossible. This our reasoning: in order to really scale, with every kind of DB, if data access is evenly distributed like happens in many web 2.0 apps, the DB must fit in memory anyway (being this OS disk cache, or directly used memory like in Redis). What really matters is persistence and replication. Asynchronous disk dumps gives us all this: data is persistent, replication is fast and non blocking and trivial to setup. And because everything is in RAM we are free to do high level stuff without to care about how data is stored on disk. On disk we have just a "stupid" dump that we reload a startup.

In some way this is what Paul Graham described in some comment years ago: take all the data in memory in your application, and just write a file in append-mode to reload it later. Basically this is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Redis :)

spoiledtechie• 17 years ago

Ya but what happens when Solid state hard drives come into play. And they will be soon.

I, a asp.net guy will be able to compete with Redis then.

pierrefar2• 17 years ago

Interesting. Thanks for the detailed explanation!

I'll be checking it out in detail (hopefully) soon.

petercooper• 17 years ago

On disk we have just a "stupid" dump that we reload a startup.

In a lazy fashion, or not? That is, if I have a 10GB data set, will I be either waiting X minutes for a basic restart or dealing with sluggish queries for a slightly lesser time?

(Update for anyone who wants an answer too: I dug around the docs and found that it's a full read into memory on restart, not lazy loading. They give an example of "It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID." So if frequent restarts are in your plan for some reason - not that they should be, of course - it might not be a goer.)

Simple Twitter clone using only the Redis key-value store as db and PHP

Related Stories