LightCloud: Distributed key-value database built on Tokyo Tyrant

http://opensource.plurk.com/LightCloud/

by amix • 17 years ago

62 26 17 years ago

opensource.plurk.com

amix10• 17 years ago

Comments, questions and reviews are very welcomed.

bdr• 17 years ago

What does LightCloud add that Tokyo Tyrant does not already provide? I've read the websites of both products, it's just kind of confusing.

From some other comments, it seems I'm not the only one confused. Tokyo Cabinet+Tyrant are pretty new on the scene and there isn't a lot about them in English yet. So, if you add a level of explanation to the site that seems excessive to you, people would probably find it more useful than you expect.

amix10• 17 years ago

LightCloud adds horizontal scaling. If you just use Tokyo Tyrant then you can only scale by buying bigger servers. If you use LightCloud you can scale by buying extra servers.

When scaling upwards you would generally _really_ want to scale horizontally, since the vertical scale has a limit and you can quickly reach it (plus, buying bigger machines is generally much more expensive than buying extra machines).

catch232• 17 years ago

it's down!

btw, i'm also using tyrant myself, a very cool thing indeed!

siong19872• 17 years ago

I am wondering. From the benchmark, it is obviously slower than memcached. Why someone wants to use this instead of memcached which has better support?

amix10• 17 years ago

memcached is a memory based key-value database. LightCloud is persistent i.e. data is saved to disk. I'll make this more clear on the website.

nuggien• 17 years ago

there is memcachedb (http://memcachedb.org/) which provides persistence through berkeleydb.

siong19872• 17 years ago

I will be more excited to see how this key-value database actually helps to scale plurk. I am interested to integrate this into Rails if this really works very well.

henning• 17 years ago

Do you therefore see LightCloud as a possible alternative/complement to memcached? Certainly buying 500 GB of disk space is cheaper than buying 500 GB of RAM.

sebastian• 17 years ago

Isn't memcachedb faster?

Comparing the posted benchmark results with http://memcachedb.org/benchmark.html

You get around 2800 r/s using LightCloud vs. around 64000 r/s using Memcachedb

and around 1080 w/s using LightCloud vs. around 23500 w/s using Memcachedb.

I would be interested in seeing some benchmarks that compare both.

I really like LightCloud's idea of automatic scaling, failover and load balancing.

amix10• 17 years ago

Please do see http://news.ycombinator.com/item?id=498699 (memcachedb != LightCloud). memcachedb should be compared with Tokyo Tyrant and not LightCloud.

And if you liked, you could extend LightCloud with memcachedb support (which we also had at one point and ran it in production [see my posts on memcachedb mailing list for proof]), but really, when it comes to key-value databases, it's really hard to beat Tokyo Tyrant, which is the fastest and most feature complete key-value database out there (IMO and I have looked at most of the popular solutions).

jwinter• 17 years ago

That comment says: "memcachedb is not distributed, meaning that you can only scale vertically (i.e. by buying bigger machines)." Is that true? The docs on memcachedb seem to imply the opposite.

moonpolysoft• 17 years ago

How does it deal with events like disk failure, network partitions, and concurrent updates? The design documentation is rather light, so it's really hard to make out how this actually distributes data.

You say it doesn't have any concept of eventual consistency. Yet how does it coordinate updates to nodes? Does it do two phase commit? Paxos?

amix10• 17 years ago

Every node in both hash rings is replicated using master-master replication - i.e. node A and node A' can both receive updates and reads. Node A and node A' sync their updates via an update log and can fail at any time and come back at any time without taking down the system.

Additionally, if high availability is really a big issue, then a node A''' can be introduced that can be in another data center.

If you add nodes to the storage ring, then some of the existing keys will be invalidated. To solve this issue and the issue of routing a lookup ring is created. Lookup ring holds a pair (key, storage_ring_location). The system will automatically update (key, storage_ring_location) if it's at some point invalidated (such as that key does not point to node A, but node D).

I have tried to find an easy solution for a rather complex problem. Keeping membership state, doing Paxos and keeping routing tables would have been much more time consuming to make - so I have tried to solve the problem from another angel (by using master-master replication for high availability).

binarray2000• 17 years ago

amix, thanks for your effort (regarding both the development and the explanations here). Upvoted and will be considered for the next project. Keep up the good work!

trezor• 17 years ago

Like I said on reddit (http://www.reddit.com/r/programming/comments/814no/lightclou...), the performance seems somewhat lackluster. Especially considering the extremely small test-load.

I'd be more interested and might provide a somewhat less negative attitude if you were to do some real testing on a proper dataset (several hundred megabytes or gigabytes, not 10 bytes) and could show that adding servers actually improves performance.

The current test-data and test-script is simply insufficient to the point of being useless.

amix10• 17 years ago

Like I have already stated I am interested in how the system will run in production. Generally, you will do lots of small updates and lots of small fetches with key-value databases. You won't do batch operations - which makes your benchmark pretty irrelevant.

Try to benchmark your relational database by doing this: - create a new connection - fetch one row - close connection

And try to compare this to selecting multiple rows at once. The result will be MUCH different. And this basically outlines the difference between your benchmark and mine.

This said, you will only hit limitations with a relational databases if you are having lots of data. If you run a blog, a low traffic site or can keep all your data in memory, then you won't have any problems. And I do have experience in the world of relational databases and using MSSQL won't solve this problem for you (else you would see Facebook, Friendfeed, Twitter and Google etc. use MSSQL or Oracle).

leej• 17 years ago

Tokyo Tyrant has different kinds of databases LightCloud has support for all? I think so but just be sure.

Do you have any plans for developing a PHP API for LightCloud?

Thanks for your excellent work. I hope documentation will be improved.

dh2k• 17 years ago

Hey,

my test results are the following:

[root@server test]# tcrmttest write -port 1978 localhost 100000 <Writing Test> host=localhost port=1978 tnum=1 rnum=100000 nr=0 ext= rnd=0

......................... (00010000) ......................... (00020000) ......................... (00030000) ......................... (00040000) ......................... (00050000) ......................... (00060000) ......................... (00070000) ......................... (00080000) ......................... (00090000) ......................... (00100000) record number: 200001 size: 6928736 time: 22.460 ok

[root@server test]# tcrmttest read -port 1978 localhost <Reading Test> host=localhost port=1978 tnum=1 mul=0 rnd=0

......................... (00020000) ......................... (00040000) ......................... (00060000) ......................... (00080000) ......................... (00100000) tcrmttest: tcrdbget: error: 7: no record found record number: 200001 size: 6928736 time: 21.996 error

[root@server test]# tcrmttest remove -port 1978 localhost <Removing Test> host=localhost port=1978 tnum=1 rnd=0

......................... (00020000) ......................... (00040000) ......................... (00060000) ......................... (00080000) ......................... (00100000) tcrmttest: tcrdbout: error: 7: no record found record number: 100001 size: 6928736 time: 22.692 error

how can I reach the 1M put/get?

Looks like TT is around 2-3K records / sec in read/write. I've tested with all kind of table structures (on-memory hash, b+ tree, disk based hash, b+ tree, table, etc). and it was the same speed all the time.

LightCloud: Distributed key-value database built on Tokyo Tyrant

Related Stories