Redis 4.0

https://groups.google.com/forum/#!msg/redis-db/5Kh3viziYGQ/58TKLwX0AAAJ

by fs111 • 9 years ago

588 58 9 years ago

groups.google.com

indeyets• 9 years ago

LFU policy sounds really interesting!

antirez2• 9 years ago

Thank you, there is more info in this blog post: http://antirez.com/news/109

leesalminen• 9 years ago

Awesome write up. I can't wait to try out LFU with our usage. I have a suspicion that our cache hit ratio will see a nice bump.

I've said this before and I'll say it again...thank you for all you do!

andyzei• 9 years ago

The point around visualizations for measuring and tuning is an excellent one with a great example. Thanks for the link -- great writeup.

Dowwie• 9 years ago

@antirez: Congrats! Are you going to modularize disque now that v4 is ready?

antirez2• 9 years ago

Yes, one of the top items in the Redis 4.2 roadmap is to port Disque as a Redis module, so the steps are: 1) Implement a Redis Cluster modules API (I've already some draft design, basically there are low level stuff to enlist nodes, broadcast messages, send one-to-one messages, and have callbacks called when we receive a message), and then port Disque as Redis module on top of such API, so that we can both validate that the Cluster API is good enough with a real project, and can make Disque a first citizen of the Redis ecosystem without all the code duplication, that in the end, made the project extremely hard to sustain. Thanks.

acidus• 9 years ago

I'm very excited about this!

ericfrederich• 9 years ago

I look forward to Aphyr scrutinizing it. No disrespect to you, his posts are always fun to read and show how hard it is to get this stuff correct.

StevePerkins• 9 years ago

"9) Active memory defragmentation. Redis is able to defragment the memory while online..."

I'm so amazed that this is a thing.

ihsw23• 9 years ago

Memory fragmentation is largely related to the allocator you're using (ie: glibc malloc, jemalloc, tcmalloc) and previously it was up to the OS to manage this (ie: freeing up unused memory).

Now with active memory defragmentation things are a bit more pleasant, specifically with high delete load actually freeing up unused memory in a timely manner without impacting performance too much.

Previously, to fully recover unused memory, you would have to restart the Redis server. Obviously this is not feasible but when Redis is using >50% more memory on a 120GB machine than it should then you will have to consider this an occasional housekeeping option -- now, as mentioned, this ridiculous task is no longer necessary.

xxs3• 9 years ago

120GB on a single thread, yeah. Redis has been abused for quite some time but 120GB is way out of reasonable reach for current CPU arch if you use a single core only (even if you switch off hyper threading)

FLUX-YOU• 9 years ago

Now with active memory defragmentation things are a bit more pleasant, specifically with high delete load actually freeing up unused memory in a timely manner without impacting performance too much.

Is Redis managing this itself using simple calls to Jemalloc, or is Jemalloc doing it on its own because it has better algorithms than the OS?

tgjsrkghruksd• 9 years ago

That's only part of the "os" if you mean part of the installed libc, as opposed to part of the kernel. Managing heap is done in userspace.

jackmott2• 9 years ago

It is one of those things where people say "Java can be faster than C/C++ because it can compact memory" and the C/C++ lady says "here, hold my beer"

xxs3• 9 years ago

To be fair only a compacting GC can do that

faragon• 9 years ago

Memory fragmentation can be avoided by construction, e.g. using trees allocated in linear storage (e.g. like std::map is implemented, or the maps in libsrt [1]). For strings a binary tree (or a B/B+ could be faster), and it would not require defragmentation nor rehashing (and worst access time would be log(n), instead of O(n)). Make growing the tree using realloc() is "free" on systems with virtual memory map/remap.

[1] https://github.com/faragon/libsrt

modarts• 9 years ago

Seriously, that sounds magical

praseodym• 9 years ago

Could anyone explain me why this is needed with virtual memory? Wouldn't the OS take care of handling gaps of free space?

frou_dh• 9 years ago

Maybe with the new modules support there will emerge some explicit way to do robust worker/job queues? So you don't have to remember your BRPOPLPUSH/LREM dances (or whatever it is) just so.

zedpm3• 9 years ago

Antirez has talked about moving Disque to a Redis module, which should solve that problem neatly. I'd love to drop our RabbitMQ dependency and move queueing into Redis.

juliensab• 9 years ago

Why?

Hates_• 9 years ago

Anyone know when we might see this on AWS (Elasticache)?

Thaxll• 9 years ago

Probably not soon because they use a fork of Redis.

actuator• 9 years ago

I don't like the Elasticache offering of Redis at all. It is severely limited if you are not just using Redis as a cache. There is no way to scale up/down the size of the node without putting it down(this is possible if you run it on EC2 as well as you have access to commands related to replication that are not allowed on Elasticache). We have had master restarting instead of failing over to slave and in the process coming back with no data. Even the snapshot is only taken once in a day.

dvirsky3• 9 years ago

If you want to try out some of the modules already available: https://redis.io/modules

jokoon• 9 years ago

It seems it even supports nearest search for lon/lat points by default... Quite nice since most RDMS don't even support it be default.

Although I'm curious to know what algorithm it uses for nearest search, it doesn't talk about it in the doc.

I don't really understand what redis should not be used for, I guess it's not for complex queries? Conventional RDMS really seem to belong to the hard disk drive age. So the difficulty resides in having well designed data schemes.

dvirsky3• 9 years ago

The entire redis geo capabilities are based on geohashing.

simonw• 9 years ago

I adore redis, I use it for tons of stuff, but I still like to keep my single-point-of-truth for critical data in an ACID, transactional relational database.

I frequently denormalize that data INTO redis so I can answer certain classes of queries quickly. I'm also very happy with redis for caching, rate limiting, inter-process-communication, distributed locking and tons of other fun use-cases.

infocollector• 9 years ago

Is there a Redis PPA for Ubuntu 16.04 that is supported by the redis team?

cihangirb• 9 years ago

docker images should be ready soon in case that is of interest. PR just went in - https://github.com/docker-library/redis/pull/98

ing33k• 9 years ago

Redis is of the the very few applications , that I install manually almost every time.

trick is to use the utils/install_server.sh

dvirsky3• 9 years ago

hey, I wrote that little script years ago, glad to hear someone (besides me) is still using it! :)

brango2• 9 years ago

Redis Cluster connecting to nodes via DNS instead of IP would vastly simplify deployment on K8s.

iagooar3• 9 years ago

How do you solve this problem? Can you use headless K8S services for that?

brango2• 9 years ago

You can use a StatefulSet which gives a fixed hostname per pod. Then you can have a sidecar pod that monitors the cluster, discovers the IPs of all pods, and uses redis_trib to create and maintain the cluster. It's a hassle to make it robust though.

anirudhgarg• 9 years ago

Any news on when this will be on Azure Redis Cache ?

https://azure.microsoft.com/en-us/services/cache/

tracker1• 9 years ago

Wouldn't expect it in anything less than 3-6 months tbh... depends on how anxious MS is to git this into place.

fapjacks• 9 years ago

I've said it before and I'll say it again: I am so smitten with antirez (and redis)! One of my favorite projects for sure.

HankB99• 9 years ago

Interesting coincidence (or maybe not...) Redis was discussed on the Floss podcast that I listened to earlier today and now I have an inkling of what Redis is. My first exposure to Redis was to ponder where it came from after I tried to run Gitlab on my puny (J1900, 4GB RAM and spinning rust) file server. It was spectacularly non-performant with most page loads timing out. I suppose it was because Redis had insufficient RAM for operation. Redis may be scalable toward large busy systems but seems less so in the other direction. I thought it would be cool to have a real Git server but this one was not it.

During the podcast the Redis guy mentioned that 4.0 was on the verge of being released.

blaenk• 9 years ago

Any reason why you lay blame on Redis and not Gitlab? Gitlab is what's in charge of serving and rendering the pages, after all.

notamy• 9 years ago

The issues you had may have come from GitLab not having enough resources available to handle everything, as opposed to Redis itself; anecdotally, I have an application with a couple thousand keys in it that only uses ~5MiB of RAM.

pfooti• 9 years ago

Yeah, I'm in the same boat - I have a redis I use for caching auth tokens and some other values for a small service, and it doesn't take much ram at all. (5-10 MiB)

bschwindHN• 9 years ago

I suppose it was because Redis had insufficient RAM for operation. Redis may be scalable toward large busy systems but seems less so in the other direction.

This is a pretty weak conclusion to draw. Did you try running Redis on its own without Gitlab? If not, I would encourage you to try it out as it takes up much less RAM than you think. The Raspberry Pi is even a supported platform...

Redis 4.0

Related Stories