In-memory database Redis wants to dabble in disk

https://www.theregister.com/2023/10/19/redis_disk_support/

80 58 3 years ago

It is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added.

Seems way off the mark for why people use Redis. Developers use it precisely because it's not like their classic databases.

dimgl• 3 years ago

I don’t think that’s necessarily always true. I was recently looking into using Redis as a regular database. Unfortunately data is still kept in memory even if it’s persisted on disk, so it was a no-go. You’d be surprised what people want!

ReflectedImage• 3 years ago

Perhaps you are looking for LMDB?

tayo42• 3 years ago

What were you trying to accomplish? Curious about your use if you can share it

super_scooper• 3 years ago

You’d be surprised what people want!

If what you want is persistent k/v, then there are already tons databases that can do that. Redis has a different use case, so what are you trying to say? That you want to misuse tools?

reconditerose3• 3 years ago

I'll bite. I think with most relational and document databases, my view of the "classical world", you are often storing all of your data in tables or documents in a structured way. Normalizing the data where possible. You're then building indexes on that data to pull it out in a useful operational, optimizing for the most common operations. If a new use case comes up, you add some more tables, add some constraints, and create some new indexes.

I think with Redis, you are starting knowing that your use case needs to be fast. You start with the access patterns and fit that into key/value lookups (or search with modules). You're typically denormalizing the data, duplicating it or aggregating it from other places, so that you can serve operational data fast. Otherwise, just use in-memory SQL.

willsmith72• 3 years ago

The lines between data "stores" are blurring. I've seen it before where x team needs a fast key value store, so the obvious choice is redis. A year down the line, the product is evolving and the devs want to do some "classic" queries. Maybe it makes sense to keep it in redis based on their support for the queries you need, rather than migrating/replicating to a whole other system.

karmakaze• 3 years ago

They mention both flash and disk, which finally gets explained as tiers 2 and 3. Also less clear though I assume this applies to their hosted service offering and not the core software.

"We can take the lesser-used data that hasn't been touched in a while and shuttle it off to flash where it can sit for a while. When the user comes back eventually, it's very easy for us to seamlessly move it from flash back into memory. And that allows the company to save costs," he said.
Redis is now planning to extend the concept to disk-based memory to offer support for a three-tiered architecture.

SteveNuts• 3 years ago

I would like at least two more tiers: S3, and an LTO tape drive.

b112• 3 years ago

What about the stenographer tier? A simple terminal, a little transcription, and I bet we'd keep restore delay down under a few seconds + 5 minute file lookup of stored paperwork.

hivacruz• 3 years ago

For those interested in an alternative to Redis on disk, compatible with the Redis protocol, have a look on Kvrocks[1] which was recently accepted in the Apache foundation. It is based on RocksDB and works quite nicely for us.

[1] https://github.com/apache/kvrocks

chocolatkey• 3 years ago

Thanks for this, it seems to even support redis' lua functionality with is essential for us. I wish I hadn't decided to use redis stack since even kvrocks isn't compatible with its extensions

9dev• 3 years ago

What I actually need is a better way to bootstrap a cluster using Docker containers …

wutwutwat• 3 years ago

https://redis.io/commands/restore/

mrinterweb• 3 years ago

I was confused because I know Redis already offers persistence. https://redis.io/docs/management/persistence/ The main difference here is how its runtime will shift from runtime data only in RAM to not storing as much data in RAM and looking up on disk. Current Redis persistence looks like the runtime is stored in RAM and changes are periodically or append only persisted to disk. I hope I got that right.

whalesalad2• 3 years ago

One main criticism of Redis had been its lack of support for SQL, the ubiquitous query language. Trollope said that was fixed now. A module RediSQL is available on GitHub.

I think redisql has been supplanted by zeesql - https://zeesql.com/

Xeoncross2• 3 years ago

I don't understand this. How can you want SQL for something that you are using because it's faster than SQL?

If your dataset fits in memory and you want relations then just use PostgreSQL/MySQL. Why would you pick the unique data structures and performance of redis sets/lists/pubsub/HLL and want SQL in front of it?

rezonant• 3 years ago

because it's web scale

djbusby• 3 years ago

Memory only sqlite is a (rad) thing. Redis for KV things but one can have a shared, multi GB sqlite thing when you want fast sql things.

whalesalad2• 3 years ago

SQL is a query language. It has no real bearing on the underlying implementation.

thrixton• 3 years ago

SQL the language of choice for data manipulation, to me at least, although I wonder if I’m one of the dinosaurs after 20+ years.

After working with MongoDb, CosmosDb and DynamoDb, I’m very happy to be back in a proper SQL world with Postgres.

ddorian435• 3 years ago

Pay $7/month for the license for each index/db after the first 3. I bet oracle ends up being cheaper.

iot_devs2• 3 years ago

Ahahah

Author of zeesql/redisql here!

Unfortunately I was not able to figure out a clear business model for the module.

If any of you guys need it also the PRO version, just write me.

ohnoesjmr• 3 years ago

I've tried Redis-on-flash (enterprise version), and performance was very poor. Its just lsmt underneath and suffers from all the usual lsmt issues.

In the end we went with Aerospike which actually while backed by NVMe, outperforms redis backed by memory. Aerospike in memory is completely absurd speeds and throughput. Sadly the setup is a bit weird and query model is a bit wonky but we worked around it. A bit sad on how expensive the enterprise version is.

otterley• 3 years ago

Anyone know how their plans differ from the Virtual Memory plans that antirez had about 13 years ago, implemented in Redis 2.6, and then (wisely) abandoned?

https://redis.io/docs/reference/internals/internals-vm/

rjh29• 3 years ago

NVMe is a thing now.

infomaniac• 3 years ago

memcached has recently gained the ability to spill to disk: https://github.com/memcached/memcached/wiki/Extstore

we recently implemented this to grow our caches to >50TB

donatj• 3 years ago

It is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added. This initiative aligns with Redis's ambition to be seen as more than just a fast, albeit expensive, cache

This sounds to me like not understanding your place in the market or why people use your product. I can name half a dozen classic SQL databases off hand. I can’t name a tool that competes with Redis at their niche. Why aim for an already oversaturated market when you already have a good profitable niche.

vidarh• 3 years ago

Why aim for an already oversaturated market when you already have a good profitable niche.

Presumably because there are a lot of things that straddles the two, and where you risk losing people to databases or hybrid setups where the Redis use will be simple enough that people won't want to pay for support.

E.g. cases where you can't afford to keep it all in memory where you today might end up resorting to using Redis as just a dumb-ish in front of a database, or even avoiding Redis entirely in favour of in-memory tables or aggressive cache settings for your database even in cases where you might prefer Redis if it could intelligently handle a larger working set more cheaply.

And also to create a buffer given that the level of oversaturation in the classic database market may make their niche a tempting target to try to expand into for some of those to set themselves apart.

jstummbillig• 3 years ago

Assumedly, because they do know their place in the market and decided it's not great and that the outlook is worse.

Recently DHH of Rails fame gave a talk about Redis and how at their company it's being replaced in favor of long term storage on the now much faster and cheaper SSDs.

The takeaway was that the long caching it enables yields far better real world results in their products. He even had some charts.

geenat• 3 years ago

URL to talk?

tehbeard• 3 years ago

How would the example of a scoreboard work with tiered memory?

Surely you need the sorted set in memory to derive rank/manipulate it etc?

Or would you have to shard it and add other elements on top in app code to focus on only a subsection of the data?

pzmarzly• 3 years ago

Surely you need the sorted set in memory to derive rank/manipulate it etc?

If you store it as an augmented balanced tree, then you only need to load the path from root to leaf to get or change rank. For example, if each node stored the number of leafs of its subtree, then you can avoid visiting most branches by simple logic: if you want to find element with rank=14, and left subtree only has 10 elements in total, ignore it and go straight to the right subtree, searching for element with rank=4 there.

fumeux_fume• 3 years ago

Redisk... Sounds like a good April fools joke

iot_devs2• 3 years ago

How bad is Redis with a swap file on SSD?

I would expect it to be a very reasonable middle ground that requires almost no work and it is already enough and ready for most use cases.

reconditerose3• 3 years ago

It's really bad. The problem is that the swapping of pages is constantly stalling the main Redis thread, which is not built for parallelism. Redis is built around a single highly efficient thread, so stalling it for 250us to pull a page from disk instantly reduces the throughput by an order of magnitude.

andrelaszlo• 3 years ago

It's not mentioned in the article but this is Redis Enterprise which is more like a superset of open source Redis.

remram• 3 years ago

I think I would rather they added a write-ahead-log to make writes durable, than disk lookup to support bigger-than-RAM datasets (slowly).

In-memory database Redis wants to dabble in disk

Related Stories