RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

http://www.rethinkdb.com/blog/1.5-release/

by josegonzalez • 13 years ago

212 102 13 years ago

Very cool.. I'm going to try this.

manacit• 13 years ago

I'm really excited to see this project take off - I played with it a few months ago and it seemed like it was getting close to prime time. The two issues I see now are:

* More packages for more distros - It's great that I can install it on my OS X machine for testing, but I can only roll it out on an Ubuntu machine, and last time I checked the packages were a bit iffy when I tested them.

* More languages - It looks like you guys are working hard on this one, which is great.

Hopefully it'll be production ready soon! Great work guys.

mglukhovsky2• 13 years ago

Mike @ RethinkDB here. Great feedback-- while we've been a bit behind in rolling out releases to other distros and languages, we've been delighted to see the community pick up our slack:

- Our new build system has people successfully compiling from source and building community packages for CentOS, Slackware, Gentoo, Arch, and lots of other distros.

- One intrepid soul is porting RethinkDB to FreeBSD, where it builds successfully (almost ready to merge in): https://github.com/rethinkdb/rethinkdb/pull/688

- We've seen an enormous number of client drivers for different languages pop up in a short time, with some impressive collaboration:

    * rethink-net (C#/.NET): https://github.com/mfenniak/rethinkdb-net

    * php-rql (PHP): https://github.com/danielmewes/php-rql
    
    * rethinkgo (Go): https://github.com/christopherhesse/rethinkgo
    
    * cl-rethinkdb (Common Lisp): https://github.com/orthecreedence/cl-rethinkdb
    
    * haskell-rethinkdb (Haskell): https://github.com/atnnn/haskell-rethinkdb
    
    * lethink (Erlang): https://github.com/taybin/lethink
    
    * librethinkdb (C): https://github.com/unbit/librethinkdb
    
    * Lots of emerging efforts for Java, Perl, and other languages.

It's great to see people passionate about bringing our project to other platforms / distros, and we're doing whatever we can to support them. If anyone's interested in a similar project, shoot me an email at mike [at] rethinkdb.com

rdudekul• 13 years ago

I love RethinkDB API. It makes joins possible without going through mapreduce. Though I am using MongoDB currently, I plan on switching to RethinkDB as soon as it is more stable and provides better performance. Excited to see the team making progress on all fronts.

dude_abides• 13 years ago

It makes joins possible without going through mapreduce.

This is pure gold https://twitter.com/shit_hn_says material :)

mintplant2• 13 years ago

...Compared to MongoDB, which is referred to in the very next sentence.

RyanZAG• 13 years ago

I'm not sure what you think is pure gold here - the statement is correct. When dealing with big data, joins are the first thing to go as you need to shard your data. Whether RethinkDB will really solve joins across sharded data is up in the air though - ultimately you still need to do filtering per shard and then reduce the result somewhere - and doing filtering across two attributes means you are going to be doing a lot of data transfer, but RethinkDB is about the only database I've seen really making progress in this area.

hendzen5• 13 years ago

Seems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer.

Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.

jdoliner10• 13 years ago

Hi, I implemented secondary indexes in RethinkDb.

This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later.

Another reason though is we're not totally sure we're sold on the state of the art of query optimizers. From what we've seen they can introduce a number of non intuitive behaviors and it can be hard to tell which operations will use which index. Query optimizers are nice when they work correctly but they don't always work correctly so for the first release of secondary indexes we opted to make something that had very clear precise semantics and then build on it. We will eventually have a query optimizer but it's a big problem and we don't just want to implement what others have because we think we can do better.

antirez• 13 years ago

+1, a good approach could be to still make users able to specify manually what index to use even when the optimizer will be available. Sometimes humans just know better.

hendzen5• 13 years ago

Thanks for the reply. Do you have any resources you could share regarding the pitfalls of query optimizers?

My experience with them comes primarily from studying the System R optimizer where the literature presented query optimization as a boon to performance without mentioning such drawbacks.

Congratulations on the release by the way.

gleb• 13 years ago

A.k.a. "worse is better" approach. Seems like a good call to me.

sergei3• 13 years ago

A decent cost based optimizer will be right most of the time, and you have hints for the cases where it falters.

Your approach is simply hiding the query optimizer choices inside of the app, making it more fragile.

Let's say I have two fields: a and b, both of which have an index. And then I have a query with predicates across both a and b. For example, a = 10 and b = 20.

One of the core facilities within an optimizer is selectivity estimation. By looking at the statistics, the optimizer will see that a = 10 might look at 10,000 rows while b = 20 might only look at 10. So the optimal and desired choice would be to use index over b.

However, the exact same query construct with different parameters (e.g. a = 50 and b = 3) might flip the index selection.

Now let's imagine I had to implement this inside of my app. Every time I have such a constraint, the app has to become aware of selectivity to know which indexes to use based on input parameters.

mfenniak• 13 years ago

Hi Joe,

I just wanted to pipe in and say that's excellent reasoning for this design, especially since the ability to create indexes on arbitrary ReQL expressions would make an intelligent query planner into a lot of work. I'm looking forward to giving these a try.

wpietri• 13 years ago

Bless you.

The thing that makes me crazy about database engines is the "60% the time, it works every time" phenomenon, wherein things are magically fast until they are magically slow.

Then the discussion goes seamlessly from "don't worry your pretty head, just build your app" to "of course you have to really understand <proprietary, opaque, uncontrollable internal technology hidden by abstractions>." With no recognition that these are basically contradictory messages.

If you can really hide the complexity so it almost never matters, great. But here, where you couldn't, I think you made the right choice by exposing it.

JulianMorrison3• 13 years ago

Deterministic semantics are something I approve of. I'd rather have indexes under my control than the DB's.

Is it one index per chained operation (filter with index A then join with index B, for example)? Or can you only use one index per complete query?

hcarvalhoalves• 13 years ago

RethinkDB is looking great. The feature gap is closing, now I'm itching to use it in a real project.

agildehaus2• 13 years ago

Build fails on a "brew upgrade" for me.

http://pastebin.com/3GWetUTE

mglukhovsky2• 13 years ago

Thanks for the report, I've sent up the Bat-Signal for our @atnnn (our Homebrew expert) to sort this out.

coffeemug14• 13 years ago

Ok, we've got a fix for this. We'll release 1.5.1 shortly.

agildehaus2• 13 years ago

Looked into it and discovered it was a simple lack of the pyyaml python module. Should probably put that in your build instructions and have the configure script throw an error.

rb2k_• 13 years ago

    pip install pyyaml

fixed it for me

craigyk• 13 years ago

It's not always 100%, but who else think homebrew is awesome?

I miss it when I go to my linux machines.

Game_Ender• 13 years ago

What do you miss about it?

nickpresta• 13 years ago

I like Homebrew as well and I don't want to detract from how awesome it is, but dpkg + aptitude is much better in my opinion and I find I actually miss aptitude on OSX.

dscrd2• 13 years ago

Does RethinkDB do multiple atomic updates?

hendzen5• 13 years ago

Nope, operations on a single document are atomic, but not on multiple documents. MongoDB works the same way.

AFAIK, the only NoSQL DB with support for multi-doc transactions is HyperDex [0].

[0] http://hyperdex.org/

nanodeath• 13 years ago

OrientDB also has multi-document transactions, I believe.

prodigal_erik• 13 years ago

Also http://www.rethinkdb.com/docs/advanced-faq/. RethinkDB started out as a MySQL driver that was lock-free and did schema updates non-stupidly, but it sounds like they've abandoned ACID and schema enforcement entirely like almost all NoSQL systems (as if what I wanted is faster wrong answers).

sturadnidge• 13 years ago

I believe MarkLogic does too.

bdarfler• 13 years ago

http://foundationdb.com/ supports multiple doc transactions.

leif2• 13 years ago

Our (Tokutek) build of MongoDB with Fractal Tree indexes has atomic multi-document updates: http://www.tokutek.com/2013/04/mongodb-transactions-yes/ Come talk to us if you want to give it a spin!

Don't mean to steal the thunder from RethinkDB, they're doing great work too.

dscrd2• 13 years ago

And HyperDex only supports them via a proprietary plug-in.

dkhenry• 13 years ago

Looks like one by one the features are being checked off to make rethink a real contender. Keep up the good work guys.

jdoliner10• 13 years ago

Thank you!!

themgt• 13 years ago

We've updated our demo to 1.5 as well: http://rethinkdb-demo.pogoapp.com/

dennis822• 13 years ago

this is a good update and not to rain on their parade, but doesn't this seem market redundant and reinvent-the-wheel?

coffeemug14• 13 years ago

RethinkDB is definitely late to the party, so this is a great question. Here's how we think about it.

1. Technically, Rethink already does very useful things leading NoSQL contenders don't do. An extremely expressive query language, massive query parallelization, distributed joins, etc. You can't get that anywhere else, and we have lots of features in the pipeline to keep raising the bar.

2. I believe we have a unique take on usability and design that is very valuable to users but isn't present in other products. Rethink is pleasant to set up and (hopefully) pleasant to use. To us, design matters, and we put an inordinate amount of effort to make the product beautiful. It turns out users care about this a lot.

3. I feel like leading NoSQL contenders are somewhat stagnant. There is an enormous amount of innovation that could be done, needs to be done, and isn't being done. We'll continue releasing really useful features that nobody has, and (I think) nobody expects. I think we bring a unique philosophy to product design that's very valuable.

4. Details matter (which is why Rethink is late to the party). Once you outgrow the ten-minute blog stage, a lot of underlying architectural decisions start to really matter, and we took the time to do them right.

TL;DR: Rethink is already really good for building apps on top of it and offers things nobody else does. It will continue getting better. We'd be honored if you took it for a spin!

mintplant2• 13 years ago

I'd rather have had them take their time to get it right than rush it out the door to be first-to-market.

papsosouid3• 13 years ago

Wow, I thought this was a joke press release making fun of "web scale" fad craziness. They seriously gave a 1.0 release number to a "database" that you couldn't search?

apendleton• 13 years ago

No, you've always been able to execute queries. They just weren't indexed before.

jdoliner10• 13 years ago

Data has always been indexed it just was only indexed by a single primary key. Before this release the indexing scheme wasn't more sophisticated than a key value store now it is. I feel like there are several other products on the market that just do key value indexing.

papsosouid3• 13 years ago

I didn't say "execute queries", I said search. Unless you have a tiny amount of data, no indexes means searching is not feasible.

LeafStorm• 13 years ago

> Early in the development of RethinkDB, we made the design decision to be extremely conservative about durability and safety of users's data.

I'm glad that the Rethink engineers are taking this approach to development instead of "let's win all the benchmarks!" Overall, it's apparent that they are really thinking through the consequences of their decisions and the tradeoffs in database systems, and that they want to build something that will last instead of something that will just become popular in the short term.

jdoliner10• 13 years ago

Thanks. If we can succeed in being Mongo without the gotchas we'll be very happy.

Ecio78• 13 years ago

Congrats for your release, RethinkDB seems very interesting and I really appreciate how coffeemug and all the other team members genuinely and clearly reply to questions here.

Is there already a list of sites/services that use RethinkDB in production?

coffeemug14• 13 years ago

Hi :)

There isn't a list yet we can share, but we're starting to work very closely with a small group of early customers -- shoot me an email to slava@rethinkdb.com if you'd like to see if we can work together!

troyk2• 13 years ago

Am I the only one that hardly ever needs a case-sensitive query/index on string data? This is the one feature I miss from the MySQL glory days. With postgres, I use the citext type (case-insensitive text) but I guess Mongo and ReThinkDB expect you to either store the data as all lower or upper, or duplicate the field (for indexing).

coffeemug14• 13 years ago

The reason we (and probably Mongo folks) do it this way is because JSON is case-sensitive, so making indexes case-insensitive wouldn't work very well.

EDIT: it looks like I misread your comment. I was talking about index/field names, while I think you were talking about field values. In this case Sam's comment below is correct -- once we add string.toLower() (which is very easy) you'll be able to do case-insensitive indexing if you like because the indexing system supports arbitrary reql expressions. If you're using rethink and this is important to you, let me (slava@rethinkdb.com) know, and we'll prioritize this.

pixelcort• 13 years ago

Are the indices based on reql expressions updated incrementally?

How about making indices based on map-reduce? Would that work? Would they be updated incrementally?

troyk2• 13 years ago

Installed and played with (1.3 I think) and it looks very promising but the ability to query and order by case-insensitive text is a requirement for us. Our app deals with user provided names of people, places and things that are commonly searched and sorted.

So not using RethinkDB today, but I am 100% in love with what you guys are trying to accomplish.

shrughes• 13 years ago

If or once a string.toLower() function is added to RQL, you'll be able to have case-insensitive indexes in RethinkDB, because you can index on arbitrary functions of your rows.

psykotic• 13 years ago

Did anyone notice the name of the example database in the video demo? The RethinkDB crew must be Dota fans.

hoodoof3• 13 years ago

I like it alot but even simple queries are hideously complex. It doesn't need to be this hard. The ReThink developers are exceptionally helpful but goodness. Try to do a search returning all items where an array contains a value. In simpledb it's just select * where field='value', even if field is an array. In RethinkDB I had to ask for developer help and ended up writing a deep, sophisticated query. Querying has to be dramatically more simple.

coffeemug14• 13 years ago

We're actually working on adding commands to ReQL to make queries a lot simpler. (It turns out that just a few commands that are very easy for us to implement can make enormous difference)

If you can, would you mind giving examples of things you'd like to do that are hard? We'd love to make it easy, and we think we've got most common things covered in the next release, but if there's something specific you have in mind we'd really appreciate it.

hoodoof3• 13 years ago

The basic stuff needed to let a beginner developer write a web app should be super super simple. REALLY simple. No map no reduce nothing hard.

Here are some of the basic examples of typical things people might need to do when writing a web app:

How do I select records based on a value in an array?

How would I find all records whose name begins with "star" i.e. 'Star Trek' 'Star Base' 'Star Wars' 'Star Anise'

I have records which contain an array of email addresses. How would I get all documents which contain one or more of a list of email addresses?

How would I add a where clause to my queries to ensure for example that the "status" field in my document equals "open"?

How would I implement paging through a result set?

How would I implement count to control the number of records returned during paging?

What is the right way to store dates in ReThinkDB so that they can be queried?

How would I select all documents in a date range?

How do I do queries on fields that are within other fields in my JSON structure?

How do I do wildcard searches?

How do I define which fields are returned from my query, or does it always return the entire document?

How do I make sure that I do not add documents to the database that are missing required fields, thereby breaking all queries that do not contain those required fields?

It should be incredibly easy (and well documented) to do all the above.

I wanted to work with ReThinkDB but chose a different database because I was so intimidated by the query writing.

anatari• 13 years ago

So is this production ready? Excited about this, but I don't have time at the moment to dabble on things I can't use and the last comment regarding this is a post on Quora from the founder saying that it is not production ready yet. 1.0+ version numbers makes you think this isn't beta software anymore, but that doesn't seem to be the case.

coffeemug14• 13 years ago

We'll bump the version to 2.0 when the product is production ready (the 1.x versioning scheme is an accident that unfortunately we can't do much about).

RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements

Related Stories