RethinkDB: An open-source distributed database built with love over three years

http://www.rethinkdb.com

by coffeemug • 14 years ago

739 237 14 years ago

rethinkdb.com

Heff• 14 years ago

Congrats on the launch guys!

sutro• 14 years ago

How does RethinkDB perform when compared to open-source distributed databases built with hate?

bsg75• 14 years ago

Maybe not built with hate, but used in anger?

pc• 14 years ago

This took tenacity. Congrats on shipping.

jbellis• 14 years ago

I'll ask the obvious question not in the FAQ: How is this different from MongoDB?

coffeemug37• 14 years ago

Hey, this is Slava, founder of rethinkdb. There are some obvious high level differences:

* A far more advanced query language -- distributed joins, subqueries, etc. -- almost anything you can do in SQL you can do in RethinkDB

* MVCC -- which means you can run analytics on your realtime system without locking up

* All queries are fully parallelized -- the compiler takes the query, breaks it up, distributes it, runs it in parallel, and gives you the results

But beyond that, details matter. Database system differ on what they make easy, not what they make possible. We spent an enormous amount of time on building the low-level architecture and working on a seamless user experience. If you play with the product, I think you'll see these differences right away.

Note: rethink is a new product, so it'll inevitably have quirks. We'll fix all the bugs as quickly as we can, but it'll take a few months to iron things out that didn't come up in testing.

mej103• 14 years ago

What do you see as the potential areas where RethinkDB will shine?

Also, I am excited to try this out. I always enjoyed your writings and I am sure you + team have made something awesome.

fooyc• 14 years ago

> All queries are fully parallelized

Does it means that every query touches all servers ? Or does it sends queries to only a subset of servers when possible ? (e.g. range queries on PK)

lacker• 14 years ago

One apparent difference between RethinkDB and MongoDB is that in RethinkDB, you can only index on the primary key. I imagine secondary indexes will be coming along soon.

coffeemug37• 14 years ago

We're planning to add secondary indexes in the next few releases. Doing them well in distributed systems is really hard. It's relatively easy to check off a box and introduce it as a feature, to make it actually behave well and allow for good performance is incredibly tricky. This is why they didn't make it into this release -- since it's a core feature of every db, we wanted to take the time to do this right.

pspeter3• 14 years ago

Are there any performance tips or information? This looks really cool

masukomi• 14 years ago

Yeah, I'm curious too as to how it stacks up against MongoDB performance-wise and why I'd want to use one over the other.

3amOpsGuy• 14 years ago

I'm sure they'll provide performance examples in due course but we should be conscious how difficult it is to do this in practice.

Sure they can take the shortcut and just produce some single case and claim 'faster than a.n. other db' and that will probably satisfy the 'my db is faster than your db' brigade.

However, based on the effort that appears to have gone into this product to get it to this stage, they probably wouldn't be happy with that.

More power to them!

coffeemug37• 14 years ago

Performance analysis of complex systems is really tricky, and good benchmarks are even trickier. We'll be working through posting numbers, docs, tips, etc. over the next few months. Documenting this well is very hard work, so it'll take a little bit of time.

In the meantime, you can always chat with us and we'll help you work through any issues you might run into.

shawn-butler2• 14 years ago

Can I provide an intelligent sharding algorithm in place of a naive partition by primary key? I always get into trouble encoding things into primary keys eventually.

dested• 14 years ago

This certainly seems interesting enough to look into!

One note, there's a typo in the code in the tutorial on top

r.table('users).insert({'name': 'Slava', 'age': 29 }).run()

users needs a closing quote.

coffeemug37• 14 years ago

Thanks -- fixed.

wildmXranat2• 14 years ago

If authors are lurking here, link from github to a quick start page is broken: http://www.rethinkdb.com/docs/guides/basic_quickstart.html

GH issue https://github.com/rethinkdb/rethinkdb/issues/2

edit: fixed already!

coffeemug37• 14 years ago

Thanks -- fixed.

l0stman• 14 years ago

I was just wondering if you've planned on open-sourcing the code since the very beginning or if the idea came much later. Anyway congrats on launching.

coffeemug37• 14 years ago

We wanted to open-source all along, but there were concerns raised by investors about IP, etc. It took us a bit of time to work through all the issues people raised, which is why it took so long.

whyleyc2• 14 years ago

How do you plan on monetizing the business ? Are you looking at some kind of consultancy play, ala Redhat / MySQL (pre-Oracle) ?

mmorearty• 14 years ago

I don't know much about RethinkDB yet, but I will say that I have been a big fan (online) of one of its founders, Slava Akhmechet, for years. I've never met him, but he wrote some terrific articles on his website, http://www.defmacro.org/ , a few years ago. Start at the bottom of the list of articles, with "The Nature of Lisp."

Slava is a deep thinker, which makes me very excited to take a look at RethinkDB.

aberman• 14 years ago

I wish I could up-vote this a thousand times. Slava is one of the most genuine founders I've met, and I wish him and RethinkDB all the best.

jgw• 14 years ago

Indeed - he mentions in the article that he set himself a goal to convert 10 programmers into Lispers. Sounds like he probably has that many just in this thread! Kudos, sir!

samstokes• 14 years ago

He didn't make me a Lisper (I'm more a Haskell fan these days), but reading Slava's Lisp articles years ago was a significant part of what set me on my current career path.

He helped get me into functional programming, which got me a contract job [1], which is how I met one of my current co-founders.

[1] http://martin.kleppmann.com/2009/09/18/the-python-paradox-is...

reinhardt• 14 years ago

So is RethinkDB written in Lisp?

NathanKP• 14 years ago

No according the website it is written in C++.

ch0wn• 14 years ago

This looks really interesting. I'm interested to see how their license choice works out. The server is AGPL-licensed while the drivers are under Apache 2.0. This should at least avoid the issues we all know from libmysqlclient.

tbrock2• 14 years ago

Same licensing that 10gen uses for MongoDB (AGPL) and drivers (apache).

jedberg• 14 years ago

Suggestion: It would be great to have a page on your website that explains why RethinkDB is better than the other prevailing options. Right now I don't know why I'd want to invest time setting up yet another database.

coffeemug37• 14 years ago

Thanks -- will do in the next few days.

eventdroid• 14 years ago

yeh, that would be interesting to know.

stock_toaster• 14 years ago

Agreed. It would be very useful as a potential user.

As an example, I like how Basho provides some comparisons[1] of Riak vs other popular options.

http://docs.basho.com/riak/latest/references/appendices/comp...

xutopia• 14 years ago

What's the elevator pitch? Maybe we can help you with those advantages if you can tell us right now.

dkhenry• 14 years ago

I don't see much on the documentation on indexes. Also this looks awesome, I would love to see a option to let it be eventually consistent and still keep the great querying ability.

coffeemug37• 14 years ago

We currently support primary key indexing, but no secondary indexes. This is definitely planned -- it will take a few months to get this out.

EDIT: also, you can run queries with an out_of_date_ok flag, which will give you what you want. This only works for read queries though, the architecture is pretty much set up in a way where this would be very very difficult to do for write queries.

aterreno• 14 years ago

From the homepage it already looks more usable than mongo, liking the query syntax. Looking forward in trying the performances. Well done.

codewright• 14 years ago

I'm hoping this'll be a viable replacement for MongoDB. (Sparse/Schema-free is incredibly useful for me, as is JSON-centric modeling)

jedberg already asked for a compare/contrast, but let me provide some specifics I care about that you might be able to answer.

1. Is it fair to say that thanks to MVCC, running an aggregation or map-reduce job isn't going to lock the whole damn thing up like it does on MongoDB?

2. You've got a distributed system that is seemingly CP, do the availability/consistency semantics compare with HBase? Master-slave? Replication? Sharding?

3. Latency is a big one for us and is a large part of why we use ElasticSearch. How does the read-latency on RethinkDB compare with Mongo/MySQL/Redis/et al ?

coffeemug37• 14 years ago

1. Yes -- that was the main motivation for MVCC. We wanted to allow people to use rethinkdb for analytics and map/reduce on top of the realtime system without dealing with having to replicate data into something else.

2. Short answer: we favor consistency (via master/slave under the hood). It allows for much easier API, much fewer issues in production, etc. The user experience is just better. If you're ok with out of date results, you can do that too without paying the price of consistency guarantees. The downsite of our design is that you might lose write availability in case of netsplits (if the client is on the wrong side of the split). Longer answer: checkout the FAQ at http://www.rethinkdb.com/docs/advanced-faq/

3. Read latency should be equivalent to other comparable master/slave systems. We don't do quorums, so latency will be much better than quorum/dynamo-based designs.

rbranson• 14 years ago

I want to preface my comment: this is impressive work, congratulations on shipping, and this is what MongoDB should have been from the start.

In reality, most transactional database deployments are heavily skewed towards read workload, so reading from hot slaves is basically a requirement for master/slave databases. So, in most real world applications at scale, apps already deal with inconsistencies between slaves and the master and are making the "difficult" choice of dealing with CAP trade-offs. Asynchronous replication also creates a potential for difficult or impossible to recover from data loss in the sense that masters & slaves always have a continuous possibility for split-brain.

RethinkDB does not provide multi-shard transaction atomicity and/or isolation, which in my experience is the biggest difficulty thrown up in front of developers coming from single-node databases. I feel like the difficulty of dealing with inconsistencies across multiple versions of a single object is far more familiar as most developers have at least dealt with cache invalidation in some form. It's really having to ensure and deal with potentially out of order operations (inconsistency in the ACID sense) across a "graph" of data that's more insidious.

StavrosK3• 14 years ago

By the way, how safe is the JS interpreter? Can you get into trouble by running untrusted code in map/reduce queries?

gruseom4• 14 years ago

The most exciting news in quite a while! It will be interesting to hear what people think as they try it out and differences begin to emerge. In the meantime, congratulations on both the release and the open-sourcing.

continuations2• 14 years ago

* In the previous incarnation of rethinkdb the focus was on maximizing performance on SSDs. Is this still the case - does rethinkDB perform better than other databases on SSDs? Do you have any benchmark numbers?

* How does rethinkdb compare to MySQL Cluster? Both are distributed, replicated databases with a sql-like query language.

* Any plan to offer a java client?

coffeemug37• 14 years ago

* The SSD-optimized storage engine is running under the clustering engine. I'm wary of saying 'better' or 'worse' in case of benchmarks, because they're really tricky to do right. We'll be publishing well-researched benchmarks as soon as we can, but it will take time.

* RethinkDB has flexible schemas and a query language that integrates straight into the host programming language and doesn't require string interpolation. As far as clustering goes, RethinkDB is a) really really really easy to use, and b) does a lot of query parallelization and distribution that MySQL cluster doesn't do. The product feels totally different, I think in a good way. The downside, of course, is that rethink is new and it will take some time to work out all the kinks.

* I can't commit to a timeline yet, but yes, absolutely.

Xorlev• 14 years ago

I'm impressed that you're taking the time to do proper, well-researched benchmarks. They're really tough to get right. It really comes down to your own specific workload that matters anyways.

This feels new and refreshing, I hope things turn out positively for you.

hagope• 14 years ago

Great work, can't wait to try it out! :-)

tjic• 14 years ago

What the heck does "built with love" even mean?

Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages?

I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.

jacquesm• 14 years ago

Releasing a project like this means working very hard for a long time without anybody patting you on the back saying you're doing a good job. It's exhausting, a labour of love.

Get it now?

haberman4• 14 years ago

As someone who's been doing that for several years on a side project that I believe in, this really hit home.

whyleyc2• 14 years ago

I reckon $1.2 million in funding helped soften the pain :)

gruseom4• 14 years ago

You could not have gotten these guys more wrong. They are serious technologists who have been working day and night for years to build something that they deeply believe in. Every hacker's heart should be warmed by the fact that they kept at it.

When you have a vision of something great that ought to exist and set about bringing it into the world, you are in an isolated position: other people don't yet see what you see. This leads to a lot of doubt by others and by yourself too. The longer it takes, the more exposed you are. To make it through that you are going to need a deeper source of motivation – an underground spring. Love is a fine word for this, and it makes me happy that Slava put it in his title: it's a clue to this experience that rarely gets mentioned, especially in the land of pivots and MVPs and weekend hacks.

daslee1969• 14 years ago

cannot agree more. saw firsthand how hard this team kept at it!

benatkin• 14 years ago

Since it's under the AGPL it will mostly be built by people that have been vetted. By switching to this people are one step closer to having a machine that boots without assholes. http://rusty.ozlabs.org/?p=196

VeejayRampay• 14 years ago

You look like an angry person man, chill the fuck out.

Rule of thumb is if you build something this nice and with that order of magnitude in complexity you can put My Little Poney stickers on your homepage and still get respect. Who cares about the "attitude" and the "language" for Christ's sake, they BUILT stuff with their own hands and are offering it to the world, they can do whatever they damn please.

ketralnis• 14 years ago

What does hipster even mean here?

oh_sigh2• 14 years ago

"People who I don't like and people who I think they think they're cooler than I am"

amazedsaint• 14 years ago

Made with love = Made not just for money, but Made for making the world a better place.

eknkc• 14 years ago

So, no secondary indexes right?

Will doing a query like "age > 25" perform something equivalent to a full table scan?

coffeemug37• 14 years ago

Yes, there are no secondary indexes right now (they're coming, but it will take a little time). Currently "age > 25" will do a scan, but if the db is sharded it will touch only the relevant shards and the query will be completely parallelized. We also do internal sharding for multicore performance, and the query will be parallelized across cores as well.

All of that is only good for relatively small amounts of data though, so we'll be adding secondary indexes soon.

spicyj2• 14 years ago

How does it know which shards are relevant? I thought that data was sharded by primary key.

jdoliner12• 14 years ago

Correct currently no secondary indices. Ranges on fields other than the primary key will require a full table scan.

-Joe Doliner, engineer at RethinkDB

jamesli• 14 years ago

Great work! One question: is there any manual that explains the implementation details of the internals? Some manual similar to those Oracle, MySQL, Postgres, etc. provide?

The only docs I found in the company website that goes deep into the internals are Advanced FAQ (http://www.rethinkdb.com/docs/advanced-faq/). It is more of an architecture view, though.

The reason I ask is that with a good understanding on the internals, the engineers who understand database internals and distributed systems will have an "more" accurate idea on the capabilities and the limits of the features. Thus, if they decide to adopt RethinkDB, the understanding will help them design their applications to take advantages of the benefits and avoid the potential issues (or surprises!). MongoDB was not very good at documentation. It claims this or that feature works smoothly. Then, people found out many potential issues and limitations. That is one reason it leaves a bad tastes to many engineers.

coffeemug37• 14 years ago

There currently isn't, beyond the advanced FAQ. This isn't by design -- writing really good detailed architecture papers takes a lot of time, and we were 100% focused on getting the product out. We'll get much better at documenting the internals, but it will take some time.

sidcool• 14 years ago

Catchy title, I must say.

wildmXranat2• 14 years ago

Is the package up on Ubuntu PPA already ? it seems that the installation instructions use the ppa, but apt-get doesn't find the package.

edit: Indeed, my architecture i386 doesn't match the only available amd64 binaries. Thanks

coffeemug37• 14 years ago

The package is up. Which ubuntu version are you using? We support 11.04 and above. Anything less than that is missing some kernel features we use.

EDIT: the main thing missing from earlier ubuntu versions is TCP_USER_TIMEOUT. We can work around it in the server, but we haven't done it yet.

arzvi• 14 years ago

Query language and ruby like function-chaining are what I feel the selling factors. I like the ease with which I added a node to the cluster. But naming the version as Rashomon scares me..

lobster_johnson• 14 years ago

A release named Rashomon should be the one where they introduce versioning support, in my opinion.

joevandyk2• 14 years ago

Why would you use this over PostgreSQL, especially with pg's new json support?

tcwc• 14 years ago

JSON support in Postgres is currently limited to a validated plain text field, it doesn't let you efficiently query inside the json object.

shawn-butler2• 14 years ago

While technically correct I assume the parent was referring to hstore [0] which has both GIST and GIN indexes as well as btree and hash for equality.

With some simple formatting functions (its just json after all) it's sixes to me.

[0] http://www.postgresql.org/docs/9.2/static/hstore.html

joevandyk2• 14 years ago

There's nothing stopping you from writing functions that can index json documents. plv8 makes this easy.

perfunctory2• 14 years ago

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

crazygringo• 14 years ago

I have nothing to do with RethinkDB, but what are you talking about? Just represent them as strings. (What databases support colors as native types, anyways?) If you format dates YYYY-MM-DD, then you can do string comparisons for ranges.

And JSON has the huge advantage of supporting hierarchical data -- arrays with objects inside, etc. It seems a like a huge step forward.

kemiller2• 14 years ago

But there's no standard way of representing the dates as strings, or indicating that this field here is a date and not a string that happens to look like one, so nothing can rely on what you do. You can represent literally anything with a string, but you lose type information when you do.

lttlrck• 14 years ago

Or unix epoch if you want an easy life when it comes to querying.

optymizer• 14 years ago

So because you have no idea how to represent some arbitrary things with JSON, we should all stop using it? You, sir, are not making any sense.

Also, you seem to be confusing primitive data types with complex data types. Yes, JSON doesn't have a 'color' data type. But guess what? Neither does C, nor Java. If you want a 'color' type, you'll have to create one yourself! Mind blowing, I know.

So here, let me suggest a possible solution:

dates: string

timestamp: integer

colors: string (RGB,BGR,RGBA,...), integer, object

Part of 'data modeling' is to model your data out of basic types. Shocking! If JSON had types for every type of object under the Sun (like you seem to want), JSON parsers would be a lot more complicated and little thinking would be required in the process of modeling your data.

perfunctory2• 14 years ago

json definitely has certain advantages. e.g. arbitrarily nested data structures. However. json is not extensible by design. There is NO standard way to define new custom data types in json.

jedahan• 14 years ago

Last I heard RethinkDB was a tail-append style engine for MySQL that was optimized for SSDs. Interesting to see a drastic pivot like this. Looks good, and good luck.

szopa2• 14 years ago

Nice work! It seems that you are well aware of the tradeoffs that you are taking and communicating it openly in your documentation (and your choices seem to be very reasonable). I really like the tone of your communication – it seems essentially BS/koolaid free.

1. How much data can you put in one instance before seeing performance degradation? I know that you still working on good benchmarks – but do you have any ballpark figures?

2. How does replication work? Is it closer to row/document or statement based (or something completely different)? How fast is the replication?

3. What is your envisioned used of the replication? Are replicas supposed to serve read traffic, or their goal is to keep the data safe in case of a catastrophe?

4. Can you tell me something more about cluster configuration propagation? The Advanced FAQ answer doesn't get into much detail.

5. Am I correct to assume that you are using protocol buffers? What motivated your choice?

coffeemug37• 14 years ago

1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset is < ~80GB the system will perform well. Once things get out of RAM, everything will work well if you have an SSD. On a rotational disk, performance will degrade very very quickly. This is a bottleneck on all modern databases, but clustering makes this problem go away because you can effectively increase the amount of RAM at linear cost by just adding more nodes (e.g. two nodes at 100GB of RAM each cost about four times less than one node with 200GB of RAM).

2. We do do block-level replication. On each node of the btree we store replication timestamps. When a node asks for new data, we can cull away parts of the tree the node has almost instantly. So replication is very very efficient for most OLTP workloads. We don't have statement-level replication yet, so if you do a range update on a large table, we'll have to replicate data block by block. It'll take a while to add statement-based replication - we'd have to do a pretty significant refactoring to make it happen.

3. Either. Replicas are great for failover -- if the master dies, you just failover and a replica picks up where the master left off. If you're ok with out-of-date reads, you can also hit replicas directly (e.g. for reports, etc.) and spread out the read load across the cluster.

4. This is a really complex question - we didn't document this because doing it properly would take a lot of time. I'll ask jdoliner to chime in -- he designed the architecture and wrote most of the code, perhaps he can describe it succinctly while we write deeper docs on this :)

5. We use protocol buffers between the client drivers and the server. We picked that because there were libraries for the initial three languages we picked (Ruby, JS, Python), they were really easy to use, and very efficient. We could also have a single spec for the client/server API. Internally we use our own serialization scheme which allows us to dump arbitrary C++ objects on the network. It doesn't support other languages (which we didn't need), but is much more versatile for writing complex cross-machine code.

jdoliner12• 14 years ago

Small correction to this. There actually wasn't a library for JS we had to write that for ourselves.

jdoliner12• 14 years ago

Hi, here to answer question number 4.

Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge.

Long Answer Cluster configuration is stored in semilattices which are a neat mathematical structure with a few very desirable properties. Semilattices support have a join operator. For our cluster metadata joining is the means by which metadata is updated. When one server connects to another the two swap metadata and each joins the other's metadata into his own. In essence learning what the other knows.

There are two properties in particular of the joining that are nice. First off joining is commutative. This means machines can exchange data in whatever order they want and get the same result at the end. Secondly they're indempotent. That means machines can resend their data without fear. The value doesn't change if the same value is joined in twice. These help us with a lot of the worries of distributed systems.

szopa2• 14 years ago

Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times?

If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (from the point of view of different instances) there are two masters at the same time and some writes are sent to the wrong instance?

EDIT:

One solution for such things is to use something like Zookeeper (or some other system whose documentation mentions "Paxos" ;)). Have you considered that? How does what you are doing compare with that?

yxhuvud• 14 years ago

Now that you mention it, it would be very nice to have a database suited for configuration that behaved like git in that branching, restoring old states, reverting selected commits was built in, while at the same time supporting ACID features and replication?

Does anyone know if something like that exist?

krob• 14 years ago

I'd really like to know why they don't have a PHP library, considering it powers half the web. The key shouldn't be to promote someone elses own language when using infrastructure products but rather to support everyone in using their tool. Mongo supports everything under the sun, so should this.

alexpopescu5• 14 years ago

... and it will.

We had to start from somewhere and we also wanted to get a feel of what are the most requested libraries so we can focus our energy in those directions.

alex @ rethinkdb

sergius• 14 years ago

No C++ API???

DaemonXI• 14 years ago

Will there ever be a 32-bit version?

coffeemug37• 14 years ago

It should be easy to do a port, but it's one of the many things to do. I can't commit to a timeline, but it will probably happen.

shin_lao• 14 years ago

Congratulations! How do you plan on making money?

harryh2• 14 years ago

If I were you guys I'd strongly consider adding support for hashing of the shard key. There are many cases where you care about distributing your writes(1) a lot more than fast range queries on the PK.

-harryh

1. Yes, I know there are other ways to do this besides hashing the shard key, but this is often the best way.

coffeemug37• 14 years ago

We actually support hash sharding underneath -- each range shard is further broken down into hash shards internally to support multicore scalability. This isn't exposed to the users currently, but it can be. Another option is to allow the user to provide a hash function on the PK (I think this is what you're suggesting).

We'll be addressing this at some point, we have to sort through the list of feature requests first. It's a long list :)

harryh2• 14 years ago

I don't feel the need to provide the hash function, as long as it's something reasonable. I understand that it's a long list! Just adding a vote to one particular item. Good luck!

bjhoops1• 14 years ago

The querying capabilities here look amazing. Having to manually figure out how do joins and group by in something like CouchDB is really a pain, but this looks really slick. Very impressed and I will be trying this out!

jdoliner12• 14 years ago

We spent a long time trying to reimplement other people's protocols but with our engine underneath. Being able to have features like this is one of the things that eventually convinced us it would be worth it to control our own. Bear in mind that our joins are also distributed which we think is really cool.

d0m• 14 years ago

I absolutely love the website. Congrats on the public launch. In the FAQ, I would suggest a "How do you compare with Mongo?" I've read the intro, the faq and a couple of quick guides to find out what was different (read better). If I'm a happy mongo user, why would I switch to RethinkDB?

alexpopescu5• 14 years ago

1. If you are a happy mongo user, we'd still be happy to hear your feedback about RethinkDB.

2. We're putting together some comparisons, hope to add them to the site soon.

3. There are already a couple of answers to this question on this thread:

https://news.ycombinator.com/item?id=4764137

https://news.ycombinator.com/item?id=4763939

alex @ rethinkdb

shykes• 14 years ago

I am very excited about this. The RethinkDB team is rock-solid and the market is only going to get bigger.

I particularly like the perspective of an easy onramp to get started, knowing that I will never have to leave because of scale or reliability.

Please, please give me a SQL adapter! My marketing team needs SQL. My business app developers need SQL. Give them an adapter and I will get them to use RethinkDB - knowing that 1) my data is safe and I'm not 6 months away from a painful re-architecture and migration, and 2) as my developers hit the limits of SQL they can gradually (gradually!) peel the paint off and start using your more powerful query language.

sandstrom• 14 years ago

On the contrary, I hope you [RethinkDb] don't spend time on an SQL adapter. I'd rather see time spent on improvements to the database itself.

haberman4• 14 years ago

Is schemaless a win over an object schema like a JSON schema (or a Protocol Buffer .proto file)?

Schemaless is clearly a convenience win over SQL because SQL's way of modeling nested/repeated data doesn't map as easily onto programming languages. But for all the people who are using JSON-based databases these days, I'm curious how many of them couldn't easily write a JSON schema or a .proto file that describes their de facto schema.

I ask because a lot of things become easier to reason about (and optimize) if you know that a field won't be a string in one record and a number in another. And writing a .proto file (or equivalent JSON schema) would give you an authoritative place to document what all the fields actually mean.

I don't have any actual experience with JSON-based databases, so I was interested to hear the opinions of people who do.

embwbam• 14 years ago

this. I don't like SQL columns. They make life hard. But I'm spending time learning TypeScript specifically so I can add some types/schemas to my JavaScript.

That doesn't mean I want to deal with the implementation detail of columns, but I definitely wouldn't mind some type safety.

ollysb• 14 years ago

I hear a lot of talk about how hard it is to maintain sql schemas and columns. I've used mongodb on projects previously and while it was interesting I didn't find that the lack of schema made life any easier. Writing a migration in rails is so easy, I just can't understand how managing a schema really makes life more difficult. What is it about managing a schema that makes people so eager to jump to schema-less?

coffeemug37• 14 years ago

There is of course no fundamental reason why JSON-based db's has to be schemaless. This is one interesting direction that might be worth exploring.

m0th874• 14 years ago

I would love a system that is schema-less by design, but has guards that can be enforced at insert/update. That way, the underlying data structures don't have to be locked up from complex migrations (as needed w/ ALTER TABLE), but you still get type safety. A migration instead would simply involve a change in guards and an asynchronous update of existing entries. Plus you'd get all the wins of something resembling optional types: you only enforce guards if you want.

banachtarski• 14 years ago

Would love a comparison of this to couchbase which seems to have a similar sharded distributed setup.

Congrats on shipping!

m0th874• 14 years ago

How do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.

old-gregg• 14 years ago

Nope, they build an AST for filter expressions and compile it on the server IIRC. The client gets filtered data from the server.

jdoliner12• 14 years ago

Piping all the data to the client would be extremely inefficient. Fortunately we don't do that.

When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter out the rows which do not meet the query predicate. This data gets returned to the coordinating node and eventually to the user. Thus only the data the will actually be returned is ever transferred over the network.

Furthermore this process is done lazily. On the client side rather than getting back a huge array with the results of your filter you get back an iterator. This iterator stores a buffer of data which will be refilled as it is incremented.

coffeemug37• 14 years ago

To add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there.

It's rather difficult to integrate into a host language like that smoothly from the driver implementation perspective, but once the driver is written the user experience is amazing because you can write queries that look exactly like Python, but they're executed entirely on the server.

jpettersson• 14 years ago

Wow, looks great! I'm excited to try it out.

The Github graphs are really interesting too, that's a lot of love/work right there!

https://github.com/rethinkdb/rethinkdb/graphs/impact

dbro• 14 years ago

I wonder what happens in the RethinkDB office on Monday evenings? https://github.com/rethinkdb/rethinkdb/graphs/punch-card

mcartyem• 14 years ago

Are there plans for an Arc driver?

jdoliner12• 14 years ago

PG said he'd post facto reject us from YC if we don't.

In all seriousness while we'd eventually like to have support for every language Arc is farther down our list than others such as PHP and Java. We will get to it eventually though.

If Arc could be made to use protocol buffers it wouldn't be too hard for a contributor to write the driver themselves though.

coffeemug37• 14 years ago

Hey guys, Slava here. I've been up since yesterday, so I'm going to clock out (though some of the team members are still lurking here). I wanted to thank everyone for great feedback. We're working hard to improve Rethink over the next few months. FYI, you can always hop on IRC (#rethinkdb on freenode) or github tracker (https://github.com/rethinkdb/rethinkdb/issues) with questions and we'll help you out.

biturd• 14 years ago

Thanks for this work, it looks really nice.

I was looking at the github comments about a home brew recipe in which it was stated that aside from a recipe creating a VM, the Mac OS X port would take a bit longer.

Is that a full port from one language to another? Or just an issue of the different flavors of *nix that need dealing with and probably some of the dependency tree issues that come with it?

I'm curious what needs be done to get it building on Mac OS X — perhaps I could assist somehow.

I see a few dependencies that don't immediately sound familiar. You may have better luck with MacPorts, which uses tcl as the language for their portfiles.

Portfiles are just like homebrews recipes, but MacPorts always builds new, including the entire dependency tree ( and dependencies of dependencies etc., etc. ), for which they have thousands of working portfiles. Since those are completed and working, you wouldn't have to worry about those until you wanted to be able to make a binary outside of any package manager.

MacPorts can build binaries now ( new feature ), so you could just as easily instruct it to create a standard Mac OS X installer .pkg which makes sure everything goes in the right place, on the right platform, for the right architecture.

They are an exceedingly friendly and helpful group, I'm sure they would live to see this software in their package/portfiles list.

i3862• 14 years ago

Seems to me that most of us who have used MacPorts have moved to Homebrew or that could just be the bubble I'm living in. Is there anyone still who still uses MacPorts who could chime in and say why they never made the switch?

coffeemug37• 14 years ago

Most of the issues come from kernel differences. Some are relatively big, like epoll vs. kqueue, and some are very subtle (some syscalls have subtly different behavior on strange edge-case scenarios). I don't think building this natively would be too hard, but in the context of everything else left to do, it isn't an absolutely trivial project.

dxbydt• 14 years ago

Thought I'll share this with you.

A yc company hired me. I showed up at their mountain view office. The founder said "This is the former office of RethinkDB! I hope we are as successful as them."

I didn't know who/what RethinkDB was, so I said ok, sure.

3 days later he asked me to clear my desk and leave. He said "You are the sort of person who should work in RethinkDB".

So I asked "What does that mean ?"

He said "RethinkDB is trying to solve very deep algorithm problems. They want somebody with CS knowledge to do deep research. That is what you are good at. But here we are just trying to run a business. You are not a good fit for that!".

So I left.

fragsworth• 14 years ago

I know lots of engineers who have trouble talking to people who don't share their knowledge. This problem is extremely pervasive - I'd say a good 25% or more have this problem to some extent. It's not a good thing when this happens - you need to be able to speak to laymen or you're gonna have a bad time.

I am going to go out on a limb here and suggest you try to work on being a bit more practical. Don't complicate things for the sake of solving difficult problems. Don't try to shower people with your engineering knowledge when it's not necessary, and don't expect everyone to know everything you do. And don't be an asshole about it either.

borplk2• 14 years ago

I adjust the level of details and technical stuff pretty well according to who I am speaking to.

But sometimes someone with very limited knowledge of something asks me a detailed question about X.

What they ask is too difficult and complex to be described in a simple way. Either I have to overly simplify it which may insult them and will do no good or I have to go on and step by step give them digestible chunks of explanation which will inevitably be a bit technical even though I try to minimise that.

I think it's a two-way thing. The person should also consider their own level of knowledge before asking for an explanation of something and adjust their question based on that.

I don't ask my doctor to tell me why my heart does this and that because I simply don't have the knowledge to be able to understand his answer. I ask, my heart is doing ok? cool!

flogic• 14 years ago

Wow that's special. Sounds to me like they're tools. You can't really expect an employee to know they're way around the code base after only 3 days. Hell most places you find yourself sitting on your thumbs the first week due to everyone being too busy to spend much time orienting you.

tsiki• 14 years ago

My thoughts exactly. I sometimes read about these people getting fired from IT jobs after a couple days and keep wondering why their bosses even hired new people. Either the boss is too overconfident in their ability to gauge someone's skills based on a couple of days, or they never really needed new people.

erichocean7• 14 years ago

I find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type and a timestamp/date type, both of which SQL provides (and are used A LOT).

Sure, you can put that stuff in strings, but then you'll run into limitation with queries where you want to, e.g., aggregate a total, or do timestamp arithmetic.

I could do everything with strings, custom map-reduce, etc., if you're inclined to suggest that as a workaround. Still doesn't mean JSON's a good idea.

cuu508• 14 years ago

Or you could use fixed point math: 600 in database means $6.00 Then aggregates and comparison operators would work, but you would have to decide upfront how much precision you might ever need

erichocean7• 14 years ago

That sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days.

A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.

erichocean7• 14 years ago

The other thing that bothers me about all these new JSON databases is they aren't really novel anymore.

Clustered databases are essentially a solved problem, and have been for years. What's needed today are databases solving the problem that Google Spanner addresses – global consistency across distributed clusters in separate data centers. If you want a challenge in the DB world, that's where it is.

But another clustered, schema-less JSON database? Might as well open up Intro to Algorithms and run through the exercises -- it's no longer a challenge, algorithmically or otherwise.

Sorry to be a downer on this, and it does still take a strong coder to implement one, so well done on that front. :)

jchrisa• 14 years ago

Try keeping it running while growing to millions of users in weeks. The simple data model lets us focus on elasticity and performance. There's a lot more to production quality software than the algorithms, but there are only a few NoSQL databases that get the algorithms right.

oh_sigh2• 14 years ago

Just store it in a string. Or a special map structure: {"date": "2012-11-01", "format":"YYYY-MM-DD" }

alexpopescu5• 14 years ago

We thought of supporting data types that are not part of JSON (date/time/timestamp/deltas, etc.), but we wanted to take the time to do it right so these didn't make it to this version.

alex @ rethinkdb

javajosh• 14 years ago

What is the business model, if any? (This question is not addressed in the FAQ, and I believe has at least some relevance to the longevity/shape of the reDB community over time.)

Also, have you talked to the Meteor folks about swapping Mongo out for this? Or would this be 'newness overload'?

ww5207• 14 years ago

I like their DSL-style query API. It's fantastic!

Disclosure: Ok. I'm biased. :) I've designed a similar DSL-style query API in another project. https://github.com/williamw520/jsoda

ww5207• 14 years ago

Congratulate on releasing. Well done!

A few questions:

1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query.

2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients?

3. Related to 2. How can loosely-sequential Id be generated using a table?

4. Will some transaction support be added? Don't need full ACID, just grouping updates (intra-table and/or inter-tables) in one shot would be nice. Should be feasible with MVCC already in place.

5. Do all the clients hit a central server to initiate queries which then farms out the requests to different shards? Or the client library knows how to get to different shards directly? First case has a single-point-of-failure, and bottleneck in scaling.

6. Do you support automatically re-balancing of shard data (data migration) when new shards are added or old ones retired?

7. How are authentication and authorization done? Or any clients can come in?

8. Internal detail. For out-of-date distributed query on the slave replicas, is there a cost-based (or load-based) decision process to pick the most idle replica to do the sub-query?

9. Internal detail. Do you use Bloom Filter to optimize distributed joins?

DanielRibeiro• 14 years ago

Did they not launch a while ago: http://techcrunch.com/2011/06/06/rethinkdb-expands-beyond-ss... ?

ww5207• 14 years ago

Well any product release is a huge effort, especially database product. Things got done and pushed out of door. Congratulation well deserved.

jdoliner12• 14 years ago

> 1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query.

Secondary indices are one of the most asked for features so they'll probably be added in the next release. No promises though secondary indices are tough to do right and we won't ship them if they're not great.

> 2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients?

Updates can be done with conditions on the row. For example: table.filter(lambda x: x['age'] > 25).update(lambda x: {"salary" : x["salary"] + 25)

> 3. Related to 2. How can loosely-sequential Id be generated using a table?

Loosely-sequential IDs would have to be generated client side for now.

> 4. Will some transaction support be added? Don't need full ACID, just grouping updates (intra-table and/or inter-tables) in one shot would be nice. Should be feasible with MVCC already in place.

Eventually. No concrete timeline for this right now though.

> 5. Do all the clients hit a central server to initiate queries which then farms out the requests to different shards? Or the client library knows how to get to different shards directly? First case has a single-point-of-failure, and bottleneck in scaling.

A client makes a connection to a specific server and all queries go through that server. However every server can file this role so connections can be distributed and there's no single point of failure. An even better option is to run a proxy on the same machine as the client. For more info run:

rethinkdb --help proxy

> 6. Do you support automatically re-balancing of shard data (data migration) when new shards are added or old ones retired?

Right now sharding is a manual process. You tell the server how many shards you want and it handles figuring out how to evenly split the data, picking machines to host them and getting the data where it needs to go. What it doesn't do is readjust the split points when the data distribution changes. This will be a feature in RethinkDB 1.3.

> 7. How are authentication and authorization done? Or any clients can come in?

RethinkDB has no authentication built in to it. You should not allow people you don't trust to have access to it.

8. Internal detail. For out-of-date distributed query on the slave replicas, is there a cost-based (or load-based) decision process to pick the most idle replica to do the sub-query?

Right now we just select randomly. This is slated as a potential upgrade for 1.3. Especially if it proves to be a problem for people. Thus far it hasn't been for us in profiling runs but this is the type of problem that's more likely to show up in real world workloads.

9. Internal detail. Do you use Bloom Filter to optimize distributed joins?

We do not currently use bloom filters to optimize this.

coffeemug37• 14 years ago

1. Yes. It's a matter of doing this right, which will take some time.

2. Yes. There is no special command, you just combine update and branch (http://www.rethinkdb.com/api/#py:control_structures-branch) Here's an example in Python:

  r.table('foo').get(5).update({ 'bar': r.branch(r['baz'] == 0, 1, 2)})

This will set attribute bar to 1 if baz is 0, or to two 2 otherwise. Everything is atomic on that document.

3. Currently the server doesn't support a sequential (or even loosely sequential) id autogeneration. You'd have to do that on the clients, but using a timestamp for example.

4. I don't know yet how to do this really efficiently. It's relatively easy to do on a single shard, but cross-shard boundaries make this really hard.

5. Any client can connect to any server. The server will then parse and route the query. There is no central server, everything is peer-to-peer. The client library doesn't know about multiple servers now, so responsibility is on the user to hit a random server. Alternatively you can run "rethinkdb proxy" on localhost and connect the client to that. The proxy will then route queries to proper nodes in the cluster.

6. In the web UI, if you click on the table and reshard, everything will be rebalanced. You don't even have to add or remove shards, it'll just rebalance data for the number of shards you have. The UI has a bar graph with shard distribution, so you can see how balanced things are.

7. Currently there is no authentication support - we expect users to use proper firewall/ssh tunneling precautions.

8. Yes, that's how queries get routed. Currently this isn't very smart, but it will get much better over time. If something breaks for you performance-wise, just reach out and we'll fix it.

9. No, not yet. If you run eq_join on a small subset of the data (99% of OLTP workloads) it will be very fast. Other joins work ok, but there's A LOT of room for optimization.

Phew!

ww5207• 14 years ago

Thanks for your and jdoliner's detail answers! Hope I didn't ask too many questions. :) I'll respond to both here.

For 2 and 3, I think I didn't make it clear. Let me clarify. A common db problem with multiple clients is dealing with concurrent update on the same piece of data. E.g both client1 and client2 read D as D=15 at the same time. Client1 adds 1 to D as 16 and saves it. Then client2 adds 1 to D as 16 and save it as 16, which is wrong. It should be 17.

Conditional update is one feature db usually provides to let clients deal with this problem, i.e. the update would only go through if certain condition is met otherwise abort. Update D=16 if D==15. Client1 would succeed while client2 would fail, where it can retry the whole read-increment-update cycle again with the new read value.

The litmus test to see if a db system can handle this problem is to try to implement a sequential Id generation feature run by multiple clients at the same time.

For 8, if the query is parsed into a query execution plan, you can ship the plan to all equivalent replicas to ask them to estimate the execution cost based on their current load. After they reply, pick the lowest cost one and send the execute command. Even a simple approach of asking for machine load of all replicas and picking the lowest one could have adaptive utilization of all the servers.

For 9, Bloomer Filter is a relative simple technique that can dramatically reduce the amount of data to ship across peers to do join. You basically filter out the vast majority of the non-matching data before shipping.

It's a good start. Good luck going forward!

Meai• 14 years ago

Could you guys offer a C driver? (preferably not c++)

lambda• 14 years ago

Taking a look at their API, a C++ driver would actually fit much better with their style, which relies on operator overloading to make queries involving arithmetic easy to write in the host language.

A C driver would definitely be possible, but would be a little bit clunky.

nnash• 14 years ago

  joe@alchemist~$ rethinkdb
  joe@clockwerk~$ rethinkdb -j alchemist:29015

Dota player?

coffeemug37• 14 years ago

Oh yes!

StavrosK3• 14 years ago

Wanna get a game going sometime?

I play DOTA 2, by the way, hopefully you do too.

tehansen• 14 years ago

am I the only one that noticed the lambda at the bottom right of the webpage? :)

willytobler• 14 years ago

This looks like very interesting. And a nice interface to deal with. But ... I do not find anything about authentication. Whoever wants can fiddle with this 8080 Port.

Did you rethinkAuth or am I just too stupid to RTFM?

coffeemug37• 14 years ago

No authentication support yet unfortunately - you'd have to do an ssh tunnel for the web admin. This is one of the things on the todo list.

hcarvalhoalves• 14 years ago

It seems very interesting, and having to deal with ORMs daily makes me appreciate the clean API.

I feel being based on JSON is a big con though. While it's popular, it was never meant to be a rich serialization format, just simple. How to implement more complex fields like dates, and query efficiently on RethinkDB?

shrughes3• 14 years ago

A benefit to JSON is that people know immediately what it is. It certainly does have its deficiencies (binary data being one that outweighs dates in terms of immediate importance). Being limited to strict JSON is not a permanent decision (I'm saying this as a member of the RethinkDB engineering team), it's a conservative one in terms of API design, and in terms of limiting scope for the first release.

juzfoo• 14 years ago

Congrats guys! Looks great from the specs. Any thoughts on architecting for multitenancy?

goranp• 14 years ago

I am a bit late to the game but how does this compare to HyperDex in terms of scalability and sharding?

RethinkDB: An open-source distributed database built with love over three years

Related Stories