RethinkDB 1.3 is out, now available on OS X

http://www.rethinkdb.com/blog/rethinkdb-13-release/

by coffeemug • 14 years ago

178 58 14 years ago

This is an example of an excellent website for a developer tool. Even the blogpost contains a hello world example in the header. The front page contains everything you need to know in just a couple of sections.

eberfreitas• 14 years ago

Is anyone over here using it? What is your impressions?

StavrosK2• 14 years ago

I used it for a bit, my first impression was very good, if buggy. The team is very responsive and extremely accommodating, and I'm hoping they make a very solid database. The featureset looks great, and the admin interface is fantastic.

I would definitely recommend giving it a try.

m0th872• 14 years ago

Could you expand on the buginess? Was it aesthetic issues, or did you ever face problems like data corruption?

bjourne• 14 years ago

It's great to see all these new ideas in the db arena. But one thing I don't understand at all is why they diss the idea of having data integrity. Am I the only developer who actually likes working with relational data and do not think of a schema as a straight-jacket? Am I the only one getting burnt by MySQL and SQLite playing loose with types and silently coercing data almost randomly?

Check constraints are also great and has saved my ass so many times. It's so much easier to debug a problem when the crash occurs when you are inserting the data than in a wrong calculation much further down the line. Postgres for example, allows you to create regular expression constraints which you could use to ensure that a text column only contains a valid variable name. Exclusion constraints are even better, if you have a table with allocations (date ranges, integer ranges, etc) you can check so that none of the allocations overlap. Plus about a million other ways you can constrain your data.

That's the killer features for me. They have helped me debug problems so many times that I consider them almost essential to writing good web applications. So why does almost all new database projects seem to neglect them?

m0th872• 14 years ago

Schemas have saved my ass a million times too, but schema migrations are a pain.

I'd love a database that's loose with types by default, but allows you to add declarative specifications for types. Documents are checked at insert/update time, but otherwise the specifications play no role in how the data is stored.

That way you get schemas, but only when you need them. And migrations don't require heavy collection-wide locking; instead, you just change the constraints you enforce and lazily update old documents.

Postgres does to an extent with CHECK constraints, but you still have to do an ALTER TABLE if you change the underlying type of a column.

petergeoghegan• 14 years ago

Yes, you still have to do an ALTER TABLE to change the type of a column within Postgres, because, well, that's just how it's done. This does not necessarily mean that the table will be rewritten, which I assume is what you're concerned about. See this item from the Postgres 9.2 release notes:

Reduce need to rebuild tables and indexes for certain ALTER TABLE ... ALTER COLUMN TYPE operations (Noah Misch)

Increasing the length limit for a varchar or varbit column, or removing the limit altogether, no longer requires a table rewrite. Similarly, increasing the allowable precision of a numeric column, or changing a column from constrained numeric to unconstrained numeric, no longer requires a table rewrite. Table rewrites are also avoided in similar cases involving the interval, timestamp, and timestamptz types.

lwat• 14 years ago

> ...you still have to do an ALTER TABLE if you change the underlying type of a column.

I don't see why this is a problem? How often do you update really big (hundreds of millions of rows) columns' data types in practice? Most of the time you'd change type early in development when you don't have that much data in the table yet which makes it pretty much instantaneous. If you really need to update the data type for a supersized table you could always make a new column instead and migrate lazily to that in the background. But all of this happens so rarely that in my view it's a silly excuse to dump all the great benefits you get from having an enforced database schema.

lucian1900• 14 years ago

Migrations are the main reason I like schemas! Trivial migrations (like adding columns) are trivially managed by a good library (SQLAlchemy + Alembic or Django + South) and more complex migrations are possible.

Without a explicit schema, it's very hard and sometimes impossible to change the shape (schema) of your data.

ibejoeb• 14 years ago

I'm with you. I guess it depends on what you work on. When I'm counting thousandths of a penny, I use something that provides integrity and consistency. I've worked on things that don't require precision or timely consistency, and sometimes it's fun to just jam things into a non-rel system; I like couch.

If I were to work on a general-purpose technology product, it'd be a dbms that provides these, first class: referential integrity, arbitrary precision arithmetic, interval datatypes, dependency graphs/recursive DS, and out-of-the-box reporting (formatting, grouping, serialization, etc.)

jdoliner4• 14 years ago

RethinkDB is filled with developers who have conflicted thoughts on this. The difference between structured and unstructured database systems has always seemed very analogous static vs. dynamic type systems to me. Most of us at RethinkDB like our type systems static, Haskell being one of the favored languages, and we've had many conversation about why our database doesn't reflect these same principles.

I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving. People seem to really like developing in dynamic languages using structureless databases. I think that a lot of times this gives people more ways to shoot themselves in the foot (and I sense you agree). But even if this type of development was purely a band of fools shooting off each other's feet we still might have to respond to it.

That being said I don't think it is quite so foolish an endeavor. The thing about structured databases that sucks, and a place where the analogy to dynamic languages breaks down, is that your data integrity becomes entrenched. Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure. This can be a very cumbersome process made more cumbersome by the fact that a lot of structured databases make optimizations based on things like knowing the size of each row (very hard to change on the fly). So while RethinkDB doesn't dislike data integrity we do really dislike data stasis and that's what we wanted to avoid. We do have plans for the future of having a way to enforce arbitrary constraints over you database, we just aren't willing to do them in such a way that they're hard to change. Unfortunately doing them in that way is time consuming and we haven't gotten to it yet.

jdoliner @ rethinkdb

saurik5• 14 years ago

> I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving.

Is this really true? I mean, it is clear the majority of stuff you read about is from people who don't like types, but honestly as time has marched on development has been "democratized" to a large extent... yet, if I think about "people with both a lot of experience and a lot money who are willing to both pay for things to use and people to maintain those things" (what I would call "the market" for a database solution) I see lots and lots of rigid types.

I mostly see "schemaless" from highly-popular (to the masses) database system where I don't really see much money being made or even moved (on database software). It could be, however, that I'm just totally misunderstanding how well some of these companies are doing. However, when I see a company evaluating things to plunk money down on, they seem to be 1) clustering support for PostgreSQL, 2) hosted management of MySQL, and 3) system administrators and licenses for Oracle.

In contrast, people providing NoSQL "schemaless" systems seem mosty to be living in a world where the developers expect everything to be free; they sell support contracts, but I never hear of anyone actually buying them... do a lot of companies do this? It also always sounds like a magical accomplishment when you hear one being used a large company with a lot of money (such as the developers at UbiSoft giving a talk on how they are using CouchBase at re:Invent)... I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years.

Note: to be clear, I do hear of companies with "NoSQL" systems who have a lot of money, but they are all using Casaandra, HBase, or SOLR. SOLR let's you make define dynamic columns, but everything is still rigidly typed. HBase and Cassandra both have schemas with fixed columns; sure, Cassandra allowed you to largely ignore the schema, but if you've seen CQL3 they are actually nigh-into turning into an RDBMS: it is hilarious, and I think awesome. Even then, it isn't clear to me how much these companies pay money for their databases.

> Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure.

This is not actually true, as the structure can be metadata stored describing the data. When I add or delete fields in PostgreSQL, the operation is instantaneous, as is renaming existing fields. AFAIK, only MySQL sucks at this. The only time I need to do a table rewrite is if I change the datatype of an existing field, and I am pretty certain that you can also handle the majority of those cases without a rewrite. Even with the rewrite, you shouldn't have to block incoming queries.

It isn't like not having a schema fixes this, as there is a schema after all... my code certainly has an assumption about how the data is stoted. It just means I either a) can't change the schema at all, b) can change the schema, but have to do so by doing the table rewrite manually, or c) can manually implement the metadata behavior (but then why not just add this to the database server). Given how the patterns one uses to solve these problems don't really rely on the data being stored, it simply makes sense to standardize them in the database server.

astral303• 14 years ago

I have to agree with the last sentiment.

In my experience, one never ends up having all of one's update/migration (to a new version) changes constrained only to schema alterations. This is because the static database structure can only capture so much metadata. One inevitably ends up facing putting migration logic into the application itself, or logic that says "if old data, treat differently." So now one ends up with a mixed data migration situation: both database-level and application-level.

Thus, it's a quite valid, pragmatic tradeoff to make: handle all data migration/schema changes in the application, let the store be completely dynamic and have it all in one level.

alexpopescu3• 14 years ago

This is an as wide discussion as static vs dynamic typing in programming languages. Both approaches come with their pros and cons and their impact varies from project to project. (re-reading this part feels pretty plain unfortunately)

It's also interesting to mention that there are multiple sub-topics related:

1. schema vs no-schema

2. type checking

3. referential integrity

4. data integrity

Now in reverse order:

4. every database should guarantee data integrity (i.e. you get back what you put in).

3. referential integrity is an interesting and challenging problem in the space of distributed systems that has impact on the trade-offs that the system has to make

2. type checking: while I'm not aware of any non-relational database supporting this, it is a feature that could be added at some point by basically tagging specific fields/attributes with typing information. For the time being I think people that use non-relational databases and requiring type checking are implementing it in the database access layer. But there's no good reason why it couldn't or shouldn't be supported by the database itself.

1. schema vs no-schema: IMO these two represent the two ends of a spectrum where what I think would actually make sense is support for optional schema definitions--in a way this could be similar to the way Protocol Buffers/Thrift/Avro define their protocol.

alex @ rethinkdb

jdoliner4• 14 years ago

Tagging fields in json as different types is actually something we've talked about (as you may well know given they you're my coworker). But there's another aspect of it that I think is worth mentioning. Right now RethinkDB has a very limited number of types that it can return, it's limited to JSON types so basically just object, array, strings and numbers. There are several other types that could be encode as these but you're going to wind up needing to do conversions when you insert and extract this values from the database. For example you might wind encoding and RGB value as an array of 3 numbers and your code will wind up looking like:

    table.insert({"color" : colorToArray(c)}).run()
    arrayToColor(table.get(...).run()["color"])

This obviously is cumbersome. This is the type of problem that an ORM seeks to solve. However ORMs have the big flaw of being limited to a single language and needing to be kept in sync with the database. If you did this in the database layer by having the driver convert the color object to a tagged array then it would know to return it as a color object rather than an array. Other languages could do it too using the same database since they would see the tag and could convert it to a color type in their language.

donpdonp• 14 years ago

Schemaless databases move the integrity checks up into the middleware, which I dare say is a more flexable place to have it. Most ORM libraries have integrity checks so the db checks end up being redundant and more difficult to change. I too have been saved on occasion by such 'redundant' checks, but over all I find its easier to manage the structure of data when its kept in the middleware. Keep the storage layer as simple as possible with a focus on consistency, availability, and partition tolerance (CAP).

theshadow• 14 years ago

While ORM level validations are nice to have, they are in no way substitutes for database level constraints. Just relying on ORM level validations is like saying "hey I'm doing error checking in my javascript code and therefore checking for errors on the server side is redundant and unnecessary". It's not, DB level "redundant checking" is worth it. All it takes is for one guy to "just quickly try something" in the db by trying to manipulate it directly without the ORM layer to have your db end up in an inconsistent state. ORMs have their own conventions on how they handle validations which maynot be completely obvious to a new developer on your team who is not familiar with it, whereas everyone understands how db level constraints operate. I'd rather put a few lines of extra code at the db level to make sure the data is safe than risk having the db end up in an inconsistent state, you are not only doing this for the customer but also for yourself. You don't want to be the developer who has to go in and figure out why certain queries are not returning the correct results after your db is in an inconsistent state.

scott_w• 14 years ago

People rejecting SQL/schemas tend to fall into two camps:

1) People who don't know what they're doing. People in this camp tend to be inexperienced, and reject structured data because "it's easier to just deal with it in code". People in this camp, tend not to realise that they're still maintaining a schema, but making it harder because they have to manually code all the constraints.

2) People who are trying to solve a specific problem that is easier when you weaken constraints on the data. You could be dealing with naturally unstructured data such as web crawling. People in this camp have actually thought about what they want to build, and are making a conscious choice.

I think that people fall into the first camp more often than not. Using myself as an example, the team I was leading decided that CouchDB would be a good fit for a use-case involving "unclean" data. It turned out that the data could be cleaned manually quite trivially, and the overheads of utilising CouchDB massively outweighed the ever-shrinking benefit.

Note that I'm categorising people who choose to use NoSQL in production situations, not people experimenting or people not using NoSQL at all.

philjohn• 14 years ago

There's also a third camp, for those of us dealing with structured data that doesn't fit into a "tables and rows" idiom.

tworats• 14 years ago

I'm looking for a middle ground - SQL/structured data extensible with non-structured data. The ability to extend sql tables with unspecified json-like data. The love child of mysql/postgres with Mongodb.

Any such thing exist?

te_chris• 14 years ago

Postgres with HBase/JSON storage potentially fits what you're describing.

ksev2• 14 years ago

Does this include the new wire format? That was talked about here: http://www.rethinkdb.com/blog/rethinkdb-wire-protocol-call-f...

I'm itching to write a toy client in clojure.

coffeemug9• 14 years ago

No, this doesn't include the wire protocol updates yet. That's slated for 1.4 -- sorry!

ksev2• 14 years ago

Alright, gives me more time to nail down a nice query DSL :)

Nice to see OSX support, simplifies development quite a bit for us mac users.

SnowLprd2• 14 years ago

I commend the RethinkDB folks for supporting Mac OS X explicitly. However, the .pkg installer, while handy for folks that prefer that installation method, installs to /usr/bin/ and /usr/share/ -- areas that are traditionally reserved for system tools provided by Apple. Sure, the DMG also contains an uninstall-rethinkdb.sh script, but I'm still left quite unwilling to kick the RethinkDB tires on my primary Mac workstation.

A better method, in my opinion, would be for the RethinkDB devs to contribute a Homebrew formula. That way everything would be installed into /usr/local/ by default, and updates would be a quick "brew upgrade rethinkdb" away. Am I the only one who would prefer that?

UPDATE: It seems they had originally planned on a Homebrew formula but thought the .pkg format was better. Clearly, I disagree with that. Want to offer both? Awesome. But if you're going to pick one or the other, I think Homebrew is the vastly superior method. More here: https://github.com/rethinkdb/rethinkdb/issues/5

coffeemug9• 14 years ago

The reason we did that is because /usr/local/bin isn't on the default path on many OS X systems, and we didn't want the installer to modify the user's PATH.

The thing about Homebrew is that it's great once you have it, but the user experience of actually installing it can be quite a pain, and we didn't want users who don't have it to have to go through the trouble. That being said, I still think it would be great to contribute a Homebrew formula.

SnowLprd2• 14 years ago

Regarding the first point, which OS X systems are you referring to? The default PATH on Mountain Lion definitely contains /usr/local/bin, as you can see here:

/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

With respect, I don't share the view that installing Homebrew is a pain. Even the most inexperienced would-be developer can follow the very simple guide I (very coincidentally) posted yesterday: http://hackercodex.com/guide/mac-osx-mountain-lion-10.8-conf...

But it seems you guys are amenable to offering both the .pkg installer and a Homebrew formula, which would be great. :^)

mscarborough2• 14 years ago

Are you folks open to that?

I'd be happy to do a pull request that does paths for homebrew and ports, so that either OS X party can be OK with it.

ra• 14 years ago

I'm currently running rethinkdb via vagrant, and it's actually pretty great.

Personally I wouldn't install the pkg for the reasons stated by SnowLprd, but I definitely would switch to a homebrewed formula if it was available.

Great work on getting OSX support out so quickly!

patrickg• 14 years ago

When installing via .pkg, you can place a file in /etc/paths.d that adds to the path. For example, when you install in /usr/local/rethink/, you just add a file (e.g. /etc/paths.d/rethink) with one line: /usr/local/rethink/bin. Thats it, very easy.

emp_• 14 years ago

I for one am extremely happy for, on my new mac mini, being able to run sublime text 2, nodejs and npm (pkg), postgres (app) and the heroku toolbelt (pkg) without installing XCode.

themgt• 14 years ago

http://rethinkdb.a.pogoapp.com/ - our demo has been updated to 1.3.1

DeepDuh• 14 years ago

I can't find information on how (or whether) you handle users. Is there some built-in per-document read/write lock based on user access for example? This is the most important feature I miss in CouchDB.

coffeemug9• 14 years ago

Do you mean permissions (i.e. granting access to tables based on authentication)? If so, Rethink doesn't have that yet, but it's definitely on the list.

gummydude• 14 years ago

can't wait for final golang driver, kudos for listening.

coffeemug9• 14 years ago

You might want to check out community one for now -- https://github.com/christopherhesse/rethinkgo (perhaps even contribute -- hint hint)

cshesse• 14 years ago

This driver attempts to match the official drivers as closely as is reasonable. It has almost all of the functionality of the official drivers now, but won't have a stable-ish API or actual tests until the end of December.

Let me know if you have any feedback, this is the first Go language library that I have built.

RethinkDB 1.3 is out, now available on OS X

Related Stories