The thrill of a new technology: CouchDB

http://tempe.st/2009/03/the-thrill-of-a-new-technology-couchdb/

by intinig • 17 years ago

27 47 17 years ago

mapteo• 17 years ago

I work only with relational db...but this is a great tech!

Tichy6• 17 years ago

Hm, flat files will rule the world? No more messy SQL, just iterate over the file's entries with a for loop - even Java programmers can understand that.

Sorry, but I am not convinced yet.

intinig3• 17 years ago

Tichy, give it a spin. The ease of execution (and the sheer speed, even over giant amounts of data) is impressive.

dagheti6• 17 years ago

What do you mean by impressive "ease of execution"?

How terse the code is? How easy the code is to read? How likely the code is to be correct? How flexible the data model is to new requirements? How simply you can reason about what the code does?

sounddust• 17 years ago

I simply don't believe that this model scales "over giant amounts of data", at least in any sort of real-world usage scenario. Can you back up this claim with benchmarks?

Tichy6• 17 years ago

I can imagine some benefits, but as for speed of execution I am really doubtful. Your example suggests a "select all" followed by some analysis done in code. I can't imagine this scales well.

Tichy6• 17 years ago

Reading CouchDB documentation now and can't stop reading. It sounds like a lot of fun. Haven't understood yet how to handle the performance problems, but would like to use it just for the fun of it.

jshen5• 17 years ago

You may not be convinced yet, nor am I, but your strawman argument makes you look ... well a little silly.

Tichy6• 17 years ago

That is the way I understood it. What do you mean by "it creates an index for each view"? I just would like to avoid reinventing the wheel, and relational databases abstract away some of the problems of dealing with large data sets.

Edit: OK, read the CouchDB introduction now. Did not know that the "views" are server side. I have a past as a Java developer, and I have actually come across the "select all, then compute in Java" pattern several times in the real world.

noss• 17 years ago

A document based DB is not like a flat file. They do index for fast retrieval.

It is just that the indexes are added based on what queries are made. You can be explicit about what indexes you want maintaned, but it is often good-enough to just perform a query and have the db figure out how to index to make that query faster.

It's pretty much dynamic-typing vs static-typing for databases (plus the document-entries instead of row-column-entries).

jrockway10• 17 years ago

No, graph databases will rule the world. Couch only supports trees, which is annoying if you want to have relationships between "documents".

Also, Couch's implementation of map/reduce arbitrarily limits the kinds of queries you can run.

If you actually want to ditch your relational database, take a look at things like KiokuDB, Elephant, AllegroCache, and so on. They may not have exciting web 2.0 screencasts, but the technology is much better. (I am biased towards KiokuDB, since I helped write it, but it has been very easy to use KiokuDB instead of a relational database; there has been significantly less code in our apps, tests have been much easier to write, and I don't think we've lost any runtime speed either. I really need to write a long blog post about this, but I haven't had time.)

It is good that the world is gradually working their way up to object/graph databases, though. Last week it was "OMG KEY VALUE STORES SOLVE EVERY PROBLEM", this week it is "DOCUMENT DATABASES WILL RULE THE WORLD", so hopefully the blogosphere is only a few weeks away from enlightenment ;)

dagheti6• 17 years ago

Hierarchies and graphs are both fundamentally navigational data models, which force a physical-logical coupling of concerns, hampering flexibility significantly.

The network model recognizes the flexibility issue with hierarchies, however I'm not sure how it can guarantee the same scalability benefits. Due to this it doesn't have the flexibility of the relational model, nor the performance characteristics of the hierarchical model.

This is probably why it isn't ruling the world.

jrockway10• 17 years ago

This is probably why it isn't ruling the world.

Technical merit and popularity are only occasionally related in the computer world, so I won't comment on this.

I will elaborate further on the use case of graphs versus a relational database. Basically, I see relational databases as especially useful for cases where you have a big pile of data, and have no idea what it really means. You do queries to learn what you have. ("Aha, in March, people from Illinois buy more of product 23894735 than people from California.") The relational model is good for this, since it doesn't build any preconceived notions into your data.

However, most applications don't need anything like this. They have a well-defined data-model, and rarely run "queries" (except to work around the fact that that's the only interface to their data). In this case, the relational model is basically being used as a dumb key/value store supplemented with joins. This results in a ton of code in the application to translate in-memory structures to something that can be stored in the database.

If you use a graph database, you can just store your in-memory structures directly, and get them back later. (Most good object database give you other features, like the ability to index your data so you can still run searches efficiently. Kioku and Elephant do this, anyway.)

So really, you should use the right tool for the job. Want to persist and search in-memory structures? Use an object database. Have a big pile of data you need to make sense of? Use a relational database.

You are allowed to use more than one, they are tools, not religions.

(FWIW, I see relational databases as filling a very specialized role, and I see object databases as the general thing you should use when you want to persist some state. The rest of the world seems to have gotten this backwards, which is somewhat depressing. I think it's because people think databases are magical, as a result of not understanding how they actually work.)

gcv• 17 years ago

I've looked at AllegroCache before, and really liked it. Can you briefly discuss how a graph database of type you describe (AllegroCache or KiokuDB) differs from the object-oriented databases which have not exactly taken the world by storm?

Edited: I also just Googled around a bit and found something called Neo4j. Do you know anything about it?

jrockway10• 17 years ago

Can you briefly discuss how a graph database of type you describe (AllegroCache or KiokuDB) differs from the object-oriented databases which have not exactly taken the world by storm?

Basically, object databases and graph databases are conceptually the same. I have only used graph databases to store object graphs. Instances are nodes, and "has a" relationships are edges.

I don't know why this hasn't caught on, as it's made my life significantly easier (by making my software simpler and easier to test). My guess is that there hasn't been a good implementation in a popular language. Perl and Lisp are widespread, but not nobody is going to start using either because there is a nice object database for them.

(Neo4j looks nice, and I don't know why its not more popular. Perhaps it needs a good object-storage frontend, and this is hard to write since Java's MOP is pretty limited. Perl and Lisp both have excellent MOPs, Moose and CLOS, so this makes it easy to write good object databases. With Java, you will have to build your own MOP, which is hard. Same goes for C++ and C#, which are the other "popular" languages.)

moe• 17 years ago

Well, just to state the obvious, no particular database will "rule the world".

They are different tools for different tasks.

RDMBS will not go away in data warehousing for a while. Key/Value stores are useful for many webapps where they match the access pattern. Graph DBs have yet their own area of application.

There is not "one tool to rule them all". Albeit it would be an interesting project to wrap up all those engines under a common API (SQL?) and have the server choose the optimal one either on user demand or even by magically analyzing the workload.

I'm saying that because the real uglyness that many of us are facing is that we need not one but several of the aforementioned tools for our particular app. For parts of the data we like the guarantees and integrity of RDMBS, for other parts we need the scalability of a key/value store.

jrockway10• 17 years ago

Well, just to state the obvious, no particular database will "rule the world".

I don't think this either, I was just parodying the overly-editorialized title.

hendler• 17 years ago

I agree, in principle that the most dominant, important technology will be graph databases. The reason I think so is because of experience working in technologies related to the semantic web.

However, there are reasons one might a Distributed Document Store to implement a graph store - it's the index that matters. MySQL is not the same thing as InnoDB/Sphynx. Graphdb product X could be built on mapreduce +CouchDB+Lucene(+somegraphindex) .

The only thing I'm sure of, is that if anything stands out above the rest, Oracle and IBM (and maybe MS) will try to buy it.

mattlanger• 17 years ago

With high expectations I clicked a heading entitled "Document based DBs will rule the world" but found someone lifecasting "hello world" on CouchDB.

dasil0034• 17 years ago

Total clickbait title. To be fair, the article doesn't make this claim anywhere. Flagged with extreme prejudice.

mattmcknight• 17 years ago

I have run into serious issues with the document based approach. Imagine a domain where you have several different types of documents. You still end up doing joins. In the example the author gives, doing a query to get you all of the unique tags can be slow. Or, if they are dates, trying to get a max(date) can be tricky. The biggest problem though, is document size. For a complex domain, there are big tradeoffs between keeping the entire graph in the document, which is slow to read, or creating a variety of document types, which brings you back to joining.

A hybrid approach can give you a little of both. For most of the kind of applications I build, I use relational databases, but I dump a denormalized view of the data into Lucene for searching and such and use a caching layer for most reads. This way I get some document type performance on many operations, but still have the normalization and simple transactions of a relational model.

leej• 17 years ago

We need basic guides for the graph/document database engines like this code in sql is translated as that code.

The thrill of a new technology: CouchDB

Related Stories