No to SQL? Anti-database movement gains steam

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9135086

by johns • 17 years ago

84 100 17 years ago

mblakele2• 17 years ago

This seems oversimplified to me. When I think "database" I think of ACID properties. You can have SQL without ACID, and you can have ACID properties without SQL.

evgen2• 17 years ago

The problem with this simplification os the problem-space is that ACID properties cannot be maintained (see CAP) when you start moving to a distributed model. The "no SQL" group is just using it as a catchy hook upon which to hang their hat, it is not SQL per se but the traditional RDBMS that is reaching its limits, but it was hard to make a memorable name that included "RDBMS" :)

gaius9• 17 years ago

ACID properties cannot be maintained (see CAP) when you start moving to a distributed model.

You need to pop back to the 1960s and tell IBM that before they wrote TPF.

mblakele2• 17 years ago

I think SQL has its limits too, for example in hierarchical use cases. While it is possible to write extremely complicated CONNECT BY clauses in SQL, I would prefer XPath and XQuery for that job.

neilc• 17 years ago

The problem with this simplification os the problem-space is that ACID properties cannot be maintained (see CAP) when you start moving to a distributed model.

That assumes you need the system to remain available in the presence of partitions. How often is that really true?

petern• 17 years ago

Fully agree. look at graph databases like Neo4j, which has ACID semantics but no SQL interface. Then, IMHO SQL is not only geared towards a special RDBMS type of underlying database but even against a special problem domain that is not always applicable, leading to query languages like SparQL.

wvenable4• 17 years ago

Performance is always a trade-off; wou rarely get something for nothing. These object stores are great when all your doing it storing objects, but that's a pretty simplistic use-case. If you need to do more with your data than simply read it back in object-sized chunks, you're going to find that a relational database probably provides better performance. For web sites especially, retrieving and constructing objects is a fairly expensive operation if you're just generating list pages.

I wonder if we just have a generation of programmers who can't think in SQL. They're used to using ORMs (which are useful abstractions) but can't work at a lower level. Sending a query over the line to get exactly the results you want, and no more, and have it optimized and run entirely on server is pretty damn efficient.

johnnybgoode• 17 years ago

> I wonder if we just have a generation of programmers who can't think in SQL. They're used to using ORMs (which are useful abstractions) but can't work at a lower level.

There is probably some truth to this, but another reason is that they want to avoid certain scaling hassles or costs.

Edit: I'm not saying this is necessarily a good thing.

trezor5• 17 years ago

I fail to see how using a proven solution like relational databases involves hassles with scaling.

gaius9• 17 years ago

I wonder if we just have a generation of programmers who can't think in SQL.

I think that's probably true. SQL requires thinking declaratively, not imperatively, and in sets, not objects. Kids these days don't understand the difference between a table definition and a class and the difference between a row and an object. The declarative/functional style isn't really taught anymore; everyone just wants to learn Java and get a job.

If anything, ORMs are less "scalable" than SQL because under the hood the system is based on SQL, and in an effort to be completely generic, the ORM generates really bad SQL.

jrockway4• 17 years ago

in an effort to be completely generic, the ORM generates really bad SQL.

Maybe your hand-rolled ORM does, but most off-the-shelf ORMs generate excellent SQL.

rmaccloy3• 17 years ago

Indeed, "kids" (by which I assume you mean "assembly-line CS grads") these days are pretty much using Hibernate, ActiveRecord or (I guess?) LINQ. This is a good thing: The O/R mismatch exists. For good programmers ORMs are more productive in most cases (and you can write raw SQL for the exceptions); bad programmers have never gotten the relational model (and there's plenty of legacy code to prove it.)

However, the discussion about "kids" is a total red herring, because I'm pretty sure all of the people involved in talking up Tokyo Cabinet, memcachedb, Couch etc on one end and Cassandra, Hypertable or HTable on the other are fully aware of the difference between a class and a table. The truly clueless don't even know the discussion is happening.

The object/document database revival circa 2009 is about three things:

A. People are already using object-based access for almost everything, so if they're not using relational features it's trivial to drop in an object store and get better performance/less overhead.

B. On the low-end, sometimes a relational database is too much overhead -- if not in performance, in administration. (SQLite is cheap, sure, but sometimes you want just a disk-backed hash table.)

C. On the high-volume-data-analytics-end, RDBMSs don't perform well enough- unless you shell out serious cash for Oracle or Greenplum, Aster Data, Teradata, etc, and even then they still can't handle the volumes MapReduce and Bigtable-alikes can.

ubernostrum2• 17 years ago

"I wonder if we just have a generation of programmers who can't think in SQL."

Maybe we have. I'm not so sure.

What I am sure of is that we've had multiple generations of developers who've seen that real-world data almost never fit into a uniform, enforceable schema, and so are happy to have solutions which accept that reality instead of demanding that developers and DBAs try to shoehorn the data into uniformity anyway.

fhars• 17 years ago

"I wonder if we just have a generation of programmers who can't think in SQL."

And they don't even teach set theory in elementary school anymore like they did in the early seventies, which may be among the root causes of the malaise. Understanding things like relational databases and conditional probabilities is so much easier if you have been taught a solid foundation from early on.

Nelson69• 17 years ago

> I wonder if we just have a generation of programmers who can't think in SQL. They're used to using ORMs (which are useful abstractions) but can't work at a lower level. Sending a query over the line to get exactly the results you want, and no more, and have it optimized and run entirely on server is pretty damn efficient.

Yeah, it's really obvious if you ever work with a really really senior DBA. I don't think it's the queries though, it's the modeling. I think a lot of software teams model their database sort of like they model data structures and then use various ORM tools to be the glue. DBAs tend to model the data in ways to make it most acceptable to the database and minimize the loss of any information, it's almost always more complex. The database as become an object persistence engine in a lot of cases that has some relational properties that may or may not be used. Instead of writing and managing files in the filesystem, you shove stuff in to a database.

Just for starters, your ORM will model an object for a row of data, how many times do you get the whole row when you're really interested in just a column or two? Does your ORM let you just specify the parts of the row you're interested in or does it hydrate an object and populate all of the columns? (Those extra columns being copied does add up...) ORMs are a religious war, as a software engineer it's a really beautiful idea, in reality I've never seen one that really works well with the database, they're too softwarey.

The other thing that seems to be rampant is the traditional 3 tier application model has kind of collapsed. It's not the case everywhere but I've seen it at more than a couple places where there is a persistence tier and then kind of a combined presentation/business tier. With a more traditional data model the business layer is absolutely critical and shows a lot of value, you might have to glue some more complicated queries together in to objects inside a transaction rather than just hydrating a row from a table. When you use a database as a store for your data structures a business layer just doesn't seem to be as valuable.

If you just want keyed data storage, a relational database does become the wrong tool. I would think, and this might just be parochial thinking, that at some point keyed data storage would essentially reinvent the relational database as the problems grow in complexity.

trezor5• 17 years ago

I don't know about (n)hibernate or other ORMs, but Linq for SQL let's you specify exactly what you want, and when you inspect the SQL generated, it is usually quite efficient, although not 100% optimal for complex queries.

There is also the issue with nested objects (relations) and if they should be prefetched or not (to avoid in effect nested-loop type SQL), but that is perfectly controllable.

My biggest issue with ORM layers is transaction-handling when you are doing some things in the DB, some things with data from this query and some data from this other query. This can quickly promote to what would in DB be a simple transaction to a distributed transaction.

When used correctly ORMs are very nice tools indeed, but to use them efficiently, you have to know how they work and how the underlying DB they interface with works as well.

sant0sk1• 17 years ago

My biggest hold up to investing time learning how to harness "noSQL" data stores is that there are so many young-ish projects and I'm afraid of picking a loser. Couchdb, mongoDB, Redis, TokyoCabinet, etc, etc.

Seems like it's too early to cast lots, and I'll just stick with what I know for the time being. Is that prudent of me or just naive?

evgen2• 17 years ago

The tricky thing is that there is not going to be a single winner here. What we are seeing is data moving out of a creaky (albeit very capable given certain conditions) silo and the managing it is becoming more task-specific. Each of the databases you mention has its own strengths and weaknesses depending on what you want to get out of it, the nature of your data and access patterns, and the environment in which it will be deployed. Right now I would suggest that you just watch the space, become conversant in the various models (especially the weaknesses of each) and take each for a small test drive so that you have a basic understanding of what each particular model is capable of.

mahmud• 17 years ago

The noSQL school includes both key/value databases and bonafide object stores. None of this stuff is new, BerkeleyDB, circa 1986, is as industrial strength as they come. O2, GemStone, Statice, etc. all go back to the 70s and 80s.

Just because something is on github doesn't mean it's ground breaking ;-) The future is full of wonderful things that are yet to reach us from Lisp and Smalltalk research of the past.

silentbicycle2• 17 years ago

The sticking point with BDB is typically its license - if used commercially, you suddenly need a commercial license from Oracle, and the cost is unspecified.

Devilboy2• 17 years ago

I'm not convinced. Of course these projects are useful for some applications but most online services will never grow to the size where you HAVE to dump your SQL server for something more scalable. And in the meanwhile SQL is just sooo convenient and easy to develop on. Why would I want to switch unless I have to?

rjurney3• 17 years ago

You'll still be able to use SQL. You will still use RDBMses for things they are good at. You will stop using SQL for things SQL is not good at.

You will start collecting much more data. You'll be able to throw as much processing power at your problem as you want to, for short periods of time. You'll be able to effortlessly pull in disparate data-sets and work with them without the effort required to import new data and pre-define relationships and indexes in a SQL schema. You will be able to query unstructured data in multiple formats including video, images, audio, html, plaintext, etc. with more structured data. You will live in a culture of data, where large numbers of PCs are brought to bear on all of your data whenever and however you want. You will discover new relationships you did not know were there. You will get increased value from data you didn't know had value because you didn't bother collecting it, because your DBMS wasn't good at working with that kind of data.

All these are new opportunities that exist in addition to traditional RDBMSes, not in place of them.

gnaritas4• 17 years ago

> And in the meanwhile SQL is just sooo convenient and easy to develop on.

No, it isn't, it's a pain in the ass because it doesn't match how applications are programmed. I know SQL pretty damn well, and I know how to tune a database very well, and SQL still sucks.

dasil0034• 17 years ago

It has it's pain points to be sure, but the fact that "it doesn't match how applications are programmed" is because it's modeling data not code. Yes your object store will make the application much easier to write initially. But then what happens when you need reporting or worse, some feature that you never anticipated that requires a different view into the data.

That's not to say SQL is the end-all-be-all, but people are so quick to dismiss it, and I bet there is going to be a lot of pain felt as code based on simpler databases ages.

Devilboy2• 17 years ago

My experience with SQL has been mostly awesome. I like it way more than anything else I do (coding wise).

mdasen• 17 years ago

Cassandra can write 50GB of data in 0.12 milliseconds, more than 2,500 times faster than MySQL.

Really? Cassandra can write 416TB, 666GB per second? If Cassandra can write 50GB in 0.12ms, then it can write over 416TB in a second and 25petabytes per minute. Which clearly isn't true. Of course, that quote would also mean that MySQL can save 166GB per second.

The article is a piece about technology that the writer doesn't understand. They all have trade-offs. Key-value stores eliminate your ability to access data by anything other than its key. So, you have an article with an id of 5. What if you want to look up the articles by author 9? You're just unable. Column based databases aren't magic either. They just change the orientation of the data. Rather than seeing 1,Adam,West;2,Mark,Twain;3,Will,Smith you see 1,2,3;Adam,Mark,Will;West,Twain,Smith. That offers some advantages - such as being able to do metrics on a column easily - but it also means that if you want to get the data for one row, it has to do MORE work than a regular row-based database. To retrieve 1,Adam,West it has to do multiple lookups since the data isn't located next to each other. And frankly, random access of a row is what you're likely to want.

They run on clusters of cheap PC servers. PC clusters can be easily and cheaply expanded without the complexity and cost of "sharding," which involves cutting up databases into multiple tables to run on large clusters or grids.

That's like saying that if you plant a hamburger you'll get a McDonalds. They don't just magically run on clusters of cheap PCs. The fact is that there are tradeoffs. If you put 50% of the data on server1 and 50% on server2 you'll get faster query speeds except that you then have to know where to look for a specific piece of data.

So, sharding: basically, you split your data along logical (or non-logical) lines into separate databases. Like registration at a conference, users A-M are in DB1 and users N-Z are in DB2. So, when someone comes up with a query, you can easily tell which place to send them and then the query is run there. Or you can non-logically split and have a map that tells you where to go. So you query metadb and ask "where's Frank" and it says "Frank is in DB1" and then you query DB1 for Frank.

BigTable isn't so different. It does do some awesome auto-splitting of tables, but when it comes down to it, it's much the same. There's a single META0 tablet that gets queried to find the location of the META1 tablet which knows the location of the actual data. And, in order not to overload the META0 tablet, one needs to cache the location that you get back.

Often times, the problem is that you have web programmers who wouldn't know a B-Tree from a linked list. All of these technologies exist because there are certain things that they're good at. However, what this article purports to know is that these NoSQL databases are just superior. In most ways, they're inferior. A key-value store has its place, but it's severely limiting in a way that you can't just use a key-value store; you need more querying power. Likewise, whether a column-oriented database is right for you depends more on how you want to access the data (by column or by row).

Oh, and really, learn SQL indexing and check whether your queries are doing full table scans. That's the root of a lot of problems. I mean, a good index should do lookups in log(n) time which means that on 10,000 rows a query should take 1/769th the time; on a million rows a query should take 1/50,000th the time; on a billion rows it will take 1/33,333,333th the time of doing a full table scan of a table that size. Yeah, indexes make a huge difference, but they aren't magic either. They merely order the data in a certain way that makes it easy to pluck out certain rows since you don't have to look at every value. For example, dictionaries are alphabetized and so you don't have to look at every word. You just start jumping toward the area of the word you're looking for. Indicies work the same way. Now imagine if the dictionary was unordered. You'd have to look at every single word to see if it was the word you were looking for.

Oh, and just saying "put an index on stuff" isn't enough to know. It isn't too complicated, but there's a Google talk that goes over some of the mistakes one can make with assuming that indexes are doing what you want (http://www.youtube.com/watch?v=u70mkgDnDdU&feature=chann...). It's in the middle somewhere, but it's a good watch for the whole thing.

Really, is this the type of article we've become? I mean, column-oriented datastores and key-value datastores are both valuable technologies, but a NoSQL flame written by someone who doesn't know that Facebook is still MySQL backed and uses Cassandra for only specific things and that they use many different technologies where they're good (like heavy use of memcached as a key-value store to reduce load on MySQL where queries aren't needed). Or perhaps that the article doesn't seem to acknowledge the drawbacks that the authors of these systems would talk about. For example, mongoDB states that they lack ACID integrity and that it's more suited for "High volume, low value data" - that should be read as whether someone has upvoted or not; if that data gets lost it doesn't really matter so much because the person can click the vote button again and all is well or just because it isn't super important data.

Why are we supporting "we hate this technology just because"? Technologies are tools that have their places. It's foolish to write off a good technology just because you want to make fun of it or deride it. Use the right tool for the job. In your web application, you might find that using multiple tools in concert is the way to go.

archon810• 17 years ago

I also noticed this outrageous claim and left a comment in the article. It's shocking that the author can just blurb out random BS and post it without verifying.

drawkbox4• 17 years ago

"They all have trade-offs. Key-value stores eliminate your ability to access data by anything other than its key. So, you have an article with an id of 5. What if you want to look up the articles by author 9? "

For cases like that you don't have one source of data, you do almost like an AFTER INSERT trigger to update other flattened tables of data that you need.

For instance, where you would normally perform a JOIN you would just after save update a JOIN table or a VIEW that is created by incremental updates.

So then you could filter by many other things and really aren't limited too much by space to make those duplications.

It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

But yes the article is off and you are right, most likely systems will approach this with multiple solutions for specific problems.

gaius9• 17 years ago

It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

More like 5Tb these days.

andersnawroth• 17 years ago

The interesting thing about the article is that the NoSQL movement has succeeded in reaching outside the database nerd circles! To get some insight in the actual trade-offs and uses cases for non-RDBMS database systems, this article is a much better read: http://thinkvitamin.com/dev/should-you-go-beyond-relational-...

fendale• 17 years ago

> It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.

I work every day with a > 4TB database and its all on 1 machine (a very big machine) with some very fast SAN. If you have money, Oracle can get very big on the right hardware. I am willing to bet MySQL or Postgres could do very well on that sort of hardware too.

patcito2• 17 years ago

> So, you have an article with an id of 5. What if you want to look up the articles by author 9? You're just unable.

Yes you can, at least in CouchDB, just create a view that emits the author id and you're done.

sho9• 17 years ago

After the server has crawled over every single doc in the DB submitting them to the new view, then yes, you're done : D

rmaccloy3• 17 years ago

This article is really terrible, and it's sad because most of the pieces of software discussed are at least interesting. Several are indispensable to anybody in the business of dealing with PB-scale data. (I'm pretty sure Hadoop is seeing healthy enterprise adoption.)

That said, I think that calling the whole event NoSQL was kind of asking for it; of course people are going to try to squeeze some drama out of that stone. (I would have gone to the meetup anyway, though, if I hadn't already been booked.)

alexgartrell2• 17 years ago

To be fair to the noSQL guys, the technologies that drive the alternatives are very, very young. People are just starting to play with it. There's no reason to think that techniques for performing more traditional sql-style queries on distributed data won't quickly surpass it.

And as far as a lookup occuring in log(n) time, imagine a broadcast "hey everyone, row associated with 1030923". Each node checks its bounds (in parallel) and then, if it exists w/i those bounds, does a log(n) search. That's pretty reasonable.

In short, my prediction is that these things will grow way outside of key-value pairs and into a complete solution that scales very well (and is _incredibly_ redundant).

dasil0034• 17 years ago

Are you kidding? I wouldn't say it's impossible, but "no reason to think that techniques for performing more traditional sql-style queries on distributed data won't quickly surpass it" is quite a stretch. There's plenty of reason to believe that. Mainly because of the number of databases and the sheer volume of thought put into RDBMS', both theoretical and practical. Sure, when it comes to massive scalability there are dozens of promising new technologies that beat SQL databases, but they all make tradeoffs, there's just no way not too when you're talking about something as powerful as SQL.

I imagine there will be some great improvements that come out, but I think we're past the point of any major data store revolutions unless something fundamentally changes like quantum computing becomes a reality and we have to all basically start over.

fauigerzigerk3• 17 years ago

I agree, the article is really sloppy, a typical piece of journalism I would say.

I just want to add that column stores are not in any way anti SQL. They don't change the data model at all. They're just a different implementation of the relational model, more suitable for analytics than row stores. For OLTP apps they're bad. Reconstructing one row isn't necessarily expensive (as column stores don't just store individual columns), but writing one row is expensive.

blasdel• 17 years ago

You build and maintain indexes, just like you do with an SQL db.

jrockway4• 17 years ago

If you click through to the slides, they are actually referring to latency. The slides say that for accesses of > 50GB, Cassandra's latency is 0.12ms, and MySQL's is around 300ms.

This, of course, has very little to do with the rate at which data is written.

(Yay for IT journals... quality information for PHBs...)

by• 17 years ago

"Cassandra can write 50GB of data in 0.12 milliseconds ..."

These numbers have been taken from p.21 of the linked PDF by the Facebook engineer Avinash Lakshman where it shows this MySQL vs Cassandra comparison:

"MySQL > 50GB Data Writes Average : ~300ms Reads Average : ~350ms Cassandra > 50GB Data Writes Average : 0.12ms Reads Average : 15ms"

Which I'm guessing actually means: with a database table which is more than 50GB total size they have measured individual row accesses at these speeds. Maybe Avinash has accidentally transposed the read and write figures - I can't think why the read would be slower than the write.

15ms to write a row to disk seems possible given that a fast modern disk

http://www.seagate.com/docs/pdf/datasheet/disc/ds_cheetah_15...

has a latency of 2ms and seek time of 4ms and a sustained transfer rate of about 100MB/s. The track to track time is only 0.4ms so maybe if you just wrote all the data to disk serial-log-style you could reconstruct from the log after a failure and handle all reads from memory. I don't know Cassandra. Obviously, from these figures, the disk couldn't do a row read in 0.12ms.

jbellis• 17 years ago

Cassandra uses log-structured merge storage, so reads really _are_ slower than writes. (But still much faster than MySQL!)

You can (usually) make reads faster by throwing things like memcached at the problem. Writes are harder. So I think this is the right tradeoff for a modern system.

/Cassandra dev

pj• 17 years ago

I agree with you. There's a lot of hype around these nosql databases, because a lot of people simply don't understand sql databases or how to make them performant or they haven't run into the problems RDBMSs solve yet.

I would like to add one quick correction or maybe clarification to, [Indexes] merely order the data in a certain way that makes it easy to pluck out certain rows since you don't have to look at every value.

Indexes don't always order the data, only clustered indexes order the data and there can only be one clustered index on a table. You can have lots of unclustered indexes that are merely pointers to data within the larger table. http://en.wikipedia.org/wiki/Index_(database)#Clustered

jimcaruso• 17 years ago

Great post. Clear, sensible, and informative.

messel• 17 years ago

So I'm working on a small project that will develop into a startup if the prototype works. It's heavily based on information handling and to supplement the functionality we plan on having an internal user profile database. You are obviously familiar with the functionality of modern database tech. Was thinking SQL, might there be faster/lower latency lookups? Haven't nailed down all the fields yet but baseline will include taxonomy(language topics), historical preferences (general and specific) I can go into more detail by email if you're curious)

geebee• 17 years ago

This movement sounds reactionary, and perhaps for good reason. SQL has been the default method for storing and retrieving data for a long time, but obviously it isn't the best fit for every case.

That said, I like sql. I find it extremely easy and natural to keep relational models in my head and write queries. While I do appreciate some of the tools that free me from writing very basic CRUD stuff or persisting objects, I don't like to be too insulated from my database, and I'll gladly drop to sql when I need to.

The problem is that sql has been used for all data persistence and retrieval for far too long, when all kinds of different models might have been appropriate. So I guess this is a kind of backlash. And eventually, there will be a backlash against the backlash - or, more likely, people will forget there ever was an argument, and we'll get a new generation of developers who think they've discovered relational algebra for the first time.

silentbicycle2• 17 years ago

Another thing is that some people have problems with SQL itself, not relational databases per se. SQL is just the compromise-of-a-standard that came out of several RDBMS query languages competing for mindshare.

And, naturally, in some peoples' minds any problems that MySQL has are joined to SQL, and in fact all relational databases.

schammy• 17 years ago

This article is the biggest load of fucking bullshit I've ever read. That's all I'm going to say about it. The author couldn't tell his ass from a hole in the ground.

sho9• 17 years ago

I am glad you are getting modded back up from the grave; I agree and appreciate someone calling a spade a spade.

We don't always need to get into thesis-length technical arguments about the merits of so and so viewpoint - sometimes an article is just bullshit and deserves to be dismissed as such.

sho9• 17 years ago

Quite a turnaround. The GP was at -2 when I wrote the above.

HN is a fickle mistress indeed!

fauigerzigerk3• 17 years ago

There seem to be two quite contradictory arguments for NoSQL. One camp says, we already have what we need in the form of [programming language] objects, so why would we take everything apart and map it to something as reductionist as relational tables?

The other camp says, relational tables are soooo complicated, we need something even more reductionist and free form, so key-value pairs are just right.

I can understand the second argument, but the first is a step backwards unless your programming language is something like lisp, which has a reductionist data model itself.

Object oriented models are fine if you write one application. But they tie functionality to data very closely and that makes repurposing of data for seperate applications much more difficult. So how do they analyse their data or use it in more than one application?

They have to write a lot of code using an application specific object oriented API instead of a general purpose reductionist data model. That's horrible but how horrible it is only shows a few years down the road. It's a disaster in the making. A whole army of programmers will be required to extract data from all those big APIs of legacy apps.

I'm convinced that data has a different life cycle than procedural code and therefore needs to be expressed in a simple, uniform, reductionist model independent of all application code.

ubernostrum2• 17 years ago

"Object oriented models are fine if you write one application. But they tie functionality to data very closely and that makes repurposing of data for seperate applications much more difficult. So how do they analyse their data or use it in more than one application?"

This was an extremely poorly-written and poorly-researched article. One problem is that the projects being discussed are not object databases in the sense of storing straight-up serialized representations of business objects.

Take CouchDB for example; the basic "thing" you store in it and get back from it is a JSON object, which is a set of key/value pairs. In this sense it's not too far removed from SQL-based DBs, because you still choose which bits of data you're going to store (the keys in the object). It departs from the SQL point of view in not requiring that all the records have the same schema, and in not having a representation (at the data-storage level) of relations between records.

This turns out to offer some big advantages: in my experience, the number-one cause of unwieldy SQL is a schema that's had to grow over time to accomodate ever more edge cases. Maybe it's been done through lots of nullable columns, maybe it's been done through lots of related tables or some other mechanism, but it frequently has to be done and ends up making the database painful to work with.

CouchDB throws that out the window: edge-case records simply go in like anything else, and if they don't have some particular field present in other records, so what? The "query" is actually hitting a map/reduce in which the "map" function can take whatever action it wants with a record that's missing some particular key. It can skip that record, it can spit out a default value for a missing key, it can do anything it likes.

"I'm convinced that data has a different life cycle than procedural code and therefore needs to be expressed in a simple, uniform, reductionist model independent of all application code."

Simple and independent, yes. Reductionist and uniform, no.

To run with CouchDB as the example, those assumptions are thrown out because CouchDB essentially adds a layer to the stack. Traditionally, you have application code over here, querying data over there. In CouchDB, you have application code which queries a CouchDB view which returns data, but the query doesn't necessarily know anything at all about what the actual data in the DB is, or how it's structured (or even if it is structured in any sensible way; maybe it's just a bunch of random key/value pairs). The view layer is the part which cares about that.

And views are not static (or mostly static) things like the schemas in relational DBs; views are free to evolve over time, you're free to add or remove views in response to changing needs, and the underlying data never has to change as a result. And so you don't need to agonize over the most efficient way to reduce your data to a uniform schema. You don't need to "migrate" your underlying data storage representation to change the types of things you can store or the types of queries you can run.

fauigerzigerk3• 17 years ago

I think data independence all but implies a reductionist data model (uniform probably isn't a very clear term). And when I say data model I don't mean a particular schema but the primitives that are used to model the data, like relations and attributes in the relational model or key value pairs in Berkeley DB.

I cannot say anything useful about CouchDB as I don't know it nearly well enough. What I think doesn't work is to hide data behind a procedural API when it comes to read access (write access is a different matter).

A procedural API is a black box that you cannot reason about and has a very application specific purpose that doesn't lend itself to analytics apps. Analytics apps should know as little as possible about particular applications. They cannot easily call arbitrary functions.

My experience with data centric apps is that it's a good thing to have that situation where everything is a table and each of the few operations you have creates another table. Tables in, tables out. The same thing works with lists, key/value pairs, etc.

My battle cry would be "No applications specific APIs" (for data access)

laforge49• 17 years ago

>>Object oriented models are fine if you write one application. But they tie functionality to data very closely and that makes repurposing of data for seperate applications much more difficult.<<

I've taken a different tack in my research--create a generalized user object and then build applicatin objects on top of them, with the base user object responsible for all persistent data. (These objects can each be mapped into an XML document, so there's a fair amount of flexibility).

The work has gone very slow, but here's the site: http://agilewiki.ning.com/ and mind, I have mixed in some hype/BS, though mostly it is intentional things which I've previously prototyped (been working on this over 6 years now).

antirez2• 17 years ago

At the point you see the load drop on the following graph, we switched from MySQL to Redis:

http://db.lloogg.com/load.dir/load-1month.png

Alternative DBs are not good for everything, but SQL databases are not good for everything as well...

sounddust2• 17 years ago

I'm sorry, but this graph says nothing. Load average is meaningless; for example, one process using 100% CPU at all times would show a load of 1.0, and one process that repeatedly spawns threads but uses little cpu can push the load average very high.

What's the difference in maximum number of pages served per second using ab? Or average CPU usage? Or average disk throughput?

I'm not saying I disagree with your conclusion, but this doesn't prove it.

antirez2• 17 years ago

got your point actually, you can see all the data here: http://db.lloogg.com, there are many stats about load, disk I/O, and so on.

trezor5• 17 years ago

MySQL can hardly be called representative for good relational databases. That MySQL is underfeatured and performs like shit for any non-trivial query is hardly news.

c00p3r• 17 years ago

No to SQLServer and Oracle. =)

hello_moto• 17 years ago

As far as I know, HBase is the software that they should point out in the key-value discussion, not Hadoop.

gsteph22• 17 years ago

I wrote a fairly detailed post about the Death of the Swiss-Army RDBMS on my blog, http://www.roadtofailure.com :)

The RDBMS has been used for way too long as a generic "bucket" for any sort of data storage and analytics. As a result, speed and scalability have suffered. With Social Media and the advent of "web-scale" data, engineers need to realize that the storage and analysis solutions they use must be driven by the type of data they're storing, and what they want to do with it.

As I stat, "The ACIDy, Transactional, RDBMS doesn’t scale, and it needs to be relegated to the proper dustbin before it does any more damage to engineers trying to write scalable software."

gits_tokyo• 17 years ago

Easy in != Easy out ... didn't your mum ever tell you, "there's no such thing as an overnight success"

---

Personally, I have absolutely no rush in dropping MySQL as a persistent store.

No to SQL? Anti-database movement gains steam

Related Stories