Codd's relational vision – has NoSQL come full circle?

http://www.opensourceconnections.com/2013/12/11/codds-relational-vision-has-nosql-come-full-circle

by lukaseder • 13 years ago

107 33 13 years ago

opensourceconnections.com

yepguy2• 13 years ago

Author thinks he is being contrarian by defending SQL... huh?

andrewflnr• 13 years ago

It's a NoSQL conference, so yes.

yepguy2• 13 years ago

Ah, I missed that bit.

lafar65022• 13 years ago

This happens when someone is trying to actually use NoSQL instead of just praising it ;)

softwaredoug2• 13 years ago

hey thanks for your comment (I'm the author)

I must be biased by my peer group :). Maybe I'm less concerned about SQL being a default choice than Mongo or something being the default choice? I say this as somebody that helps people migrate to NoSQL databases as a job.

sowhatquestion2• 13 years ago

Excellent article. As promised, the author brings a historical perspective that's typically lacking in this debate (at least for someone like me who has casually followed it on HN).

Not to stray too far off topic, but this article raised a question that I've had since starting software development in earnest... Why do we have to choose between heavily "normalized" relational databases that structure all data & thus allow more arbitrary queries, OR un-structured databases that are fast and flexible but often slow to query in complex ways? Why can't there be a hybrid "smart" database that dynamically generates indexes (to use the term loosely) based on how it's queried, in order to speed up similar queries in the future? With some kind of additional weight given to the most frequent queries. Granted, it would need some time to "warm up" (not unlike a tracing JIT compiler?), and the implementation might be fairly complex, but other than that, I can't think of any downsides...?

lukaseder6• 13 years ago

You might enjoy PostgreSQL, then. It supports a variety of "weakly" structured data types, such as JSON, hashmaps (key/value types, also called hstore in PostgreSQL).

Many databases also support XML data types. And of course, you can always store unstructured data in CLOBs and BLOBs

sowhatquestion2• 13 years ago

Actually, you read my mind! I'm currently working on a Rails project that uses PostgreSQL with hstore for semi-structured data such as user information, configuration, etc. Haven't gotten too far into it yet, but it looks promising. :)

What I was trying to imagine in my post, though, was some hypothetical DB where all this would be abstracted away--i.e., it would expose a NoSQL-like interface for all data, while working behind the scenes to provide SQL-like speed for frequent queries. Hence the analogy that it would be the "tracing JIT" of databases.

lafar65022• 13 years ago

Yes, but usually you don't want to use these 'weakly structured' BLOBS if you have an alternative. Relational dbs have some weak points but structured data storage is not one of them. I can't imagine a database without a fixed schema, it always exists even if you don't have to declare it upfront.

a8da6b0c91d• 13 years ago

It's probably possible to incorporate very smart query engines in purely relational database systems that would obviate many of the complaints about relational databases. But SQL does not really implement the relational model and makes that infeasible. See "The Third Manifesto" and co.

smilliken• 13 years ago

PostgreSQL is the closest to what you're looking for.

It's really fast; if it doesn't meet your requirements you're probably already aware that you need a specialized solution.

Besides explicit indexing, it also analyzes your data and maintains stats that inform the query planner. As your tables grow and your schema evolves, it will change its query plans accordingly. It can be quite smart. (A few examples: it maintains histograms of common values, null percentages, correlation between physical ordering and logical ordering or rows, and much more).

Of course, you also have a lot of flexibility to denormalize and get the best of both worlds.

MehdiEG• 13 years ago

AFAIK RavenDB, which describes itself as a "2nd generation document database", does exactly what you describe. Whenever you run a query that's missing an index, it will automatically create a temporary "dynamic index" to service it. If it finds that this index is used a lot, it will automatically promote it to a permanent index. I haven't tried it yet however so can't comment on how well it works.

But yes, generally speaking, I'm also surprised to see that in 2013 we're creating our indexes manually. While there are of course applications where you really want to ensure that all the right indexes are created ahead of time, for most applications having indexes created automatically based on query patterns would seem like a much better solution.

lukaseder6• 13 years ago

I agree that automatic creation of indexes looks like a reasonable thing to do for a database like Oracle. Oracle could gather statistics and auto-create indexes where it seems fit, i.e. where there is a lot of querying and little writing going on.

In a way, this is what caches can do. Besides that, Oracle is already very good at giving you statistics and tuning hints to help you assess where you could add an index:

http://stackoverflow.com/a/2937047/521799

DB2 and SQL Server probably have similar tools. As far as I know, they all don't go as far as automatically creating or dropping any indexes.

pepijndevos• 13 years ago

Googles appengine datastore will suggest indexes, but it's rather bad at it IMO.

sehugg• 13 years ago

Well, Oracle has its SQL query tuning tools... but in a production system you often don't want too much magical optimization leading to erratic behavior/performance problems. Oracle also has a "plan stability" feature for this very reason.

jandrewrogers2• 13 years ago

The major downside is that it is an inherently non-scalable behavior for a database engine. A few databases do what you suggest but they are limited in the amount of data they can organize in this way without performance becoming problematic. Since data is becoming quite large rather quickly, it is not a sensible architecture choice these days.

There are several reasons why you would not want to design a large-scale database this way. It implies an enormous amount of extra data motion which is arguably the major killer of performance in parallel databases; it is the same reason Hadoop is so slow and inefficient for analytic queries. Additionally, secondary indexing structures and similar types of de-normalization offer poor performance in distributed environments due to the implied consistency and coordination problems; it will severely degrade insert/update/delete operation performance as the data structures grow. Most database applications also value a high-degree of predictability of performance under load, which this definitely will not allow. These tradeoffs are unacceptable for many (most?) applications.

In short, while you could design a database that does what you suggest, it would only be useful at tiny scales or for read-only databases where consistent performance characteristics are not a requirement.

rusabd• 13 years ago

The major downside is that it is an inherently non-scalable behavior for a database engine

well, that is true for the whole SQL, but certainly not true for some subset of SQL which is rather big. And how much is "quite large"? We have client who claims that 160GB is big (he is running cluster of 8 machines now, poor bastard)

softwaredoug2• 13 years ago

Hey thanks for liking the article (I'm the author)

It seems maybe some of the NewSQL stuff is doing this. You use "SQL" but the data is still organized by some kind of hierarchy (like in F5). I'm trying to get into Google's Dremel paper to learn more about how they do it.

In general, though, it seems if there's any chance something might be normalized, but behind the scenes maybe its actually denormalized a bit, then folks try to layer on some kind of SQL-likeness to the database.

Spooky23• 13 years ago

Fundamentally, the two problems (max throughput for transactions vs. query flexibility) are different. So while you may build a jack of all trades system that can do it all, but if you keep growing, eventually you'll need to make tuning decisions that negatively impact queries or throughput.

The solution in most cases is to maintain transaction and query optimized databases separately.

alok-g• 13 years ago

See also:

SQL and NoSQL are two sides of the same coin, mathematically

https://news.ycombinator.com/item?id=3313234

lukaseder6• 13 years ago

That seems more like a plea for LINQ than anything else ;-)

One of LINQ's main visions is to unify querying regardless of the data source - and it does so quite well. At the time the author of that article conceived LINQ, he intended to unify querying RDBMS, XML, and objects. "NoSQL" (whatever that is) is a type of data store that was not part of LINQ's original master plan. Thus, LINQ has to be retrofitted / enhanced to accommodate new requirements. It would be all too nice if things were as easy as a simple duality, specifically given the fact that the author of that article has now also created a company called Applied Duality Inc.

But history will teach us where these things go. I currently don't see a second E.F. Codd to solve the complexity introduced with the new abundance of NoSQL data stores - yet.

(As always, when Erik Meijer writes something: Very interesting read, nonetheless)

brasetvik• 13 years ago

See also: What Goes Around Comes Around (Stonebraker and Hellerstein)

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113....

That one's from the time where XML-databases were all the rage. Unfortunately, its publication year is missing. (The youngest cited article is from 97)

lukaseder6• 13 years ago

Gotta love Stonebraker!

SeanLuke• 13 years ago

This is neat because, in my Artificial Intelligence class in college, this kind of “reasoning” is presented as something advanced. In fact there’s a whole language, Prolog, that’s whole job is to take assertions about the world and give you extra facts. I had to learn all that stuff in AI when mundane-old SQL was right under my nose giving me the tools to learn anything I want about the world my relations represented!

[Sigh]

I guess broken worldviews like this are why Datalog never really got off the ground in the DB community.

coldtea• 13 years ago

Which reminds me of Fabian Pascal, probably somewhere drinking heavily to forget the current state of the DB industry.

(Context: for more than a decade he tried to educate people on the relational model, the ill-thought back-to-the-past ideas that are NoSQL DBs, and more besides. Unfortunately his old site was scrapped, it seems, and has as of 2012 turned it into this blog: http://www.dbdebunk.com/ )

m_mueller2• 13 years ago

My point is that it’s dangerous to just grab a database off the shelf because its fun and trendy. You might end up creating a lot of technical debt for yourself. Do you know you’re handling an extremely high volume of a single somewhat-well-defined “thing”? Then the hierarchical model might be an amazing bet for you. Do you have many different kinds of records that link together, but don’t necessarily contain parts of the whole? A relational data store might be the ideal solution.

This seems to me the most important point of the article and I fully agree here. What I'd like to add is that a DBMS like CouchDB is IMO fully suitable to the use case you describe. ArangoDB, which was also presented at that conference, appears to be even easier for those kinds of datastores - I haven't been able to play around with it however.

There is no such thing as a perfect DBMS - and that applies to RDBMS as well. With Couch you give up sharding, Ad-hoc querying and per-document security, in return for conditionless replication, relative fool proofness in terms of indexing, the ability to have fully normalized relational data (compared to other NoSQL) and mobile offline capabilities.

lukaseder6• 13 years ago

One thing about ArangoDB. Why don't they implement SQL instead of their weird COBOLESQUE AQL language?

m_mueller2• 13 years ago

I have been asking myself the same question - they tried to explain it but I didn't really understand the point to be honest. From what I get SQL was what they first pursued, but gave up at some point when it became clear that it wouldn't be ideal for their use case of a multi paradigm database (document as well as graph database). In general I agree that when developing a new product, it's good to question everything rather than blindly repeat what everyone has done for 20+ years. I'd love to read a blog entry about this decision though.

Delphiza• 13 years ago

One thing that is never directly mentioned in the history of SQL is the available memory, cpu and storage at the time. When a mainframe has 192KB of core memory and a 300KB/s disk transfer speed, coming up with a way to access all that data was important. The OP example of data in a text file would be okay(ish) today because you could load up the whole in memory and be done with it. Yes, we have more data, more rows, and larger records but are solving different problems with different tools and significantly more horsepower at our disposal. This is not anti-SQL, but the SQL-as-a-solution in history needs more context. With that context, those lessons may be less relevant.

Codd's relational vision – has NoSQL come full circle?

Related Stories