Predictions on the future of databases

http://gigaom.com/2013/12/14/5-predictions-on-the-future-of-databases-from-a-guy-who-knows-databases/

by thewarrior • 13 years ago

49 52 13 years ago

I still think its amazing that Facebook is running off MySQL. Such a tremendous amount of user data, placed in a system that many deem inferior.

I'm not saying they made a bad choice, I'm just impressed at how they've managed to make MySQL fit their needs.

GuiA• 13 years ago

I'm guessing the way they use MySQL is similar to the way they use PHP?

Impressive indeed.

RyanZAG2• 13 years ago

I was under the impression they use it more as a sharded "key-values" store?

ie. no joins - the part of the database that makes it relational.

gbog• 13 years ago

I concur. In our shop we also use MySQL without the joins. That's like an efficient key-value store, but there must be a better solution somewhere: all the joins are done in client code, and caching too. That's a great amount of error-prone code that probably shouldn't be necessary.

rimantas• 13 years ago

  > that many deem inferior.

I guess most of those "many" never actually run anything close to the scale of the Facebook (or Wikipedia, which also runs MySQL). If you bother to pay attention what FB DB engineers say you will know that they have their arms shoulder deep into the innards of the database itself, I/O stack, all that jazz.

davidw2• 13 years ago

If you bother to pay attention what FB DB engineers say you will know that they have their arms shoulder deep into the innards of the database itself, I/O stack, all that jazz.

Most of us can't afford that, and want something that's pretty good out of the box. I've used Mysql a number of times, but it always seems to have more gotchas for what I need to do with it than Mysql.

eknkc• 13 years ago

I wonder if it fits their needs or they are stuck with it? I mean, at some point when you hit petabyte scales, is it even plausible to move to other data stores / rebuild indexes etc? I would not dare to do that.

h2database2• 13 years ago

What about H2?

http://www.h2database.com/html/main.html

I can't think of a better reason NOT to use it. Can someone who knows a lot about db please comment on this.

h2database2• 13 years ago

and also check this chart:

http://www.h2database.com/html/features.html#comparison

ddorian434• 13 years ago

http://www.slideshare.net/PostgresOpen/postgres-is-different...

bsaul3• 13 years ago

Could it be that facebook complete lack of innovation is due to them being stuck for years with the same issues on the db ?

ddorian434• 13 years ago

aren't they using many other dbs for different stuff, like hbase for messagess, some other for graphing etc

bsaul3• 13 years ago

I don't know. That was an honest question. The original post seemed to mention facebook being stuck on mysql as a big issue. So that made me wonder if that could be a reason why the only useful feature facebook released was facebook chat, and it was more than two years ago.

threeseed4• 13 years ago

You don't really understand how popular websites work do you ?

Their users don't want innovation. They want a stable platform with moderate and infrequent changes.

ddebernardy• 13 years ago

Best I recollect, FB is quite innovative in the data center.

Also, it would be nice of you to step in and contribute to an open source database engine if your expertise is such that you can dismiss what FB did with them as lacking innovation. I'd wager they'll be all ears open for your explanation how to do multimaster replication and distributed joins efficiently while remaining ACID compliant.

bsaul3• 13 years ago

I wasn't talking about facebook technical innovations, but rather end-user new features, sorry i wasn't clear.

VLM4• 13 years ago

This is an example of an interesting problem with "innovative" databases which is vaguely like the OSI networking model

The first layer is business need, without one there is no point. You need to start here. Facebook is empty here, so no need for innovation and no concrete ideas about what to innovate about, means no point discussing arch or later tech.

The next layer is architecture. Do you need joins? Well then use them. Do you need transactions, or not? Basically a big key value store?

The final step / layer is tech. You can do a boring key value store on mysql just fine. There might be a slightly faster or cheaper software choice. Or maybe not. Or maybe some nosql thing fits the architecture.

This is a huge problem I see with "the whole market is going to be disrupted totally and only I know how" type articles. So I herd cats around a quasi-geographic database at work (not directly geographic, but related). The only way to change its scale or business needs is to modify the geography of the United States or dramatic demographic changes that would even theoretically take generations to breed the eventual customers. So layer one can't change for that job.

The arch doesn't seem like it can be improved very much. So layer two can't change for that job.

Therefore because layer one and layer two are static, layer three will utterly change completely. Because he says so. LOL

siddboots2• 13 years ago

The future is a fickle thing. 15 years ago I, along with many others, would have told you that this was "the future of databases":

https://en.wikipedia.org/wiki/The_Third_Manifesto

The difficult problems since that time turned out to be scalability and parallelism, neither of which were addressed by The Third Manifesto. So instead of D, we got Map Reduce and NoSQL.

Who knows what the hard problems of the next 15 years will be.

dangoldin• 13 years ago

Interesting - thanks for sharing that. It highlights the disconnect between the theoretically correct solution and the practical one.

lmm3• 13 years ago

The object-relational mismatch is still a huge problem. I'm quite interested by this supposed solution. But I spent 5 minutes chasing links and couldn't find an example of what the syntax actually looks like, only a suggestion that I buy some book to understand the language. This "D" thing sounds pretty cool, but it seriously needs some more user-friendly marketing (and a non-colliding, googleable name).

dragonwriter2• 13 years ago

Most of theninterestin info is linked from the reference material section of www.thethirdmanifesto.com; what you Semmelweis to be looking for is:

http://www.dcs.warwick.ac.uk/~hugh/TTM/Tutorial%20D-2013-05-...

Which provides a description of Tutorial D, an implementation of the prescriptions.

siddboots2• 13 years ago

Tutorial D is an example language spec, but the syntax itself isn't really the point.

D would overcome the object-relational mismatch by making the relational side much, much more powerful. It's really about unifying the ideas of "tables of data are relations" and "relations are logical propositions", so that one could have an über-powerful general purpose language that just happens to also be a natural interface for databases.

D is essentially a set of low level prescriptions that the authors felt would be required in order to achieve such a language.

CmonDev• 13 years ago

"Obamacare, for better or for worse, is being launched on a NoSQL database system" - silver bullet did not help (along with HTML5 etc. whatever).

Ygg2• 13 years ago

So are you implying that a regular DB would have fared better?

I think it was a failure because of the complex law background and interfacing with all kinds of legacy systems it has to draw data from.

threeseed4• 13 years ago

Obamacare website failed because of bad capacity planning.

And from reports coming out it was the Oracle Identity system that was causing many of the headaches.

digitalzombie• 13 years ago

It's just going to be different dbs in the backend working with what they do best in tandem.

Lucence base for search. RMDB for money/transactions. And other noSQL type for just data that isn't critical, don't care much about relation, and just need it to be fast.

Of course, perhaps PostgreSQL and other RMDB can have noSQL attributes then noSQL would have some good competition.

PostgreSQL if they can make it easier to clustered, like cassandra or elasticsearch. And get a better fuzzy string and other text search capability in there then now we're talking. But I guess it's just me dreaming.

ddorian434• 13 years ago

postgresql XC is master-master, shardable, cross-shard transactions, data partitioning across nodes, distributed queries

aquadrop2• 13 years ago

Yes, but it's side project so it doesn't get such attention\confidence as postgresql. But I hope the project won't get abandoned and will evolve into something very powerful.

threeseed4• 13 years ago

It's also not part of the mainline code base which gives it the impression of being unsupported and untrustworthy.

perfmode• 13 years ago

RMDB?

ddorian434• 13 years ago

please explain what does the comment mean

yxhuvud• 13 years ago

Probably a misspelling of RDBMS.

threeseed4• 13 years ago

By the sounds of it you don't understand how banks work today. They are eventually consistent and would work great with databases like Cassandra. And the growth of NoSQL today is in enterprises which are almost entirely mission critical.

It's actually the smaller SQL databases (MySQL, PostgreSQL) that are being used for non mission critical uses.

aquadrop2• 13 years ago

By the way, as long as we're speaking about DB future, did anyone try foundationdb? Does it live up to expectations?

JackMorgan• 13 years ago

Anyone who cares about databases absolutely must check out Datomic. If you want to know the future of relational databases for the next 15 years, I think it is here. Practically infinite client side scaling, logic based querying, ACID, runs on several foundations, supports joins, all data is permanent so it stores the entire history of every fact. You can query "right now" or in the past at any known time. The database lives on your application stack the same way git checks out your code on a development machine, and you push and pull data to bring your application up to date with other stacks.

It doesn't meet every single need you might ever have, but it sure meets a whole lot more then any other database I've seen. http://www.datomic.com

dmiladinov• 13 years ago

Also, "as of" reporting is a built-in feature requiring no additional configuration.

misframer• 13 years ago

I'm really skeptical about Datomic. I think it has some great features, but I have a hard time believing that ACID transactions is one of them.

I've asked about this on their Google Group: https://groups.google.com/forum/#!topic/datomic/zkMV50VKw2s

As of right now, there are still questions left unanswered.

stuarthalloway• 13 years ago

http://docs.datomic.com/acid.html

VLM4• 13 years ago

The most popular corporate world database system at this time is Excel. And its not a very good DBMS, and the world is full of spreadsheet guys who know nothing about database theory, designing horrific abominations.

On a large scale I think the future will be interesting, will "real databases" remain in IT land or will it move out in the world?

Don't laugh, I'm just barely old enough that when I started out, printing, file management, physical media management, static network address configuration (not just IP in the old days), and backups were done solely by IT professionals in the machine room, and now for better or worse every noob does it (or tries to do it) themselves. If in the future someone comes up with software to automate away DBAs...

Or it might go the other way and the corporate business database of the future will be super turbo hyper i-Excel.

tsahyt• 13 years ago

There's a lot going on and that's the way it has been for quite a while now but as someone who's working with databases only marginally at the moment, I'm left wondering: What's wrong with RDBMS? Sure it's nice to have a simple Key-Value Store for applications that don't need much more than this but bottom line, the relational model is very powerful, albeit quite abstract, and most data we need to deal with can be made to fit into this model. We have a working, declarative query language that Just Works(TM), for which we have written very good optimizers. The various instances of such DMBS range from small-scale use (for example, SQLite) to really-huge (Postgres, Oracle, etc). So, to sum it up: Exactly why should we abandon this?

VLM4• 13 years ago

Most boots on the ground DBAs don't know anything about rational DB design. Codd Normal form? Whats that? Can't I just make one giant table? Indexes? Why would I want indexes? I heard they take up disk space and CPU so I won't use any, to increase performance. So they (poorly) implement key value stores regardless what the problem actually requires (if the only tool you know how to use is a nail...). Nothing funnier that watching a noob do a select * on a large table transferring gigs of data and use his RoR app to implement the WHERE using nested ruby if statements and string comparisons (or better regexes without understanding regexes, LOL) and god help me a bubble sort instead of an ORDER BY clause and of course hit another table once for each row selected from the first table because he doesn't know JOINs even exist, or heard they're "slow".

A RDBMS is a very slow key value store, so anything that can't do transactions or can't do anything else, is always going to be faster at its extremely limited set of abilities.

From an engineering perspective if you need 10 HP to run your 5 KW generator, a RDBMS is like installing a 10000 HP marine diesel and then not having any idea how to start or maintain it or even where to get the fuel. Obviously every objective performance metric would be better for a 10 HP lawnmower engine in that app, if all you'll ever plan to use is 10 HP and you have no idea how to use anything more advanced anyway.

There are also very loudly trumpeted anecdotal situations or contrived thought experiments where certain unusual technologies fit a unusual situation very well. This is, oddly enough, unusual.

I've read and experimented with the "7 DBs in 7 weeks" book and it IS very interesting but I can't find any business cases to actually use any of it, which is somewhat frustrating. And my experience is how you end up with people writing CRUD apps to store cooking recipes that none the less use NEO4J because they really, really, want to add a line to their resume that they used NEO4J, not because the app needed it.

StefanKarpinski2• 13 years ago

Condescend much?

lmm3• 13 years ago

the relational model is very powerful, albeit quite abstract, and most data we need to deal with can be made to fit into this model

Maybe, but it's awkward. Graph or tree structures are still painful to store in relational databases, and a lot of problems turn out to involve those shapes.

We have a working, declarative query language that Just Works(TM), for which we have written very good optimizers.

Maybe, but the tooling is still terrible. Where are the libraries? Where are the IDEs? Where's the integration that makes it easy to call procedures in my application language from SQL and vice versa? General-purpose programming languages have got better and better in the last 25 years, while SQL has stayed static.

So, to sum it up: Exactly why should we abandon this?

Look at the ORM problem. Why do people continue to use these horrific bloated, leaky tools? Because it's really nice to be able to store and query in the native language of your application. We need storage tech that's better at supporting this.

Predictions on the future of databases

Related Stories