How NoSQL forced the evolution of a scalable relational database

http://blog.memsql.com/nosql/

by rickn • 8 years ago

253 207 8 years ago

blog.memsql.com

rickn• 8 years ago

My name is Rick Negrin. I run the Product Management team at MemSQL, a scalable relational database. I recently wrote a blog on my thoughts regarding NoSQL vs. Relational Databases and I'd love to hear the community’s thoughts on this.

sqlcook3• 8 years ago

Hi Rick, I've been following Memsql for a few years now, are there any plans to release "community" edition? Last time I checked about 1.5 years ago json support was very basic and EE pricing (dont remember exact #s) was rather high. Thanks

siscia6• 8 years ago

What are you looking for in a "community" edition?

berns• 8 years ago

There already was a community edition[0]. And it was replaced by a "developer" edition that can no longer be used in production. It seems it didn't pan out as a marketing strategy and I don't think it's coming back.

[0] https://news.ycombinator.com/item?id=9577663

AtlasBarfed• 8 years ago

I notice not a single mention of the CAP theorem. I have a hard time taking any "scalable" RDBMS solution seriously without a discussion on how you scale well-known problems with deletes and updates, maintaining b+tree indexes, distributed joins, distributed data, node loss/partitions, and distributed transactions/updates.

If you are a new distributed system and don't have Jepsen tests or a similar level of discussion on how your database handles partition events, then that tends to be harbingers of snake oil in distributed systems.

Granted I only did a quick search on CAP on your article, but the initial paragraph of your article didn't exactly invite further investigation or time investment.

If your "NoSQL" is just a single node MongoDB, then you should state that rather than your blanket statement. As is your categorization of "NoSQL" is unspecified and unqualified, leading me to believe this is a management-level article with little regard for the real issues in large-scale distributed databases, and why would I think your software also considers such problems?

stephenr2• 8 years ago

NoSQL came into existence because the databases at the time couldn’t handle the scale required.

Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD.

In the case of Directory Services, it's a completely different approach to both SQL and the common "Document Store" approach of modern "NoSQL" options. Unless you go cowboy-mode, you're still adhering to schemas (albeit with the option to combine schemas), you still use indexed attributes to conduct searches, but you have a literal tree of objects with a more flexible layout (e.g. multi-value attributes without needing to use a JSON column) than with a regular SQL RDBMS.

The rise in popularity of 'modern' NoSQL options (most commonly document stores) is IMO driven by a combination of lazy developers and too much Kool-aid consumption.

logophobia7• 8 years ago

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql.

There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.

exclusiv2• 8 years ago

It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch.

ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer.

I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.

threeseed8• 8 years ago

Not sure what you are talking about here.

There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL).

And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.

1897239544• 8 years ago

don't want to take the time to properly learn and optimize sql.

This sounds like an old-wives tail at this point.

mikekchar• 8 years ago

There were NoSQL databases well before LDAP was a thing. We called them "Object databases". The idea was that doing object relational mapping was a PITA and wouldn't it be better to store your domain objects directly in the database? (As an aside, in case you are curious, the answer is: No. Generally it's not a good idea) Of course if your data is not relational, you can't do SQL. :-)

pjmlp• 8 years ago

Memories of POET and Gemstone spring to mind...

dragonwriter2• 8 years ago

Of course if your data is not relational, you can't do SQL.

The relational model is, as I understand it, fully general, so this literally cannot be the case. There is a problem of “if you do not know the shape of the data in advance” and “if the RDBMSs available to you do not efficiently handle data of the particular shape and access pattern you are using”, but “relational” isn't an inherent property of data, it's a property of a data model, and any data that can be modelled can be modelled relationally. That doesn't mean it is efficiently handled by every, or any, existing RDBMS, though.

threeseed8• 8 years ago

So the developers at Google (BigTable), Facebook (Cassandra), Yahoo (HBase), Apple (FoundationDB), Redis etc are "lazy" and have drunk "too much Kool-aid".

Because these technologies every day power the most demanding applications on the planet.

mmt6• 8 years ago

I think the accusation was against the developers who caused the rise in popularity (i.e. the users), not the developers who created the technologies (i.e. the authors).

In that light, it might be best to exclude BigTable, since it's only available publically as a service.

hnaccy• 8 years ago

I believe Yahoo is Sherpa not HBase.

1897239544• 8 years ago

lazy developers and too much Kool-aid consumption.

When your argument is based on personal attacks of the people using that thing, it sounds more like you have an emotional investment in the alternative rather than a rational argument.

qaq6• 8 years ago

It is a very rational argument people want to copy what Big boys are doing often fairly rationally as they want that stuff on the resume. I've seen countless projects using "BIG data" tools for dealing with 100GB or less of data that has very modest grows rate.

nostalgeek2• 8 years ago

Well, this is exactly why competition is good, in every domain. That's why PostgreSQL got its column mode and JSON type, because there was a clear use case for performance for the former and schema-less data for the latter. EAV pattern is a plague and I'm glad documents are replacing it.

zurn• 8 years ago

What do you mean by PG's column mode?

icebraining• 8 years ago

Postgres has had hstore for schema-less data since 2003.

igl• 8 years ago

This is a very blunt statement and overvalues hstore. Keys AND values in hstore are and can only be text strings.

amirouche• 8 years ago

Why EAV is plague? Isn't JSON type a wrapper of EAV?

nostalgeek2• 8 years ago

JSON type is JSON type, EAV isn't a data type, it's a data modelling pattern.

uncle_d2• 8 years ago

My own experience chimes with this somewhat, having used MongoDB on my last project - quite often application data storage requirements are relatively trivial - and it is a boon to do away with the ORM layer and be able to vary any given object schema without touching the database (although in practice a .js data migration script may be involved, so this is moot). As a sidenote local cultural issue, the fact that we can operate the MongoDB servers ourselves, whereas RDBMS instances are with a central team and buried under a layer of bureaucracy was also probably an operational consideration.

When it comes to joins, these have lately been added to MongoDB, as has SQL - although the functionality is still rudimentary compared to a mature RDBMS.

But where we experienced pain was when the business decided they wanted to do live reporting. We ended up piping the data into a SQL Server instance and using SSRS.

lyqwyd• 8 years ago

The fact that you could operate the mongo instance, but not the DB is definitely a cultural issue, or more likely a managerial issue. Definitely NOT a DB issue. I’ve worked at places where we could manage the DB ourselves, and other places where we couldn’t eve talk to the DBAs without beuraucracy. It’s all about the management in those scenarios. And the company results were as you would expect.

uncle_d2• 8 years ago

Sure - I’m not claiming that as a database issue at all. It’s just a side addendum to note that sometimes technical decisions can have a political aspect.

matwood7• 8 years ago

As a sidenote local cultural issue, the fact that we can operate the MongoDB servers ourselves, whereas RDBMS instances are with a central team and buried under a layer of bureaucracy was also probably an operational consideration.

This is clearly an oversight in the organization and will get fixed. I don't see it as neutral in the argument for or against some system.

But where we experienced pain was when the business decided they wanted to do live reporting. We ended up piping the data into a SQL Server instance and using SSRS.

This is common even in RDBMs. You have a database designed to handle the running of whatever application. That design may not be the most efficient for reporting off that application. Flattening the data, and pre-aggregating is usually a good first step. Many data warehouse solutions can do some of this automagically if set up with a common schema type (like star).

pavbelshippable• 8 years ago

Perfect! We at Shippable moved from NoSQL MongoDB to PostgreSQL for several reasons. Here is a small story,

It started with small problems...

Even though we had the ability to add features at a lightening pace, we started seeing occasional downtimes which always seemed to come down to MongoDB. For instance:

We were very happy to have 24x7 availability with primary and secondary instances of MongoDB. However, our perf suddenly deteriorated one day and retrieval started taking more than a second per document. We tried using many tools and profilers, but could not figure out what was happening. Finally, we rebuilt a new server, switched that over as primary, and rebuilt our secondary. Retrieval times dropped to 150ms again. This is still an unsolved mystery! Our Mongo instance reached 4TB and we were proud of our growing adoption. Due to the lack of tooling around managing large DBs, we relied on indexes to keep the search times low. When NoSQL DBs first became popular, there was no way to create uniqueness, so these features were built as an afterthought. Some of the bloating of our MongoDB was actually due to indexes, but rebuilding them was primitive and the entire DB would lock down. At one point, we needed to reboot our DB server and it took MongoDB 4 hours to come back online. This led to an extended downtime for our service, and we had very little visibility into the MongoDB process or status.

And then came the knockout punch! The biggest advantage, and disadvantage, of MongoDB is that it has a flexible schema. This means that documents in the same collection (aka table in the old world) do not need to have the same set of fields or structure, and common fields in a collection's documents may hold different types of data. In a nutshell, there are no strict schema rules and this opens it up to a lot of cowboy tinkering. While many developers love the flexibility, it also puts a very high degree of responsibility on their shoulders to get things right.

The whole experience we have written on our blog http://blog.shippable.com/why-we-moved-from-nosql-mongodb-to...

mmt6• 8 years ago

Worth mentioning the two links toward the end of your story:

https://www.sisense.com/blog/postgres-vs-mongodb-for-storing...

https://www.enterprisedb.com/node/3441 (mongo-pg perf comparison)

nojvek2• 8 years ago

Almost everything has a schema. Whether you explicitly write it down or have it in someone’s head.

If you really want schemaless then use JSON type in a relational DB.

Also with huge memory and risk machines, one could go multi terabytes with a 3 machine cluster on a relational database.

fogetti3• 8 years ago

"And then came the knockout punch!"

I am trying to understand what you say. So basically this was not clear for you from the beginning?

elvinyung• 8 years ago

I feel like this topic was covered better by Stonebraker :p https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...

In the late 90s/early 00s Eric Brewer also wrote a bunch of papers about his experiences scaling stateful systems at Inktomi. The most famous one is probably the one that introduced/codified the CAP theorem: https://pdfs.semanticscholar.org/5015/8bc1a8a67295ab7bce0550...

keithnz• 8 years ago

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ).

Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've been watching https://www.twitch.tv/nybblesio ( who I first came across here on HN on a thread about live coding via twitch streaming ). One of his projects he works on is a SASS product which is largely done in postgresql leveraging https://github.com/PostgREST/postgrest . Seems pretty good, going to be interesting to see how it turns out ( though I haven't seen him work on it lately )

I tend to like SQL Server, ASP.NET Core and found https://blogs.msdn.microsoft.com/sqlserverstorageengine/2018... and I've been playing around with this kind of idea and using a lightweight mapper when needed ( https://github.com/StackExchange/Dapper which the stackoverflow people created )

I gotta say, I kind of like it

One of the problems of many stacks is that the frameworks wrap general purpose languages over SQL, which, is not really a good idea, SQL is a vastly more capable language for dealing with relational data and layers built over the top often dumb down the database.

The REAL problem SQL wrappers need to solve is that most languages don't have a good interface with SQL. So most often SQL is handled in strings and manipulated with string manipulation and there is no type safe way transitioning data from the DB to a general purpose language.

Microsoft at one stage had Linq to SQL which was quite good..... but they killed it :)

sidstling• 8 years ago

Linq to SQL is still a thing, you may need to click it’s package when you install visual studio 2017+, but it’s still there and it’s still the best SQL interface they have build in my opinion.

We rarely use anything else, but I can see the value of entity if you’re changing your DB a lot. We typically don’t do that though, and to be honest, working with changes with linq to sql, typically goes smoother than entity migrations.

Disclaimer: it’s entirely possible that we simply suck at entity.

pbz• 8 years ago

LINQ to SQL is a dead end. Look into LINQ to DB https://github.com/linq2db/linq2db which is way faster than LINQ to SQL ever was (faster than even Dapper). If you must you could give EF a try, but stay away from LINQ to SQL.

1897239544• 8 years ago

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript.

I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future.

Here are the reasons:

Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a big, complicated, and slow piece of software that tries to make a SQL database into something else. A NoSQL database is like a native version of that. It starts off with you being able to define collections and schemas in code and so on.

SQL is a shit way to query a database. I used to write massive queries that took an hour or so to write just so that the non-programming people could put them into analytics software that didn't support any other methods of input. While I was writing these queries, I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more". The fact is that programming languages are much better at querying databases than SQL.

Joins are slow and complicated, and ORMs are slow and complicated, so you can never actually use any of these things in a high-traffic environment. As you say, not many people will reach these levels of traffic, but it is still a factor.

Most of the comments about "I moved from mongo to postgres" lately are first-time programmers who didn't know any of the basic concepts of databases. They then discovered patterns that the SQL database forced on them and declared that mongo sucks when in reality, they just didn't know what they were doing.

One of the problems of many stacks is that the frameworks wrap general purpose languages over SQL, which, is not really a good idea, SQL is a vastly more capable language for dealing with relational data and layers built over the top often dumb down the database.

This is not true at all. Programming languages are vastly superior at querying a database. Just go write a complex query and compare it to the one in linq or whatever you use.

Microsoft at one stage had Linq to SQL which was quite good..... but they killed it :)

They have entity framework, which uses linq and is the same thing. It is very slow though.

richmarr2• 8 years ago

This is basically a big, complicated, and slow piece of software that tries to make a SQL database into something else.

Just to nitpick a little, but I think it's a worthwhile distinction, ORMs don't try to make a SQL database something else... either literally or philosophically. To use them in any more than a trivial way you still to understand RDBMSs. They just make the queries less verbose and the output more convenient to work with.

CuriousSkeptic4• 8 years ago

Linq works for trivial left join project style of queries. But for anything actually using the features of modern sql-databases it comes short quickly.

As a example try to express something like “sum(x) over (partition by y order by z)”

I do agree that SQL the syntax leaves a lot to be desired, and a proper relational language with a syntax optimized for actual development, and even better, optimized for 6NF style databases, would be awesome, but linq is not by any stretch sufficient to be that

matwood7• 8 years ago

Everybody uses an ORM with a SQL database.

I'm ripping ORMs out of any backend code and using things like jOOQ.

I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more".... Joins are slow and complicated

You know what's really slow? Creating joins in code.

This is not true at all. Programming languages are vastly superior at querying a database. Just go write a complex query and compare it to the one in linq or whatever you use.

It makes zero sense to pull a bunch of records back from the db using multiple network calls to join and then filter. Let the db do its job.

orf• 8 years ago

The fact is that programming languages are much better at querying databases than SQL.

They really are not though. Sure, perhaps for simple single table scans, bu nothing beats the pure optimizing potential of SQL. Try joining three tables in JS neatly and fast.

It's a bit terse, but you're lying to yourself if you think hodgepodge mess of ad-hoc JavaScript written to poorly replicate a single specific query is in any way better than what we have now.

flavio81• 8 years ago

Everybody uses an ORM with a SQL database.

That's not true.

While I was writing these queries, I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more"

Well, no, because SQL is a 4th generation language of higher level than most general-purpose languages. It is a domain-specific language focused on database handling.

A simple SELECT with a few joins and indexes involved encompasses a query execution plan that would be a few hundreds of code in a regular programming language.

I chose Mongo for a project recently.

Storing relational data on a document store is a bad idea.

And if you need a document store, Mongo is pretty much a bad option.

collyw5• 8 years ago

Joins are slow and complicated, and ORMs are slow and complicated, so you can never actually use any of these things in a high-traffic environment. As you say, not many people will reach these levels of traffic, but it is still a factor.

I have to clean up crap written by people like you. Go learn how to use a relational database properly instead of writing tosh like this and crap buggy code in the application layer.

Sjenk2• 8 years ago

Not only easy in Javascript, it is easy in any language that has easy serliazation to JSON. I always use it for my own hobby projects because I dont have that much SQL knowledge. Which makes Nosql easy choice for me, just serialize your object into json and push it to your DB.

lotsofcows2• 8 years ago

That's not a choice. To make a choice you need to be aware of alternatives. You use NOSQL because it was easy to achieve your goal.

ben5093• 8 years ago

Pushing a serialized object into a distributed hash definitely solves some problems.

And you have finite time, so if the choice is between that solution that you can do now and searching for another solution that may or may not be better, it's often reasonable to go with the known solution. In economics, this is known as "rational ignorance."

The value proposition of learning will change, though. You will find it's worth it, especially if you move beyond hobby projects and have to deal with any kind of business logic, to spend some time learning how databases work, and maybe even the math behind them, the relational model.

There are a lot of mediocre or awful references out there. If you need a good text on the subject, I recommend CJ Date's "Introduction to Database Systems." Note, though, I said good, not easy.

sametmax2• 8 years ago

it is super easy to program with javascript.

It would be more honest to say that the weird js design, its lack of features and ridiculously small stdlib made it harder to do anything complicated at the time (no ES6 or webpack, remember ?).

Hence, not good tooling existed for relational databases: no decent abstraction layer, migration lib or ORM. Lost of languages had a 2 liner to pop an sqlite db without needing to install anything, but JS didn't even have that.

So here you are, with thousands of front end devs that were just learning to declare their variable correctly, coming to the server. They had the choice between a technology that needed a lot of boiler plate to get anything done, and provided a painful experience with many early errors.

Or something that just seemed to work out of the box. You just dump your thoughts in it. You don't need to think. No error back. It just says "ok".

Seemed is, however, important here. The first years of mongo, it failed miserably, but you only saw it down the road of the projects using it. And of course the inexperience of the css wizards using it as a "dump-my-data" store prove to be a terrible curse.

At the time, if you said any of those things, people would have called you an old grumpy looser that didn't want to live in the future.

But they learned that, just like "var" was important to avoid bugs instead of using your variable whenever you felt like it, deciding of a formal schema was, even for document data store, an important thing to do.

Fast forward, the JS community has now matured and include modern constructs, a rich ecosystem and people with way more experience. Mongo has changed its design and has now sweet spots use cases.

But we still don't have good tooling for relational DB in JS.

And it's still important to have an explicit document stating which schema you use for your document store. Schema less means you can do whatever you want. It doesn't mean, however, that you should.

kapv89• 8 years ago

True that Node.JS didn't have, and still doesn't have a 2 liner to pop an sqlite db. But what it did have(a few years after its launch) was a pure-JS sqlite client which can leverage V8's event loop and libuv to make parallel execution of db-queries a trivial operation.

There is a lot of mettle in Node.js for server side programming.

PS: have you looked at TypeORM(http://typeorm.io/)

mpweiher• 8 years ago

kicking out as many layers between a relational db and a REST Api

Cool, is there anything we can see? I pretty much stumbled upon something along those lines back in 2003, and developed it into what I now call "In-Process REST"[1][2] and "Polymorphic Identifiers"[3][4].

The idea being that you can stay within a "storage-oriented vocabulary" for as much as possible, and that can include seamlessly all the way from provided interface to various storage mechanisms.

What's neat is that it reduces friction a lot, greatly enhances composability and reusability and just makes things blatantly obvious.

[1] https://link.springer.com/chapter/10.1007/978-1-4614-9299-3_...

[2] https://github.com/mpw/SoftArchLon2016-In-Process-REST/blob/...

[3] https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...

[4] http://objective.st/URIs/

ElectronShak• 8 years ago

The REAL problem SQL wrappers need to solve is that most languages don't have a good interface with SQL. So most often SQL is handled in strings and manipulated with string manipulation and there is no type safe way transitioning data from the DB to a general purpose language.

For me this is (was) the main problem. When building Android apps and there was need to store some data locally, I always used Realm (https://realm.io/blog/realm-for-android/) because their API was way easier to use, other than the default SQLite database. Of course this made the apk significantly big, but at least i knew how to manage the data. SQLite on the other hand, wasn't easy to work with, and worse when it came to versioning / updating the local database schema / definition without messing things.

I say (was) because late last year, I started using Room (https://developer.android.com/topic/libraries/architecture/r...), and working with SQLite on Android has never been way easier. So yes, this has been a problem, but the solutions if built well, are great.

matwood7• 8 years ago

Microsoft at one stage had Linq to SQL which was quite good..... but they killed it :)

They also had a rest API directly into the db server at one point, but killed that also.

infinite8s2• 8 years ago

This is similar to the problem of macro programming by eval'ing a string (in dynamic languages). Basically you are trying to construct a tree (AST) by glueing together linear strings of text (instead of working natively with trees)

_Codemonkeyism2• 8 years ago

Same here, using Mongo helped us a lot to iterate fast, JSON is much more agile than SQL.

We now migrate to PG using 'core' attributes in tables and 'flexible' attributes in JSONB to get the best of two worlds.

I agree to the API problem, JDBC and PG/SQL is not as nice as a interace compared to Mongo libs.

sytse• 8 years ago

When you say JSON is more agile is that because you don't need to write migrations? Do you use a framework that supports migrations like active record or south?

ben5093• 8 years ago

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript.

NoSQL caught on because of what it doesn't do: attempt to implement the relational model.

A lot of people were using MySQL in infamous LAMP stacks to set up websites with virtually no business logic.

So you had:

1. An application that didn't need the relational model 2. A DBMS that had a very poor implementation of SQL, which itself is a poor implementation of the relational model 3. A large community who didn't know anything about database theory.

When you're using MySQL as a glorified hashtable, you get none of the benefits of the relational model, and all the downsides.

From their understanding of the technology, throwing all that out made a lot of sense. It's pretty reasonable to use a distributed hashtable if what you want is a distributed hashtable.

I think the backlash now is that NoSQL marketed themselves as a capable alternative to SQL DBMSs. But they had no query optimizer, no transaction support, no reliable backups, and no schemas. The last was supposedly a selling point, but the reality is that your data always has a schema, and code will crash if the data doesn't match it.

Great marketing, but people were sold half a car and then told building the rest themselves was a feature.

One of the problems of many stacks is that the frameworks wrap general purpose languages over SQL, which, is not really a good idea, SQL is a vastly more capable language for dealing with relational data and layers built over the top often dumb down the database.

Another problem is that object-orientation is fairly broken, and ORMs tried to figure out how to shoehorn relational databases into inheritance hierarchies. If current notions like value classes had been around, a more straightforward mapping would have been possible.

d4l3k2• 8 years ago

MongoDB certainly was missing a lot of things when they first launched and were very hyped, but now pretty much everything you could want in a database is now in MongoDB.

Multi-document transactions + strong consistency: https://docs.mongodb.com/manual/core/write-operations-atomic...

Passes Jepsen test: https://www.mongodb.com/mongodb-3.4-passes-jepsen-test

Schema: https://docs.mongodb.com/manual/core/schema-validation/

Query Optimizer: https://docs.mongodb.com/manual/core/query-plans/

Backups: https://docs.mongodb.com/manual/core/backups/

rbyrne• 8 years ago

This is exactly how I feel.

I am currently working on a project for a client where they expect to have about 3,000 users each with some basic profile information, and the ability to upload some documents etc - nothing too crazy. I don't think they ever have to worry about scaling to 50,000 users and performing any data-intensive database queries. Why not use something like Mongo in this situation?

collyw5• 8 years ago

NoSQL caught on because our industry is like the fashion industry and we need to follow the fads year after year. Plus a lot of FUD about relational databases not scaling (it's true that NoSQL will scale certain problems way better than relational, but the vast majority of people were/are not writing applications at that scale).

uptownJimmy• 8 years ago

Dapper is a great tool for getting an app started and up to speed, and even later once the app gets bigger if sometimes it makes sense to bypass the ORM and hit the DB straight-on.

bunderbunder3• 8 years ago

Microsoft at one stage had Linq to SQL which was quite good..... but they killed it :)

I think that might be like when rock stars die at 26 instead of hanging around long enough to become the subject of a VH1 "Where are they Now?" episode.

I was on a project that used LINQ to SQL long after Microsoft stopped development on it, and consequently fell out of love with it. The caching mechanism was flaky, and could cause really weird problems that manifested in non-obvious ways. Also, changing the database schema in a LINQ to SQL project could be a chore.

I ended up migrating us away from it and toward Dapper. We ended up bypassing it way too often to be able to straight-facedly claim it was saving us from having to understand SQL (if that was ever possible), It was the warts, but even more than that, it was the many situations where, to get the query right, we needed to use a recursive CTE, or a temp table, or a table-valued parameter, or any of the oodles of other things that are awkward or impossible with an ORM.

soundwave106• 8 years ago

Dapper is great IMHO. However, the Microsoft-recommended solution after Linq to SQL ended up being Entity Framework... which was an even heavier ORM.

My personal opinion is akin to the parent: the more you abstract away from thinking like your data store, the easier it is to write poorly performing code, because some of the fundamentals get lost in the process.

For instance, in EF and other ORMs, one classic pitfall is "N+1" queries. Even though it's a well known issue to avoid, I still run into poorly performing loops quite often. Here, I feel the abstraction is partially responsible. Most for loops run in code (in memory at least) very quickly, even against a lot of records. However, those who develop against SQL know that you avoid cursors etc. as much as possible, and that for loops / cursors against a SQL database will bite you the instance you are dealing with thousands of records to loop through. I don't think it's always "obvious" to, say, a relatively new C# developer with less experience writing SQL, that this for loop against a SQL database is going to literally translate into thousands of select statements that will perform poorly.

I think it's possible to use a heavy ORM like Entity Framework and write well performing code, but there has to be some awareness of when the task is too complex for the ORM to perform well, and you need to switch to a stored procedure. And there has to be at least a little bit of understanding on how to write well performing interactions with the DB.

spikej• 8 years ago

Its been mentioned here before, but check out Rezoom.SQL https://github.com/rspeele/Rezoom.SQL

All the power with static typing

antirez• 8 years ago

Redis is not mentioned here at all so maybe the author is thinking mostly of other NoSQL software here, but well, in the case of Redis the whole point was not just the in-memory performance part, but the data model as well. My claim is that you can't really exploit the advantage of using memory if you perpetuate in using the memory to represent the same data model that you were using with relational databases. For instance think at Redis sorted sets in the use case of leaderboards in popular games (the same pattern is used in a number of applications that have nothing to do with games). To use SQL, even an in memory one, in such use case is not going to work. So in the case of Redis the point was to remove the interface between the user and the way the data is actually fetched from data structures, to make the user do the choices in a very direct way. Thus this article does not apply to Redis in my opinion. I've the feeling that many other NoSQL products could argument like that, but in other areas of their research and difference. For instance I've issues to see how modern SQL systems can replace CRDTs based stores.

threeseed8• 8 years ago

This is the problem with people that use the term NoSQL.

Do you mean Cassandra (BigTable), MongoDB (Document), Riak (Key/Value), Redis (Mix), Kafka (Log Structured) ? There are dozens of fundamentally different systems many of which are closer to an RDBMS than their NoSQL peers. And many of them have rigid schemas so it definitely isn't that either.

janemanos2• 8 years ago

Don't forget native multi-model databases like ArangoDB

charlesvdv• 8 years ago

You can also add Neo4j (graph) as part of the NoSQL family, right?

zzzcpan2• 8 years ago

In general understanding performance is way more critical for NoSQL/distributed systems, where one wrong assumption can have a huge performance penalty. SQL as a language and abstract relational model just can't work in this situation well.

For instance I've issues to see how modern SQL systems can replace CRDTs based stores.

They can't of course, but it's not only CRDTs. SQL is incompatible with a lot of things that came out of distributed systems research. Just like POSIX filesystem APIs or pretty much any legacy stateful tech. And no amount of PR articles can fix it.

SkidanovAlex• 8 years ago

To use SQL, even an in memory one, in such use case is not going to work.

Have an index on the score and select with ORDER BY ... LIMIT?

pluma• 8 years ago

I'm surprised how many people still conflate NoSQL with "document database" and only think of MongoDB or CouchDB. I'd love to hear how MemSQL addresses graph data, for example.

hinkley• 8 years ago

Because when the fox has been in the henhouse this long, it’s a fox house, not a henhouse.

pull_my_finger• 8 years ago

I keep looking at Tarantool longingly, but I can't figure out when I would need it.

davidgould3• 8 years ago

I have the same feeling. If you ever figure this out, let me know. It looks very interesting, but I don't think my clients will let me remake their production with it and it is difficult to evaluate databases without data or a workload.

janemanos2• 8 years ago

What concernes me is that the article is strongly biased in the sense, that modern developments of non-relational databases just get ignored while ranting on their state from a few years ago. Schemaless databases do support full ACID transactions (MongoDB, ArangoDB), with some you can also enforce a schema and not everybody loves SQL. So having competition even on the query language-side, can only improve the status quo.

Yes, thinking about your data model will for sure increase your understanding of what you are actually doing... and every professional developer is doing this as well when choosing a non-relational database. Anythings else would be stupid.

For me the article rather shows that the modern developments in non-relational databases do affect vendors on the relational side of the spectrum. Otherwise, it would not make sense to invest so much time in writing such a long article.

ChicagoDave2• 8 years ago

NoSQL is an excellent technology for rapid R&D, but once a domain “settles”, data should be modeled and data stores should be switched to relational or graph backends.

If a system already has a well-defined domain, NoSQL adds little value.

Of course you could leverage AWS DynamoDB and reduce cost, but you still have downstream implications for things like reporting, which requires a known schema with relational paradigms.

wild_preference3• 8 years ago

I don’t see how NoSQL could be better for rapid dev when something like Postgres can migrate its schema and data at once while enforcing constraints. That’s clearly better precisely when your schema is changing.

ChicagoDave2• 8 years ago

In R&D scenarios, NoSQL wins hands down because devs are jamming schema changes hourly. That level of frequency in a relational model is a massive drag on time and it’s wasted.

If it’s a redevelopment of a well-known domain, depending on reporting requirements, I’d slightly side with relational.

But AWS DynamoDB can be a huge cost savings, so as any good architect will say, “It depends.”

gwbas1c• 8 years ago

I used to tell people that MongoDB was the "visual basic of databases." Easy and useful for very simple applications.

stickfigure7• 8 years ago

Everyone seems to want to compare against MongoDB, but when I think NoSQL I think about Google Cloud Datastore and Amazon DynamoDB. Databases which are fully hosted, infinitely scalable, zero-maintenance, transactional, reliable, and - at least with Google's offering - scales down to a free tier. They aren't perfect or applicable in every situation, but they're cheap and easy enough to allow a one- or two-programmer team to achieve massive scale without hiring devops.

MemSQL is so expensive you have to call for a quote.

If it's just "the ability to scale" then sure, SQL is back on the menu. MemSQL, Spanner, CockroachDB are leading the charge and that's great! But you have to pay a pretty penny for it (in TCO). There's still a lot of value in a cheap fire-and-forget scalable database, and there are not currently any SQL options there.

Regarding schemaless: I think this is a divide between people who work in dynamic languages vs people who work in statically typed languages. Schemaless databases are just fine in languages like Java; your classes define the schema with enough rigidity to keep you out of trouble. They wouldn't be my choice for Javascript though, that's for sure.

qaq6• 8 years ago

Outside of very niche use cases pushing enforcing consistency to the app layer adds enough complexity to dev process that a one/two programmer team will not generally handle it well. That's why even Google ended up building Spanner.

stickfigure7• 8 years ago

Having built several large, complicated, high-traffic apps on the Google Datastore, I disagree. Eventual consistency was at worst a minor nuisance.

Rather, I found the major pain points to be 1) no joins and 2) no aggregations.

nojvek2• 8 years ago

I don’t understand the “call to get a quote” sales cycle. I bet memsql has a sales time the size of their engineering team trying to get those fat Enterprise deals but in the grand scheme of things why should I care?

It doesn’t have a free tier, I can’t use it easily and just seems like a lot of hype without any 3rd party to back up their claims. Its the kind of things mongodb used to say to gain mindshare.

Most popular databases like MySQL and Postgres gained its mindshare because they were open source and anyone could verify they did what they said. Only a couple closed source databases have won it big. Behind them were huge monoliths like Microsoft and Oracle with armies of sales teams.

I’m not saying MemSQL will die. They are probably quite profitable. I work for an analytics company with a custom database. It’s just the average joe doesn’t get much from it. The cloud based dbs gets the average joe very far.

adrianpike• 8 years ago

I don’t understand the “call to get a quote” sales cycle.

It's about maximizing revenue from enterprise clients - when the customer LTV is huge but the volume is low, it's smart to focus your resources on those dollar amounts. When the volume is high but the dollar amounts are low, it's smart to focus on optimizing the volume funnel.

manigandham• 8 years ago

They're focusing on customers that can afford it, so it's not about you caring but you not being their primary customer.

You can download the developer edition and it does everything except cross-region replication if you want to test it.

It's a great product and the real deal, we used it before (as a startup too) and it did the job and is one of the most polished databases out there.

scarface742• 8 years ago

I am an AWS true believer but I hate DynamoDB. It is far more limiting than Mongo DB.

MongoDB works great with C#. Using a JSON based database with a statically typed object oriented language goes hand to hand. You don’t have the object relational mismatch and the languages type system helps enforce a “schema” on the database.

With C# you work with a MongoCollection<T> And all of your inserts, Linq queries, etc are statically checked by the compiler.

52-6F-622• 8 years ago

It's been a pleasure working with MongoDB in TypeScript as well

kodablah4• 8 years ago

allow a one- or two-programmer team to achieve massive scale without hiring devops

Know what isn't massive scale? Subjecting yourself to the off switch of a single company. So in that light, AWS-specific or Google-specific options are the worst scale you could ever have.

mylons• 8 years ago

a meteor could also hit your datacenter, neutrinos can fly through your RAM, a person with malicious intent could detonate an EMP next to your servers, etc etc etc.

eikenberry• 8 years ago

Just as an FYI DynamoDB has a free tier as well; per-month you get 25GB storage, 200 Million requests and 2.5 million stream requests.

https://aws.amazon.com/dynamodb/pricing/

commandlinefan• 8 years ago

the need to retrain people

Last I checked, we stopped doing that back in the early 90's - learning new things is your problem and you better know all of them by next Tuesday, or we'll replace you with somebody who does.

beamatronic• 8 years ago

If you want a scalable database that holds JSON and lets you query it with SQL, you should check out Couchbase

richpimp• 8 years ago

Someone please help me understand. I use relational databases everyday for my projects (db, api, frontend). I'm well-versed in SQL and general best practices, and my projects scale well (admittedly I'm not in the petabytes level of scale). Does the desire to use NoSQL databases come from a disdain for having to think in terms of declarative programming instead of imperative? Is it desirable to have dynamically typed fields in a database? I've always thought that the biggest motivation to use NoSQL is simply due to high-level language programmers' unfamiliarity with relational database logic and SQL syntax. Once I grasped the concepts, the relational model has proven indispensable for me.

I will say, though, that it is important to get your schema correct at the beginning. I've also always worked full-stack, so I don't know what it's like not to have the ability to manipulate the database.

wild_preference3• 8 years ago

NoSQL is used by two camps. One wants a denormalized doc store for their technical needs, and the other because it has an easier quickstart on day 1 when you don’t know SQL. Here we are usually talking of the latter.

The beauty of something like Postgres is that you don’t need to get your schema right on day 1. How could you anyways unless tou already know all future data access pattern?

kodablah4• 8 years ago

For me, it's two main reasons: 1) Generating and storing massive amounts of pre-computed permutations of the same data based on how I'll read it with easy linear growth by adding servers and 2) Transparent multi-master reads/writes providing easy HA failover. Note, not all NoSQL options offer this. If existing free relational DBs had these options easy and out of the box, I would use them instead.

I just need a ton of data, often redundant, that I never delete, that supports distributed writes and reads (ACID not required). The things I use Cassandra for would require huge Postgres servers and custom sharding/partitioning setups and Citus HA and all sorts of other things.

njharman• 8 years ago

what it's like not to have the ability to manipulate the database.

In general "not wanting to understand, depend on, or work with other systems/teams" (often cause of dysfunctional eng/pm organizations) has, over 20years experience, been the number one reason devs (esp front end devs cause they tend to move the fastest (faster than traditional backend teams are prepared for) and have least experience in back end systems) go for something new, something they can control, something they can spin up fast and quick. NoSQL, cloud, anything they can do themselves.

jokoon• 8 years ago

Can't you have ACID with write on ram first, write on disk second, delay the commit by committing after writing on disk, and have some tolerance for the time between write on ram and write on disk? How probable is it that a crash happens in that window? The data loss would seem very minimal and rare.

And is ACID always so important for most databases? I don't think most databases in the world are related to banks or require that much guarantees and safety. Better hardware often achieves that level of safety.

Other side note, I wish there would be alternative to the SQL "english sentence" flavor of the language, so that one could give more precise parameters for querying data, instead of building a sentence query which doesn't always make a lot of sense when compared to a non-sentence programming language.

bcheung• 8 years ago

How are people running migrations in these distributed SQL environments? Especially with something like Kubernetes that has canary / rollback functionality. Do people run their migrations as a manual step, is integrated into the CI/CD system?

I can't find anybody talking about how they handle it in a scalable way. Not having a good DB migration story for dev/prod environments with automatic rollout / rollback makes using a SQL DB a non-starter for me.

Most Docker / Kubernetes documentation and their ecosystem assumes the app is fully developed and development doesn't exist. Real world companies develop products and have to perform DB migrations on the production database on a regular basis with each release.

What am I missing?

fogetti3• 8 years ago

It's hard to understand the point of this article. First of all this is a good example of a completely biased ranting.

Secondly after that it's bashing NoSQL through many chapters later it is using those same features that NoSQL was capable of in the first place as a selling point for MemSQL... Seriously? This is a really biased marketing opinion piece at best.

Not to mention that it talks about new types of analytics but it completely ignores (or at least it doesn't mention any of) the category of low latency streaming analytics applications like Apache Storm, Apache Spark, Apache Kakfka et al.

"To do this requires a new breed of analytics systems that can scale to hundreds of concurrent queries, deliver fast queries without pre-aggregation, and ingest data as it is created. On top of that, they want to expose data to customers and partners, requiring an operational SLA, security capabilities, performance, and scale not possible with current data stores"

Guess what! This is what streaming analytics was invented for.

fogetti3• 8 years ago

"To give an analogy, imagine libraries saying they are doing away with the Dewey Decimal System and just throwing the books into a big hole in the ground and declaring it a better system because it is way less work for the librarians"

No, to give an analogy imagine a library where the librarian only accepts books which are below a given size, have a specific color and weight, and he rejects any books which have even a cm bigger size, a different color or it's just a gram heavier. Also from time to time (let's say each month) the librarians are sitting together to decide what other books shall they accept (sometimes extending (and by that inherently limiting) the acceptable ranges to other properties like title or smell). Also to give a good analogy the librarians also reject those books which are returned without their softcover. That's RDBMS for you. It works, but not everyone is satisfied with such a service.

How NoSQL forced the evolution of a scalable relational database

Related Stories