Call me maybe: MongoDB (2013)

http://aphyr.com/posts/284-call-me-maybe-mongodb

by cirwin • 12 years ago

93 72 12 years ago

This is ancient stuff though, is this still relevant today?

leif3• 12 years ago

Yes, there are still problems with the election protocol, e.g. [1]. The right kind of network partitions can cause multiple primaries to stay up indefinitely, accepting writes on both sides of the partition, which will eventually be rolled back. There is another problem with the election protocol that allows writes acknowledged by a majority of machines to be rolled back after an election.

Both of these problems can be fixed by using something like Raft[2] or Paxos for elections, rather than the ad hoc mechanisms used today.

In TokuMX[3], we're currently working on replacing the election algorithm with something similar to Raft, that will eliminate these sources of data loss. We've heard that MongoDB is also working on fixing replication, but we don't know what their exact plans are (they have a bigger challenge since they need to stay compatible with their existing replication algorithms, which use timestamps as transaction identifiers) or whether these fixes will end up in 2.8 or in a later version.

[1] https://jira.mongodb.org/browse/SERVER-9848

[2] https://ramcloud.stanford.edu/wiki/download/attachments/1137...

[3] http://docs.tokutek.com/tokumx

ericingram2• 12 years ago

Once again, a TokuMX engineer steps up to explain issue and offer a potential solution. I can't help but wonder why MongoDB engineers aren't doing this. But no matter, just glad we're using TokuMX.

onedev2• 12 years ago

People still use MongoDB in 2014?

aioprisan• 12 years ago

really, what's the point of that comment?

adamors• 12 years ago

For what it's worth, I think it an interesting question. Considering just how hyped Mongo was a few years ago, and how many other NoSQL databases (CouchDB, Riak, Cassandra perhaps even Postgres with its JSON support) gained mind share since then.

kiyoto• 12 years ago

Enough people use it to have the company valued above a billion dollars, have their first MongoDB World (http://world.mongodb.com) and have a HN darling like Stripe as a power user.

Sure, MongoDB has its shortcomings, but so does <Someone's favorite database>. It's much more productive to understand the pros and cons of using it and use it (or not user it) appropriately.

xtrumanx• 12 years ago

Sure, MongoDB has its shortcomings, but so does <Someone's favorite database>.

I've been learning Postgres lately and have been liking it so far. What are its shortcomings? Haven't run into something bad about it yet.

ulisesrmzroche2• 12 years ago

Yup. Super big names too like Verizon, Forbes, and the British Government too. So...

gaius3• 12 years ago

I'll wager all of those have more COBOL than they do any other technology. Probably 1000x more than they're running on MongoDB.

poolpool• 12 years ago

Having one small team in a 6 figure sized organization using mongo doesn't mean a single thing, but makes your statement true, which is how most of these flagship customers are able to be advertised.

olegp4• 12 years ago

For all the hate MongoDB gets, I have to say building queries in server side JS using MongoDB JSON syntax rather than SQL style queries is the way to go.

Check out these two examples:

https://github.com/olegp/stick-blog-pg/blob/master/lib/serve... - uses Postgres https://github.com/olegp/stick-blog/blob/master/lib/server.j... - uses MongoDB - much easier to construct dynamic queries using the JSON syntax

There are of course plenty of other things a relational DB like Postgres has going for it, so I've been experimenting with having a MongoDB interface to Postgres with data stored using the JSON datatype. It's now feature complete and passes all unit tests with read performance exceeding that of Mongo in some benchmarks:

https://github.com/olegp/pg-mongo

goldenkey• 12 years ago

Your comparison is not apt; an ORM can be used for any SQL flavor.

olegp4• 12 years ago

Having written an ORM myself back in 2000 and used Hibernate and JPA for many years, I'm now of the opinion that ORMs are an unnecessary, leaky abstraction in the modern day of dynamic languages.

One is better off talking directly to the data store and effectively streaming data out via a REST API.

buster• 12 years ago

It's probably nice if you only write JS (say, you write a node.js app). To me SQL is still the best query language we have. It's usable amongst a wide range of database servers, from tiny embedded sqlite to oracle. It's _easy_.

One side that NoSQL fails on is to not have a common query language, imo. You chose MongoDB in your project and want to migrate to another database? Not that easy. Switch from MySQL to Postgres? Probably not a drop-in replacement, but much easier to do.

brickcap2• 12 years ago

To me SQL is still the best query language we have

I agree 100% there. I think orient db proves that it is possible to use sql in a no sql database. Though I think not all databases are designed to support entire sql querying mechanism.

twic3• 12 years ago

To me SQL is still the best query language we have.

OQL! Okay, so nobody ever implemented OQL. But there are OQL-inspired query languages in production which i prefer to SQL for routine use, such as JPQL.

One reason for that preference is the ability to join through foreign keys with a syntax which resembles property access on objects:

  select e
  from Employee e
  where e.department.head.manager.level = 'VP'

This beats the equivalent SQL:

  select e.*
  from Employee e
  join Department d using (department_id)
  join Employee h on d.head_id = h.employee_id
  join Employee m on h.manager_id = m.employee_id
  where m.level = 'VP'

Admittedly, whilst JQPL is nice for this sort of routine fetch-and-filter stuff, it lacks the more powerful features of SQL like window functions, recursive common table expressions, etc. I don't often need those, but when i do, it would be rather painful to do without them.

noelwelsh• 12 years ago

You're comparing concatenating strings vs building data structures. Naturally the latter is preferable. It's fairly trivial to represent SQL as data structures in your host language (an embedded DSL if you like) and then you have the benefits of compositionality you currently have with JSON.

Pacabel2• 12 years ago

That's not a very convincing example.

Let's ignore the fact that it's a trivially small example to begin with. I don't think it's very representative of larger apps, especially ones that are far more complex than a very simplistic blog system.

And let's ignore the fact that the SQL-based one is around 10% shorter than the MongoDB one, too, in terms of their line counts.

The SQL-based one makes the queries very clearly visible. It's easier to find the queries, and for anyone who knows SQL it's very clear at a quick glance what they're doing. It's much harder to isolate the queries in the MongoDB one. The visibility of the SQL queries makes maintenance easier, and performance tuning easier.

And I don't buy the dynamic query argument. Those are often the most fragile ones used, and should generally be avoided, if possible. It's better to have two or more similar queries than it is one that's built dynamically. Even then, it's often possible to write a single query with more complex filtering that can handle numerous very different cases.

At best, the MongoDB example is roughly equivalent to the SQL one. In practice, you're trading off a huge number of benefits of relational databases (SQL usually being one of them) for little to no gain when opting for MongoDB instead.

olegp4• 12 years ago

Thanks for checking out the example and I agree that it may be overly trivial. We do plan to investigate this further and will blog about it at https://starthq.com/blog/

I really think that depending on one's requirements it should be possible to query relational database using MongoDB syntax and use (basic) relational queries on Mongo data.

There have been tools like Metrica (https://starthq.com/apps/metrica) that do this, but we're working on implementing that as a Node package.

lucian19002• 12 years ago

That's why there are nice libraries for building SQL ASTs and generating code for various databases, like SQLAlchemy. They also benefit from not being ambiguous and arbitrarily constrained (like Mongo's joke query language).

Gonzih• 12 years ago

With rise of RethinkDB it would be lovely to see similar post on it.

hopeless2• 12 years ago

I've looked into RethinkDB and I really like it but… you have to migrate your data between each 1.x -> 1.y release which might be trivial early on but impossible at a larger scale :-/

http://rethinkdb.com/stability/

neumino• 12 years ago

This will not be needed for the next releases (if everything goes as planned) https://github.com/rethinkdb/rethinkdb/issues/1010#issuecomm...

garretraziel• 12 years ago

I'm building small application using Node.js and MongoDB and I'm planning to host it on Openshift or Heroku. All that hate that MongoDB takes here on HN makes me reconsider technologies I am using.

I will not have many relations in my database (model User, model Document, User owns Document... and that's all) so I thought that NoSQL databases will do. Plus, MongoDB lets me use GridFS - I'm planning to store pdf presentations in it.

If I should drop MongoDB, what other technology should I use? Or should I fall back to Postgres + ORM and manage my files in filesystem manually?

I don't want to start a flame, I am looking for an advice. I have considered MongoDB to be "good enough" as GridFS lets me store my files without a hassle, but after all that I read on the Internet, now I am not so sure.

Rapzid3• 12 years ago

PostgreSQL actually works quite well as a noSQl/document store :) I believe the best answer though is "it depends". I'm not sure this is the best place but if you gathered your requirements and listed them I'm sure a dstabse that will keep your data safe can be recommended.

barkingcat• 12 years ago

I don't think it's hate - it's a call for users to understand and examine the implications of assumptions that the developers of the tools have made.

Don't take the advertising taglines and slogans as a panacea - there are still a ton of things to understand about how mongodb might or might not work for your circumstances. The advertising buzzwords are just that - advertising. You need to examine each and every technology that you chose as part of your project.

And I totally recommend the entire Jepsen series from that site http://aphyr.com/tags/Jepsen - it's an eyeopener!

In my opinion, if you are starting some experiments, etc, just use postgres! It removes a lot of doubt and new learning/confusion from your development.

reqres• 12 years ago

I was in a similar situation to you when MongoDB was a lot less mature. Postgres was my "go to" store and I was a very skeptical of the Mongo hype.

My view would be to use Mongo if it's just a small project. It's a good time to learn exactly how Mongo is different and get a first hand experience of what the trade-offs are.

Pacabel2• 12 years ago

It's not about "hate".

There's no emotion involved, in fact.

It's about rationally looking at a given tool, considering its capabilities, considering how it can be beneficial and harmful in a given situation, and considering how it compares to similar other tools.

Sometimes this analysis is in favor of a given tool. PostgreSQL is a good example of this. It excels in many different ways, including some that overlap with MongoDB and other so-called NoSQL databases.

Sometimes this analysis is not in favor of a given tool. MongoDB is a good example of this. It has some pretty serious issues, and there are often better alternatives.

When people suggest to not use MongoDB, it's generally not some emotional response due to "hate". It's usually because they've considered what it can do and how it works, and it turns out that in pretty much all cases there is a better alternative available that should be used instead.

threeseed• 12 years ago

Most of the MongoDB hate comes from the PostgreSQL crowd.

It's actually a fine database for many domain models i.e. lots of nested data and GridFS does work pretty well. That said don't use GridFS. Use something like S3 and reference the files.

And if you are planning to use MongoDB then look at MongoLab/MongoHQ. Anyone who says you should run your own database should NOT be listened to. Use a hosted solution if you are starting off small. You don't want to be spending your valuable time testing your backups (just one of the many operational activities most people don't do).

jamesaguilar4• 12 years ago

Most of the MongoDB hate comes from the PostgreSQL crowd.

Or anyone who doesn't like losing their customers' data.

sandstrom• 12 years ago

I've used MongoDB quite a bit, and find it convenient. That said, it also has flaws (many has been pointed out on HN and elsewhere).

It's schemaless and malleable, easy to restructure the data. Fairly easy to operate, replication is easy to setup.

Don't use GridFS though, I'd go for S3 or similar instead.

StavrosK3• 12 years ago

I've tried to go non-relational for things that required a few relations and keeping things consistent was a pain. Owners would drop and documents referring to them would still be in the database, causing problems. I'd go with Postgres until you know it's not enough (and I do, nowadays).

jamesaguilar4• 12 years ago

If it's a small app, just use Maria or Postgres. IMO, the only time to even think about using one of these non-traditional stores is when you've scaled past what SQL systems can offer. And even then, I'd use the non-traditional store like I use a C extension in Python -- only for the parts that can't scale without it.

StavrosK3• 12 years ago

Is there any datastore in this series that behaved correctly in a partition? I've seen ElasticSearch, Redis, Riak, Mongo, and all of them crapped their pants.

misframer• 12 years ago

Zookeeper? Not in the same category as MongoDB, of course.

nemothekid• 12 years ago

Cassandra, Zookeeper and Kafka didn't shit the bed.

tveita• 12 years ago

Kafka did lose a lot of data in the Jepsen test, as it will by design to preserve availability under some specific circumstances.

The behavior in question will be configurable from Kafka 0.8.2, by setting "unclean.leader.election.enable" to false.

See https://issues.apache.org/jira/browse/KAFKA-1028

dbenhur2• 12 years ago

On Cassandra, some CQL collection operations behaved well (adding elements to a set). Everything else Kyle tested there was demonstrated to lose data.

Kafka lost data all over the place under Jepson; though Kyle offered great respect to the team and expects them to deliver optionally configured safer semantics (at a performance cost) in future releases.

Riak was totally solid when using CRDTs and turning off the insane LWW default.

zardosht3• 12 years ago

Another thing to keep in mind is that not all systems are meant to be CP systems.

brickcap2• 12 years ago

I was hoping that aphyr would cover couchdb at some point. But since he didn't I assume he tried it and was satisfied with it.

But it might be that it is an ap system and there was nothing much to say there.

jamesaguilar4• 12 years ago

So far as I can tell, no traditional SQL systems have been tested. I'd lay a small bet that Postgres wouldn't lose data, although I don't even know if it has any kind of masterpool capability.

StavrosK3• 12 years ago

Nah, he already tested Postgres: http://aphyr.com/posts/282-call-me-maybe-postgres

dbenhur2• 12 years ago

Never observed shitting the bed -- Zookeeper: http://aphyr.com/posts/291-call-me-maybe-zookeeper

"As with any experiment, we can only disconfirm hypotheses. This test demonstrates that in the presence of a partition and leader election, Zookeeper is able to maintain the linearizability invariant.

Recommendations

Use Zookeeper. It’s mature, well-designed, and battle-tested."

Doesn't Shit the bed when used correctly -- Riak: http://aphyr.com/posts/285-call-me-maybe-riak

"Is there no hope? Is there anything we can do to preserve my writes in Riak?

Yes. We can use CRDTs.

If we enable allow-mult in Riak, the vector clock algorithms will present both versions to the client. We can combine those objects together using a merge function.

... CRDTs preserve 100% of our writes. ... Moreover, CRDTs are an AP design: we can write safely and consistently even when the cluster is totally partitioned–for example, when no majority exists. ...

Strategies for working with Riak

Enable allow-mult. Use CRDTs."

There exist methods where by it wont shit the bed -- Cassandra: http://aphyr.com/posts/294-call-me-maybe-cassandra/

"CQL and CRDTs

Without vector clocks, Cassandra can’t safely change a cell–but writing immutable data is safe. Consequently, Cassandra has evolved around those constraints, allowing you to efficiently journal thousands of cells to a single row, and to retrieve them in sorted order. Instead of modifying a cell, you write each distinct change to its own UUID-keyed cell.

Cassandra’s query language, CQL, provides some collection-oriented data structures around this model: sets, lists, maps, and so forth. They’re CRDTs, though the semantics don’t align with what you’ll find in the INRIA paper–no G-sets, 2P-sets, OR-sets, etc. However, some operations are safe–for instance, adding elements to a CQL set: All 2000 writes succeeded. :-D

That’s terrific! This is the same behavior we saw with G-sets in Riak. However, not all CQL collection operations are intuitively correct. In particular, I’d be wary of the index-based operations for lists, updating elements in a map, and any type of deletions. Deletes are implemented by writing special tombstone cells, which declare a range of other cells to be ignored. Because Cassandra doesn’t use techniques like OR-sets, you can potentially delete records that haven’t been seen yet–even delete writes from the future. Cassandra users jokingly refer to this behavior as “doomstones”.

The important thing to remember is that because there are no ordering constraints on writes, one’s merge function must still be associative and commutative. Just as we saw with Riak, AP systems require you to reason about order-free data structures. In fact, Cassandra and Riak are (almost) formally equivalent in their consistency semantics–the primary differences are in the granularity of updates, in garbage collection/history compaction, and in performance.

Bottom line: CQL collections are a great idea, and you should use them! Read the specs carefully to figure out whether CQL operations meet your needs, and if they don’t, you can always write your own CRDTs on top of wide rows yourself."

_JamesA_• 12 years ago

Does anyone have experience/comparisons with OrientDB? http://www.orientechnologies.com/orientdb/

It seems to be much lesser known but it ticks all the right boxes compared to other NoSQL datastores.

Call me maybe: MongoDB (2013)

Related Stories