The genius and folly of MongoDB

http://nyeggen.com/blog/2013/10/18/the-genius-and-folly-of-mongodb/

by throwawayeau • 13 years ago

339 270 13 years ago

bsg753• 13 years ago

MongoDB is easy to make fun of.

I think more often its easy to poke fun at _how_ its used.

When any tool or tech is used globally, before knowing its limitations, problems are likely. Attempting to use MongoDB in all storage or persistence scenarios is no more sensible than using MySQL in all cases.

Yes, there is marketing around this product that must be looked at critically - after taking into account that many newly developed technologies won't solve all the problems older tech have worked for decades to solve.

hayksaakian• 13 years ago

I think the difference is you don't get made fun of online for using MySQL for everything

cincinnatus• 13 years ago

MySQL is richly deserving of ridicule as well. It's ubiquity is unfortunate.

eli• 13 years ago

Nonsense, this is a religious debate. People get made fun of for using MySQL as a regular relational database.

bsg753• 13 years ago

Empathy perhaps.

crucialfelix• 13 years ago

people make fun of MySQL all the time. especially postgresql people :)

anecdotally in the more than 10 years I've used MySQL I've never had any issues. whereas with postgresql I've had a few major downtime incidents. it can be very stubborn and arcane. but at least I didn't lose any data.

rsynnott12• 13 years ago

Attempting to use MongoDB in all storage or persistence scenarios is no more sensible than using MySQL in all cases.

Substantially less sensible in many cases. MySQL has its issues (it has a lot of issues), but people have been able to get it to work surprisingly well in roles that it wasn't designed for (albeit sometimes by just building a database on top of it, as with Twitter's thing).

theatrus2• 13 years ago

MySQL fills the "networked reliable transnational BTrees on disk" niche fairly well.

mattdeboard7• 13 years ago

Really like this article. I try not to dump on MongoDB too much because frankly I have never taken the time to understand its internals. I constrain my criticisms to particular unnecessary failures/inadequacies that I personally have experienced (or any "I'll just use mongodb so I don't have to worry about my data" sentiment).

Funny punchline at the end there too.

calinet62• 13 years ago

I like this too. There's very little to criticize, since it basically tells it exactly like it is without too much embellishment, and makes intelligent, honest conclusions.

bithive123• 13 years ago

This article doesn't really make a case for "genius" -- "saving grace", maybe. And in what universe are the Redis data structures "crazy"?

fiatmoney• 13 years ago

Redis internal data structures are quite sophisticated.

bhahn• 13 years ago

How does being sophisticated imply crazy?

nbevans3• 13 years ago

If you consider sets and trees from CompSci 101 to be "quite sophisticated".

bsg753• 13 years ago

I read it as "crazy useful".

mattdeboard7• 13 years ago

I think the genius referred to is in its simplicity. This simplicity let MongoDB get a product to market very quickly, as well as the inherent goodness of making simple things.

I think in MongoDB's case, the getting-to-market part pushed a little too hard on the make-it-simply part. Simple is good but a thing should be as simple as possible, no less.

rsynnott12• 13 years ago

Its apparent simplicity to _developers_ was certainly a good marketing tool, but it did come with tradeoffs; some of its limitations imply a considerable amount of developer and operational complexity if you actually want to use it.

meowface3• 13 years ago

Most databases don't give you the option of storing an array, a hash, a set, or a sorted set. Postgres does, but it's kind of hacky in my opinion.

lucian19003• 13 years ago

SQL databases usually give you all of those (and more) in the form of indexes. Merely a different way of looking at the problem.

functional_test11• 13 years ago

He's right that MongoDB could use improvements like string interning so you don't need to worry about field names. But overall, I think this article is very misleading.

If you use MongoDB in production, you should definitely take he time to learn about the durability options on the database side AND in your driver. By using them appropriately, you can have as little or as much as you like. Data sets larger than 100GB are no problem either -- right now I'm running an instance with a 1.6TB database.

As always, use the right tool for the right job. If you need joins/etc. and don't need unstructured data, Mongo probably isn't a great choice (even with the aggregation framework).

yummyfajitas6• 13 years ago

For what use cases is Mongo the right tool?

mattdeboard7• 13 years ago

It's "a" right tool in any case where distributed storage of unstructured data in JSON format is wanted, where database-level locks won't be an issue of concern, and availability is the primary, overriding concern.

jbooth5• 13 years ago

From what I've seen, the data model has a lot of utility as long as you don't need super high concurrent performance. Basically, the same area as where rails is the right tool - we want easy features and rapid development, will worry about scaling later.

functional_test11• 13 years ago

I use it to store a lot of historical time series data that doesn't change once written (at least, not often). I can easily achieve the write performance necessary to record the data streams live. Since it's all append-only, I don't need to worry about fragmentation. With replication, it's possible to access the data with very high throughput which is useful when the data is being accessed by a cluster, for example.

I also use it as a metadata "scratch space" for highly available applications (things where failures are not acceptable and must run for days at a time). Again, with replication and automatic fail overs, I've been able to maintain 100% uptime outside of maintenance windows. Obviously that can't last, but so far it's been >2 years with no major problems.

EDIT: I should point out that although the size of the metadata objects can be highly variable, since I usually had a small number of them relative to the time series, fragmentation was still not an issue.

JulianMorrison4• 13 years ago

You have a smallish number of documents where some particular field of fixed size gets overwritten a lot, the old values are uninteresting, and it wouldn't really be a tragedy if your data got trashed. For example, it's the player's score.

You want a fixed-size, rolling backlog of time series data such as logs.

hannibal5• 13 years ago

Like the article says, ZFS is pretty damn good.

Keystore where the "engine" is ZFS works mighty well and is reliable. There is little need for simple solutions like MongoDB if the filesystem rocks.

justincormack• 13 years ago

Filesystems are not great as kv stores if your values are usually very small (under page size). Well, they might still be better than some systems...

rogerbinns• 13 years ago

There are also people using MongoDB and finding it meets their needs well, and don't feel the need to keep writing about how everything sucks or is wonderful. (I'm one of them.)

None of how MongoDB works is a secret. And just like everything else it has sweet spots and problem areas. And like many others, development continues and it gets better.

The database does not get the job done - it is a tool to help get the job done.

orthecreedence2• 13 years ago

None of how MongoDB works is a secret.

Maybe not now, but this hasn't always been the case. The fact that they had (have?) a global write lock was completely buried on the doc site for ages. Benchmarks were waved in front of developer's faces to distract them from the "drivers don't actually write data, they just blast it out in every direction and hope it lands somewhere good" BS.

I don't use Mongo anymore, and I think a lot of it has not to do with the database itself, but with the way 10gen used their marketing machine in a dishonest way. They incurred a lot of trust-debt, and now have a serious amount of work to do to pay it back.

jbooth5• 13 years ago

That's all true, but they were giving 90% of their users exactly what they wanted:

"We value feature-set and expressiveness much more than scalability at our data size, but we want to feel like we're big data too so say some of that stuff"

And that's their brilliance, they listened to what people said they wanted and then gave them what they really wanted.

rogerb• 13 years ago

Sorry, but the global write lock has been public knowledge since the very early days. Mongo's done nothing to hide that. Also (i worked for 10gen at the time) the "marketing department" that you refer to, was a single person organizing events to give developers what they wanted: knowledge and a community around mongodb.

rit• 13 years ago

I worked for 10gen (now MongoDB) for over 2 years (I left in December).

Never once while I was there did they publish a benchmark: There was a [publicly] stated company policy to not publish or comment on benchmarks.

If you have evidence otherwise (i.e. benchmarks published by the folks working on MongoDB) fine, but I take this as a deliberately inflammatory (and false) statement.

EDIT: The global write lock was removed ~last August; there is now a database level lock. Future releases will likely make that more fine grained. Additionally, the drivers no longer do "unsafe" writes, but check w/ server.. as of the same release.

calinet62• 13 years ago

It's important to make the inner workings of our tools visible and understandable, so that we can make the right decisions about them.

In this case, this article does nothing but elucidate the truth, quite plainly and clearly, in fact. That's important, in and of itself.

How you use it is up to you, but we must not vilify truth.

invisible2• 13 years ago

Mongos (the routing for MongoDB when clustering) has a bunch of drawbacks that make MongoDB worse. One of which is dropping all connections when a master switches. Have you dealt with those problems yet? MongoDB is generally perfect at small scale is what I have perceived.

The new database level lock in 2.2 is also annoying (and arbitrary) but it is better than the global lock.

ak2173• 13 years ago

I'm curious, how do other DBMSs handle a master switch/other cluster updates? I'm familiar enough with mongos to know how it works but not what e.g. redis or postgres or mysql does.

weddpros• 13 years ago

Most people here seem to complain that their champion lost a benchmark, or that Mongodb does not follow their ideology.

I've used Mongodb for the last 18 months, never lost any data, and it made it obvious that "enough durability" is sometimes... enough. When I switched to MongoDB from MySQL, performance rose 10+ times and I switched from "don't hit the db too much" to "give it more work, 'cause it's idle".

I know it's an ideology problem: otherwise, I can't see why people would complain about a tool they don't use.

the12• 13 years ago

It took minutes to mongoimport 10k small documents. import gets slower. not sure what's going on.

willvarfar2• 13 years ago

Varnish famously demonstrated how to use the kernel page cache effectively. MongoDB, though, is Squid-like. Its an interesting comparison.

Every single MongoDB step has had the old timers groaning.

Even with something solid like Tokutek's storage engine in it, its going to be a hard sell.

leif3• 13 years ago

I'm an engineer at Tokutek

I'm confused by your comment. The beginning acknowledges the fact that MongoDB has a weak storage engine, but your conclusion is that, even with a strong storage engine like ours, there is still a problem. What other problems do you see? Are they something we could work on?

jpgvm2• 13 years ago

This is going to come off as abit negative but I kinda feel it has to be said. I would first like to say I do love the Fractal tree indexing, very cool and could have alot more intesting usecases outside of databases (I'm thinking logical volume/block storage etc.. I'm always thinking in kernel land..)

The problem is that Mongo advertised itself as a database and wasn't one. Once you do that reputation of the product is dead forever.

TokuMX is a real database as far as I can see, MVCC, great indexing story etc.

By association TokuMX is probably not regarded as highly as it should be. Which is a shame but it's a people problem, not a technical one. People can very easily lose trust in a technology at which point it's effectively dead, it might take a long time to die due to lock-in but it's dead.

For instance I have recently started playing with RethinkDB over TokuMX almost purely because of Mongo association.

Now technically that might not sound like good reasoning but when you think about the kind of person that writes a database that doesn't fsync your writes by default and relies on the page-cache over doing direct I/O when building a database.. doesn't really inspire confidence in the network stack, the query planner.. or well anything.

If anything it makes you insistent on not having ANYTHING to do with that sort of codebase.

Just replacing the storage engine might actually be good enough, but restoring my trust in the rest of the codebase is almost a forgone conclusion at this point.

lafar6502• 13 years ago

Maybe MongoDB is clown's shoes, but so are 99% of all technology startups. They all fail before reaching limits of the database.

anatari• 13 years ago

Article is spot on about mongodb being ideal for online games. We use it as the main datastore for our latest game, and it has worked out very well for us. My main gripes with it has been key values taking up too much space and how difficult it is to shard. I think Rethink DB will be even better once that matures.

meowface3• 13 years ago

RethinkDB looks like a much better database than MongoDB.

Unfortunately though, I believe Mongo is still beating it at performance, which is the one thing keeping me away.

dodyg7• 13 years ago

Read performance or write?

nawitus2• 13 years ago

So, what's a good NoSQL database for e.g. node.js use? The only alternative I know of is CouchDB. (Yes, I should give more parameters about the intended use, but I really don't know any alternatives).

egeozcan• 13 years ago

RethinkDB[1], perhaps.

[1] http://www.rethinkdb.com/

bpicolo• 13 years ago

You know, if you like it with node, ignore the haters and use it anyway. It's not some atrocious mess. I love mongoose in addition to it, use it all the time.

Touche• 13 years ago

Couch is a great choice. So is Redis. I would go with Redis if you can afford the cost of ram and you know you'll always be looking stuff up by key. Otherwise I'd with Couch since the queries are just map/reduce functions and the api is just rest.

patrickod• 13 years ago

I'm unsure why you say there are no other alternatives. Surely the choice of database should preference your data and what you want to do with it over the language you'll use? There are node libraries for any database I've ever wanted to use (SQL and NoSQL alike)

nawitus2• 13 years ago

I tried to say 'no NoSQL alternatives'. And I noticed that while there's node middleware for most libraries, the ease of use is different for each database.

jackowayed• 13 years ago

Node.js isn't really a usecase that dictates what datastore you should use, it's just a certain way of writing a totally different piece of your app.

In most node.js apps, the best answer is probably a SQL database. Sorry.

If you're working with timelines or other cases where Redis's data models can help you, consider it, though beware that if your data is large, things will get more expensive fast since you're keeping everything in RAM.

HBase, Cassandra, and Riak are all reasonable in similar cases and have their own tradeoffs.

And yes, Couch fits a similar niche as Mongo. You might even be able to use something simpler like BerkeleyDB (quite mature) if you think you want a document store.

RethinkDB may be a nice choice too. It's fairly young but looks like it's going good places.

But your choice should be mostly dependent on what kind of data you're storing and what kind of guarantees and access models you need.

It should not be based on someone on HN telling you "Riak is the best NoSQL database for Node.js" because their idea of what most Node apps need may not be what yours needs.

nostrademons11• 13 years ago

What's wrong with the Viaweb/Arc/HackerNews/Mailinator approach of just using in-memory datastructures (hashtables, linked lists) and then journaling out changes to the filesystem as records that are read in on startup? It's incredibly simple and blindingly fast as long as you stay on one server, and you can get several thousand QPS of capacity on that one server (vs. like 10 with a Django/Rails + SQL database solution).

Another highly underrated solution is using MySQL/PostGres as a key-value store. Just create one table for each entity type, with the primary key as the key and a JSON or protobuf blob as the value. You're using completely battle-tested solutions, you've got bindings in basically every language, you're doing basically the same work (at the same speed) as your NoSQL solutions, but you have a lot more flexibility to add additional indices and can rely more on pre-existing functionality than a MongoDB or CouchDB solution.

rsynnott12• 13 years ago

What's wrong with the Viaweb/Arc/HackerNews/Mailinator approach of just using in-memory datastructures (hashtables, linked lists) and then journaling out changes to the filesystem as records that are read in on startup?

That works for some things. However, it's no more a foolproof magical solution than MySQL or MongoDB or Cassandra or Oracle or... It just has different tradeoffs (non-primary key queries will tend to be a problem, you'll have to make your own replication, sharding will be a problem, etc etc).

sahil_videology• 13 years ago

MySQL now supports the key-value use case via the memcached API since MySQL 5.6.

https://blogs.oracle.com/MySQL/entry/nosql_memcached_api_for...

philwelch4• 13 years ago

It's incredibly simple and blindingly fast as long as you stay on one server and you can get several thousand QPS of capacity on that one server (vs. like 10 with a Django/Rails + SQL database solution).

Wait, what? Even if vertical scaling was a good idea, scaling is far from the only reason you should have more than one server for anything serious.

byroot3• 13 years ago

What's wrong with the Viaweb/Arc/HackerNews/Mailinator approach of just using in-memory datastructures

https://news.ycombinator.com/x?fnid=cjVXpi8HxVR5TTze3bqSCa

  Unknown or expired link.

Oh I remember now...

rsynnott12• 13 years ago

So, what's a good NoSQL database for e.g. node.js use?

Well, this is the thing; 'NoSQL' is really a pretty unhelpful term. It tends to just mean "not relational", and covers a vast number of things.

So, for instance, you might be okay with having to have your data set fit in RAM (with MongoDB you'll suffer if it doesn't, anyway), and not care too much about availability. In that case, Redis might be good. Or maybe you care deeply about availability; in that case, one of the Dynamo paper databases might be good, if you're willing to put in the work dealing with the consistency issues. Or...

I could go on for a bit. 'NoSQL' is verging on a meaningless term.

yummyfajitas6• 13 years ago

One viable option:

    CREATE TABLE mongodb (
        key VARCHAR(256) PRIMARY KEY,
        value JSON
    );

mkhattab• 13 years ago

I think you're onto something here.

joevandyk2• 13 years ago

I'd use a uuid for the key.

Also, see https://github.com/umitanuki/mongres

Postgresql speaking the mongodb protocol.

meowface3• 13 years ago

Mongo (and other document stores) let you do pretty complex querying within JSON objects. Postgres unfortunately doesn't let you do so; that's really the only thing keeping me with Mongo at the moment.

Lazare2• 13 years ago

Well, the obvious answer to your question is: CouchDB! It's a brilliant, underrated database, and hey, it backs NPM!

On the other hand, Redis, Cassandra, Riak, and many more are also excellent NoSQL databases. But none of them, including CouchDB, are excellent at everything. What are you planning on making? You can write a lot of different things in node.js. If you're writing, say, a blogging engine you probably should look into flat files, or maybe Postgres, and forget the NoSQL kool-aid. :)

dodyg7• 13 years ago

Actually if you are making a blog engine or CMS, a document database is an obvious choice.

the12• 13 years ago

ejdb or solr.

mistercow• 13 years ago

But in that case, it also wouldn’t be crazy to pull a Viaweb and store it on the file system

I've done this before when I was doing work for a client using an existing simple web host with no built-in options for databases. It works well, and the nice part is that there's a simple, obvious way to do any query. The bad part is that anything other than a primary key lookup is slow unless you add a lot of complexity.

programminggeek2• 13 years ago

I've used MongoDB for various projects and found it nice to use. Lately though, I've found MySQL to be pretty enjoyable too, so honestly, what's all the fuss? It's a database.

Nobody writes about the filesystem like they do the database, and yet they do the same job - store and retrieve data.

PhilipA• 13 years ago

It depends on your needs, neither MySQL or MongoDB is the silver bullet.

brandonbloom2• 13 years ago

The fuss is that a "proper" database does so much more than store and retrieve data.

If you've only ever used Mongo, a filesystem, and/or mysql, then you've never really used a database. Postgres (and mssql, oracle, etc) are so much richer; they are so much more than storage systems. I'm not saying you no one should ever use Mongo or MySql, I'm just saying that they are generally far inferior choices for problems bigger than mere storage.

It's high time we started thinking about storage systems and the higher-level functionality of databases separately. We can make different, more informed, and generally better tradeoffs than we're currently making by viewing this broad category of software through such a foggy lens. For instance: Take a look at how Datomic utilize pluggable storage to provide a sensible information model, with raw index access and powerful, pluggable querying.

rsynnott12• 13 years ago

so honestly, what's all the fuss? It's a database.

Different types of databases are useful for different things.

Nobody writes about the filesystem like they do the database

You must have missed the last decade of people going on about ZFS.

programminggeek2• 13 years ago

You must have missed the last decade of people going on about ZFS.

I guess I did, what's the big deal about ZFS?

nbevans3• 13 years ago

I would put it out there by suggesting that you probably have low standards when it comes to databases. Have you used anything else other than MongoDB and MySQL? These are basically two of the worst NoSQL and SQL implementations in existence.

Recommendation: Stop following all the hype and what all the other blind sheep are doing.

Regarding filesystems... yes they do. Tons of information out there about filesystems you just have to look for it. Read up about ZFS, ReFS and that should get you started.

mcot2• 13 years ago

A lot of these downsides are fixed by Tokumx. Real transactions, document level locking, compression and disk optimized indexes. I suggest everyone take a look at it.

ericingram• 13 years ago

I agree. TokuMX has filled many holes in Mongo and (at least in my experience so far) performs very well. It's got great documentation, backed by a brilliant team. As a drop in replacement for mongo binaries, it's really easy to install and offers professional/enterprise support if you need it.

rgo• 13 years ago

Previous versions of my startup's enterprise product used to be based on relational DBs (mostly Oracle, MySQL also). This year we switched to Mongo and dropped RDBMS support.

RDBMS performance was fine most of the time as we're not doing big data really. Our problem was developing and maintaining a schema that holds lots of metadata many levels deep. Our app allows for unlimited user defined forms and fields, some of which may hold grids inside which hold some more fields... Our app also handles lots of logs and large file dumps, which slowly made data, cache and fulltext search management mission impossible. Even though we had considerable previous experience with Mongo, it took us a long time to switch because we were utterly scared. It's nice to sell a product that is Oracle-based, as that sent out a message about our "high-level of industry standardization and corporate commitment" bullshit that (we thought) is quite positive for a startup competing against the likes of IBM, HP, etc.

To our surprise, our customers (some Fortune 500 and the like) were VERY receptive to switch to a NoSQL, opensource database. Surprise specially given it would be supported by us instead of their dreadfully expensive and mostly useless DBA departments. It even came to a point where it has changed their perception of our product and our company as next generation, and surprisingly set us apart from our competition even further.

In short, as many people here know, not all MongoDB users are cool kids in startups that need to fend off HN front page peak traffic day in day out. Having a schemaless, easy to manage database is a step forward for sooo many use cases, from little intranet apps to log storage to some crazy homebrew queue-like thing. 10-gen superb, although criticized, "marketing effort" also helps a lot when you need to convince a customer's upper-management this is something they should trust and even invest on. I can't express my gratitude and appreciation for 10-gen's simultaneous interest in community building, flirting with corporate wigs and getting the word out to developers for every other language. Mongo is definitely a flawed product, but why should I care about the clownshoeness of its mmapped files when it has given us so much for so long?

oijaf888• 13 years ago

Perhaps your customers were receptive because you were supporting it and they were happy with the existing support you got and unhappy with their internal DBA department? Its possible that they didn't care about the technology and just realized that the support was going to be better.

andrewvc• 13 years ago

Well written post. Even as a detractor of mongo I'll agree that it works for your use case. But the key is "Our app allows for unlimited user defined forms and fields, some of which may hold grids". That really isn't a very common case. SQL is not great at representing large groups of documents without any common structure.

The vast majority of apps just don't deal with that problem. If MongoDB was really only used by people that its a good fit for (like yourself), it'd really be a niche product. They're marketing it as a general purpose product, which is why they've earned scorn from so many.

badclient• 13 years ago

Bingo. They should call themselves mongodocs, not mongodb. The way I see it mongodb sees widespread misunderstanding about its use cases and instead of make the use cases more clear, they seem to take an interest in seeing mongodb being used unnecessarily.

digitailor2• 13 years ago

Thanks for sharing your experience on this. We make a lot of custom enterprise intranet applications, and we've been considering adding MongoDB to our toolchest. My concern has been what you say- there will be customer resistance. That they'll have fear for the future of their DB, since it's not SQL. Based on your post, it seems like it may be my fear and not necessarily the clients'. I was very interested in NoSQL at the beginning and what that did is make me realize I need to up my SQL game. I want to make sure I don't move into NoSQL just because it's hip and relational DBs annoy me.

We are evaluating Mongo to be the persistence for single page web applications, which is how we'd like to start making the majority of our intranet/private enterprise jobs. Are you using it in this context and has it been helpful? Your example of nested fields being easier made me warm and fuzzy- we had a project last year that had growing, fluid, user-defined data structures. We made it work (well) with Postgres but there were several kludges that really bothered me. One of them was handling delete dependencies gracefully on user-defined, nested structures. Did you encounter this issue pre-Mongo as well and if so, did it help with it?

continuations• 13 years ago

Having a schemaless, easy to manage database is a step forward for sooo many use cases

Can you explain why can't you do schemaless with an RDBMS?

From what I understand MongoDB is schemaless by storing all fields as one single JSON document. So what stops you from doing the same in an RDBMS - have a catch-all field "JSON" and store all your data there?

aaronem2• 13 years ago

That gets you halfway there, but you still don't have the ability to query your datastore by structure, unless you've installed PostgreSQL 9.3 and are using its JSON field type, which does have that capability, thus entirely demolishing the NoSQL USP as far as I can determine.

Goopplesoft• 13 years ago

That'd be an unqueryable string field in most sql databases. Not at all the same thing.

bkanber7• 13 years ago

I posted this further down the thread, but I thought I'd share my thoughts on why I like mongo.

Most people don't like mongo because 10gen gives the impression that mongo is better than it actually is, many people feel that mongo is not reliable enough for at-scale applications. They're right; it's not. But that's ok, because:

Mongo's really great for rapid prototyping. You don't need to worry about updating the schema at the db level, it can store any type of document in any collection without complaining, it's really easy to install and configure, the query language is simple and only takes a couple of minutes to learn, it's pretty fast in most use cases, it's pretty safe in most use cases, and it's easy to create a replica set once your prototype gets usage and starts scaling.

Mongo does everything well up until you reach the level where you need heavy-hitting, at-scale, mission-critical performance and reliability. Most projects out there (99 in 100?) will never reach the level of scale that requires better tools than mongo. And since the rest of it is so easy to use, that makes mongo a great starting point for most projects. You can always switch databases later, but mongo gives you the flexibility to concentrate on more important things in the early stages of a project.

swanson• 13 years ago

Couldn't you argue that e.g. Postgres and ActiveRecord give you the same rapid prototyping ability but with an easier (and more established) path towards scalability? It is easy to change your schema with migrations at the beginning of a project - just go edit the original ones and nuke your database. And I don't have to worry about properly configuring write-locks, replica sets, or writing map reduce javascript.

threeseed4• 13 years ago

Configuring write-locks ? I guess you have never actually used MongoDB before which explains why absolutely none of what you said makes any sense. MongoDB is far easier to use, manage the schema with and scale than PostgreSQL.

bkanber7• 13 years ago

Of course you could argue that. But so what? Having an easier path towards scalability is nice, but irrelevant for the vast majority of projects; not every project is going to turn into a startup or a real product or even something you work on for more than a few weekends!

The last time you hacked together a blogging engine in Node.js one weekend, were you worried about future scalability, or just playing with new technologies because it's fun?

And while it's pretty easy to do schema migrations, it's not easier than _not_ doing them. And what is it you really want to worry about? Making sure your DB is production ready, or tinkering with Express.js and Backbone?

So because of that, many people use mongo as their de facto database. It's just what I use when I need a persistence layer for anything I build, because I already have a mongo db running for like 30 different defunct projects on my dev server.

And then, by happy accident, one of your side projects turns into a real product, and then mongo handles you really well for the first year or so, just up til the point of having to hire a real devops engineer; at which point you swap out your ORM layer and switch to postgres.

rapind• 13 years ago

Actually postgres can be a bit of a PITA, but so can mongo. At the risk of sounding reckless, unless the app needs to support high CUD throughput I sometimes opt for sqlite. Doesn't get much easier than that and it's read performance is impressive from what I've seen.

Even then you can sometimes get away with staying on sqlite for your admin side CRUD and redis for the heavy / concurrent writing from the public facing side (obviously situational).

danpalmer• 13 years ago

The case for using it as a prototyping database is the best use-case I've seen for Mongo, however I'm not sure it's always a good idea.

For a hack-weekend sort of project, fine, but if you are in any way attempting to make a product, it strikes me as the sort of thing that would be really difficult to change later down the line, and so worth investing the very little extra effort it takes to include your schema in the database, and use something like Postgres/MySQL/etc.

threeseed4• 13 years ago

I've never understand people that worry so much about the schema. It's like they've missed the last decade of computing. Everybody these days uses ORM. Which means that (a) data migration between databases is a relatively simple task and (b) schemas often just get in your way.

bkanber7• 13 years ago

I feel the opposite way. If your application's data layer is sensibly designed, it shouldn't be too bad to switch to postgres when you need to. It may be tedious if you have a large codebase, but it won't be difficult.

I think the up-front benefits of using mongo (especially as a sole developer/devops/sysadmin person) outweigh the difficulty of the changes you'll need to make later on, which will only happen as you hit scale and have more resources to nurture the devops side of the tech stack.

spamizbad• 13 years ago

It is really difficult to change down the line. My company tried and failed. Now we're stuck with MongoDB. Huge mistake, but a lesson was learned.

Edit: "tried and failed" in the political sense. You don't change horses in midstream etc.

nbevans3• 13 years ago

The problem with that argument is that prototyping rarely needs a database. Just store stuff in memory. Who the fk in the real world is seriously writing data-layer code for a prototype?

lebski88• 13 years ago

Mongo is also brilliant for internal tools that need to change rapidly but you absolutely know will never require significant scale. I've been badly burnt trying to use mongo on a large dataset but it's genuinely great for getting things done quickly.

I also used it to write a service that had to go from nothing to working in a couple of days. I then spent the next two days swapping it back out again. It was surprisingly painless to go from an object store to a relational model

bcoates• 13 years ago

I just use the filesystem for that sort of thing. Everyone justifies using MongoDB because its easy and general but compared to the tool and compatibility ecosystem around files it's awkward and primitive.

functional_test11• 13 years ago

The file system is great for some things, but, for example, how would you handle the same data to having multiple indices?

JulianMorrison4• 13 years ago

Have you considered RethinkDB? For most of the advantages of MongoDB that don't specifically come from mmap and overwrite-in-place, it's an equal or better.

ahoge2• 13 years ago

RethinkDB still doesn't support Windows. [1]

Most developers who use Windows will just go with something which doesn't require a virtual machine. For example, MongoDB, CouchDB, OrientDB, Cassandra, and ArrangoDB work fine everywhere.

[1] https://github.com/rethinkdb/rethinkdb/issues/1100

ozataman• 13 years ago

Application design for me almost always begins with data and data structures. Whether my database has an explicit schema or not, I always have one in mind, documented or otherwise reified in the table-data structures I have in my code. I just don't get why people would want a schema-free database that is in almost every way inferior to the rock-solid power beast that is Postgres. Just use a library with proper migration support so you can propagate changes to your schema rapidly during development. You'll thank us later when you learn a little bit of SQL and start analyzing your data, running circles around the no-sql guys.

Cassandra et. al. are completely different, in that you don't use them because they are more fun to use. You use them despite their awkward, low-level interfaces because you're going to dump billions of data cells into your database from day one with no end in sight and want all the easy scaling/availability features provided.

bkanber7• 13 years ago

Do you always start with the perfect data structure? I find myself adding, removing, and restructuring schema often. Just as you think it's silly to use an "inferior" db during prototyping, I think it's silly to have to jump through hoops -- even minor ones -- while I'm just trying to experiment with a new technology or play with a concept, product design, or pet project. 99 times out of 100, I don't care if my project survives the weekend. Let me use that database I want to use!

You'll thank us later when you learn a little bit of SQL and start analyzing your data, running circles around the no-sql guys.

That's a little condescending... do you know a single mongo user who doesn't have experience with SQL? Plus, I love the fact that I can literally run javascript against my database. Good for production? Certainly not. But that doesn't mean it's not fun or useful.

Not every project requires such rigor. If that's how you enjoy development, that's great! Very few of my projects put the db layer to the test, and so I'm happy with the balance that mongo gives me. I use it in about 4/5 of my experiments and side projects.

rsynnott12• 13 years ago

Mongo's really great for rapid prototyping.

Has any phrase ever struck more fear into the heart of a programmer? Rapid prototypes have a nasty tendency of accidentally becoming products...

You can always switch databases later

shudder

nostrademons11• 13 years ago

That's why you use a language or technology that's politically unfeasible for your rapid prototypes, like Clojure or Haskell, or...for that matter...MongoDB. ;-)

EGreg2• 13 years ago

I think they are upset with their marketing, like "web scale". Even the name "Mongo" is derived from "Humongous" -- but that's exactly the scale at which you'd switch away from Mongo.

Reminds me of this: http://www.youtube.com/watch?v=URJeuxI7kHo

Want NoSQL? Use RIAK!

rsynnott12• 13 years ago

Want NoSQL? Use RIAK!

See, you're just perpetuating The NoSQL Problem. :) Riak is well-suited to some tasks, but it is no more a magical fits-every-problem thing than MongoDB is.

sargun2• 13 years ago

Riak also has massive problems. Realistically, figure out your data that you want to stick in a database, why, and how you're going to query it, and then work from there.

lucian19003• 13 years ago

None of the things you listed are any harder with Postgres and SQLAlchemy. Learning to use MongoDB isn't exactly trivial anyway, so why choose the thing that is known to be broken at all, when it's neither easier nor faster?

rdtsc3• 13 years ago

The problem with MongoDB is their shadiness. The shipped with unacknowledged writes up until not too long ago. In other words you would write to it and there wouldn't be an ok or fail response, you'd just sort of hoped it would go in.

They fixed that problem but it was too late. In my eyes they proved they are not to be trusted with data.

Had they called themselves MangoCache or MongoProbabilisticStorage, fine, can silently drop writes, I don't care it is not database. But telling people they are a "database" and then tweaking their default to look good in stupid little benchmarks, and telling people they are webscale, sealed the deal for me. Never looking at that product again.

functional_test11• 13 years ago

I understand some of the reasons people didn't like Mongo, but this always vexed me. The default write level was very clearly documented and you could always change it as necessary. Surely it would be necessary to read the documentation of a database before rolling it out to production?

justinsb3• 13 years ago

Other databases are forgiving; they are configured "safe", even at the expense of speed. The intention is that you can deploy a small system immediately; if/as you grow you will see that the database is going too slowly. You can _then_ look at the performance/safety dials you can tune and choose appropriate trade-offs.

These are systems designed for the real world, where people don't read the manual until they have to.

When people assume MongoDB was similarly designed with their best interests in mind, that's when things go wrong.

rdtsc3• 13 years ago

Surely it would be necessary to read the documentation of a database before rolling it out to production?

You buy a car. It comes with brakes disabled because for whatever reasons that also lets it get to a higher top speed. You are expected to read you car owner manual and on page 54 you find that you have to hold "enable brakes" button under the console for 10 seconds to turn on your brakes. Would it vex you that people might be slightly critical of that car. Clearly they are silly for not reading their car manual until page 54.

That "feature" is not something that should be discovered by reading docs or when you get a crash and then load a backup from another week and still get a crash and then you start hitting your head on your desk.

Anything calling itself a "database" should not have shipped with those default settings _ever_. If they did they might have gotten away with it in my book by having a big flashing red warning on the front or download page. I don't remember one.

taspeotis• 13 years ago

I understand some of the reasons people didn't like Mongo, but this always vexed me. The default write level ... Surely it would be necessary to read the documentation

I don't have much sympathy for people who can't RTFM but storing data is kind of a thing for databases.

chaffneue• 13 years ago

And it's not even good or recommended as a cache at any kind of profile. So I guess their most valuable niche is low traffic/prototype sites with poor architecture discipline or genuinely unrelated data sets. There are so much better tools for caching (memcached/redis), durable persistent storage (postgres), session storage (memcached/redis/browser hybrids) and document storage (postgres). Mongo is just one of those brands that quickly solved a problem that most of the better technologies missed - the user interface.

tracker1• 13 years ago

I think being able to have flexible data storage with indexing is where they are better than most other options. There's something to be said for some of what they do offer. I was able to replace the search system for a site that used SQL to MongoDB, which often includes geolocation, it works fairly well, I had considered using a ElasticSearch, or something similar, Mongo was a better fit.

Today, I would be inclined to use PostgreSQL with JSON support, and some triggers to update an aggregate search table, or look more seriously towards RethinkDB.

With any NoSQL system you give up something.. you just need to be aware of what you are giving up, why and for what gains.

dsaber• 13 years ago

Note that even with the changed default to 'acknowledged', data is not guaranteed to have been written to the journal. So, there is still no full durability in writes (by default) and there is a chance data might be lost (e.g. a mongod instance crashes).

adron• 13 years ago

Mongo was sort of great at first, it's a not bad solution for low traffic sites. It's also great for prototyping. But when anything grows up into high availability, high traffic or anything high.

It's kind of funny, albeit interesting that it took this long for much of the industry to start knocking down the house of cards that is built around the database. However many of the databases that are coming out these days have a lot of the same cultural issue of "hide the issues" and "talk about the cool parts".

I'm expecting any minute now for the anti-schema-less pro-schema movement to rise up...

IMHO, it's all about what you're doing at the time, and making the right decision... albeit it helps if the decisions for a database isn't glossing over the issues as insignificant. :-/

btw - "MongoProbabilisticStorage" is a great name for a product!

dlau1• 13 years ago

Can someone comment on how mature rethinkdb is at the moment?

I'm considering moving away from MongoDB before I have to implement what seems to be an incredibly complicated architecture to get it to scale on the level tens/hundreds of millions of documents.

leif3• 13 years ago

RethinkDB is not yet "ready for production use" but reportedly will be soon.

If you're currently on MongoDB but need more performance, concurrency, or compression, please try TokuMX: http://www.tokutek.com/products/tokumx-for-mongodb It's a drop-in replacement server that uses a better storage engine but speaks the same protocol and query language.

dodyg7• 13 years ago

RethinkDB doesn't have Geospatial index yet (if it's relevant to your needs)

https://github.com/rethinkdb/rethinkdb/issues/1158

yeukhon• 13 years ago

I am so happy Postgres is adding support for JSON. This is a big change. The sole benefit of mongo to me is that you can be flexible with your schema at the beginning. But the consequences are

* you have to learn to do indexing right later (if you have to scale)

* failure and miss starting to occur (as you scale)

* more code to write to manage legacy schema and optional fields

The last is painful and ugly. Whereas if you start out with a good schema that last point is in a good hand. When you use SQL you always have the restraint that "xyz" attributes are repeating and you can just make a new relation, whereas with mongo you'd stuff 20 fields into a single collection. The refactoring is harder.

I will begin to migrate back to SQL for new projects.

Also ecosystem is richer in SQL. I have not seen a good ORM for Mongo. MongoEngine is fine but implementation + db have a lot of issues make that ORM a bit unusable from time to time. SQLAlchemy is good.

PS: For quick PoC and Hackathon projects sure prototyping with mongo is fine.

warmwaffles• 13 years ago

I have not seen a good ORM for Mongo

Uh, Mongoid is good. I have used it on past projects. It's nice. (mainly use some form of SQL now)

danbmil99• 13 years ago

I feel like the discussion of MongoDB is a bit like the discussion around the Affordable Care Act (aka "Obamacare"). The conversation always shifts between whether the very idea of a noSQL db is a good one, to the question of Mdb's implementation faults and (I guess?) its strengths.

Whether 10gen are vapid spin-meisters or not, even whether they have developed a usable product, seems orthogonal to the question as to whether a schemaless persistent storage layer might be a better fit for some projects than a relational database.

joeblau• 13 years ago

One thing I always do is try to scope a database for the problem. This site[1] has been a valuable resource for myself and colleague who were evaluation the best way to store/access our data depending on what we need back.

[1] http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

The genius and folly of MongoDB

Related Stories