MongoDB’s new query engine

https://laplab.me/posts/inside-new-query-engine-of-mongodb/

by subset • 3 years ago

135 115 3 years ago

laplab.me

romanovcode4• 3 years ago

Serious question: Why use MongoDB when Postgres supports indexed dynamic json?

dzogchen• 3 years ago

Why use a non-FOSS database at all?

romanovcode4• 3 years ago

Also, but main point for MongoDB 10 years ago was the JSON. But now?

malermeister• 3 years ago

Isn't mongodb open source?

https://github.com/mongodb/mongo

Edit: ah it's source-available, not open source.

JackMorgan• 3 years ago

My question exactly. No sense building a house on someone else's land. Too many projects I've worked on were shattered because of technology that stopped being offered or got too expensive to justify.

nemo44x4• 3 years ago

Because just about no one cares and I want to burn down a backlog quickly and often mongo enables that. Developer productivity is far more valuable that open source purity.

maxloh• 3 years ago

There is FerretDB. But they are not fully compatible to Mongo yet.

https://www.ferretdb.io/

jamil72• 3 years ago

If you're only storing and querying JSON, then it's cheaper to run DynamoDB instead of RDS.

salil9993• 3 years ago

This is completely false and has nothing to do with the article. DynamoDB and MongoDB might both serve JSON but they're used for wildly different things. DynamoDB is a key value store whereas MongoDB is much more than that.

threeseed4• 3 years ago

MongoDB:

a) has a proven, supported, easy-to-use horizontal scaling solution. PostgreSQL doesn't.

b) is ridiculously faster than PostgreSQL at per-tuple document updates.

c) has clients which are tailored for operations and data structures around documents.

d) is easier to install, configure and manage.

coldtea4• 3 years ago

I thought all those points died along with the hype ("MongoDB is web scale"), as all were wrong.

Those were the marketing points people mentioned in their "honeymoon phase" posts. Then after using it in production, actual benchmarks and comparisons coming in, came the regret and moving on posts. In fact, most of those mentioned moving from Mongo to Postgres, and there was a full blow "yeah, NoSQL was a dumb idea for 99% of use cases, and Mongo even more so" discussion.

In the end MongoDB was the butt of a joke, there were whole memes about it.

So this comment is like a trip down memory lane, or into an alternate universe, where it's like 2012 and these things never happened.

sambeau2• 3 years ago

Thanks. Just what I was looking for.

riku_iki6• 3 years ago

MongoDB:

does mongodb support ACID transactions? I think this is one of the key question in making decision.

varelaz2• 3 years ago

PostgreSQL will be horizontaly scaling if you can avoid joins and index range locks. But the thing is you don't want to, or why do you need SQL database then. I found MongoDB good for cases when you need a lot of upserts, your data model is looks like document and you don't need joins. Like you store events data for further processing or collect stats or counters with tags/indexes.

okr• 3 years ago

e) MongoDb is web-scale.

;->

(personally i like the schema-lessness)

frankjr• 3 years ago

I gave my answer a few years ago. The summary is that its query and update operations over documents are miles ahead of what PostgreSQL has to offer.

https://news.ycombinator.com/item?id=23271085

throwaway2990• 3 years ago

3 year… almost 4 years ago you commented about a few years ago. So not really relevant.

For the idiots downvoting.

2018 was version 11. Right when PostgreSQL JSONB support was beginning to get good.

nemo44x4• 3 years ago

Because the developer productivity is great because the language drivers that interface with the database are seamless with your code. MongoDB lets you burn down a backlog better than any other database. It’s not right for every problem but when it is right it’s the best tool because the language drivers are so good. That’s their secret sauce.

dubcanada• 3 years ago

Why is there no conversation on the topic at hand and instead conversation on why use MongoDB.

This authors post has nothing to do with pro/cons of mongodb, and is entirely around a new query engine and computer science. And instead we've devolved into a discussion on the pros and cons of PostgreSQL vs MongoDB.

christophilus• 3 years ago

Same as it ever was. Someone posts a new JS framework, and the top 50 comments are about how bad JS is, and why don't we do RUST with WASM instead? Someone posts about a new Go release, and the top 50 comments are about how terrible Go is, and why don't we use a real language instead? Mongo... same thing.

I get it. People don't like certain tools. But it would be nice for the rest of us to be able to actually discuss the topic at hand without all of the tangential critical noise.

sambeau2• 3 years ago

It's completely understandable. This is a significant and interesting upgrade to MongoDB that naturally gets people interested in a, "Hmmm… maybe I should take another look at MongoDB" way. Before wasting time, it seems sensible to use this forum full of experts to ask whether it would be worthwhile, and what the advantages or disadvantages would be. I didn't sense any negativity in the question, and it was the exact question I had in my head too. After skim-reading the article I went straight to scrolling down the comments to look for this exact question.

hot_gril5• 3 years ago

The original question was fine, asking what's the reason to use MongoDB. Like, it got a new query engine, please convince me to try using it / how does the new engine compare to Postgres.

The sub-questions about FOSS and font choice are annoying, in fact I removed my replies on them because I didn't want to feed the problem.

h1fra• 3 years ago

I'm a postgres fan but I have to admit the JSON syntax is not the best and not well supported by all the tooling around this database. Doesn't mean it's bad but the DX could be better.

Other than that, I have never seen a proper justification to use schema less db, good for prototyping and get started, not incredible as a long-term solution.

synthmeat2• 3 years ago

Schema-less is a bit wrong term, since you always end up with one but, indeed, schema-on-read vs schema-on-write discussion for individual use cases is far from a settled thing, even though the zeitgeist is that schema-on-write won sometime in the 70s.

You say "prototyping", but I would generalize that to "faster evolving" in the long term. Of course, not without tradeoffs.

I personally have zero issues writing a custom marshaller when needed for any the schema-less document collections I have. Constraints your application has on the data are a superset of db schema anyways.

synthmeat2• 3 years ago

Just to add to sibling comments, one killer feature for me is ChangeStreams[1]. It's miles ahead of what Postgres[2] offers, and it enables really interesting use cases. Some of my services built around ChangeStreams end up not doing a single query to the DB. Data is right there in the program memory, indexed how I need it to be, and program is immediately reactive to any changes in the DB.

[1] https://www.mongodb.com/docs/manual/changeStreams/

[2] https://blog.sequin.io/all-the-ways-to-capture-changes-in-po...

Lolol00• 3 years ago

I remember adding changestreams a few years back to my project, they were amazing .

splix• 3 years ago

MongoDB is much easier to setup as a HA cluster

heipei• 3 years ago

Serious counter-question: How do you run Postgres in a native and hands-off replicated failover setup? With MongoDB you create a three-node replica set and you're done. I've yet to see a simple guide for Postgres. But I might be wrong, so happy to hear how you would achieve the same with Postgres.

Too2• 3 years ago

This isn't just a server property. It need to be implemented across the whole vertical, including client libraries and how queries are constructed in the application. With mongo it all is, and it's there out of the box.

Client libraries are made to do this failover and retry under the hood. Queries are more often made as upserts instead of assuming transactions, eg ObjectId rather than auto increment and so on. Even non upserting operations like $inc are safely retried if handled by queries and not application code.

It's magic to see it happen, with no extra code, the application just keeps running even when one server is down. Starting it up again and it reconciles with no hands-on.

I would love to see a postgres that does this, because relational data is often more useful than the document model.

masukomi• 3 years ago

Dear Geeks with interesting things to say on your blog:

Please, for sake of the people you want to hear your words. DO NOT USE A SANS SERIF MONOSPACED FONT FOR ANYTHING BUT CODE.

I get it. We stare at text (code) formatted in this kind of font all day long, and many of us find fonts that we truly enjoy. But, most of our monospace sans-serif fonts are designed to make sure individual characters aren't misread. We don't read prose the same way we read code. There is far more pattern recognition going on than actual parsing of individual letters, and monospaced fonts break that. We can debate the aesthetics of serifs but they actually do help provide context clues to the pattern recognition systems in our brain.

Convincing you all to start using serif fonts on prose is not a battle i'm likely to win, but maybe I can convince you to only use monospaced fonts for your terminal, and your code.

Please.

colesantiago• 3 years ago

Just use reader mode?

All modern browsers have this.

weinzierl• 3 years ago

You exaggerate the issue. Proportional fonts were a necessity when lead, ink and paper were expensive. In lead typesetting they are easy to do. This caused the success of newspaper fonts like Times New Roman.

Contemporary fonts have much less variance in the width of characters, except for a couple of outliers like the i. From there to a completely monospaced font is not as big a leap as you make it seem. For me, I'm fine with monospaced fonts for prose.

TylerE• 3 years ago

Disagree. Reading monospaced prose sucks big time. For those of us with a little bit of vision deficiency the vary character widths are helpful.

Also, the reduced character width (as a monospaced font inevitable has to be spaced at what the widest chatacters, such as w, require, means that you have to use a smaller font size to get the same amount of info per scroll/line/page whatever, again compromising readability.

Here's a comparison between the posted site, and the same site with the font size bumped up 4px, proportional (just browser default, I didn't cherry pick) and the ridiculously wide line spacing reduced.

Even with all that, the easier to read version is quite a bit more compact. Could likely bump the font size 2 more px and still be smaller.

https://i.imgur.com/DBFI2RU.png

ltbarcly33• 3 years ago

I'm surprised when people notice fonts. If the font is weird, or very decorative then sure, I would notice a font in Papyrus or Comic Sans. On my kindle I set it to the modern sans sarif font because its easier to read, for me. If there was no option to change the font I probably wouldn't have noticed, though.

I don't want say this in a way that comes off as insulting, but I'll just say it and please don't take it as a put down: if my kid came to me and expressed this much frustration and difficulty because of a sans sarif or monospaced font I would be concerned they had a problem with their vision or an issue processing what they saw, and I might research and/or take them to get checked out.

wheels3• 3 years ago

I wouldn't be bothered about it in this case (I'm much more offended when the font size or margins are stupid), but to use your phrase "I don't want say this in a way that comes off as insulting, but I'll just say it and please don't take it as a put down..."

You probably just don't have a strong sense of aesthetics. Some people notice design-y stuff and some people don't. A lot of people do notice fonts, or misaligned margins, or whatever. And a lot of people don't. The latter category are probably never going to be good designers or artists or whatever. People are different.

glitcher• 3 years ago

Sentences in all caps are much worse IMO.

salil9993• 3 years ago

This is great and all but I'm curious on the performance improvements. I'm surprised there are no graphs or charts when show improvement in latency, CPU usage, disk reads, etc

tjpnz• 3 years ago

Does MongoDB still lose data?

salil9993• 3 years ago

Providing some context would be nice

endisneigh4• 3 years ago

Our most recent report on MongoDB 3.6.4 focused on causal consistency and linearizability in sharded collections. We found that sharded clusters appeared to offer linearizable reads, writes, and compare-and-set operations against single documents, so long as users ran with read concern linearizable and write concern majority. However, any weaker level of write concern resulted in the loss of committed writes. MongoDB’s default level of write concern was (and remains) acknowledgement by a single node, which means MongoDB may lose data by default. Although the write concern documentation does not make this clear, the rollback documentation states:

https://jepsen.io/analyses/mongodb-4.2.6

jabradoodle2• 3 years ago

Your comment is ambigous.

Since 5.0 the default write concern is majority, if that's what you were referring to.

cj• 3 years ago

No [0]

[0] https://www.mongodb.com/docs/manual/reference/write-concern/...

nevi-me• 3 years ago

Heh, I missed that there's a new query engine.

From [0] it looks like it was available from 5.1, also interesting is that you can't choose which engine to use, so I suppose it only works on subsets or queries that meet certain conditions.

This isn't the first time I hear something to the effect of "LLVM JIT is great, but it introduces a lot of query latency". I wonder if there are other JIT engines more suitable for compiling potentially small/simple queries.

[0] https://www.mongodb.com/docs/v7.0/reference/sbe/#std-label-s...

__s• 3 years ago

re LLVM: Yes, Cranelift was designed to address these issues

https://github.com/bytecodealliance/wasmtime/blob/main/crane...

https://blog.benj.me/2021/02/17/cranelift-codegen-primer

https://jason-williams.co.uk/posts/a-possible-new-backend-fo...

https://news.ycombinator.com/item?id=25130528 30% faster rust build time

jandrewrogers• 3 years ago

LLVM has many advantages for JIT but it does not prioritize very low latency, choosing to optimize other properties instead. Consequently, LLVM tends to be more popular for systems targeted at analytical workloads where queries commonly have intrinsically high overhead latency. The latency isn't terrible but it is noticeable if you are running a low-latency "fast-twitch" workload.

There are other specialized JIT compilers that are essentially purpose-built to provide very low latency; for systems like MongoDB which are rarely used for serious analytical processing, you'd probably want to use one of these instead.

matthewcford• 3 years ago

can it sort by id?

maxbond3• 3 years ago

I'm sure it can[1], do you mean can you get documents in insertion order? The default _id is a random right-leaning identifier[2], so sorting by it would only get you an approximate insertion order (except in a deployment with a single server). However as the linked documentation demonstrates you could supply a strictly incrementing ID, if your architecture abides.

[1] https://www.mongodb.com/docs/manual/reference/operator/aggre...

[2] https://www.mongodb.com/docs/manual/reference/method/ObjectI...

winrid4• 3 years ago

no, that means you can sort on a max of 32 fields per sort request, like fieldOne, fieldTwo, fieldThree...

There's no limit on number of sortable documents. Use an index to sort and it'll also be fast and use no extra memory. If you want to do an in memory sort there are memory limits, but you can also tell it to overflow to disk.

bit_flipper2• 3 years ago

This article seems to have inspired others to look at MongoDB again, so I'll give my thoughts after using it recently.

MongoDB Atlas is a surprisingly good managed database product. I'm not a huge fan of someone else running my databases, but I think it might be the best one you can run across any cloud. If you like MongoDB (and, ignore the memes, there is a lot to like nowadays), and are OK paying a bit more to have someone run your database, I'd strongly consider Atlas.

winrid4• 3 years ago

The problem is when you grow. They can be really tough to work with on pricing. Also, their licensing does not allow servers past a certain size. Can you imagine Oracle telling the CIA they can't use servers with more than 256gb of ram? Just silly.

bit_flipper2• 3 years ago

I'm not sure what experience you have, but I've run both their Enterprise licensed database on prem as well as migrated to Atlas and there have never been any licensing issues preventing vertical scaling of databases. One of our clusters on Atlas right now has machines larger than 256GB of RAM -- you're more limited by what your cloud vendor has available than Atlas.

avereveard• 3 years ago

Yeah atlas with federated queries and what not makes thinking about the entirety of an application storage layer a breeze. And gpt is even better at generating mongo queries than it is at sql, which is a nice unexpected facilitation in day to day usage.

wackget• 3 years ago

No support for collation is an enormous dealbreaker. It means you can't have case-insensitive searches or keys (e.g. foo@bar.com is treated differently than Foo@bar.com). You also can't rename databases.

The query syntax is also a massive pain IMHO. Especially when you get into nesting expressions. E.g. I can never remember if it's `{$id: $regex{'/pattern/'}}` or `{$id: ${regex: '/pattern/'}}` or `{$id: $regex{ pattern: '/pattern/' }}` or something totally different.

SQL is still superior.

willio582• 3 years ago

It means you can't have case-insensitive searches or keys (e.g. foo@bar.com is treated differently than Foo@bar.com). You also can't rename databases.

Can't you just query for a lowercase version of the input and sanitize data going into the DB so it's only lowercase? I'm not a mongo user but that doesn't seem like a dealbreaker to me

oron• 3 years ago

I love Postgres and use it for different projects but for inboxes.com which has a very high insert rate coupled with auto delete by time stamp Mongo up until now has been very kind to me. We sometimes have 1000 incoming emails per second and high usage of our API and it just works.

pphysch• 3 years ago

Did you actually migrate from PostgreSQL to MongoDB for inboxes.com prod?

MongoDB’s new query engine

Related Stories