computerfan494

By "patch" I am talking about the public commit. Updated binaries were made available when the CVE was published.

That's a good question. I suppose that posting the commit makes it incredibly obvious how to exploit the issue, so maybe they wanted to wait a little bit longer for their on-prem users who were slow to patch?

The author of this post is incorrect about the timeline. Our Atlas clusters were upgraded days before the CVE was announced.

If this is a result of that vulnerability, Ubisoft only have themselves to blame. Our support contacts ensured that we knew about the vulnerability as early as possible and gave us a clear guide to remediation for our self-hosted clusters. Our Atlas clusters were automatically patched before this was announced publicly. You'd have to be running your database open to the internet (already a mistake), ignore the advice to simply turn off zlib, and ignore the fixed versions that have been available for over a week.

If you're going to be in the business of running your own critical infrastructure, you better have spent a lot of effort planning for these situations, because they are inevitable. Otherwise, it's easier to just pay a vendor to do it for you.

We received communication that all Atlas clusters were upgraded with the fix before the vulnerability was announced.

Postgres violated serializability on a single node for a considerable amount of time [1] and used fsync incorrectly for 20 years [2]. I personally witnessed lost data on Postgres because of the fsync issue.

Database engineering is very hard. MongoDB has had both poor defaults as well as bugs in the past. It will certainly have durability bugs in the future, just like Postgres and all other serious databases. I'm not sure that Postgres' durability stacks up especially well with modern MongoDB.

[1] https://jepsen.io/analyses/postgresql-12.3

[2] https://archive.fosdem.org/2019/schedule/event/postgresql_fs...

Yes, and my point is that this customer switching to running their own MongoDB instances on EC2 like Atlas does would reduce the bill by less than 50% because the rates that they are charging mean that their cut is less than what AWS is getting from this customer.

You can look at this particular bill and observe that more than 50% of the cost was going to AWS.

There's definitely MongoDB markup, but a full 33% of their bill was AWS networking costs that have nothing to do with Atlas.

Most of the cost in their bill wasn't from MongoDB, it was cost passed on from AWS

I can assure you that Stripe does not regret the decision.

Maybe it's only me, but I just don't write that much code. I try to change less than 100ish lines per day. I try to keep codebases small. I don't want to run a codebase with hundreds of thousands of lines of code in a production environment.

I'm not planning to because there is no documented protocol. If there were, I might! As a result, I can't use FoundationDB.

The hard part is that there is no client spec you can follow as a third-party. Everything is implementation-defined. If you're out-of-tree, your code can break at any time. If the FoundationDB project committed to a protocol, client authors could write libraries outside of the main project.

FoundationDB is very cool, but I wish it didn't require linking in their C library to talk to it. The client story is not good.

MongoDB is a very good database, and these days at scale I am significantly more confident in its correctness guarantees than any of the half-baked Postgres horizontal scaling solutions. I have run both databases at seven figure a month spend scale, and I would not choose off-the-shelf Postgres for this task again.

The reason to change is scale. If your Postgres spend is in the six digits a month range, JSONB is probably very painful for you. It does not perform very well. Additionally, you've probably spent a good amount of engineering time painstakingly sharding your database at the application level and hoping you've done it correctly. MongoDB solves these problems out of the box, among others. I have run both of these above 1M a month in spend and I would certainly choose MongoDB for this over Postgres. If you plan on never spending a lot on your database, I do like Postgres a lot, still.

I will copy and paste a comment I wrote here previously:

"MongoDB ships with horizontal sharding out-of-the-box, has idiomatic and well-maintained drivers for pretty much every language you could want (no C library re-use), is reasonably vendor-neutral and can be run locally, and the data modeling it encourages is both preferential for some people as well as pushes users to avoid patterns that don't scale very well with other models. Whether these things are important to you is a different question, but there is a lot to like that alternatives may not have answers for. If you currently or plan on spending > 10K per month on your database, I think MongoDB is one of the strongest choices out there."

I have also run Postgres at very large scale. Postgres' JSONB has some serious performance drawbacks that don't matter if you don't plan on spending a lot of money to run your database, but MongoDB does solve those problems. This new documentdb extension from Microsoft may solve some of the pain, but this is some very rough code if you browse around, and Postgres extensions are quite painful to use over the long term.

The reality is that it is not possible to run vanilla Postgres at scale. It's possible to fix its issues with third party solutions or cobbling together your own setup, but it takes a lot of effort and knowledge to ensure you've done things correctly. It's true that many people never reach that scale, but if you do, you're willing to spend a lot of money on something that works well.

I have been using it in production for more than a year. There were some early hiccups associated with my use-case that were resolved on upgrade and with a bit of tuning. Otherwise it has been very stable. I'll say that I wish the official language bindings were tested more like rqlited itself. They've been more problematic than the database.

MongoDB ships with horizontal sharding out-of-the-box, has idiomatic and well-maintained drivers for pretty much every language you could want (no C library re-use), is reasonably vendor-neutral and can be run locally, and the data modeling it encourages is both preferential for some people as well as pushes users to avoid patterns that don't scale very well with other models. Whether these things are important to you is a different question, but there is a lot to like that alternatives may not have answers for. If you currently or plan on spending > 10K per month on your database, I think MongoDB is one of the strongest choices out there.

Also want to add that you can definitely use MongoDB (or any other database) in a way that doesn't scale well. I have personally run MongoDB at petabyte scale and had a relatively great experience.

My understanding of what the OP was discussing is an optimistic locking system where the nodes only accept commits if the last issued token matches the token included in the commit. While agreeing on the last token requested requires coordination, unlike monotonically increasing tokens you could have well-behaved clients generate token content themselves without coordination. That may or may not be useful as a property.

I feel like you're dancing around admitting the core issue that Martin points out - RedLock is not suitable for systems where correctness is paramount. It can get close, but it is not robust in all cases.

If you want to say "RedLock is correct a very high percentage of the time when lease timeouts are tuned for the workload", I would agree with you actually. I even possibly agree with the statements "most systems can tolerate unlikely correctness failures due to RedLock lease violations. Manual intervention is fine in those cases. RedLock may allow fast iteration times and is worth this cost". I just think it's important to be crystal clear on the guarantees RedLock provides.

I first read Martin's blog post and your response years ago when I worked at a company that was using RedLock despite it not being an appropriate tool. We had an outage caused by overlapping leases because the original implementor of the system didn't understand what Martin has pointed out from the RedLock documentation alone.

I've been a happy Redis user and fan of your work outside of this poor experience with RedLock, by the way. I greatly appreciate the hard work that has gone into making it a fantastic database.

Locking without a timeout is indeed in the majority of use-cases a non-starter, we are agreed there.

The critical point that users must understand is that it is impossible to guarantee that the RedLock client never holds its lease longer than the timeout. Compounding this problem is that the longer you make your timeout to minimize the likelihood of this from accidentally happening, the less responsive your system becomes during genuine client misbehaviour.

In the above description of optimistic locking, it is assumed that it is impossible to issue the same token to multiple clients. Nodes can agree that a given token has also never been issued before just like a monotonically increasing value. The nice property about non-monitonically-increasing tokens is that nodes may generate them without coordinating if you can make other assumptions about that system. A good example is when nodes use an ID they were assigned beforehand as part of the token generation, guaranteeing that the leasing tokens they mint will not conflict with other nodes' as long as node IDs are not reused.

To be honest I've long been puzzled by your response blog post. Maybe the following question can help achieve common ground:

Would you use RedLock in a situation where the timeout is fairly short (1-2 seconds maybe), the work done usually takes ~90% of that timeout, and the work you do while holding a RedLock lock MUST NOT be done concurrently with another lock holder?

I think the correct answer here is always "No" because the risk of the lease sometimes expiring before the client has finished its work is very high. You must alter your work to be idempotent because RedLock cannot guarantee mutual exclusion under all circumstances. Optimistic locking is a good way to implement this type of thing while the work done is idempotent.

I don't like Taylor Swift's music, but I do hope this author experiences joy in something, sometime.

I'm not sure how to convince you to get over your hangup around how other people feel about the word "delivery", but this is exactly why others distinguish between "delivery" and "processing". If you think there are better terms than those two to describe "the system that receives a message and must be responsible for de-duplication" and "the system that can rely on messages being already de-duplicated", then feel free to propose them and have people debate, that I suppose. But because of what I noted earlier, this is a very useful distinction to maintain for people working on systems that are responsible for de-duplication (likely most people on this forum), and these words seem to make sense for most individuals.

It explains why most people seem to disagree with you on what "delivery" and "you" mean in this context. For the majority of contexts, "delivery" means that a system responsible for de-duplication receives the message.

You're correct, but in my experience the vast majority of code written is not downstream of a usable de-duplication mechanism.

At the end of the day the author and those they are arguing with mostly agree, they simply disagree on what the word "delivery" means. Given the author's background, I wonder if the issue is that they're focused mainly on lower levels of the stack, while those who disagree mainly work with traditional applications that do things like send email, respond to webhooks, update databases, etc.

The reason I think it's important to be pedantic about distinguishing between "delivery" and "processing" is that I have seen plenty of higher level systems that have incorrectly not implemented idempotency and had bugs as a result. I have seen many folks be confused by Kafka's "Exactly-Once Semantics" feature and introduce major bugs into message processing pipelines. The author, who clearly understands these fundamental design challenges, is not my problem. It's everyone else who struggles to design safe, idempotent exactly-once systems.