The four horsemen behind Postgres outages

https://malisper.me/the-four-horsemen-behind-thousands-of-postgres-outages/

40 17 13 days ago

If you understand Postgres's problems so well (though in reality, such problems can occur in many applications, and it's unclear how your rewritten version attempts to solve them), then why not address them in the main branch by becoming a contributor. Less code and more value.

zaik• 10 days ago

It is explained in the article: Any of those changes require massive or incompatible changes.

hoppp2• 10 days ago

So they want 100% compatibility first and then implement features that require massive and incompatible changes?

dmitrijbelikov3• 10 days ago

No, this note is another declaration without exact explanation. If you have real bugs you should send a bug report, that's all.

Maybe you should start from bug report without LLM-slop?

studmuffin650• 10 days ago

I think the issue isn’t a lack of want for fixing these issues, but the inertia and friction of getting them fixed.

The issues around the transaction ids and process per connection are well known, but the changes to the codebase to fix them would either constitute a backwards incompatible change that would change storage needs or an incredibly large rewrite of the codebase that breaks with decades of assumptions.

The json issue is a lot less of a problem as that’s net new. But some of these changes have been debated for years with no movement (and no lack of willing developers to tackle it) and at some point a fork or rewrite like this will happen. In my mind, all LLMs have done is made this work easier to do. If you have reservations about LLM’s doing this kind of work, no one is forcing you to use it, and I think it shows the utility of LLMs in that these kinds of things now can exist.

dmitrijbelikov3• 10 days ago

TID (XID) - VACUUM, AVTOVACUUM (configuration), process per connection - pgbouncer, odyssey

bananamogul• 10 days ago

This article states multiple times that spinning up a process is expensive/very expensive. Is that really true? I ask out of ignorance.

“Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.”

“Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.”

Also:

“There’s been years of people talking about switching Postgres from a process model to a threading model, but nothing concrete has come out of that.”

I’ve read several times that on Linux, the cost between a process and a thread is relatively small.

dudul• 10 days ago

A process is significantly more expensive than a thread on a Linux system. The main cost is memory. Processes own their memory which forces the system to duplicate resources, as opposed to threads that are able to share resources.

Where have you red that "the cost between a process and a thread is relatively small."? I would be curious to see a link because the most cursory internet search would show you that it is not true.

guenthert• 10 days ago

"significantly more expensive" and "relatively small"

Well, both are true of course. You will have to put numbers on those to make sensible decisions.

Historically, creating processes in Linux has been cheaper than in other Unix-like OS (you might find results of an old Byte benchmark) and much cheaper than in Windows. Creating a thread will be undoubtedly cheaper still, but whether that will change the user experience or is worth the cost depends on details (how large the memory footprint is, pagesize, how many files are open, how many threads are to be executed, their lifetime, how many execution units are available, etc.).

wbkang• 10 days ago

It’s all relative. It’s cheap if you think about spinning up new processes to run one off commands. It gets very expensive when you have to spawn it hundreds of times per second, and it can waste cpu cycles that could be used for query processing instead.

titanomachy• 10 days ago

It’s also important to distinguish between an os thread and a userspace thread. The author said “thread” without qualification, so I don’t know which he meant. Userspace threads can be many times lighter than processes.

ubercore• 10 days ago

I think tokio does userspace threads running on a thread per core. At least by default.

spongebobstoes• 10 days ago

the ability to even attempt this ambitious project is what free software is all about

The four horsemen behind Postgres outages

Related Stories