JoelJacobson

Here is the full pgsql-hackers mailing list thread where you can follow our work from initial idea to commit: https://www.postgresql.org/message-id/flat/9d8a4a42-c354-41f...

Back in 2024, I was trying to optimize PostgreSQL's NUMERIC data type, which is base-10000, using Karatsuba. The problem of finding the optimal threshold of when to switch to Karatsuba turned out to be really hard, since it depends on the size of both factors combined. After some hundreds of hours, I gave up, and started thinking about if there could be a simpler solution. I came to think about another idea I'd had before but abandoned, about 64-bit modernizing the digit base from 10k to 100M, but that would be a challenge due to existing data on disk. Desperate of finding a solution, I wondered if it could be fast enough to do on-the-fly conversion back and forth between base-10k and base-100M, and then realized that, yes, of course, it will be fast already for quite small N (testing shows already between 3-6 base digits). The trick basically reduced the N in O(N^2) into half, i.e. O((N/2)^2), with some O(2*N) cost for the conversion back and forth.

I had a lot of fun hacking on this idea together with the maintainer of the NUMERIC data type, and after two months the patch finally was ready and got committed:

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

I was curious what the claim

"10-100x uplift in terms of speed compared to Postgres on things like regexp_matches()"

was about, so I checked, and DuckDB's regexp_matches() is not the same as PostgreSQL's regexp_matches(). DuckDB's version "Returns true if string contains the regexp pattern, false otherwise." [1] while PostgreSQL's "returns a set of text arrays of matching substring(s)" [2].

I think the closest think in PostgreSQL to DuckDB's regexp_matches() is `string ~ pattern` or `regexp_like(string, pattern)`.

[1] https://duckdb.org/docs/lts/sql/functions/regular_expression... [2] https://www.postgresql.org/docs/current/functions-matching.h...

Sorry, should have emphasized that it was the "much easier" part I didn't agree with in that interview.

This article made me think of a strange claim by Elon Musk at 07:08 in this [1] interview:

"Cooling is actually much easier in space than it is on earth. You can just radiate to the vacuum."

I don't think that follows. The radiator is only the final heat sink. You still need to move heat from very dense chips into a deployable, space-rated radiator, and handle pumps, loops, leaks, redundancy, radiation damage, replacement, eclipses, Earth IR/albedo, and launch mass.

[1] https://youtu.be/D_1j5dVWNYI?si=R77VeVKlRXRhaBk5&t=428

I think we can't rule out the explanation that all the ideas of space data centers could be connected to a desire by some of finding additional applications for rockets that can transport stuff to space.

Here is a tl;dr as well: https://keyjoin.org/tldr.html

Shame on The Netherlands: ~89% of homes still use natural gas in some way for heating [1], and their government are now "scrapping the obligation to purchase a heat pump in 2026" [2].

[1] https://www.cbs.nl/en-gb/news/2025/50/ever-more-gas-free-hom... [2] https://www.abnamro.nl/en/personal/specially-for/preferred-b...

Shameless plug: In the upcoming release of PostgreSQL 19, LISTEN/NOTIFY has been optimized to scale much better with selective signaling, i.e. when lots of backends are listening on different channels, patch: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

I agree! It should be very stable, IMO. If not, then please send a bug report and we'll look into it. Also, now it scales well with the number of listening connections (given clients listen on unique channel names): https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

Rust without async maybe?

What a really like about concurrent(), is that it improves readability and expressiveness, making it clear when writing and reading that "this code MUST run in parallel".

It's a common misconception that the single queue is a poor design choice. The user reports, of seeing notifications/second severely degrade with lots of backends, cannot be explained by the single-queue design. An efficient implementation of a single-queue, should flatten out as parallellism increases, not degrade and go towards zero.

In the linked "Optimize LISTEN/NOTIFY" pgsql-hackers, I've shared a lot of benchmark results for different workloads, which also include results on how PostgreSQL currently works (this is "master" in the benchmark results), that can help you better understand the expectations for different workloads.

The work-around solution we used at Trustly (a company I co-founded), is a component named `allas` that a colleague of mine at that time, Marko Tikkaja, created to solve our problems, that massively reduced the load on our servers. Marko has open sourced and published this work here: https://github.com/johto/allas

Basically, `allas` opens up a single connection to PostgreSQL, on which it LISTEN on all the channels it needs to listen on. Then clients connect to `allas` over the PostgreSQL protocol, so it's basically faking a PostgreSQL server, and when clients do LISTEN on a channel with allas, allas will then LISTEN on that channel on the real PostgreSQL server on the single connection it needs. Thanks to `allas` being implemented in Go, using Go's efficient goroutines for concurrency, it efficiently scales with lots and lots of connections. I'm not a Go-expert myself, but I've understood Go is quite well suited for this type of application.

This component is still being used at Trustly, and is battle-tested and production grade.

That said, it would of course be much better to avoid the need for a separate component, and fix the scalability issues in core PostgreSQL, so that's what I'm currently working on.

Thanks for the report. For that use-case (if you have a single application using a single connection with a LISTEN) then it's expected that is should perform well, since then there is only a single backend which will be context-switched to when each NOTIFY signals it.

Here is the Commitfest entry if you want to help with reviewing/development/testing of the patch: https://commitfest.postgresql.org/patch/6078/

It works, but suddenly your query times explode! Instead of doing 1 million transactions per second* you can now do only 3 (*These numbers were exaggerated for dramatic effect)

In general, a single-queue design doesn’t make throughput collapse when you add more parallelism; it just gives you a fixed ceiling. With a well-designed queue, throughput goes up with concurrency, then flattens when the serialized section (the queue) saturates, maybe sagging a bit from context switching.

If instead you see performance severely degrade as you add workers, that typically means there’s an additional problem beyond “we have one queue” — things like broadcast wakeups (“every event wakes every listener”), global scans on each event, or other O(N) work per operation. That’s a very different, and more serious, scalability bug than simply relying on a single queue.

The problem with Postgres' NOTIFY is that all notifications go through a single queue!

Even if you have 20 database connections making 20 transactions in parallel, all of them need to wait for their turn to lock the notification queue, add their notification, and unlock the queue again. This creates a bottleneck especially in high-throughput databases.

We're currently working hard on optimizing LISTEN/NOTIFY: https://www.postgresql.org/message-id/flat/6899c044-4a82-49b...

If you have any experiences of actual workload where you are currently experiencing performance/scalability problems, I would be interested in hearing from you, to better understand the actual workload. In some workloads, you might only listen to a single channel. For such single-channel workloads, the current implementation seems hard to tweak further, given the semantics and in-commit-order guarantees. However, for multi-channel workloads, we could do a lot better, which is what the linked patch is about. The main problem with the current implementation for multi-channel workloads, is that we currently signal and wake all listening backends (a backend is the PostgreSQL processes your client is connected to), even if they are not interested in the specific channels being notified in the current commit. This means that if you have 100 connections open in which each connect client has made a LISTEN on a different channel, then when someone does a NOTIFY on one of those channels, instead of just signaling the backend that listen on that channel, all 100 backends will be signaled. For multi-channel workloads, this could mean an enormous extra cost coming from the context-switching due to the signaling.

I would greatly appreciate if you could please reply to this comment and share your different workloads when you've had problems with LISTEN/NOTIFY, to better understand approximately how many listening backends you had, and how many channels you had, and the mix of volume on such channels. Anything that could help us do better realistic simulations of such workloads, to improve the benchmark tests we're working on. Thank you.

If you call pg_notify or NOTIFY inside a trigger, it will get called 100,000 times and send out 100,000 notifications if you change 100,000 rows in a single transaction which from a performance perspective is ... not ideal.

This is only true if those notifications are different; if they are identical, such as in the same the notification is to alert listeners some table has new data (for cache invalidation), they are sent out as one notification only. See source code comment in async.c:

     *   Duplicate notifications from the same transaction are sent out as one
     *   notification only. This is done to save work when for example a trigger
     *   on a 2 million row table fires a notification for each row that has been
     *   changed. If the application needs to receive every single notification
     *   that has been sent, it can easily add some unique string into the extra
     *   payload parameter.

I think the sweet spot is to use exceptions for bugs. If the error is expected, make it data.

It made me happy to see the pg_get_acl() function that I was involved in adding, is appreciated by users. I think there is still much improvement in the space of querying privileges. I think most users would probably struggle to come up with the query from the article:

    postgres=# SELECT
        (pg_identify_object(s.classid,s.objid,s.objsubid)).*,
        pg_catalog.pg_get_acl(s.classid,s.objid,s.objsubid) AS acl
    FROM pg_catalog.pg_shdepend AS s
    JOIN pg_catalog.pg_database AS d
        ON d.datname = current_database() AND
        d.oid = s.dbid
    JOIN pg_catalog.pg_authid AS a
        ON a.oid = s.refobjid AND
        s.refclassid = 'pg_authid'::regclass
    WHERE s.deptype = 'a';
    -[ RECORD 1 ]-----------------------------------------
    type     | table
    schema   | public
    name     | testtab
    identity | public.testtab
    acl      | {postgres=arwdDxtm/postgres,foo=r/postgres}

Do people here need pg_ownerships and/or pg_privileges?

[1] https://www.postgresql.org/message-id/flat/bbe7d1cb-0435-4ee...

I'll add comments to this thread with Github projects that I can find by searching for "postgres listen/notify" on Github.

https://github.com/graphile/worker

    // Line 535 in src/main.ts
    client.query('LISTEN "jobs:insert"; LISTEN "worker:migrate";')

Thanks! I found a link to a GitHub Issue where they described their experienced problems: https://github.com/pulp/pulpcore/issues/6805

Fun fact: CBOR is used within the WebAuthn (Passkey) protocol.

To do Passkey-verification server-side, I had to implement a pure-SQL/PLpgSQL CBOR parser, out of fear that a C-implementation could crash the PostgreSQL server: https://github.com/truthly/pg-cbor

Hey folks, I ran into similar scalability issues and ended up building a benchmark tool to analyze exactly how LISTEN/NOTIFY behaves as you scale up the number of listeners.

Turns out that all Postgres versions from 9.6 through current master scale linearly with the number of idle listeners — about 13 μs extra latency per connection. That adds up fast: with 1,000 idle listeners, a NOTIFY round-trip goes from ~0.4 ms to ~14 ms.

To better understand the bottlenecks, I wrote both a benchmark tool and a proof-of-concept patch that replaces the O(N) backend scan with a shared hash table for the single-listener case — and it brings latency down to near-O(1), even with thousands of listeners.

Full benchmark, source, and analysis here: https://github.com/joelonsql/pg-bench-listen-notify

No proposals yet on what to do upstream, just trying to gather interest and surface the performance cliff. Feedback welcome.

Thanks for creating, really useful!

Would be nice with a [Download Combined Rendered] button to download a self-contained .html web page of the rendered combined page.

Thanks for sharing your perspective—I really appreciate it. In hindsight, I should have given more weight to the "Any better approaches?" part of the original post. I fully acknowledge that my proposal involves trade-offs and isn't a one-size-fits-all solution.

That said, I’d like to explore how we might achieve security by design without sacrificing user experience. First, let’s agree on one core principle: if a user decides to share their screen, the OS should treat that choice uniformly across all apps—meaning it must always share the entire screen.

Given that, let's think about other ideas to address the risk scenario: a user might unwittingly share their screen with an adversary and then start a top-secret chat, accidentally leaking sensitive information. Ideally, users handling top-secret data would be exceptionally cautious, but in practice, mistakes happen.

Here's an alternative approach: a "Secret Chat Room" feature, that would rely upon OS checks, explicitly authorized by the user. Think of it as akin to physical secret meeting rooms with soundproof walls and Faraday cages—places where sensitive conversations are truly isolated. When a user enters such a room, they'd see a prompt like:

    You are now entering a Secret Chat Room.
    This room is designed to ensure that no eavesdropping (such as keystroke logging, microphone tapping, or unauthorized screen sharing) is occurring.
    To proceed, please authorize the OS to perform an integrity check. You’ll be allowed in only if this check is successful.

    1: The user authorized the check AND it succeeded.
    0: Either the user did not authorize the check OR the check failed.

What do you think about this approach? I'd love to hear your thoughts on refining it further to balance robust security with a seamless user experience.

I agree that having options is crucial, but we also need to consider how these options fit together to ensure system-wide safety by design. To actually mitigate screen-sharing risks, you need a combined hardware/OS and secure communication app where the “No Screen Sharing” feature can’t be turned off—and that setup has to be widely used for it to matter at scale.

Hardware/OS choice is the first step. Some platforms, like iOS, are already more sandboxed than, say, Android. I personally use iOS and feel comfortable trusting Apple’s approach, even though zero-day exploits remain a possibility. For my server needs, I use Linux and appreciate full control and root access—but that’s a separate use case.

App choice is the second step. Secure messengers like Signal prioritize privacy as a core feature, while many other messaging apps don’t. If a few high-profile apps enforced “No Screen Sharing,” the people who genuinely need or want to share their screen could always switch to a different app. So in practice, this feature wouldn’t prevent screen sharing entirely; it would just block it in contexts where security is paramount.

All of which is to say: optionality still exists—you can choose a less-restrictive OS or a different communication app. But for those who opt into a more locked-down environment, having a secure messenger that outright prevents screen sharing can make all the difference in avoiding accidental leaks or social engineering attacks.

https://grok.com/

500 Internal Server Error

nginx/1.27.4

I tested PostgreSQL vs Neo4j with a friends-of-friends query two years ago, and to my surprise it was Neo4j that crashed, not PostgreSQL.

https://github.com/joelonsql/graph-query-benchmarks

I haven't tried the latest versions of both databases though.