PostgreSQL 16 Beta 1

https://www.postgresql.org/about/news/postgresql-16-beta-1-released-2643/

by jkatz05 • 3 years ago

154 54 3 years ago

postgresql.org

klysm• 3 years ago

Glad to see the move off of autoconf to meson.

itsthecourier• 3 years ago

COPY from files up to 300% faster

Load Balancing from client libs

Support for CPU acceleration using SIMD for both x86 and ARM architectures, including optimizations for processing ASCII and JSON strings

new pg_stat_io view that provides information on I/O statistics

craigkerstiens• 3 years ago

For being "boring and stable" technology, Postgres 16 includes 200 features, which is on par with each of prior years releases. If you're looking for the full set of features it's available here - https://www.postgresql.org/docs/16/release-16.html.

Personal favorites on the list include:

- load_balance_hosts, which is an improvement to libpq so you can load balance across multiple Postgres instances.

- Logical replication on standbys

- pg_stat_io which is a new view that shows IO details.

Fgehono• 3 years ago

Could you elaborate on the load balancing?

Is this a replacement for PG bouncer and similar?

dalyons3• 3 years ago

IMO the biggest reason folks use pgbouncer is not for load balancing (which it can do, -ish) but instead for connection pooling. Postgres connections are expensive for the db server (one process per connection not one thread) so if you have say thousands of web application pods you need to use pgbouncer or similar as a proxy to multiplex those thousands of connections down onto a more manageable number (~200). So no, not really.

(EDIT: if you don't know this already - the _establishment_ of connections is also super expensive. so another reason to pgbounce is to keep connections persistent if you have app servers that are constantly opening and closing conns, or burst open conns, or such like. Even if the total conns to pg doesnt go super high, the cost of constantly churning them can really hurt your db)

garyclarke27• 3 years ago

Postgresql is amazing but I must say the last few updates have been a bit disappointing.

Still waiting for automatic incremental updates for materialized views - been worked on for several years but still not released!

https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...

https://github.com/sraoss/pg_ivm

tensor• 3 years ago

To be fair view update is a very hard problem in computer science. It's not as though it's just a lack of time to implement it.

samwillis• 3 years ago

https://materialize.com/ are doing interesting things with this - although not inside Postgres (they use a few bit of Postgres as components).

cryptonector2• 3 years ago

I've done some work in that space, which was mainly to build an alternative view materialization scheme[0] in PlPgSQL that reifies the internals of CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW but with the following properties:

  - the materialized view is a plain table, so
     - you can write to it from triggers
     - you can have triggers on it
  - refreshing a view records the deltas in a
    history table (which is useful as a poor
    person's logical replication scheme)
  - you can mark a view as needing a refresh

Then in the application I have hand-coded triggers to either update the view's materialization directly or to mark the view as needing a refresh. A background job can asynchronously refresh views as needed.

I've also spent some time thinking about the AST form of view queries that PG stores and how one might automatically generate triggers on source tables that update the materialization or mark it as needing a refresh.

As you note, many queries can be very difficult to transform into queries that compute incremental deltas. Moreover, even where it's possible to do that, the time it takes to execute the delta computation might be unacceptably long. For example, if you have a recursively nested grouping schema and you want to maintain a view of the expanded transitive closure of that data, then removing a large from from another might require thousands of row deletions from the materialized view, and that might make the transaction take much too long in a UI -- the obvious thing to do here is to say "sorry, that kind of update takes a while to propagate, but your transaction will complete quickly", so just mark the view as needing a refresh and refresh it asynchronously.

[0] https://github.com/twosigma/postgresql-contrib/blob/master/m...

sophacles• 3 years ago

On the one hand, I totally understand.

On the other, it's pretty confidence-inspiring that they don't put stuff in until they're sure it's ready.

simonw2• 3 years ago

Yeah, that's one of my favourite characteristics of PostgreSQL: if a feature makes it into a release, you can be VERY confident that it works well even across weird corner-cases.

pawelduda• 3 years ago

I have one use case where this functionality would be huge in terms of performance, patiently waiting for it

cryptonector2• 3 years ago

You can use https://github.com/twosigma/postgresql-contrib/blob/master/m... and write triggers to update your views or mark them as needing asynchronous refreshes. It's not as automatic as what you'd like, but it's a lot better than having to keep waiting.

h1fra• 3 years ago

Postgres getting good updates at slow but steady pace over many decades is really impressive.

jasfi• 3 years ago

I look forward to seeing some benchmarks. Especially to see the effect of the SIMD optimizations.

tuetuopay• 3 years ago

This is an exciting release to be sure. Very happy to see that the CPU improvements include ARM, diversity is good!

Given the current release pace, I'd love to have another upgrade path than `pg_dump | psql`. That would remove a great deal of friction in prod.

hans_castorp2• 3 years ago

I'd love to have another upgrade path than `pg_dump | psql`

pg_upgrade

djbusby• 3 years ago

Or logical replication to the new box then migrate traffic.

ComputerGuru• 3 years ago

pg_upgrade is not an option on many systems if you are installing with the package manager as it requires both old and new binaries to be simultaneously present.

jeltz2• 3 years ago

There is pg_upgrade and logical replication. I have used both and they both work excellently. Logical replication can be a bit of a hassle to set up though, especially if the database is huge.

LunaSea3• 3 years ago

Can't wait to have it available in RDS in 2030.

phamilton• 3 years ago

AWS has gotten much better in recent years. Pg15 is available on Aurora as of last month. That's about 6-7 months after it was released, a significant improvement over years past.

LunaSea3• 3 years ago

Yup, but the lack of access to beta versions also means that we will basically get access to access to PostgreSQL 16 in RDS by next year around this time.

pritambaral• 3 years ago

My list of unexpected but very welcome changes:

- pg_hba.conf and pg_ident.conf can include other files

- Logical replication apply can use non-PK btree indexes

- Integer literals in non-decimal bases

- Underscores in numeric literals

- Subqueries in the FROM clause can omit aliases

- Addition and subtraction of timestamptz values

- pg_upgrade can override new cluster's locale and encoding

MuffinFlavored• 3 years ago

Not a bad list!

https://commitfest.postgresql.org/19/1741/ (index skip scan/loose index scans) would be very welcomed... I think. Not sure how many people run into it in the wild.

It says "target version: 16" but "returned with feedback" and hasn't been bumped in 14 months. :( First opened in 2018.

https://wiki.postgresql.org/wiki/Loose_indexscan

seanhunter• 3 years ago

- Subqueries in the FROM clause can omit aliases

This is great. It never made any sense to me that this was required. For people who are unaware, say you want to understand a table a natural way of doing it might be

    select * 
    from the_table
    order by some_metric desc 
    limit 10

so you'd think you can do the same for queries like

    select * 
    from (
      select blah blah blah the rest of the query
    ) a
    order by some_metric desc 
    limit 10

you need to put the alias 'a' to placate existing postgres even though it's never actually used, which never made any sense to me.

paulddraper• 3 years ago

It never made sense.

And FYI this comes from ANSI SQL.

jimktrains2• 3 years ago

It makes sense in that every table needs a name to reference, however if you only have the one table there isn't any ambiguity.

rektide• 3 years ago

Bunch of interesting loosenings ups on logical replication. Allowing standbys to also have subscribers; great for fan out! Bunch of perf improvements (honestly a bunch of it is removing rather onerous limitations I didn't know about!) At the end there seems to be a developer flag for allowing seeming bidirectional replication too.

simonw2• 3 years ago

"parallel execution of the string_agg and array_agg aggregate functions" caught my eye - that feels like one of those minor improvements that might enable some pretty interesting new applications.

array_agg is particularly interesting, because it lets you implement patterns like this: https://til.simonwillison.net/sqlite/related-rows-single-que...

tracker13• 3 years ago

Have to say, I find the SQL/JSON support in Postgres (jsonb) is probably the single biggest killer feature IMO. I can't believe MS (still) doesn't have an implementation (their SQL to JSON functions suck exponentially by comparison).

Nice to see the continued advancement and progression all around.

sureglymop2• 3 years ago

Is MSSQL used often in production? And where are all the windows servers this would run on?

tracker13• 3 years ago

Yes... many, many internal deployments are definitely using MS-SQL services. StackOverflow.com uses MS-SQL, for a prominent, public example. There are client libraries via ODBC or direct in many languages and platforms, including Node.js (mssql/tedious) and Rust.

As to where, AWS offers Windows as do many other cloud providers, including a significant portion of Azure VMs. Not to mention, that MS-SQL and SQL-Edge both run on Linux. IIRC, Azure Cloud SQL is also MS-SQL running a non-windows version. There's also Linux/x86_64 Docker images.

Aside: if you're willing to write a big check, MS-SQL replication configuration is far easier than pretty much anything else to setup and configure (UI based flows or scripted). While I personally advocate for PostgreSQL, I've used and mostly like MS-SQL fine.

cjblomqvist• 3 years ago

FYI: MS is releasing support for a native JSON data type in next release.

But yeah, obviously they're behind in that area!

jeltz2• 3 years ago

I am personally very excited to see that work on direct IO has started. It is a huge undertaking so I do not expect any production ready thing any time soon but nice to see that some code landed.

anarazel2• 3 years ago

Yeah- we've been working on that for quite a while, and it'll take a few more years to finish it. I hope we can get some important parts into 17 though.

FWIW, there have been smaller prerequisites merged into 15 already, and 16 has a number of improvements that are part of that work. E.g. the more scalable relation extension (making COPY scale much better), and the related buffer mapping changes, come from the AIO effort.

Personally I think the feature is using asynchronous IO and direct IO support is part of that :)

darksaints• 3 years ago

Meanwhile, PG15 is still in preview mode on Azure

PostgreSQL 16 Beta 1

Related Stories