The Wonders of Postgres Logical Decoding Messages

https://www.infoq.com/articles/wonders-of-postgres-logical-decoding-messages/

by rbanffy • 3 years ago

88 13 3 years ago

simonw• 3 years ago

I remember hearing years ago that Facebook used a trick like this for cache invalidation: they would write custom "invalidate cache X" messages (maybe even as SQL comments?) into their MySQL replication streams that fanned out to multiple data centers, such that a data center would know to clear a cached record out of its local memcached cluster when the relevant MySQL rows were updated.

xyzzy_plugh• 3 years ago

That doesn't seem particularly novel. A replication stream by definition announces changes to data, simply listening to it gives the listener the full story.

Through a certain lens, a read replica is just a cache.

oulu2006• 3 years ago

It's quite a common pattern, we did the same thing years ago using RMQ to achieve eventually consistent cache invalidation.

cryptonector• 3 years ago

NOTIFY also can be used this way.

klysm• 3 years ago

Notify is really hard to use correctly, and I think logical replication works better in most cases

semiquaver• 3 years ago

Correct me if I’m wrong but even though NOTIFY is transactional, it isn’t persisted to WAL or anywhere else —- so any clients that were not LISTENing at the time NOTIFY was called cannot receive it retroactively. WAL is, well, a log which has nicer properties for many applications that allow clients to “fall behind” or recover from transient network disconnects without losing data.

paulddraper• 3 years ago

That's so cool.

aynyc3• 3 years ago

Does it support some form of checkpointing without Flink? I would like to run debezium without Flink.

jakehilborn• 3 years ago

Yes. Postgres stores the checkpoint on the source database replication slot. You can configure Debezium Engine to send checkpoints to the file /dev/null to disable the default external system checkpoint behavior.

aynyc3• 3 years ago

Any documentation on this? We ran an assortment of outbox pattern for data distribution, I don’t want to manage that in data producers anymore.

gunnarmorling2• 3 years ago

Author here: yes, you can use Debezium without Flink, via Kafka Connect. Alternatively, if you're not using Kafka but Kinesis, Apache Pulsar, etc., via Debezium Server (Discussing the different options on this slide: https://speakerdeck.com/gunnarmorling/change-data-streaming-...).

The Wonders of Postgres Logical Decoding Messages

Related Stories