Don't add a read replica until you've read this

https://incident.io/blog/dont-add-a-read-replica-until-youve-read-this

by shintoist • 2 days ago

14 6 2 days ago

lawrjone• 2 days ago

I work with Johanna (author) on the incident team. It’s been pretty cool watching this roll out, been particularly great that I’ve barely had to think about the replicas when working with the code.

The wiring to route to either replica or primary is very transparent. Seen this done several times before and it’s never been quite as easy to have your code just do the Right Thing.

mbfisher• 2 days ago

Also work with Johanna and agree this was fun to work on!

Biggest win for us now IMO is that we can just horizontally scale reads. Instead of trying to play whackamole with micro-optimisiations, we just buy more computers. (Optimisation still has its place obvs!) And, the way this is set up, load balancing across multiple replicas can also be completely hidden from callers.

Neat

mnbbrown• 2 days ago

You guys sound like cool guys.

joehart• 2 days ago

Its a shame they didn't go with a Bladerunner replicant pun for the title. "I've seen database performance you wouldn't believe"

rorymalcolm• 2 days ago

Disclaimer: I worked with Johanna on this

I had a lot of fun “long tail whackamoling” our way through edge queries that our model of “this is eligible for a read replica” wasn’t accurately catching. Honestly just one of those “keep reading traces and working through them” development processes that just keeps drive a graph down and to the right.

patrickhamann• 2 days ago

Disclosure: I work at incident.io with the author. My favourite part of how we implemented this is that consistency stopped being something engineers had to continuously think about and became part of the Go context object in our codebase; so any subsequent write in the same context taints without them having to think, and tainted reads go to the primary.

Don't add a read replica until you've read this

Related Stories