Towards zero-downtime upgrades of stateful systems

https://stevana.github.io/towards_zero-downtime_upgrades_of_stateful_systems.html

by todsacerdoti • 2 years ago

96 21 2 years ago

stevana.github.io

Use cassandra?

vlovich1237• 2 years ago

This seems like a problem you can’t solve generically and you always end up making trade offs. The two main challenges I can think of:

* State in old system isn’t representable in new system (not a backwards compatible upgrade or more likely a bug exists in handling the new state)

* There’s state outside of the program that’s impossible to transition gracefully (e.g. dirtied IO socket where you don’t know what it’s state is & it’s a resource owned outside of your program)

* Transitioning state means there’s a possibility of failure because the program never reaches a graceful transition point to snapshot the state. So you either have to choose between running the old program forever or abandoning the graceful state transition anyway.

Distributed systems I’ve observed pick one of two strategies:

1. Using the load balancer strategy of migrating off the old version & then terminating it after some grace period.

2. Use a formal distributed state system like CockroachDB, Yuggabyte, DynamoDB, S3, etc etc.

This is probably a big reason why most programs use external storage solutions even if they’re less efficient - it centralizes maintenance of state onto a system that has well defined semantics and can handle repair transparently.

naasking• 2 years ago

This is probably a big reason why most programs use external storage solutions even if they’re less efficient - it centralizes maintenance of state onto a system that has well defined semantics and can handle repair transparently.

Indeed, now contrast with the article on HN discussing orthogonal persistence.

mdaniel• 2 years ago

because the front page is a fickle thing: https://news.ycombinator.com/item?id=39615228

stevan4• 2 years ago

This seems like a problem you can’t solve generically and you always end up making trade offs.

That shouldn't stop us from solving the problem in the cases where it's possible though? We can tackle the corner cases separately with manual overrides.

This is probably a big reason why most programs use external storage solutions even if they’re less efficient - it centralizes maintenance of state onto a system that has well defined semantics and can handle repair transparently.

This is certainly the case today, what I'm asking is: does it always have to be like that in the future?

vlovich1237• 2 years ago

I suspect that it’s impossible in the sense that the “possible” space will look like a distributed storage solution and the rest will look similar to graceful handoff of new connections to new version + shutdown of old version after some time (with forceful disconnect of sessions hanging around).

Rygian2• 2 years ago

* State in old system isn’t representable in new system (not a backwards compatible upgrade or more likely a bug exists in handling the new state)

New system starts in a backwards compatible mode where it accepted all state that was representable in old system. Transition is achieved after the upgrade, with flag variables.

vlovich1237• 2 years ago

Yes, but I’ve seen people regularly struggle to write code that accepts all back compat state + handles it correctly. It’s a very hard problem and bugs are very real. Not convinced it’s a better strategy to go for seamless transition for every session vs other approaches.

zubairq• 2 years ago

I love this and often think about how we can write more robust systems without having to rewrite the whole thing when the implementation of just one part changes. State has a way of becoming a global variable which all modules depend on

KingOfCoders• 2 years ago

"There’s one exception, that I know of, where upgrades are talked about from within the language: Erlang/OTP"

Java OSGi. Great tech, didn't take off, probably to complicated with not enough gain.

Towards zero-downtime upgrades of stateful systems

Related Stories