Solving durable execution's immutability problem

https://restate.dev/blog/solving-durable-executions-immutability-problem/

by stsffap • 2 years ago

52 28 2 years ago

stsffap3• 2 years ago

The hardest problem in durable execution, as in many areas of infrastructure, is safe updates.

pavel_pt4• 2 years ago

Not just in durable execution, but in control systems in general! Anything that operates on persisted state in a distributed environment has similar issues. Durable execution frameworks make them more apparent - and may also help us chart a path for how to deal with updates more rigorously.

whoiskatrin3• 2 years ago

Ok, this actually looks interesting

JackHopkins2• 2 years ago

Cool! why is it crucial to keep old versions of code, and what are the risks of running outdated code?

p10jkle4• 2 years ago

If don't keep old code around, you are forced to either drop pending requests that started on them, or try and replay them on the updated code, which can't be known to be safe. So it's better to keep it around, but this brings in a new set of infra and security challenges, eg where will it run, what will it cost, will it have a vulnerable dependency, etc

JackHopkins2• 2 years ago

Can't you replay pending requests? How can you mitigate the issue of differing side effects generated by the new / old versions?

paidcompute• 2 years ago

Can you dive deeper into the technical details of how Restate integrates with Lambda's versioning system to ensure in-flight requests?

stsffap3• 2 years ago

Every Lambda you deploy to AWS Lambda gets a version assigned. The corresponding arn would look like `arn:aws:lambda:my-region:123456789101:function:my-function:my-version`. Once you have deployed the Lambda, you register it with Restate by providing this very same arn. Now, when Restate invokes a Lambda function, it remembers the arn with which it started the invocation. So on any subsequent retry, Restate will always invoke the same arn it started the invocation with.

akbirkhan• 2 years ago

this is pretty important when you move to LLMs as microservices (e.g. calling gpt-4 for some plan / then 3.5 for easier sub component).

Would be keen for a python implementation

danism• 2 years ago

Is there a timeline for a Python SDK?

stsffap3• 2 years ago

We are currently gathering feedback on which SDKs to prioritize next. Python has been asked for a couple of times already. Once we decide on the next SDK, we'll let you know.

baran-y• 2 years ago

Will there be a Go SDK?

p10jkle4• 2 years ago

It's probably the next on the list :)

threepointone• 2 years ago

Curious how you think about state that changes outside of a workflow’s scope (like user info in a database)

p10jkle4• 2 years ago

In general, if you're changing database schemas, you need a code version running that supports old and new. If you have some old code running to serve in-flight requests, you probably need to wait for those requests to complete. This is one of the main areas that makes me feel that requests > a few hours are a real problem (and that we shouldn't be writing workflows this way)

DaiPlusPlus• 2 years ago

I'm of the opinion that actual database schemas should be a private implementation detail of any given service within an organization, whereas workflows are not services themselves, but instead are clients of the services and so should only ever interact with services via their API (be it a web-service or otherwise (no love for CORBA, eh?) and never directly talking to an RDBMS.

(obviously the rules are different for KV/object/blob-stores and don't apply here).

-----

That said, if you need a stopgap solution for a problem like that, then take advantage of existing compatibility-shim features in a decent RDBMS: things like VIEWs and SYNONOYMs exist for this reason (and yes, you can INSERT INTO-a-VIEW).

-----

Another option is to constrain your DB designs such that all DDL/schema changes are always backwards-compatible (i.e. disallow dropping anything, disallow alter-column except to widen an existing column (e.g. int-to-bigint is okay but not string-to-UUID), all domain-model attributes must be initially modelled with a m:m multiplicity whereever possible, etc etc) - and you can enforce these rules using a DDL trigger too - and any changes made by a workflow's DML can be repaired by a background job.

----

(Just throwing some ideas around)

taeric2• 2 years ago

Funny to see this continue to get so much energy. I was strongly opposed to making some AWS Step Function workflows fully immutable in the first project I used it for. This was all the more my feeling with some of the gigantic state machines that teams around me were building. Not shockingly, situations would arise where we could know ahead of time that a large portion of outstanding state machines were cruising to a crash at step $BIG_NUMBER.

Ironically, I'm still largely about idempotent calls that should be able to trigger something. And those idempotent identifiers can be stored such that you can replay or serve up the relevant data, as needed. But going all in and forcing a long compute chain that cannot be changed is something you do in places you have to do it. No reason to choose to do it everywhere. And certainly don't build larger and larger compute chains between durable writes to the backing data store.

k8svet2• 2 years ago

That... Just sounds like a horrible implementation/engine/execution/strategy whatever you want to call it.

I don't see any reason whatsoever that a temporal workflow composed of activities would be subject to the things you've described. And I don't even think it's that impressive of an implementation of the idea.

taeric2• 2 years ago

And that sounds like a no true scotsman criticism. :D

My complaint was less that it isn't a compelling idea. The problem seems to be that people want to expand the one system they are working on to be a fully comprehensive system. I remember one team tried to get their entire state reflected in a Step Function state machine that could run for as long as necessary. Idea literally being a state machine that would last years or so. They compromised for one that would only last a week or so. And then faced the consequences rather quickly.

And note, given enough time, I think this can work. On a deadline, though, I've seen it fail too many times for it to be my goto solution. All the time by people that are objectively smarter than I am.

Solving durable execution's immutability problem

Related Stories