abrgr

Great article!

There exists a higher level problem of holistic system behavior verification

This is the key observation. Strict, code-level verification of every line, while valuable in the small, doesn't contribute very meaningfully to understanding or building confidence in higher level system properties.

I see a future of modules with well defined interfaces and associated contracts. Module level contracts can be formally verified or confidently believed via property based testing. Higher level system behavior can then be proven or tested by assuming the module contracts are upheld.

Module boundaries allow for varying the desired level of confidence module by module (testing, property based testing with variable run time allotments, formal verification).

LLMs can handle each step of that chain today, allowing humans to invest specifically where we want more confidence.

Modeling workflows as state machines is incredibly powerful.

In addition to everything the author mentioned, the constraints of state machines allow a workflow platform to provide a ton of additional guarantees and capabilities around consistency, state propagation, reliable timers, inter-instance messaging, etc.

We built our workflow execution platform [1] around state machines and we've seen great results. We find our workflow code is incredibly simple and easy to understand.

[1] https://www.statebacked.dev

This is really cool. Love to see usage of state machines on the backend!

It's such a good fit for multi-step LLM apps and a really nice abstraction for generic backend flows as well.

Rust ADTs and pattern matching are so much better than other mainstream languages I find that once my code compiles it actually is almost always correct.

The next step is to encode your transition logic in the From impls between the enum structs and you've got yourself a first-rate state machine.

Great points.

It's very similar to a DAG you'd make for a workflow but not necessarily acyclic and events can be sent from external sources as well as internal processes.

States can be grouped into parent states, which gives you some additional nice things like parallel states and also a totally reasonable way to zoom out and hide info in visualizations.

When you have a pure function or a single effect, that should definitely just be a function rather than its own machine. Sate machines are most useful for orchestrating those effectful functions and for situations where you want to accept external events. So you're still writing regular code but you're writing it in small functions that get invoked by the state machine as needed.

That is, state machines don't replace code. They're just a really nice way to organize it.

Very true!

Well put and thank you!

Thanks! Running systems with this pattern for the past few months, it has been so nice being able to cleanly upgrade instances.

The old Joel on Software blog has some gems.

Thank you! These workflow systems were such an amazing advance over what came before but we can do even better.

Great point! State machines are a really nice way to orchestrate multi-step LLM invocations.

They're also a great way for LLMs to produce code. It's human-readable so you can vet that it's doing what you want it to and it's high-level enough that the remaining bits to fill in tend to be small functions that AI can easily generate.

So happy that others are getting behind this idea. Protecting critical backend state by establishing a state machine that owns it makes so many things so much easier to reason about.

Recently opened up access to a backend as a service [0] for running arbitrary state machines as reliable workflows or real-time backends. After building a bunch of systems on top of this, I'm definitely convinced that this is a better way to structure a lot of systems.

[0] https://www.statebacked.dev

Love sketch.systems!

We built https://www.teampando.com out of a similar love for state-based thinking. We turn natural language requirements into state machines, have a Figma plugin to connect designs to states (didn't realize sketch.systems had embeds that can support that - very cool!), and can export XState state charts.

State-based thinking is already helpful just to get your thoughts organized as you're building something but it's super helpful to give everyone (eng and non-eng) on the team something concrete to talk about and play with.

Thanks for the recommendation.

Ontology should take up the most mental effort.

Unfortunately, much of the advice in vouge today argues against thinking about ontology. That advice comes from an epistemological stance that you can't know the right ontology beforehand so you shouldn't even bother trying. Instead, I would argue, in those situations, you should design for the ability to easily evolve your ontology and system.

Sorry, more investigation prior to mitigation than mitigation. I think moving the lambda aspects of sst to a lambda extension would allow for a workflow like this: 1. Find production issues 2. Replicate production issues in user facing tools 3. Run a local `sst start <fn> --filter '.userId == 'my-user-id'` to have remote <fn> connect to my local for any requests that match `evt.userId == 'my-user-id'` 4. Replicate issue and debug with the request sent to this particular lambda, potentially many calls away from the user action 5. Patch and deploy.

Being able to debug in situ would be fairly incredible in a complex application.

None of that is meant to take away at all from what you've built. What you have here is really awesome and I'm excited to use it.

This is amazing. Congratulations! I've been thinking along similar lines and so happy this exists.

Have you considered extracting the lambda portion into a lambda extension so that you can deploy this as part of any cdk/cloudformation stack or standalone lambda and toggle on/off local debugging based on some registration in dynamo (e.g. store a predicate applied to the event and proxy to local if the predicate evaluates true)? This would allow something like this to be hugely useful in mitigating outages.

I didn't catch it in the repo yet but I'd you're proxying local AWS calls through the lambda to allow quickly catching IAM and security group/routing issues, that would be incredible.

Happy to help with this stuff and will definitely be a user!

Our galaxy has ~10^70 atoms. At an atom per compute element, we will use 1/10^10 of our entire galaxy for our own compute? 10^60 is a bit more than the number of atoms in our sun. Given our sub-exponential space advances over the last 50 years, it seems... ambitious... to turn our sun into a computer in the next 300.

If they're saying we'll be an interstellar spacefaring civilization in much less than 300 years, they really buried their lead :)

10^60 computing elements (presumably doing some computing that we humans want done) with only ~10^80 atoms in the universe? I'm as optimistic a technologist as they come but that seems a bit... extreme

As RyanZAG says, all production has diseconomies of scale in unit complexity. The production process has economies of scale, meaning that churning out more units of equivalent complexity reduces the marginal cost of churning out the next unit of equivalent complexity.

Software exhibits this same economy of scale in production. Take Google's machine learning platform. They allow multiple functional teams to churn out roughly-equivalently-complex machine learning-powered widgets in less and less time. Contrast that with a startup building a single machine learning-powered widget and the marginal cost to Google is significantly lower.

I definitely agree that we'll always need people who think like programmers. We can develop tools in the vein of those you mention to significantly enhance the productivity of those people though. I haven't seen many tools like that that help in distributed systems or that allow one to easily visualize and understand an entire system.

"It’s not going to be too much longer before we declaratively describe out systems as well as our code. I am looking forward to that."

Amen! Been doing that to the extent possible for a while and it is terrific!

The difficulty of conceptualizing an entire programming system gave rise to a paradigm of small pieces fit together through composition. Each piece should operate at its own level of abstraction. So too with management. Without it, everyone needs to simultaneously operate at all levels of abstraction just to be able to decide what's most important to work on.

Well done trying an experiment and well done acknowledging when it didn't work out and fixing it.

We're in a world of APIs and de facto protocols now. The bulk of the interesting communication at this point is about data. REST over HTTP provides a decent mechanism for interacting with arbitrary nouns. Anything significantly more structured would, I imagine, essentially be an exercise of modeling a particular domain of data. With rapid innovation, it becomes difficult to codify a particular data model ahead of time, so we end up in a world of RESTful APIs, some well-documented and some not-so-well-documented, and, when one emerges as the winner, it becomes the de facto protocol.

That said, the walled garden approach of all platforms today (especially Apple) certainly endangers future innovation. Though, I see that as a content delivery problem, not a protocol problem.

The world would be better without autonomous drones just as it would be better without guns or without bombs. The issue is that in a world in which the knowledge required to make a gun exists, it is in there interest of all of civilization to manufacture them and ensure that the "good guys" have enough of them and are sufficiently trained in their use to deter and exact vengeance on the "bad guys" who will develop, build, and use them regardless of their legality. This situation is identical. Knowledge cannot be destroyed (and it's very unclear of the damage caused by the destruction of the knowledge required to build autonomous drones outweighs the benefits humanity would derive from other applications of the same knowledge), so our only option as purported "good guys" is to arm ourselves. Once armed, we can attempt to construct incentive compatible mechanisms to make the use of drones of extreme negative utility, but we must be armed in case some actors have unexpected/unconventional/un-deter-able utility functions (terrorists). To propose that no smart people apply their knowledge to a particular end is idealistic and may leave us dangerously exposed.

Socialism is a "bogeyman" in the US because it has contributed nothing to the success of our country and capitalism and private property, its antithesis, has been responsible for our rise as the greatest economic power the world has ever seen.

Half of the comments on here are extolling Elon Musk and this forum exists precisely because of the advancements of similar private tech pioneers. The Netherlands is a great place and I loved the time I spent there, but no one talks about replicating the inventiveness and penchant for the creation of world-changing technologies of the Netherlands. The US is the innovation capital of the world for a reason.

As I read it, this article is, quite intelligently, recommending (or at least a consequence of their logic would be a recommendation for) using a point-free style wherever it's applicable. Programming is, primarily, data transform and a focus on the transform, rather than naming the intermediate data items, is typically a lot nicer to read. Name your functions, avoid naming your data if you can.