reubenbond

Would GC.TryStartNoGCRegion work for you? https://learn.microsoft.com/en-us/dotnet/api/system.gc.tryst...

We have a section in the FAQ about it: https://learn.microsoft.com/en-us/dotnet/aspire/reference/as...

In short, these are complimentary things. Think about it like this: you use aspire to define the pieces which your app is comprised of (services, databases, etc), and that includes your Orleans silos & clients. Aspire lets you configure clustering once and then pass it to the services which consume it - rather than having to configure clustering identically in every silo & client app, for example. Here's an example of the work-in-progress: https://x.com/reubenbond/status/1724479093247549470?s=20

We should chat more about Reminders v2 - I want to learn more about what you'd like it to provide. Can you comment on the issue (https://github.com/dotnet/orleans/issues/7573)? If you'd like to see some of what is in the works, please message me on Discord and we can chat more about it (https://aka.ms/orleans/discord)

There were a bunch of new features added to Orleans in the past year, including Live Grain Migration, IAsyncEnumerable support, Cosmos DB & Redis providers.

I think workflows (durable async/await) are more useful than Reminders v2 alone (in some sense, workflows are Reminders v2), but an enhanced reminders system is likely part of that, as is the new log-structured storage system. The log structured storage issue discusses this: https://github.com/dotnet/orleans/issues/7691. We've been experimenting with a programming model for workflows and intend to share that more broadly soon. Currently, we are planning for .NET 9, so feedback is welcome (best provided via GitHub rather than here). Aspire will make it easier to build and deploy Orleans apps, which is one of the harder points for people getting started with Orleans currently.

Which systems use Calvin-like protocols?

Anything but RabbitMQ.

Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

Not the same, but this reminds me of the Epoch Protection Version Scheme discussed in this paper: https://dl.acm.org/doi/fullHtml/10.1145/3533737.3535091

EPVS is used in Microsoft FASTER, a KV storage engine that uses a hybrid log as its primary data structure: https://dl.acm.org/doi/10.1145/3183713.3196898, which is OSS and used by Microsoft (where I work), with implementations in C++ and C#: https://microsoft.github.io/FASTER/

I used it to test an idea I had to implement the Fast Paxos optimization on top of CASPaxos and it was quite useful for that: https://github.com/ReubenBond/fast-caspaxos (the code is bad, it's just an experiment)

try virustotal.com

Coyote is a great tool/library from Microsoft for testing large numbers of schedules in .NET programs: https://microsoft.github.io/coyote/

If task A & B perform IO (eg, a DB call) and the alternatives are running them sequentially on one thread or running them concurrently (via async/await) on one thread, then running them concurrently can both decrease end-to-end latency and increase throughput.

If you just synchronously run A then B, the overall time would be shorter (higher throughput) because of less context switching overhead.

There are no context switches: async/await isn't threads. The compiler generates state machines which are scheduled on a thread pool. Basically, each time an event happens (eg, database request completes or times out, or a new request arrives), that state machine is scheduled again so that it can observe that event. This doesn't involve context switching: you can have 1 thread or N threads happily working away on many concurrent tasks without needing to context switch between them.

Consider that the minimum grain of a Task.Delay is 1 millisecond.

The minimum here is contingent on a few things. The API can accept a TimeSpan which can express durations as low as 100ns (10M ticks per second: https://docs.microsoft.com/dotnet/api/system.timespan.ticksp...). The actual delay is subject to the timer frequency, which can be as high as 16ms and depends on the OS configuration (eg, see https://stackoverflow.com/a/22862989/635314). However, I'm not sure how any of this relates to "go[ing] as fast as possible", since surely you would simply not use a Task.Delay in that case.

There is a shitload of context switching and other barbarism that occurs when you employ async/await.

Async/await reduces context switching over the alternative of having one thread per request (i.e, many more OS threads than cores) and it (async/await) exhibits the same amount of context switching as Goroutines in Go and other M:N schedulers. If there is work enqueued to be processed on the thread pool, then that work will be processed without yielding back to the OS. The .NET Thread Pool dynamically sizes itself depending on the workload in an attempt to maximize throughput. If your code is not blocking threads during IO, you would ideally end up with 1 thread per core (you can configure that if you want).

Async/await can introduce overhead, though, so if you're writing very high-performance systems, then you may want to consider when to use it versus when to use other approaches as well as the relevant optimizations which can be implemented. I'd recommend people take the simple approach of using async/await at the application layer and only change that approach if profiling demonstrates that it's becoming a performance bottleneck.

The Opus OpenSubtitles corpus was very useful when I was creating this Chinese-English dictionary app: https://github.com/ReubenBond/HanBaoBao. The tool which creates the dictionary database aggregates several sources, including processing Chinese subtitles for word frequency to inform the most likely cuts when performing word segmentation.

Given how Ray "provides [...] exactly-once semantics" for its actors, you could draw similarities between it and workflow-as-code frameworks such as https://temporal.io. The way that Ray splits up actors and tasks looks similar to Temporal's Workflows + Activities split: Workflows (Ray actors) contain orchestration logic and have their method calls/results durably logged. Activities (Ray tasks) perform the expensive computations and any interaction with external systems and are not durably logged.

If you're in the .NET ecosystem or interested in distributed systems in general, you may like Orleans (https://github.com/dotnet/orleans), which I work on at Microsoft. Orleans contributes the Virtual Actor model which other modern actor frameworks are starting to adopt since it is well suited for the hectic, failure-prone environment of distributed systems (which those so-called Cloud Native Apps live in). The Ray paper linked from the article (https://www.usenix.org/system/files/osdi18-moritz.pdf) discusses some similarities. Slight correction on the paper: it states that "For message delivery, Orleans provides at-least-once [...] semantics". It's at-most-once. At-least-once messaging semantics (usually implemented via automatic retries) aren't ideal for these kinds of systems, in my opinion.

I was watching a talk from Microsoft on the topic where they would refer to actors as “cloud native objects” and the model itself as an ideal way to do OOP on distributed systems. The marketing feels a little cringe at first but the mental model works well.

Was that my Orleans Deep Dive video? https://youtu.be/R0ODfwU6MzQ

I started referring to Grains as Cloud Native Objects as a way to better convey them to people who aren't familiar already. My view is that Grains are different enough from Erlang/Akka style actors that lumping them under the Actor terminology can cause confusion. In the research publications, the term Virtual Actor is coined, which better describes them.

That video doesn't discuss it, but Orleans also supports distributed cross-actor ACID transactions, largely thanks to Phil Bernstein: https://www.microsoft.com/en-us/research/publication/transac... and more recently https://www.microsoft.com/en-us/research/publication/resurre....

The Raft paper is there. It's the first paper under the "Expository papers and blog posts" section

Orleans and Service Fabric are different. Orleans has been running in production for some time now and is actively developed on GitHub: https://github.com/dotnet/orleans. Teams inside Microsoft run it on top of Service Fabric (and Kubernetes, etc.) More details in this talk: https://youtu.be/KhgYlvGLv9c

Service Fabric has something called Reliable Actors which are heavily inspired by Orleans.

Source: I'm the project lead for Orleans

Have you seen Orleans? https://github.com/dotnet/orleans

why it took up to 6 years for MS to release an experimental fix [1] to run on Kubernetes? where I can find the outstanding issue in the release notes that [1] tries to fix? - if .NET Orleans runs fine on Kubernetes, why is there a need for an experimental fix?

There is no experimental fix, just improvements. Things which can make life easier for developers running on Kubernetes by automating some things (setting addresses), and taking advantage of information that's available in a Kubernetes cluster (whether or not a pod has been deleted) and feeding that into the cluster membership system.

The reason the latter is useful is that it addresses something which can occur during initial dev/test, but which does not come up in production cases: when an entire cluster is deleted and redeployed with the same identity, the new instances try to contact defunct instances for a few minutes as a safety measure. The enhancement is to query Kubernetes to determine if it's worth trying to contact those nodes, or whether they're almost certainly dead.

- why Microsoft doesn't write and release an Operator & Helm chart, instead asking the community?

Helm charts aren't something we see requested often. Perhaps because Orleans is a framework which is embedded into the developer's application and not a service which gets deployed and stands alone (compared to, for example, a database). There are no separate Orleans pods, just the user's application pods. Internal users have been building applications on Kubernetes with their own Helm charts. Microsoft is not asking anybody to create those things, or anything, unless they want them for themselves.

Interesting take. We run Orleans on Kubernetes in production at Microsoft on multiple services. Other services run Orleans on Service Fabric in production - SF is similar to Kubernetes in terms of lifecycle. The project is open source, so things like Helm charts and k8s operators would be welcome contributions.

If you want to discuss, feel free to message me on Twitter (https://twitter.com/reubenbond) or Gitter (https://gitter.im/dotnet/orleans).

I think if you look at the consistent development history, you can use that as an indicator. Internal teams host it on Kubernetes (Linux), Service Fabric (Windows), and other places. The "Orleans at Microsoft" talk from August covers where teams are hosting it and how they're using it: https://youtu.be/KhgYlvGLv9c

No, Service Fabric Reliable Actors are based on the Orleans API and model

There is a new package which adds improved Kubernetes integration and fixes the issues you mentioned specifically. It's in beta right now, but we expect a stable release soon. Feel free to contact me and I can help you with running Orleans on Kubernetes

There are developers using Orleans with F#, so it does work. It is not as nice of an experience as it ought to be, though. For example, it requires a dummy C# project to hold the generated RPC/serialization code. PRs to improve the experience for F# developers are greatly appreciated.

Orleans doesn't have any ordering guarantees. While this is not a firm requirement of the Actor model itself, both Erlang and Akka guarantee ordering between a given Sender-receiver pair (i.e. messages sent from A to C will be sent in order)

Ordering has a performance cost. It needs to either be maintained at all levels, or reconstructed from unordered messages at a later point. Even something simple like an m:n thread pool scheduler can ruin ordering guarantees. My view (Orleans core developer) is this: if you want ordering, await your calls. That way, you are guaranteed ordering regardless of any message reordering that can occur in the scheduling or networking layers, or due to failure and recovery of a host. So you can choose when to pay that cost and when to reap the performance benefits of not paying it (by firing off multiple calls in parallel).

A stable and properly configured Akka Shard cluster will never have more than one of the same entity actor alive at a time.

Likewise for Orleans, but the caveat of a "stable cluster" doesn't do much for users. "Stable cluster" falls apart frequently in real scenarios, which can be as simple as a single machine being abruptly restarted. Developers must account for the error scenarios.

Akka and Erlang have the concept of Supervision.

Orleans does not have supervision, since each grain stands on its own (no hierarchy) and have an eternal nature (managed lifecycle). Grains are not destroyed when a method throws an exception: the exception is propagated back to the caller and the caller can use try/catch to handle the exception. This is similar to what .NET developers are used to, since regular objects are also not destroyed when a method throws an exception, and the caller is able to handle the exception. My belief is that this kind of exception handling is usually appropriate, since the caller has context which can be useful in handling the error. The developer can also write per-grain or global call filters which can operate on all calls and handle any exception, so if that's preferred, then it's available.

If the link is being updated, I think the github repository provides a much better overview: https://github.com/dotnet/orleans/

That's a good question. I wasn't around at the time the project was formed. At that time I was working on Microsoft's internal metrics system. Years later, I became involved as an external contributor (I had left MS) and eventually rejoined MS to work on Orleans full-time.

My opinion is biased, as a core developer, but I do not think it's packed full of design patterns. Orleans and the core team are relatively unopinionated on design patterns that developers employ. I'm much more concerned with developers writing code which cannot perform well at scale, or which becomes unreliable at scale, where machine failures are common, than I am with whether or not something fits some set of prescribed design patterns.

All communication in Orleans is asynchronous, similar to communication in gRPC, so in .NET world, you're using async/await with methods that return Task/Task<T>/etc. The boundary there is hopefully apparent: async calls can incur IO and have a cost and the 'async' keyword can hopefully make that cost apparent to the developer.

References to grains are represented by interfaces and if the machine you're communicating with fails, the grain will be re-activated on a surviving host the next time you need to call it. In other words, they are location transparent and the application won't get stuck in some failure state when a machine crashes.

Microsoft Research has a tradition of naming projects after cities. Eg, if you type "microsoft research project" into Google (Bing appears to work better for this) and let it autocomplete, you can see some other projects which appear to be named after places: malmo, athens, tokyo, and others.