Reactive Cloud Actors: No-nonsense micro-services

http://byterot.blogspot.com/2014/04/reactive-cloud-actors-no-nonsense-microservice-beehive-restful-evolvable-web-events-orleans-framework.html

by aliostad • 12 years ago

41 31 12 years ago

byterot.blogspot.com

bananas3• 12 years ago

My bullshit detector was tripped by the number of buzzwords.

What does this actually solve versus say traditional MQ (EIP stuff), service bus systems and normal SOA?

I've dealt with NServiceBus extensively and really big scale .Net stuff and if it's anything remotely close to that architecture, I'd rather hang myself than use it again.

aliostad5• 12 years ago

Yes, so many buzzwords together - I admit.

The difference is replacing Commnad/Query with events. Replacing Imperative with Reactive.

Did you read the whole post or you were put off by the buzzwords?

bananas3• 12 years ago

Perhaps it's because it is Sunday but I phased out after the first couple of paragraphs. I've read it now and am still unaware of why this is fundamentally better.

Command/Query pattern is an imeplementation concern. The architectural pattern that backs it up isn't necessarily related (for example event sourcing). At that point it isn't imperative so why is this different? It just appears to be fragmenting lots of concerns into even smaller ones which are vastly more complicated to debug and manage.

The bold highlighted text of: "in a reactive actor system, each actor only knows about its own Step and what itself does and has no clue about the next steps or the rest of the system"

Isn't that the same for a message channel but without a transactional guarantee? (going back a decade or more to EIP).

mml2• 12 years ago

He's certainly on to something. As he pointed out, it's been around since about '73, and many people are happy with the design of such systems. Amazon even has an actual for-real implementation of one (see Simple Workflow), he should probably try it out before rolling his own.

It really is an elegant way to do things from what I can tell. It forces you to keep your state in one, carefully managed place (hairball), rather than sprinkled around god-knows where, being poked and prodded by god knows what....

IMHO, There should be more talk about the design of actor based systems than the glut of OO yammering we are all exposed to.

th0br02• 12 years ago

Isn't this what http://akka.io does?

aliostad5• 12 years ago

Yes, but they do them differently. BeeHive is much more opinionated about the message (must be event and has an enveloper) and the Basic Data Structures.

Also BeeHive avoids stateful actors. State is always persistent in HA queues or Basic Data Structure stores.

platz• 12 years ago

Any comments about the recent Orleans announcement? There must be differences but also I assume a lot of similarities

As you mention there in no state in beehive actors, so that is one difference.

http://blogs.msdn.com/b/dotnet/archive/2014/04/02/available-...

th0br02• 12 years ago

How exactly do you mean "stateful actor"? Whether an Akka actor has a state is actually up to you!? E.g. you could have your router fire up a new actor to process each request up to a maximum number of n concurrent actors etc...

hibikir• 12 years ago

I thought Rails had taught us that extremely opinionated infrastructures are doomed to at best be influential, but not widely used.

Sometimes actors have to be stateful. Sometimes they should be the interface to large, complex computations that take 20 minutes to run.

The market is choke full of techs that solve very narrow cloud problems. What we need is something that isn't so terribly narrow. Microservices is very, very narrow. Even people working for Fred George admit to that.

orasis• 12 years ago

Sorry, I couldn't get past the terrible whitish text on blackish background. Instant headache.

mamcx2• 12 years ago

The big problem with this kind of stuff is how debug them. Also, that despite the intention, all is really coupled: Don't make sense to cancel a order before create it, so the order of the actions are important.

The benefit is (only ?) easy of scaling, IMHO. How solve this? I understand that GO keep the ordering of the calls and is not harder than doing imperative...

aliostad5• 12 years ago

Well, first of all, debugging while developing an actor is actually easier: you just have to debug an actor in isolation and make sure it does the right thing and sends the right event in response to the incoming event.

In terms of debugging issues when in live, you have to rely on tracing and let something such as Logstash handle search, etc.

The ordering of the workflow is preserved. As I explained, FraudCheck is only ever done after an order's payment has been authorised.

Hope this makes sense.

twic• 12 years ago

No. No, this is terrifyingly incorrect. Making each component do the right thing in isolation is easy. Most of the difficulty in systems - in developing, testing, and debugging them - arises from what the components do when they're together, because it turns out your ideas about how they need to behave in isolation were mistaken.

mercurial• 12 years ago

Fun. I accidentally used this pattern (minus the timestamp, since it wasn't relevant for my use case) to implement a non-distributed data processing framework on top of POE. It does work pretty well, however, you need to have a way for the actor to block until enough messages have arrived (sometimes buffered operations are the most efficient). I'm not sure how you would do that with the model proposed here.

mantrax44• 12 years ago

This guy seems uneasy with Actors. His example is wrong and then he tries to solve it with a solution that's worse than the problem he thinks he has.

Let's untangle that stuff. Actors are about conversations between folks organized, like you would find in a good company. So let's use the conversation metaphor.

His example of "classical" actors:

- PaymentActor (to herself): Someone told me to process a payment from this order. I'm done.

- PaymentActor: Hey, FraudCheckActor, check if this order I'm giving you is a fraud.

- FraudCheckActor (to herself): Damn, this is a fraud.

- FraudCheckActor: Hey, CancelOrderActor, cancel that order I'm giving you!

- CancelOrderActor (to himself): I canceled the order.

Now I see an immediate problem here. This is anarchy. No one is in charge of the whole process, everyone runs to talk to someone else, and passing around that order data and there's no conversation happening here. Would someone in a real company organize things this way? Not in a good company, for sure!

Let's see what his solution is:

- PaymentActor (to herself): Someone told me to process a payment for this order.

- PaymentActor (screaming): Everyone! Payment is complete for this order! (screams the order info to everyone)

- FraudCheckActor (to herself): I heard that, I'll check it. Damn, this is a fraud.

- FraudCheckActor (screaming): Everyone! This order is a fraud! (screams the order info to everyone)

- CancelOrderActor (to himself): I heard that, I'll cancel that order.

- CancelOrderActor (to himself): I cancelled it.

Now I see an even bigger problem here. It's still a god damn anarchy, but now also everyone is screaming at everyone, including the entire order data, or talking to themselves. How is this better again?

Plus notice, there's still state in the system - the subscription setup. The only thing achieved here is that it'll be nearly impossible to reason about how this thing works once it becomes more like a real-world scenario.

Here's how you design it with Actors, once you have some experience:

- OrderActor: Hey, PaymentActor, process this payment and tell me what happened.

- PaymentActor: Hey, OrderActor, I processed it. It's good!

- OrderActor: Hey, FraudCheckActor, check this order for fraud and tell me what you found.

- FraudCheckActor: Hey, OrderActor, damn! This is a fraud!

- OrderActor: Hey, CancelOrderActor, cancel that order and tell me when done.

- CancelOrderActor: Hey, OrderActor, I canceled it!

Now what do we see here? We see structure. We see clear responsibilities. We see teamwork. No screaming and no chaos. Only OrderActor has the entire order data. He's passing along only the needed bits to each actor according to their responsibilities.

And best of all, PaymentActor, FraudCheckActor and CancelOrderActor are not coupled to each other. They just respond to their events, limited to their own responsibilities.

Good job OrderActor & co.

OrderActor is a supervising actor (that's an actual term). In my example, he's managing the workflow and he is stateful and as you see it's not that scary and out-of-order as the author is telling you. After all, he needs to know the payment went through, and then remember to ask for a fraud check and know what to do after that.

(Note: technically the supervisor in this example doesn't have to have state, he can just react to the events by Payment, FraudCheck, and Cancel, but this only works in this very simple example. Real world scenarios never fit into a completely stateless world. We're talking about the order state here!)

The other actors have no state (that is seen in this example). So proper design isolates concerns, and may have stateless actors, but stateful actors are essential to the model.

Of course, the linked article demonstrates how you can shoot yourself in the foot even with the best model out there. I find it ironic that the author was unhappy with over-complications and coupling, yet coupled everything in a complicated way with his "subscriptions" model, instead of simply using a supervisor.

mml2• 12 years ago

This is pretty much exactly how AWS SWF handles it.

The java implementation of the workflow "decider" (supervising actor) is somewhat interesting as well. It actually creates a bunch of future/promise objects and replays the workflow from the top as the promises are delivered (which of course has the implication that workflow decider processes must be deterministic!).

I suspect, but don't know, that AMZN has been dogfooding this product in a big way for quite some time.

(full disclosure: just spent some time with the swf guys in seattle)

On the other hand, It seems like the long way around to inventing Erlang ;)

karmajunkie6• 12 years ago

There's nothing particularly wrong with his design, except for appropriating the terminology from the Actor model for what is basically CQRS.

You're just advocating for an imperative solution which may or may not be asynchronous. Nothing particularly wrong with that, but its not the only "right" way to do it. In point of fact, the underpinning of languages like Haskell is that state is conveyed by the arguments to functions (i.e. events), and the terminal "state" of a process is just the integration of those events over time or composition of functions. State is useful as a convenience. Its not essential.

Also, not for nothing, but oversimplified metaphors like shouting in an office are just obnoxious, and also irrelevant. Shouting might be a problem in an office. Software, it turns out, is capable of paying attention to whatever it wants to pay attention to and ignoring the rest.

mantrax44• 12 years ago

Actually, shouting in the office means you coupled the FraudCheckActor to a PaymentCompleteEvent, and you coupled CancelOrderActor to FraudDetectedEvent.

Your lower level actors are still coupled with each other, through quite specific events, and on top of that, dependent on a subscriber setup.

Instead of declaring your own protocol and accepting messages in that protocol, now you need to understand everyone else's protocol (in the form of their event types) and react to it if you're subscribed to it.

It's the worst of both worlds.

Now, sure, you're right. Actors aren't like people in an office. Actors can't complain when you entangle them in a hot mess, they just carry on with whatever they're told to do.

But the more experience I gain, the more I see the systems of the "real world" are in fact exactly like the systems of the "computer world". If something makes no sense in an organization made of people, then almost inevitably it'll make developers cry when having to figure out the sequence of events when implementing the same organization as a set of actors.

Take it as a useful heuristic and humor my analogies. You might find a nugget of gold there.

Also, there's nothing blasphemous if my supervisor can be implemented as synchronous imperative code. When people hear about a new paradigm, they believe they have to abandon everything and do things differently for no reason. No. Find some balance, what you knew still works, don't throw it out, just add the new benefits Actor offers to you.

I.e. while Actors mean we can have objects that do things that you can't do in imperative/synchronous fashion, this doesn't mean we should never use actors for something that might be done in imperative/synchronous fashion. 90% of the interactions between actors still look like a request/response pattern. But those 10% when things don't look like a request/response really matter when you need that ability.

Also keep in mind Actors don't care how far its subordinates are. In synchronous programming things have to happen fast, or the thread blocks. With actors a request may be sent this week, and you may get an answer next month. Things can happen very fast, or very slow, the model is not biased. How fast is fast enough is now entirely up to your app's domain logic.

Also, some Actors implement persistence (which is far more trivial than trying to persist the callback graph or call stack in "normal" code), so they don't even care if the system shuts down and it has to start over from its last state.

Imagine OrderActor sends a call to FraudCheckActor and then the system crashes, or (if it's a business workflow process) it's just powered down for the night. When the system is restarted, OrderActor is already waiting for that answer from FraudCheckActor.

Do that with imperative/synchronous code.

zenbowman• 12 years ago

Another case of reinventing the wheel and calling it something new. This is just a reimplementation of what used to be called the "blackboard architecture".

http://en.wikipedia.org/wiki/Blackboard_system

I don't see how broadcast is more "reactive" than singlecast. And the solution to tight coupling in actor systems has already been thought of - it is a message broker. This is just an actor system where every actor shares a single message broker.

Its cool that you built your own system for actors, and I certainly wouldn't discourage people from reimplementing something for the sake of learning, but every criticism you have of traditional actor systems is inaccurate. Strangely enough, you criticize Orleans in particular, which is especially good about tackling the coupling problem by making actor references virtual.

"Perhaps the central problem we face in all of computer science is how we are to get to the situation where we build on top of the work of others rather than redoing so much of it in a trivially different way."

-Richard Hamming

Reactive Cloud Actors: No-nonsense micro-services

Related Stories