Ambrosia: Robust Distributed Programming Made Easy and Efficient

https://github.com/Microsoft/AMBROSIA

by algorithmsRcool • 8 years ago

24 8 8 years ago

github.com

algorithmsRcool3• 8 years ago

Research Paper: https://www.microsoft.com/en-us/research/uploads/prod/2018/1...

Basic intro: https://github.com/Microsoft/AMBROSIA/blob/master/docs/Ambro...

algorithmsRcool3• 8 years ago

I gave the paper a full read through last night.

Essentially this is a language agnostic framework for building data processing systems that are highly-avalible, distributed, topologically static (no dynamic scaling), and features exactly once processing.

You define a message handler that will always produce the same output sequence given the same input sequence and the framework provides delivery, serialization, buffering, durability and transparent recovery. They even provide a nice way of wrapping non deterministic behavior so that you can seamlessly continue even if you fail in the middle of processing a message.

That being said, the really are sloppy with their performance numbers, the comparison to gRPC isn't really fair at all due to their dynamic batching. And the code examples in the paper have some really silly errors.

But the paper is still a great introduction to reliable stream processing and basic strategies for delivering exactly once delivery.

Also, there is some interesting code that the paper glosses over in this repo: https://github.com/Microsoft/CRA

rrnewton2• 8 years ago

Thanks for the feedback on the draft.

Regarding "topologically static", the system doesn't assume a fixed set of communicating endpoints (like MPI ranks). It will all you to dynamically add new participants to the network.

Why is the gRPC comparison not fair? Shouldn't they do dynamic batching also? It can be done without unduly affecting latency. (I did my phd in stream processing studying this proposition, and Jonathan Goldstein and others have demonstrated the same thing in Trill.) In the case of AMBROSIA, our latency increase vs gRPC is not because of batching, but is because on waiting for the log to persist in georeplicated storage.

huukhiem• 8 years ago

Hi,

Did you have a chance to read about Google's Dataflow paper[1] and their new Streaming Engine[2]?

From a layperson's perspective, it seems like they are tackling some of the same ideas (separation of state && computation, applying optimisation techniques used in functional world, etc).

I'd be interested in learning where and how AMBROSIA differ!

[1] https://storage.googleapis.com/pub-tools-public-publication-...

[2] https://cloud.google.com/blog/products/data-analytics/introd...

zunzun• 8 years ago

What is the difference between "robust" and "highly robust"?

algorithmsRcool3• 8 years ago

Enthusiasm?

rrnewton2• 8 years ago

Hi all! I'm a member of the team making the release. Note that this was a bit premature, as we haven't announced the project yet. So expect a lot more documentation around how to use AMBROSIA to appear shortly!

Nevertheless, please give it a try and help us out by filing issues for anything you run into. You can get off the ground by simply running `Scripts/run_*_ci.sh` (which will need you to set up your Azure storage connection string for pushing service metadata to Azure).

This Thursday, December 13th, you can find us from 8-noon at the AKS KubeCon booth. https://events.linuxfoundation.org/events/kubecon-cloudnativ...

raheegp• 8 years ago

We have now officially announced AMBROSIA! Check it out and let us know what you think! blog post: https://cloudblogs.microsoft.com/opensource/2018/12/13/intro...

Ambrosia: Robust Distributed Programming Made Easy and Efficient

Related Stories