Docker libswarm

by fredkelly • 12 years ago

106 15 12 years ago

shykes5• 12 years ago

Hi all, a quick warning: this is very rough and early, and not yet ready for production use.

What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).

TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!

[1] libswarm is built on a small library called libchan: https://github.com/docker/libchan

kansface• 12 years ago

How does libswarm compare to etcd/fleet?

akerl_2• 12 years ago

This is a layer "above" etcd/fleet/geard/shipyard/etc.

The idea is to standardize the way that administrators/users/developers "talk" to the various underlying containers. The intent is for it to have plugin support for etcd and fleet and all the other tools in that layer.

shykes5• 12 years ago

The author of Fleet have started contributing a Fleet adapter :)

libswarm exposes a standard interface to systems like Fleet, Mesos, Consul etc. It also has adapters for popular infrastructure providers, and for the Docker API itself. That way you can compose your architecture from standard building blocks, and change the building blocks over time without being locked in.

There was huge demand for this in the Docker system because there are so many orchestration tools being announced and competing for the attention of developers and sysadmins. Just at Dockercon, New Relic announced "Centurion", Spotify announced "Helios", Google announced "Kubernetes". Having a standard interface allows for these tools to continue competing without the risk of fragmentation and lock-in.

Of course the interface has to be thin enough to not get in the way. That's why we're focusing on keeping the codebase small, and the API as simple as possible.

philips• 12 years ago

fleet and etcd are orthogonal to the problems that libswarm is trying to solve.

libswarm has a future as a nice way of doing composition of networked services that are represented as objects in pipelines. The first demo is a `swarmd` daemon that exposes a docker API endpoint on one end and aggregates between a lot of "real" Docker endpoints.

The big open question is how this sort of composition works when the objects have different interfaces. So, how do we expose a "swarm" API that can be shared between different consistent data stores like etcd, zookeeper or consul. Or how do we encapsulate things like a runtime configured load balancer like nginx, apache2 or vulcand using a unified interface.

My hope for the project is to look at the interface model of Go or the DBus protocol and try to get consensus around the interfaces that make sense and hide complexity.

dominotw• 12 years ago

Organize your application as loosely coupled services from day 1, without over-engineering.

Seems a little counterintuitive

shykes5• 12 years ago

This alludes to the fact that you can do this within the same process, simply by running goroutines and communicating over Go channels. Later, when you actually need to scale, your services already support the interface necessary to be exposed over unix sockets, websocket or spdy/tls: you just need to change the plumbing.

To do this we take advantage of a low-level communication library called libchan: https://github.com/docker/libchan

moe• 12 years ago

What if most of my services are not written in Go?

What if most of my services (postgres, redis, etc.) are not even written by myself?

shykes5• 12 years ago

We introduced libswarm at Dockercon today, along with a few other projects. Here is the presentation: http://www.slideshare.net/shykes/docker-the-road-ahead

tlrobinson• 12 years ago

Will videos of Dockercon talks be posted?

Tombar• 12 years ago

akerl_2• 12 years ago

They did confirm this morning that videos and slides will be posted. I've not heard where/when that will happen.

ndyg• 12 years ago

What does libswarm offer over unix sockets?

Is the idea for the project to be a rallying point to normalize protocols for different services that already talk over unix sockets?

philwhln• 12 years ago

Blog post interview with Mr Hykes on libswarm can be found here http://bit.ly/libswarm

Docker libswarm

Related Stories