Weave – The Docker Network

https://github.com/zettio/weave/

by ferrantim • 12 years ago

197 63 12 years ago

github.com

ferrantim3• 12 years ago

These are the same people who built RabbitMQ.

rabbitmq• 12 years ago

:-) The founding team, yes.

hintjens• 12 years ago

RabbitMQ has always been an extraordinarily good piece of software, from the very first versions. I think this is very good news for anyone using Docker.

netcraft• 12 years ago

as someone who is more developer than ops, I feel like the docker stuff is still changing fast and that the way you would use docker today will be very different a year from now; but that containers seem to be the way of the future - if I have no pressing need to change my server architecture does it make sense to wait for things settle or would it be more beneficial to get in and learn now and experience the changes and why they were necessary?

weavenetwork8• 12 years ago

It is definitely changing fast, but that is not a reason to wait before trying out the technology in order to learn about it. If nothing else, the community will welcome your feedback :-)

ipedrazas• 12 years ago

precisely because you're a developer you whould embrace docker with open arms.

Because docker removes all the trouble of running applications that you need for your development: databases, application servers, queues...

I love the fact that I can focus on my code and not on all those details that stole so much of my time.

pas3• 12 years ago

I'd like to see a post/writeup (or even an essay!) on how to do the magic "backing services" (http://12factor.net/backing-services) with Docker. And that's where I find Deis and other Docker orchestration systems lacking very much.

Sure, you can run MySQL in Docker, but it's a far cry from running it on native xfs with aligned partitions and whatever fancy you feel configuring. And since docker containers are very reusable, whereas backing data is by default should be persistent, my impression is that it's too easy to accidentally remove a docker container.

lemieux2• 12 years ago

So, basicaly, you're saying that you don't want to mess with that setup and use existing stuff... which you could achieve with everything as long as someone else do the setup for you. I don't see where Docker is better for a developper in this case... providing a vagrant box is exactly the same.

Someone1234• 12 years ago

Docker hopefully will be a little different in a year as it will offer solid security isolation (which is not the case now).

Right now if you run a Docker container and run something within it as root, if that application gets compromised someone can break out of that container and alter the master (due to the way Docker links container-root and system-root, a root user in a container is effectively a root user on the whole system).

Docker are working on allowing containers to run entirely in user-mode (thanks to improvements in LXC). This would mean that you can run a process as root within a container, and if that gets compromised there is near zero chance of leveraging that into damaging the master OS (since it will just have normal user privileges).

Here's an article about their progress (to usermode):

http://s3hh.wordpress.com/2013/07/19/creating-and-using-cont...

To quote Docker's own documentation[1]:

However, it has been pointed out that if a kernel vulnerability allows arbitrary code execution, it will probably allow to break out of a container — but not out of a virtual machine.

In other words, right now, a root process is likely able to escape a Docker container. You can use SELinux, AppArmor, and similar to somewhat mitigate that when it happens but neither are near as powerful as having that usermode isolation on the master.

If Docker is able to get usermode containers working, it will be very difficult for a Docker container to either alter other Docker containers or the master system (other than over the network, maybe).

[1]http://blog.docker.com/2013/08/containers-docker-how-secure-...

SEJeff2• 12 years ago

It has nothing to do with LXC, but the Linux kernels. LXC simply bundles all of the linux kernel namespace features together into one shiney thing to make containers. Conceptually, docker, via libcontainer, does the exact same thing.

jpgvm2• 12 years ago

I hope all of these Docker overlay networks start using the in-kernel overlay network technologies soon. User-space promiscuous capture is obscenely slow.

Take a look at GRE and/or VXLAN and the kernels multiple routing table support. (This is precisely why network namespaces are so badass btw). Feel free to ping me if you are working on one of these and want some pointers on how to go about integrating more deeply with the kernel.

It's worth mentioning these protocols also have reasonable hardware offload support, unlike custom protocols implemented on UDP/TCP.

shykes2• 12 years ago

I would love to take you up on that. I want to bake vxlan support into Docker upstream (as an optional plugin, like everything else).

Edit: Hi Joseph! Just realized it was you :)

jpgvm2• 12 years ago

Haha for sure Solomon, I have some other ideas now too that I am using Docker in production. I will hit you up to have a chat. :)

kijiki2• 12 years ago

If you're going down the path of VXLAN support in Docker, I'd love to talk. The company I founded built a Linux distribution for commodity hardware switches that can do VXLAN encap/decap in hardware at 2+ Tbit/sec. The same configuration that works in a Linux container host or a hypervisor works on the switches.

nolan@cumulusnetworks.com

darren0• 12 years ago

You have to pick either kernel or user space, not both. Either implement it purely in the kernel or purely in user space. In reality pure user space is faster, just look at Snabb switch https://github.com/SnabbCo/snabbswitch/wiki

wmf2• 12 years ago

Snabb and DPDK aren't magic though. Because they poll you have to dedicate a whole core to the vSwitch. Containers are a different case than VMs because the packets start in the kernel TCP/IP stack; to get into a userspace vSwitch they'd have to exit the kernel.

vishvananda• 12 years ago

Since you seem to have some kernel expertise, do you know if there is an easy way (via an iptables/ebtables plugin or some such) to get packets to switch namespaces? It seems like you could do a whole lot with just simple kernel packet rewriting if you could have an in-container-namespace rule to jump into another namespace before routing. You could do some analog of this with a veth device, but it seems like it would be much faster to just switch the namespace.

kijiki2• 12 years ago

"just switching namespaces" isn't easy, since a packet (in the kernel represented by an SKB) has to have an interface it came in on. The main role an veth pair has is to move the packet between namespaces, and to provide a new in interface, one that is visible in the new namespace.

Unless someone did something crazy, traversing a veth pair should just be doing a little bookkeeping on the SKB, no data copies at all.

roque• 12 years ago

OpenContrail can be used as an overlay network for docker: the overlay is implemented as a kernel module and comes very close to the theoretical maximum iperf performance on a server with 2x10G links. This script https://github.com/pedro-r-marques/opencontrail-netns/blob/m... can be used to associate any docker container created with "--net=none" with an overlay network. Better yet you get all the semantics of the OpenStack neutron API: floating-ip, dhcp options, source-nat, LBaaS. The kernel module also collects flow records of all the traffic and there is a web-ui that can display the analytics of all the traffic flows in your network. Install guide: https://github.com/Juniper/contrail-controller/wiki/OpenCont... Support on freenode.net #opencontrail.

thu• 12 years ago

This seems very nice. What would be the pros and cons of using Weave instead of Tinc ? I have used Tinc for a while[0] and, the end result looks very similar (i.e. there is not a nice command-line tool dedicated to use Tinc with Docker, but the high level description match).

[0] https://gist.github.com/noteed/11031504

weavenetwork8• 12 years ago

Just having a look at Tinc.. one difference is completely non-technical - Tinc is GPL but Weave is Apache licensed so aligns better with the whole Docker ecosystem. More comments to come.

grkvlt• 12 years ago

This is really interesting. I've been looking for a way to build in support for networking between Docker hosts in my clocker.io software, to simplify deploying applications into a cloud hosted Docker environment. I'd been young with adding Open vSwitch, but am going to try weave as the network layer in the next release. Will there be any problems running in a cloud where I have limited control over the configuration of the host network interfaces and the traffic they can carry, such as AWS only allowing TCP and UDP between VMs?

weavenetwork8• 12 years ago

TCP and UDP is all you need. (And the UDP can actually be quite broken, just not completely)

We've created weave networks spanning hosts on EC2, GCE and local data centres.

zobzu• 12 years ago

What this really means security wise:

http://i.imgur.com/Cko02do.png

cschneid• 12 years ago

Each `W` node in that graphic is a different physical host. So 3 kernels.

saryant2• 12 years ago

Question for weavenetwork: are containers addressable by hostname from other containers? Is there a good way to do that? I didn't see anything about it in the readme.

I suppose service discovery is out-of-scope for this project but having some sort of weave-wide hostsfile would certainly simplify it. Am I misunderstanding the project?

weavenetwork8• 12 years ago

Weave itself does not provide addressability beyond IP. That is the situation now, but this area is very much high on the agenda for us - service discovery is definitely in scope for weave.

Meanwhile, two points of note:

1) In weave the IP addresses can be much "stickier" than in other network setups, i.e. a moving a container from one host to another can retain the containers IP. That means it is quite amenable to relatively static name resolution configurations, e.g. via /etc/hosts files.

2) Since weave creates a fully-fledged L2 Ethernet network between app containers, name resolution technologies like mDNS that rely on multicast should work just fine.

So, in summary, while weave currently does not have any built-in service discovery, existing solutions and technologies for that should be relatively easy to deploy inside weave application networks, until weave itself grows these capabilities.

SEJeff2• 12 years ago

Perhaps you'd consider looking for the best way to "weave" weave into consul?

http://consul.io

greenimpala• 12 years ago

Err can anyone spot the tests in the repo? I cannot.

t0mas882• 12 years ago

This looks like a great idea. For me this was a missing piece two months ago when playing with Docker.

However I have strong doubts about the network performance, not only the overhead of the UDP encapsulation (that should be quite small), but mostly the capturing of packets with pcap and then handling them in user-mode. Looks like a lot of context-switches, copying and parsing with non-optimal code paths. Are there any benchmarks available?

My feeling is that this will consume large amounts of CPU for moderate network loads and thus be unusable with most NoSQL kind of systems that benefit from clustering across hosts?

weavenetwork8• 12 years ago

re benchmarks...publishing some is on the TODO list. See https://github.com/zettio/weave/issues/37. Informally, weave is pretty fast but it's not saturating Gbit Ethernet. As you say, capturing with pcap and handling packets in user space carries an appreciable overhead.

We've got some issues filed to look at pcap alternatives and also generally aim to improve performance.

re suitability for NoSQL clustering... depends on where the bottlenecks are; if you want to cluster for HA rather than scale, i.e. there aren't any real bottlenecks, then weave will work well. Same if you want to cluster because of CPU or memory bottlenecks. If, otoh, networking is the bottleneck then adding weave into the mix isn't going to improve matters.

t0mas882• 12 years ago

Ok, good to know. I think the challenge in taking another route than pcap is that you would need to do complex tricks with the existing network stack. Because if I understand the way Weave works you would really only need to do processing at the beginning of a connection and for some ARP requests etc while you don't need to do anything to existing TCP streams apart from encapsulating and forwarding?

baq• 12 years ago

can you compare this with openvpn, or any other vpn if we're at it?

brazzledazzle• 12 years ago

Has anyone compared this to rudder (https://coreos.com/blog/introducing-rudder/)?

bboreham• 12 years ago

Rudder has a central configuration (via etcd); Weave communicates network changes across all nodes by itself - as long as each new node knows how to contact at least one other node, it will learn of all other nodes and connect to as many as it can.

Also Rudder is Layer 3 and Weave is Layer 2 and Weave can encrypt traffic.

weavenetwork8• 12 years ago

The most significant conceptual difference is that in rudder sub-nets are tied to hosts. So containers on different hosts will always be on different sub-nets. By contrast, in weave containers belonging to the same application reside in the same sub-net, regardless of what host they are running on. In other words, weave makes the network topology fit the application topology, not the other way round.

GrantNelson• 12 years ago

Yikes, this looks scary. Just because you can do something doesn't mean you should. Networks are finicky, perf is king.

Weave – The Docker Network

Related Stories