Is the Network the Limit? Dealing with Weave, CoreOS and Azure

https://arjanschaaf.github.io/is-the-network-the-limit/

by SanderMak • 11 years ago

23 13 11 years ago

arjanschaaf.github.io

jamescun• 11 years ago

Is the poor performance of Weave really surprising? Userland networking will always be lesser performing than that of native kernel networking (the show is somewhat better with technologies such netmap/PF_RING/DPDK). Weave itself would probably improve in performance if it utilised a TUN/TAP device + static routes as opposed to its current mangling of packets with pcap and BPF.

dpw2• 11 years ago

We did try TUN/TAP. It seemed to be slower than pcap.

Our answer to weave performance concerns is on its way: http://blog.weave.works/2015/06/12/weave-fast-datapath/

We're really interested to hear how weave fast datapath works out in all kinds of environments, which is why we put the preview out. It's a shame that Arjan's post did not include numbers for FDP, but maybe he'll be able to include them in an update.

justincormack2• 11 years ago

Indeed tun/tap performance is terrible, it seems to take about a whole CPU at gigabit.

slintes• 11 years ago

regarding fast data path: Arjan's colleague Paul posted some new numbers: http://paulbakker.io/docker/docker-cloud-network-performance...

api• 11 years ago

User space networking overlays impose a penalty just like and for much the same reason as dynamic language, garbage collection, etc. -- it's a trade-off between versatility and performance.

The best networking performance is achieved by binding directly to the metal using SOCK_RAW, but that also gives you the least flexibility.

pyvpx• 11 years ago

oddly, there is a growing trend of going userland and skipping native kernel because, well...the kernel is slow. userland can be made fast.

I think seeing Weave implemented in something like say, Snabb Switch, would at the very least be interesting -- from a performance standpoint.

bboreham• 11 years ago

A major point of weave is that you don't have to change your code, so it has to work with the kernel network APIs.

jbeda• 11 years ago

I suggest you also look at GCE for network performance. It is something that the Google networking team has been working hard on and, because of the unique network fabric at Google, should scale to lots of VMs.

Also, you can get equivalent "IP per container" using advanced routing with GCE. Flannel has a GCE specific back end where it inserts routes for each subnet assigned to each range. That will eliminate the extra level of encap.

(Disclaimer: I started GCE but am no longer at Google.)

jsolson2• 11 years ago

Hi Joe :)

Conveniently, Amin Vahdat recently shared some content publicly about why it should scale to lots of VMs: https://www.youtube.com/watch?v=FaAZAII2x0w

(And, to others, disclaimer: I work on some bits of GCE networking at Google)

__monadic• 11 years ago

+1 to @jbeda, but let me offer some comments for the avoidance of possible confusion among potential Weave users.

Weave aims to deliver a completely portable network. In other words if you create an application using Docker container and a Weave network, that should be able to run anywhere. And all this should be 'magically simple'. Once you have decided where to run your app, you may wish to trade portability for performance gains. For example by using GCE networking (or Azure, or ...).

To date Weave achieved portability by sending some packets (inter host) through user space. This has big benefits in terms of ease of use eg. dealing with wide area networks, multicast and firewalls. But under load it performs worse than kernel-only models.

We now have Weave "fast data path" - http://blog.weave.works/2015/06/12/weave-fast-datapath/ ... This aims to deliver close to line performance with portability and extreme ease of use.

There may of course be yet more optimisations that end users wish to investigate. If you are willing to sacrifice some portability you could certainly make use of fast networks provided by a specific public cloud. We haven't yet seen a strong need to support this, but it is certainly a reasonable thing.

(Disclaimer: I work at Weaveworks)

Is the Network the Limit? Dealing with Weave, CoreOS and Azure

Related Stories