Gravitational Wormhole: WireGuard for Kubernetes

https://gravitational.com/blog/announcing_wormhole/

by aberoham • 7 years ago

170 31 7 years ago

gravitational.com

kevin_nisbet7• 7 years ago

Author here. Feel free to reach out if you have questions or thoughts about the project.

CreepGin• 7 years ago

Not gonna lie... I paused at reading the title and only clicked when I was expecting to read about physics.

(I had no idea of any of the Kubernetes products mentioned in the article prior to reading)

aberoham• 7 years ago

Hi Kevin, how does this compare to other ways of doing encrypted overlay within Kubernetes? Things such ebpf-based tools or sidecars?

kevin_nisbet7• 7 years ago

This is discussed a bit in the rationale and alternatives section of the original spec: https://github.com/gravitational/wormhole/blob/master/docs/r...

To try and summarize without speaking too much for projects that I'm not intimately familiar with, Wormhole takes the approach of automating the configuration of WireGuard to create a full mesh encrypted network.

Other approaches take a much more comprehensive approach, of trying to solve key signing, identity, revocation, offload to the kernel and intercept via sidecars (using ebpf), etc. These tools come with many features and an inherent complexity, where I believe there is room for a simpler tool that is easier to use and troubleshoot.

FiloSottile• 7 years ago

Hi! It’s good to see Wireguard become more usable in more scenarios.

Are you in contact with the Wireguard author, and how do you plan to support the Wireguard project?

kevin_nisbet7• 7 years ago

Hi.

No I'm not in touch with the WireGuard maintainers. I'd be happy to start a dialog.

As for contributing to the project, I'll likely take a crack at porting the netlink interface to the golang netlink project (https://github.com/vishvananda/netlink) so that the wireguard interface can be automated through golang. Right now, the project just automates this by execing the wg cli.

We're also integrating this with our commercial offering of gravity, which if we see traction I'll be making the case that we should contribute to the project financially as well.

otterley• 7 years ago

What advantage do you see using WG over TLS? A nice feature of TLS is that you can perform AuthZ on a per-service basis, and you get encryption for free.

TLS also has the nice property of working with all sorts of clients, even those outside the cluster that might not be using WG encryption at the network later.

kevin_nisbet7• 7 years ago

I would say it really comes down to the use case. If you're already using TLS and have the infrastructure for key management, issuing certs, etc. It's not worth replacing.

To me, where WireGuard comes in is for services that aren't traditionally encrypted, or where an extra layer of security is desired.

The example case that we've seen a few times is cluster DNS. The DNS traffic internal to the cluster is just plain DNS, meaning it leaks internal information and can be tampered with on an untrusted network. Even though CoreDNS does support DNS over TLS, it's a large effort to update clients in some organizations, that is better spent elsewhere.

This is where we see the opportunity for WireGuard, is ensuring all internal traffic is encrypted. For taking applications where the security model traditionally requires a secure network, and allowing them to be forklifted into a kubernetes cluster that provides those capabilities. I'm not saying this is an ideal model over using mTLS for every connection, just not everyone has what's needed to deliver mTLS everywhere.

Unfortunately, we do lose the identity features that TLS offers in the x509 certs. Nothing prevents running TLS for services that need it though on top of the WireGuard network.

So like everything else, I think there are pros and cons to both approaches, and it depends on what applies to the specific application.

I hope that helps.

orra• 7 years ago

I had previously come across an in-tree WireGuard extension for Flannel, https://github.com/coreos/flannel/blob/master/dist/extension....

However, Flannel's WireGuard extension looked way too experimental, and unmaintained. I realise it is also early days for Gravitational Wormhole, but Gravitational Wormhole looks promising.

solatic• 7 years ago

Hi Kevin, I'm wondering where you see Wormhole fit into the ecosystem alongside heavier systems like Istio, which offer many more features than just mTLS. Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so? Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired? Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?

kevin_nisbet7• 7 years ago

I'm wondering where you see Wormhole fit into the ecosystem alongside heavier systems like Istio?

I see wormhole as a simpler / lighter weight solution. Like flannel, but building an encrypted network instead of a vxlan (or xx plugin) network. The trade offs are optimized towards simplicity and being a lighter weight option with less capabilities.

Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so?

I don't think I have a deep enough understanding of Istio to know for sure how easy it is to use them together and have never tested this sort of configuration. I wouldn't consider wormhole a replacement for mTLS, as wormhole isn't capable of embedding identity of either end into a connection. You can think of it like establishing ipsec tunnels between networks.

Where I think wormhole would come into play in this theoretical setup, is if you want a subset of traffic to be encrypted, which either doesn't play well with istio for various reasons or you benefit from additional layers to the security model.

Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired?

Yes, exactly.

Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?

Yes. Although using wormhole doesn't require any application involvement. Wormhole operates as the networking configuration for the cluster, and simply uses an encrypted protocol for doing so. So any applications that run on say a flannel cluster, should not realize wormhole is there, we just provide a kubernetes network. Of course there is always an implementation detail we may do slightly different, but that's likely a defect in the implementation we would need to address.

leetbulb• 7 years ago

Very cool to see WireGuard being used in a mesh implementation. This reminds me of Weave[0] which has worked well for me. I'll definitely be experimenting with Wormhole.

[0] https://www.weave.works/oss/net/

OJFord2• 7 years ago

I thought I understood this, and that it replaced (and no doubt did a better job of) what I'd already done - WG to get nodes on the same network, CNI on top.

But requirement 2 confused me: > A Kubernetes cluster with IPAM enabled (--pod-network-cidr= when using kubeadm based install)

So, do node machines need to already be on the same network or not?

kevin_nisbet7• 7 years ago

Sorry for the confusion, this requirement isn't about the node networking. This is about assigning subnets to each node for the pod IP addresses.

Kubernetes includes an optionally enabled IPAM, that can allocate subnets from a large subnet, into the /24 subnet assigned to each node. This becomes the spec.podCIDR field in the node object when enabled.

Instead of implementing another IPAM for the node cidrs, wormhole uses the built in IPAM within kubernetes. The only thing is, the ipam needs to be turned on. On a kubeadm based install, this is done by setting the pod-network-cidr flags during init. This sets --cluster-cidr / --allocate-node-cidrs on kube-controller-manager.

OJFord2• 7 years ago

But what's the value of that flag if I'm going to use Wormhole only afterwards?

I set up a Wireguard /17 (iirc) mesh, and then kubeadm'd with --pod-network-cidr as that node's wg endpoint /24.

I suppose what I'm not understanding is how I can set up the cluster sufficiently to be able to kubectl apply wormhole if I haven't already setup a wireguard (or...) network between the nodes?

badloginagain• 7 years ago

Doesn't Istio provide inter-pod encryption, or am I totally off the reservation here?

nhoughto• 7 years ago

We were recently discussing creating something like for this, the current set of CNI options is wide but shallow. As mentioned in the article CNI is something you want to be as simple as possible, we’ve had trouble with weave and all it’s complexity. Flannel plus encryption is perfect!

sloppycee4• 7 years ago

Check out Scrambler: https://github.com/jameskeane/scrambler

It's basically this, but with ~100 lines of bash. As simple as it gets.

sloppycee4• 7 years ago

Big warning to readers:

If Gravitational asks you to complete an 'engineering challenge' they are using you for free labour.

See: https://news.ycombinator.com/item?id=19784787

alexk• 7 years ago

James, we respectfully disagree.

I think it's helpful if you (or anyone reading this) compare two projects side by side.

This is the project you have submitted:

https://github.com/jameskeane/scrambler

It is a prototype of IPSec CNI plugin using Bash. We are happy you are promoting your project online and as we've discussed during the interview, you could use it as you wish as it's your work.

This is the project that we have created at Gravitational:

https://github.com/gravitational/wormhole

It is using Go, Wireguard and is designed for production use under Apache 2 license.

I understand that similarities in the problem statement may have led you to believe that we could re-use the project idea, I'm assuring that you that we have not.

There are many OSS CNI plugins in the market solving the similar problem in a similar fashion (e.g. flannel with IPSec backend).

As a matter of fact that's the main reason we have picked this as a problem statement - to make it easier for candidates to look around for prior work.

For example, this is a submission of the engineer we've hired based on their similar execution of the assignment from April 2018:

https://github.com/webvictim/k8s-ipsec-cni

James, I wish you the best and hope that my response clarifies some confusion around this.

Sasha, CTO and co-founder at Gravitational

sloppycee4• 7 years ago

So you're admitting that you asked candidates to work on a project you intended to monetize?

Were you lying when you explicitly told me the opposite or did you decide after my submission (going by commit dates) to make this a "real" project?

You didn't steal my code; you stole my _ideas_ and are now presenting them as your own original work. Shame on you.

fulafel• 7 years ago

If you’re running Kubernetes in a network you don’t fully trust or need to encrypt all pod network traffic between hosts for legacy applications or compliance reasons, Wormhole might be for you

Is there a good analysis somewhere about how typical kubernetes setups trust the network and what badness an advesary could do with kubernetes network access? How sound is this default deployment setup from security POV?

For example I think DNS is used internally for service discovery, and incoming TLS is often terminated and proxied onwards as HTTP - those could be both MITMed, right?

laacz• 7 years ago

Just afraid that time will come, when searching for black hole, neutrinos, gravity will primarily yield these kinds of topics. Not actual ones.

Gravitational Wormhole: WireGuard for Kubernetes

Related Stories