Kubernetes Networking: Behind the Scenes

https://itnext.io/kubernetes-networking-behind-the-scenes-39a1ab1792bb?source=friends_link&sk=b5b666c68375f7c60f76faa8571baa21

by onig1990 • 7 years ago

182 33 7 years ago

itnext.io

raesene92• 7 years ago

If getting a better understanding of exactly how Kubernetes network is of interest, I'd also recommend this episode of TGIK https://www.youtube.com/watch?v=IhbJ3ll4usI

In general actually, I've found TGIK really useful in improving my understanding of the underpinnings of k8s. They have a load of deep dive episodes that focus on a specific part of the solution and generally have a load of hands-on examples.

Already__Taken2• 7 years ago

Shame the audio got ruined somewhere along the way.

raesene92• 7 years ago

it fixes about 2:30-3:00 in.

k__2• 7 years ago

Did anyone do some measurements on k8s/container latency overhead in networks?

loriverkutya• 7 years ago

Yes. And they are _really_ easy to find. https://machinezone.github.io/research/networking-solutions-... https://itnext.io/benchmark-results-of-kubernetes-network-pl...

Edit: added one more link

ITNEXT• 7 years ago

I appreciate that you also shared Alexis Ducastel benchmark.

nazka• 7 years ago

I love the idea of Cilium using BPF and XDP. They are the right tool for this type of software. And they extend it to do really cool things with it.

k__2• 7 years ago

I'm sorry that I'm an _idiot_.

Thanks for the links.

CameronNemo• 7 years ago

The upcoming IETF hackathon in Singapore has a project for this subject.

https://mailarchive.ietf.org/arch/msg/hackathon/PjbOkxx-5O0o...

Already__Taken2• 7 years ago

It's surprising k8s being so new seems to have no concept of ipv6

the_duke• 7 years ago

It does, in alpha.

Soon to hit beta though: https://github.com/kubernetes/enhancements/issues/508

Most internal networks still run IPv4, and will continue to do so for a long time.

sascha_sl2• 7 years ago

Because ipv6 is not a simple addition, but a complex suite of new protocols meant to replace v4, except that is never going to happen in our lifetimes.

v6 should've simply been v4 with more bits

forgot-my-pw• 7 years ago

No backwards compatibility could be IPv6's biggest mistake. Now everyone is forced to maintain 2 IP addresses.

mrweasel• 7 years ago

IPv6 should not be an addition to Kubernetes, but the default. IPv4 could then be an addition for when you need external communication to IPv4 networks. There's no reason why IPv6 isn't the default for internal communication between containers.

We've been deploying IPv6 for at least 10 years and have hosts that are exclusively IPv6. Then along comes Docker and Kubernetes with their retro IPv4 only stack and now customer wonder why we're struggling to make containers work or suppliers that can't understand why we can't easily deploy the latest version of their software, which is only distributed as a Docker image.

londons_explore• 7 years ago

I feel like k8s networking is 10x more complex than it needs to be...

I think it's because they still want to use the socket API, but to "pretend" you can open a socket to a pod, or to a service, etc.

Instead, they should have defined a new API, like "connect('k8s:service:foobar?selectionstrategy=leastloaded')"

Then they could have made simple LD_PRELOAD shims to get their magic API to work with existing software without modification.

There would be no need for millions of IPtables rules, virtual IP addresses, port remapping, proxy processes, etc.

justinsaccount4• 7 years ago

That is basically how consul connect works: https://www.consul.io/docs/connect/native/go.html

atombender5• 7 years ago

Not as I understand it. It uses DNS resolution, so you connect to something like foo.web.service.consul. Kubernetes's KubeDNS works exactly the same way with Kubernetes service objects.

atombender5• 7 years ago

How is that different from today's use of KubeDNS?

Kubernetes' networking would arguably have been less complex (multiple CIDR subnets, proxying, etc.) if they'd embraced IPv6. But Google doesn't seem big on IPv6.

0xEFF• 7 years ago

API incompatibility is largely why AppEngine failed. Cloud adoption suffers greatly when the platform is not compatible with the data center.

pas3• 7 years ago

It's complex because K8S is used in a wide variety of complex environments.

Secure dynamic overlay networking, plus egress is inherently not a trivial thing. Then add customization/extendability on top.

The important thing to get is that it uses really simple building blocks, very UNIX-y. Every time a new container starts it starts a program (the CNI driver) and pipes in a JSON. And the driver does what it wants (executes iptables, BGP announces a prefix, commits something to its internal DB) and spits out some JSON that tells what IP the container got.

And .. that's it. No magic. Everything has to be configured, arguments passed to some daemon at cluster setup time, and so on. Yes, kubelet and apiserver both have hundreds of command line args, but [almost] all of them are very clearly associated with tuning a functionality (or passing some data forward to some daemon/driver/component) - and thus if you don't touch it but it works, yaay, you don't have to care about it (though of course it might turn out that you'll need to scale up/down/faster/lighter).

The socket API is fine, even with a new shiny API the network fundamentals would have to be managed just the same way. The Linux kernel would be the same, just as Ethernet, IP and so on. Reinventing all of those wheels just to provide transparent pod-to-pod networking doesn't seem to ever worth it.

schmichael• 7 years ago

On the Nomad project we've taken a distinct approach to networking: treat the IP layer as an implementation detail and attempt to abstract it away. This has led to a few design decisions:

1. Global node addressing - Given a region + node ID you can query any node from any other node.

2. Global allocation (pod) addressing - Given a region + allocation ID you can interact with individual allocations and tasks (metrics, files, logs, remote exec, etc).

3. Global name-based service addressing (via Consul) - whether via Consul's DNS interface, catalog API, or Connect service mesh, services discover one another by name instead of by IP or port. The service's address (ip+port) is hopefully an opaque implementation detail that never needs to be externally observed.

With all 3 of these taken together there's no need for overlays, distinct subnets, or unique IPs-per-allocation. Datacenters may even have overlapping subnets and Consul Connect will still allow services to communicate via global name based addressing. Even without Consul, almost all Nomad commands will route to a remote region and execute on the correct node regardless of subnet overlapping.

This does cause some friction at the edge: when translating from external IP based routing to internal services (aka ingress). However, there are lots of load balancers that integrate with Consul and therefore translate between IP and name based routing adequately.

Really any service unhappy with our "route by name, not by ip" design is going to be a bit awkward to use, but routable IPs and ports are selected and exposed by Nomad's scheduler - or advertised in Consul if the address is overridden at runtime.

Kubernetes Networking: Behind the Scenes

Related Stories