Connecting Kubernetes services with linkerd

https://lwn.net/SubscriberLink/719282/d8444c95089cd014/

by dankohn1 • 9 years ago

67 47 9 years ago

alpb3• 9 years ago

Great article, I've been able to learn more about why a cluster setup could utilize linkerd. However this statement confused me a little:

    However, Finagle is written in Scala, which Gould concedes is "not for everyone".

linkerd is written in Scala [1] and linkerd is written in Rust [2]. If he is optimizing for other people’s contributions, is this really the best way to go?

[1] https://github.com/linkerd/linkerd [2] https://github.com/linkerd/linkerd-tcp

kevinbowman• 9 years ago

Maybe it means at runtime, as opposed to at contribution-time?

daenney2• 9 years ago

linkerd is written in Scala [1] and linkerd is written in Rust [2]

Though your links are correct your statement isn't. Linkerd is written in Scala, but the Rust bit is linkerd-tcp, a TCP loadbalancer, which linkerd (the Scala one) doesn't handle as that covers HTTP and gRPC with optionally encrypting connections between linkers using TLS.

If he is optimizing for other people’s contributions, is this really the best way to go?

But is he optimising for other people's contributions? Using a C/Rust for something like a TCP proxy makes a lot more sense to me than having a JVM do that. And if he perceives Scala as "not for everyone", maybe adding Rust means he might get more contributions to those components?

I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two. By spinning it out as a standalone component that acts as a proxy, like linkerd is doing, Scala doesn't get in the way and what Finagle offers can be used with a mesh of microservices written in a myriad of languages.

alpb3• 9 years ago

Sorry, not typing -tcp on linkerd-tcp was completely a typo on my end.

I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two.

Thanks, this is the answer I have been looking for.

olix0r2• 9 years ago

Yep! That is exactly what I meant by that ;)

cookiecaper12• 9 years ago

I appreciate the sentiment behind linkerd et al but I think we are really over-complicating things and tying ourselves into knots by going head-first into this world. I can't help but think we're going to regret it in a few years.

The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated.

Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't.

For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok.

There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard.

One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with.

My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic.

Source: been converting entire company to Docker/k8s

kinghajj• 9 years ago

Building "from scratch" off the base image distributed by the distro itself is considered "the old way"

Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.

rubiquity• 9 years ago

Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the space is way out of wack.

SEJeff• 9 years ago

Well you can simply use GKE and that is all managed for you. Seemed to work quite well for Pokemon Go as they scaled out massively.

cookiecaper12• 9 years ago

I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important.

My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere.

Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines.

I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.

mwpmaybe3• 9 years ago

If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance)

I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1].

My understanding is that most public image repositories are unmoderated.

To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both.

most Dockerfiles are built on opaque images registered with an image registry

Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.

So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on.

0. https://hub.docker.com/_/ruby/

1. https://github.com/docker-library/official-images/tree/maste...

cookiecaper12• 9 years ago

I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images.

Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?

Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?

While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict.

Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.

Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is

a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)

and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort.

Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).

[0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...

lobster_johnson5• 9 years ago

It's perfectly normal to use base images and build your own. We do this for most things.

One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily.

Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one.

Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.

cookiecaper12• 9 years ago

I recognize that Docker and k8s are distinct, and I think rkt looks much smarter than Docker. Now that k8s has released a stable version capable of running non-Docker containers (as of last week), it's fair to separate them.

However, I'm still struggling to see the real value proposition offered by either.

The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.

jimktrains26• 9 years ago

Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics.

Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly.

Also, regarding https://linkerd.io/, what's with landing pages having almost no useful information on them. "linker∙d is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications" what's that even mean?

Fast, lightweight, and performant Handles tens of thousands of requests per second per instance with minimal latency overhead. Scales horizontally with ease.

Well, since you're calling yourself "transparent" I would hope it'd be fast!

Any language, any environment Runs as a transparent proxy alongside existing applications, integrates with existing infrastructure.

So, it's just proxying and retrying things and doing tls for me? Why isn't this better handled in the library I'm using to make the call in the first place?

Latency-aware load-balancing Balances request traffic using real-time performance, reducing tail latencies across your application.

Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.

Runtime traffic routing Provides dynamic, scoped, logical routing rules, enabling blue-green deployments, staging, canarying, failover, and more.

OK, some of that does sound useful, but some of it still feels like something the caller should handle.

Drop-in service discovery Integrates with most service discovery systems, decoupling applications from specific implementations.

What's this mean? Why not just use SRV records?

Production-tested and proven at scale Powers the production infrastructure of banks, artificial intelligence companies, social networks, government labs, and more.

I know why this is here, but it feels like _everyone_ claims the same thing, so, to me, it just becomes noise.

"linkerd allows you to drop in a transparent layer of application resilience and provides the operational affordances critical for modern, cloud native environments." —Oliver Gould, CTO, Buoyant

This sounds doesn't make sense to me, you can't "drop in" resiliency. It needs to be built. The whole thing just feels like it's all marketing and doesn't tell me exactly what problems it's solving for me as a developer.

------------

Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.

cookiecaper12• 9 years ago

Disclaimer: I'm still researching linkerd and haven't tried to use it.

One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level.

You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd.

Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.

Quick, turn back now from this whole containerization thing, because I assure you this is basically the answer for everything in this space. And you will be aggressively accused of being an old fogie for resisting it. Simplicity is just not appreciated anymore; at this point, "containers" may as well be an antonym for it (and by extension, an antonym for "sane engineering").

jimktrains26• 9 years ago

I'm personally a big BSD jails fan. The idea of containerization is nice, but people often forget that it requires tooling and care, just like anything. And isn't a magic bullet

daenney2• 9 years ago

> Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.

Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.

Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.

So? If you're happy with your solution, keep it! But maybe others haven't solved all these problems, or don't like your approach to how you've solved them for other reasons or don't even want to spend time solving these problems in the first place. For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.

Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.

jimktrains26• 9 years ago

Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.

Wouldn't those stats get out of date really quickly if you're not constantly talking to that machine?

For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.

But now you have a new component that you need to test and get to know. You didn't get rid of those 10 things, you just compartmentalized them in something else that needs to be maintained.

Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.

Very true, I just don't see the particular advantages to using more basic and well-known tools like DNS and TLS.

alpb3• 9 years ago

No programming language's HTTP/RPC stack supports SRV records. They are just not a thing. Changing your microservices to know about SRV is not a great solution either. Using something like linkerd frees your application from concerns like how discovery and transport is done. SRV is also not compatible with most components out there. e.g. you can't tell memcached/redis to resolve SRV records. You should take a look at how Kubernetes also does not do SRV records, instead it creates an IP address per service for intra-cluster communication. A/AAAA names work with legacy applications and are backwards compatible. That's pretty much why open source community has not been rolling out a major solution based on SRV.

jimktrains26• 9 years ago

Multiple A records are also fine. Also more supported as you pointed out.

Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.

I'm still not sure what service discovery is required beyond knowing the name of a service.

Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.

2202• 9 years ago

I don't think SRV records are the right answer; your networking layer should be k8s aware and issue a watch command on Endpoints; it'll be updated immediately-ish when the servers changes. This is similar to how finagle's zookeeper server set is supposed to work.

What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)

jimktrains26• 9 years ago

I don't think SRV records are the right answer;

OK, A/AAAA are more common.

your networking layer should be k8s aware and issue a watch command on Endpoints;

Or, I could use standard networking concepts and not build my entire app to suit one new thing that I may not want to stay with forever, or even more than just test.

What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production

There are already TCP/IP, DNS, DHCP, &c libraries for all languages, and that aren't tied to a specific stack. Why should I build into this stack and not into a more common subset?

lobster_johnson5• 9 years ago

The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.

Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices. Linkerd replaces this with something that is explicitly designed to deal with inter-app communication. (I believe one way to run Linkerd is as a small sidecar app inside the same Docker container the man app, as opposed to a big central server, which is also a neat way of doing it.)

Re "why not a library" — sure, you can build all sorts of discovery, distribution and resilience into the client itself. But that consolidates all of this intelligence into just that client. What if you use multiple languages? What about third-party client libs (e.g. Memcached, Redis, PostgreSQL) that you don't control and that you don't want to reimplement? You'll find yourself reimplementing the same thing over and over.

you can't "drop in" resiliency

I recommend reading the documentation, because drop-in resilience is really the whole point. The guys who built Linkerd previously developed Finagle [1], which is Twitter's RPC library, and when they talk about network resilience, they actually know what they're talking about [2].

[1] https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rp...

[2] https://monkey.org/~marius/funsrv.pdf

jimktrains26• 9 years ago

The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.

That's just DNS.

Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices.

Why not DNS round-robin?

I recommend reading the documentation, because drop-in resilience is really the whole point.

Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.

Artemis2• 9 years ago

Lyft has a tech similar to linkerd called Envoy: https://github.com/lyft/envoy. It is a bit lower level than linkerd (based on APIs rather than already integrated with common services) and written in C++.

I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.

stock_toaster• 9 years ago

  > Not having to run the JVM on every machine is also nice.

Personally, Envoy compiling to a binary makes it much more likely that I would experiment with it than with linkerd. I have hesitated trying out linkerd because we don't run anything JVM based in any of our stack, and we aren't currently excited about introducing any.

I am definitely going to take a look at Envoy. Thanks for referencing it!

zweizeichen• 9 years ago

Folks at Lyft/IBM/Google are building a service mesh partly based on Envoy over here: https://istio.io

webo• 9 years ago

The content of the article does not have anything to do with the title. There is not mention of how linkerd and k8s work together.

Connecting Kubernetes services with linkerd

Related Stories