Scaling Kubernetes to Thousands of CRDs

https://blog.upbound.io/scaling-kubernetes-to-thousands-of-crds/

by hasheddan • 4 years ago

139 22 4 years ago

blog.upbound.io

This is great work! Even with just a few dozen CRDs kubectl can get a little slow on versions prior to these changes.

I am excited about crossplane because handling infrastructure via a reconciliation loop addresses most of my problems with terraform. Namely, managing/sharing a state file, importing resources that already exist, and ensuring that reality doesn't drift from terraform state.

theptip• 4 years ago

I do like this direction. What's the current state of the art for rendering the diffs? E.g. in Terraform your `terraform plan` will show whether the change requires a delete/create or just an edit.

Naively, if I just look at the raw CRD diff, that information won't be apparent.

negz5• 4 years ago

We've thought about a client-side tool to diff desired from most-recently-observed actual state, similar to tf plan. Nothing built yet though.

FWIW though we never automatically delete-and-recreate in Terraform fashion. Our thinking is that once Kubernetes supports it (or once we build an admission control webhook for it) we'd like to explicitly mark immutable fields as such, which would require you to explicitly delete-then-recreate. Little bit less declarative at the expense of being a lot less surprising given the constantly reconciling nature of our system.

moondev• 4 years ago

Awesome post. It's very cool to see how Crossplane is pushing the boundaries of CRDs, as well as sharing their insights with the community and upstream.

From the post it seems like interacting with Crossplane Managed Resources is done via a simplified "userspace" CRD/controller and not directly. Is direct interaction a Crossplane anti-pattern? It seems like both approaches have their own advantages.

negz5• 4 years ago

We certainly design Crossplane with the intent that end-users will interact with a "Composite Resource" API, not use MRs directly. Little bit like end-users using a Terraform module curated by their platform team rather than writing resources directly.

tchasnyk2• 4 years ago

Thank you for an excellent post on Crossplane usage in real life! This piece of technology is just starting its way to production systems, but it's definitely a great way to manage application infrastructure.

As far as I understand, use cases for Crossplane - it supposed to be used together with CD systems, such as ArgoCD. ArgoCD on its own add quite a lot of CRDs to cluster, so the problem highlighted in the article is very relevant to heavy used clusters

negz5• 4 years ago

Post author here - thanks for the kind words!

I wouldn't say Crossplane is _supposed_ to be used together with CD systems (that's optional) but that's certainly a common and practical use case.

tchasnyk2• 4 years ago

Is there a convenient way to limit the number of CRDs that Crossplane provider installs? For instance, someone wants to expose XRD only for one specific AWS service and completely ignore other AWS services

hasheddan• 4 years ago

Crossplane maintainer here -- this has come up a number of times in the community, but ultimately we would prefer for the system to be able to handle an arbitrary number of CRDs, whether they are being used or not (as evidenced by the work outlined in this post). Moving to filtering CRDs that a provider installs can lead to a confusing experience where the presence of a given package does not mean the same thing across clusters, which also complicates the dependency model Crossplane packages implement. We view Composition + RBAC as the mechanisms to define what a given entity can access (i.e. the API line).

paulfurtado• 4 years ago

I can actually say that not supporting this really does hinder crossplane adoption. At work we operate large shared kubernetes clusters. A team attempted to use crossplane and we immediately had to remove it from our clusters immediately because it made kubectl and a variety of other tools completely unusable due to all the rate limiting and how frequently it refreshes cached discovery data. They wanted to use crossplane for like 10-20 of its supported object types. Instead, they had to actually run crossplane inside of a vcluster because there is no option to filter the number of CRDs it creates.

So while I can get behind this sentiment philosophically, until something changes upstream in kubernetes, this makes it really difficult to use crossplane in a cluster used for anything else and it probably makes sense to offer a workaround until then. Also, in practice, any security conscious users running crossplane in production are probably going to give it AWS credentials scoped to only the resources they want to allow it to manage, so even if you do install all of the CRDs in the cluster, 90% of them won't work due to their AWS credentials anyway.

jontro• 4 years ago

Knit pick, the screen shot in the beginning of the post shows a postgresql server transformed into an mysql server.

negz5• 4 years ago

Haha I noticed that also shortly after I made this graphic. I've used it in a few talks and posts now and always worried someone would notice. Good eye.

andix• 4 years ago

Do I understand it correctly, that the kubernetes API is only used as a configuration management tool? So just that part, and not as container orchestration tool?

negz5• 4 years ago

Correct. We take the control plane of Kubernetes and extend it so that it can be used to configure/orchestrate anything, not just containers.

FBISurveillance2• 4 years ago

This looks like a multi-cloud variant of Google Config Connector?

tennix• 4 years ago

This is particularly problematic for hosted Kubernetes services like Google Kubernetes Engine (GKE), which often limit the CPU and memory available to the API server. These services can gracefully scale the API server up when they predict it will require more resources - e.g. when more nodes are created. Unfortunately at the time of writing most don’t factor in more CRDs being created and won’t begin scaling until the API server is repeatedly “OOM killed” (terminated for exceeding its memory budget).

Yeah, that's the limitation of hosted kubernetes services. Internally we used crossplane a lot. Though we haven't reached this limitation yet, we used another approach by deploying a standalone etcd + k8s apiserver as k8s (eks/gke) pods. And register the CRDs into this apiserver. So we can scale the etcd and apiserver easily.

dmlittle• 4 years ago

Depending on the size of your cluster, scaling out the apiservers is not necessarily cheap as the watch caches need to be initialized which puts load on etcd. Generally speaking, it's probably better to scale the apiservers vertically rather than horizontally.

voigt• 4 years ago

A couple of KubeCons ago there was a talk about a "Kubernetes without Containers". The idea was to utilize Kubernetes' API and extensibility, without the necessity to run a container runtime.

I can not recall the name of the project. Would love to see whether this idea got any traction. Sounds like it would be a nice fit for a crossplane maintained infrastructure.

smarterclayton• 4 years ago

You might be thinking about kcp?

https://github.com/kcp-dev/kcp https://www.youtube.com/watch?v=oaPBYUfdFE8

The team is developing the idea and has made a lot of progress. Fair warning, some of the recent progress is not accurately represented in the demos / docs.

I think the ideas behind kcp, along with this work from crossplane, is a great example of the Kube ecosystem exploring new opportunities (also vcluster and cluster api nested). Infrastructure control planes have a lot of potential advantages for streamlining complex environments for consumers.

edit: This work from crossplane is awesome because it benefits kcp which is trying to hit that next level (an example of giving back that benefits everyone)

CSDude• 4 years ago

We had a in house DAG of Pods that created thousands of CRDs. EKS team scaled our Api and master nodes to c5.18xlarge. It was in 2018.

pkrishnath• 4 years ago

i would create a vcluster with crossplane (using init helm deployments: https://www.vcluster.com/docs/operator/init-manifests )and control k8s API's CPU/Memory to manage thousands of CRDs (may be vcluster per provider)

Scaling Kubernetes to Thousands of CRDs

Related Stories