The Guide to Kubernetes Labels

https://blog.kubecost.com/blog/kubernetes-labels/

by stumarkin • 5 years ago

60 24 5 years ago

blog.kubecost.com

ben_olsen2• 5 years ago

What's the general thinking around putting private identifiers (eg user name) in kube labels? That data would be going to a monitoring stack, eg Prometheus or hosted Cortex?

michaelmdresser• 5 years ago

I'm a little confused by this use case. Do you spin up individual pods for each user/group? Is this for other resources like secrets?

ben_olsen2• 5 years ago

We spin up a new namespace for each company/user on our platform. Looking to add labels as identifiers for general analysis and alerting.

AjayTripathy3• 5 years ago

Each use case is different, but I think expiring the data after 60d in your monitoring stack makes sense both from a scaling perspective of your monitoring stack and from a privacy perspective.

I wouldn't necessarily put _user_ data in labels, but team/product names and contact info of the coworkers responsible for the service seem fine to me.

brancz2• 5 years ago

Unless you explicitly allow them (via a flag) in your kube-state-metrics deployment, Kubernetes labels and annotations are not translated into Prometheus labels.

jrockway• 5 years ago

kube-state-metrics doesn't specifically matter. Prometheus has built-in Kubernetes service discovery, and you can drop labels/annotations or not. Most people never see this because they take an opaque blob from a vendor (helm chart, prometheus-operator), etc. -- but it's there: https://prometheus.io/docs/prometheus/latest/configuration/c...

kube-state-metrics does its own service discovery and outputs prometheus metrics that it cares about, but any app could do that.

alexjplant• 5 years ago

Uhhh...

Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod

...wat? The k8s docs say

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.

I've only ever seen them used in this fashion, i.e. declarative configuration facilitated by the control plane. Furthermore why would you have a _phone number_ as an annotation on a _pod_? If you're managing pods individually without a controller of some sort then you'd better have a really good reason.

GauntletWizard2• 5 years ago

I wouldn't use it on a pod directly, but I recommend that annotations on deployments include the oncall rotation that is responsible for them - And there's no real reason not to cascade that down onto pods as well.

wdb• 5 years ago

Interesting, do you have an example how this would look like?

webb2• 5 years ago

Yeah, I agree. I actually think that labels are a much better fit for this reference.

Instead, Kubernetes annotations are meant for sharing information with other colleagues. For example, you may use an annotation to record the phone number of the person responsible for a pod

But it can be super helpful to put owner/contact info for alerting purposes as teams scale.

I also agree that having this metadata on the controller is helpful.

dan_quixote• 5 years ago

Yup, used quite a lot for things like loadbalancer configuration (for instance): https://kubernetes-sigs.github.io/aws-load-balancer-controll...

stumarkin3• 5 years ago

We discussed and agree, PR just merged to update this reference: https://github.com/kubecost/blog/pull/26

mikesabbagh• 5 years ago

some monitoring tools like kiali expect some standard labels to be available like app=nginx. This article misses out on the very standard label "app"

stumarkin3• 5 years ago

Good catch, we're working on an update to the article that includes this label!

alpb• 5 years ago

I've been working on Kubernetes stuff for about 5 years now and I have never seen people "organize" their stuff with labels.

It sounds great in theory ("lemme query all my pods in staging", "lemme find secrets by the marketing team") but it's just not used for some reason. I think people like flat organization (e.g. "find deployment X") more than searching by labels.

stumarkin3• 5 years ago

Interesting! Does that present challenges when it comes to monitoring or managing at scale? Or when a project spans multiple clusters?

webb2• 5 years ago

I've actually worked with 100+ companies that use labels heavily. It's way more common when you have multiple engineering teams and nobody everybody knows all projects in a cluster. Also, it's common to start with namespace labels which are generally easier to manage.

raitom2• 5 years ago

Funny. I'm working on a specific issue with GCP GKE Autopilot. I'm running a bunch of workloads every day with 1 pod per client. Each pod has various running time and I wanted to know the pod cost per client.

I tried to use Kubecost but could not find any option at all to do what I wanted. My labels never appeared on the Kubecost UI. It's funny now to see an article about labels knowing that Kubecost itself doesn't seem to support them.

AjayTripathy3• 5 years ago

Kubecost fully supports labels by default-- however until recent versions we relied on kube-state-metrics for that data, which requires users to whitelist the labels they want to see. If you upgrade to our latest version, I'd be surprised if this still wasn't working for you. Once you've upgraded, you can reach out to support@kubecost.com if labels still aren't appearing or me personally ajay@kubecost.com and I'd be happy to help troubleshoot.

raitom2• 5 years ago

Ok my bad then. Thanks! I will take a look and see if it works for me.

reilly3000• 5 years ago

Labeling is incredibly useful but there is such a thing as overuse of them. One time I encountered a particularly fanatical user of labels and it made all of the logs difficult to view, index, and query. Also it can be oenerous to set all of those labels on every resource according to each key’s conventions. 3-5 taxonomies are great, 20 not so much.

The Guide to Kubernetes Labels

Related Stories