Kubernetes Hardening Guidance [pdf]

https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF

by cjg • 4 years ago

237 60 4 years ago

media.defense.gov

SoftTalker• 4 years ago

I didn't read this, but it's really tiresome to hear about having to "harden" systems in 2022. They should be "hard" by default. If you need to soften them to make them easier to work with internally, that should be what needs a checklist and instructions.

Kalium3• 4 years ago

Looking at the guide, much of the hardening is above and beyond anything any Kubernetes configuration can be expected to do. Thus there is literally no way to ship a pre-hardened Kubernetes that has to be softened with a checklist guide.

Much of it is about development practices. Kubernetes cannot scan your containers for vulnerabilities and misconfigurations for you. Kubernetes cannot ensure lease privilege practices in your containers for you. Kubernetes cannot do regular reviews of logs and configurations and security patching status for you. Kubernetes cannot monitor audit logs for you.

With all this said, it's worth taking a look at the guide. It goes far beyond suggesting a few changes to default settings. Perhaps it could have been better phrased as "Hardening the context and practices around Kubernetes" to avoid this confusion.

itintheory• 4 years ago

I think this is true, and the issue is deeper for kubernetes. A k8s cluster by itself is not particularly useful. In order to deploy real software you're going to need a bunch of other components, extensions, and software. Things like persistent storage, ingress controllers, service mesh, certificate managers, DNS, etc. All of those components require consideration from a security perspective.

numbsafari• 4 years ago

Even OpenBSD ships with post-install “hardening” guidance…

https://man.openbsd.org/afterboot

throwaway894345• 4 years ago

I didn't read this, but ...

stonemetal12• 4 years ago

Yeah, but then everybody bitches about ease of use. That other project down the street just works out of the box, while your project is mired in configuration hell.

More like we need a better Dev mode vs Production mode switch. Dev mode would be fairly insecure but would also refuse to run on the internet. Production mode would ease deploy but also "self harden".

Kalium3• 4 years ago

That kind of divide ultimately falls apart. Over time, things built in dev mode rely more and more on its insecurity and production systems get pushed that way.

This is why development systems need to be as production-like as possible. Otherwise people ship boring webapps that inexplicable rely on running as root in privileged containers and expect prod to enable this.

splix• 4 years ago

Is it possible to configure a Kubernetes cluster to run only _signed_ images? I.e., if someone has replaced a Docker in registry is should not be accepted by cluster.

kryptn• 4 years ago

You can probably use an Admission Controller with the ImagePolicyWebhook.

https://kubernetes.io/docs/reference/access-authn-authz/admi...

schainks• 4 years ago

Came here to ask the same thing. If I can sign git commits, can we sign images or even individual layers?

dilyevsky• 4 years ago

Yes, see https://github.com/sigstore/cosign/blob/main/USAGE.md for one example

cheriot• 4 years ago

There's an ecosystem of policy control tools built on top of k8s' ValidatingWebhooks. Check Open Policy Agent and Kyverno.

https://www.openpolicyagent.org/

https://kyverno.io/docs/writing-policies/verify-images/

mfer• 4 years ago

The short answer is yes. There are multiple tools that let you do this.

My personal favorite tool for this is Kubewarden[1] because its policies are web assembly. There is a specific policy just for verifying signatures[2].

[1] https://www.kubewarden.io/

[2] https://artifacthub.io/packages/kubewarden/verify-image-sign...

AgentME• 4 years ago

Being able to specify images by hash would be a simpler alternative.

Requiring signed images seems like an arbitrary place to require signatures, given that there's plenty of parts of kubernetes deployment configs that could be used to do damage and you need the whole thing authenticated. I guess a benefit of having signed images instead of content-addressed images is that they could be updated by a trusted person without needing to update any kubernetes deployments, but presumably you'd want to tell kubernetes to switch its running instances to the new images so that sounds like an incomplete solution.

paulfurtado• 4 years ago

If you use cri-o as the runtime along with an openshift container registry, it will actually verify signatures at the runtime layer. In addition to crio, podman and anything based on containers/image supports this too.

Really that just means a registry that sends back a header indicating it supports signatures and serves up the right signature endpoints. It's shocking this isn't more common.

But if you just want to check signatures at the cluster's point of entry, you can use an admission controller to block the pods from being created with unsigned images.

Havoc• 4 years ago

I find the fact that the Defense dept issues stuff like this almost more interesting than the content itself. Says a lot about what keeps them up at night

anonporridge• 4 years ago

The Space Force is cool, but honestly, we desperately need a Cyber Force.

e12e• 4 years ago

https://www.cybercom.mil/About/Mission-and-Vision/ ?

core-utility• 4 years ago

https://www.nsa.gov/

boston_clone• 4 years ago

https://www.cisa.gov/

mattpallissard• 4 years ago

A lot of what a cyber force would do falls under DHS

kodah2• 4 years ago

The Marines created a hacking MOS and a programming MOS in the 2200 specialty I believe.

Link: https://www.marinecorpstimes.com/news/your-marine-corps/2014...

jason-phillips• 4 years ago

Having worked exactly in this space, this is a perfectly normal document. Architects and principal engineers aren't bothered to the point of distraction but it is definitely something that's taken seriously. And it isn't just the DoD, it's every federal agency.

triggercut• 4 years ago

You may also find the US Government Accountability Office interesting.

They do a mix of various reviews and audits of government services and projects, and publish and targeted recommendations; But to form a basis for those reviews and audits they have a position on what "good" looks like and publish that as guidance.

[1] https://www.gao.gov/

kodah2• 4 years ago

It means the defense department or its vendors are using Kubernetes, which they are. That should keep them up at night.

preisschild• 4 years ago

Yup. Apparently even their newest stealth bomber uses it internally: https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe...

albrewer• 4 years ago

I used to work in a really niche engineering field. The amount of stuff I used from documents released by the US military was non-trivial. Never anything directly, but most of the time the docs were useful enough to shed deeper light and context on the trade publications and standards I was using when the history and intent behind a lot of calculations wasn't clear.

wingmanjd• 4 years ago

Another guide may be the CIS benchmark guide [1].

I can't attest to efficacy of this particular benchmark from defense.gov (we don't use k8s at $DAYJOB), but we've leveraged other benchmarks from CIS for various flavors of Windows/ Linux.

[1] https://www.cisecurity.org/benchmark/kubernetes

raesene92• 4 years ago

This is one of the standards and compliance guides you can use for k8s.

The other ones I'm aware of are

- CIS Benchmarks, there's coverage for Kubeadm, AKS, EKS, GKE, OpenShift and some others. This is a compliance guide focused on just k8s

- DISA STIG for Kubernetes, another compliance guide, they don't mention which distribution but it's kubeadm from looking at the paths mentioned.

- PCI Guidance for containers and container orchestration, this one is recent, it's a generic guidance targeted at container environments (docker, k8s etc) for PCI in-scope organizations but TBH it should work for most places (if that one's of interest, some more info https://raesene.github.io/)

Some more details on these https://www.container-security.site/general_information/cont...

Making security guidance for k8s is kind of tricky due to the number of distros and changes between versions (https://raesene.github.io/blog/2022/09/20/Assessing-Kubernet...)

alpb• 4 years ago

Prior discussion: https://news.ycombinator.com/item?id=30692794

jackconsidine• 4 years ago

Kubernetes, frequently abbreviated “K8s” because there are 8 letters between K and S

I'll be damned. I thought it was because the end kind of sounded like "8-es"

deathanatos• 4 years ago

Same as i18n (internationalization) and l10n (localization).

cmehdy• 4 years ago

a11y for accessibility also (and perhaps "ally" too?), but is a bit ironic given that screen readers wouldn't be making much sense out of that one.

paxys• 4 years ago

And a16z. It's a pretty common silicon valley convention. I was even asked to implement this in an interview once.

davewritescode• 4 years ago

XML-Canonicalization is xml-c14n too!

adolph• 4 years ago

* According to Tex Texin, the first numeronym of this kind was "S12n", the electronic mail account name given to Digital Equipment Corporation (DEC) employee Jan Scherpenhuizen by a system administrator because his surname was too long to be an account name. By 1985, colleagues who found Jan's name unpronounceable often referred to him verbally as "S12n" (ess-twelve-en).*

https://en.wikipedia.org/wiki/Numeronym

nazka• 4 years ago

You are not alone! I didn’t know

multani• 4 years ago

For those who are implementing these security guidelines: how do you ensure they have been correctly implemented?

Do you have any kind of static check program that can check beforehand that you are going to deploy a hardened kubernetes cluster? Do you have a "live" checker that can verify the actual configuration of a running cluster? Does it run all the time oronce in a while? Also , if you have an automated way of verifying your configuration, which program do you use?

I only know about Chef's Inspec and the CIS profiles that are available online, but the experience wasn't extraordinary and I was wondering what is used in the wild?

outworlder• 4 years ago

Maybe you can add them to OPA? (https://www.openpolicyagent.org/)

linuxftw• 4 years ago

The people that define the standards don't implement them. The people that implement them don't validate them. Government systems don't typically have any kind of system that periodically checks system configuration, though the systems might get reviewed every few years.

Kalium3• 4 years ago

A lot of these things can't be meaningfully statically checked. It has to be done by a working policy apparatus. If your org can't adopt, implement, and adhere to policy then there's no amount of standards documents in the world that will help you much.

There's no way in the world to statically and automatically check if your org regularly reviews configurations, responds correctly to monitoring alerts, ensures your developers adhere to least privilege principles, and so on. It has to be policy.

raesene92• 4 years ago

So there are various commercial systems that fall under CSPM (or KSPM sometimes) that are designed to assess compliance with different standards.

My purely personal opinion on this is that it's difficult to do well as even with compliance standards automating assessment isn't always possible

For example the CIS benchmark for k8s can't say "Never use cluster-admin" as there are some legitimate use cases, so instead it says "minimize the use of cluster-admin" which can't be fully automated as a check.

To do it well, a company should come up with their own spin on applicable standards, automate where possible (either with 3rd party or internal tooling) and then manually review the things that can't be automated on a periodic bassis (either with internal resource, or consultants)

sngz• 4 years ago

I'm implementing it right now actually and it's a combination of automated tools and someone third party actually going through and verifying all the controls.

gz52• 4 years ago

Well done. From the control plane section:

The Kubernetes API server runs on port 6443, which should be protected by a firewall to accept only expected traffic.

How are folks doing this in practice at scale? Managing ACLs for kubectl, admins, workflow systems, distributed worker nodes etc?

PubliusMI• 4 years ago

There's actually quite a lot.

gz52• 4 years ago

meaning lots of ACLs on both sides of the api server, or meaning lots of better (simpler, more secure) ways to do it?

ly3xqhl8g9• 4 years ago

For a bare-metal cluster one simple set and forget about it solution is to port forward another external port (e.g., randomish 51203) to the internal control-plane-ip:6443 and block port scan attacks using fail2ban, DenyHosts, psad, etc. This should prevent most of the attacks.

efortis• 4 years ago

The example of appendix A is a PoLA violation (Principle of Least Authority).

It has source code in the container. Use an external build server.

Kubernetes Hardening Guidance [pdf]

Related Stories