Real-world stories of how we’ve compromised CI/CD pipelines

https://research.nccgroup.com/2022/01/13/10-real-world-stories-of-how-weve-compromised-ci-cd-pipelines/

by usrme • 5 years ago

265 117 5 years ago

research.nccgroup.com

cerved• 5 years ago

a hardcoded git command with a credential was revealed

cries in security

i_like_waiting• 5 years ago

reminds me of tons docker tutorials, where all of them are doing default password in plaintext in docker-compose file

rietta• 5 years ago

I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git. The goal is a developer who has just cloned the repo should be able to run the setup script and have the whole system running with random seed data without futzing with copying secrets from coworkers.

dlor• 5 years ago

This is a great resource. I'd love to see more reports like it published. CI/CD pipelines often run with highly elevated permissions (access to source code, artifact repositories, and production environments), but they are traditionally neglected.

kevin_nisbet• 5 years ago

I suspect this is also an under considered area even in organizations with lots of attention to security. So would be good to also get more mindshare, as after we discovered some of our own CI/CD related vulnerabilities[1], it feels like most approaches we looked at had similar problems, and it took alot of research to find the rare solution that we could be confident in.

[1] https://goteleport.com/blog/hack-via-pull-request/

0xbadcafebee4• 5 years ago

Been there, done that, bought the t-shirt....

We had this "deploy" Jenkins box set up with limited access for devs, because it had assume-role privs to an IAM role to manage AWS infra with Terraform. The devs run their tests on a different Jenkins box, and when they pass, they upload artifacts to a repo and trigger this "deploy" Jenkins box to promote the new build to prod. The devs can do their own CI, but CD is on a box they don't have access to, hence less chance for accidental credential leakage. Me being Mr. Devops-play-nice-with-the-devs, I let them issue PRs against the CD box's repo. Commits to PRs get run on the deploy Jenkins in a stage environment to validate the changes.

This one dev wanted to change something in AWS. But for whatever reason, they didn't ask me (maybe because they knew I'd say no, or at least ask them about it?). So instead the dev opens a PR against the CD jobs, proposing some syntax change. Then the dev modifies a script which was being included as part of the CD jobs, and makes the script download some binaries and make AWS API calls (I found out via CloudTrail). Once they've made the calls, they rewrite Git history to remove the AWS API commits and force-push to the PR branch, erasing evidence that the code was ever issued. Then close the PR with "need to refactor".

In the morning I'm looking through my e-mail, and see all these GitHub commits with code that looks like it's doing something in AWS... and I go look at the PR, and the code in my e-mails isn't anyware in any of the commits. He actually tried to cover it up. And I would never have known about any of this if I hadn't enabled 'watching' on all commits to the repo.

Who'd have thought e-mail would be the best append-only security log?

Kalium• 5 years ago

I wouldn't say they are traditionally neglected, precisely. CI/CD systems are often treated as a place where devs hold infinite power with developer convenience prioritized above all else. Developers, who are generally not security experts, often expect to wholly own their build and deployment processes.

I've seen few things get engineer pushback quite like trying to tell engineers that they need to rework how they build and deploy because someone outside their team said so. It's just dev, not production, so why should they be so paranoid about it? Sheesh, stop screwing up their perfectly good workflows...

lox3• 5 years ago

We’ve been using Sysbox (https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.

xmodem3• 5 years ago

Could you elaborate a bit more how you get the containers into their own IAM roles?

nijave• 5 years ago

Not sure if this applies to the parent, but one way this Buildkite

Queues map pipelines to agents. Agents can be assigned IAM roles. If you want a certain build to run as an IAM role, you give it a queue where the agents have that role. For AWS, Buildkite has as a Cloud Formation stack that sets up auto scaling groups and some other resources for your agents to run.

orf• 5 years ago

AWS has IAM service accounts for containers. Comes for free with EKS, not sure how you’d do it without EKS.

Basically it adds a signed web identity file into the container which can be used to assume roles.

selecsosi• 5 years ago

For ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

lox3• 5 years ago

Yup, we have a sidecar process/container that runs for each job and assumes an AWS IAM Role for that specific pipeline (with constraints like whether it’s an approved PR as well). The credentials are provided to the job container via a volume mount. This allows us to have shared agents with very granular roles per-pipeline and job.

lima4• 5 years ago

Never head of Sysbox before. At a first glance, the comparison table in their GitHub repo and on their website[1] has a number of inaccuracies which makes me question the quality of their engineering:

— They claim that their solution has the same isolation level ("4 stars") than gVisor, unlike "standard containers", which are "2 stars" only (with Firecracker and Kubevirt being "5 stars). This is very wrong - as far as I can tell, they use regular Linux namespaces with some light eBPF-based filesystem emulation, while the vast majority of syscalls is still handled by the host kernel. Sorry, but this is still "2 stars" and far away from the isolation guarantees provided by gVisor (fully emulating the kernel in userspace, which is at the same level or even better than Firecracker) and nowhere close to a VM.

— Somehow, regular VMs (Kubevirt) get a "speed" rating of only "2 stars" - worse than gVisor ("3 stars") and Firecracker ("4 stars"), even though they both rely on virtually the same virtualization technology. If anything, gVisor is the slowest but most efficient solution while QEMU maintains some performance advantage over Firecracker[2]. These are basically random scores, it's not a good first impression–if you do a detailed comparison like that, at least do a proper evaluation before giving your own product the best score!

— They claim that "standard containers" cannot run a full OS. This isn't true - while it's typically a bad idea, this works just fine with rootless podman and, more recently, rootless docker. Allowing this is the whole point of user namespaces, after all! Maybe their custom procfs does a better job of pretending to be a VM - but it's simply false that you can't do these things without. You can certainly run a full OS inside Kata/Firecracker, too, I've actually done that.

Nitpicking over rating scales aside, the claim that their solution offers large security improvements over any other solution with user namespaces isn't true and the whole thing seems very marketing-driven. The isolation offered by user namespaces is still very weak and not comparable to gVisor or Firecracker (both in production use by Google/AWS for untrusted workloads!). False marketing is a big red flag, especially for something as critical as a container runtime.

Anyone who wants unprivileged system containers might want to look into rootless docker or podman rather than this.

[1] https://www.nestybox.com

[2] https://www.usenix.org/system/files/nsdi20-paper-agache.pdf

lox3• 5 years ago

I don't spend a lot of time on those comparison-style charts if I'm honest, but that is good (and valid) feedback for them. I also hadn't heard of it, I discovered sysbox via jpettazo's updated post at https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-d..., he's an advisor of nestybox the company that develops sysbox.

For the CI/CD usecase on AWS, sysbox presented the right balance of trade-offs between something like Firecracker (which would require bare metal hosts on AWS) and the docker containers that already existed. We specifically need to run privileged containers so that we could run docker-in-docker for CI workloads, so rootless docker or podman wouldn't have helped. Sysbox lets us do that with a significant improvement in security to just running privileged docker containers as most CI environments end up doing.

Just switching their docker-in-docker CI job containers to sysbox would have mitigated 4 of the compromises from the article with nearly zero other configuration changes.

pritambaral2• 5 years ago

They claim that "standard containers" cannot run a full OS. ... this works just fine with rootless podman and, more recently, rootless docker.
Anyone who wants unprivileged system containers might want to look into rootless docker or podman rather than this.

Perhaps I'm missing something, but I have been running full OS userlands using "standard containers" in production for years, via LXD[1].

[1] https://linuxcontainers.org/

ctalledo3• 5 years ago

Thanks for the feedback; I am one of the developers of Sysbox. Some answers to the above comments:

- Regarding the container isolation, Sysbox uses a combination of Linux user-namespace + partial procfs & sysfs emulation + intercepting some sensitive syscalls in the container (using seccomp-bpf). It's fair to say that gVisor performs better isolation on syscalls, but it's also fair to say that by adding Linux user-ns and procfs & sysfs emulation, Sysbox isolates the container in ways that gVisor does not. This is why we felt it was fair to put Sysbox at a similar isolation rating as gVisor, although if you view it from purely a syscall isolation perspective it's fair to say that gVisor offers better isolation. Also, note that Sysbox is not meant to isolate workloads in multi-tenant environments (for that we think VM-based approaches are better). But in single-tenant environments, Sysbox does void the need for privileged containers in many scenarios because it allows well isolated containers/pods to run system workloads such as Docker and even K8s (which is why it's often used in CI infra).

- Regarding the speed rating, we gave Firecracker a higher speed rating than KubeVirt because while they both use hardware virtualization, the latter run microVMs that are highly optimized and have much less overhead that full VMs that typically run on KubeVirt. While QEMU may be faster than Firecracker in some metrics in a one-instance comparison, when you start running dozens of instances per host, the overhead of the full VM (particularly memory overhead) hurts its performance (which is the reason Firecracker was designed).

- Regarding gVisor performance, we didn't do a full performance comparison vs. KubeVirt, so we may stand corrected if gVisor is in fact slower than KubeVirt when running multiple instances on the same host (would appreciate any more info you may have on such a comparison, we could not find one).

- Regarding the claim that standard containers cannot run a full OS, what the table in the GH repo is indicating is that Sysbox allows you to create unprivileged containers (or pods) that can run system software such as Docker, Kubernetes, k3s, etc. with good isolation and seamlessly (no privileged container, no changes in the software inside the container, and no tricky container entrypoints). To the best of our knowledge, it's not possible to run say Kubernetes inside a regular container unless it's a privileged container with a custom entrypoint. Or inside a Firecracker VM. If you know otherwise, please let us know.

- Regarding "The claim that their solution offers large security improvements over any other solution with user namespaces isn't true". Where do you see that claim? The table explicitly states that there are solutions that provide stronger isolation.

- Regarding "The isolation offered by user namespaces is still very weak and not comparable to gVisor or Firecracker". User namespaces by itself mitigates several recent CVEs for containers, so it's a valuable feature. It may not offer VM-level isolation, but that's not what we are claiming. Furthermore, Sysbox uses the user-ns as a baseline, but adds syscall interception and procfs & sysfs emulation to further harden the isolation.

- "False marketing is a big red flag, especially for something as critical as a container runtime." That's not what we are doing.

- Rootless Docker/Podman are great, but they work at a different level than Sysbox. Sysbox is an enhanced "runc", and while Sysbox itself runs as true root on the host (i.e., Sysbox is not rootless), the containers or pods it creates are well isolated and void the need for privileged containers in many scenarios. This is why several companies use it in production too.

mvdwoord• 5 years ago

The company I currently do contract work for, decided it would be best to have one large team in Azure DevOps and subdivide all teams in repositories etc with prefixes and homegrown "Governer" scripts, which are enforced in all pipelines.

Global find on some terms like "key", "password" etc were great fun. It really showed most people, our team included, struggled with getting the pipeline to work at all. Let alone doing it in a secure manner.

This is a 50k+ employee financial institute. I am honestly surprised these kind of attacks are not much more widespread.

MauranKilom• 5 years ago

Interesting to learn that credentials in environment variables are frowned upon. I mean, makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. I guess you'll want to limit the blast radius, but drawing these boundaries seems like a nightmare for everyone...

imachine1980_• 5 years ago

say from everbody to all sec teams

xmodem3• 5 years ago

makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too.

The answer is very much, 'it depends'. For oen thing, developers can run whatever code in CI before it's benn reviewed. I could just nab the env vars and post them wherever. If there are no sensitive env vars for me to nab and you have enforced code review, then I need a co-conspirator, and my change is probably going to leave a lot more of a paper trail.

Another risk is accidental disclosure - I have on at least two occasions accidentally logged sensitive environment variables in our CI environment. Now your threat model is not just a malicious developer pushing code - it's a developer making a mistake, plus anyone with read access to the CI system.

I don't know about your org, but at my job, the set of people who have read access to CI is a lot larger than the set who can push code, which is again a lot larger than the set of people who can merge code without a reviewer signing off.

but drawing these boundaries seems like a nightmare for everyone...

As someone currently struggling with how to draw them, yup.

nickjj3• 5 years ago

For one thing, developers can run whatever code in CI before it's been reviewed.

Yeah I don't think this gets talked about enough.

If you're talking about private repos in an organization then CI often runs on any pull request. That means a developer is able to make CI run in an unreviewed PR. Of course for it to make its way into a protected branch (main, etc.) it'll likely need a code review but nothing is stopping that developer who opened the unreviewed PR to modify the CI yaml file in a commit to make that PR's pipeline do something different.

Requiring a team lead or someone to allow every individual PR's pipeline to run (what GitHub does by default in public repos) would add too much friction and not all major git hosts support the idea of locking down the pipelines file by decoupling it from the code repo.

Edit: Depending on which CI provider you use, this situation is mostly preventable -- "mostly" in the sense that you can control how much damage can be done. Check out this comment later in this thread: https://news.ycombinator.com/item?id=29967077

marcosdumay• 5 years ago

Wait. Isn't any CI step done before review expected to be configured for a test environment?

I'm failing to understand how that procedure even works. How do you run the tests?

majkinetor2• 5 years ago

Some CI systems like GitLab can protected against this. It allows that only maintainers can run highly privileged pipelines that expose secretes as env vars. Many systems mask or remove passwords (Gitlab, Rundeck etc.)

asimpletune• 5 years ago

If there's malicious code already excepting with that level of privilege then, yes, you're pwn'd. However, I don't think that's what the main threat is. Probably the main threat is totally legitimate code, that has access to those variables, and now it may do something bad on accident.

It's sort of like how I tell my mom, "Even you don't want to know your passwords", when explaining that she should use a password manager. The reason in this case is because she may get fished and redirected to a fake website. If there's a password manager, and she doesn't even know her own password, then it won't recognize the certificate and she's safe. However, if she does know it, then she has the ability to leak it.

This is the same with environment variables, and a reason why secret storage is preferred over them. Think of environment variable as your mom knowing her password, and secret storage to her using a password manager. More or less haha.

detaro• 5 years ago

The article is more specific than that: They shouldn't be shared with code run by people/jobs who shouldn't have access to it. I.e. don't have secrets used for deploys in the environment that runs automatically on every PR if deploys are gated behind review by a more limited list of users.

mulmen2• 5 years ago

I mean, makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway?

Maybe. Back in the old days if you had the commit bit your badge didn’t get you into the server room. I get the impression a lot of shops are effectively giving their devs root but in the cloud this time, which isn’t necessary.

Lucasoato• 5 years ago

Is it just my impression or security in Jenkins seems much more challenging and more time-consuming than in GitLab? This post gives many examples where GitLab was attacked, so of course bad practices like privileged containers can lead to the compromise of a server independently by the technology used, but from my experience with Jenkins, I've seen using passwords in plaintext so many times, even in big companies.

ramoz2• 5 years ago

I don’t really like either. Both have traditionally been bad & related to on-prem legacy workloads. Building for SVN apps or teams new to git. It’s usually a mess.

nathanlied• 5 years ago

As someone adjacently interested in the field: care to elaborate on what systems you do like? It's always interesting to get new perspectives.

kiallmacinnes• 5 years ago

but from my experience with Jenkins, I've seen using passwords in plaintext so many times, even in big companies

I reckon this has to do with how the CI tools are configured.

Everyone knows you shouldn't commit a secret to Git, so tools like GitLab CI which require all their config be in git naturally will see less of this specific issue.

formerly_proven3• 5 years ago

Jenkins was also affected by numerous Java serialization vulnerabilities. It also used to be that any worker could escalate to the main Jenkins server pretty much by design, not sure what the current situation is.

0xbadcafebee4• 5 years ago

Jenkins is a tire fire, and security is just one of their tires on fire. Every aspect of it encourages bad practices.

2ion• 5 years ago

Jenkins is security game over if you overlook a small crucial configuration option or if you install any plugin (and it's unusable without some plugins), as plugin development is a free-for-all and dependencies between plugins are many. We basically decided that one instance of Jenkins plus slaves was unfixable and unconfigurable to use securely across multiple teams with developers of differing trust levels (external contributors vs normal in-house devs) and started fresh with a different CI design.

Jenkins is a batteries excluded pattern in one of its worst possible incarnations.

Jenkins is basically a CI framework for trusted users only. Untrusted workloads must not have access to anything Jenkins.

INTPenis2• 5 years ago

Many of these points are about running pipelines in privileged containers. Something I actually took extra time to resolve for my team. That's when I discovered kaniko first, and shortly after podman/buildah.

After that podman and buildah have gotten a lot of great reviews from people so I think they're awesome.

For an old time Unix sysadmin it just doesn't make sense to run something as root unless you absolutely have to.

Which also makes the client excuse in the article so strange, they had to run the container privileged to run static code analysis. wtf. Doesn't that just mean they run a tool against a binary artefact from a previous job? I fail to see how that requires privileges.

wdb2• 5 years ago

If you start fresh would you use Kaniko or Podman? Trying to find out the best way to avoid docker —privileged on custom Gitlab runner

INTPenis2• 5 years ago

Podman all the way. I abandoned kaniko as soon as I found podman.

That also eliminates the risk of accessing the host docker.

rawgabbit• 5 years ago

You would think by now we would have better credential methods. I still see username and passwords for system credentials. I see tokens created by three legged auths. I don’t get how that is an improvement. The problem is that most deployed code doesn’t have just one credential but a dozen. Multiply that with several environments and you get security fatigue and apathy.

mdoms• 5 years ago

The credentials gave the NCC Group consultant access as a limited user to the Jenkins Master web login UI which was only accessible internally and not from the Internet. After a couple of clicks and looking around in the cluster they were able to switch to an administrator account.

These kinds of statements are giving major "draw the rest of the owl" vibes.

https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d...

staticassertion7• 5 years ago

Thank you for writing this up.

Some thoughts:

1. Hardcoded credentials are a plague. You should consider tagging all of your secrets so that they're easier to scan for. Github automatically scans for secrets, which is great.

2. Jenkins is particularly bad for security. I've seen it owned a million and one times.

3. Containers are overused as a security boundary and footguns like `--privileged` completely eliminate any boundary.

4. Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak. I've thought about this a lot lately, especially after log4j. I think one pattern that may help is clearing the variables after you've loaded them into memory.

Another I've considered is encrypting the variables. A lot of the time what you have is something like this:

Secret Store -> Control Plane Agent -> Container -> Process

Where secrets flow from left to right. The control plane agent and container have full access to the credentials and they're "plaintext" in the Process's environment.

In theory you should be able to pin the secrets to that process with a key. During your CD phase you would embed a private key into the process's binary (or a file on the container) and then tell your Secret Manager to use the associated public key to transmit the secrets. The process could decrypt those secrets with its private key but they're E2E encrypted across any hops between the Secret Store and Process and they can't be leaked without explicitly decrypting them first.

teddyh3• 5 years ago

Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak.

The two real problems with environment variables are:

1. Environment variables are traditionally readable by any other process in the system. There are settings you can do on modern kernels to turn this off, but how do you know that you will always run on such a system?

2. Environment variables are inherited to all subprocesses by default, unless you either unset them after you fork() (but before you exec()), or if you take special care to use execve() (or similar) function to provide your own custom-made environment for the new process.

otterley2• 5 years ago

A foreign process’s environment variables are only readable if the current UID is root or is the same as the foreign process’s ID. As user joe I can’t see user andrea’s process’s envvars.

staticassertion7• 5 years ago

1. Environment variables are traditionally readable by any other process in the system. There are settings you can do on modern kernels to turn this off, but how do you know that you will always run on such a system?

I think that this would require being the same user as the process you're trying to read. Access to proc/pid/environ should require that iirc. You can very easily go further by restricting procfs using hidepid.

And ptrace restrictions are pretty commonplace now I think? So the attacker has to be a parent process or root.

2. Environment variables are inherited to all subprocesses by default, unless you either unset them after you fork() (but before you exec()), or if you take special care to use execve() (or similar) function to provide your own custom-made environment for the new process.

Yeah, this goes to my "easy to leak" point.

Either way though you're talking about "attacker has remote code execution", which is definitely worth considering, but I don't think it matters with regards to env vs anything else.

Files suffer from (1), except generally worse. File handles suffer from (2) afaik.

Embedding the private key into the binary doesn't help too much if the attacker is executing with the ability to ptrace you, but it does make leaking much harder ie: you can't trick a process into dumping cleartext credentials from the env just by crashing it.

colek42• 5 years ago

When you start doing security this way you end up chasing your tail. There are so many ways to mess it up.

There is a really good article that explains a different way of securing these systems though sets of attestations.

https://grepory.substack.com/p/der-softwareherkunft-software...

staticassertion7• 5 years ago

Start doing security what way, exactly? I defined a threat model and a mitigation. And it's pretty straightforward - a single keypair that ties environment variables to their deployment.

The article you linked to is about signing. It doesn't solve "I need to put an AWS key into the environment of a process".

notreallyserio• 5 years ago

I think your agent idea is good. I'd want to add in a way for the agent to detect when a key is used twice (to catch other processes using the key) or when the code you wrote didn't get the key directly (to catch proxies), and then a way to kill or suspend the process for review. Would be pretty sweet.

thomasmarcelis• 5 years ago

>In our final scenario, the NCC Group consultant got booked on a scenario-based assessment:
>“Pretend you have compromised a developer’s laptop.”

Most companies will fail right here. Especially outside of the tech world security hygiene with developer's laptops is very bad from what I have seen.

movedx2• 5 years ago

This is exactly why I both love and hate CI/CD.

Ultimately most CI/CD setups are basically systems administrators with privileged access to everything, network connected and running 24/7. It's pretty dangerous stuff.

I don't have an answer though, expect maybe to keep the CI and CD in separate, isolated instances that require manual intervention to bridge the gap on a case by case basis. That doesn't scale very well though.

hinkley3• 5 years ago

I think in general we put too much logic into our CI/CD configurations.

There is an argument to be made for a minimalist CI/CD implementation that can handle task scheduling and dependencies, understands how to fetch and tag version control, count version numbers and not much else. Even extracting test result summaries, while handy, maybe should be handled another way.

For many of us, if CI is down you can't deploy anything to production, not even roll back to a previous build. Everything but the credentials should be under version control, and the right people should be able to fire off a one-liner from a runbook that has two to four sanity checked arguments in order to trigger a deployment.

jeffjeff2• 5 years ago

A place I was at recently used makefiles in each project containing entries for "build", "deploy", "test" etc. All the CI did was use shared generic jobs to call the relevant makefile command.

These could just as easily be run from a developers machine with the right credentials.

contingencies• 5 years ago

A distributed system is one where the failure of a machine you've never heard of stops you from getting any work done. - Leslie Lamport

... via https://github.com/globalcitizen/taoup

jiggawatts4• 5 years ago

A weakness of modern secret management is that it isn’t.

A secret value ought to be very carefully guarded even from the host machine itself.

.NET for example has SecureString, which is a good start — it can’t be accidentally printed or serialised insecurely. If it is serialised, then it is automatically encrypted by the host OS data protection API.

Windows even has TPM-hosted certificates! They’re essentially a smart card plugged into the motherboard.

A running app can use a TPM credential to sign requests but it can’t read or copy it.

These advancements are just completely ignored in the UNIX world, where everything is blindly copied into easily accessible locations in plain text…

pxx2• 5 years ago

Except afaict SecureString doesn't reliably do that and shouldn't be used. https://github.com/dotnet/platform-compat/blob/master/docs/D...

jiggawatts4• 5 years ago

“It’s not perfectly secure so use a totally insecure alternative instead” seems like terrible advice.

tialaramex• 5 years ago

A recurring theme is that they obtain secret credentials from a service which needs to verify credentials, and then turn around and use those to impersonate the entity providing those credentials. For example getting Jenkins to run some Groovy discovers credentials Jenkins uses to verify who is accessing it, and then you can just use those credentials yourself.

To fix this - almost anywhere - stop using shared secrets. Every time you visit a (HTTPS) web site, you are provided with the credentials to verify its identity. But, you don't gain the ability to impersonate the site because they're not secret credentials, they're public. You can and should use this in a few places in typical CI / CD type infrastructure today, and we should be encouraging other services to enable it too ASAP.

In a few places they mention MFA. Again, most MFA involves secrets, for example TOTP Relying Parties need to know what code you should be typing in, so, they need the seed from which to generate that code, and attackers can steal that seed. WebAuthn doesn't involve secrets, so, attackers who steal WebAuthn credentials don't achieve anything. Unfortunately chances are you enabled one or more vulnerable credential types "just in case"...

Real-world stories of how we’ve compromised CI/CD pipelines

Related Stories