Thanksgiving 2023 security incident

https://blog.cloudflare.com/thanksgiving-2023-security-incident

by nomaxx117 • 2 years ago

643 317 2 years ago

blog.cloudflare.com

jshier• 2 years ago

Fascinating and thorough analysis! I guess if you think an account is unused, just delete it!

phyzome2• 2 years ago

Probably safer to rotate the credentials and then schedule it for deletion later. Then if you discover it wasn't unused after all, you have an easier recovery... :-)

sebmellen• 2 years ago

The most surprising part of this is that Cloudflare uses BitBucket.

Cthulhu_2• 2 years ago

How so? It integrates well with the other Atlassian products they use.

infecto2• 2 years ago

Maybe but maybe not. I don't like Bitbucket but there are a number of large companies where they worry about using services owned by competitors in one of their verticals.

kccqzy3• 2 years ago

Bitbucket doesn't have to be a service. It can be an old-fashioned downloaded software that you install on your own machines. Not everything is SaaS.

toyg5• 2 years ago

Integrates with Jira and the rest of Atlassian's stuff, and it's just another git server at the end of the day.

gempir• 2 years ago

A lot of very big companies use Bitbucket, it's just a lot more cost effective than Gitlab/Github.

aitchnyu2• 2 years ago

Wonder how powerful is Scriptrunner for Jira. They got the security certifications but I cant tell how sandboxed it is.

itomato• 2 years ago

As well as can be expected in a company that gives Smartsheet access to Jira with an Admin Service Account.

https://github.com/BishopFox/sliver

"Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin."

https://blog.cloudflare.com/thanksgiving-2023-security-incid...

sevg• 2 years ago

Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket).
The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.

readyplayernull• 2 years ago

The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

Aha, the old replace-your-trusted-hardware trick.

zitterbewegung• 2 years ago

Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.

AzzyHN2• 2 years ago

In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high.

Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.

syncsynchalt• 2 years ago

Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been.

You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.

barkingcat• 2 years ago

I think they did have to do that far though.

Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.

Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

vasco• 2 years ago

What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come.

It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.

swyx3• 2 years ago

do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?

tgsovlerkhgsel• 2 years ago

It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.

Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

nolok2• 2 years ago

It would be a company ending event

Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.

(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)

sneak4• 2 years ago

Imagine getting in at the centre of a new Meet-Me room and having persistent access to key switches there.

This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).

meowface4• 2 years ago

Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit.

I can just imagine the attackers licking their lips when they first breached the data center.

Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)

tptacek5• 2 years ago

This is why old secops/corpsec security hands are so religious about tabletop exercises, and what's so great about BadThingsDaily† on Twitter. Being prepared to do this kind of credential rotation takes discipline and preparation and, to be frank, most teams don't make that investment, including a lot of really smart, well-resourced ones.

If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive.

† https://twitter.com/badthingsdaily?lang=en

akira2501• 2 years ago

It'd be more impressive if they actually got all the credentials.

It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.

schainks• 2 years ago

Having seen the small number of DEFCON talks that I've seen, I would have absolutely gone that far.

orenlindsey2• 2 years ago

Cloudflare is showing how to correctly respond to attacks. Other companies should take note.

ldoughty• 2 years ago

The nuclear response to compromise should be the standard business practice. It should be exceptional to deviate from it.

If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this.

Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.

marcinzm• 2 years ago

we were (for the second time) the victim of a compromise of Okta’s systems

I'm curious if they're rethinking being on Okta.

Icathian• 2 years ago

The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.

amluto• 2 years ago

Why not? Cloudflare already operates a system that can help customers to require SSO for access to their services — why not try to capture more of that vertical by becoming an IdP?

whalesalad• 2 years ago

They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/

BytesAndGears7• 2 years ago

My company will only give us new laptops that are preinstalled with Okta’s management system.

I am grandfathered in to an old MacBook that has absolutely no management software on it, from the “Early Days” when there was no IT and we just got brand new untouched laptops.

They offered me an upgrade to an M1/M2 pro, but I refused, saying that I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.

Since that would hugely disrupt my work, I can’t upgrade. Maybe I can use incidents like this to justify my beliefs to the IT department…

verve_rat• 2 years ago

Why do you need personal passwords on your laptop to do your work? I'm not understanding this.

yjftsjthsd-h2• 2 years ago

if I have my own personal passwords or keys anywhere on my work computer.

Well... don't do that? Why would you ever have personal anything on a work computer?

samcat116• 2 years ago

new laptops that are preinstalled with Okta’s management system

Okta doesn't make device management software, thats made by companies like Jamf. Okta can integrate with them but Okta isn't what manages your laptop at all.

I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.

Do not do this, its not a personal device.

ocdtrekkie• 2 years ago

I wouldn't use Okta at work, but as a network administrator, I also wouldn't allow your improperly managed laptop to talk to business resources (and I'd demand anyone overruling me sign a written statement demanding it to exempt me for responsibility for it). Wild you work somewhere that is letting you get away with that.

prg204• 2 years ago

IT departments do not care for the musings of self-proclaimed neckbeards. The arrogance of such users to comfort to security policy is a well known risk and usually grounds for swift disciplinary action.

giancarlostoro• 2 years ago

At a former employer, the company wanted everyone's machine to have full hard drive encryption, thankfully we were on Linux and the vendor they wanted to use was not compatible. Even so, I always encrypt my installs on laptops, not so much workstations (I didn't install it, and reinstalling would be a mess) since at that point you're physically in the office building and we have more serious problems.

twisteriffic• 2 years ago

This wasn't really an additional failure at Okta. This was credentials lost during the original Okta compromise that CloudFlare failed to rotate out.

Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.

jamiesonbecker• 2 years ago

Agreed, but Okta's still a $14B company.

cowsandmilk3• 2 years ago

This wasn’t a new compromise, but there were still two Okta compromises that impacted CloudFlare

January 2022: https://blog.cloudflare.com/cloudflare-investigation-of-the-...

October 2023: https://blog.cloudflare.com/how-cloudflare-mitigated-yet-ano...

bigbluedots• 2 years ago

They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023

It's fair to "punch down" imo as that's how the credentials were originally compromised. I'd agree with you if CF were trying to minimize their own mistake but that doesn't seem to be what is happening here

sophacles2• 2 years ago

How is Cloudflare ($384M in revenue, q3 2023) punching down at Okta ($584M in revenue, q3 2023) by stating exactly what happened.

If anything Okta is a bigger company (by revenue, by employee count) and they were founded a year earlier.

jrockway4• 2 years ago

Which "nation state" do we think this was?

meowface4• 2 years ago

For these kinds of attacks it's nearly always China, Russia, US, or sometimes Iran. 95% chance it's either China or Russia, here.

2OEH8eoCRo07• 2 years ago

When has it been the US?

toyg5• 2 years ago

Their response program being called "Code Red" is likely a hint.

godzillabrennus• 2 years ago

China.

jedahan• 2 years ago

The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.

tomschlick2• 2 years ago

No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.

CubsFan1060• 2 years ago

M247 is commonly used by VPN providers (https://www.reddit.com/r/PrivateInternetAccess/comments/8xwn...)

lijok• 2 years ago

Which nation state has good enough employment protection laws that they can take weekends off while doing recon on a top value target?

icepat• 2 years ago

Yes, they must have been a member of the Norwegian Foreningen Svartehattehackere. They are a very strong union.

toyg5• 2 years ago

Might be a coincidence. A certain nation-state is currently engaged in all-out war; the intruder might have been summoned to another, more urgent task.

papertokyo• 2 years ago

I assume the break is to have less chance of their activities be discovered and/or connected.

wubbert• 2 years ago

Israel.

belltaco• 2 years ago

Great write up.

Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories
They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).

Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.

Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin.
This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL.

Ouch. Full access to a server OS is always scary.

Aeolun5• 2 years ago

Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.

I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand?

2M Jira tickets is probably easily achieved if you create tickets using any automated process.

trollied• 2 years ago

They might create a JIRA ticket for each customer support interaction. Would make sense.

ummonk• 2 years ago

The number of repositories sounds really high. The number of tickets doesn't.

spenczar5• 2 years ago

12k git repos can happen if the team uses github enterprise with forking internally.

It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc…

Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.

sesm• 2 years ago

I can relate to this, it seems that code hosting providers push their users into having more repos with their CI limitations. I’ve noticed that with GitHub Actions, I assume Atlassian does the same.

OJFord4• 2 years ago

They probably have thousands of devrel/app engineer type example/demo/test repos alone - it doesn't say they're active.

2M tickets - in my 4.5y at present company we've probably averaged about 10 engineers and totalled 4.5k tickets. Cloudflare has been around longer, has many more engineers, might use it for HR, IT, etc. too, might have processes like every ticket on close opens a new one for the reporter to test, etc. It sounds the right sort of order of magnitude to me.

pkkim• 2 years ago

At least 25% of that is a single Jira project consisting of formulaic tickets to capture routine changes in production, and a large portion of that is created by automated systems. There may be other such projects too.

Source: former Cloudflare employee

keltraine• 2 years ago

Blog updated:

They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 194,100 pages)

Aeolun5• 2 years ago

Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand?

Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.

Cthulhu_2• 2 years ago

The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.

Aeolun5• 2 years ago

Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.

lavezzi• 2 years ago

Since December 18, new SEC rules require companies to report “material” cybersecurity incidents on a Form 8-K within four business days of their materiality determination.

The rule does not set any specific timeline between the incident and the materiality determination, but the materiality determination should be made without 'unreasonable delay'.

BytesAndGears7• 2 years ago

Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business.

Yes, they aren’t perfect. They do some things that I disagree with.

But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this.

Thank you for the blog post!

Xeyz0r2• 2 years ago

Nobody is perfect, but Cloudflare indeed inspires confidence. Especially thanks to cases where they don't hesitate to talk about the issue and how they resolved it. It's precisely these descriptions of such situations that demonstrate their ability to overcome any challenges.

overstay89302• 2 years ago

We're one of their larger enterprise customers and stuff like this makes it easy to get renewals approved easily, keeping engineers in the loop makes it such an easy sell.

nimbius• 2 years ago

Then, the advertisement worked.

- Insist that you have better integrity than your competitors

- share a few operational investigations after your latest security event

what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.

They mention a third-party audit was conducted, but thats not because they care about you. Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information, it needs to pass a yearly on-premise audit to ensure PCI compliance. if they didnt, major credit houses would stop processing their payments.

eastdakota5• 2 years ago

I was the one who made the call to bring in CrowdStrike. It had zero to do with PCI/DSS or any other compliance obligation. It was to 1) bring in a team with deep experience with a broad set of breaches; and 2) to make sure our team didn’t miss anything. The CrowdStrike team were first class and it was good to confirm they didn’t find anything significant our team hadn’t already. And, for the sake of clarity, no system breached touched customer credit card or traffic information.

autoexec2• 2 years ago

Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.

They did, and they admitted that it was their fault. I have to give them credit for that much.

They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023...The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

JakeTheAndroid2• 2 years ago

I am not sure where you're getting your information on requirements for PCI service providers. There isn't anything inside of PCI DSS that requires some sort of SOC report to be generated and distributed to customers. And Cloudflare does make their PCI AoC available to customers.

They clearly defined the scope of impact, and demonstrated that none of this impacts systems in scope for PCI. There was no breach to change management inside of BitBucket, and none of the edge servers processing cardholder data were impacted. They will have plenty of artifacts to demonstrate that by bringing in an external firm. So I am really not clear why you're bringing up PCI at all here. They made it clear no cardholder data was impacted so your perspective on the required "on-site" audits is moot.

Cloudflare operates two entirely different scopes for PCI; The first being as a Merchant where you the customer pays for the services. This is a very small scope of systems. The second is as a Service Provider that processes cards over the network. The network works such that it is not feasible to exfiltrate card data from the network. There are many reasons as to why this is, but they demonstrate year over year that this is not something that is reasonably possible. You can review their PCI AoC and get the details (albeit limited) to understand this better. Or you could get their SOC 2 Type 2 Report which will cover many aspects of the edge networks control environment with much better testing details. After reading that you can then come back to the blog and see that clearly no PCI scoped systems were impacted in a way that would require any on-prem audit to occur.

And they are not a card network. They are a PCI Service Provider because cards transit over their network. They are not at risk of being unable to process payments or transactions for their Merchant scope even if there are issues with their Service Provider scope. Because, again, these are two separate PCI audits that are done, testing two different sets of systems and controls.

And, as an aside, Cloudflare effectively always has on-prem PCI audits occur. Because the PCI QSA's need to physically visit Cloudflare datacenters to demonstrate not only the software side of compliance, but the datacenters deployed globally.

mynameisvlad• 2 years ago

I’m sorry did we read the same write-up?

Like I get cynicism, but they very clearly explained the lead-up to the accounts being compromised and the mistakes that caused that. They took full accountability of it. Which is frankly more than most companies dealing with security incidents. This entire write-up is more than most companies obligations or responses.

bimguy• 2 years ago

Nimbius, you sound like you work for a Cloudflare competitor.

No competitors were mentioned in Cloudflares article, they explained what kind of information was breached, nothing to do with payment/card info... so I doubt you even read past the first few paragraphs/conclusion.

ziddoap• 2 years ago

Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information

No payment card information was compromised.

tptacek5• 2 years ago

None of this would have had anything to do with PCI (nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches). At much smaller company sizes than this, insurance requires you to retain forensics/incident response firms. There's a variety of ways they could do that cheaply. They brought in the IR firm with the best reputation in the industry (now that Google owns Mandiant), at what I'm assuming are nosebleed rates, because they want to be perceived as taking this as seriously as they seem to be.

It's a very good writeup, as these things go. Cloudflare is huge. An ancillary system of theirs got popped as sequelae to the Okta breach. They reimaged every machine in their fleet and burned all their secrets. People are going to find ways to snipe at them, because that's fun to do, but none of those people are likely to handle an incident like this better.

I am not a Cloudflare customer (technically, I am a competitor). But my estimation of them went up (I won't say by how much) after reading this writeup.

encom• 2 years ago

Better hope you stay on their good side, and don't say anything their CEO doesn't approve of.

zelon882• 2 years ago

You actually believe that an intruder gained access to their KB/Tickets and didn't manage to get valuable information? That's not what Jira is for. If you know what Jira is for, and you're willing to run it on-prem, then you know the purpose of doing all that work is because you have something valuable to store in there.

I don't believe they didn't lose anything. That's not how this works, and most Jira/Confluence I've seen is loaded with secrets.

OJFord4• 2 years ago

The one service token and three accounts were not rotated because mistakenly it was believed they were unused.

Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?

stepupmakeup• 2 years ago

Rotating could have been manual and the person in charge wanted to save time. Stress could be a factor too.

htrp3• 2 years ago

blameless post mortem most likely

Great call out too

Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

OJFord4• 2 years ago

It can still be blameless though? The 'because' makes it sound like that's a correct reason to leave it; that the only error was thinking they were unused. (i.e. that it's fine to leave them if unused, only a problem if they're used)

i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.

phyzome2• 2 years ago

Betting they have a new item in their compromise runbook. :-)

OJFord4• 2 years ago

No I don't think so, I do think something's just difficult to say because of what they can't say, or they just neglected to say/didn't word it well, or something. i.e. a bug in the writing, not the post mortem itself.

Because if you take it exactly as it's written it's just too weird, I'm not a security expert with something to teach Cloudflare about err maybe don't leave secrets lying around that aren't actually needed for anything, that's not news to many people, and they surely have many actual security people for whom that would not even be a fizzbuzz interview question reviewing any kind of secret storage or revocation policy/procedure. And also the mentioned third-party audit.

crdrost• 2 years ago

I would assume that "believed" is not meant to be interpreted in an active personal sense but in a passive configuration sense.

That is, I'd expect there was a flag in a database somewhere saying that those service accounts were "abandoned" or "cleaned up" or some other non-active status, but that this assertion was incorrect. Then they probably rotated all the passwords for active accounts, but skipped the inactive ones.

Speaking purely about PKI and certificate revocation, because that's the only similar context that I really know about, there is generally a difference between allowing certificates to expire, vs allowing them to be marked as "no longer used", vs fully revoking them: a certificate authority needs to do absolutely nothing in the first case, can choose to either do nothing or revoke in the second case, and must actively maintain and broadcast that revocation list for the third case. When someone says "hey I accidentally clobbered that private key can I please have a new cert for this new key," you generally don't add the old cert to the revocation list because why would you.

londons_explore2• 2 years ago

Am I the only one who just sees a totally blank page?

Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...

chankstein38• 2 years ago

No, that's also what I see. I'm not sure why you're getting downvoted.

EDIT: re-opened the link a few minutes later and now I see the post

overstay89302• 2 years ago

Happened to me on my iPhone too

londons_explore2• 2 years ago

So after the Okta incident they rotated the leaked credentials...

But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.

fierro2• 2 years ago

The one service token and three accounts were not rotated because mistakenly it was believed they were unused.

This odd to me - unused credentials should probably be deleted, not rotated.

pbhjpbhj3• 2 years ago

This smells weird, surely? I'd be looking at who chose not to rotate those particular credentials.

1: "what are these accounts?"

2: "oh they're unused, they don't even appear in the logs"

1: "we should rotate them"

2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons"

pphysch2• 2 years ago

More likely: "no one has any idea what these old credentials do, so let's not touch them and potentially break everything"

mparnisari• 2 years ago

Agreed. This whole post reads as "I'm the victim" but they don't admit on the one mistake that snowballed

muzso• 2 years ago

The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).

In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.

kccqzy3• 2 years ago

Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold.

For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company.

Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.

toyg5• 2 years ago

Not if such citizens are sanctioned. Code Red. Hint hint.

elashri• 2 years ago

I think this probably was a name after the famous Code Red worm [1], not a reference to China.

[1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm)

eep_social2• 2 years ago

we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”.

Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat.

The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.

neilv3• 2 years ago

Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.

They had government engagements with Google's consent, and all those various engagements could be disclosed to each other?

If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?

slowbdotro• 2 years ago

Knowing google employees, it's coke. Lots of coke

_kb• 2 years ago

Payroll? You guys are getting paid?

Australians get the 'opportunity' to be part of that sort of that sort of espionage as a base level condition of citizenship [0].

As an upside, I guess it helps with encouraging good practices around zero trust processes and systems dev.

[0] https://en.wikipedia.org/wiki/Mass_surveillance_in_Australia...

owlstuffing• 2 years ago

For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company

Precisely. Particularly in the case of US businesses. Why bother picking a lock when you have both the key and permission?

orenlindsey2• 2 years ago

Cloudflare being compromised would be enormous. Something between 5 and 25% of all sites use CF in some fashion. An attacker could literally hold the internet hostage.

htrp3• 2 years ago

They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44.

Okta hitting everywhere

wepple• 2 years ago

They mention Zero Trust, yet you can gain access to applications with just a single bearer token?

Am I missing something here?

There’s no machine cert used? AuthN tokens aren’t cryptographically bound?

This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”

_cenw2• 2 years ago

these were service accounts used by third parties to provide jira integrations, not a user account

Bluecobra3• 2 years ago

If they are using Active Directory, wouldn’t a service account be no different than a regular employee account? Both a Jira service account and the CEO of Cloudflare are still Domain Users in AD. Granted, a service account should be way more locked down and have the least amount of access possible.

Bluecobra3• 2 years ago

Yeah it seems odd to me that their internal wiki, code repo, and Jira is exposed directly to the internet and arbitrary IPs could connect to it. Atlassian had a rash of vulnerabilities recently, who knows how many undiscovered ones still exist.

If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.

prg204• 2 years ago

You're not. The article makes no sense. They claim robust security controls but apparely lacked a proper accounting of service accounts with external access, especially with admin access to freakin' Jira.

Bluecobra3• 2 years ago

Thank you! The more I think about his it makes no sense to me. If a service account needs external access why can't they also whitelist connectivity to specific public IP addresses?

mmaunder2• 2 years ago

Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.

burnished• 2 years ago

Whats your point?

mmaunder2• 2 years ago

That this is messaged and received as a net win. It’s not.

arp242• 2 years ago

The source code next year is not the same as source code this year.

The customer data next year is not the same as the customer data this year.

BandButcher• 2 years ago

agreed, to me this is a big deal for CF. especially coupled with confluence documentation which most likely includes future plans and designs, org charts, meeting minutes... you could also find other easter eggs in any legacy code, almost all companies have undocumented backdoors

obviously a customer data breach would be worse but this is really no bueno

jbverschoor• 2 years ago

It's almost valentine

wowmuchhack• 2 years ago

Such a beautiful report and beautiful ownage.

Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise.

Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.

culopatin• 2 years ago

Im growing more and more annoyed at Cloudflare and their stupid “are you a human” crap.

joshbetz• 2 years ago

What makes them think it's a nation state?

j-rom• 2 years ago

To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

The thoroughness is pretty amazing

lopkeny12ko• 2 years ago

The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

This seems incredibly wasteful.

Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.

rjzzleep• 2 years ago

It is, but for most of these components there is no other choice since there is no way to guarantee that nothing was changed. lvrick would say that's what why want to attest everything.

Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.

perlgeek• 2 years ago

The sentence before...

To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers.

It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad.

Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.

this_steve_j• 2 years ago

This is an excellent report, and congratulations are due to the security teams at CS for a quick detection, response and investigation.

It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain.

These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now!

Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack.

Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius.

[1] https://docs.github.com/en/actions/deployment/security-harde...

prg204• 2 years ago

Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”.

Why didn't they start this effort BEFORE there was an incident?

we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials

Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this?

To ensure these systems are 100% secure

Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag.

Nothing was found, but we replaced the hardware anyway.

Well that is just plain stupid and wasteful.

We also looked for software packages that hadn’t been updated

Why weren't you looking for that prior to the incident?

we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials.

And yet they continue using Okta. The jokes just write themselves.

The one service token and three accounts were not rotated because mistakenly it was believed they were unused.

Wait, wait, wait. You KNEW the accounts with remote access to your systems were UNUSED and yet they continue to be active? Hahahahaha.

The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.

Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.

zelon882• 2 years ago

What I don't understand is how they got access to Jira yet you still insist there was no compromise.

The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data.

So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach".

You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.

nullpointer00• 2 years ago

I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand). --

Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases.

Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting?

TA hijacked a session token of a CF employee from a support ticket.

Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform.

How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA?

Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code.

Did the stolen employee credentials only have access to Atlassian?

Blog: TA gained access to a set of credentials

Does this mean multiple credentials got uploaded to the Okta support system?

Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November.

Does this mean the compromised token was long-lived?

Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.

I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise?

Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system.

Is this service token a bearer token? And without expiry. Is this like an API key?

TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway.

2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance,

So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry

TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment.

Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks.

3. A Bitbucket service account, which was used to access our source code management system

4. AWS environment that had no access to the global network and no customer or sensitive data.

Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA.

The only production system the TA could access using the stolen credentials was our Atlassian environment.

Mitigations:

Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.

What does hardening security protocol mean here? Is it techniques for D&R or something else

Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials)

I believe this means forced resets on employees (Okta users), right?

mmvasq• 2 years ago

Really good write up

Thanksgiving 2023 security incident

Related Stories