“Worst cloud vulnerability you can imagine” discovered in Microsoft Azure

https://arstechnica.com/information-technology/2021/08/worst-cloud-vulnerability-you-can-imagine-discovered-in-microsoft-azure/

by fortran77 • 5 years ago

272 81 5 years ago

arstechnica.com

runnerup• 5 years ago

I always wonder if these types of vulnerabilities affect the most secret DoD contracts, or if those accounts are somehow sharded onto sufficiently separate systems/networks?

jeffbee• 5 years ago

DoD only has idiotic requirements like there has to be a fence around their private racks, not useful requirements like the global database isn’t open to the world.

dharmab• 5 years ago

Azure runs their Gov and China clouds as separate infrastructures with separate capabilities.

reallydontask• 5 years ago

I'd guess that the azure us gov cloud defaults to stuff not being open to the web, so if you spin up an azure function this is only available internally, but this is a guess

nonameiguess• 5 years ago

Nope. Look here (https://docs.microsoft.com/en-us/azure/azure-government/comp...) and search for Cosmos and you'll see this service is only approved for DoD Impact Level II systems. That is the lowest impact level the DoD offers and includes only systems exposed to the public, like the websites for recruiting and career descriptions. Any system handling controlled unclassified information or PII would not have been allowed to use this service.

And when you're talking about "most secret" contracts, those are all classified systems, which are on totally separate networks in totally separate private data centers located on military installations. Unless you've figured out how to break strong symmetric encryption using hardware-generated, hardware-loaded, pre-shared keys controlled in military arms rooms, that means you need physical access. It doesn't necessarily mean you need to break into a military installation. You can always try to break into a contractor SCIF instead, but that still isn't all that easy. My wife once saw some AT&T contractors digging too close to the wrong fiber line at her facility when she was working for the Navy at a contractor site and unmarked black SUVs were there to take those guys away to God knows where within two minutes.

That said, I don't doubt people try. When I was at Raytheon working at a secure facility, a Chinese company bought the property across the street, built a hotel at exactly the same height with windows facing us, and it was conspicuously almost always empty. I don't think demand for hotel rooms was financing that place.

csharptwdec19• 5 years ago

My wife once saw some AT&T contractors digging too close to the wrong fiber line at her facility when she was working for the Navy at a contractor site and unmarked black SUVs were there to take those guys away to God knows where within two minutes.

I used to do design/permitting for fiber networks and going around DOD areas/Fiber was always fun. You wind up having to submit your routes, and get vague feedback as to what you need to move, but never how far away/etc. Usually your best bet is to just go to the other side of the road if possible.

karmakaze• 5 years ago

It's bothersome that they keep saying 'primary key' in regards to what seems to be an 'access key' relating to a database.

Disappointing to see that it's arstechnica.

mikeryan2• 5 years ago

It's the term that MS uses, agree unfortunate name but you can't blame Ars.

https://docs.microsoft.com/en-us/azure/cosmos-db/secure-acce...

nuker• 5 years ago

You guys open DBs to internet?

mikeryan2• 5 years ago

If I had to guess I'd think the 30% of customers that Microsoft did notify we're the ones that didn't have access in some way secured via private network or IAM controls.

edoceo• 5 years ago

Sure, some of them. But usually have a pgbouncer in front.

oefrha• 5 years ago

Dupe: https://news.ycombinator.com/item?id=28322550

wiredfool• 5 years ago

Worst I can imagine would be this, but for active directory or IAM.

arduinomancer• 5 years ago

Whenever stuff like this happens I see people saying there should be legal consequences for leaking data.

By that logic should there be legal consequences for a company if someone breaks into their office and steals paper records?

geofft• 5 years ago

https://www.law.cornell.edu/wex/negligence

inetknght• 5 years ago

By that logic should there be legal consequences for a company if someone breaks into their office and steals paper records?

Comparing cloud storage and paper records is worse than comparing apples to oranges; it's comparing apples to celery. Sure they're both edible but they are vastly different organisms.

And yes, there can be legal consequences for a company if someone breaks into their office and steals paper records.

jereees• 5 years ago

An analogy of an analogy. Could somebody go one level deeper?

aranchelk• 5 years ago

It depends, in a regulated industry if you fail to do stuff like install a commercial firewall, keep facilities secured with proper door locks, etc. you may very well face fines or loss of a license to operate.

I think when people talk about consequences, they’re not regarding these companies as victims of crimes, rather as negligent actors.

vorpalhex• 5 years ago

If there is negligence involved, yes - an insecured building being broken into can absolutely have legal consequences.

That computers are involved should not remove negligence as a possibility to recover damages.

stan_rogers• 5 years ago

You mean like in bailment? That's a thing.

doliveira4• 5 years ago

If we're into bad physical analogies, I feel a better comparison would be the company itself sending an indexed copy of the records of all their clients and relying on ethics alone to keep someone from reading others' data.

ljm• 5 years ago

There already are consequences. Auditors will check the physical security of your office buildings if you're dealing with anything sensitive. If a breach happens later on and it turns out you cut corners on that physical security (or even somehow unwittingly compromised it) then you're not going to have a good time.

turminal• 5 years ago

This is more akin to leaving the door unlocked than to having a break in.

donmcronald2• 5 years ago

$40k? Lol. I’m poor and I’d have to think twice about disclosing it for that. How many government lists does having the ability to discover that type of exploit get you on?

I bet Microsoft would claim damages of $1+ billion if someone used that type of exploit maliciously by damaging data and undermining customer confidence in Azure.

What a joke. This should pay $1+ million.

cagenut• 5 years ago

The context you're missing here is the company/research-team that found this are ex-MS employees who started a company (Wiz.io) to help other companies secure their cloud hosting/environments. This is some of the most pure-gold viral content marketing they can dream of, they don't care about the $40k at all, its just to acknowledge this is non-trivial.

doliveira4• 5 years ago

That sounds like paying artists with "exposure".

scrollaway• 5 years ago

Uh, yeah, maybe, I'm not sure how much this changes things to be honest. I fully agree with grandparent this is $1MM reward territory. (Edit: sibling put it really elegantly, this is pay-with-exposure justification)

PR campaign or not, you don't spit on those kinds of rewards. And it's a bad look on MS to award 40k on one of the worst vulnerabilities to ever hit a cloud provider...

woofie113• 5 years ago

The context you're missing is it doesn't matter. Next person to discover a similar vulnerability in Azure will have a choice:

1. Disclose to Microsoft for $40k

2. Disclose to an intelligence agency for several times that

3. Disclose to criminals for several times that, in turn

The incentives are now publicly known to be misaligned, and as a potential Azure customer, I have to contend with the simple reality that a significant number of vulnerabilities will be exploited rather than reported.

$40k doesn't even come close to covering engineer time here. This should be a $1M payout.

oaiey3• 5 years ago

Because it is unethical and in most countries illegal :)

But you are right. They should pay them more.

donmcronald2• 5 years ago

I wouldn’t use it maliciously, but I would honestly think twice about disclosing it. I think that’s especially true for anyone that doesn’t have a way to gain from the publicity.

ramraj07• 5 years ago

What if you sell to the NSA itself?

Jyaif• 5 years ago

Maybe they have already sold the exploit months ago to everybody that would buy it?

YetAnotherNick• 5 years ago

The question is if this wouldn't have been reported what are the chances that this would have been exploited. Or what are the chances that Microsoft got saved from a hack in the wild from this knowing this vulnerability. If we assume that the codebase has one vulnerability for 10k lines, we would still get 10s of thousands of vulnerabilities. Any one of those could cost billion dollar to Microsoft, but patching one of those doesn't make the chance of getting hacked much different.

didntknowya2• 5 years ago

well the black market is always going to pay more. that's kinda why criminals tend to go there...

jtsiskin• 5 years ago

Yes, the next person will contact someone like https://zerodium.com/program.html instead

muststopmyths2• 5 years ago

I'm curious if Microsoft is suffering from a massive loss of generational expertise. At least right after XP we had to go through a security standdown where all code was reviewed and audited throughout the company. Subsequent features and services had to go through a pretty thorough security review at design time as well.

Over the past few years the number of security fiascos has been increasing. Is the internal Security team (forget what the org was called) dead now ?

I'm sure lack of dedicated testing is also a major factor, but do launches no longer have to satisfy a security review ? Maybe in the name of agile development ?

foepys• 5 years ago

I'm wondering about this, too. There are so many things being redone from scratch that I'm scratching my head about the why. Maybe Microsoft lost so many engineers from the 90s that they don't have the people anymore that understand the old code.

ta9882• 5 years ago

The problem here is due to a lack of basic security practices. There is nothing related to old code, it is brand new code and infrastructure that was deployed without audit.

pmlnr• 5 years ago

Understanding old code rarely gets one promotions.

stingraycharles• 5 years ago

At the time of XP, Microsoft had a fierce monopoly in the world and was absolutely dominating.

I wouldn’t be surprised if they started embracing the “ship fast” mentality with the cloud a bit more over the past years, in order to corner the market more quickly (which they did).

Additionally, I can also imagine that the release processes for cloud are fundamentally different than something like an OS. With the cloud, there’s a much larger mentality of releasing often, and it may be difficult to translate rigorous security audits to this workflow.

bink2• 5 years ago

I think this is more of an industry problem than a Microsoft problem. This was a feature added onto an existing service. The old waterfall method of security approvals might have caught this, but for most orgs that has gone the way of the dodo (and probably for the better).

Cosmos DB probably went through security review during the design phase and then again regularly as the code was written and improved. The Jupyter notebook functionality was also likely reviewed by security teams during the design, testing, and implementation phases. But once you're through those approvals most security review is going to be done via automated tooling with only occasional re-reviews and penetration testing at scheduled intervals. Automated testing is great at detecting vulnerabilities that have been discovered in the past, but really not good at detecting new classes of vulnerabilities, hard-to-detect authorization vulnerabilities, or how code integrates with other services.

Once the initial approval and code reviews had been done developers would still be committing code to the service and each line of code is probably not receiving a manual code review. Vulnerabilities like this are hard to detect even with a manual review as the testing team may not have great knowledge of all interconnected services, especially if it's an outside vendor.

mistrial9• 5 years ago

this sounds reasonable but, in this case stealing a key and using it for a man-in-the-middle attack, is what happened

Wistar2• 5 years ago

One thing: Given the rise of MS stock in the recent past and the pandemic, I have seen many long-time MS folks decide to retire, especially in the last 10–12 months.

redwood• 5 years ago

One thing is for sure, putting its astronomical revenue growth on the side, which presumably involves some fuzzy math... Azure is definitely the worst of the big three cloud providers

gatvol• 5 years ago

AZURE seems to be trying to play catch up -at least in feature parity - with AWS and to some extent GCP. IMO this sort of outcome is inevitable when moving at speed. While they do seem to have some sort of feature parity (on the tin type) and some good ideas, many of the services they have a pretty half baked once you scratch the surface, plus they're expensive.

riho• 5 years ago

Is it possible that there's also just more software in different domains being written, hence making it more difficult to keep track of everything that is going on?

muststopmyths2• 5 years ago

Shouldn't be. If I recall correctly, the process was fairly logical. One of the things you wrote down in your design doc was all the external surfaces (endpoints) your application/service had and how you were securing them.

You could literally not ship anything without this review, so they must have abandoned the process by now. Or just never extended it to "the cloud"

AzzieElbab• 5 years ago

After reading into details, this sounds like a fundamental design flow that was made easily exploitable via a new feature

Wistar2• 5 years ago

"The 8 most important execs who left Microsoft in 2021, and 3 new power players who joined"

https://www.flexi-news.com/the-8-most-important-execs-who-le...

varispeed• 5 years ago

I am the only one who thinks these bounties are ridiculously low? They should at least add a couple of zeroes to that one. Just wow.

jobigoud2• 5 years ago

If you add two zeroes you run the risk of rogue employees planting vulns themselves.

mirekrusin• 5 years ago

What kind of private key is it, and can it be called _private_ key if it exists remotely?

oaiey3• 5 years ago

It's private in the tenant. A SSL certificate encrypted web server also has the private key in the remote machine. Potentially hosting millions of other pages.

tialaramex2• 5 years ago

I don't see the phrase "private key" in this article. The only mention of the word "private" is in a mention of how the problem was reported to Microsoft.

I don't believe private keys were in use here, instead I expect these are secrets and the thing about secrets is, as Bono sang, "A secret is something you tell one other person" which means now there's another party that can lose it.

This is a reason to avoid using shared secret schemes and instead prefer private keys, because if access to Cosmos DB required a private key you as the only one with the key know if you gave that to Jupyter, if you didn't then it can't very well have given copies to anybody else. Unfortunately this is also why systems often don't use such an approach, enabling Jupyter across Azure likely earned somebody a promotion.

rixed2• 5 years ago

From the article:

A privilege escalation vulnerability allowed anyone with a Cosmos DB account to filch the private key for any other Cosmos DB account, by way of the Jupyter notebook functionality.

Could we not be served the same article?

doliveira4• 5 years ago

It's even hard to imagine how this might have happened. Is the service a shared Jupyter server (or group thereof) that somehow has access to everything and it's within this service that the access/authorization is implemented? I wouldn't expect IAM to work this way in a cloud service

I don't know how these security boundaries are usually implemented, but I would expect this bug to be way less plausible. Isn't this an architectural smell?

m0zg• 5 years ago

Microsoft isn't helped by the fact that there are also FB and Google in town, and they pay at least 20% more and have better engineers to work with as well, which makes work far less aggravating. So MS gets FB/Google rejects at this point, and there's a constant brain drain on top of that. But their internal culture has always been, "if we pay managers well enough things will work out". This breaks down when you actually have to do something hard (rather than rearrange buttons on Office app toolbars), as managers aren't the ones who do the actual work.

srigi• 5 years ago

I remember how one of the local Azure advocates blogged how there are two specialised security teams in Azure: 1. constantly improving security by patching products 2. acting like hacker(s) are already in the system and looking for them

I believe a 3rd group should be founded - our programmers did something stupid, looking for that.

“Worst cloud vulnerability you can imagine” discovered in Microsoft Azure

Related Stories