Learnings from kCTF VRP's 42 Linux kernel exploits submissions

https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html

by tim_sw • 3 years ago

206 85 3 years ago

security.googleblog.com

jpgvm• 3 years ago

Shame but understandable. io_uring (like netlink) is essentially a completely new interface to kernel-land things so until security boundary stuff is sorted out this is probably the best move.

hedora• 3 years ago

No. It was completely new a few years ago, and they’re running kernels from three years ago that are missing security patches from two years ago.

viraptor• 3 years ago

I'd like to see a source for this. Even if they do run older kennels, it doesn't mean they're missing patches. Google has significant involvement in security and enough engineers to roll their own systems. I don't expect they'd be missing any relevant patches, much less for years. (I know this was an older patch in this case, but they did mention other mitigations, so it may have been a considered choice not to backport)

Given how large they are as a target, we'd also hear about them being exploited all the time if that was true.

sirdarckcat• 3 years ago

[citation needed]

* ChromeOS, COS and Android are opensource, and they are all up to date. Mind elaborating?

greesil• 3 years ago

Linus?

jsiepkes• 3 years ago

There is a steady stream of security issues with io_uring since its inception. You can find security issues less then 3 months old for it [1].

[1] https://seclists.org/oss-sec/2023/q2/135

compiler-guy• 3 years ago

Google actively patches all its kernels and keeps them pretty up to date. The problem isn't the vulnerabilities Google or the public knows about, its the ones it doesn't know about, and the rate of discovery is too high for io_uring to be used in certain situations.

"As such, io_uring continues to be actively developed, but it is still affected by severe vulnerabilities and also provides strong exploitation primitives."

summerlight• 3 years ago

At least for my workstation, it doesn't seem so? The kernel is only 2 month ago, and I tend to postpone my OS update to the very last moment unless it's forced. I'm pretty sure that the production even got much more frequent updates.

ajross• 3 years ago

[Google is] running kernels from three years ago that are missing security patches from two years ago.

This is ridiculous. No, they aren't. No one is. Kernels on production Linux devices[1] are cut from an LTS branch and get prompt and expertly audited security backports of everything anyone knows about. Now, sure, that process does break down occasionally; but this is an area being shepherded by some of the best engineering effort in the world. In fact I'd go so far as to say a regularly updated LTS kernel is probably the software product that is most responsive to security issues.

[1] All the ones I know about anyway. Surely there are some gadgets being less prudent, but Android and ChromeOS are absolutely not on the list.

tptacek2• 3 years ago

This is an editorialized title, and should instead be "Learnings from kCTF VRP's 42 Linux kernel exploits submissions".

KerrAvon• 3 years ago

The io_uring finding is the only novel tidbit.

dang2• 3 years ago

Thanks! Changed now. Submitted title was "io_uring disabled across Google platforms".

NavinF4• 3 years ago

IMO the original title was way better since it doesn't bury the lede. For better or worse, very few programmers care about "Learnings from kCTF". A lot of programmers care about io_uring being disabled.

DrBazza2• 3 years ago

"Learnings" - should be lessons. We don't go on 'drivings' or 'travellings' we go on journeys.

Which lead me down this rabbit hole - https://english.stackexchange.com/questions/546959/why-is-le...

Observation bias, but I first noticed this word popping up in tech talks from Microsoft first.

insanitybit4• 3 years ago

How odd that a language stackexchange wouldn't just immediately quote Fowler's, which has a whole section on such words, as I recall.

tveita• 3 years ago

There's viewings, showings, hearings, shootings, cuttings, smearings, belongings, holdings... It's not like the pattern doesn't exist even if it doesn't hold for all verbs.

p-e-w6• 3 years ago

As the Linux kernel is a key component not just for Google, but for the Internet, we started heavily investing in this area. We extended the VRP's scope and maximum reward in 2021 (to $50k), then again in February 2022 (to $91k), and finally in August 2022 (to $133k).

So an award roughly equal to four months' salary for an average Silicon Valley engineer is considered "heavily investing" in security, when it comes to potentially industry-shattering exploits?

I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.

vasco3• 3 years ago

Who is lawfully paying out more for bounties on open source projects they didn't even start?

You also seem to discount the monetary value of not becoming a criminal by selling exploits unlawfully. Do you want to be the person that reported a vulnerability to increase security in the world or to give more power to spy agencies or crime syndicates?

TechBro8615• 3 years ago

How do you define "selling exploits unlawfully?" I'm not aware of any law that says only Google is allowed to purchase information disclosing exploits in open source software.

vlovich1232• 3 years ago

Who is lawfully paying out more for bounties on open source projects they didn't even start?

That’s kind of begging the question. Google invests a non trivial amount employing Linux kernel developers. How do these bounty payouts compare in terms of the amount they pay for their own engineers and security researchers? This also isn’t a donation. This is rewards for work Google finds valuable enough that they’ve changed several products based on this work. They also benefit immensely from all the free labor other companies and volunteers out into the kernel.

I think OP raised an extremely fair critique of the amount being too small.

p-e-w6• 3 years ago

I'm pretty sure it's "lawful" to sell exploits to intelligence agencies of your own country. It wouldn't make any sense for it not to be, considering that the same people who control those agencies also control what's legal and what isn't.

gretch3• 3 years ago

Yup, GP underestimates the difficulty of being a serious criminal.

If you try to sell your exploit on the black market for 133k but mess up and walk into a federal agent honey pot, now your in jail for 10 years.

Even if you did successfully do it, imagine how hard it is to explain where an extra 133k came from in your bank account.

eru• 3 years ago

Those are just the maximum rewards, not the total rewards paid?

jplona• 3 years ago

Reading a bit futher, the total for the program comes to almost 2 million which seems heavier than the per-award number.

Veserv• 3 years ago

That is not very much money. You can not even start a single McDonald’s restaurant with that little money.

txdv• 3 years ago

One way is legal, the other is not wirh shady buters, etc.

renewiltord2• 3 years ago

Note how none of the companies that don't do this find themselves criticized. This is called the Copenhagen Interpretation of Ethics.

In fact, despite so many of us personally relying on this software and giving nothing, we are all immune from criticism.

I suspect that if I had offered $10 into a bounty I would be responsible too for not paying more than lunch in SF. Haha!

p-e-w6• 3 years ago

My criticism doesn't concern Google's ethics, it concerns their practicality.

Common sense says that with bounties like that, they won't attract the most serious exploits, which isn't an ethical problem, but can absolutely be a security problem.

NavinF4• 3 years ago

It's a lot more than the $0 or negative reward that most companies offer. There are many examples of security researchers being rewarded with threats and police reports.

Funny how when a person/company does a small good deed, they get criticized for not doing more but when they don't do anything and implicitly offer $0 they don't get criticized.

vmfunction• 3 years ago

Just wonder if the the reward for security bounty would increase such as 1 - 10 million dollars to match the black market, do you think it would attract more serious security researchers?

udunadan2• 3 years ago

You don't include reputation and publicity benefits into the overall profit formula. If you sell stuff on grey market, you'll have to keep your lips tight over that deal. It isn't going to your resume, so if one day you'll decide to join a security division in some corp, you'll have nothing to show them. Whereas a public achievement like this may be a door opener for many places. Same thing with things like pwn2own or security conferences. Why burn knowledge there? Because it promotes you.

jsnell• 3 years ago

I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.

If that were the case, why did even a single person report a vulnerability to Google? Why didn't these muppets just sell it to somebody else for ten times as much? I would suggest tha it's because what you claim isn't correct. It's not actually "easy", and the payoffs won't be "a lot more".

Now, it's pretty hard for anyone to actually refute your argument since literally the only evidence you're providing is your imagination. The one thing we do know is that tens of serious vulnerabilities were reported to Google via this program. Those are a concrete existence proof of this VRP being a sufficient incentive for bugs to be reported via this legit channel. Surely that outweighs your complete lack of evidence.

saagarjha6• 3 years ago

It’s also possible that people prefer the channel that makes them feel better about themselves?

laeri2• 3 years ago

Some people also have moral values and selling these would conflict with their personality. Not saying that the payout is high enough for skilled individuals to invest time.

rfoo4• 3 years ago

I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.

This is unfortunately false. The running price for Linux kernel exploit isn't that high.

twunde• 3 years ago

For anyone who wants an easily digestible read on io_uring exploits I love my former colleague's post: https://chompie.rip/Blog+Posts/Put+an+io_uring+on+it+-+Explo...

stevekemp• 3 years ago

That was an outstanding writeup, thanks for sharing it.

nitinreddy88• 3 years ago

This deserves its own thread HN. I don't have minimal knowledge on Kernel yet I understood many things about io_uring

fragmede• 3 years ago

https://news.ycombinator.com/item?id=36353845 I'm surprised it hasn't been submitted before.

klooney• 3 years ago

Oof. I hope this doesn't get wedge io_uring into a place where people can't use it because it's disabled, so they don't, and the whole improvement gets stuck and wasted.

laserbeam• 3 years ago

As a reader... I can't tell if there's a fundamental design flaw with io_uring, or just growing pains because the implementation is newish. Like how speculative execution had fundamental vulnerabilities which had to do with timing your code, not a specific implementation.

The title makes it look like a design problem with io_uring, Google's reaction makes it feel like a design issue. But the article itself sounds like growing pains.

sylware5• 3 years ago

Speculative execution is not fixed in modern CPUs? Or this is related to old ones?_

wahern• 3 years ago

Speculative execution is not fixed in modern CPUs?

Very generally speaking, not really. It's a huge class and the pace of discovery of new vectors has continued nearly unabated since Spectre. See https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner... The effectiveness and cost of various remediations--software, microcode, hardware--varies depending on the particular vector, so some vectors can be considered fixed, while other vectors, such as some of those relating to SMT, might never be fully fixed without effectively giving up some of the benefits of SMT.

Some of the more robust theoretical remedies, like cache coloring and kernel scheduler-based process affinity schemes, still haven't seen much investment or attention, so I would expect the state of things to remain relatively crappy. It just may seem less crappy because the industry has become inured to the threat, and also perhaps because there's usually much lower hanging fruit for attackers (i.e. traditional exploits like with io_uring), so these vectors are rarely used in the wild.

insanitybit4• 3 years ago

It's complex. No, there's nothing so fundamental about io-uring that it could not be made safe. But also, there are significant factors that are leading to it being ripe for exploitation.

1. Security has been sort of an afterthought. io_uring was being built out as a complex, powerful capability and only much later were things like auditing, selinux hooks, and ebpf even considered.

2. io_uring is inherently complex code. It's highly optimized, concurrent, and new. It does a lot of work to avoid things like copying across the kernel boundary. This code is hard to get right under the best of circumstances.

3. io_uring is very different. This is basically a new system call interface that behaves in a different way from the other interface, we're playing catch up in security.

So in theory this could all be fixed but it's going to take a lot of work and, importantly, a shift in priorities - io_uring can't just be developed as the "gotta go fast" interface, it needs to be designed around safety.

tlamponi2• 3 years ago

  > While io_uring brings performance benefits, and promptly reacts to security
  > issues with comprehensive security fixes (like backporting the 5.15 version
  > to the 5.10 stable tree), it is a fairly new part of the kernel. As such,
  > io_uring continues to be actively developed, but it is still affected by
  > severe vulnerabilities and also provides strong exploitation primitives. For
  > these reasons, we currently consider it safe only for use by trusted
  > components.

As often in the kernel (and probably everywhere): there are people caring for performance and there are people caring for security, and a lot of the time there's not that much overlap.

Ensuring that io_uring is covered by the Linux Security Modules (LSM) framework, that provides hooks for things like AppArmor or SELinux (and a variety of others, some even can even get stacked), was only done after it was already quite popular.

So FWICT, even if io_uring is existing since a while, 1) the security experts vetting it are looking only closer since relatively recently and 2) it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel, especially as the project still wants to stay high performance.

IMO, this doesn't mean io_uring is inherently unsafe, or has no use, but that one certainly needs to evaluate if great async submission model is worth the potential security implications. E.g., in an isolate environment, where no untrusted third-party apps or people have access, it can be fine, while especially in environments that Google runs it isn't worth the trouble for now.

But, I think that the features which io_uring brings are too nice to have for quite some (I/O or syscall heavy) workloads even in those environments, so there will be an incentive it to the integration into the existing Linux security stack and mechanisms, and evolve the latter to be also a good fit for this rather different interface.

udunadan2• 3 years ago

Yes, it is no different than OPSEC with heavy procedures in place which take the actor's mobility at the sake of security. A universal security property.

rfoo4• 3 years ago

integration into the existing Linux security stack

It's less about integration into the security stack. It's just that the implementation of io_uring is hard to get right, and at 2023 there seems to be an endless stream of race condition bugs. Quite like user namespaces ten years ago. It doesn't matter how good your security design is. Hackers just find a memory corruption bug and get around everything.

To be fair it isn't endless, finding an io_uring bug which can be exploited to cause memory corruption and ultimately kernel code execution in 5.15 tree is measurably harder now. But io_uring moves very fast and it's an entirely different story for mainline kernels.

tlamponi2• 3 years ago

  > It's less about integration into the security stack.

Yes, that's why I wrote:

  >> it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel

But user namespaces are actually a very good examples, they also broke with a lot of things that were often used as axiom, so there really are some parallels to io_uring from that POV.

  > Hackers just find a memory corruption bug and get around everything.

If the Rust experiment pans out well, it could make a lot of such bugs straight out impossible, at least if devs aren't bending backwards to lower the risk to roughly the same as C now has in the best case. At least at $work this pans out quite nicely.

retrocryptid• 3 years ago

"Lessons." The word you're looking for is "lessons." As in "Lessons from kCTF VRP's 42 Linux kernel exploits submissions."

dang2• 3 years ago

"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting."

https://news.ycombinator.com/newsguidelines.html

nathants• 3 years ago

when i was implementing lsm hooks for mighty-snitch last year, i remember attempting the io_uring hooks but quickly gave up.

the normal lsm hooks all do something well scoped, specific, and take meaningful parameters.

the io_uring hooks take what appear to be a turing complete language snippets as a parameter.

the hooks now all return false (disabled).

Learnings from kCTF VRP's 42 Linux kernel exploits submissions

Related Stories