Problems with the heap

https://rachelbythebay.com/w/2025/03/26/atop/

by todsacerdoti • 1 year ago

211 90 1 year ago

rachelbythebay.com

bee_rider2• 1 year ago

RachelByTheBay underestimated how much we hang on her every word over here, I think. Haha.

anitil2• 1 year ago

I believe she hasn't had the best behaviour from our members unfortunately

simoncion3• 1 year ago

I expect she's had the best behavior that the members in question are capable of.

wglb• 1 year ago

I think it has gotten better of late.

neuroticnews25• 1 year ago

Can you expand on this? I'm new here and I have been binge reading her posts recently.

ultrarunner• 1 year ago

This reads like it's referencing something specific. Am I out of the loop or is this just about heap exploits in general?

dan353hehe• 1 year ago

A post from them yesterday made it to the front of HN.

https://news.ycombinator.com/item?id=43477057

Edited to be the HN post.

bee_rider2• 1 year ago

There was a post yesterday

https://news.ycombinator.com/item?id=43477057

Which was sort of strongly reacted to (atop is wide spread and the blog author has a bit of a following here).

baltimore• 1 year ago

Same author in March 2014 was having segfault issues with atop apparently: https://rachelbythebay.com/w/2014/03/02/sync/

pdpi• 1 year ago

Rachel's been a reliable source of interesting issues like these for the better part of eternity now. Her blog's well worth reading.

usefulcat• 1 year ago

After reading that: atop should've used SQLite.

th0ma5• 1 year ago

Check out this https://github.com/Atoptool/atop/issues/330 possibly related GitHub issue?

OptionOfT• 1 year ago

There haven't been any releases of atop since Jul 27, 2024, v2.11.0.

Are people building atop themselves?

https://archlinux.org/packages/extra/x86_64/atop/ shows 2.11.0-1, not updated since 2024-07-31 11:02 UTC

https://packages.debian.org/sid/atop & https://packages.ubuntu.com/oracular/atop show 2.10.0-3, not updated since Fri, 31 May 2024 13:42:28 +0200 (https://metadata.ftp-master.debian.org/changelogs//main/a/at..., https://changelogs.ubuntu.com/changelogs/pool/universe/a/ato...)

john-h-k• 1 year ago

as there seems to be some confusion, this is my interpretation:

atop is (for some reason) touching memory of processes it monitors.

atop is touching this in an insecure way. An executable can cause atop to corrupt its memory.

this has high potential (although not guaranteed) for allowing RCE within atop via a correctly crafted process that atop monitors.

atop is often run in root, and so this otherwise meaningless RCE becomes privilege escalation, which is bad

either this is correct, or cunningham’s law will bring out the correct interpretation

shanemhansen• 1 year ago

This is a complete shot in the dark but wild speculation is fun. If atop had a buffer overflow when reading a process name (changeable at runtime using $0 in perl for example) this would be the kind of issue I expect.

Similarly, some other value that was expected to be null terminated but wasn't.

twic• 1 year ago

My guess is that it tries to decode the malloc metadata, which involves chasing pointers, and doesn't do enough sanity checking, so if a process accidentally or deliberately sets up corrupt metadata, atop will dereference an invalid pointer and explode itself.

dang• 1 year ago

Recent and related:

You might want to stop running atop - https://news.ycombinator.com/item?id=43477057 - March 2025 (131 comments)

judofyr• 1 year ago

Okay, first off, everybody breathe. Everyone is freaking out. This is not the way to do this.

Why are you doing it this way then?

keyle• 1 year ago

Yeah that's a half-assed apology. Yesterday's post might have unintentionally sent a lot of heads spinning, "this is not the way to do this" is not an apology, it's a double-down.

I understand there is a potential heap overflow with atop, thanks for letting everyone know; but you're also letting the people capable of taking advantage of it aware that there is possibility to do this in the wild. Due process is let the developers fix it and then tell everyone to upgrade.

Anyway a C process that runs as root for lifestyle purposes (e.g. not a critical service) is a big no-no. And I say this as I like to write C, I love C. But I wouldn't push my C code on anyone else's computer, especially requesting root access. I'm not that good.

achierius• 1 year ago

I would give her the benefit of the doubt and presume that the responsible parties were informed some time ago.

simoncion3• 1 year ago

Yeah that's a half-assed apology.

That makes sense. That's because it's not an apology; It's an admonition to those who are blowing their stacks.

simoncion3• 1 year ago

Why are you doing it this way then?

I'd argue that it'd be a beneficial life lesson for the people who are freaking out over a "Hey, maybe stop using 'atop'." comment to learn how to enhance their calm.

Blowing one's stack over every little thing shortens your lifespan! It's best to learn how to take friendly warnings about bad things in stride.

NobodyNada2• 1 year ago

Both of the error messages we're given indicate that the "top" chunk of the heap was corrupted, which is a special internal allocation used by glibc malloc to represent any unused capacity from the last time malloc decided to grow the heap:

    high memory addresses ("top" of heap)
    | unmapped memory |
    |-----------------|
    |                 |
    |    top chunk    |
    |  (mapped, but   |
    |     not yet     |
    |    allocated)   |
    |                 |
    |-----------------|
    | allocated chunk |
    |-----------------|
    |   freed chunk   |
    |-----------------|
    | allocated chunk |
    |                 |
    |-----------------|
    | allocated chunk |
    |-----------------|
    |   freed chunk   |
    |-----------------|
    |     tcache      |
    |-----------------|
    low memory addresses ("bottom") of heap

That likely indicates a heap buffer overflow. If a call to malloc() doesn't find a freed chunk, it will split the top chunk in two and return a pointer to the bottom portion. If you then write past the end of the returned allocation, you clobber the metadata of the top chunk, and get errors like the ones in the article.

With the exploit mitigations built into modern Linux and glibc, it's a lot of work to go from here to arbitrary code execution; but it very well may be possible, depending on exactly how much control the attacker has over what atop does. The attacker can probably trigger the heap buffer overflow multiple times by spawning multiple processes, and if the length and contents of the heap buffer overwrite are attacker-controlled, they can probably play some games to overwrite any data stored in the heap. If that's true, the only thing preventing full arbitrary-code-execution is ASLR; there are many clever ways to get around that, but it's often quite difficult and may or may not be possible here.

This year's LACTF had a challenge with essentially this exact setup. My solution writeup is a good example of what it takes to defeat the exploit mitigations and turn a heap buffer overflow into an RCE: https://jonathankeller.net/ctf/lamp/

progmetaldev2• 1 year ago

I thought I would ask, does this mean that memory restricted environments can be a better target for attack than one with a large amount of memory available? In my mind it seems like this would be the case, but I'm not sure if there is anything in place to protect these types of environments, like intentionally breaking up and segmenting memory so it's not possible to read much linearly. I admit that I haven't touched low-level code since the early 2000's, and earlier than that for anything other than a course requirement, so I apologize if you explained it in your linked article and I don't understand.

NobodyNada2• 1 year ago

Yes, with the caveat that virtual memory restrictions matter a lot more than physical memory restrictions.

Heap exploitation is difficult because 1) glibc malloc is hardened to try to defeat many common exploit strategies, and 2) ASLR means that even if you have the ability to corrupt a pointer, you might not be able to control what the pointer points to.

Regarding #1, memory constraints don't really make a difference until you're so constrained on memory that you can't run glibc anymore, so you use an allocator that is optimized for low overhead/code size rather than performance and security.

For #2: ASLR works by placing every section of the process (heap, stack, program binary, and each library) at a randomized virtual address so that attackers can't forge pointers. ASLR is much more effective on a 64-bit system than on a 32-bit system, simply because there's so much virtual address space available to choose. If memory addresses are only 32 bits, it's feasible to just brute force guess the memory address of some data you're interested in; with 64 bits, it's not. And if your system doesn't have virtual memory at all (like a microcontroller), you probably don't have any kind of ASLR.

intentionally breaking up and segmenting memory so it's not possible to read much linearly

This is typically done at the level of individual sections (stack, heap, program binary, library binaries) but not within sections; mainly for performance, memory overhead, and cache locality reasons. The entire heap is contiguous, so if you overwrite past the end of one heap allocation you can overwrite adjacent allocations, but you can't overwrite the stack (without more work). Breaking up the heap into smaller chunks wouldn't really help that much; it just means an attacker has to manipulate the heap layout so they can be sure the allocation they're targeting ends up in the chunk they're targeting.

Exploit hardening in the heap allocator is something of a last-ditch stopgap measure: "we've already lost, but let's see if we can minimize the damage to the user/maximize the difficulty to the attacker". There's certainly much more you could do to harden the heap, but remember that any security measures implemented in glibc are applied to every program, with no way to opt out (except bringing your own libc). So glibc is designed to maximize performance and compatibility; exploit mitigations are only included when they doesn't compromise these goals.

If you don't mind giving up some performance for the sake of security, you probably would already be using a garbage-collected language instead of C. (And now that Rust is sufficiently mature for most use cases, you probably should be using that if you're in a situation where you need both performance and security).

LPisGood• 1 year ago

It’s a lot easier to use an exploit if you know your chunk is from fast bins if I recall correctly.

ianbutler5• 1 year ago

Hey guys so I work on a tool call Bismuth along with my co-founder for finding and fixing bugs and we think we have this. At the very least we have a bug in atop which mimics what is being described.

We're going to throw this sha down right here: 1d53b12f3bc325dcfaff51a89011f01bffca951db9df363e6b5d6233f23248a5

And now we're going to go responsibly disclose what we have to the maintainers.

ianbutler5• 1 year ago

We've reached out to the maintainer over e-mail.

Retr0id• 1 year ago

Based on the bug you've found, do you think it's exploitable beyond DoS?

__turbobrew__2• 1 year ago

Thank you for doing this instead of just vagueposting and wasting everyone’s time.

vanderZwan2• 1 year ago

I'm inclined to expect that we should put the blame for that on whomever used legal channels to force Rachel to shut up, although obviously the jury is still out until we know more.

spyc4• 1 year ago

Hi! Three things:

- There is no commit with a SHA1 like that in atop Git history and what you shared is too long for a SHA1, it looks more like a SHA256. Did you share the right checksum? The only other way I can read this is that it's a SHA256 checksum of one of the past atop release tarballs or artifacts. I have not yet checked those.

- I have tried finding your tool Bismuth but all I find is things KDE and crypto currencies. Please share a link to the Bismuth that you are working on.

- You technically said that you are working on Bismuth /and/ found something, not that you found the bug /through/ Bismuth. Please clarify if and how that was the case.

Thank you!

spyc4• 1 year ago

Update: I found https://www.bismuth.sh/ at https://news.ycombinator.com/user?id=ianbutler .

throwawayben• 1 year ago

pretty sure it's just a hash of some text they can reveal later, to prove that they had something at this point in time. not referring to any release or commit

ianbutler5• 1 year ago

- That SHA is just a proof marker so if it turns out we are correct we can prove we had it at that time

- Bismuth did indeed find the bug, our bug scanning feature in particular. Obviously we're going to sit on our hands until the maintainer gives the all clear but we'll write something up after this is all squared away

- https://www.bismuth.sh is our tool, we're still relatively new

ianbutler5• 1 year ago

We did in fact find the bug:

https://news.ycombinator.com/item?id=43519522

nsxwolf• 1 year ago

I found this hard to follow, like it was in the middle of something.

mary-ext• 1 year ago

A vulnerability can be high risk, a vague disclosure means that people can only assume the worst.

Maybe don't vaguepost about vuln disclosures

EdiX• 1 year ago

I gave her the benefit of the doubt initially, she usually posts good posts, but this is not the way to do things. Vagueposting about a security vulnerability without properly disclosing it to the mantainers: (1) damages their reputation, (2) sends every blackhat on the hunt like a real life worldwide CTF event, (3) leaves sysadmins in the dark unless they are following this specific random blog and (4) since the details aren't known even if they know it's impossible to determine if they really are affected.

Something like this would be justified if the maintainers were unresponsive and it was a remotely exploitable bug. Now it turns out this is probably a minor thing (local privilege escalation if you happen to be running atop as a privileged user).

It seems to me like an irresponsible, egocentric way to handle things.

ynik• 1 year ago

At least on Debian, installing the `atop` package will automatically install a background service running atop as root. (by default, logging some stats to /var/log/atop/ every ten minutes)

nukem222• 1 year ago

Eh, finger pointing does nobody any good, emphatically including this comment. Finger pointing towards someone who actually found a vulnerability is just bleak. I would not willingly associate with anyone who engaged in such behavior.

Maintaining software is hard, but this does not imply a right to be babied. People should simply lower their expectations of security to match reality. Vulnerabilities happen and only extremely rarely do they indicate personal flaws that should be held against the person who introduced it. But it's your job to fix them. Stop complaining.

whatnow3737310• 1 year ago

I never quite understood why computing is so different from literally all other branches of reality. Systems need to be secure, I get it. But if we have a bunch of folks dedicating their life to breaking your shit I don't get how that is in any way acceptable and why the weight of responsibility solely lies with people responsible for security.

We apparently have a society/world that normalizes breaking everyone's shit. That's not normal - IMO.

If I break into a factory or laboratory of some kind and just walk out again I have not found a "vulnerability" and I certainly won't be remunerated or awarded status or prestige in any way shape or form. I will be prosecuted. Everyone can break into stuff. It's not that stuff is unbreakable, it's that you just don't do that because the consequences are enormous (besides obvious issues with morality). Again, breaking stuff is the easy part.

I am certainly completely ignorant and should be drawn and quartered for it, but for me it is hard to put my finger where I'm so wrong.

I can see how the immaterial nature of software systems changes the nature of the defense, but I don't see how it immediately follows that breaking stuff that's not allowed to be broken by you is suddenly the norm and nothing can be done against that. We just have to shrug and accept our fate?

frontfor• 1 year ago

I second this. The pompous holier-than-thou I-know-better attitude some members of the computer security community has always rubbed me the wrong way. This behaviour of complaining is a manifestation of the typical “putting down” and dismissing someone who isn’t part of the tribe.

gruez3• 1 year ago

Finger pointing towards someone who actually found a vulnerability is just bleak. I would not willingly associate with anyone who engaged in such behavior.

Nobody is "finger pointing" Rachel for the vulnerability. They're calling her out for how she communicated it. I feel that's totally justified. For instance if someone found a critical RCE, but the report was a barely coherent stream of consciousness, it's totally fine to call the latter part out. That's not "finger pointing".

But it's your job to fix them. Stop complaining.

It's the developers job to respond to bug reports in the form of vaguely written blog posts?

amiga386• 1 year ago

Fingerpointing is bad, but we have to have an honest conversation.

One person posted the vague post. They clearly did not expect the reaction it got, though they could have anticipated some of it, they are aware their blog is widely read. Their reaction is commendable, to quickly post a followup appealing for calm and sharing some details, to quell the problems caused by the intense vagueness.

What people from HN did, because of the vagueness, was assume this a super-secret-squirrel mega-vulnerability and Rachel is gagged by NDAs or the CIA or whatever... and they've gone off and harrassed the developers of atop while trying to find the issue.

Imagine a person of note saying "the people at 29 Acacia Road are suspicious", then a mob breaks down the door and start rifling through all the stuff there, muttering to themselves "hmm, this lamp looks suspicious... this fork looks suspicious"... absolute clowns, all of them.

For example, this asshole who went straight in there with bad-faith assumptions on the first thing they saw: https://github.com/Atoptool/atop/issues/330#issuecomment-275...

No, you dummies, it's not going to be in the latest commit, or easily greppable.

This is exactly why CVEs, coordinated disclosure, and general security reporting practises exist. So every single issue doesn't result in mindless panic and speculation.

There's now even a CVE purely based on the vaguepost, assigned to a reporter who clearly knows fuck all about what the problem is: https://www.cve.org/CVERecord?id=CVE-2025-31160 - versions "0" through "2.11.0" vulnerable, eh? That would be all versions, and the reason the reporter chose that is because they don't know which versions are vulnerable, and they don't know what it's vulnerable to either. But somehow, "don't know", the absence of information, has become a concrete "versions 0 to 2.11.0 inclusive"... just spreading the panic.

I don't know why Rachel is vagueposting, but I can only hope she has reported this correctly, which is to:

1. Contact the security of the distro you're using. e.g. if you're using atop on debian, then email security@debian.org with the details.

2. Allow them to help coordinate a response with the packager, the upstream maintainer(s) if appropriate, and other distros, if appropriate. They have done this hundreds of times before. If it's critically important, it can be fixed and published within days, and your worries about people being vulnerable because you know something they don't can be relieved, all the more quickly.

ptx• 1 year ago

a minor thing (local privilege escalation if you happen to be running atop as a privileged user)

I seem to be hearing this sentiment a lot lately. How is local privilege escalation a minor thing?

If it's such a minor thing, is the old advice to not run as root considered passé? Should we just run everything as root? Should we discard the entire Unix security model and chmod all files to 0777?

ayende• 1 year ago

In most scenarios, you are no longer running with multiple users on the same machine. Either this is a server, which has an admin team, or a client machine, which _usually_ have a single user.

That isn't 100% true, and local privilege escalation matters, but it is a far cry from remote code execution or remote privilege escalation.

red1reaper• 1 year ago

Should we discard the entire Unix security model and chmod all files to 0777

It depends, but for most use cases... yes, actually.

heavyset_go2• 1 year ago

All of that etiquette sounds nice if you're being paid to do this work, but I don't think anyone is obligated to "properly" disclose a vulnerability they found on their own time/dime, nor do I think it's a moral imperative for anyone to do so.

halayli• 1 year ago

You need to establish the causal link between user1 doing something and atop segfaulting. That link is what determines whether there's a potential exploitability to take advantage of. it's easy to think of a scenario where user1 is using almost all the memory on the system and user2 runs atop and segfaults because atop is missing a memory check and overcommit is disabled, or user1 deletes a file that can cause another app to crash given the wrong permissions on the file.

trismegisti• 1 year ago

It would be very surprised if this isn't just an atop bug.

Can it be exploited? Considering the error messages, the possibility is high.

I don't like to see it as "Problem with the heap". As someone who played lots of CTF and is quite sufficient in exploiting such bugs, i would much rather see those things framed as "Problems with the glibc allocator".

If we just wouldn't use inline metadata, or verify it's integrity (smth like scudo but not broken), all of the security issues with the heap would just be gone. If you get to work with frameworks/languages that are more flexible when it comes to allocate memory (thinking about zig and it's amazing allocator abstraction here) you quickly realize that malloc and free are an insanely simplistic api for what is one of the most difficult and important problems in programming: memory allocation.

There is also no excuse for the error messages being that bad. Because the reality is that most systems programmers will have to debug those at some point.

titaphraz• 1 year ago

Responsible disclosure has been a thing for a long time. This is not a professional behavior.

Problems with the heap

Related Stories