Much ado about NULL: An introduction to virtual memory

https://blogs.oracle.com/linux/post/much-ado-about-null-an-introduction-to-virtual-memory

by todsacerdoti • 5 years ago

28 31 5 years ago

blogs.oracle.com

saagarjha2• 5 years ago

Note that Linux has since added a "mmap_min_addr" tunable that is intended to prevent applications from mapping in the zero page and exploiting NULL dereferences.

gpvos3• 5 years ago

It is already mentioned in the article, or maybe in part 2.

bhaavan• 5 years ago

The follow up post: https://blogs.oracle.com/linux/post/much-ado-about-null-expl...

gpvos3• 5 years ago

^^^ This is where the actually interesting bits are, and it's not directly linked from part 1.

unwind• 5 years ago

I drives me slightly crazy that an article about a fantastically technical subject such as "NULL pointer dereferences in the (Linux) kernel" immediately uses incorrect C code with undefined behavior to illustrate something.

This:

    printf("The argv pointer = %d\n", argv);

is not okay, you cannot pass a pointer-sized value when you tell printf() you're passing an integer. It needs to be:

    printf("The argv pointer = %p\n", (void *) argv);

You should use %p which is for printing pointers, and cast to void pointer.

Edit1: Oh, and I realize it "worked" for the author, no nasal demons [1] appeared and the code printed a number. That's just being lucky, and it's not a very nice foundation to base an argument on.

Edit2: Okay, now I read a little bit further and I'm pretty sure the author lives in the "pointers are just integers, and NULL is zero" world, which is of course not a world that co-exists with our own.

There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", that is implementation-defined and completely up the the architecture you're running on (and the compiler).

I realize that in practice, on common hardware, NULL is typically implemented as all-bits-zero, but that is not defined by the language. I don't even know if I make sense, obviously I'm a C buff and take these things rather seriously.

[1] http://catb.org/jargon/html/N/nasal-demons.html

saagarjha2• 5 years ago

In the context of vulnerability analysis it is often more convenient to mark undefined behavior with something implementation-defined. This is why you can say "a buffer overflow will overwrite the return address" or "this integer will wrap to be negative", or, in this context, NULL is represented with the zero address/bit pattern.

vlovich123• 5 years ago

The article is from 2010 (missing tag) so the author likely is talking about 32 bit x86 machines. Also compilers from that era were less likely to exploit UB if at all. Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it with Chris Partner’s blog about it following about a year later [1].

[1] http://blog.llvm.org/2011/05/what-every-c-programmer-should-...

guerrilla• 5 years ago

Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it

It was common knowledge among C programmers long before then, for both security and portability reasons but optimization was also done. Old MacOS even used the higher 8-bits of pointers for MM flags. Other popular systems did equally weird things (think VAX.) That discussion linked to by the Jargon file is from 1992.

tialaramex2• 5 years ago

Also compilers from that era were less likely to exploit UB if at all

No. Here's an LWN discussion of a CERT advisory in 2008 trying this nonsense, claiming it's a problem that GCC 4.2 elides futile late NULL checks. https://lwn.net/Articles/278137/

In fact several other compilers already had this optimisation, and CERT had to go back and revise the advisory because in practice all that's going on is, as now Real Programmers think their undefined program ought to do whatever it was they intended, and that's not how programming works.

masklinn2• 5 years ago

The article is from 2010 compilers from that era

Funny.

tcc was created in 2001 as a specifically non-optimising compiler.

pedrocr• 5 years ago

There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", that is implementation-defined and completely up the the architecture you're running on (and the compiler).

I always wondered if all the C codebases that do "if (somepointer)" to check for NULL were actually wrong of if the standard guaranteed that NULL always evaluates to false even if it's not implemented as zero. Apparently it's valid and everything works even in a theoretical architecture where NULL was not implemented as 0:

http://c-faq.com/null/ptrtest.html

masklinn2• 5 years ago

Yep, NULL doesn’t have a defined value, but that value must be false in a boolean context.

tialaramex2• 5 years ago

Under the hood, for this to work there's an implicit cast to boolean, even though at the time it was invented C doesn't have a formal boolean type.

So, (almost?) every expression in C has an implied boolean value, and the implied boolean value of pointers is false if they're NULL or true otherwise.

A better design is to have a boolean type, and require that the if condition tests only actual booleans, but this requires a substantial eco-system wide commitment, if you just insist on actual booleans you make programming needlessly tiresome and programmers say this language is bad because I have to write so much boilerplate for no gain. You need to actually provide the rest of the language features that make this pain worth it.

That commitment seems relatively affordable in 2021, but it was probably too big an ask when C was invented and by the time C was standardised it's too late. Likewise for C++.

gpvos3• 5 years ago

which is of course not a world that co-exists with our own

The "our own" world you are describing has recently been created by UB-exploiting compilers; it's a very strange and unintuitive place, is only described in inscrutable standards documents, and compilers don't properly warn you if you do something wrong even if you ask them to. A bit more understanding of people making mistakes seems appropriate.

mpweiher6• 5 years ago

And in fact, most of "us" don't inhabit this world voluntarily.

And the standards in question are actually have very clear and simple language disallowing those shenanigans.

"Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)."

Alas, that language was made non-binding and now the people who made it non-binding pretend it doesn't exist at all.

https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...

mpweiher6• 5 years ago

you cannot pass a pointer-sized value when you tell printf() you're passing an integer.

Actually you can. You probably shouldn't, particularly now that many 64 bit C machine models have different sized pointers and ints. If they were going to use an int conversion, they should have used %ld.

You should use %p which is for printing pointers,

Agreed.

[no nasal daemons] That's just being lucky,

Actually it isn't "just lucky". Well, to be precise it wasn't just lucky back when C compiler writers didn't stretch the idea of UB way beyond the breaking point, and way beyond certainly the intent and arguably the letter of the standard.

(The cop out is that the part of the standard that explicitly says all these shenanigans are not OK was made non-binding in later versions of the standard, it originally was binding, and thus people pretend that this part of the standard doesn't exist at all. But it does.)

There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero",

There is no guarantee of this in the C standard. However, the C standard is intentionally incomplete, i.e. conformance to the standard is not enough for an implementation to be "fit for purpose". I think it even used to say that somewhere in the standard.

I realize that in practice, on common hardware, NULL is typically implemented as all-bits-zero,

For a very not to say overwhelmingly large value of "typically". I can't name a single implementation that does not, and my guess would be that total market share of such implementations would be less than 0.1%, and I believe I am being very conservative with that estimate. Which also means that if you are on such an architecture, you almost certainly know it.

but that is not defined by the language.

...because the language definition is intentionally incomplete. People talk about the "C abstract" machine as if it were a thing. It's not. C maps to real hardware, that's the point of the language, and so taking the real-world behavior of the hardware that you are going to run on into account is perfectly fine.

SAI_Peregrinus3• 5 years ago

Android on 64-bit ARM uses the MSB of pointers for a tag. There are 2^8 NULL pointer values at the hardware level there.

mrslave• 5 years ago

There are 2 types of C programmers: those who have read Van der Linden, and those who have not.

da_chicken• 5 years ago

Note that this article is from 2010. Title should probably include a "(2010)".

twoodfin• 5 years ago

HP-UX actually had an executable attribute bit that would control whether dereferencing a pointer to address 0 would accvio or result in the value 0.

I learned this fact when at least one library (OpenSSL? Kerberos?) shipped with the OS relied on the non-crashing behavior, unconditionally dereferencing an optional pointer-to-struct argument.

That was a fun debugging session.

nickcw• 5 years ago

NULL is only slightly special as a pointer value: if we look in stddef.h, we can see that it's just defined to be the pointer with value 0.

Your C compiler is at liberty to use a different value than 0 for the actual NULL pointer stored in memory or in a register. It has to pretend to you that it is 0, but could be 0xDEADBEEF in reality.

Before ANSI C, NULL was #define-d as the actual bit value required. In modern C NULL is always 0.

In all the modern architectures I can think of a NULL pointer is represented by a value with all bits 0, but there are lots of examples of older architectures where this wasn't true.

There is a section in the comp.lang.c FAQ about this:

http://c-faq.com/null/machexamp.html

Filligree• 5 years ago

Needing to care about a wart that's only there due to 40 year old, dead architectures is yet another problem with C.

At least in this case the standard goes to some length to make sure you don't have to care.

SAI_Peregrinus3• 5 years ago

40-year old dead architectures like Android on 64-bit ARM?[1] Pointer tagging isn't that rare, and as memory expands in practice I'd expect we might even end up wanting >64-bit systems to allow for >48-bit real address spaces and/or >16-bit tags.

The C standard means you don't have to care that there might actually be 2^8 NULL pointers since you're running on ARM with Top Byte Ignore in use.

[1]https://source.android.com/devices/tech/debug/tagged-pointer...

Much ado about NULL: An introduction to virtual memory

Related Stories