Why is my x64 process getting heap address above 4GB on Windows 8?

http://blogs.msdn.com/b/oldnewthing/archive/2015/07/09/10624293.aspx

by ScottWRobinson • 11 years ago

54 34 11 years ago

blogs.msdn.com

the8472• 11 years ago

Why are they truncating 64-bit pointers to 32-bit values?

Maybe to get more compact data structures? But in that case they shouldn't rely on that and try detecting that behavior at runtime.

E.g. the hotspot jvm tries to map memory in the low end of address space to compress pointers but falls back to full pointers when that fails.

electrograv• 11 years ago

Maybe to get more compact data structures?

Even if this is why, there is never a valid excuse for blindly truncating 64-bit pointer to 32-bits. It's just like saying "I'm gonna just go ahead and truncate all bank account balances to 16-bit integers to save memory in my data structures." It may be the reason, but that reason is still incredibly stupid.

cnvogel• 11 years ago

It may be the reason, but that reason is still incredibly stupid.

Sometimes there are stupid reasons, but sometimes even reasons which are incredibly stupid from a technical standpoint make sense in a broader context.

For a short-term fix, for example, I can very well imagine to prefer this "trick" forcing sub-4G allocations if the other alternative would be to change 1000 places in undocumented legacy code doing crazy casts... And if it's for a product (or product component) on life-support only needed for a forseeable future, to me it makes perfect sense to ask the initial "crazy" question, even if it makes me cringe...

cesarb• 11 years ago

As mentioned on the blog post's comments, it's often caused either by storing a pointer in a DWORD variable (very common on 32-bit Windows), or by storing a pointer in a "long" variable (on most common 32-bit and 64-bit platforms, a "long" is at least as wide as a pointer; the main exception is 64-bit Windows).

The correct variable type to store a pointer would be intptr_t/uintptr_t for portable software (and DWORD_PTR for Windows-only software), but the "intptr" types didn't exist until the 1999 C standard (and, as far as I could find, the default Windows compiler didn't add the corresponding header file until around 2010). Software older than that would often assume that a "long" (or a DWORD, which IIRC was just another name for "long") was big enough to store a pointer, or perhaps tried to use the C99 types and used "long" if the header wasn't found.

TillE• 11 years ago

storing a pointer in a DWORD variable (very common on 32-bit Windows)

Is it? I mean, if you're lucky it won't cause a compiler warning on x86, but the sane thing to do is to use the types which the API docs specify. Which would usually be things like LPVOID or HANDLE.

That's basically how I managed to port a service and several drivers to 64-bit Windows many years ago with almost no changes.

netheril96• 11 years ago

There are tons of C/C++ codes that use `int` for every type of integer, including indexing arrays and storing pointer values. A relic of the time where everything was 32-bit. It is common on Unix system as well. When you compile C/C++ library, a Python c-extension, there are usually many warnings about integer truncation.

kardos• 11 years ago

aka bugs!

aardvark179• 11 years ago

Hotspot is really quite subtle, and thorough in its strategy. Because of the alignment of java objects it can both shift and truncate pointers (reversing the shift when they are actually used), but can also handle the heap not being allocated in the low end of address space by storing the offset of the base address, and adding/subtracting that as needed as well.

phs2501• 11 years ago

This bit me in a library of mine. Some third-party code I used was using long as memory offsets which failed on Windows 8. Unfortunately I was developing on Windows 7, so it didn't get noticed until the customer stumbled over it. :( (Fortunately said third-party code had an update that switched to ptrdiff_t in all the right places, so it was an easy fix.)

I'm a Unix person, so the notion of long being less than word-length seems pretty silly to me, but I understand why they did it, sort of.

sjolsen4• 11 years ago

I'm a Unix person, so the notion of long being less than word-length seems pretty silly to me

It's silly for Windows to do something a certain way just because that's not how Unix does it?

What's silly is assuming that long and unsigned long are more than 32 bits wide, since neither C nor C++ has ever guaranteed anything more.

sp332• 11 years ago

If it's not specified, implementers have the option of making it wider. Matching the width of a pointer seems like a better choice.

michaelt• 11 years ago

I don't write much C, so perhaps someone can educate me.

What are the advantages of having data types that behave differently on different platforms, like C does? It seems to make code less portable and I'm having difficulty seeing what the benefits are.

I can appreciate that, on an 8-bit processor, you'd want access to an 8-bit data type for speed - but wouldn't that be better accomplished by explicitly choosing an 8 bit data type?

copx2• 11 years ago

Pointer packing is popular with people wanting to write efficient interpreters for programming languages. The memory use and performance gains are massive and it was perfectly safe during the 32-bit era.

LuaJIT does it for example. Interestingly enough some programmers do this without even understanding how memory allocation works. For example the author of the newish wren language: https://github.com/munificent/wren

Somehow most programmers - including almost everyone who frequents reddit.com/r/programming - believe that "virtual address space" means that the addresses returned by malloc() will restart from (somewhere close to) 0x0 for every new process i.e. that it is safe to truncate pointers if you never intent to allocate more than ~2GB per process anyway.

beagle32• 11 years ago

LuaJIT does something weirder. It does double packing - using 48 bits (IIRC) of value inside a "NaN" double encoding. It is going to be a while before this becomes a problem, even on 64-bit systems - you need more than 128TB addressable space before this becomes a problem (and I suspect Pall will switch to a 51-bit (1024TB=1PB) pointer when that future is on the horizon).

copx2• 11 years ago

No that is wrong. Currently the x64 port of LuaJIT is limited to about 1GB [1] in the best case and can fail to allocate any Lua objects at all if the lower region of the address space is already completely consumed.

[1] http://lua-users.org/lists/lua-l/2010-11/msg00241.html

daniel02216• 11 years ago

This is one reason why OS X enforces a hard page-zero on 64-bit programs - it is a hard error to map or allocate anything in the lowest 4 GB of memory, so programmers will immediately discover pointer truncation bugs when porting 32-bit code to 64-bit.

cokernel_hacker• 11 years ago

What if you have a load command in your mach-o which asks for an address below the 32-bit boundary? Will loading the image fail?

fsloth• 11 years ago

Long was not 64 bits? This is why C standard has stdint.h which typedefs numeric types of specific length (for 64 bit integers use int64_t, etc). It seems Visual Studio also has it nowadays https://msdn.microsoft.com/en-us/library/hh874765.aspx.

Why is my x64 process getting heap address above 4GB on Windows 8?

Related Stories