Making our own executable packer, part 13, Thread-local storage

https://fasterthanli.me/blog/2020/thread-local-storage/

by lukastyrychtr • 6 years ago

58 23 6 years ago

fasterthanli.me

drudru11• 6 years ago

Why is he making a packer?

royjacobs• 6 years ago

The better question is, why not?

scintill76• 6 years ago

amos loves to tinker

killercup2• 6 years ago

Start at part 1.

No, he doesn't give you an answer, but you might just find it interesting enough to no longer have the qusstion.

chrisseaton2• 6 years ago

I don’t understand what a ‘packer’ is - is it a linker?

rustybolt• 6 years ago

From what I understand, it's something that 'packs' an application, either to preserve space or to obfuscate it (making it a little bit harder to reverse engineer it since you have to peek at the extracted contents in the memory instead of at the file contents).

So when you would run a packed application, there application would essentially start to unpack an encrypted/compressed blob of memory, and then jump to it once it's unpacked.

haecceity• 6 years ago

It compresses executables.

killercup2• 6 years ago

I don’t understand what a ‘packer’ is

You've found the right series of articles then!

chadcmulligan• 6 years ago

The first few paragraphs on the first part explains it?

Executables have been fascinating to me ever since I discovered, as a kid, that they were just files. If you renamed a .exe to something else, you could open it in notepad! And if you renamed something else to a .exe, you'd get a neat error dialog.

Clearly, something was different about these files. Seen from notepad, they were mostly gibberish, but there had to be order in that chaos. 12-year-old me knew that, although he didn't quite know how or where to dig to make sense of it all.

So, this series is dedicated to my past self. In it we'll attempt to understand how Linux executables are organized, how they are executed, and how to make a program that takes an executable fresh off the linker and compresses it - just because we can.

platz2• 6 years ago

Would someone explain what the primary intent and current best-case usages of thread-local storage are?

Is this more suited for certain languages than others?

Someone2• 6 years ago

https://en.wikipedia.org/wiki/Thread-local_storage mentions the two most common uses: errno (historically a global variable, but on modern Unixes thread-local) and having separate threads accumulate a value, to only merge them at the end, thus decreasing contention for the merged value.

platz2• 6 years ago

having separate threads accumulate a value, to only merge them at the end, thus decreasing contention for the merged value.

That sounds like just having scoped state in each thread that isn't global.

dblohm7• 6 years ago

This explanation is a bit hand-wavy, but "correct enough" for our purposes.

Let us suppose that you wanted to store some per-thread data.

A naive way to do that could be to have a global mapping from thread ID to a pointer to your data. To look up your data, you would obtain the id of your current thread and look it up in the map. Of course, since that map is shared across threads, you'd also need to protect it with a mutex.

Operating systems themselves need to maintain per-thread data. When context-switching between threads, the OS will also switch out the current block of per-thread data. Applications can piggyback on this, so instead of needing to lock a shared map and then do a lookup to retrieve their data, it will already be available to them by virtue of the fact that it is essentially part of the current thread's context. No locking or global map lookups necessary!

cryptonector• 6 years ago

It's super useful for many reasons:

  - to help make thread-unsafe APIs thread-safe

    E.g., errno, strtok(), and so on.

  - to store thread-private mutable globals

    (Thread-locals are more like thread-globals.)

  - to keep a reference/copy of global immutable
    data that might get replaced in the middle of
    a thread's handling of a request

  - to keep contextual data that you can't retrofit
    an API to pass explicitly (see also first item)

spacechild1• 6 years ago

Pure Data made heavy use of thread local storage to turn the original application singleton (with lots of globals) into a threadsafe and reentrant library (libpd).

pjc50• 6 years ago

IIRC "jemalloc" achieves a speedup by using thread-local storage for (pointers to) bookkeeping and memory pools in preference to a heap shared between all threads - that makes it possible to do an allocation without using any synchronisation.

elteto• 6 years ago

Great article. I particularly enjoyed the “beyond the basics” use of gdb.

I know the basics just like anyone else whose done some light debugging with gdb but it is hard to find information that goes beyond that. There’s a gazillion of half-baked gdb introductions but IMO not enough advanced tutorials.

Making our own executable packer, part 13, Thread-local storage

Related Stories