Should you be scared of Unix signals?

http://jvns.ca/blog/2016/06/13/should-you-be-scared-of-signals/

by ingve • 10 years ago

155 82 10 years ago

js2• 10 years ago

Except for SIGKILL. When you get sent SIGKILL nobody communicates with you, you just die immediately. But the rest of the signals you're allowed to install signal handlers for.

SIGSTOP also cannot be caught nor ignored.

wtf_is_frp• 10 years ago

or blocked.

puppers• 10 years ago

Signals are not scary if you read the docs! Perhaps you could tone it down with the exclamation points!

eric_the_read• 10 years ago

This is FoaF-level stuff, but:

I used to work with a guy who in a past life was an HP-UX dev. He told me that the guys who worked on the signals support in the OS had a 10-foot pole between their cubicles that had a flag on top reading: "You must be THIS tall to use signals."

vikiomega9• 10 years ago

What's FoaF? (RDF?)

ludamad• 10 years ago

Friend of a friend

marios• 10 years ago

I've linked to this before, but AFAIK it's still relevant as the gotchas regarding signals haven't changed. Slides from a talk titled "Signal Handlers" from OpenBSD developer Henning Brauer : http://www.openbsd.org/papers/opencon04/index.html To answer the articles question: should you be scared of Unix signals ? No. But you shouldn't do anything complicated in the signal handlers.

jasonzemos3• 10 years ago

On linux, the behavior for locking a mutex (I tested a GNU C++11 std::mutex) from a signal handler is to consider it (at least for the interrupted thread) to have been unlocked. This allows intuitive synchronization from handlers and avoids deadlocks, which I'm assuming is facilitated in the kernel-futex design. If any kernel hackers want to chime in on why this works (and is safe (is safe?)) in the face of most unix specifications, generic docs and articles like this it may be enlightening.

0x04• 10 years ago

Do you have a source for this? Are you talking about pthread_mutex_lock() ? Maybe I'm misreading you, but it sounds like you are saying it would be safe to treat a mutex as unconditionally unlocked inside a signal handler, which doesn't make much sense (what if a signal is delivered between _lock() and _unlock()? )

gpderetta11• 10 years ago

You are well into UB land. The behaviour you describe is very dangerous as the signal handler will be accessing the mutex protected data structure while it is in a potentially inconsistent state.

The right, portable way to signal from a signal handler are POSIX semaphores that on glibc are a thin wrapper over futexes. Any data structure access must be non blocking.

Edit: autocorrect

jasonzemos3• 10 years ago

I should mention I was using x86. My initial assumption was that the kernel simply references the robust list (even if it was initially entirely resolved in userspace), and yields back to the interrupted thread -- I should emphasize my test showed the kernel breaks into the lock and presents a semi-coherent as-is structure entering and exiting the handler's lock. Of course this is all way way UB for portability or complex structures indeed...

Animats• 10 years ago

Signals are like interrupts, and like interrupts, they're handled in an unusual environment. That's the main problem. You can be inside some nonreentrant library when a signal handler is called.

Most programs that do something complicated with signals generate an event in the signal handler and put it on a queue to be handled later. The queue should be lock-free, or there's a risk of deadlock.

ambrop72• 10 years ago

Interrupts typically do not observably interrupt currently running code. In a simple system (e.g. embedded system with no threads, all event-driven), the interrupt handler will run, then the processor will go back to running whatever it was running before. This is not so for UNIX signals, in case you are in the middle of a system call, because the mere occurrence of the signal will change the behavior of the interrupted code.

Yes I know it's not the same because in my example single-threaded system there's no such thing as a blocking call.

Actually I don't see a good reason why signals in unix would have to cause EINTR errors in system calls. Perhaps a better solution would be to let the system call go on normally. Since the signal doesn't observably interrupt code not in a system call, why would it observably interrupt code in a system call?

In case anyone thinks, "so you can detect the signal in the main code", that is a bad answer because whatever you do you will have race conditions if the signal happens just before you enter the system call. Your only chance is to use things like ppoll() which are designed for proper signal handling, and these things could work just as well in a hypothetical unix design with no EINTR.

gpderetta11• 10 years ago

You can siglongjump out of a signal handler [1]. If you sigsetjump right before doing a blocking call, you can reliably detect signals.

Another way to avoid the race condition in poll/select, before p{poll/select} were standardized, was to store the timeout parameter in a global variable and have the signal handler set it to zero. Finally there is the self pipe trick, which admittedly doesn't require EINTR at all.

[1] This is historical unix behaviour. At one time it was specified by the SUS, but it seems that it was dropped from more recent SUS/Posix standards.

JdeBP4• 10 years ago

... and, thence, SA_RESTART.

scottlamb5• 10 years ago

Actually I don't see a good reason why signals in unix would have to cause EINTR errors in system calls. Perhaps a better solution would be to let the system call go on normally. Since the signal doesn't observably interrupt code not in a system call, why would it observably interrupt code in a system call?

You can request this behavior via the SA_RESTART flag. I'm not sure if it applies to all syscalls (the Linux manpage suggests not). I'm also not sure if things that take a relative timeout internally subtract the elapsed time when interrupted; one could imagine that each signal snoozes the timer on nanosleep or select/poll/epoll_wait so that you oversleep.

breadbox• 10 years ago

Also, signals become much easier to deal with if your program is single-threaded. Once threads get involved, it becomes more complex to know which thread(s) will receive a given signal.

signa11• 10 years ago

Once threads get involved, it becomes more complex to know which thread(s) will receive a given signal.

well, one approach that might be worth looking into would be to designate a special thread as a signal-handling-only thread. others just block every signal that can possibly be blocked. this signal-handling-thread then communicates the signals etc. to others as needed.

prima-facie, this boils down to signal handling for single threaded programs. what might be the downsides ?

scottlamb5• 10 years ago

People say "signals" as if they're just one thing, but I find it more useful to break them into two categories:

* process-directed signals such as SIGHUP, SIGINT, SIGWINCH, SIGTERM, SIGQUIT, SIGCHLD. They come from outside the process, including the `kill` command, the init system, and the terminal. For these, a dedicated signal-handling thread is a common, practical approach. Even if your program is single-threaded before implementing signal handling, creating a new thread might be the best approach. Or you could integrate signal handling with an event loop via the self-pipe trick.

* thread-directed signals such as SIGSEGV, SIGFPE, SIGBUS (the preceding are all machine exceptions), SIGPIPE, SIGPROF, or anything sent by pthread_kill / pthread_sigqueue. If you need to handle these signals (usually for diagnostics), by definition you have to do it in the thread in question. And you almost certainly need a traditional signal(2) / sigaction(2) style signal handler.

wyldfire• 10 years ago

As others have said, there's peril to be had there for sure, so tread carefully. Minimize the scope of your handler is the best advice, certainly also refer to "Async-signal-safe functions" in signal(7) if you must use libc funcs.

One challenge in a distributed system when there's (ab)use of signals is finding out which process issued a signal. There might be a better facility to do it now but I've used systemtap [1] to find out who the sender was with satisfactory results.

[1] https://sourceware.org/systemtap/examples/process/sig_by_pid...

pm215• 10 years ago

If you've ever read the Lions' Book commentary on 6th Edition Unix, you'll notice that many parts of the API as implemented back there are pretty solid -- quality, well designed interfaces that have stood the test of time.

Signals are not one of those parts. The 6th Ed signal handling code reads to me as somewhat of an afterthought whose use cases were mostly "kill the process for a fatal signal or terminal ^C", "ptrace for a debugger" and maybe SIGALRM. The data structures don't allow a process to have more than one pending signal -- if a new one comes along the fact an old one was pending is simply dropped. Running a signal handler automatically deregistered it, leaving a race condition if two signals arrived in close succession (this is a well known bug fixed by BSD later). And EINTR is an irrelevance if signals are generally fatal but its effects spread like poison through every other kernel API if you need your program to be reliable even with signals being delivered.

The worst bugs and races were fixed up by the BSD folks and others, but the underlying concept is an unfortunate combination of "basically irredeemable", "indispensable" (you have to have some kind of "kernel tells you something has happened" API, and signals are what we got) and "insidious" (thanks to EINTR). I think they're a strong candidate for "worst design decision in unix".

(PS: one of the reasons they stand out in 6th Ed is that so much of the rest of that code is so good!)

silvestrov2• 10 years ago

I think signals are a "poor mans" implementation of threads and queues. They didn't have threads and queues back in the 70'ies, so they resorted to an ugly hack. Ditto with interruptable system calls.

It's somewhat like the interrupt-level callbacks classic MacOS (version 1-9) had for network and file i/o. They too didn't have threads back then, so interrupt-callbacks was the only way to avoid blocking calls which would make the UI hang while waiting for network requests.

pvg• 10 years ago

I'm pretty sure they had threads and queues in the 70s. Or they wouldn't have bothered inventing primitives like semaphores in the 60s.

mturmon• 10 years ago

This is an interesting perspective. I think of signals the other way around -- that they came out of people realizing how useful hardware interrupts could be in allowing devices to talk to the kernel, so they engineered a similar affordance (signals) to allow the kernel to talk to processes.

In the case of hardware interrupts (like for old keyboards), if you didn't grab that character off the hardware right away, it would vanish, or another one would take its place. As long as you were fast, it worked. Same with signals. As long as you don't try to do too much in the handler, it works. In other words, it was designed to be "good enough" if used properly (just like all hardware).

adrianratnapala3• 10 years ago

...so interrupt-callbacks was the only way to avoid blocking calls

I don't know how early MacOS works, but on Unix its not the _only_ way: that's what poll(), select() and the like are for.

And while threads their uses, they don't really change this aspect. Eventually the thread must report a result, and the best way to do that is to report into to some queue that gets polled somewhere.

Koromix• 10 years ago

Neil Brown published an interesting series about Unix design mistakes a few years ago: https://lwn.net/Articles/414618/

The first part of the "Unfixable designs" article is about Unix signals.

Qwertious• 10 years ago

The worst bugs and races were fixed up by the BSD folks and others, but the underlying concept is an unfortunate combination of "basically irredeemable", "indispensable" (you have to have some kind of "kernel tells you something has happened" API, and signals are what we got) and "insidious" (thanks to EINTR). I think they're a strong candidate for "worst design decision in unix".

Suppose you were to throw the whole thing out and write a good replacement (and backwards-compatibility be damned), what would it be like?

jbarham• 10 years ago

FWIW Plan 9 uses what it calls a "notification" mechanism: http://plan9.bell-labs.com/magic/man2html/2/notify

reacweb• 10 years ago

Maybe, the signal coming from another process should be handled in an independent thread (maybe created the first time a signal is raised). Something would still be needed to interrupt pending system calls.

ktRolster• 10 years ago

systemd?

edit: Actually there are some fairly decent systems for IPC now, with the Mach kernel from CMU probably being the most popular example. Most systems with micro-kernels probably have something similar, and of course dbus is there, and Android intents.

I think Applescript deserves a mention here, because although it's a rather weird language, the IPC stuff is really really easy.

pjc504• 10 years ago

Suppose you were to throw the whole thing out and write a good replacement (and backwards-compatibility be damned), what would it be like?

Steal the best bits from Windows NT, and improve the existing mechanisms.

Kill signals in their current form. Build a general-purpose notification mechanism consisting of a mutex and a message. Possibly allow a process to have more than one message queue (Windows makes this really, really easy).

All IO, networking and informational signals (SIGWINCH, SIGCHLD etc) then come as messages (these may have to be fixed size, but anything from a few words to a 4k page would do). select, poll etc are replaced by waiting on a mutex. You can put all your worker threads waiting on that mutex. A message arrives. The kernel wakes one waiting thread and gives it the message (via an atomic deque-or-block syscall). You don't have to do any O(n) processing to work out which socket it relates to as the kernel has helpfully put it in the message.

In the deluxe 4k page version, a 1500-byte ethernet frame arrives, is DMA'd into the top half of a page, the kernel inspects it and sets message headers to say where the data is, and hands it directly into the userspace of a waiting process.

The one downside of this is that UNIX pipe programs become slightly more complicated. Rather than just doing "while(read()) write()" you'd have to switch on the type of message recieved and implement your own abnormal exit functionality. This could probably be tidied away for you inside the standard library.

External process control mechanisms would have to be built for killing processes and suspend/resume.

rtpg• 10 years ago

Considering that you can basically establish no invariants in the signals model, to me it sounds like a cooperative model is the way to go.

Have a way to poll for "outstanding" signals and deal with them accordingly, in some sort of queue (and the OS drops older messages or something, if they're important you'd deal with them).

The "freeze-the-world-and-do-this-thing-instead" model seems like the best way to introduce "my entire program is now broken"-style bugs. The OS deals with interrupts but the OS is also running by itself.

JoshTriplett• 10 years ago

Suppose you were to throw the whole thing out and write a good replacement (and backwards-compatibility be damned), what would it be like?

Keep the "you can always kill or stop a process" provided by SIGKILL and SIGSTOP/SIGCONT, using dedicated system calls. Handle every other kind of message (from the kernel or otherwise) via file descriptors, similar to signalfd.

gmazza• 10 years ago

(k)dbus is the most likely replacement here and systemd is a major vehicle for its adoption beyond DBus' current niche use in Gnome.

pjmlp• 10 years ago

No time to read it back again, but if I remember correctly not even POSIX defines portable semantics for signals across implementations, it always leaves some room for implementation specific behavior.

bitwize• 10 years ago

YES. Especially when there are much better ways of handling OS interrupts to your program -- like Structured Exception Handling under Windows.

wscott• 10 years ago

BitKeeper uses signals to implement a paging data structure from a compressed backing store. I allocate the memory for my data structure that is backed by a file on the disk and then use mprotect() to mark that memory as read-only. Later when trying to access that memory a signal handler traps the access and loads and decompresses the data from disk into memory.

This is only done for unix systems that implement the sigaction() POSIX signals. It is tricky to get right, but it does work.

BTW I did find I could never get OSX 10.4 to work correctly, but by 10.7 Apple had finally fixed the bugs in their signal code.

monk_the_dog• 10 years ago

Oodbs used a similar technique to translate addresses from a large "global" address space into smaller "local" one. Here is a paper if you're interested (ftp://ftp.cs.utexas.edu/pub/garbage/swizz.ps). Did you discover this technique on your own? At any rate, very cool.

zAy0LfpBZLC8mAC2• 10 years ago

I would say: Oh, yes, be as afraid as you can be. But don't let that stop you from figuring out why it is perfectly rational to be afraid of signals ;-)

Now, it is not impossible to use signals, but there are many opportunities to screw it up, often in non-obvious ways (so, things seem to work, but it's not actually reliable). And at the same time, signals almost never give you any advantage over alternatives if you do it correctly. That profiler thingy might be one of the rare cases where it actually makes sense.

In particular, what tends to be so tempting about signals is that they are executed "immediately", so you get to react without any further delay, no matter what else your program is currently doing--who wouldn't want that? Except that doesn't actually work, because you need to somehow access the state of your program in order to actually do anything useful with the signal notification. But you cannot access that state unless you can be sure it's in a consistent state and that your accesses won't interfere with what your program is doing in some unpredictable way. Just like in multithreaded programming. You have to somehow coordinate with your program to make sure things happen in an orderly fashion. Which essentially means that you only can access the program's state at certain times when the program isn't currently using it. Like, when it is unlocked. Except the signal handler potentially preempts your program, so you can't use locks to perform the coordination, because that could deadlock. Except if you were to use locking primitives that also block signals, so that preemption during critical sections can't occur. But then, you effectively have a weird polling solution (the unlocking at the end of the critical section/before entering the event loop dispatcher effectively acts as if you were polling for signal events).

Also, you cannot even reliably queue signals without potentially dropping some. Now, the kernel does that anyhow, so you can't rely on all signals being delivered individually anyhow, but it still is important to understand why that is (which is also why the kernel behaves the way it does): If you consume events, you have to either have some mechanism to slow down the source to prevent it from generating events at a higher rate than you can handle (like, if you can't keep up reading from a pipe, the writer end of that pipe will block in order to stop it from producing more data), or you would need potentially infinite amounts of memory to be able to store all those events for later processing. Now, the latter isn't really possible, of course - but it's even worse in signal handlers because you cannot really allocate memory there because there is only one memory allocator in the libc and that most definitely is not reentrant (like, you cannot allocate memory right in the middle of your thread freeing memory).

What this boils down to is that you always have to somehow defer processing of signals to some point in time where you can actually safely access your program's state, which is something that you can achieve with pipes and sockets much more easily.

euske• 10 years ago

I found a paragraph in this article http://www.linusakesson.net/programming/tty/ very apt at describing what Unix signals are like:

  In *The Hitchhiker's Guide to the Galaxy*, Douglas Adams 
  mentions an extremely dull planet, inhabited by a bunch of 
  depressed humans and a certain breed of animals with sharp
  teeth which communicate with the humans by biting them very
  hard in the thighs. This is strikingly similar to UNIX, in
  which the kernel communicates with processes by sending
  paralyzing or deadly signals to them.

blucoat• 10 years ago

SIGSEGV is a very important signal. It happens when your program tries to access memory that it does not have. An appropriate reaction might be to

    allocate more memory
    read some data from disk into that memory
    do something with garbage collcetion (but what? I'm confused about this still.)

What? Are there any Real World Programs which do anything other than print a stacktrace and exit? I don't think this person gets what a segfault is.

barrkel• 10 years ago

Windows uses page faults in the stack guard page to lazily commit stack pages. Compilers allocating large structures on the stack need to generate loops touching each allocated page in turn to guarantee the allocation. On Windows the lazy allocation can be done entirely in user code - it doesn't need to be an OS feature. I believe pthreads uses the same technique on Linux; very far from sure though.

Generational GC can use segfaults to detect writes to older generations and mark pages that need scanning for references to younger allocations. They can also act as a way of triggering a safe point without polluting the branch prediction cache: unmap a page when you want an interrupt, and periodically touch the page in code that needs interrupting (loops etc.). Virtual machines for languages like Java can and do use these techniques.

kaushiks• 10 years ago

While I agree that it'd be unusual for regular applications to have to resort to using SEGVs to implement features, low level systems code, especially VMs often do, for performance reasons. The Hotspot JVM for instance uses SEGVs to force a thread into a safepoint. The JIT inserts a read instruction, among other places, at backward branches, which tries to read from a page in memory called the polling page. Said page is mapped during normal operation of the application. When the VM needs to bring threads to a "safe point", say to perform a GC, it does so by unmapping the polling page. This causes each of the active threads to fault on the read and enter the SEGV handler, which notices that the faulting address falls within the polling page and executes appropriate "safe point" actions. Libc implementations use a similar technique to commit pages for a thread's stack lazily.

hornetblack• 10 years ago

If you had a Green threaded program and one of the threads segfaulted. You would probably want to catch Segv and kill that thread. (Not killing the OS thread running it).

I've also seen is used to implement a distributed malloc. When a segfault occurs, the handler messages the programs peers asking if they have the data for that address. If so the peers sends the page and the handler maps in a new page for that address with the correct data in it. This is essentially implementing a page fault handler in user space. (For some network backed memory).

gpderetta11• 10 years ago

Why would you only want to kill that green thread? On any thread implementation I'm aware of, an unhandled segfault kills the whole process. Anything else is disaster waiting to happen.

bediger4000• 10 years ago

I've read that one of the original Unix shells (Thompson's or Bourne's) used a combination of sbrk()/brk() system calls and SIGSEGV to do dynamic memory allocation for itself. I can't find a reference to this via Google, as any information about old shells and SIGSEGV is swamped by modern people talking about bash and bad programs, or trapping SIGSEGV in scripts or some such. The "heirloom sh" code doesn't have anything like that, but it's clearly been tinkered with, as it uses sigaction(), a BSD innovation.

So, feel free to ignore this vague memory.

known• 10 years ago

Consider this code highly experimental and yourself highly mental if you try and use it in a production environment

http://www.kegel.com/c10k.html#examples.nb.sigio

0x04• 10 years ago

I've never seen a safe signal handler beyond a SIGTERM that sets a "volatile int time_to_quit = 1" for a main loop to pick up on later...

JdeBP4• 10 years ago

volatile int isn't necessarily safe. volatile sig_atomic_t would be, however.

jfoutz• 10 years ago

This is one of the great counter examples to betteridge's law.

ambrop72• 10 years ago

The worst thing about signals is that they interrupt whatever system call the thread is currently inside (EINTR). This is not typically observed but can have dire consequences randomly. For example, last time I checked, in Python 2.7, a signal that invoked a signal handler will cause a running print() to throw an exception. Here you should consider signals like SIGCHLD which you want to handle and not kill the process.

A particular case this happens is when doing event-driven programming. The only way to be sure that you don't have such bugs lurking is to setup signal handling such that a signal cannot possibly interrupt unsuspecting code. Currently, I'm aware of two solutions, both involve blocking signals:

1. Block relevant signals in the main loop (or generally all threads) and use signalfd to detect and consume signals (or similar mechanisms on other platforms, e.g. kqueue).

2. Start a dummy thread whose only purpose is to handle signals, leave relevant signals unblocked in this thread and block them in all other threads. Write the signal handler to communicate the signal to your main loop via the self-pipe mechanism or similar.

Note that solution (2) can usually be implemented for an existing framework without changing that framework - you only need to add code to main which starts that thread then blocks signals, before any other threads are started.

I consider "fixing" code to be robust to signals a non-solution, because you would have to verify every single piece of code running in your program, including third-party libraries.

ptx• 10 years ago

For example, last time I checked, in Python 2.7, a signal that invoked a signal handler will cause a running print() to throw an exception.

Python 3.5 fixes this – system calls are now automatically retried:

https://docs.python.org/3.5/whatsnew/3.5.html#pep-475-retry-...

gpderetta11• 10 years ago

The big problem with unix signals is that they have been abused to deliver some messages that should really be delivered via a message pipe (e.g. SIGCHD, all the terminal/tty specific signals). The other is that the set of signals is limited and signal handlers are a process (or thread) wide resource, so it is hard to make use of them in a library.

Other than that, the general ability of interrupting and delivering a message to a thread no matter what is doing is necessary and signals are a way to implement that. Exceptions are another way, but that can be implemented on top of signals.

edit: but there is really no excuse for EINTR. The "Worse is Better" essay has something to say about this.

jhallenworld• 10 years ago

When I first started programming UNIX I thought SIGIO made sense as a good mechanism for I/O multiplexing. I thought this because of much previous experience with interrupt handlers in the embedded world. However at the time, it just did not work (no sigsuspend) and even today it's a big mess. SIGIO should just be removed- it's the wrong way to handle I/O in UNIX.

ausjke• 10 years ago

I think these days we all should use sigaction instead of signal()? why is this totally missed in the post.

http://stackoverflow.com/questions/231912/what-is-the-differ...

Should you be scared of Unix signals?

Related Stories