Beej's Guide to Unix IPC (2010)

http://beej.us/guide/bgipc/output/html/multipage/index.html

by adamnemecek • 11 years ago

148 31 11 years ago

beej.us

pizza• 11 years ago

Also check out his guide to C programming [0] and his guide to network programming [1]

[0] http://beej.us/guide/bgc/ [1] http://beej.us/guide/bgnet/

rubiquity3• 11 years ago

I owe my career to Beej. I was getting burnt out of web development and his guides (along with Jesse Storimer's books) made programming fun for me again. I highly recommend learning how Unix systems work. It's a lot of fun and opens up a whole new world of programming!

shabbyrobe• 11 years ago

I'm feeling the same way about web, I'd like to transition to something similar. I quite enjoy programming in C, so hopefully that helps. What are you doing at your job now? How did you go about making the switch from web to systems? Has it paid off with an increase in enjoyment?

rubiquity3• 11 years ago

Truth be told, a significant portion of my day job is still web application stuff. The web, whether that's HTML/CSS/JS applications or APIs, is inescapable at this point for a huge portion of programmers. Only the lucky few get to hack on databases, hypervisors and file systems for a living!

But, armed with the knowledge of sockets, processes, etc. and how this correlates to performance/scaling, I've been able to carve out roles where I get to spend a decent amount of time working on problems that I get excited about.

But even before I was able to work the knowledge into my day job, nothing beats the dopamine rush of learning things that fascinate you! The rabbit holes that learning the basics of Unix have opened feel like they could occupy my hobbyist hours for the rest of my life.

ridiculous_fish5• 11 years ago

Something I've struggled to implement on Linux is cross-process multicast notifications, where any process can post and multiple subscribed processes receive it. FIFOs and SysV IPC are unicast, and I think DBus is too.

On OS X there's notify(3) which is very nice. Any good options on Linux, other than writing my own socket server?

troglobit• 11 years ago

TIPC is pretty neat, it's available in the kernel since quite a long time (AF_TIPC) and works between processes on one or many nodes.

http://tipc.sourceforge.net/

pdkl953• 11 years ago

The solution for this kind of problem will depend heavily on what type of messages you're trying to send. When messaging becomes that complex, there are often other things that impact the overall design in important ways that need need to be considered.

For example, if I was implementing something that is usually associated with user events (rare-ish, basically zero bandwidth, complex signal with stateful messaging semantics), I would probably just write a simple server to manage it all. This has an advantage of centralizing any messaging complexity and lets you manage any multi-message state easily. Rebroadcasting messages to allow peer-to-peer messaging would be a trivial addition. This would probably UNIX sockets if the connections are persistent (which can changed into AF_INET{,6} sockets easily, if you wanted to add network support).

For something that requires very low latency (e.g. audio/MIDI), in the past I have had to use shared memory, which has zero overhead once it is setup (no syscalls or context switches). Here, the need for low latency dictated the design. Of course, this means managing locks. Not fun, but a cost that is sometimes worth paying.

There really isn't a one-size-fits all solution.

edit:

Or, as troglobit said, TIPC. I keep forgetting that we now have it has an option. :(

ridiculous_fish5• 11 years ago

Thanks for the reply. My use case is very simple: a stateless "something happened" notification, which can be delivered asynchronously. Coalescing or even occasional drops are fine.

I did originally use a Unix domain socket server, but that added a lot of complexity: one has to arrange for it to be launched, guard against the possibility that it gets stuck, version it, deal with permissions, etc.

My new solution on Linux is a total hack: there's a FIFO, and to post a notification, you write to it. Clients see that the FIFO became readable, and that change represents the notification. The sender then drains the data it wrote, so that the FIFO becomes unreadable again. This is a total abuse of FIFOs, but it's proven to be much simpler than trying to manage a separate server.

I've never heard of TIPC. From a little searching it looks like it's very capable but geared towards clusters, and is overkill for my use case. What do you think?

ekiru• 11 years ago

dbus signals can be multicast. Specifically, dbus signals without a destination are routed to all connections with match rules (added with org.freedesktop.DBus.AddMatch on the message bus) which match the signal.

ploxiln• 11 years ago

Classic signals are tricky... but you can send a signal to all members of a "process group".

see: man killpg, man setpgid

also, "zeromq" has a pubsub mode, though I've never used it and I'm not sure about its limitations

easytiger• 11 years ago

zeromq over loopback or with pgm is prob as good/fast as UDS

gnachman• 11 years ago

I haven't tried it, but you should be able to have multiple processes listen on the same UDP port bound to localhost using SO_REUSEPORT. Send broadcast UDP packets (using SO_BROADCAST) so they all get it.

From here: http://www.kohala.com/start/mcast.api.txt

"More than one process may bind to the same SOCK_DGRAM UDP port if the bind() is preceded by:

int one = 1; setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one))

"In this case, every incoming multicast or broadcast UDP datagram destined to the shared port is delivered to all sockets bound to the port. For backwards compatibility reasons, THIS DOES NOT APPLY TO INCOMING UNICAST DATAGRAMS -- unicast datagrams are never delivered to more than one socket, regardless of how many sockets are bound to the datagram's destination port."

rdtsc• 11 years ago

You can do it with shared memory. One writer writes and multiple readers can observe. I did this for both low latency and throughput reasons.

In general you have to be very careful how you handle it and consider various consistency and failures scenarios.

The main part of memory layout looks something like this:

  [write_counter][......buffer.....]

This is owned and updated by the writer. Readers have a read_counter that they maintain in their own context (not shared).

You'd probably have to declare this using the 'volatile' keyword. Otherwise compilers will optimize away access to part of this (seemingly) unused variables.

Then it works like this:

The writer_counter value is always counting up when writer writes. Data gets written to buffer. Then "write_counter" is incremented. Both reader and writer index into buffer by using {write|reader}_counter%buffer_size.

Also, note these counters also function as total counts of items written and so each reader cand determine how far ahead the writer is.

Another note: depending on the sizes of your counter it will not necessarily be updating atomically. Compiler could separate the update as multiple instructions and say, increment the lower part of the value, then the upper part. Writer could get pre-empted between those two instructions, so you could get this strange torn value. In this case because of the % you'd still fall into the range of the buffer. But, you might be reading data you didn't expect. Whether that works for your use case or not you'll have to see.

EDIT: Don't also forget about the slow and stupid multicast mechanism -- writing to a file. Some file operations can be atomics (renaming a file). And some operating systems let you watch the files for changes.

vezzy-fnord• 11 years ago

You can actually implement a basic pubsub (both one-to-many and many-to-many) mechanism using FIFOs and file system permissions in a particular fashion known as a fifodir:

http://skarnet.org/software/s6/fifodir.html

http://skarnet.org/software/s6/ftrig.html

ladybro• 11 years ago

Beej helped me pass my Operating Systems final last semester. Really quality and entertaining guides he puts together.

alekratz• 11 years ago

Beej is great. I've printed this book out and have it sitting in a binder on my bookshelf.

jrapdx3• 11 years ago

I'll add to the chorus of praise for Beej's work. I've often consulted this particular guide and the companion piece for networks as well. Helped me write web and other servers, a great way to learn about important technologies, providing knowledge that stays useful even if the production server runs on Node.js.

Besides, I'm biased. "BJ" happens to be my wife's unofficial name. She's a remarkably smart person, so I was predisposed to think "BeeJ" would know what he's talking about and it turns out he did.

paraiuspau• 11 years ago

Ahh.... summer 98, discovering sockets, too poor to afford Stevens, 14.4 kbps modem, netscape 4.x rendering Beej's .edu site (forgotten now), me staring my future in the face.

Thank you for your efforts, Brian!

am185• 11 years ago

beej! how can i forget the his network programming guide!

stuaxo• 11 years ago

Thanks for reminding me, I really need to put this into practice, so much great stuff in here.

ausjke• 11 years ago

what about an update to posix IPC instead of sysV? I thought Posix IPC is "better" these days?

Beej's Guide to Unix IPC (2010)

Related Stories