* is Unix

http://jacobian.org/writing/star-is-unix/

by jacobian • 17 years ago

62 47 17 years ago

jacobian.org

_b8r0• 17 years ago

No. * is most definitely not Unix. Unix is Unix. Anything else is not Unix.

tvon• 17 years ago

I think you might be taking the title of the post a bit to literally.

jrockway5• 17 years ago

Sometimes it pays to click through to the article before commenting.

wooby• 17 years ago

The alternative to preforking is select(2), which lets you get away with a single-process server and avoid IPC in certain scenarios: http://www.lowtek.com/sockets/select.html

I think Boa and thttpd use select instead of forking for clients.

bretthoerner• 17 years ago

You have to combine select with processes or threads if you want to take advantage of SMP like these examples do.

maw• 17 years ago

select doesn't scale particularly well, though. (And I'm not wild about its interface either; poll conceptually similar but much nicer to program to.) http://www.kegel.com/c10k.html (a now fairly old document, by internet standards) discusses a number of other options.

silentbicycle5• 17 years ago

You might want to look at libevent (http://www.monkey.org/~provos/libevent/) and libev (http://software.schmorp.de/pkg/libev.html). They're wrappers to kqueue (on BSD) and epoll (on Linux), both of which improve upon select (http://www.monkey.org/~provos/libevent/libevent-benchmark.jp...).

mace• 17 years ago

Libevent is great for multiplatform code. PyEvent is the Python wrapper module for libevent also written by dugsong.

antonovka11• 17 years ago

This meme is a silly amateur-hour wheel re-invention and ignorance of history.

The fact is, pre-forking a pool of processes without implementing a per-process multiple-connection event mechanism is going to be grossly inefficient in terms of memory utilization (ie, you need to use select, poll, kqeueue, etc.).

You can claim otherwise (COW pages et al), but history is not on your side (look at the scalability of existing pre-fork/connection-per-process network daemons). Think about it -- how much of the runtime can truly be shared across connection handlers?

The fact that this model doesn't scale is not news. It doesn't require rediscovery. It shouldn't be surprising. Anyone who has run Apache, sendmail, or any 80s/90s networking software -- much less authored something similar -- will know better.

Moreover, there's nothing wrong with threads -- threads are a low-level tool that can be used to implement high-level concurrent systems. They're cheaper than processes because they do support mutable state where it's useful, but you don't have to use fork() to avoid avoid mutable state.

If you don't want shared mutable state (and you don't), then don't use it.

jacobian• 17 years ago

This meme is a silly amateur-hour wheel re-invention and ignorance of history.

Yeah, well fuck you too.

As it turns out, I am an amateur, at least when it comes to POSIX, but more importantly I find this stuff fun. I hadn't realized that having fun and learning about new topics was such an offensive thing to do.

Maybe this is just frivolity, but it's surely a better way to spend a few minutes than crapping all over somebody else's fun.

antonovka11• 17 years ago

My intention was to elucidate why the fork(2) non-event-based "mini-meme" is known dead-end.

claymation2• 17 years ago

Right on. I'm all about learning, sharing knowledge, and having fun. Kudos to you and Ryan for having the balls to study in public.

What I find interesting, and somewhat alarming, though, is that experienced web developers such as yourself seem to only now be learning the fundamental, bedrock principles of how the technology you use every day actually works.

Maybe it's because high-level abstractions weren't widely available, or because the books I had on hand were Stevens' classics, but I learned network programming starting with socket(2), and to this day still ask what each of socket/bind/listen/accept do when I'm interviewing sysadmins, much less developers.

It's remarkable to me that a whole breed of web developers today might not have any exposure to networking below the HTTP stack offered by their favorite programming language.

davidw2• 17 years ago

Err... Apache does not simply fork for each incoming request.

antonovka11• 17 years ago

Apache traditionally used a pre-fork model. Later work has been expended on moving away from this model: http://httpd.apache.org/docs/2.2/mpm.html

I attempted to address both pre-fork and fork-per-request above -- both involve mapping one connection to one process -- this is resource intensive, and should be replaced with more modern event-based multiple-connections-per-process (select, poll, kqueue, etc.) approaches.

viraptor• 17 years ago

While I agree in general, there are of course exceptions. "look at the scalability of existing pre-fork/connection-per-process network daemons" -- exactly, look at SER/opensips/kamailio - while they work on a per-packet, not per-connection basis, the architecture is mostly the same - fork a pool of processes on startup and share the sockets. And they could probably still handle a small city's phone connections on a single average machine ;)

antonovka11• 17 years ago

the architecture is mostly the same

Well, not exactly. Per-event delegation of processing to subprocesses (as in the case of opensips) is an multiple-connection-per-process event-based model. It's also a very complicated way to achieve it, and not what's being discussed in these "... is UNIX" blog posts.

catzaa2• 17 years ago

> This meme is a silly amateur-hour wheel re-invention and ignorance of history.

Just a note on the tone of the discussion:

Everyone is a silly amateur at one stage when learning something – there is nothing wrong with it. You have to do things that are silly and immature to learn. Most of us did some silly things in areas where we are not yet specialists.

antonovka11• 17 years ago

I think it's a responsibility of everyone in any scientific or engineering field -- specialists and "not-yet-specialists" alike -- to think long and hard about advocating an approach without studying the large body of academic and practical work that exists for nearly every subject imaginable.

jrockway5• 17 years ago

Indeed. I hate to contribute to this, but when I get home, I am going to blog about my coroutine-based echo server. Fewer lines of code than the original, scales linearly over active connections (and constant time over idle sockets), and my tiny eeepc laptop can easily handle 1,000,000 open connections.

Sometimes "worse is better" isn't.

apgwoz• 17 years ago

If you aren't joking, I want to see this.

omouse2• 17 years ago

Are you using coroutines in Perl or some other language? And I wanna see that code in any case :p

ilyak• 17 years ago

nginx is Unix

It's obviously designed by Unix orthdoxes and it shows

No dynamic linking! No threads! Not much of mem alloc, either!

claymation2• 17 years ago

I love an in-depth, technical blog post as much as anyone, and kudos to Ryan for dissecting Unicorn, but I didn't know whether to laugh or cry when I read that Unicorn has, "teh Unix turned up to 11". zOMG! fork!

This series of blog posts underscores my point that the Rails community, perhaps the Web 2.0 community at large, is just now learning things the rest of us have known for years. accept(2)-based pre-forking socket servers are not news.

silentbicycle5• 17 years ago

> "teh Unix turned up to 11". zOMG!

Yeah. Ruby people suddenly feeling all hardcore after touching Unix syscalls for the first time. Ick.

* is Unix

Related Stories