Why Ruby App Servers Break on MacOS High Sierra

https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/

by mef • 9 years ago

259 126 9 years ago

blog.phusion.nl

joshmn• 9 years ago

This issue has been addressed on ruby-head

https://github.com/ruby/ruby/commit/8b182a7f7d798ab6539518fb...

akvadrako• 9 years ago

It's been addressed, but it's a hack. To be safe, apps must stop all other threads before forking.

The only correct fix I can see is to have pre- and post- fork hooks in every library, which make that guarantee.

nateberkopec• 9 years ago

In Puma, we emit a warning if we detect any additional threads running prior to fork.

jasonmp85• 9 years ago

Right; this article is almost a week old at this point and some progress has been made in addressing this. In particular, I found the characterization that other app servers are not taking action on this problem to be a lie in poor taste.

This kind of talking up of one's own product and saying "we're first; we're best!" rather than working within the ecosystem to help fix a shared issue confirms my feeling that Passenger is a bit of an outlier in the Ruby community.

oblio2• 9 years ago

This is so incredibly Apple :)

The breakage, I mean. To clarify a bit, for better or for worse, this is what Microsoft does, totally different psychology: https://blogs.msdn.microsoft.com/oldnewthing/20031223-00/?p=...

mrpippy• 9 years ago

Apple isn’t breaking old/existing binaries, the new behavior only applies to binaries compiled against the 10.13 SDK.

gradstudent• 9 years ago

is that so much better? I dread every new release of macos because it always breaks some part of my toolchain (either gcc and friends, or valgrind or whatever). It wasn't so bad when a new os came every few years. now they break stuff every year. six months since I bought a new machine and my gdb still isn't working quite right.

abritinthebay5• 9 years ago

It is very different but I wouldn’t argue better. It’s lead to a lot of their problems with Windows APIs

pilif• 9 years ago

I honestly prefer an early crash to random memory corruption

grandinj3• 9 years ago

that is exactly what I dislike about Microsoft. The right answer is to detect those dodgy applications and disable them, and notify the user that the program is "bad", thus providing a severe disincentive to software authors to ever do such things again.

nimish• 9 years ago

Why should users pay for developers' poor choices? This is a great way to piss off people who would blame Microsoft.

There isn't a great answer to this that satisfies everyone.

ygra2• 9 years ago

That mechanism also exists, notably in Vista and 7 there were a number of older applications where Windows would display a warning prior to running it that the application has known incompatibilities and may not run correctly.

However, a user who just upgraded Windows and half their programs stop working will simply not upgrade. In many cases such dodgy applications are also never fixed because the vendor doesn't exist anymore, doesn't care, or the flaw is in some library they use and they don't really want to invest that much time to change it.

Besides, detecting such behaviour is pretty much the same as fixing it with a compat shim, since you have to do more or less the same work. So from MS's perspective the benefit of working around the buggy behaviour is much more prominent than trying to discipline developers by making users suffer.

geocar5• 9 years ago

So Unix people had this function called `gets` that was defined like this:

    char *gets(char *at);

In the early days, if someone wanted a string you could do:

    x = gets(sbrk(0));brk(x + strlen(x) + 1);

And this is perfectly safe, but it is perhaps the only safe way to use `gets`. See, most people wanted to write:

    char buf[99];
    x = gets(buf);

And is this not safe because `gets` doesn't know that `buf` only has room for 99 bytes.

The API has a choice:

a) They can make it harder to do the wrong thing; make `gets` generate errors, warnings, crash if you use it, etc. This requires people fix their programs. That's what GNU did for `gets` and it's what Apple is doing here.

b) They can change the rules: It's possible to modify the ABI and require markers or a mechanism to detect where the edge of the buffer is. This requires people recompile their programs. I think Zeta-C had "fat pointers" so gets should be safe there[1]

c) They can work around: If you have enough resources, you can look at the programs using this API and figure out what their buffer sizes are and hardcode that into the API provider. Microsoft has famously done this for SimCity[2]

That's it. They can't really do anything else: The function is difficult to use right and programmers do the easiest most-obvious thing they can. oh i need to get a string, so i'll use gets... but i need two strings so....

Anyway, I'm not aware of any good rules for choosing which way to go: Less total effort seems like a good metric, but this is very hard to estimate when you live on an island and don't know how other people use your software.

Memory corruption is serious though: It's often very easy to turn into a security hole, so I generally advocate doing something. All of the people just disabling this security feature make me nervous. I wonder how many of them run websites that I use...

[1] http://www.bitsavers.org/bits/TI/Explorer/zeta-c/

[2] https://news.ycombinator.com/item?id=2281932

fanf2• 9 years ago

Surely that gets(sbrk(0)) will not work since sbrk(0) returns a pointer to unmapped memory. Maybe you wanted sbrk(BUFSIZ)?

throwme2113454• 9 years ago

fork() is not gets and you don't f*ck with process creation. If a programmer commits an error that deadlocks or corrupts data by using fork() unsafely they really should have known what they were doing in the first place.

jjnoakes• 9 years ago

Modern brk() can return -1 to indicate there's insufficient space above the current break address to allocate the requested additional space. I assume "in the early days" brk() could also fail to allocate space (after all the space available is not infinite) so "perfectly safe" comes with caveats.

ScottBurson2• 9 years ago

Wow, somebody remembers Zeta-C! Yes, you're right, 'gets' was safe in Zeta-C. (I'm the author.)

wruza2• 9 years ago

If I ruled MS or was a big boss of Raymond, I would ask him to show a big screen-dimming warning before running these programs, like “This application uses unsafe methods of interacting with your computer and works only because we decided to support it, spending two weeks to make it work. Still no safety guarantees, btw. Please ask Vendor, Inc. to read http://doclink on how to do his shit the right way.” [Okay] [Damn, FTS] [Send email for me]

No chances I make it to the top of MS though.

oblio2• 9 years ago

As he nicely explains in some other blog posts, sometimes Vendor, Inc. went out of business in the 80's but the apps are still in use. So you'd just annoy the poor dweeb from Human Resources that has to use the OS and app provided by his employer, the only job he could find in Podunk :)

valleyer3• 9 years ago

Apple does this too. Run "strings" on various libraries in Apple's OSes and you will find plenty of instances of bundle identifiers of non-Apple apps.

the_mitsuhiko• 9 years ago

Python suffers from the same issue but the response there has largely been to stop using modules that use objc.

captainmuon2• 9 years ago

I was thinking about Python when reading this, the real culprit is not objc but the third-party modules.

If I understand correctly, this problem is caused when you call initialize in another thread, and while that is running, you fork.

It affects these app servers because this is triggered at "require" (or "import") time. This is madness, no module should run code just when you import it. OK, I have been guilty of that, too. But if absolutely neccessary, keep it to a minimum. It breaks all kind of things when module imports are not side-effect-free.

Launching a thread and (indirectly) taking a mutex is definitely not something you should do on module import!

dchest2• 9 years ago

As far as I understood, the module isn't launching threads at import time. On the contrary, the problem is that since pg is linked against macOS's Kerberos and LDAP frameworks, as soon as the Objective-C runtime realizes that NSPlaceholderDictionary initialize method was called after fork(), it crashes:

One of the rules that Apple defined is that you may not call Foundation class initializers after forking.

pantulis• 9 years ago

I love this geek-porn stuff, and the Phusion guys never fail to deliver it ;)

But my question is: is this really that important? I mostly use macOS for development, don't feel that the preforking model has that impact in the development cycle.

digger250• 9 years ago

You are not affected, but there are people in the world besides yourself who do things differently than you.

FooBarWidget• 9 years ago

It is important for dev-prod parity. You will want to test whether your code is compatible with preforking, which you will likely use in production.

lima• 9 years ago

Why do people run applications servers on macOS?

bantunes• 9 years ago

Maybe they're developers running a local copy of the projects they're working on?

geocar5• 9 years ago

The underlying problem also affects Linux.

Apple is probably tired of bug reports to their software that really stem from some other place.

tonyedgecombe• 9 years ago

https://www.apple.com/uk/macos/server/

giancarlostoro• 9 years ago

I've mostly seen it for managing multiple Macs in a classroom environment. Not sure I've seen it used heavily in production to run web servers though. Was curious and was able to find sites that do let you rent Macs that you remote into, interesting setup since you know exactly the hardware you're getting into.

mhandley3• 9 years ago

Interesting discussion. If these were user-space threads, like FreeBSD ~20 years ago, there'd be no problem. When fork() is called, the whole user-space threads package would be forked, along with all the threads.

So the obvious question is whether it's fundamental that with kernel threads the fork() system call doesn't clone all the other threads in the process? Yes, that's not how it's done, but could Apple choose to implement a new fork_all() system call? I imagine it wouldn't be easy - you'd need to pause all the running threads while you copied state, but is there a reason it's actually not possible?

mhandley3• 9 years ago

Thinking about it some more, one problem would be what to do with threads that have just called a system call and are waiting for a response. To have both parent and child process system calls complete without errors would require cloning the kernel state, and that's not trivial. It some cases, there's probably no good answer, short of causing one of the two system calls to return an error, and that in itself would confuse a lot of user-space software. I suppose you could return something similar to EAGAIN, but you'd be dependent on user-space handling that cleanly. Maybe you could hide this under libc?

wruza2• 9 years ago

EINTR? I suspect some calls are defined to never return it, but it may be just de facto, not strict rule.

Edit: simply signalling all threads with SIGFORK, waiting for a handler to return and freezing a thread at SA_RESTART point feels right way to do it. Why did POSIX decide to fork() only one thread by default at all?

ScottBurson2• 9 years ago

So this is a worse-is-better design decision driven by a previous worse-is-better design decision. If system calls were interruptible, then the problem of threads in system calls would go away.

geocar5• 9 years ago

Is this what you want most of the time?

If you're just going fork() && exec() then why would you copy all that state just to run some subprogram?

Is that what you want any of the time?

This prefork implementation is silly. Either do prefork after initialisation so you can take the benefits of COW, or don't bother: Just use SO_REUSEPORT and run 10 copies of your server. This distributes the TCP traffic and provides you an excellent way to upgrade-in-place (just roll up the new version as you roll-down old ones).

mhandley3• 9 years ago

I agree - most of the time, the current fork() implementation is what you want, so it's the right choice. The conventional wisdom is that you shouldn't spawn threads and then fork(), and so long as you stick to this, you're fine.

Today though, everyone loves threads. It's hard to know precisely what happens in some low-level library, so it's hard to be sure that you don't have any threads left running after initialization. This is the subject of the OP.

My question though is broader - if we chose, could we add a version of fork() that did clone all the threads? I'm not entirely sure what it would be used for, but I'm sure there would be uses. Likely some of those would be for increased security, as processes provide stronger isolation.

gcb02• 9 years ago

because that what everyone that uses fork want. memory access. otherwise they would use posix_spawn or something

4ad2• 9 years ago

Solaris has forkall(2) and forkallx(2) system calls.

krisives• 9 years ago

What does Linux do right now?

dchest2• 9 years ago

If you fork after launching threads, your program is an unexpected state, so it may corrupt data or crash.

The same happens on macOS, it's just that Apple added a contract that you can't initialize ObjC classes after forking.

throwaway613834• 9 years ago

fork() fundamentally does not make sense as the de-facto method of starting a new process. Why aren't people using posix_spawn() by default?

tedunangst3• 9 years ago

Because that doesn't share memory.

twic• 9 years ago

You can share memory in other ways, via mmap or SysV shared memory.

You could imagine an implementation of the Ruby interpreter which kept all the expensive and shareable state (bytcode and class metadata, i suppose) in a shared memory segment, and used spawning to create worker processes with their own process heaps, but sharing that segment.

It would be a lot of work to implement. It would probably be easier just to get Ruby to use threads properly.

segmondy• 9 years ago

implementation of fork is defined by the POSIX standards. Apple should respect it. There are tons of apps that have been written using fork, are you going to change all of them to use posix_spawn()?

tedunangst3• 9 years ago

Posix standard says you can't call anything but async functions after fork() in a threaded program. Application developers should respect that.

saagarjha6• 9 years ago

Greg Parker, who works on the Objective-C runtime, has a blog post that goes into more detail: http://www.sealiesoftware.com/blog/archive/2017/6/5/Objectiv...

hinkley• 9 years ago

So, basically Objective-C doesn't like the RAII pattern if the resource is a process? Ouch.

I got the gist of his summary but his writing is a bit... awkward. Is English not his first language?

saagarjha6• 9 years ago

Is English not his first language?

As far as I know it is.

valleyer3• 9 years ago

C has this same problem. Most of the C standard library is unsafe to use between fork and exec of a multithreaded program. This usually includes malloc and printf.

ransom1538• 9 years ago

I agree the bug should be fixed. But, why not just use docker, then run rails like its on ubuntu/linux on your mac? It miserable having windows/mac/etc specific issues.

sillysaurus32• 9 years ago

Docker grows to >8GB. I have 13GB free, 486GB used. My free space fluctuates by as much as 20GB depending on how much RAM I'm using. (And by "I'm" I mean "Chrome.")

I've never felt the need to install docker on my local dev environment. It's great for production, and I'm sure it's great for people who can afford the disk space. But when space became tight, Docker was the first to get the axe. I haven't missed it yet.

jasonjei• 9 years ago

I'm just curious how you would deploy on a container production environment if you can't test all the platform-specific issues while developing on Mac instead of using Docker to develop for a container platform. Isn't the whole point of using Docker to minimize the differences between production and development?

throwme2113454• 9 years ago

LOL. mac osx breaks fork() to avoid state inconsistency in threaded applications. How about pthread_atfork() semantics? But ,as usual, apple heavy hands userspace and breaks things. Nothing new to see here, move on.

throwme2113454• 9 years ago

What you don't like the fact that apple sucks for breaking userspace (as usual) or that pthread_atfork() type approaches should be in every programmers toolbox?

oshepherd• 9 years ago

POSIX has deprecated pthread_atfork because it is unworkable. In particular, atfork handlers can only call AS-safe functions, which means they're useless.

mkj• 9 years ago

Do developers just call whatever function seems to work without reading the docs? It doesn't work for low level programming.

  ~ man fork

CAVEATS: There are limits to what you can do in the child process. To be totally safe you should restrict yourself to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented...

tedunangst3• 9 years ago

Yes. Programming by coincidence is the norm. Seems to work, must be right.

Dangeranger• 9 years ago

My suspicion is that this behavior is left over from early implementations of Ruby where threading wasn’t as common and so wasn’t as large of a concern.

ProAm• 9 years ago

Welcome to the stackoverflow era of programming, where technical specs don't matter and frameworks are life.

mrguyorama• 9 years ago

This has been around since well before stack overflow

Here are a plethora of examples from the Windows 95 days: http://ptgmedia.pearsoncmg.com/images/9780321440303/samplech...

viraptor• 9 years ago

It's silly to point at this documentation as if this is something that people will read in this case. People, or specifically web developers are not calling fork(). What they're doing is using supported language, a webapp server, using a database access library, and enabling the preloading option. As seen in the bug report in Ruby, this comes down to coordination between those projects. Each one is correct on their own. Putting them together as a web developer shouldn't involve analysing syscalls you're going to make either.

booleanbetrayal• 9 years ago

for other fun low-level High Sierra issues, see the PostgreSQL msync() thread: https://www.postgresql.org/message-id/flat/13746.1506974083%...

salarycommenter• 9 years ago

Interesting. Mapping 64k at a time seems a bit excessive. If I call mmap it's to map everything I'm going to need, but maybe I am missing context.

snuxoll2• 9 years ago

PostgreSQL still supports 32-bit platforms, heap and index files can get large enough that it's not feasible to mmap them into a 32-bit process. As far as the specific number of 64k though, I don't know why they choose that, writing 8 pages to disk at a time (especially sequentially) doesn't really take advantage of modern hardware well.

bpicolo2• 9 years ago

I've hit similar issues with uwsgi in recent memory (though pre high-sierra), where an OS upgrade caused it to start segfaulting when using the `requests` lib inside CoreFoundation somewhere (though of course entirely unrelated to the new forking changes).

Maybe this? Though the resolution was to disable uwsgi proxying globally... https://stackoverflow.com/questions/35650520/uwsgi-segmentat...

olivierlacan• 9 years ago

The discussion on the Ruby core team issue tracker is also very informative: https://bugs.ruby-lang.org/issues/14009

dep_b• 9 years ago

This cryptic error

Well....I don't think I've seen many programmer-to-programmer errors that are less cryptic than the one described in the article. It's actually quite amazing how much explanation you sometimes get from Cocoa!

LeoNatan25• 9 years ago

I think it's mostly unclear to non-Apple developers.

semanticfact• 9 years ago

It's only cryptic if you don't read it, right?!?

Why Ruby App Servers Break on MacOS High Sierra

Related Stories