Mach's designers simply assumed that systems would be rebooted often enough

http://clozure.com/pipermail/openmcl-devel/2011-February/012567.html

by zachbeane • 15 years ago

127 70 15 years ago

jarin3• 15 years ago

I wonder if that's been fixed in the OS X/Darwin version of Mach, seems like the only time I ever reboot my MacBook is for system updates.

zackb• 15 years ago

I wonder if there's an update for my MkLinux box!

http://mklinux.org/

tianyicui• 15 years ago

Wow. There are still MkLinux users? What's the motivation?

mikeryan• 15 years ago

This: the only time I ever reboot my MacBook is for system updates

Seems to indicate that it has ;-)

eibrahim• 15 years ago

really man? maybe you are not using your macbook long enough or you are simply browsing the internet with it?

I reboot my macbook pro (running leopard) as often as I reboot my windows 7 PC which is not much. Except I use the Win7 machine 90% of the time and for much more "intensive" stuff.

jarin3• 15 years ago

I use it probably 14+ hours a day, and it's variously running MySQL, Redis, MongoDB, Photoshop, 600 million Chrome tabs, Xcode and Starcraft 2 whenever possible :)

I did experience memory leaks with pretty much every Bittorrent client, but I haven't torrented any uh, Linux distros in forever.

blago• 15 years ago

As a fellow mac user: If you are rebooting your Mac for reasons other then system upgrade, something is wrong with your machine. I say this as someone who compiles from source and runs server software on my mac almost every day. mogodb, apache, nodejs, mysql, tomcat is just the tip of the iceberg in the server department.

CountSessine• 15 years ago

The original poster in that thread was running 10.5.8. No one in the thread posted saying that they couldn't repro the port leak. The thread is quite recent, the latest post being from earlier today.

I think we have to assume that the port leak is still there in the most recent version (10.6.6).

I do hope that the poster had misunderstood Avie Tevanian in his conversations with him. Tevanian was one of Apple's chief architects and was one of the big wheels responsible for NextStep's transformation to OS X. It would be kind of outrageous for an OS architect to believe that it's at all acceptable to allow user-mode processes to cause kernel resource leaks.

jrockway• 15 years ago

It was an engineering trade-off like any other. Would you be willing to reboot your computer once a month if it meant everything would be 1000x faster? Of course. So were the Mach designers, apparently.

jwatzman• 15 years ago

While the discussion in that thread may or may not apply to OS X, be very careful when saying "X is true about Mach therefore X is probably true about Darwin". A friend of mine put it very well:

OS X was Mach once in the same way that orcs were elves once.

eschaton• 15 years ago

You're misinterpreting the post; Mikel wasn't talking about the same leak the original thread was. He was just talking about how he fixed a leak many years ago.

Using Mach ports correctly without leaking is hard, similarly to how using C's malloc/free without leaking is hard. The former no more implies a current flaw in Mach than the latter implies a current flaw in C stdlib.

klodolph• 15 years ago

Kernels should not have resource leaks. A web server that leaks memory implies nothing wrong with the clients, a kernel that leaks memory implies nothing wrong with the user space programs.

It just means that there are cases that the kernel developers didn't think of, just like the people who made Apache didn't think of the Slowloris attack. Heck, a few months ago someone exhausted Linux kernel fds in just by shuffling socket pairs around (just 40 lines of C with three syscalls).

These are problems, ranging from remote DOS attacks, local DOS attacks, to accidental DOS attacks.

mobilemonkey4• 15 years ago

I'm not going to pretend I know what Mach is, but around here, (big company that you're familiar with), rebooting/bouncing the servers is pretty much how issues are dealt with. "Response times outside of SLA: bounced the server." "Database connections timing out: bounced the server." "Users experiencing high load times for pages: restarted JVMs. Then bounced the servers."

Root cause seems to be "server up too long."

derleth2• 15 years ago

Mach is a microkernel. That means it's a way to design an OS such that most of the things application programs rely on are outside the kernel, provided by servers that can be restarted and replaced without rebooting the whole system. For example, the majority of the filesystem is provided by a filesystem server, most of the networking code is in a networking server, and so on.

You begin to see why a microkernel that needs to be bounced more often than, say, Linux, which is not a microkernel, begins to lose some of the appeal of having a microkernel in the first place.

(Aside: Mac OS X is based on Mach, but practically all of the stuff applications see is provided by a single server which (to the best of my knowledge) can't be restarted without taking down the whole system. It's a hybrid design.)

Also, just to confirm a stereotype, do you use Windows for your servers?

stcredzero4• 15 years ago

Also, just to confirm a stereotype, do you use Windows for your servers?

Related tangent: There was a time when the the US military's COTS (Civilian Off The Shelf) initiative resulted in AEGIS missile cruisers running largely on machines running Windows NT. Yes, the ships responsible for screening carrier task forces against attack -- had to be rebooted on a weekly schedule.

To be fair, the unmolested NT kernel is rock solid. I know people used to run RAID arrays with it, but for the desktop, Microsoft allowed the crossing of certain architectural lines.

reeses• 15 years ago

Mach isn't a microkernel. For example, NeXTSTEP was based on Mach 2.5, which is a monolithic kernel. Mach 3.0 was the only microkernel version, and to my knowledge, no one has adopted it with any success.

mobilemonkey4• 15 years ago

Hmm, nope. Solaris of some flavor I think.

mobilemonkey4• 15 years ago

Thanks for the great explanation, too, btw.

jwhitlark• 15 years ago

At my last sysadmin job, we had a policy of bouncing a server if it's uptime exceeded one year, not to clean up any resources, but to make sure the config in the files matched the config that was running. Of course, not only were we running Linux (gentoo, in fact), but a lot of our stuff was DJB services, including his service manager.

I'd argue that a 5 why's analysis of "server up too long", leads to "server wasn't written well." YMMV

mpk2• 15 years ago

Now this is an excellent reason and timeframe for doing periodic reboots.

It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk.

If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point.

Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.

jacques_chester• 15 years ago

Mainframe shops regularly reboot their systems on a rolling basis.

A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on.

The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls.

Meanwhile, the other LPARs will keep running as if nothing is going on.

gwern• 15 years ago

Richard P. Gabriel makes that point somewhere - that one of the downsides of an 'organic' system like a Lisp or Smalltalk image where you can rewrite or do anything dynamically is that you tend to do just that, and can wind up in a situation where you are no longer able to stop using the image because it has too many valuable changes which are too difficult to reverse-engineer.

JoeAltmaier2• 15 years ago

Yahoo reboots IM servers daily. Easier than bothering to find all the horrible leaks.

Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.

Twirrim• 15 years ago

See that's not actually solving the problem. Thats more like pretending that the problem doesn't actually exist. Instead of investigate and fix permanently so that customers don't experience problems they'd rather just keep letting the problems fester.

doyoulikeworms• 15 years ago

Not everyone is "pretending that the problem doesn't actually exist." There are limited resources available to any and every organization. Many times, a proper fix can cost more than a band aid, sometimes even over time.

Here's a hypothetical, but not ridiculous, scenario. Which is a better way to spend resources? Track down and stamp out an extremely difficult resource leakage bug? Or simply bounce the server? The costs associated with the former may be HUGE. The costs associated with the latter may be small in comparison (cost of potential down time, minimal labor cost of bouncing the server(s), and costs associated with user reliability).

I wouldn't always chalk it up to whatever it is that your comment implies (laziness? willful ignorance?).

pinko• 15 years ago

Relevant: http://www.usenix.org/event/osdi04/tech/full_papers/candea/c...

Microreboot – A Technique for Cheap Recovery

"A significant fraction of software failures in large-scale Internet systems are cured by rebooting, even when the exact failure causes are unknown. However, rebooting can be expensive, causing nontrivial service disruption or downtime even when clusters and failover are employed. In this work we use separation of process recovery from data recovery to enable microrebooting – a fine-grain technique for surgically recovering faulty application components, without disturbing the rest of the application."

bm98• 15 years ago

My favorite Linux bug (since fixed):

https://bugzilla.redhat.com/show_bug.cgi?id=97373 (System UPTIME reported incorrectly):

"Steps to Reproduce: 1. Boot Linux system; 2. Go away for 497 days; 3. check uptime"

stcredzero4• 15 years ago

Take it off the queue and go on a 16 month sabbatical to somewhere really beautiful. Come back, check uptime, report to your boss that you've recreated it and you'll start digging in the code.

http://xkcd.com/583/

aidenn0• 15 years ago

"Hello IT, have you tried turning it off and on again?"

fleitz• 15 years ago

There are some great reasons due to memory fragmentation and other issues to reboot your systems every two or three days anyway. Plus, if you pride yourself on having failover, DR systems, etc, rebooting lets you know that your failover systems are working.

Mach's designers simply assumed that systems would be rebooted often enough

Related Stories