macOS Internals

https://gist.github.com/kconner/cff08fe3e0bb857ea33b47d965b3e19f

by JoelMcCracken • 3 years ago

773 141 3 years ago

gist.github.com

nikodunk• 3 years ago

It's impressive how so many years after he left, the legendary articles by John Siracusa are still having an impact.

cocacola12• 3 years ago

If you don't, would recommend listening to ATP, which he's a cohost on. Still a joy to listen to.

karlshea2• 3 years ago

Neutral and then ATP is what got me into listening to podcasts. Still a great one to this day!

Their interview of LLVM co-founder/Swift architect Chris Lattner is def a good listen for historical background: https://atp.fm/205-chris-lattner-interview-transcript

sbuk2• 3 years ago

Used to listen to it. Stopped because I cannot stand Marco Arment.

microtonal• 3 years ago

For newer macOS changes, Howard Oakley's blog is really awesome:

https://eclecticlight.co/category/macs/

saagarjha7• 3 years ago

With a somewhat lower bar for accuracy, unfortunately.

internetter2• 3 years ago

I've loved reading blogs and microblogs of Asahi Linux developers, like https://social.treehouse.systems/@marcan, https://asahilinux.org/blog/, and https://rosenzweig.io/

They all have intimate insights into macos internals from reverse engineering it.

latexr• 3 years ago

As does his old podcast, Hypercritical. It finished almost a decade ago, yet even the early episodes continue to have interesting insights.

https://hypercritical.fireside.fm/

If you’re interested but don’t know where to start, “The Bridges of Siracusa County” is a classic.

https://hypercritical.fireside.fm/15

krackers• 3 years ago

If you want to go even deeper there's the book series by Jonathan Levin

DavidPiper• 3 years ago

http://newosxbook.com/home.html for the lazy :)

They are an updated and much expanded version of "Mac OS X and iOS Internals: To the Apple′s Core" by the same author.

Which in turn was an updated version of "MAC OS X Internals: A Systems Approach" by Amit Singh.

magoghm• 3 years ago

Does anybody know why they are so hard to buy from outside the US?

wkat42423• 3 years ago

Oh, interesting. I still have Amit's book here on my shelf.

Though I see this author has also given up work on this project. And they are extremely expensive in Europe. At least Amit Singh's I could just grab on local Amazon. Levin's aren't available here.

vaughan3• 3 years ago

History is such a great way to learn anything.

It’s interesting - like a mystery revealed, but also provides the why - why things are the way they are.

I was looking into accounting the other day, and you can go back to the first document that mentioned credit and debit and introduced modern accounting. There are so many topics like this that seem confusing and arbitrary but it all began extremely simply and pragmatically. Music theory is another example of a fun topic to study historically. And programming languages and operating systems.

It’s important to use two learning paths though: historical and modern scientific. Historical usually explains terminology and legacy aspects of such topics, but then topics can also be explained by using modern technology to experiment and observe.

Think about unix - it was primarily designed for very low power low memory environments. And databases too. You can learn a lot poking at an existing system, but when you watch the early content on it, you see the why.

robertlagrant2• 3 years ago

Agreed. This is why often reading a Wikipedia page is the worst way to learn something. The best way is to get a teacher to tell you the relevant history leading up to the thing, and it makes much more sense.

raarts2• 3 years ago

CS teacher here. This is my preferred way to teach stuff. Way easier to remember and understand knowledge if you know its origin story.

layer8• 3 years ago

Though many Wikipedia pages have a History section doing exactly that. (More should have one.)

aebtebeten• 3 years ago

The present is the past rolled up for action and the past is the present unrolled for understanding. — A&WJD

JadeNB• 3 years ago

A&WJD

Who's that?

victor1062• 3 years ago

This is so true.

Most of math starts to make sense once you know the history behind the concepts.

bombcar3• 3 years ago

I was able to do calculus when I first learned it, but it didn't really click until I studied Newton's Principia, and was like, oooooooooh I see what he did.

Similar with Euclid's Elements vs the modern "Cartesian" view of geometry.

vernal• 3 years ago

What was that first accounting document?

vaughan3• 3 years ago

For double-entry accounting, Luca Pacioli's Summa. The history of accounting is a bit more complex and unknown.

https://en.wikipedia.org/wiki/Summa_de_arithmetica

whatever1• 3 years ago

So the Window server is a memory shitshow for the past 20 years?

Today I learned.

astrange3• 3 years ago

Pixels use a lot of memory. Especially since they're effectively not compressible.

pimlottc2• 3 years ago

It’s a bit confusing to say “read these in chronological order” since it’s not immediately apparent that the document is ordered chronologically. It would be clearer just to say “read these in order”.

easeout4• 3 years ago

Thanks for the note; I may tweak that. Does the Highlights section's intro, "These chronologically-ordered highlights…" help?

—gist author

pimlottc2• 3 years ago

Yeah I’d find that clearer.

ilkkal2• 3 years ago

Fascinating stuff, I've never really thought about what problem e.g. frameworks are solving. It's kind of too bad that in my day job my head's in the clouds and usually somewhat removed from most of these system level details.

inb4 it's just someone else's computer -- trust me I know

saagarjha7• 3 years ago

They’re just shared libraries.

ilkkal2• 3 years ago

Thanks for the blinding insight.

ElFitz4• 3 years ago

It’s funny how I used to think that operating systems, databases, and cloud systems, as these kind of arcane things that were somehow apart from the rest of software.

I could not begin to grasp how they could deal with running programs, access control, or any of these things that once seemed so foreign and mysterious to me.

And realising that it’s actually just software, that there’s no black magic to it, just lower-level APIs, less pre-built batteries-included stuff to rely on, and probably a whole lot of more complexity, that was… both mind blowing, exciting, and kind of a let down.

And it has enabled me to have more realistic expectations from these tools.

It however doesn’t take away any of the brilliance and effort that went into the these things, or lessen the credit due to those behind them. I can only begin to imagine the complexity involved in a full-fledged modern OS.

dlivingston• 3 years ago

It's turtles all the way down. Layers of abstraction stacked on layers of abstraction.

selestify• 3 years ago

Until you get to the physics of transistors and logic gates :)

Though I suppose in a sense, even subatomic particles are abstractions over quarks. It’s just that those are no longer abstractions we control.

fatneckbeard2• 3 years ago

why i like arduino.

why i kind of like old IBM PC environment. x86 + a few bios calls, dont even need DOS after you load game.exe

apples_oranges• 3 years ago

Yeah, it's all a bunch of programs doing their thing. One draws a GUI, another talks to the network adapter, and so forth.. And when you click a launcher icon, a loader program is started that (more or less) copies a program into RAM and when all is set up, it tells the OS (and the hardware) to run that, too.

amelius• 3 years ago

Yes, a modern OS is a tangled mess of microservices.

gl-prod• 3 years ago

It hits you in the face, how do you eat an elephant? A bite at a time. One API, one abstraction at a time.

maegul• 3 years ago

For me this moment was when computers were no longer “magical” to me and that I found comments to this effect (even from those that know more than I and were more senior) odd and kinda culturally cultish.

astonex• 3 years ago

If you wanted to learn, I really recommend Operating Systems: Three Easy Pieces (OSTEP). I thought it was excellent and pretty easy to follow. https://pages.cs.wisc.edu/~remzi/OSTEP/

wkjagt• 3 years ago

There are many things in OSTEP that I found really eye opening. One thing that stuck with me was that at some point, when discussing virtualization, it referred to an OS as a virtual machine, and that really changed the way I look at operating systems now.

That is, the OS takes a physical resource (such as the processor, or memory, or a disk) and transforms it into a more general, powerful, and easy-to-use virtual form of itself. Thus, we sometimes refer to the operating system as a virtual machine.

ElFitz4• 3 years ago

Thanks!

CPLX• 3 years ago

It still kind of blows my mind that basically all this reduces to assembly and then machine language.

I’m of the age where my first computing experiences were on green screens, PET computers, networked VAX machines at the colleges my parents taught at and so on.

I was aware of what assembly was back then, and the idea that the game or basic business program I was using was just a sequence of MOVE ADD eax and so on. I just couldn’t quite get how you could be smart enough to use that to something useful.

And everything we see today is basically just a bunch of moves and adds and pokes a layer or three below the surface.

ZephyrBlu• 3 years ago

The thing that made me realize this was compilers!

"Wait, code is just text?"

"Always has been"

I still have to remind myself of this sometimes when I think "woah, how does this work?" and then I try to step through how it might be built.

eric-hu• 3 years ago

I don’t know that code was always text, but I’m thankful to have been born in the days after programming assembly on punch cards was not the only way.

dhosek• 3 years ago

I think that’s one of the advantages of coming into computing in the early 80s. Computers like the Apple ][ were completely understandable by a single person (and it’s still my mental model for how a computer works 40 years later).

ris58h• 3 years ago

Does anybody know how to force sandboxing for a 3rd party app?

saagarjha7• 3 years ago

sandbox-exec?

ideologysec2• 3 years ago

You can write your own sandbox profile if you're so inclined... but not easily, and you can't force it into a container in the Library folder, for example.

Kwpolska• 3 years ago

There are also some amusing writings not included here about modern macOS changing things in a way Siracusa dislikes. For example, mandatory file extensions (written in 2001, when the Internet was a thing and file extensions were required for all file exchanges) [0] or the Spatial Finder saga [1] about missing an antiquated way to make a mess on your screen and pretend to be managing folders.

[0] https://arstechnica.com/gadgets/2001/10/macosx-10-1/12/

[1] https://arstechnica.com/gadgets/2003/04/finder/#prev-article...

easeout4• 3 years ago

These are great :) I'd like to do without file extensions again, but internet URL norms have now made that decision for all operating systems. But I'm still kicking Finder windows back into spatial mode three times a day.

KerrAvon2• 3 years ago

He’s right about the spatial Finder. The Mac OS 9 Finder is still a better user experience. The OS X Finder only become marginally usable when they added the sidebar so that you had some starting reference points; it was much improved with Spotlight because you could then find things without spatial reference points.

But I still maintain that adding the NeXT browser view to the OS 9 Finder would be the perfect experience. The OS X Finder went down this awful .DS_Store-strewn half-assed path and it’s still just the weirdest set of choices imaginable.

bombcar3• 3 years ago

I think modern "kids" (who now are in their early 40s perhaps!) don't quite get how useful the spacial finder was to non-technically aligned people. "I left it right here" is a huge piece of memory that our brains have developed over millennia, and the spacial finder played right into that.

The "it's in the directory listing in terminal" or "spotlight will find it" or "what is a file" don't really compare, even if they can be made usable.

alin234• 3 years ago

    The window server is "lightweight" in that it does no screen drawing itself. It is, to use Apple's term, "drawing model agnostic."

WindowServer is such a behemoth nowadays, it took me ages to reverse engineer how to make rcmd (https://lowtechguys.com/rcmd) control Stage Manager.

It’s interesting to see it described as lightweight. I guess every lightweight solution becomes heavy and complex with enough usage.

shinycode• 3 years ago

Rcmd is the best app for window management productivity I found in years. Thank you so much !!

alin234• 3 years ago

Thank you! Happy to hear it has become so useful ^_^

victor1062• 3 years ago

Love rcmd.

Once I started using it I was wondering how I worked without it

catoc2• 3 years ago

I love your website's privacy page. Also lightweight.

["rcmd does not collect any personal information."]

That's the complete page. Great policy!

alin234• 3 years ago

Thank you! Yes, I believe privacy policies should be written for end users, not for lawyers.

I tried to do the same with my other more complex app as well: https://lunar.fyi/privacy

ronyfadel• 3 years ago

How does reversing WindowServer allow you to make rcmd control Stage Manager?

Figuring out which CG calls WindowServer is making?

Wouldn't these calls fail unless you have special entitlements?

alin234• 3 years ago

This is about a work in progress for a version of rcmd that I intend to distribute outside of App Store. As you guessed, calling private functions is not allowed by the App Store review guidelines.

I want to add hotkeys for creating Stages with the needed windows, because as it is right now, mousing around to move windows is too slow for me.

I'm looking inside SkyLight.framework and WindowManagement.framework to find the right sequence of SLS* or CGS* functions that does what I need.

neurostimulant• 3 years ago

Man, I miss those 20+ pages long Ars Technica reviews. They don't seem to publish these kind of stuff anymore.

kreddor• 3 years ago

The macOS Ventura review doesn't count?

galad87• 3 years ago

Not really. It lacks the insight and the attention to small changes and under the hood improvements of the Siracusa's reviews, in my opinion.

TazeTSchnitzel• 3 years ago

If this kind of thing interests you: late-2000's/early-2010's Apple had some fantastic documentation, but it's all hidden in the “documentation archive” (https://developer.apple.com/library/archive/navigation/).

pjmlp3• 3 years ago

Only available to us old timers really, as newer devs on the ecosystem don't even know what to search for.

I don't get it how they messed this up.

easeout4• 3 years ago

My hunch is SEO combined with placing less responsibility on the programmer.

The new docs have a page per code symbol, and much more focused single-page topic articles, rather than 20-page programming guides. Docs seem to be structured to answer a question right when you have a problem, but then they don't offer a good way to study in depth before you begin.

The other day a younger teammate asked a group about some API documentation that said the results are undefined if you call this wrong, and had we ever heard of such a thing. And yes, of course, using framework code in unintended ways is always a garbage in, garbage out affair. This made me realize we have come a very long way in eradicating boilerplate and ceremony in code.

There used to be so many things you just had to use the way the docs explained, or else. Doing the boilerplate correctly was a primary focus of code review. Today we have ARC, we don't call alloc separately from init, we never handle KVO without the compiler's help, and we have no IBOutlets to forget to bind. To begin writing for iPhone 15 years ago, I had to study for a week or so with the programming guides, one for the language and another for the platform. Today you can do a one hour tutorial.

So, if you can stumble your way through building an app by typing the period key and seeing what methods are available, and the whole platform SDK exhibits Clarity at the Point of Use, would a newcomer even use the programming guides if they weren't down in the archive?

pornel• 3 years ago

My best guess is that as macOS aged and became harder to keep backwards-compatible, their dev communication changed from "you must not rely on implementation details" to "you must not know about implementation details".

2809• 3 years ago

Damn, I was hoping for a MacOS version of SysInternals.

greggsy• 3 years ago

Me too. I think the closest is Xcode Additional Tools.

ideologysec2• 3 years ago

Try Red Canary Mac Monitor, it does a lot (it's at least a decent equivalent for ProcMon)

Varloom• 3 years ago

That’s super interesting.

ChrisMarshallNY2• 3 years ago

That's great!

Now, bonus time: Let's do something like that for SwiftUI.

Razengan• 3 years ago

https://swiftui-lab.com and https://swiftwithmajid.com have been a great help.

glhaynes• 3 years ago

I love the Swift Talk video series https://talk.objc.io. They've done a few reimplementations of SwiftUI components that helped me feel more like I understood those things.

ChrisMarshallNY2• 3 years ago

I subscribe to them.

Well worth it.

MichaelZuo• 3 years ago

Stopping the reviews at Yosemite was a smart choice, every later version had noticeably more bugs then the previous.

mike_hearn2• 3 years ago

macOS probably doesn't get enough love from an OS design perspective. Over the years I've had reasons to work pretty closely with the guts of all of Linux, macOS and Windows and macOS is probably my favourite, design wise, although for some tasks you can't beat the flexibility and feature set of Linux. A few highlights that are lesser known:

- XPC/Mach is a pretty reasonable IPC system that avoids the huge complexity of DCOM, whilst being more widely adopted throughout the platform than DBUS.

- Bundles are frustratingly ad-hoc in terms of detection and layout but the basic concept does work, and is better than the (rough, not really) equivalents on Windows and Linux.

- Launch Services does a pretty good job these days of letting apps be purely declarative whilst still keeping track of all the integration points things (in the past it's been a bit rough).

- APFS is a very modern FS and the migration of the Apple ecosystem to it was so well executed it was barely even noticed. I can't think of any other platform that has managed to do a migration from one FS to another so smoothly.

- Ditto for CPU architecture migrations. Apple's skill at these has become rather legendary now. I remember a time when the computer industry assumed such transitions were simply impossible, Apple did it twice!

- The code signing infrastructure is rather complex but well thought out and flexible, even though Apple don't use all that flexibility in practice. They've chipped away at it over the years and by now it delivers concrete and real benefits to end users. In particular it enables apps to be kept somewhat separated even when they aren't sandboxed e.g. on Ventura apps can't tamper with each other unless you specifically give them that permission, and this works without any notion of admin/root escalation or installation, and that's true even if apps aren't opted in to the app sandbox.

- The way Apple live the UNIX spirit by having lots of little CLI tools along with man pages for those tools etc is pretty nice.

- The sheer quantity of APIs and frameworks that solve modern high level problems is well ahead of any other desktop platform by miles (e.g. the ML frameworks, data syncing).

- The way Apple have managed to slowly migrate code out of the kernel is well executed IMO. Likewise for SIP and their whole security/immutable OS posture, something the Linux world is still just experimenting with and Windows isn't even trying.

- Their integration API for cloud storage is solidly designed, albeit it's a pity they don't also offer a lower level FUSE-like API for experimental use cases.

There are problems too of course. It's very weak that the OS has no way to keep apps up to date except via the store, which then introduces tons of other problems. Sparkle does a great job of patching up this gap, but it should really be a core OS service in this day and age. And the lack of any high level GCd/managed language for their platform continues to be a major weakness. You can have smooth UI and 60fps whilst still offering GC as Android has now proven, and of course there are large classes of apps where developer productivity matters more anyway. Microsoft always recognized that, even the Linux community understood this hence all the bindings into Python.

maccard3• 3 years ago

I agree with most of what you've said here, except:

- Code signing. It's incredibly frustrating as a developer to try and codesign an application to pass gatekeeper, requiring multiple steps with very little information as to what's _actually_ going on or going wrong in the middle of it, including uploading your entire bundle to Apple and waiting an arbitrary amount of time for them to notarize it.

It's very weak that the OS has no way to keep apps up to date except via the store, which then introduces tons of other problems. Sparkle does a great job of patching up this gap, but it should really be a core OS service in this day and age.

App Store _is_ the solution and a core OS service. You may not like the limitations that implies, but it is the OS level solution for keeping apps up to date.

And the lack of any high level GCd/managed language for their platform continues to be a major weakness.

I think swift fits this gap nicely, no? It's not technically GC'ed, but it is refcounted, which for end user case achieves the goal of "ignore memory mangement for small tasks" I like this blog post [0] which shows a desktop app in 11 lines of code.

[0[ https://www.amimetic.co.uk/blog/swiftui-for-small-internal-d...

pjmlp3• 3 years ago

Code signing is coming for Win32 as well, as announced at BlueHat IL 2023.

mike_hearn2• 3 years ago

A core OS service in my books is an API that can be used by any app, can be adopted in a backwards compatible way and which is used by the in-house apps too. The App Store isn't an API or service, it can't be used by every app and the built-in macOS apps aren't necessarily store apps. Yes, it's Apple's strategy but I maintain that from a tech perspective it's a very weak one.

Cryptography DX is always terrible on every platform, I don't know why, it seems to be an unwritten rule that crypto tools must have as many sharp edges as possible. But as far as these things go, Apple's infrastructure actually is effective and flexible. Notarization got a lot faster lately and is certainly way better for both devs and end users than client side virus scanners.

Swift is a good upgrade, but not exactly a high level managed language. Most of those languages have very fast or non-existent compile times as well as GC, they tend to be simple-ish, perhaps they have optional types. The Swift DX is quite different to a C# or a Java for example.

kitsunesoba• 3 years ago

Automatic reference counting is very well streamlined into Swift. Even in more substantial projects in most code I'm not putting much active thought into memory management at all with it. The main area where some extra attention is required is in closures, and there in most cases the main thing is to not strongly reference self which is easy to avoid.

As for the compile times mentioned by the sibling comment, they're pretty tiny for small utility things, especially incremental builds. On more recent hardware (M1, Ryzen 5000, etc) I find they're reasonable even for moderately sized projects… on M1 Pro a complex iOS app with somewhere in the ballpark of 30 different screens can be clean-built in under a minute with incremental builds usually under 3s which seems plenty reasonable to me.

sbuk2• 3 years ago

Apple did it twice!

Thrice, in fact! 68000 -> PowerPC, PowerPC -> Intel, Intel -> ARM/Apple Silicon

pjmlp3• 3 years ago

XPC is great.

I don't really understand WinDev in regards to COM.

In abstract, it is a very good OOP ABI, and cross language interop.

In practice, one would expect that given how much they have doubled down on COM for the last 25 years, they would have come up with a development experience that isn't the worse from all IPC systems invent so far in regards to development experience.

.NET requires actually knowing COM at C and C++ level, and as of .NET Core, also write IDL files manually, as they removed type library support from .NET toolchain.

On C++ side, there have been endless reboots, none of them really productive, with exception of C++/CX, killed due politics.

WinDev really loves their bubble.

Regarding GC on OS, many others have tried as well. The problem is having a company willing to throw enough money at the problem and steer developers to adopt the platform no matter what.

Which is actually quite surprising for Android, given Google's track record in killing products before they can establish themselves.

All in all, there is still a lot of NeXTSTEP in macOS.

Maursault3• 3 years ago

It's very weak that the OS has no way to keep apps up to date except via the store, which then introduces tons of other problems.

/usr/bin/softwareupdate [1]

[1] https://ss64.com/osx/softwareupdate.html

saagarjha7• 3 years ago

This updates the OS and related content.

wkat42423• 3 years ago

Ditto for CPU architecture migrations. Apple's skill at these has become rather legendary now. I remember a time when the computer industry assumed such transitions were simply impossible, Apple did it twice!

Three times actually. Motorola -> PowerPC. PowerPC -> Intel. Intel -> Apple Silicon.

Edit: Oops this was already mentioned elsewhere in this long thread sorry.

wkat42423• 3 years ago

I still have Amit Singh's huge book on my shelf here. Completely outdated now, of course. But it was always quite interesting.

It's nice to see someone else has taken on the task of bringing that kind of insight into the modern age. Because Apple certainly won't.

macOS Internals

Related Stories