Reverse Engineering the M1 [pdf]

https://i.blackhat.com/USA21/Wednesday-Handouts/us-21-Reverse-Engineering-The-M1.pdf

by todsacerdoti • 5 years ago

224 50 5 years ago

i.blackhat.com

jchw3• 5 years ago

M1 Linux does not have a cool logo or name

Wonder if this is a subtle jab at Asahi Linux. Hopefully not, as I was thoroughly unamused by Corellium’s previous antics regarding Asahi Linux. It would be a lot nicer to see at least polite relations in the future, if not collaboration…

monocasa3• 5 years ago

Isn't the feeling mutual? I was under the impression that Asahi didn't really want Corellium's contributions because they considered Corellium's looks at proprietary Apple code to be an existential threat to the project's need for clean room RE.

rowanG0772• 5 years ago

I believe this is false. The Corellium guys are (much?) further along technically. But they don't care to put their code in a format that upstream will ever accept. I believe this was the main reason collaboration went nowhere. Corelliums forked kernel will always be little more then a marketing toy because of this.

jchw3• 5 years ago

In any case, it never had to turn petty, even if they really did feel like RE knowledge from Sandcastle was being used. Even if code wasn’t shared, knowledge could have been. I came out with the feeling that they had bigger concerns about optics above all, and they couldn’t use the same approaches that might’ve worked when they were the David and not the Goliath.

I don’t want to be the asshole dredging up drama needlessly, but the line just feels snide after all that happened. Like, if you wanted a slick logo, name and marketing page, you all could just make one; you had absolutely no trouble doing so for Project Sandcastle. I don’t see why this statement needs to be made here and now, as nobody would’ve batted an eye if they hadn’t said anything.

coldtea2• 5 years ago

Well, they objectively were.

marcan_4212• 5 years ago

Sometimes we look at Apple code too, within reason and where strictly necessary.

The problem is that there are safe ways of doing this and terrible ways of doing this. When I look at vendor code, my goal is to understand what I need to do to the hardware to get it to behave how I want. I don't care how Apple's code works; I only want to use it to help answer questions about the hardware, and then build my own driver with my own approach.

However, I've seen other people (including a Corellium co-founder, in a past project that predates the company) "reverse engineer" code by literally translating it back to compilable C code, with the same logic, functions, and everything. And that is where you run into copyright problems.

And the issue is that unless you re-do all the work from scratch, it's not easy to tell from the output whether this has happened or not. I've been bitten by this in the past. And so, as the Asahi Linux project founder, I can't afford to accept contributions from any random person that are not clean-room, without having some level of trust. I have a few long-time friends in the project who I know won't screw this up, so I'm not worried about them taking a peek at Apple's drivers in Ghidra. But if it's someone I don't know, they need to build trust and we need to have a conversation about this first.

Corellium have so far refused to have any conversations with us. They never replied to any emails they were CCed on, nor have they joined our project IRC channels.

So, without being able to build trust that their approach is kosher, I can't just take their code and use it.

Practically speaking, Corellium certainly have internal documentation on the hardware that they could share (much like we're dumping info on our wiki) that would be immensely useful to us, but I doubt they'd ever share that, as it would potentially help someone compete with their product. Asahi Linux is, in this way, fundamentally at odds with Corellium: we're trying to reverse engineer the M1 and share that knowledge, while they did so to build a proprietary product and keeping that reverse engineered knowledge a trade secret is important to them.

marcan_4212• 5 years ago

I'm pretty sure it is.

They also apparently still haven't figured out that they have a bad setting in their tunables (that macOS does not do) which disables the SError reports when I/O writes are using the wrong type. They definitely don't get "silently" ignored if you don't turn off the error reports :-)

But this presentation finally answers the question of why Corellium did this. It wasn't a fun side project or a way to contribute to the community. Their Linux port is a validation platform for their emulation/VM product. Now it makes perfect sense. Releasing it and claiming they were going to upstream it was a PR stunt; they haven't updated their repo since February and didn't reply to any of the upstreaming mailing list threads they were CCed on. I'm pretty sure they don't have any actual interest in collaborating with anyone or upstreaming anything. To them, this is a platform validation tool for their commercial product, and it only has to work once, not be maintainable or upstreamable. They have no business reason to spend time on that.

Apple have done the same, by the way. They have internal Linux ports to their SoCs that they use for silicon validation.

I should probably give a talk on m1n1 and the hypervisor I built for M1 reverse engineering...

xuki• 5 years ago

I should probably give a talk on m1n1 and the hypervisor I built for M1 reverse engineering...

Please do. Pretty please.

asddubs2• 5 years ago

haha, I have to admit I did think that the asahi linux logo has way too many different colors

kzrdude• 5 years ago

Is there a video recording to go with this?

Ristovski• 5 years ago

Afaik, BlackHat talks/presentations go public after around half a year.

Icathian• 5 years ago

First thing I was gonna ask. I don't think I'm good enough yet to get much value out of the slides without some voiceover.

mdaniel• 5 years ago

Wow, I would buy every book written by this person, as the amount of knowledge and experience required to pull off a stunt like that feels overwhelming

Does anyone know if https://dougallj.github.io/applecpu/firestorm.html is the "Dougall Johnson's work if you can find it" reference, or is there some dark web version?

igor_sk• 5 years ago

It's https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75...

ksec• 5 years ago

A14/M1 use a non-standard NVMe queue format
USB4... but Apple made it their own, as usual

I am wondering about these two points. Why ?

userbinator• 5 years ago

Think Different™

Apple has a really long history of doing stuff like that.

https://news.ycombinator.com/item?id=12924051

shawnz• 5 years ago

I remember when Microsoft were the archetypal "not invented here"-ers.

mhh__• 5 years ago

There's probably an interesting groupthink/psychology to Apple, but I think in this case the proprietary-ness of parts of M1 is them focusing on getting it out of the door: Apple have a lot of experience making ARM mobile chips but this was their first desktop part.

I say this based on nothing but gossip, of course.

jeffbee• 5 years ago

If you're going to have your own platform where you control all the parts and none of them are replaceable, why would you religiously stick to the standards when you don't need to? Maybe they saved a slight amount of power consumption by not supporting 64K queues, like the nvme standard supports. A standard will try to be all things to all people, but it will never be the best thing for any given application. There is always room to specialize.

8note• 5 years ago

For USB, apple makes proprietary connectors so that competing cable makers have to pay them to license it (along with other licensing terms)

SilverRed• 5 years ago

This is not true of any M1 device.

marcan_4212• 5 years ago

The non-standard NVMe queue format is because they support offloading encryption/decryption to the NVMe controller, so they had to increase the queue entry size to be able to fit an encryption key in there.

svenpeter3• 5 years ago

The "queue" format itself is incredibly similar to the normal NVMe queue.

The normal queue is (more-or-less) a ringbuffer with N slots in memory and a head/tail pointer. You append the command to the next slot and increase the tail by writing to a doorbell. Once the controller is done it increases the head the same way.

Apple's "queue" instead is just a memory region without those head/tail pointers. Command submission now works by again putting the request into a free slot followed by just writing the ID of that slot to a MMIO register. Once a command is done the CPU again gets an interrupt and can just take the command out of the buffer again.

This probably makes the driver a little bit easier to implement.

On top of that a similar structure (which identifies the DMA buffers that need to be allowed) also needs to be put into their NVMe-IOMMU with a reference to the command buffer entry. The slightly weird thing about the encryption is that you put the key/iv into this buffer instead of the normal queue. My best guess is that this IOMMU design pushed them to also simplify the command queue to make the matching easier.

Hiding the encryption part inside the IOMMU also makes sense for them because the whole IOMMU management is hidden inside a highly protected area of their kernel with more privileges while the NVMe driver itself is just a regular kernel module which possibly doesn't have access to keys.

floatboth3• 5 years ago

Of course most of the platform weirdness is down to "we're vertically integrated and our business goal is the end product and getting that product done fast" but some of the design decisions are kinda baffling even when taking that into account. Like what exactly did they gain by making a weird non-PCIe NVMe situation? Is it really any easier/faster to make the kernel handle that crap than to put the drive on a virtual PCIe bus in hardware+firmware? Are they.. (oh no) trying to improve boot speed by not having to discover the drive on PCIe?!

wtallis2• 5 years ago

Like what exactly did they gain by making a weird non-PCIe NVMe situation?

When Intel did exactly that, there was a clear plausible chain of decisions leading to that madness. I have no clue what may have led Apple in this direction, but the excuses probably aren't any more pathetic than trying to explain why Intel has shipped two mutually-incompatible "solutions" for preventing NVMe drives from working out of the box with unmodified Windows.

floatboth3• 5 years ago

VMD is weird, but at least it doesn't require big intrusive changes like decoupling your NVMe driver from PCIe. It's more like a weird special PCI-PCI bridge. Only needs a little extra driver: https://reviews.freebsd.org/D21383

marcan_4212• 5 years ago

Why have PCIe when you don't need it? Nothing else is on PCIe internally on mobile SoCs. It's just not a thing. That would just add silicon that you don't need.

All these "crazy" design decisions only look crazy from the point of view of x86/server hardware. From the point of view of an embedded SoC this is all reasonable and standard practice.

floatboth3• 5 years ago

Well, yeah – embedded standard practice is not caring about standards and expecting the kernel to accommodate you. Makes sense for Apple with their vertical integration, but even the less integrated things do that, see the entire concept of a "BSP".

But that is what sucks. That's why embedded is creating all these piles of e-waste doomed to only run crappy outdated vendor kernels unless someone invests huge effort into reverse engineering. The whole "we don't need" attitude in regards to standard things that mainline kernels Just Work with is evil.

And now – with both Apple and Qualcomm – we have this embedded crap powering general purpose laptops…

ribit• 5 years ago

Well, their SSDs are not PCI-e devices, they are directly connected to the SoC. I think that their use of NVMe is just a compatibility stop gap, they will probably move to a custom direct-access interface in the near future.

marcan_4212• 5 years ago

They're using NVMe with customizations because NVMe is a good standard. There's no reason to reinvent it from scratch when it works; they can just make the non-standard changes they feel like making, as they have already done.

There's no fundamental reason why NVMe has to be tied to PCIe; it just happens to be that way on existing devices.

svenpeter3• 5 years ago

NVMe requires a co-processor (which Apple calls "ANS") to be up and running before it works. This co-processor firmware seems to have a lot of code and strings dealing with PCIe. Now I haven't looked at the firmware in detail but I'm willing to bet that the actual drives are on a PCIe bus (or at least used to be on a PCIe bus on previous hardware).

It's just that this bus is not exposed to the main CPU but only to this co-processor instead. The co-processor then seems to emulate (or maybe it's just a passthrough) a relatively standard NVMe MMIO space.

marcan_4212• 5 years ago

Yes, the raw NAND storage modules are connected over PCIe on all M1 machines, to dedicated root ports that are behind ANS. As far as I can tell ANS implements (parts of?) the FTL and data striping and other high-level features, and effectively RAIDs together the underlying storage modules into a single unified NVMe interface. So in this sense, the PCIe here is just an implementation detail, the physical/logical interface Apple chose to connect its Flash modules to the built-in non-volatile storage controller in the M1.

monocasa3• 5 years ago

Are there any plans to replace the Apple firmware for ANS as well, or is that so locked down with signature checks that we can't expect to be able to?

Reverse Engineering the M1 [pdf]

Related Stories