Lanai, the mystery CPU architecture in LLVM

https://q3k.org/lanai.html

by todsacerdoti • 4 years ago

287 149 4 years ago

q3k.org

CalChris2• 4 years ago

The world's best instruction mnemonic: PUNT, to switch between user and system contexts.

That's pretty good but I still prefer PowerPC's EIEIO.

https://www.ibm.com/docs/en/aix/7.2?topic=set-eieio-enforce-...

classichasclass• 4 years ago

There are many great Power mnemonics. VSX introduced one nearly as good, xxlxor (which, quite reasonably, is a logical XOR between VSX registers/FPRs). It's delightful to try to even pronounce.

The best part is that eieio's derivation is totally plausibly serious from its stated function. (Remember, IBM doesn't have a corporate sense of humour.) It's also an easy, fairly lightweight speculation barrier apart from its official usage.

thebester5• 4 years ago

My favorite PPC instruction is darn. It delivers a random number, either raw or NIST conditioned. We decided to use the same instruction name in an internal RISCV implementation, and fun times have been had complaining about the darn instruction not working.

exikyut2• 4 years ago

After mumbling "x-x-l-x-o-r" my brain decided to pronounce it "excelsior". Huh.

davidw2• 4 years ago

If Google acquired it, the cynical take is that they ended up sitting on it and doing nothing or killing it.

wmf2• 4 years ago

The fact that they developed and upstreamed an LLVM backend indicates that they probably did something with it.

rsstack• 4 years ago

The Google way: Force the rest of the world adjust to your new "cool thing", and then kill it because it wasn't that great.

ncmncm4• 4 years ago

Mentor Graphics used to buy up silicon-compiler companies just to shut them down, so that manual layout tools would have a longer shelf life. They finally had to give that up around 1990, and figure out a different business to be in. Amazingly, they did.

drewg1238• 4 years ago

I worked for Myricom 2001->2013, and Google 2013->2015. And, wow, I wish I could comment on this thread.. :)

EDIT: Scott's Medium blog post linked from this article is fascinating: https://medium.com/swlh/myricom-an-hpc-story-and-lessons-lea...

q3k4• 4 years ago

Yeah, that blog post was a great reference to get a general overview of What Happened At Myricom.

... and I just hope I didn't write anything factually wrong in TFA. :)

CalChris2• 4 years ago

I just want to know why it needs to be in-tree.

wyldfire• 4 years ago

Google uses upstream llvm as their dev repo. I don't know if they do any development downstream.

IshKebab• 4 years ago

LLVM doesn't have a plugin system so it's that, or maintain a fork and deal with horrible merges all the time.

Fnoord2• 4 years ago

Some important/big customer pays a maintainer to maintain it upstream instead of having to maintain/port it in-house. If there's two of these, it makes sense. Even with one it would otherwise require backporting. Might as well do it in public?

ncmncm4• 4 years ago

Why can you not comment? Certainly any NDA you signed must be expired by now.

PSA: If you are not getting continuing payments, any contracts you signed are void. Technically, I gather you are supposed to notify the other party that you are terminating your participation in the contract. Even if you are getting ongoing payments, you can opt out of that, too.

(This is not legal advice. Consult an actual lawyer for anything that matters.)

pcwalton• 4 years ago

It is absolutely not the case that you can just quit a company, wait a year, and then dump all their confidential information.

Even if there were no legal repercussions (which there are), that's a great way to never be hired again.

dboreham2• 4 years ago

Hard to make a more wrong post (everything apart from the advice to consult a lawyer is incorrect).

quercusa• 4 years ago

This line in that post is mystifying: they peddled Infiniband, an inferior design based on bus technology, but repositioned as a point to point network

detaro4• 4 years ago

I think that's a confusing reference to the fact that InfiniBand initially was supposed to be also a replacement for system-internal buses (i.e. a PCI replacement), before PCIe took that place instead.

dboreham2• 4 years ago

You can use a bus PHY in a p2p configuration.

NavinF• 4 years ago

Yeah that makes no sense.

drewg1238• 4 years ago

At least initially (~2005 or so?) we had customers who were given free IB for their HPC clusters ripping the IB out and replacing it with with paid Myrinet installations because they could never get their IB clusters to work reliably.

Although we had the superior product, we could not market our way out of a wet paper bag. Our CEO counted on our (then) superior engineering & expected our products to market themselves. Eventually the IB engineering caught up and surpassed ours, but our marketing never even came close to theirs, so they were the clear winner.

jandrese• 4 years ago

The CEO capped the full time headcount at 49 people so he wouldn’t be subject to California law requiring him to provide his employees with a health plan.

Can you imagine being this stingy and short sighted? The employees are largely PhDs and he is willing to cripple his company to avoid paying benefits.

drewg1238• 4 years ago

I'm not sure that part of Scott's article was accurate.

I worked remotely, and I'm pretty sure that I was offered benefits. I declined them, and used my wife's benefits because they were less expensive, and meshed better with local health care providers.

EDIT: And I know we had a 401K

foobiekr• 4 years ago

It’s not even that hard or expensive to offer 401k, health, life insurance in a startup. Been there, done that.

_tom_2• 4 years ago

I believe that There are many things that kick in at 50 people. It's oversimplifying to assume this was due to not wanting to offer benefits.

First google hit: https://www.aeisadvisors.com/california-compliance-is-my-com...

edgyquant6• 4 years ago

After the ACA passed, I remember my ex getting 29 hours at the places she worked because 30 hours demanded they be given benefits like health insurance. I really think regulations with a set number like this are the result of corruption or something. No way anyone doesn’t foresee companies playing them this way.

orangecat• 4 years ago

I can totally imagine not wanting the hassle of my company having to be a health insurance broker in addition to its actual purpose. What's wrong with paying employees a good salary and having them be responsible for their own insurance, any more than having them buy their own food, housing, and transportation?

voldacar• 4 years ago

When you create a sharp discontinuity in the marginal cost of hiring someone, expect companies to act accordingly. It would be irrational not to. Blame the system of shitty incentives

Melatonic4• 4 years ago

Interesting - I had never heard of Myricom. I have used Infiniband quite a bit in budget super high performance applications - the old hardware can sometimes be found quite cheaply compared to 40gig/100gig ethernet (or it was at least a few years ago). Much more fiddly to get working but the performance was impressive once it did.

gnufx3• 4 years ago

For what it's worth, I've had rather more trouble with Ethernet than IB, especially given IB management facilities.

phkahler• 4 years ago

Regarding the blog post: "Why did Myricom’s second CEO fail so fast, lack of experience, and loss of respect? If one were to attempt to explain the plummeting fortunes of Myricom during the term of the second CEO, two characteristics suggest themselves:"

It seems to me the second CEO didn't care about the company, the product or the risk the previous CEO wanted to take (more investment to bring out the new chip). Instead they proposed (and got) another round of dividends for the investors, a promotion to CEO, and ultimately an aqui-hire to Google. To me this reads very simply as putting profit and self-interest ahead of the product. Understandable, and it happens all the time. If you have a vision, the best way to achieve it is to remain in control. Somehow the CEO lost that control.

_tom_2• 4 years ago

When does the NDA expire? 7 years is a long time.

otterley2• 4 years ago

Most NDAs are perpetual.

speed_spread2• 4 years ago

I just want the PUNT instruction to become a standard RISC-V extension.

Fnoord2• 4 years ago

Plot twist: you were the CEO.

alduin32• 4 years ago

The moral of the story is simple, expect that in a small start-up your stock will NEVER be worth anything, and get what you can in your normal paycheck.

That's an important moral.

I wish someone had told me that back then.

dreeple16• 4 years ago

Correct me if I'm wrong, but Lanai was designed by Jakov. Who sadly and unexpectedly passed away a few weeks ago.

waych• 4 years ago

Ah shit :-( RIP Jakov

drewg1238• 4 years ago

Indeed, it was quite a shock. Jakov was a wonderful person and the best supervisor one could hope to have. We're all very sad about his passing, and very sad for his family.

vsrinivas• 4 years ago

I'm sorry to hear Jakov passed away - he was a brilliant, kind engineer.

vlovich123• 4 years ago

NDAs AFAIK typically last 5 years. Are you sure you can't comment?

wheybags• 4 years ago

I gotta say, if I was the BDFL of llvm, I would kick this out of tree without a second thought. Why on earth should a foss project support a private architecture? How many man-hours have been wasted waiting for the Lanai backend to compile? Not to mention applying project-wide refactoring to the Lanai code.

judofyr• 4 years ago

There's more context in the thread where it was upstreamed: https://lists.llvm.org/pipermail/llvm-dev/2016-February/0951...

Some notable comments:

I was going to mention it, but you guys know very well the drill, so unless this hardware changes fundamental parts of the middle end in ways that are unnatural to most other targets (doesn't seem that way), then I see no reason why not have it upstream.
I see no problem with having the backend upstream with the understanding that all the normal policies apply. Getting more people working on ToT is valuable to the community as a whole and provided it's "just another backend" with plenty of tests, the cost is low.

klausa• 4 years ago

A significant part of work on LLVM comes from Apple, which includes support for quirks of _their_ proprietary chips and platform; and the world is much better for those changes being upstreamed rather than living in a fork somewhere.

mjw1007• 4 years ago

There's a big difference between "we are the only people who sell hardware using these chips" and "we are the only people who have access to hardware using these chips".

tedunangst2• 4 years ago

How many man-hours have been wasted waiting for the Lanai backend to compile?

Probably zero? Is it compiled by default?

jcelerier2• 4 years ago

Yes. Run `strings /usr/lib/libLLVM.so| grep -i Lanai` to check on your version ; mine has all the relevant symbols

Someone• 4 years ago

Removing architectures because they’re obsolete/maintainers can’t be found, I can see (and the llvm maintainers agree. They removed Alpha at some time, for example), but because they’re proprietary? That would mean kicking out x86, x64, ARM, AMD Terascale, IBM’s z/architecture, etc.

I’m sure clang would be very happy with that.

speed_spread2• 4 years ago

Lanai is more than just proprietary, it's an entirely private architecture.

nickm12• 4 years ago

That was my first reaction, too, but then I had a second thought. Google is going to do this work on this in-tree or out-of-tree. If it is done in-tree, then there is a possibility that it will lead to core improvements that would benefit all backends. If out-of-tree, those are guaranteed to happen. I don't think it's a slam dunk, but on the balance it could make sense. And once it is in-tree you can actually validate this hypothesis by seeing the impact (or not) of contributions from Google.

IncRnd• 4 years ago

Some of my recent long-term projects revolve around a little known CPU architecture called 'Lanai'. Unsurprisingly, very few people have heard of it, and even their Googling skills don't come in handy.

I don't get this. Searching for [lanai cpu] shows tons of links on the LANai cpu architecture from Myricom, purchased by Google.

illegalmemory• 4 years ago

This seems related

https://github.com/chriseth/notes/blob/186d7ea0742336ed38e39...

"TrueBit - Off-Chain Computations for Smart Contracts"

q3k4• 4 years ago

This is https://github.com/TrueBitProject/lanai , which is in itself an interesting endeavor which I forgot to write about. I've added a mention about it to TFA.

maximilianburke• 4 years ago

Here's the thread on the LLVM development lists from when Lanai was proposed to be upstreamed with some good commentary about why it should be accepted even if Google isn't willing to go into details: https://discourse.llvm.org/t/rfc-lanai-backend/39874

nwatson• 4 years ago

For the link, Norton reports: "This is a known dangerous webpage. It is highly recommended that you do NOT visit this page." The "page full report" doesn't reveal any particulars. I rarely get this warning elsewhere.

q3k4• 4 years ago

This is an old domain, and once upon a time some private malware sample links hosted on it got indexed. Years later and some software still think I'm evil :).

trasz• 4 years ago

Random, tangentially related thing I just remembered: FreeBSD used to provide its own firmware for (iirc MIPS-like) core used in very early Broadcom NICs. There also used to be a custom firmware for some Adaptec controllers, along with an assembler tool to compile it during kernel build.

drewg1238• 4 years ago

I think you're talking about the Alteon Tigon-II NICs (Alteon was acquired by Broadcom). Ken Merry and I modified Alteon firmware so that it supported zero-copy sockets by doing page-flipping on receive.

See https://people.freebsd.org/~ken/zero_copy/

p_l• 4 years ago

All LSI "SCRIPTS" HBAs had firmware as part of kernel/driver sources among the many platforms they were used. I believe linux used a simple assembler whose sources were included in kernel to generate the binaries (if not preprocessor tricks, even). *BSDs definitely had their own solution too.

biorach• 4 years ago

The CEO capped the full time headcount at 49 people so he wouldn't be subject to a California law requiring him to provide his employees with a health care plan.

...

words fail me

cplusplusfellow• 4 years ago

I believe this was disputed by a former employee elsewhere ITT.

Melatonic4• 4 years ago

Given the timing by Google I suspect this COULD be involved in what Paul Debevec was doing there with giant camera sphere arrays

Melatonic4• 4 years ago

Interesting that this is being downvoted but not commented on.....

hogehoge51• 4 years ago

Lanai …. Sounded familiar, of course myri. I used these NICs to implement wire speed 10g packet capture. They were a fraction of the cost of dedicated packet capture boards and had a nice api.

These days I guess a Risc-v would be a perfect match for this sort of application, but back then it seemed every accelerator startup would implement a custom ISA.

kyrra• 4 years ago

Googler, opinions are my own.

Btw, the Lanai target in LLVM can be found here: https://github.com/llvm/llvm-project/tree/main/llvm/lib/Targ.... Latest commit is only 24 days ago, so it looks to still be active. Though I'm not sure how much of that is generic target updates, vs target specific changes.

zozbot234• 4 years ago

How would Lanai compare to a simple RV32I isa/core for this application area? The short description in OP doesn't really clarify what, if anything, might be specifically compelling about this arch.

q3k4• 4 years ago

Lanai is significantly older than RISC-V. These days you'd likely easily pick designing or using an existing RISC-V core for an application like this, possibly adapting it to your particular usecase.

The decision is different, however, if you happen to have the entirety of the Lanai engineering team on board and own the entire Lanai IP portfolio. :)

derefr• 4 years ago

Reminds me of the various uses that have come of the IP of the https://en.wikipedia.org/wiki/ARC_(processor) — including inside the Intel Management Engine! — since its ignominious start as the SNES SuperFX chip :)

throwawag• 4 years ago

Probably shouldn't reveal this but, Google are using it internally for their accelerated networking in their cloud, pretty much all Lanai hardware is deployed in GCP infrastructure.

jeffbee• 4 years ago

What explains why Amazon has been relatively open about Nitro but Google has never said anything about how their NICs and SSDs work? It can't be because it's secret sauce; Amazon beats them on latency etc.

jsolson4• 4 years ago

We've been fairly open about how Google Cloud Networking works -- https://www.usenix.org/conference/nsdi18/presentation/dalton

There's also some more recent coverage on IPUs that I don't have a handy link to.

Aperocky• 4 years ago

security by obscurity?

exikyut2• 4 years ago

Cool, it went somewhere very wide and deep.

I guess the only genuinely interesting question worth asking (and hoping for an answer to) is, given the back and forth about performance and Myrinet vs IB on here... how well does the design stack up in practice to an Ebit-scale environment?

jsolson4• 4 years ago

You should not have posted this, least of all because it's extraordinarily misleading.

We use Lanai elsewhere, and it's in hardware powering Google's server networking networking (inclusive of GCP), but Lanai sadly does not play a substantial role in Andromeda networking.

Feel free to ping me on corp (username in my profile here) to discuss.

Lanai, the mystery CPU architecture in LLVM

Related Stories