Why Oxide Chose Illumos

https://rfd.shared.oxide.computer/rfd/0026

by kblissett • 2 years ago

226 167 2 years ago

rfd.shared.oxide.computer

Rendello2• 2 years ago

I like the RFDs. Oxide just did a podcast episode on the process:

https://oxide.computer/podcasts/oxide-and-friends/2065190

rtpg• 2 years ago

There is not a significant difference in functionality between the illumos and FreeBSD implementations, since pulling patches downstream has not been a significant burden. Conversely, the more advanced OS primitives in illumos have resulted in certain bugs being fixed only there, having been difficult to upstream to FreeBSD.

curious about what bugs are being thought of there. Sounds like a very interesting situation to be in

taspeotis2• 2 years ago

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer.

Bryan Cantrill is CTO of Oxide [1].

I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion.

[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

gyre0072• 2 years ago

Yeah I came here to say that Bryan worked at Sun so why do they even need to write this post (yes, I appreciate the techinical reasons, just wanted to highlight the fact via a subtle dig :-))

sausagefeet4• 2 years ago

This isn't a blog post from an Oxide, it's a link to their internal RFD which they use to make decisions.

sausagefeet4• 2 years ago

Early Oxide founders came from Joyent which was an illumos shop and Cantrill is quite vocal about the history of Solaris, OpenSolaris, and illumos.

codetrotter2• 2 years ago

Joyent which was an illumos shop

And before that, they used to run FreeBSD.

Mentioned for example in this comment by Bryan Cantrill a decade ago:

https://news.ycombinator.com/item?id=6254092

[…] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […]

And I’ve seen some other primary sources (people who worked at Joyent) write that online too.

And Bryan Cantrill, and several other people, came from Sun Microsystems to Joyent. Though I’ve never seen it mentioned which order that happened in; was it people from Sun that joined Joyent and then Joyent switched from FreeBSD to Illumos and creating SmartOS? Or had Joyent already switched to Illumos before the people that came from Sun joined?

I would actually really enjoy a long documentary or talk from some people that worked at Joyent about the history of the company, how they were using FreeBSD and when they switched to Illumos and so on.

mzi• 2 years ago

@bcantrill is the CTO of Oxide.

taspeotis2• 2 years ago

Yup, thanks

panick21_6• 2 years ago

Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See:

https://www.youtube.com/watch?v=cwAfJywzk8o

As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong).

So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.

bonzini12• 2 years ago

KVM got more and more integrated with the rest of Linux as more virtualization features became general system features (e.g. posted interrupts). Also Google and Amazon are working more upstream and the pace of development increased a lot.

Keeping a KVM port up to date is a huge effort compared to bhyve, and they probably had learnt that in the years between the porting of KVM and the founding of Oxide.

elijahwright• 2 years ago

Where is Max Bruning these days?

bonzini12• 2 years ago

QEMU is often the subject of bugs affecting its reliability and security.

{{citation needed}}?

When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU.

Here's a talk I gave about it that includes that analysis:

slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf

video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y

_rs• 2 years ago

I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?

bonzini12• 2 years ago

AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances.

Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.

daneel_w3• 2 years ago

QEMU can use a number of different hypervisors, KVM and Xen being the two most common ones. Additionally it can also emulate any architecture if one would want/need that.

TimTheTinker6• 2 years ago

When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years.

So there existed known guest-exploitable vulnerabilities as recently as 8 years ago. Maybe that, combined with the fact that QEMU is not written in Rust, is what is causing Oxide to decide against QEMU.

I think it's fair to say that any sufficiently large codebase originally written in C or C++ has memory safety bugs. Yes, the Oxide RFD author may be phrasing this using weasel words; and memory safety bugs may not be exploitable at a given point in a codebase's history. But I don't think that makes Oxide's decision invalid.

bonzini12• 2 years ago

That would be a damn good record though, isn't it? (I am fairly sure that more were found since, but the point is that these are pretty rare). Firecracker, which is written in Rust, had one in 2019: https://www.cve.org/CVERecord?id=CVE-2019-18960

Also QEMU's fuzzing is very sophisticated. Most recent vulnerabilities were found that way rather than by security researchers, which I don't think it's the case for "competitors".

dvdbloc• 2 years ago

What do the big three use?

paxys• 2 years ago

AWS – Nitro (based on KVM)

Google – "KVM-based hypervisor"

Azure – Hyper-V

You can of course assume that all of them heavily customize the underlying implemenation for their own needs and for their own hardware. And then they have stuff like Firecracker, GVisor etc. layered on top depending on the product line.

daneel_w3• 2 years ago

Some more data:

Oracle Cloud - QEMU/KVM

Scaleway - QEMU/KVM

anonfordays12• 2 years ago

Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU.

QEMU typically uses KVM for the hypervisor, so the vulnerabilities will be KVM anyway. The big three all use KVM now. Oxide decided to go with bhyve instead of KVM.

bonzini12• 2 years ago

No, QEMU is a huge C program which can have its own vulnerabilities.

Usually QEMU runs heavily confined, but remote code execution in QEMU (remote = "from the guest") can be a first step towards exploiting a more serious local escalation via a kernel vulnerability. This second vulnerability can be in KVM or in any other part of the kernel.

6c696e7578• 2 years ago

Azure uses hyper-v, unless things have changed massively, the linux they run for infra and customers is in hyper-v.

cmeacham98• 2 years ago

The big three all use KVM now.

This isn't true - Azure uses Hyper-V (https://learn.microsoft.com/en-us/azure/security/fundamental...), and AWS uses an in-house hypervisor called Nitro (https://aws.amazon.com/ec2/nitro/).

hinkley• 2 years ago

If they are being precise, then “reliability and security” means something different than “security and reliability”.

How many reliability bugs has QEMU experienced in this time?

The man power to go on site and deal with in the field problems could be crippling. You often pick the boring problems for this reason. High touch is super expensive. Just look at Ferrari.

sausagefeet4• 2 years ago

While it's fair to say this does describe why Illumos was chosen, the actual RFD title is not presented and it is about Host OS + Virtualization software choice.

Even if you think it's a foregone conclusion given the history of bcantrill and other founders of Oxide, there absolutely is value in putting decision to paper and trying to provide a rational because then it can be challenged.

The company I co-founded does an RFD process as well and even if there is 99% chance that we're going to use the thing we've always used, if you're a serious person, the act of expressing it is useful and sometimes you even change your own mind thanks to the process.

transpute3• 2 years ago

Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN

  1. Xen Type-1 hypervisor is smaller than KVM/QEMU.
  2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS.
  3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware.
  4. bhyve is Type-2.
  5. Xen has Type-2 (uXen).
  6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM.
  7. pKVM (Arm/Android) is smaller than KVM/Xen.

> The Service Management Facility (SMF) is responsible for the supervision of services under illumos.. a [Linux] robust infrastructure product would likely end up using few if any of the components provided by the systemd project, despite there now being something like a hundred of them. Instead, more traditional components would need to be revived, or thoroughly bespoke software would need to be developed, in order to avoid the technological and political issues with this increasingly dominant force in the Linux ecosystem.

Is this an argument for Illumos over Linux, or for translating SMF to Linux?

bonzini12• 2 years ago

Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.

actionfromafar2• 2 years ago

I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".

packetlost• 2 years ago

The Oxide folks are rather vocal about their distaste for the Linux Foundation. FWIW I think they went with the right choice for them considering they'd rather sign up for maintaining the entire thing themselves than saddling themselves with the baggage of a Linux fork or upstreaming

netbsdusers• 2 years ago

Talking about "technological and political issues" without mentioning any

I don't know why you think none were mentioned - to name one, they link a GitHub issue created against the systemd repository by a Googler complaining that systemd is inappropriately using Google's NTP servers, which at the time were not a public service, and kindly asking for systemd to stop using them.

This request was refused and the issue was closed and locked.

Behaviour like this from the systemd maintainers can only appear bizarre, childish, and unreasonable to any unprejudiced observer, putting their character and integrity into question and casting doubt on whether they should be trusted with the maintenance of software so integral to at least a reasonably large minority of modern Linux systems.

evandrofisico• 2 years ago

I've been using xen in production for at least 18 years, and although there is been some development, it is extremely hard to get actual documentation on how to do things with it.

There is no place documenting how to integrate the Dom0less/Hyperlaunch in a distribution or how to build infrastructure with it, at best you will find a github repo, with the last commit dated 4 years ago, with little to no information on what to do with the code.

transpute3• 2 years ago

github repo, with the last commit dated 4 years ago

Some preparatory work shipped in Xen 4.19.

Aug 2024 v4 patch series [1] + Feb 2024 repo [2] has recent dev work.

hard to get actual documentation

Hyperlaunch: this [3] repo looks promising, but it's probably easier to ask for help on xen-devel and/or trenchboot-devel [4]. Upstream acceptance is delayed by competing boot requirements for Arm, x86, RISC-V and Power.

dom0less: ELC2022 slides [5] and video [6].

[1] https://lists.xenproject.org/archives/html/xen-devel/2024-08... [2] https://github.com/FidelisPlatform/xen

[3] https://github.com/apertussolutions/initrd-builder [4] https://groups.google.com/g/trenchboot-devel

[5] https://www.slideshare.net/StefanoStabellini/static-partitio... [6] https://www.youtube.com/watch?v=CiELAJCuHJg

dijit4• 2 years ago

Honestly, SMF is superior to SystemD and it’s ironic it came earlier (and, that shows based on the fact that it uses XML as its configuration language.. ick).

However, two things are an issue:

1) The CDDL license of SMF makes it difficult to use, or at least that’s what I was told when I asked someone why SMF wasn’t ported to Linux in 2009.

2) SystemD is it now. It’s too complicated to replace and software has become hopelessly dependent on its existence, which is what I mentioned was my largest worry with a monoculture and I was routinely dismissed.

So, to answer your question. The argument must be: IllumOS over Linux.

transpute3• 2 years ago

software has become hopelessly dependent on its existence

With some effort, Devuan has managed to support multiple init systems, at least for the software packaged by Devuan/Debian.

SMF is superior to SystemD ... [CDDL]

OSS workalike opportunity, as new Devuan init system?

The argument must be: IllumOS over Linux.

Thanks :)

anonfordays12• 2 years ago

Honestly, SMF is superior to SystemD

Maybe 15 years ago, not by a mile now. systemd surpassed SMF years ago and it's not even close now. No one in their right mind would pick SMF over systemd in 2024.

cthalupa3• 2 years ago

Is this an argument for Illumos over Linux, or for translating SMF to Linux?

I'd certainly like that! I had spent some time working with Solaris a lifetime ago, and ran a good amount of SmartOS infrastructure slightly more recently. I really enjoyed working with SMF. I really do not enjoy working with the systemd sprawl.

I will note the distinction between type-1/type-2 hypervisors never really made sense, and makes even less sense today. http://blog.codemonkey.ws/2007/10/myth-of-type-i-and-type-ii...

saagarjha• 2 years ago

Unrelated, but is this a homegrown publishing platform?

crb• 2 years ago

Yes; it's referred to in Oxide's RFD about RFDs [1] https://rfd.shared.oxide.computer/rfd/0001 but the referenced URL is 404 unless you're an Oxide employee.

[1] https://rfd.shared.oxide.computer/rfd/0001#_shared_rfd_rende... [2] https://github.com/oxidecomputer/rfd/blob/master/src

dcre• 2 years ago

That link is out of date. The site and backend are now open source. Only the repo containing the RFD contents is private.

https://github.com/oxidecomputer/rfd-site

https://github.com/oxidecomputer/rfd-api

benjaminleonard• 2 years ago

Yep, you can see a little more on the [blog](https://oxide.computer/blog/a-tool-for-discussion) or the most recent [podcast](https://oxide.computer/podcasts/oxide-and-friends/2065190). The API and site repos are also public.

ReleaseCandidat3• 2 years ago

Instead of stating more or less irrelevant reasons, I'd prefer to read something like "I am (have been?) one of the core maintainers and know Illumos and Bhyve, so even if there would be 'objectively' better choices, our familiarity with the OS and hypervisor trump that". A "I like $A, always use $A and have experience using $A" is almost always a better argument than "$A is better than $B because $BLA", because that doesn't tell me anything about the depth of knowledge of using $A and $B or the knowledge of the subject of decision - there is a reason half of Google's results is some kind of "comparison" spam.

actionfromafar2• 2 years ago

But everyone at Oxide already knows that back story. At least if you list some other reasons list you can have a discussion about technical merits if you want to.

ReleaseCandidat3• 2 years ago

But that doesn't make sense if you have specialists for $A that also like to work with $A. Why should I as a customer trust Illumos/Bhye developers that are using Linux/KVM instead of "real" Linux/KVM developers? The only thing that such a decision would tell me is to not even think about using Illumos or Bhyve.

The difference between

    "Buy our Illumos/Bhye solution! Why? I have been an Illumos/Bhyve Maintainer!"

and

    "Buy our Linux/KVM solution! Why? I have been an Illumos/Bhyve Maintainer!"

should make my point a bit clearer

panick21_6• 2 years ago

But Bryan also ported KVM to Illumos. And Joyand used KVM and they supported KVM there for years, I assume Bryan knows more about KVM then Bhyve as he seemed very hands on in the implementation (there is nice talk on youtube). So the idea that he isn't familiar with KVM isn't the case. So based on that KVM or Bhyve on Illumos, KVM would suggest itself.

In the long term if $A is actually better then $B, then it makes sense to start with $A even if you don't know $A. Because if you are trying to building a company that is hopefully making billions in revenue in the future, then long term matters a great deal.

Now the question is can you objectively figure out if $A or $B is better. And how much time does it take to figure out. Familiarity of the team is one consideration but not the most important one.

Trying to be objective about this, instead of just saying 'I know $A' seems quite like a smart thing to do. And writing it down also seems smart.

In a few years you can look back and actually say, was our analysis correct, if no what did we misjudge. And then you can learn from that.

If you just go with familiarity you are basically saying 'our failure was predetermined so we did nothing wrong', when you clearly did go wrong.

jclulow9• 2 years ago

For what it's worth, we at _Joyent_ were seriously investing in bhyve as our next generation of hypervisor for quite a while. We had been diverging from upstream KVM, and most especially upstream QEMU, for a long time, and bhyve was a better fit for us for a variety of reasons. We adopted a port that had begun at Pluribus, another company that was doing things with OpenSolaris and eventually illumos, and Bryan lead us through that period as well.

specialist• 2 years ago

Trying to be objective about this... And writing it down also seems smart.

Mosdef.

IIRC, these RFDs are part of Oxide's commitment to FOSS and radical openness.

Whatever decision is ultimately made, for better or worse, having that written record allows the future team(s) to pick up the discussion where it previously left off.

Working on a team that didn't have sacred cows, an inscrutible backstory ("hmmm, I dunno why, that's just how it is. if it ain't broke, don't fix it."), and gatekeepers would be so great.

blinkingled• 2 years ago

I guess that's one way of keeping Solaris alive :)

daneel_w3• 2 years ago

"• Emerging VMMs (OpenBSD’s vmm, etc): Haven’t been proven in production"

It's a small operation, but https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale.

For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single CPU core.

jclulow9• 2 years ago

Yes, to be clear this is not meant to be a criticism of software quality at OpenBSD! Though I don't necessarily always agree with the leadership style I have big respect for their engineering efforts and obviously as another relatively niche UNIX I feel a certain kinship! That part of the document was also written some years ago, much closer to 2018 when that service got started than now, so it's conceivable that we wouldn't have said the same thing today.

I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product!

notaplumber1• 2 years ago

I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product!

Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? From a secure design perspective, OpenBSD's vmd is a lot more appealing than bhyve is today.

I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet?

tonyg• 2 years ago

Nested virtualisation [...] challenging to emulate the underlying interfaces with flawless fidelity [...] dreadful performance

It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.

bonzini12• 2 years ago

It doesn't perform poorly in fact. It can be tuned at 90% of non-nested virtualization, and for workloads where it doesn't, that's more than anything else a testimony to how close virtualized performance is to bare-metal.

That said, it does add a lot of complexity.

fefe23• 2 years ago

These sound like reason you retconned so it sounds like you didn't choose Illumos because your founder used to work at Sun and Joyent before. :-)

Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't.

It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.

jclulow9• 2 years ago

It's not a blog post, it's an RFD. We have a strong focus on writing as part of thinking and making decisions, and when we can, we like to publish our decision making documents in the spirit of open source. This is not a defence of our position so much as a record of the process through which we arrived at it. This is true of our other RFDs as well, which you can see on the site there.

It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.

Yes, the whole product is definitely designed that way intentionally. Customers get abstracted control of compute and storage resources through cloud style APIs. From their perspective it's a cloud appliance. It's only from our perspective as the people building it that it's a UNIX system.

stonogo• 2 years ago

So at no point did anyone even suspect that Illumos was under consideration because it's been corporate leadership's pet project for decades? That seems like a wild thing to omit from the "RFD" process. Or were some topics not open to the "RFD" process?

leoh• 2 years ago

I’d love to use Illumos, but a lack of arm64 support is a non-starter

jclulow9• 2 years ago

Folks are working on it! I believe it boots on some small systems and under QEMU, but it's still relatively early days. I'm excited for the port to eventually make it into the gate, though!

geerlingguy• 2 years ago

In before someone asks about riscv64 ;)

ComputerGuru4• 2 years ago

I don’t mean to downplay the importance for you personally but I do want to clarify that while it might be a non-starter for you, all of arm64 is so new that it’s hardly a non-starter for anyone considering putting it into (traditional) production.

timenova• 2 years ago

You're right, however I was looking for the same information to maybe try it on a RPi to learn more about Illumos.

BirAdam• 2 years ago

I think a bigger reason for Oxide using Illumos is that many of the people over there are former Sun folks.

throw0101b5• 2 years ago

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode:

* https://www.youtube.com/watch?v=UvEKSqBBcZw

Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: "Ceph is operated, not shipped [like ZFS]".

There's more care-and-feeding required for it, and they probably don't want that as they want to treat product in a more appliance/toaster-like fashion.

pclmulqdq3• 2 years ago

Ceph is sadly not very good at what it does. The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). ZFS solves a different problem, though. ZFS is a full-featured filesystem. Like Ceph it is also vulnerable to single points of failure.

throw0101b5• 2 years ago

The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.).

There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023:

* https://www.youtube.com/watch?v=bl6H888k51w

Certainly there are some number of folks that need more than that, but I don't there are many.

Like Ceph it is also vulnerable to single points of failure.

The SPOF for ZFS is the host (unless you replicate, e.g., zfs send).

What is SPOF of Ceph? You can have multiple monitors, managers, and MDSes.

anonfordays12• 2 years ago

ZFS and Ceph is apples to oranges. ZFS is scoped to a single host, Ceph can span data centers.

ComputerGuru4• 2 years ago

It’s very possible to run a light/small layer on top of ZFS (either userspace daemon or via FUSE) to get you most of the way to scaling ZFS-backed object storage within or across data centers depending on what specific availability metrics you need.

throw0101b5• 2 years ago

ZFS and Ceph is apples to oranges.

Oxide is shipping an on-prem 'cloud appliance'. From the customer's/user's perspective of calling an API asking for storage, it does not matter what the backend is—apple or orange—as long as "fruit" (i.e., a logical bag of a certain size to hold bits) is the result that they get back.

wmf4• 2 years ago

You mean they use Crucible instead of Ceph?

jonstewart2• 2 years ago

Illumos makes sense as a host OS—it’s capable, they know it, they can make sure it works well on their hardware, and virtualization means users don’t need that much familiarity with it.

If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, there’s going to be lots of budget devoted to moving off VMWare.

parasubvert2• 2 years ago

Oracle is a $53 billion company, and never had a mass exodus, just less greenfield deployments.

Broadcom also isn't all that dumb, VMware was fat and lazy and customers were coddled for a very long time. They've made a bet that it's sticky. The competition isn't as weak as they thought, that's true, but it will take 5+ years to catch up, not 2 years, in general. Broadcom was betting on it taking 10 years: plenty of time to squeeze out margins. Customers have been trying and failing to eliminate the vTax since OpenStack. Red Hat and Microsoft are the main viable alternatives.

jonstewart2• 2 years ago

I don’t disagree much. Still, there’s a sudden weakening of the main incumbent in the on-prem virtualization market… and that is _all_ Oxide does. It will be interesting to see whether Oxide can convert some VMWare customers.

kayo_202110302• 2 years ago

Is the date on this piece correct?

The section about Rust as a first class citizen seems to contain references to its potential use in Linux that are a few years out of date; with nothing more current than 2021.

As of March 2021, work on a prototype for writing Linux drivers in Rust is happening in the linux-next tree.

kayo_202110302• 2 years ago

nm, I read the postscript. The RFD was from 2021. I wonder how correct it was, and whether decisions made, based on it, were good ones or bad ones.

alberth• 2 years ago

Isn’t it simply Oxide founders are old Sun engineers, and Illumos is the open source spinoff of their old work.

sophacles3• 2 years ago

According to the founders and early engineers on their podcast - no, they tried to fairly evaluate all the oses and were willing to go with other options.

Practically speaking, its hard to do it completely objectively and the in-house expertise probably colored the decision.

pclmulqdq3• 2 years ago

Tried to, sure, but when you evaluate other products strictly against the criteria under which you built your own version, you know what the conclusion will be. Never mind that you are carrying your blind spots with you. I would say that there was an attempt to evaluate other products, but not so much an attempt to be objective in that evaluation.

In general, being on your own private tech island is a tough thing to do, but many engineers would rather do that than swallow their pride.

Aissen• 2 years ago

Point 1.1 about QEMU seems even less relevant today, with QEMU adding support for the microvm machines, hence greatly reducing the amount of exposed code. And as bonzini said in the thread, the recent vulnerability track record is not so bad.

magicalhippo• 2 years ago

Been running Bhyve on FreeBSD (technically FreeNAS). Found PCIe pass-through of NMVe drives was fairly straight forward once the correct incantations were found, but network speed to host has been fairly abysmal. On my admittedly aging Threadripper 1920X, I can only get ~2-3 Gbps peak from a Linux guest.

That's with virtio, the virtual intel "card" is even slower.

They went with Illumos though, so curious if the poor performance is a FreeBSD-specific thing.

bitfilped2• 2 years ago

It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.

craftkiller3• 2 years ago

Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through multiple additional layers that may have slowed it down (off the top of my head: kubernetes with flannel, iptables in the VM, bhyve, and pf on the FreeBSD host).

ComputerGuru4• 2 years ago

I just spun up a VNET jail (so it should be essentially using the same network stack and networking isolation level as a bhyve guest would) and tested with iperf3 and without any tweaking or optimization and without even using jumbo frames I'm able to get 24+ Gbps with iperf3 (32k window size, tcp, single stream) between host/guest over the bridged and virtualized network interface. My test hardware is older than yours, it's a Xeon E5-1650 v3 and this is even with nested virtualization since the "host" is actually an ESXi guest running pf!

But I think you might be right about something because, playing with it some more, I'm seeing an asymmetry in network I/O speeds; when I use `iperf3 -R` from the VNET jail to make the host connect to the guest and send data instead of the other way around, I get very inconsistent results with bursts of 2 Gbps traffic and then entire seconds without any data transferred (regardless of buffer size). I'd need to do a packet capture to figure out what is happening but it doesn't look like the default configuration performs very well at all!

tcdent• 2 years ago

Linux has a rich ecosystem, but the toolkit is haphazard and a little shakey. Sure, everyone uses it, because when we last evaluated our options (in like 2009) it was still the most robust solution. That may no longer be the case.

Given all of that, and taking into account building a product on top of it, and thus needing to support it and stand behind it, Linux wasn't the best choice. Looking ahead (in terms of decades) and not just shipping a product now, it was found that an alternate ecosystem existed to support that.

Culture of the community, design principles, maintainability are all things to consider beyond just "is it popular".

Exciting times in computing once again!

craftkiller3• 2 years ago

I wonder if CockroachDB abandoning the open source license[0] will have an impact on their choice to use it. It looks like the RFD was posted 1 day before the license switch[1], and the RFD has a section on licenses stating they intended to stick to the OSS build:

To mitigate all this, we’re intending to stick with the OSS build, which includes no CCL code.

    [0] https://news.ycombinator.com/item?id=41256222
    [1] https://rfd.shared.oxide.computer/rfd/0110

implr• 2 years ago

They already have another RFD for this: https://rfd.shared.oxide.computer/rfd/0508

and on HN: https://news.ycombinator.com/item?id=41268043

Rendello2• 2 years ago

And a podcast episode: https://oxide.computer/podcasts/oxide-and-friends/2052742

yellowapple• 2 years ago

I'm surprised that KVM on Illumos wasn't in the running, especially with SmartOS setting that as precedent (even if bhyve is preferred nowadays).

mechanicker• 2 years ago

Wonder if this is more due to Bhyve being developed on FreeBSD and Illumos derives from a common ancestor BSD?

I know NetApp (stack based on FreeBSD) contributed significantly to Bhyve when they were exploring options to virtualize Data ONTAP (C mode)

https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervi...

jclulow9• 2 years ago

While we have a common ancestor in the original UNIX, so much of illumos is really more from our SVR4 heritage -- but then also so much of that has been substantially reworked since then anyway.

computersuck• 2 years ago

Because CTO Bryan Cantrill, who was a core contributor to illumos

_zagj• 2 years ago

Ctrl+f Cantrill >Phrase not found

Bryan Cantrill, ex-Sun dev, ex-Joyent CTO, now CTO of Oxide, is the reason they chose Illumos. Oxide is primarily an attempt to give Solaris (albeit Rustified) a second life, similar to Joyent before. The company even cites Sun co-founder Scott McNealy for its principles:

https://oxide.computer/principles

"Kick butt, have fun, don't cheat, love our customers and change computing forever."
If this sounds familiar, it's because it's essentially Scott McNealy's coda for Sun Microsystems.

JonChesterfield• 2 years ago

Illumos and ZFS sounds completely sensible for a company that runs on specific hardware. They mention the specific epyc cpu their systems are running on which suggests they're all ~ identical.

Linux has a massive advantage where it comes to hardware support for all kinds of esoteric devices. If you don't need that, and you've got engineers that are capable of patching the OS to support your hardware, yep, have at it. Good call.

rbatiati• 2 years ago

Letting licensing alone, I think they have a couple of killer reasons to choose Illumos: It's a fine operating system, and it's much easier for them to land the fixes/features they need on the Illumos kernel than if they've built on Linux.

Why Oxide Chose Illumos

Related Stories