Hubris – A small operating system for deeply-embedded computer systems

https://oxide.computer/blog/hubris-and-humility

by jacobwg • 5 years ago

560 238 5 years ago

oxide.computer

solmag2• 5 years ago

So this is what Cantrill has been talking about.

https://www.youtube.com/watch?v=XbBzSSvT_P0

https://www.youtube.com/watch?v=cuvp-e4ztC0

Tuna-Fish• 5 years ago

The github links don't work, are the repositories still private?

detaro7• 5 years ago

It's going to be presented at a talk in ~9 hours, so probably: https://talks.osfc.io/osfc2021/talk/JTWYEH/

pjmlp22• 5 years ago

Do you know how to watch those sessions?

The site is quite confusing, stating that registration and login are only required for speakers, but then there is no information regarding the talks beyond the description.

pjmlp22• 5 years ago

I guess we need to keep ourselves busy with some docs, as that one works.

HeavenSmile3• 5 years ago

yeah sounds as a very good suggestion to me :)

HeavenSmile3• 5 years ago

Yeah also noticed and got a little bit upset about this.. I mean publishing a website with broken links does not seems very smart nor makes very much sense to me..

HeavenSmile3• 5 years ago

That said if Hubris OS will be presented at a talk later today I guess things seems to be more clear to me.

Are kind of way very much looking forward to the talk and the presentation of the operating system btw.

bcantrill8• 5 years ago

Ha, sorry -- HN scooped us on our own announcement! We had the intention of turning the key on everything early this morning Pacific time, but we put the landing page live last night just to get that out of the way. Needless to say, we were a bit surprised to see ourselves as the #2 story on HN when we hadn't even opened it yet! It's all open now, with our apologies for the perceived delay -- and never shall we again underestimate HN sleuths! ;)

Zababa• 5 years ago

Intersting choices of names, Hubris and Humility. Combined with the style of the page, it gives to me a solemn and heavy feeling. Especially compared to most projects presented that tend to be very "positive energy and emojis". Their website is also beautiful https://oxide.computer/. Though I wonder who's the target for this. Is this for cloud provider themselves, for people that self host, for hosters? For everyone?

pjmlp22• 5 years ago

Cantrill has talked quite a few times about this, it is for people that still build their own data centers.

noisy_boy• 5 years ago

Speaking of interesting names, their control plane is called Omicron: https://github.com/oxidecomputer/omicron

barrenko• 5 years ago

Their podcasts is similarly interesting even for me who has no real (professional) interest or knowledge about building computers.

omnicognate2• 5 years ago

Their website is also beautiful

But horribly broken for me (mobile firefox), with text cut off at borders and overlaid by images.

skinkestek• 5 years ago

Same is true in mobile Safari (iOS) but I'll cut them some slack as long as it doesn't work in Chrome on iOS (since then it would be a Chrome specific hack, since Chrome on iOS uses the same engine as Safari.)

benjaminleonard• 5 years ago

Apologies, pushing a fix now. I broke this earlier today!

emptyparadise• 5 years ago

It's really a breath of fresh air.

jabl• 5 years ago

The target market is users that want to build their own cloud infrastructure, but don't have the scale required to go directly to ODM's to have their own custom designs manufactured.

thesuperbigfrog• 5 years ago

I think the names are very clever.

The OS is named Hubris. Building a new Operating System does take a lot of confidence.

The debugger is named Humility. It can be humbling to know your program is not working correctly and use a tool to discover how it is broken.

Impatience would be a great name for the task scheduler. (Because you want your task to run NOW!)

Laziness would be a great name for a hardware-based watchdog timer. (Because you keep on putting it off / resetting it until later.)

Compare: https://www.threevirtues.com/

sgt• 5 years ago

When Cantrill and his team works on something, HN listens, and for good reason. Startups like Oxide show that there's room for a lot of innovation still on a smaller scale, even within fields like HW.

sbarre• 5 years ago

The supervisor model reminds me a bit of how BEAM (Erlang/Elixir) works although I'm sure that's probably where the similarities end.

As much as most of this is way over my head, I'm always fascinated to read about new ground-up work like this.

bo0tzz• 5 years ago

Their mention of individually restarting components and "flexible inter-component messaging" really reminds me of the BEAM. Very exciting!

protomyth• 5 years ago

Reminds me of QNX. It was an amazing OS and restarting display drivers over the network was just one of its amazing abilities.

mwcampbell4• 5 years ago

No surprise, since Bryan Cantrill worked at QNX for a short time in the 90s.

hbbio• 5 years ago

There should be a retweet on HN!

QNX was one the most impressive OS I've seen, especially for its time. From the full OS in a 1.44Mb floppy disk to restartable drivers, real-time, etc. IPC with messages is built in and most things ran in userland.

It ended in the hands of BlackBerry which is probably not the best home for it...

Edit: I googled out of curiosity, and despite being closed source, it seems to still be marketed by BlackBerry and is supposedly a market leader in embedded OSes! More than 20 years later, well deserved.

https://www.automotiveworld.com/news-releases/blackberry-qnx...

pjmlp22• 5 years ago

It is an amazing OS, https://blackberry.qnx.com/en

bcantrill8• 5 years ago

Not an accident. Cliff was influenced by QNX, Minix, L3 and L4 in his design (specifically, QNX proxies directly inspired Hubris notifications). And for me personally, QNX -- both the technology and the company -- played an outsized role in my own career, having worked there for two summers while an undergraduate.[0] (And even though it's been over 20 years since he passed, just recalling the great technology and people there makes me miss the late Dan Hildebrand[1] who had a singular influence on me, as well as so many others.)

[0] http://dtrace.org/blogs/bmc/2007/11/08/dtrace-on-qnx/

[1] https://openqnx.com/node/298

panick21_7• 5 years ago

We are gettig an increasing amount of interesting Rust operating system for different uses.

- Hubris for deep embedded

- Redox OS for Desktop/Server (https://www.redox-os.org/)

- Tock for embedded (https://www.tockos.org/)

- Xous for trusted devices (https://xobs.io/announcing-xous-the-betrusted-operating-syst...)

I assume there are more.

amitpm• 5 years ago

Rust is the language of choice for dApps on Solana as well (which I thought was an interesting choice.)

rvz• 5 years ago

That is completely true. Rust is where it is going for cryptocurrencies that need to have very secure smart-contracts and scale up to millions of users with tens of thousands of transactions per second.

Even if it is completely unrelated, it is 'at least' production ready and will be used more than these projects would ever be used in terms of Rust projects.

pjmlp22• 5 years ago

Microsoft is a bit schizophrenic in Rust's adoption.

In one side of the fence you have Azure IoT Edge, done in a mix of .NET and Rust.

https://msrc-blog.microsoft.com/2019/09/30/building-the-azur...

On the other side you have the Azure Sphere OS with a marketing story about secure IoT, yet the SDK is C only,

https://docs.microsoft.com/en-us/azure-sphere/app-developmen...

To the point they did a blog post trying to defend their use of C,

https://techcommunity.microsoft.com/t5/internet-of-things-bl...

oxidethrowaway6• 5 years ago

To the point they did a blog post trying to defend their use of C,

I do a lot of Rust work, but C still occupies a prominent place in my toolkit.

In fact, this is fairly standard among all of the professionals I know. The idea that companies need to abandon C and only do Rust as soon as possible is more of an idealistic idea on internet communities.

Macha2• 5 years ago

Microsoft is a bit schizophrenic in Rust's adoption.

This is pretty famously not limited to Microsoft's views on Rust. Probably at any sufficiently large company, though Microsoft is the tech company that gets most discussed about. At a previous employer, there was a division that was gung ho on Go and React, while another was Java and Ember.

munificent• 5 years ago

Microsoft is a bit schizophrenic in Rust's adoption.

It's only "schizophrenic" to the degree that you expect an organization of thousands of individuals to behave like a single human mind.

im_down_w_otp4• 5 years ago

We also built a Rust framework called FerrOS (https://github.com/auxoncorp/ferros) atop the formally-verified seL4 microkernel.

It has a similar set of usage idioms to Hubris it looks like in terms of trying to setup as much as possible ahead of time to assemble what's kind of an application specific operating system where everything your use case needs is assembled at build-time as a bunch of communicating tasks running on seL4.

We recently added a concise little persistence interface that pulls in TicKV (https://docs.tockos.org/tickv/index.html) from the Tock project you referenced above, and some provisions are being added for some more dynamic task handling based on some asks from an automotive OEM.

Also, sincerest h/t to the Tock folks.

panick21_7• 5 years ago

Looks like a really cool idea to build on the formally verified C code and building everything else in Rust on top.

I defiantly think all the embedded Rust and even others will end up sharing lots of code.

I would like to spend some time working Datomic style on a more powerful immutable DB that could run on different KV interfaces. Lots of configuration should more immutable.

pjmlp22• 5 years ago

If you can talk about it, how does that play out with standards like AUTOSAR and its C++ requirements?

simion314• 5 years ago

Redox OS

What is interesting about this OS except it is made with Rust? I mean some interesting architecture , new exiting features that are not in Windows/Unix world?

My question is if Redox OS is more similar to other hobby Oses we see here or is something more well thought like Plan9, Fuschia, Singularity ? Is there a link where someone that is not a Rust fanoboy (but maybe an OS fanboy) reviewed/described it in more detail?

dralley2• 5 years ago

Redox is Plan9-inspired except it goes a little further with some of the core concepts.

Instead of everything being a file, everything is a scheme (basically a URL).

7thaccount• 5 years ago

Can anyone explain to a non-server person what Oxide hopes to accomplish? Is it basically just a new server with its own OS that makes it more secure?

solmag2• 5 years ago

I think this OS is intended to run in embedded context where there are significant memory constraints; read its description, no runtime allocations etc.

I linked two speeches where he goes over this in a bit more detail, but I hope the presentation opens it up even more.

detaro7• 5 years ago

Pretty much "let's redo datacenter hardware from the ground up for current requirements, cutting off legacy things we don't need anymore"

jeffbee4• 5 years ago

But the BMC would be the #1 item on my list of "things I don't need any more". How do you come up with a scratch legacy-free universe that still includes BMCs?

p_l7• 5 years ago

Their target market is essentially private cloud space combined with turnkey rack space - a pretty common on-premise setup where you order not individual servers, but complete racks that are supposed to be "plug and play" in your DC (in practice YMMV, spent a month fighting combination of manglement and Rackable mess).

You can think in this case of the final product as pretty big hypervisor cluster that is delivered complete. I'll admit more than once I'd kill for that kind of product, and I suspect that the price/performance ratio might be actually pretty good.

The operating system in this case is used for internal service processor bits (compare: Minix 3 on Intel ME, whatever was that proprietary RTOS on AMD's PSP, etc. etc) that help keep the whole thing running and ship shape.

_alex_• 5 years ago

Bingo. My guess is this is for control plane microcontrollers

ksec3• 5 years ago

a pretty common on-premise setup

I have been wondering if it will become a thing in some cloud hosting services as well. I guess we need to see their pricing.

znpy3• 5 years ago

No, it's a rack level design with the target market being not companies that buy single servers and fill racks with them but hyperscalers that need to fill whole datacenters with servers.

Basically there are a huge range of scale levels where having hardware on-prem make sense financially.

Also, Bryan Cantrill has some sort of personal nitpick with modern servers basically being x86 PCs with a different form factor, and with the fact that in modern servers hardware and software do not cooperate at all (and in some occasion, hardware gets in the way of software).

tw042• 5 years ago

but hyberscalers that need to fill whole datacenters with servers.

I strongly doubt this is aimed at Amazon, Google, or Microsoft (hyperscalers). They all already have their own highly customized hardware and firmware. If that is their target I wish them luck. There’s no margin and a ton of competition in that space and as long as they’ve been working on this that feels like a pretty poor gamble.

What I believe this is actually targeting is small enterprise and up. A company that has dozens to thousands of servers. They’re willing to pay a premium for an easier go to market.

floatboth6• 5 years ago

Hyperscalers have already moved their custom stuff in a direction quite far from x86 PCs (how many new form-factors and interconnects and whatnot are under the Open Compute Project already?) while the typical Supermicro/Dell/HPE/whatever boxes available to regular businesses are still in that "regular PC" world. This is what they're trying to solve, yeah.

zweifuss• 5 years ago

I haven't looked at Oxide in depth. Hubris seems to be about reducing the Attack Surface of a server by

* decreasing the active codebase by at least three orders of magnitude

* using no C-Code (Rust only?)

* most code is kernel independent and not privileged (e.g. drivers, task management, crash recovery)

Also: Administration is mostly done by rebooting components.

detaro7• 5 years ago

Hubris is for system management components like the BMC, not for the main CPU.

panick21_7• 5 years ago

Just to be clear, this OS Hubris is for the service processor. Its an OS for firmware, not the main OS that will run on the CPU.

However they will likely ship with a something derived from Illumos and bhyve hypervisor. You can then provision VM threw the API (and likely integrated with tools like terraform or whatever). You will likely not interact directly with Illumos.

Its basically attempt to help people make running data-center easier.

natemcintosh• 5 years ago

So for a new OS like this, how does one compile their program for it?

steveklabnik19• 5 years ago

You need to write your program for Hubris specifically; we don't support POSIX or anything. I want to have better "getting started" docs sometime soon.

tambourine_man• 5 years ago

The Hubris debugger, Humility…

That is some great naming

JulianMorrison4• 5 years ago

I really like this quote from the manual:

<<There are a class of "ideal attractors" in engineering, concepts like "everything is an object," "homoiconicity," "purely functional," "pure capability system," etc. Engineers fall into orbit around these ideas quite easily. Systems that follow these principles often get useful properties out of the deal.

However, going too far in any of these directions is also a great way to find a deep reservoir of unsolved problems, which is part of why these are popular directions in academia.

In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features.>>

mwcampbell4• 5 years ago

I wonder if the attributes of Hubris and similar systems -- real-time, lack of dynamism -- will become "ideal attractors" for developers not working in problem domains where these things are absolutely required, especially as the backlash against the complexity at higher layers of the software stack continues to grow. In other words, I wonder if a sizable number of developers will convince themselves that they need an embedded RTOS like Hubris for a project where Linux running on an off-the-shelf SBC (or even PC) would work well enough.

melony• 5 years ago

Right now we are going in the opposite direction. Web developers on HN refuse to learn proper embedded programming, and instead stack abstraction on top of abstraction with MicroPython and using Raspberry Pis for every job under the sun.

It is a shame that Arduino/AVR never bothered implementing support for the full C++ library. If the full power of C++ is available to the end user, then perhaps alternatives like MicroPython would be less attractive.

JulianMorrison4• 5 years ago

Containerisation already goes a way towards this, each executable is bundled with what it needs and nothing else. And the next step is one app per hardware, where you have maybe a minimal stripped OS that just launches your app, effectively a container in hardware. I think Google does this a fair amount.

The jump from this to RTOS is large, though. The abstractions are different. The limitations are different. You probably need to rewrite anything you need. And what do you gain? Mostly only predictable latency, and the ability to run on very limited (but cheap) hardware. Which you need why?

travisgriggs• 5 years ago

That impulse will occur as they try to do more and more complex things with systemd.

ModernMech• 5 years ago

I think you are right and I would add Turing incompleteness to that list. If your problem isn't Turing complete, then a complete language is actually probably not the best tool for the job. Incomplete languages actually give you more power than in a Turing complete language in that case. Completeness is a feature of a language and like all features there are tradeoffs. The ability to express more kinds of problems comes with the cost of not being able to leverage the contours of a particular problem (e.g. monotonic data, no cycles) to increase things like speed, debuggability, parallelization. This can enable cool features things that seem completely out of reach today. e.g. rewinding the state of your program to debug production errors. Modifying programs as they're running. Automatically parallelizing computations across an arbitrary number of cores. The ability to query the provenance of any variable in your program.

Datalog's incompleteness for example allows it to resolve queries faster than a complete language like Prolog due to the simplifying inferences it can make about the code.

AnimalMuppet3• 5 years ago

I doubt it. Real time adds its own flavor, and a small OS doesn't come with everything you might want. It's useful when it's what you want, not when you don't want something else.

hinkley6• 5 years ago

If the predictions are true that we will see more and more specialized hardware due to the end of Moore's Law, then we will see more OS services just look like services running on separate processors. Special purpose hardware doesn't need a batteries included operating system. We could argue whether a modular OS still counts as general purpose, but I'll let you guys do that.

With IPC, latency becomes the elephant in the room. An RTOS can't remove that, but it can help.

SideburnsOfDoom• 5 years ago

In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features

Huh, this is more like pragmatism than "Hubris".

steveklabnik19• 5 years ago

We'd like to think so, but it's really hard to convince people that writing an entire new OS in Rust is the pragmatic choice. The name is a good reminder of the balance there, in my humble opinion.

AnimalMuppet3• 5 years ago

"Hubris" is supposed to be a name, not a description.

ls655362• 5 years ago

no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems.

Not to be too pedantic here, but it's important to note that the absence of C code, while arguably a benefit overall, doesn't by itself guarantee anything with regards to safety/security...I suppose there's going to necessarily be at least some "unsafe" Rust and/or raw assembly instructions sprinkled throughout, but I can't yet see that myself (as of the time of writing this comment, the GitHub links are responding with 404). Nonetheless, it's always refreshing to see some good documentation and source code being provided for these kinds of things. Many companies in this space, even these days, sadly continue to live by some outdated values of hiding behind "security through obscurity", which is somehow championed (though using different words) as a benefit even to their own customers, so it's refreshing that others (Oxide among them) are really starting to take a different approach and making their software/firmware publicly available for inspection by anyone inclined to do so.

pjmlp22• 5 years ago

Which is why they state a lot of the attack and not all of the attacks.

steveklabnik19• 5 years ago

To be clear, that sentence refers to the sum total of the things in the previous sentence, not just the bit about C. And it's "a lot of the attack surface" and not "guarantee" for a reason. We don't believe that simply being written in Rust makes things automatically safe or secure.

There is some unsafe Rust and some inline assembly, yes. I imagine a lot less than folks may think.

ls655362• 5 years ago

I will admit perhaps I was a bit too loose with my own interpretation of the statement there. I think maybe this was influenced by my being tired of grand statements others have made in the past about the infallibility of writing code in Rust (even with liberal usage of "unsafe" without a proper understanding of what this implies).

It's all too often I see some cargo cult-style declaration of "no more C; it's all Rust" as if that has somehow solved all problems and absolved the programmer of the responsibility for ensuring their code is otherwise safe and correct (granted, Rust does make this easier to do), and IMHO this just ends up doing a disservice both to those proclaimers and ultimately to Rust itself. To be clear, this is not a statement against Rust by any means but rather a complaint against the conduct of some of its practitioners.

With that being said, I feel I really also need to state here that I absolutely do not believe the above to be the case with the announcement here...Even to the contrary, I would say, as the name "Hubris" says it all. It's great to see Rust used in practice like this, and I look forward to seeing more of the details in the code itself!

scudd• 5 years ago

Has Oxide released any information on the price range of one of their machines? I assume if they're targeting mid-size enterprises it would be outside what I would consider buying for hobby use, but it would be sweet in the future if there was a mini-Oxide suitable for home labs.

floatboth6• 5 years ago

AFAIK they won't even sell individual machines, the product is a whole rack.

Since they aim to open source everything, there probably will be a way to use their management plane and stuff with a homelab eventually :)

oxidethrowaway6• 5 years ago

Oxide’s work is always interesting and basically a perfect confluence of all of my combined hardware and software experience to date.

However, I can’t quite get over their policy of paying everyone the same salary of $175,000. ( https://news.ycombinator.com/item?id=26348836 ) I’d love to apply and work on these things, but I wouldn’t love the idea of sacrificing $xxx,000 per year for the privilege of building someone else’s startup.

Does anyone know if they have some variability in equity compensation at least? I’m no stranger to taking significant compensation in startup equity, but it would have to be significant enough to make up for the significant comp reduction relative to just about every other employer in these domains.

voltagex_• 5 years ago

Do you know how much money $175kUSD is to a lot of people?

oxidethrowaway6• 5 years ago

Yes! Of course it is, but employment is a market like anything else.

The difference between $175K and market rate compensation (which may be significantly higher right now, considering the job market and the skills they’re asking for) is captured entirely by the founders and investors. We shouldn’t be shaming people for expecting a higher portion of the value they create in a competitive market like this.

But the fixed salary method creates a lot of secondary problems for a company: It can become a revolving door as people join for quick experience and resume-building, but then leave as soon as they can get a higher paying job somewhere else.

hinkley6• 5 years ago

Also it looks like they did a 3% inflation bump in the last month, as it now says $180,250 on their careers page.

Another 10% salary makes a much bigger difference when your finances are out of control. Once you are living below your means it moves your retirement date around. It's not enough to move your Fuck You, I'm Moving to Tuscany date by an appreciable amount.

4ad2• 5 years ago

It's a startup, you shouldn't compare it with FAANG, but to other startups, in which case I think it's competitive, is it not?

oxidethrowaway6• 5 years ago

In my first-hand experience interviewing at other startups (Q3 and Q4 2021) it’s definitely not competitive. Startups are unbelievably well-funded right now and capital is cheap and easy to come by. Even small startups don’t hesitate to compensate their employees well because they know it’s their only shot at attractive the type of talent that gets them to exit.

Basically, it doesn’t make sense for a startup to be frugal with compensation right now.

Unless you interpret it as their way of hiring only promising junior candidates or remote workers from locations where $175K is a lot of money?

Regardless, it’s weird to pay everyone the same amount of money because you’re basically pretending experience doesn’t matter. This leads to more experienced people leaving for other companies where they can (easily) get paid more while the less experienced people won’t leave because it’s a boost over what they’d get at other places.

I’ve worked at places with HR-mandated salary caps before. The best people always leave because there’s no hope of moving up and there’s no real incentive to work any harder than anyone else earning the same amount (as long as you avoid getting fired).

hinkley6• 5 years ago

I have some friends I'd be visiting in Cupertino if they had offered that much about 3 years ago, but they didn't. The real money was going to be based off of grants and getting a promotion back to their current level, and that was all too abstract for moving to the Valley, so they passed.

Worked out as their boss quit in the interim and now they're the boss.

cute_boi• 5 years ago

I think reference provide more info than above announcement itself:

https://hubris.oxide.computer/reference

Looks amazing imo. Waiting for github code :D

moondev• 5 years ago

https://github.com/oxidecomputer/hubris

luizfelberti• 5 years ago

Refreshing to see this seems tailored for RISC-V and ARM, rather than it being just another x86 OS. RISC is the future, and the future is exiting!

floatboth6• 5 years ago

It's an embedded project. It's for microcontrollers. It's the Cortex-M kind of ARM, not Cortex-A. x86 isn't even a consideration in this space.

mlindner3• 5 years ago

RISC vs CISC is a legacy of the past. All ISAs now are a bit of RISC and a bit of CSIC.

TD-Linux4• 5 years ago

I have an embedded real-time control project that is currently written in Rust, but runs with RTIC (https://rtic.rs/), a framework which is conceptually similar (no dynamic allocation of tasks or resources) but also has some differences. RTIC is more of a framework for locks and critical sections in an interrupt based program than a full fledged RTOS. Looking through the docs, here's the main differences (for my purposes) I see:

1. In Hubris, all interrupt handlers dispatch to a software task. In RTIC, you can dispatch to a software task, but you can also run the code directly in the interrupt handler. RTIC is reliant on Cortex-M's NVIC for preemption, whereas Hubris can preempt in software (assuming it is implemented). This does increase the minimum effective interrupt latency in Hubris, and if not very carefully implemented, the jitter also.

2. Hubris compiles each task separately and then pastes the binaries together, presumably with a fancy linker script. RTIC can have everything in one source file and builds everything into one LTO'd blob. I see the Hubris method as mostly a downside (unless you want to integrate binary blobs, for example), but it might have been needed for:

3. Hubris supports Cortex-M memory protection regions. This is pretty neat and something that is mostly out of scope for RTIC (being built around primitives that allow shared memory, trying to map into the very limited number of MPU regions would be difficult at best). Of course, it's Rust, so in theory you wouldn't need the MPU protections, but if you have to run any sort of untrusted code this is definitely the winner.

Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions. I'm quite interested to look at the implementation when the source code is released.

Edit: I forgot to mention the biggest difference, which is that because tasks have separate stacks in Hubris, you can do blocking waits. RTIC may support async in the future but for now you must manually construct state machines.

JulianMorrison4• 5 years ago

I don't think they link the binaries. It's more like, put them each on executable flash in separate places and the kernel just calls them.

The intent here seems to be that each binary has no need (and no ability) to get all up in another binary's business. Nothing shared, except access to the RTOS.

TD-Linux4• 5 years ago

It doesn't need to link any symbols, but I believe it does need to do relocations if the code isn't PIC, and to relocate the task's statically allocated RAM.

panick21_7• 5 years ago

Maybe they want it to be possible to have closed source and open source task to be mixed.

steveklabnik19• 5 years ago

In general we plan on making the system as open sourced as we possibly can, so there's no specific thought currently being put into closed source tasks. While it could work, the build system doesn't actually support that at all right now.

wyldfire3• 5 years ago

Your link does not appear to work, maybe this one [1] is intended instead? I can't resolve that name, at least.

[1] https://rtic.rs/

monocasa4• 5 years ago

Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions.

What I did in a similar kernel was dynamically map them from a larger table on faults, sort of like you would with a soft fill TLB. When you turn off the MPU in supervisor mode you get a sane 'map everything' mapping, leaving all 8 entries to user code.

The way LDM/STM restart after faults is amenable to this model on the M series cores.

TD-Linux4• 5 years ago

Neat, I didn't know that the MPU fault handler was complete enough to allow for restarts.

Now that the source is available, I took a look at what hubris does - it is not actually anything fancy, just a static list of up to 8 MPU regions per task [1].

It seems that leases aren't actually shared memory, but rather just grant permission for a memcpy-like syscall [2]. This is slightly better than plain message passing as the recipient gets to decide what memory it wants to access, but is still a memcpy.

[1] https://github.com/oxidecomputer/hubris/blob/8833cc1dcfdbf10...

[2] https://hubris.oxide.computer/reference/#_borrow_read_4

steveklabnik19• 5 years ago

Hey folks! The 404s are because we were planning on actually publishing this a bit later today, but it seems like folks noticed the CNAME entry. Happy to talk about it more, though obviously it'll be easier to see details once things are fully open.

EDIT: blog post is up: https://oxide.computer/blog/hubris-and-humility and the GitHub should be open.

EDIT 2: The HN story now points to this blog post, thanks mods!

cpach• 5 years ago

If anyone else wondered about the term BMC: https://www.servethehome.com/explaining-the-baseboard-manage...

mgaunard• 5 years ago

I feel like Rust is everywhere and nowhere at the same time; how do they do it?

mlindner3• 5 years ago

"They" is just a lot of software engineers who really really like it and want to be using it, but can't use it in their day job, so continue to talk and talk and talk about it in the hopes that it's used more. I'm one of them (not using it currently, want to use it).

pjmlp22• 5 years ago

Indeed, for example, one day Rust will be part of the default VS install, and then I can justify having native components for .NET stuff written in Rust, until then C++ will have to do.

https://docs.microsoft.com/en-us/windows/dev-environment/rus...

ncmncm3• 5 years ago

MLM, effectively.

The hype machine is pushing it into some places it is far from mature enough for. The hype depends more upon bad-mouthing other technologies than its merits can justify.

This is not to discount actual merits, or to doubt the needed maturity will come, in time. And, embedded is certainly a place it is already mature enough for.

wyldfire3• 5 years ago

I'm not familiar w/the details of the Cortex-Ms -- do any of them support SMT/multicore? Does Hubris have a scheduler which can support a multithreaded/core cpu?

floatboth6• 5 years ago

Definitely no SMT (lol). Multicore is quite rare in the deep embedded space, though FreeRTOS has added some SMP support now: https://www.freertos.org/2021/10/freertos-adds-reference-imp... — note that it's just now, that fact should already tell a lot.

And often multicore in embedded is not SMP, rather MCUs with multiple cores often end up running independent programs / OS instances on the different cores.

SSLy• 5 years ago

How do the independent cores mediate access to shared I/O on buses? (that includes RAM too)

wyldfire3• 5 years ago

I asked because I was considering porting to a DSP that has multiple threads, but it wouldn't make sense unless it had a scheduler that would work.

b200002• 5 years ago

when i started working on a recent realtime project i used linux, although i wanted to do bare metal. but that was not an option because of all the drivers necessary, and i knew i wanted to use the GPU and the cortex A processor i am using. i am still wondering if there really no solution to this situation.

qayxc• 5 years ago

Depending on what you need, you might end up wasting a ton of time reinventing the wheel if you go down the bare metal route.

Maybe a unikernel (if applicable) could be a good compromise?

b200002• 5 years ago

that is true, but fine tuning linux to get what you want is a huge investment in time as well.

i don’t need all the features of a complete OS. i only need a small set of features, and i want them to work in a specific way.

throwaway8943452• 5 years ago

instead of having an operating system that knows how to dynamically create tasks at run-time (itself a hallmark of multiprogrammed, general purpose systems), Cliff had designed Hubris to fully specify the tasks for a particular application at build time, with the build system then combining the kernel with the selected tasks to yield a single (attestable!) image.

I worked briefly at John Deere, and their home-grown operating system (called "JDOS", written in C) also baked every application into the system at compile time. This was my only embedded experience, but I assumed this was somewhat common for embedded operating systems?

stusmall• 5 years ago

It's been a long time since I've worked in that world but in the micro-controller world it is common.

elcritch3• 5 years ago

Yah it seems like they're conflating high level OS'es with low level RTOS'es. Perhaps comparing their OS to a general purpose OS while doing embedded / RTOS things. Take:

Precedence for the Hubris approach can be found in other systems like library operating systems, but there is an essential difference: Hubris is a memory-protected system, with tasks, the kernel, and drivers all in disjoint protection domains.

It's great that the drivers run in a different memory domains. Still I'm not convinced this isn't that different from many RTOS'es. Zephyr RTOS which can do memory protected tasks and defines each task's stack statically.

Matthias247• 5 years ago

Yeah, its common for RTOS. All the things in automotive (OSEK, Autosar) worked the same. And OS like Greenhills Integrity can also do full isolation between modules.

spenczar52• 5 years ago

I'd like to hear more about Oxide's development process. Was this designed on an index card, and then implemented? Or was it done with piles and piles of diagrams and documents before the first code was committed? Was it treated as a cool, out-there idea that's worth exploring, and then it gradually looked better and better?

It's hard to get software organizations to do ambitious things like this, and it's impressive that this was done on a relatively short timescale. I think the industry could learn a lot from how this was managed.

mbag• 5 years ago

It was probably done using RFDs (Requests for discussion),. You can read more on the process here [1].

But someone from Oxide would need to tell you exactly how many RFDs took to desing and implement Hubris.

[1] https://oxide.computer/blog/rfd-1-requests-for-discussion

steveklabnik19• 5 years ago

There was an RFD for Hubris; it laid out the basic concepts and design goals, as well as non-goals. But after that, it's largely just iterating. When I joined there were four or five people working on Hubris regularly; we have weekly meetings where we sync up, talking about what we're working on, and discuss things.

bcantrill8• 5 years ago

So, the Hubris repo itself will show a bunch of that history, but in particular, Cliff used the "sketch" nomenclature for the earliest ideas. I think in those first days, our thinking was that we were hitting a bunch of headwind on other approaches -- and had an increasingly concrete idea of what our own alternative might look like. I think whenever doing something new and bold, you want to give yourself some amount of time to at least get to an indicator that the new path is promising. For Hubris, this period was remarkably brief (small number of weeks), which is a tribute to the tremendous focus that Cliff had, but also how long some of the ideas had been germinating. Cliff also made the decision to do the earliest work on the STM32F407 Discovery; we knew that this wouldn't be the ultimate hardware that we would use for anything, but it is (or was!) readily attainable and just about everything about that board is known.

To summarize, it didn't take long to get to the point where Hubris was clearly the direction -- and a working artifact was really essential for that.

spenczar52• 5 years ago

Cool, thanks. This matches my experience with ambitious projects that actually succeed:

- Start with a really good engineer getting frustrated with existing stuff - but frustrated in a targeted way, and over a long period of time, not just a few weeks of grumpiness.

- Let them loose for a few weeks to sketch an alternative.

- Pause, and then the hardest part - smell whether this is going in the right direction. This just takes good taste!

- Make a decisive cut - either it's not working, or Let's Do It!

I can think of four or five ambitious projects I've been on or around that have really worked well, and they all seem to have worked in this way. I don't think I realized this clearly until this comment thread - thank you.

jbott• 5 years ago

How are these docs being built? I really like how these look and it looks to be asciidoc based, but I can't seem to find a build script for these.

ABS• 5 years ago

HTML says

so I guess https://asciidoctor.org/

steveklabnik19• 5 years ago

There's an open PR with the details, I set it to deploy every push to that PR for now so we could make quick fixes. It just runs asciidoctor, nothing fancy.

https://github.com/oxidecomputer/hubris/pull/272

(specifically https://github.com/oxidecomputer/hubris/pull/272/files#diff-... )

Twisol3• 5 years ago

As someone who's only worked with a prepared hardware kit (a dsPIC33F on an Explorer 16 that came with cables and the debugging puck), if I want to pick up the board they recommend in the blog post, do I need to make sure I get any other peripherals?

This all seems very cool, and I badly want to poke at embedded stuff again, but I have whatever the opposite of a green thumb is for hardware. Advice would be appreciated ^_^

jeffrallen• 5 years ago

Right there with you, dude. I just bought an H753ZI (1), hope it's enough, we'll see.

1: https://www.st.com/en/evaluation-tools/nucleo-h753zi.html

Twisol3• 5 years ago

For anyone still reading this a week on, I got the recommended board in, and I needed a micro-USB cable -- preferably two, since it has two sockets (one for power and one for, I assume, programming). Luckily, I already have a few of these hanging around for charging my wireless headphones.

It is completely beyond me, however, why one of the sockets is mounted upside down relative to the other one.

rossmohax2• 5 years ago

Their repo is a rare case which embraced git submodules. For some reason they generate a lot of friction and not used often.

steveklabnik19• 5 years ago

They are in fact a giant pain, but sometimes, still the best option.

rossmohax2• 5 years ago

Did you try https://github.com/ingydotnet/git-subrepo ? Looks like it vendors in other repositories, making submodules entirely transparent for consumers and still allowing sumbodule workflow for authors.

Animats3• 5 years ago

That's what bootable Modula I offered on the PDP-11, over 40 years ago.

bcantrill8• 5 years ago

This needs citation, but what does the "that" even refer to? I'm genuinely curious because there's little on it; did you use it? Did it survive? And what particular aspect of Hubris and Humility reminds you of this system?

Animats3• 5 years ago

Compile all the processes together, allocating all resources at compile time. Modula I had device register access, interrupt access, cooperative multitasking (async, the early years) and it worked moderately well on PDP-11 machines.

Yes, I did use it. We wrote an operating system in it at an aerospace company.[1] It didn't work out well. Sort of OK language, weak compiler, not enough memory given the poor code generation. It was just too early to be doing that back in 1978-1982. We got the thing running, and it was used for a classified high-security application, once.

[1] https://apps.dtic.mil/sti/pdfs/ADA111566.pdf

sydthrowaway3• 5 years ago

I don't get why they had to reinvent the wheel, just use libreboot or whatever.

steveklabnik19• 5 years ago

libreboot is a great project but it's only viable on specific x86 computers, and the chips we're using for the tasks Humility/Hubris is taking on are ARM.

sydthrowaway3• 5 years ago

What do Chromebooks use?

Hubris – A small operating system for deeply-embedded computer systems

Related Stories