Python without an operating system

http://lwn.net/SubscriberLink/641244/5d1d6d20aeb0a647/

by Macha • 11 years ago

219 73 11 years ago

amyjess2• 11 years ago

Next step: Python as an operating system. Like Chrome OS or Firefox OS, just with Python instead of a web engine. I'm a big enough Python fan that the idea really intrigues me.

(also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)

Macha3• 11 years ago

That would be interesting. This isn't quite freestanding Python, in that it's running on Grub so there is a lower layer, but it's still cool despite that.

amyjess2• 11 years ago

Well, even Chrome OS and Firefox OS don't have their own kernels; they run on top of Linux.

If you were to build a Python OS, you'd probably start by writing PID 1 in Python. It would be a Python init, starting services written in Python, running gettys written in Python, and you'd log into a shell written in Python.

Hmm... I can actually see myself writing an init system in Python... (hell, I did exactly that at my last company.... I'd have to rewrite it from scratch, but that's no big deal...)

Great, I think I've found a new rabbit hole.

jacquesm2• 11 years ago

Every OS ever had a bootloader of some sort as far as I'm aware, in some cases you had to toggle them in through the front panel but I don't know of any OS that did not have a bootstrapping stage prepended.

mixmastamyk• 11 years ago

I usually forget by the time I get home

Send yourself email reminders.

seanp2k2• 11 years ago

I really like Wunderlist. In the past few months, they seem to have fixed their syncing issues, so mobile and desktop are pretty seamless now. I keep tons of ideas in Wunderlist, and the search works pretty well if I just brain-dump some keywords into the todo item.

I also made a thing to suck all the Github repos you star into Wunderlist with their descriptions, since I always want to find something I starred in the past, and I've found Github's search lacking in this regard: https://github.com/ip2k/wunder-star

e12e4• 11 years ago

Or just use "at"[1] and "mail" (assuming you have access to a computer that is connected and on after you leave for work):

    $ at 8pm <<END
    > echo "Don't forget to subscribe to lwn!" \
    >   | mail -s "LWN reminder" personal-mail@example.com
    > END

[1] http://debian-handbook.info/browse/stable/sect.task-scheduli...

striking• 11 years ago

Lisp Machines, except in 2015. The beautiful phoenix returns.

fzzzy• 11 years ago

Site's down now, but: https://mail.python.org/pipermail/python-announce-list/2004-...

voltagex_5• 11 years ago

Anyone want to take a stab at running it in QEMU?

http://uuu.cvs.sourceforge.net/viewvc/uuu/

Edit: http://dl.sourceforge.net/project/uuu/Existence/2003-09-21/e... should boot with qemu-system-i386

JoshTriplett12• 11 years ago

(also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)

If you work anywhere even remotely related to Open Source, I'd recommend getting your employer to get a corporate subscription. It's one of the single most valuable news sources I read; it summarizes LKML and numerous other first-hand sources, as well as many conferences, summits, and other in-person events.

friendzis• 11 years ago

Next step: Python on bare metal ARM targets. Imagine loading Python interpreter to an uC, quickly hacking together some scripts and happily running some DIY project. `sys` and especially `os` packages are out, maybe some high level features and something clever has to be done about interrupts, but that would be totally awesome

ul5255• 11 years ago

That is already available: https://micropython.org I have the PyBoard but one can get it running on ST Discovery boards as well.

pjc50• 11 years ago

Really would have to do something about the GIL in that case unless you want to be restricted to one core - or run several independent interpreters NUMA-style.

JoshTriplett12• 11 years ago

Yeah, the GIL is the biggest reason we haven't tried to run any Python code on non-boot CPUs; we just run tiny C functions that return information to the boot CPU.

mahouse• 11 years ago

Or a web engine made in Python.

https://en.wikipedia.org/wiki/HotJava

rootbear• 11 years ago

I seem to recall that the "Pi" in Raspberry Pi was originally "Py" and that the idea was that the system would boot straight into Python, similarly to how early personal computers booted into BASIC. This idea was later abandoned in favor of going with a full operating system.

duskwuff2• 11 years ago

There are a few real devices which boot up into interpreters, like the Espruino, which runs Javascript (!):

http://www.espruino.com/

collyw• 11 years ago

Ugggh. I love the idea, and its great that JavaScript is everywhere, by why does it have to be JavaScript?!?!

dec0dedab0de• 11 years ago

That is how the pyboard works, with MicroPython https://micropython.org

e12e4• 11 years ago

There's also https://micropython.org - for even more embedded/python on microcontrollers. Maybe one can run python on the network card rom too?

snops• 11 years ago

Maybe one can run python on the network card rom too? The network card ROM isn't microcontroller code, its just x86 code that runs on your main processor, called by the BIOS[1]. You could potentially hence have the OPs project run from the BootROM, if it would fit. It might be slightly tricky, as it runs as part of GRUB, which doesn't expect to find itself running from there.

Alternatively, the article mentions: > Environments like Mirage OS (and other "just enough operating systems") could also add Python using the BITS code without too much difficulty, he said.

You could therefore probably add this code to iPXE [2] without too much trouble to get python on your BootROM, though I'm unsure how much of the C standard library you would need to borrow or reimplement. By the looks of it, iPXE doesn't have threads or local storage support, so you would be pretty limited in usability without a lot of work.

[1] http://networkboot.org/fundamentals/ [2] http://ipxe.org

Macha3• 11 years ago

You could potentially hence have the OPs project run from the BootROM

Just pointing out as the OP that I'm not the author of the project. I just came across the article on r/python

e12e4• 11 years ago

Ah, of course. I should've said run python on your hard disk/cdrom... :-)

rjsw• 11 years ago

Mirage is built on top of the mini-os written in C that comes with the Xen sources.

de_Selby• 11 years ago

The presentation itself is on youtube[1], I'm glad I watched it, because I missed this sentence when reading the article at first:

The presentation and demos were all done in the BITS environment, so, in reality, the whole presentation was a demo, he said to a round of applause

Pretty impressive!

1 https://www.youtube.com/watch?v=bYQ_lq5dcvM

JoshTriplett12• 11 years ago

Thanks! That was definitely the most fun part of giving the talk; several people told me afterward that that was the point where the stuff I'd been talking about suddenly became real (accompanied by various degrees of "wat"), because it wasn't just talking to some abstract "hardware and firmware", it was drawing on the screen, and that made it real. And then following that up with Python-generated fractals really got people thinking about the old PEEK and POKE days.

My first programming environment was C under DOS, where you could create a pointer to video memory at 0xA0000000 or 0xC0000000 and start scribbling on the screen. I learned from a book ( http://smile.amazon.com/Microsoft-C-Programming-Robert-Lafor... ) which had an appendix showing how to draw the Mandelbrot set directly to video memory, so this was really nostalgic for me.

cmdrfred• 11 years ago

After all these years people keep finding new things to do with Python. Amazing. Maybe I can take a stab at writing my own python os.

rdtsc• 11 years ago

There is similar "Erlang as an OS" project

http://nerves-project.org/

It is used for embedded systems but can work on a larger system too I suppose.

Basically it is just a kernel (for drivers) and on top (init) is Erlang VM.

The interesting aspect is an Erlang process is similar to a modern OS process -- has an isolated heap. So it kind of maps elegantly to an safe embedded multi-process OS.

Find the Erlang Factory talk by Frank Hunleth for more details.

tlo• 11 years ago

Interesting, reminds me of HaLVM (Haskell on Xen) and Mirage OS (OCaml):

- https://github.com/galoisinc/halvm

- http://www.openmirage.org/

est• 11 years ago

Erlang on Xen is extremely cool.

voltagex_5• 11 years ago

Once TCP and hopefully "requests" support is in there I can see this competing with iPXE - load and boot images from HTTP.

leephillips• 11 years ago

Fortran predates the first examples of what we now think of as operating systems. After turning on the machine you fed in a deck of cards that comprised the Fortran compiler.

bane2• 11 years ago

You know, how awesome would it be if a RPi booted up directly into Python in the same way old 8-bits booted up into BASIC? This could really help it better achieve the goal of being an educational instrument for young students.

bduerst• 11 years ago

First thing I thought about - though if you're looking to do this with a Raspberry Pi as a purely educational tool, it may be easier just to create the pure-python tool as a layer on top of linux.

JoshTriplett12• 11 years ago

Depends on what you're trying to do; both are useful. Booting to Python under Linux gives you an exploratory environment with access to operating system services, but not necessarily one that lets you poke directly at hardware, unless you put Python in the kernel. (And even then, the kernel and drivers are doing things behind your back.) BITS gives you direct access to hardware and firmware, and it's intentionally single-tasking, running only what you tell it to.

So if you want a comfortable Python environment with more services and the ability to use the full power of Linux, you would want to boot to Linux and run Python (possibly with some extra modules); if you want raw access to hardware and firmware, but without any OS (with all the advantages and disadvantages that implies), you want BITS.

digi_owl• 11 years ago

Was that not what OLPC was about, before MS and other corporates got involved?

A small rugged laptop, booting Linux as the kernel, and a UI built entirely out of Python that the kids could modify at whim.

michaelmior2• 11 years ago

I'm curious how it would be more educational to boot directly into Python instead of logging in and starting a Python REPL.

bane2• 11 years ago

It's the same reason BASIC used to be burned into the system ROMs of old 8-bits instead of making you load it off of a disk.

Turn your question around, why would logging in and then starting Python be more educational than just booting directly into Python?

voltagex_5• 11 years ago

http://biosbits.org/ and http://biosbits.org/downloads/bits-1154.zip - no Git repo yet.

JoshTriplett12• 11 years ago

Working on that.

voltagex_5• 11 years ago

Thanks for your hard work on this project - it's really really cool.

chocolatebunny• 11 years ago

Holy cow. I've been looking for something like this. I'm currently trying to do board level testing on an embedded project with Python and the sysfs infrastructure in Linux. So far it's been decent but there are limitations to what I can do, such as testing all the ddr address lines.

JoshTriplett12• 11 years ago

Raw access to memory, MMIO, IO ports, registers, ACPI, and EFI should be pretty straightforward. If there's something you're trying to do that requires access to anything we don't expose in BITS, especially if it would require writing any new C code, please feel free to send us a patch, or just drop us a mail and we'll put it on the list to add.

nqzero• 11 years ago

python's a great operating system, all it lacks is a decent programming language

andrewchambers• 11 years ago

pretty sweet, could this make things like bootloader menus easier to implement?

JoshTriplett12• 11 years ago

It did, yeah. We're generating GRUB menus directly from Python, so that we can dynamically construct menus of tests or lists of firmware tables.

GRUB's own scripting language is mostly up to the task of enumerating OSes and displaying boot options, though.

JoshTriplett12• 11 years ago

Author and presenter here; happy to answer any questions people might have.

raverbashing• 11 years ago

Nice Work!

So maybe you can answer me this, how is memory management done in this environment? Are you free (heh) to do malloc and free or did you have to provide your own implementation? Is GRUB2 already in protected mode or there were some hacks needed on that area?

dlubarov• 11 years ago

Yeah, grub does enter protected mode for us, per the multiboot spec.

JoshTriplett12• 11 years ago

For memory management, GRUB2 fetches the memory map from the firmware (e820 under BIOS, BootServices::GetMemoryMap under EFI), and hands the usable memory regions to its memory allocator to allocate from. GRUB2's memory allocator then provides malloc, memalign, and free. (It also has some special functions for things like allocating low memory, needed for some system operations such as the CPU setup mentioned below.)

In the BIOS version, GRUB2 is entered in 16-bit real mode, and it transitions to 32-bit protected mode; it then has its own memory map. In the 32-bit and 64-bit EFI versions, the firmware transitions to 32-bit or 64-bit protected mode before calling GRUB2. (In all cases, there's no actual protection and the segment bases are all 0, though 64-bit mode does actually require a page table.)

However, GRUB2 normally only runs on the boot CPU ("BSP", bootstrap processor). We have to initialize the non-boot CPUs ("APs", application processors) ourselves. To do that, we allocate an aligned region of low memory, copy some bootstrap code and data into that memory, and send each processor an INIT (to reset them) and a startup inter-processor interrupt (SIPI) that forces them to jump to that address in low memory. We then transition them from 16-bit real mode to either 32-bit or 64-bit protected mode ourselves. (In the case of 64-bit protected mode, we set up paging by using the same page-table address that the firmware set up on the BSP, so that they all have the same memory map.) With that done, the AP can then run C code. Then we put the AP to sleep using mwait, waiting for work to do. When we want them to do something, we hand them a function pointer and a pointer-sized parameter.

pldrnt• 11 years ago

This is EXACTLY what I needed right now for a project. Does it already have a module to do the equivalent of an lspci? If not, where would I have to look to learn how to implement such a feature?

JoshTriplett12• 11 years ago

Note that PCI enumeration is a bit of an arcane art; if you want to know how to do it in practice on real systems with all their quirkiness, I'd suggest looking at how the Linux kernel enumerates PCI devices. (lspci in Linux userspace just walks the kernel-enumerated devices.) In particular, BITS doesn't have code to handle PCI-to-PCI bridges and enumerate the other side of them, or handle any quirks that crop up in real systems.

We have a very tiny Python pci module, which contains a brute-force bus-walk to find devices with a specific class; we're using that to find PCI-attached USB host controllers so that we can force them to "hand off" to the OS, which then causes the BIOS to stop polling them, which often causes the BIOS to stop doing frequent SMIs. (That allows us to tell if a BIOS's out-of-spec use of long-running SMI handlers is caused by USB handling.)

Simple PCI enumeration, though, is effectively just "walk every possible device, read the first four bytes of configuration space from function 0, if they're not 0xffffffff then they're vendor and product IDs, and then read the same from functions 1-7".

vigile_• 11 years ago

Does it mean that we gonna have a new smart generation of virus written in Python now? My my... ;-)

asgard1024• 11 years ago

These modern "GRUB + interpreter" attempts reminds me of CP/CMS. CP was a hypervisor and CMS was a small single-user operating system that runs on top of it (scriptable in REXX).

dingdingdang• 11 years ago

If this was implemented with direct access to GPU as well then a new wave of scene.org creations may arrive!

JoshTriplett12• 11 years ago

Take a look at the video: https://www.youtube.com/watch?v=bYQ_lq5dcvM&t=26m15s

We have direct access to video memory in graphics mode. Have fun.

phazmatis• 11 years ago

This seems to be a trend - python, before it erlang-on-metal. Is there a stable higher-level language runtime that runs on bare metal (that I can boot in a VM)? Or, better yet, something that compiles arbitrary LLVM code into .ISOs for deployment?

Python without an operating system

Related Stories