Rolling your own minimal embedded Linux for the Raspberry Pi

https://kevinboone.me/pi_minimal.html

by snwy • 5 years ago

161 60 5 years ago

kevinboone.me

ggm• 5 years ago

typo in early commands of the form mkdir -p staging boot (space should be /)

also, busybox doesn't demand symlinks: you can use hardlinks.

bitwize• 5 years ago

Systemd and dbus are standard parts of the Linux userland now, nearly as essential as the kernel itself as they provide vital services.

detaro• 5 years ago

Plenty embedded projects developed today don't use them (and functional desktop and server distros without them are around), "nearly as essential as the kernel" is really overstating it.

igetspam2• 5 years ago

We'll be headed down this path soon for some custom appliances. I can't find a compelling reason for systemd or dbus.

wryun• 5 years ago

Alternatively, you can use Void Linux and most of the work is done... admittedly, no read only fs by default.

sigjuice3• 5 years ago

A couple of suggestions for minimal systems for the Raspberry Pi:

https://buildroot.org build the whole Linux system from source

https://gokrazy.org pure-Go userland for your Raspberry Pi 3/4 appliances

01100011• 5 years ago

Don't forget Yocto(guide here: https://medium.com/nerd-for-tech/build-your-own-linux-image-...).

redleader55• 5 years ago

I think this should be the real tutorial here. The amount of flexibility that Yocto brings is huge compared to anything else you can do from any other distribution.

synergy203• 5 years ago

what's the usage of memory/storage comparing to buildroot for similar functionalities? I would expect something like 10x larger but I never measured it.

sigjuice3• 5 years ago

Sorry, I did not understand your question. What are the two things being compared?

Cyberdog• 5 years ago

Alternatively, if you're not stuck on Linux, just install OpenBSD or NetBSD. Both will try to install X by default, but you can easily disable this in the TUI installers. After a reboot you'll go straight to a shell with very few daemons running (and certainly no systemd). Install the packages you need and start the daemons you need from there.

Imagine having the entire output of `top` fit in a terminal window with empty space at the bottom.

seemaze• 5 years ago

I do just this with Alpine Linux. I've been running it on Raspberry Pi Zeros with great success, and less than 18MB read only ramdisk.

boring_twenties2• 5 years ago

I really wanted to like Alpine, but I can't for the life of me understand why it wants to do a ramdisk instead of just plain old read-only filesystem?

traceroute66• 5 years ago

Rolling your own minimal embedded Linux is all cute fun. But what about maintaining it ?

Assuming you're not going to be rebuilding the whole lot manually each time, you're also going to have to go through the rather cute experience of building (and maintaining) your own automation tooling around maintaining your minimal embedded Linux.

If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS.

Custom OS is fine if you've got a large team working along side you. But if its a one-man-band (or small team) then all the cuteness will soon evaporate.

If you think all Linux distros are too bloated, why not go BSD instead ?

For example, OpenBSD is famous for a minimal default install, and guess what, they maintain ready-to-go builds for Pine 64/64+, Raspberry Pi 3 and Raspberry Pi 4.[1]

[1] https://www.openbsd.org/arm64.html

PragmaticPulp2• 5 years ago

The Raspberry Pi is a learning platform, first and foremost. This is actually a great learning project for someone to understand what goes into a Linux system.

If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS

Using an off the shelf distribution is ideal your goal is rapid prototyping, but most real-world embedded projects end up requiring custom distributions sooner or later. Distributions tailored to interactive use tend not to be so great for distributing and upgrading predictable firmware. You really want all of your devices in a known state, not a random mix of package versions depending on when they last ran ‘apt-get upgrade’, for example.

In production-grade products, the OS is often built with tools like Buildroot or OpenEmbedded (Yocto), not from scratch. However, doing a tutorial like this once is a good learning exercise.

synergy203• 5 years ago

RPI also sells modules for commercial products, so anything you play with RPI can be used on products too.

johnklos2• 5 years ago

I've asked people how to easily download the sources for any given Linux distro to compile locally, and I've gotten links to sites with pages of instructions.

To build NetBSD/aarch64 on most Unix / Linux OSes (replace -j 8 with a suitable number for however many processor cores / threads you have):

  curl "https://cdn.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/src.tar.gz" -o src.tar.gz
  curl "https://cdn.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz" -o xsrc.tar.gz
  tar xzf src.tar.gz ; tar xzf xsrc.tar.gz
  cd src
  ./build.sh -m evbarm -a aarch64 -x -U -j 8 -D ../dest -O ../obj -T ../tools -R ../sets tools release

Is there really nothing this simple for Linux distros?

yjftsjthsd-h• 5 years ago

Maybe nix/guix, but most distros are going to have difficulty because they aren't a single thing to build, they're a collection of separate packages that you'd have to build individually.

1vuio0pswjnm73• 5 years ago

Cannot offer any advice to the parent^1 however the build.sh instructions he provides bring to mind a general observation: By comparison, running buildroot's script will initiate downloads of files. This is a key distinction I see with Linux/GNU distributions (cf. the Linux kernel) in general (there are exceptions). Linux distribution maintainers try to automate everything. The idea of the user downloading the source tree for the entire distribution manually is not contemplated. Another example: When NetBSD is first installed, generally no networking programs are enabled by default. Its expected the user will turn things on, if she wants them on. By contrast, when a popular Linux distribution is installed, there are usually networking programs enabled by default to listen on the network. I use both Linux and NetBSD. I am not invoking a Linux v BSD debate nor arguing for one's approach over the other's, I am just pointing out how they are philosophically different.

1 The "NetBSD way" would be to study the NetBSD source tree and then modify certain files to suit one's needs. For example, I like to create custom crunched binaries; this requires editing a list. By comparsion, with Linux, I have never read any advice to study the Debian source tree and then modify files to suit one's needs. Rather, the advice one finds directs readers to Yocto, buildroot, etc.

kbaker3• 5 years ago

Uh, with Yocto at least, it's also pretty simple for a basic bootable disk image/packages... downloading all the source and compiling the cross toolchain along the way...

    git clone git://git.yoctoproject.org/poky
    cd poky
    git checkout -b hardknott -t origin/hardknott
    source oe-init-build-env
    MACHINE=qemuarm64 bitbake core-image-sato

But after the first build you may want more - bringing in other functional layers for different features (maybe you want docker?), support for different boards (same OS distro for x86_64 and ARM, or your own custom board... plus it can build a whole SDK for you for local development/debugging?), customization of target files, pulling from updated upstreams, support in deployment and throughout the device lifecycle, custom package feeds, etc.

There's a lot going for Yocto, it has a deep learning curve but does a pretty good job of taming the complexity of managing an entire OS distro.

forty2• 5 years ago

I would use buildroot for that. It's very convenient even to just build random kernels for random targets

moondev• 5 years ago

esxi for arm edition runs on the rpi 4 and 8 gb models. This way you can use the vsphere tooling and api to make it pretty painless, including the use of snapshots and cloning

iJohnDoe• 5 years ago

Wow! Had no idea that VMware would run on a rpi. Thanks.

It’s interesting to think about the different markets for the rpi. I would never consider the rpi for learning. I only look at it as an inexpensive system to run my favorite distro. I want it to run CentOS or NetBSD, etc. rpi OS is meaningless to me. I want it to run the stuff I use every day.

spyremeown• 5 years ago

If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS.

And that's why you're probably not working on embedded. Minimal environments with almost zero footprint is a must in almost all cases. If you're rolling a custom board, you simply have to go custom, which is where OpenBSD will fall short on you. Check Variscite's or Toradex websites and see if you find any BSD in there.

foxfluff• 5 years ago

The other thing is that existing distros are rarely good for anything truly autonomous. Too many moving parts. There's always the expectation that someone with a display and a keyboard, or a sysadmin at the console, is ready to pick up the pieces when it falls apart.

LeoPanthera• 5 years ago

It claims there's no supported way to boot from a read-only filesystem with Raspberry Pi OS, but there is. Setting it up is built into raspi-config, at:

Performance Options > Overlay File System

PragmaticPulp2• 5 years ago

This is a great series of tutorials to follow if you want to gain a better understanding of what goes into a basic Linux system. I think some of the commenters here are missing the point that it’s a learning exercise.

For production-grade custom Linux firmware, two of the most popular options are Buildroot and OpenEmbedded (Yocto). I’d recommend that anyone interested in this space get started with Buildroot. It’s really easy to make a basic image and customize it to your needs, plus you’ll learn a lot along the way. Yocto is much more powerful and customizable, but the learning curve is far more difficult. It’s great for more advanced situations where you need to support multiple boards or bring in 3rd-party layers.

codetrotter• 5 years ago

I second that, Buildroot is great

unmole• 5 years ago

PTXdist is also pretty good and very similar to Buildroot. My experience is a bit dated but PTXdist had better support for some boards than Buildroot.

montecarl• 5 years ago

I have built several embedded linux distributions for raspberry pi using buildroot. I have had great luck with it. There is a bit of a learning curve, but that seems to be the case with most embedded devices. I would recommend anyone getting started to try it out.

elcritch• 5 years ago

My favorite is using Nerves system that uses buildroot and boots directly to Elixir. Buildroot can be a bit of a pain but it’s fun to get a 20 mb Linux image with a few commands.

TickleSteve• 5 years ago

I would narrow that recommendation to just BuildRoot TBH. It fits much more with the kernel-config style whereas Yocto config seems like its from another planet.

BuildRoot is much easier to work with IMO and gives equivalent or better results.

Sponge5• 5 years ago

Don't know when's the last time you've played with Yocto, but at this point all you need is to learn how to use devtool and that automates everything for you.

willjp• 5 years ago

This is really interesting! Thank you so much!

sydthrowaway• 5 years ago

This is interesting. I have always worked with baremetal systems. Are there any good courses for understanding modern Embedded Linux standards who for shops who end up using a COTS dev board?

stonecharioteer• 5 years ago

Does anyone who's read this want to weigh in on how it holds up against Linux From Scratch?

teleforce• 5 years ago

Not on Linux distribution but this Arm sponsored textbook 'Operating Systems Foundations with Linux on the Raspberry Pi' is a useful learning guide:

https://www.arm.com/resources/education/books/operating-syst...

scns• 5 years ago

Looks good, sponsored usually you get it for free though.

rvz• 5 years ago

Rolling your own Embedded Linux on a device like the Pi is all fun and games for hobbyists until you run into the most typical security vulnerabilities and maintenance hell which comes from being unable to update it to the latest firmware.

Maybe we will see improvements in this space once other secure alternative operating systems and kernels designed for running on embedded systems are available.

MrBuddyCasino• 5 years ago

If I wanted to build a RPi based system that has good support for bluetooth, audio (ALSA) and wifi, with fast boot times, what would be the most sensible route? Alpine or Buildroot? What are the advantages and drawbacks of them?

mro_name• 5 years ago

this is a remarkably well designed website, thanks for sharing.

davemtl• 5 years ago

Maybe it's just me, but I get forwarded to an advert for iFastNet and the domain gets forwarded to suspended-website.com.

leowbattle• 5 years ago

I got that problem too.

niel• 5 years ago

I suspect the flood of visitors from HN exceeded the hosting account's quota.

mcshicks• 5 years ago

https://web.archive.org/web/20210901081906/https://kevinboon...

cf100clunk• 5 years ago

Link to kevinboone.me now redirects to suspended-website.com which then cycles through crap links.

dengolius• 5 years ago

omg

link is not work, domain is for sale...

Rolling your own minimal embedded Linux for the Raspberry Pi

Related Stories