So you want to build an embedded Linux system?

https://jaycarlson.net/embedded-linux/

by olalonde • 2 years ago

127 40 2 years ago

jaycarlson.net

vegadw• 2 years ago

[2020]

asicsp• 2 years ago

Discussion from 2020: https://news.ycombinator.com/item?id=24800037

Also, another in 2022: https://news.ycombinator.com/item?id=30273301

manuel_w• 2 years ago

Since an embedded Linux system will likely require use of either Buildroot or Yocto, I'd like to ask the following slightly off-topic question:

Which SoM product line would you say has the best vendor support? I'm talking about quality of the BSP (board support package). (A meta-layer, in Yocto terms.)

Raspberry Pi has quite a community behind it, so meta-raspberrypi has a number of contributors. (None of which payed by the Raspberry Pi foundation, though.)

My latest project involved a NVidia Jetson SoM, and I was surprised to learn that the BSP doesn't see any support from NVidia at all. They rely on one motivated guy (Matt Madison) maintaining it in his free time.

I'd love to see a vendor that provides a first-class BSP maintained by their own people.

dgfitz• 2 years ago

One that comes to mind: https://github.com/nxp-imx/meta-imx

I have done quite a bit of yocto work at $dayjob, and having a vendor-supported upstream BSP is (usually) really pleasant to work with.

fullstop• 2 years ago

Yes, NXP's Yocto support has been fantastic.

nrclark2• 2 years ago

Agreed, NXP does a good job with their yocto layer.

candiddevmike2• 2 years ago

Since an embedded Linux system will likely require use of either Buildroot or Yocto,

I don't know if that's the case anymore. The kernel ships with a lot of drivers now, and Root FS for embedded devices aren't really space constrained anymore. The needs of an IoT device have also grown to include things like containers and telemetry. In my experience, forking off of an existing distribution like Debian is way easier than navigating Buildroot or Yocto's documentation and cruft, and results in faster builds.

I got so fed up with those tools I eventually built a custom tool (Etcha) to build my own meta distribution (EtchaOS) of immutable variants for Debian, Fedora, etc. It owns the entire provisioning pipeline and builds multi-arch images in less than 10 minutes:

https://etcha.dev/etchaos/explanations/build-process/

nrclark2• 2 years ago

unrelated but curious - what do you think of OSTree-based embedded distros? Any future in it?

rkangel• 2 years ago

For the projects I've done, we're not usually particularly space constrained, but we are usually at least one of:

* Cost constrained so RAM constrained * Need fast boot times * For a niche processor architecture

Starting with very little apart from an init system and busybox and then building up from there has been a good approach to get the performant (by whatever metric) platform that I need.

That said, I'm not a fan of Yocto at all. I find it painful and slow to work with. I experimented with using Nix (as in NixOS) as my build platform which had some great advantages (e.g. great caching of build steps), but also some pain.

throwway120385• 2 years ago

I've been getting a lot of traction over the past 10 years with Debian multistrap and just the kernel from the BSP. I've never needed Buildroot or Yocto.

AlotOfReading• 2 years ago

NXP is probably the best "big" vendor for this. The main issue is that their documentation solution/website is awful.

I think a smaller vendor like Toradex is more in line with what you're asking for though. I haven't used them myself, but I've heard very good things. First class yocto support is a core part of their offering.

m-ee• 2 years ago

I’ve been happy with toradex. Boundary is another similar company but haven’t used them.

phlipski3• 2 years ago

The Toradex's, Phytec's, Boundary Devices, etc... of the world all do a decent job with their BSP's but just know that they pull from NXP's mainline BSP release so there is a lag in time from when NXP makes a new release. Sometimes they'll fork their own libraries which I've seen cause some problems with application level software from NXP.

blueflow• 2 years ago

RPi has a nice community, but some of its drivers still set TAINT_CRAP on Linux.

cbsks• 2 years ago

I recently got an Nvidia Orin Nano to play with and found Nvidia’s BSP to be very well documented and easy to use… but I haven’t looked into using Buildroot or Yocto directly. https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide...

aquilum• 2 years ago

Disclosure: Raspberry Pi employee, focussed on better tooling for people building products with Raspberry Pi devices.

Raspberry Pi provide pi-gen (https://github.com/RPi-Distro/pi-gen), which allows you to build a customised Debian-based OS that uses our package list to get updates as we release them, with a low impedance mismatch between developing your software on Raspberry Pi OS and your deployment platform. Since it's debian based, your resultant system winds up being _very_ familiar.

We also provide tooling that compliments pi-gen. For example, rpi-sb-provisioner (https://github.com/raspberrypi/rpi-sb-provisioner), which will happily ingest an OS image created by pi-gen and automate the configuration of signed boot with an encrypted rootfs on CM4 devices. I've used it to flash half a dozen Raspberry Pi devices in rapid form as part of testing, and it runs on our own hardware.

I'd love to see what people make of these tools, and where we could expand support / docs / ascii-art. I don't check HN very often - but I'm pretty responsive to Github issues.

ehaskins• 2 years ago

I have current products using raspberrypi and i.MX parts, and have worked with teams using TI and Renesas parts. Some fully custom boards, some SoM based.

Their BSPs all suck to different degrees, but you can work around the issue with enough effort. If I had to choose ignoring cost, I'd pick an i.MX part, but they're priced for the industrial consumer where price isn't a big concern.

Stay away from any asian parts (Allwinner) or phone parts since they seem to have no vendor or community support.

Like others have said, some of the SoM vendors do a pretty decent job of hiding the evil, but that often comes with very opinionated BSP layers which limit your product design. We've had to ignore/replace all of our SoM vendor's layers, borrowing the few useful parts as needed.

cjk• 2 years ago

FWIW, some of the Allwinner stuff has fairly robust support in mainline Linux, and there are ongoing efforts to upstream missing bits: https://linux-sunxi.org/Linux_mainlining_effort

No vendor support is probably accurate, though.

graymatters• 2 years ago

Avoid nvidia if you can. Their attitude towards anything Open Source has been extremely unsupportive if not outright shitty. Part of company philosophy.

phlipski3• 2 years ago

Yocto is overkill for a hobbyist. It's designed to help big companies build linux distributions for multiple platforms. It works great for companies like NXP that have the resources to devote to it, but it's a f&$cking bear to wrap your hands around if you just want to build linux for a single dev-board.

Buildroot is much smaller/simpler for hobbyists IMO.

NVidia's Jetson lineup is really a double edged (disingenuous IMO) sword. They provide some nice GPU/NPU power for AI stuff and a decent price and is great to play/learn with. However if you want to take a jetson to production you can't. They don't sell the bare SOC's and the modules have limits to how many you can buy. That jetson lineup is probably a 0.05% pimple on their balance sheet...

dilawar• 2 years ago

Excellent blog! Love the fact that a great amount of background information is added to the article. Can be read as an "introduction to embedded programming" chapter but not as boring as you find in textbooks.

hejira• 2 years ago

Agreed!

amelius2• 2 years ago

Embedded systems question. I have an nVidia Jetson, and I want to expose one of the USB3 ports to the user of the system (e.g. so they can attach a USB drive). However ... the USB3 port of the Jetson is not near the enclosure of the box. Now I'm thinking of using a USB3 cable, but I know from experience that this is not going to give a reliable connection. Any suggestions on how to properly do this? I don't want to move the Jetson, in part because I am planning to use the other exposed ports internally.

spaceywilly• 2 years ago

Can you check if there is another USB port exposed via a pin header? That is probably the easiest way if it is available. Otherwise, I think your best best is to solder wires onto the same pads used by the existing USB port. I wouldn’t try to desolder the port that is there, that could cause more problems. But if you can find a location on the board to tap into that USB port’s signals, you could solder wires to run another usb port using the same connections.

amelius2• 2 years ago

I don't think any USB3 ports are exposed on the headers.

But I also have to wonder how that would improve the situation over running a cable between the actual USB3 port and a socket mounted in the enclosure.

ab71e53• 2 years ago

Question for anyone who has done something similar : how feasible would the hardware part be for someone who has some basic electronic / breadboarding knowledge but no pcb design experience? Besides this excellent article what would be a good resource to learn to do something like this myself?

beryilma2• 2 years ago

If you want to learn about the mechanics of designing PCBs, install KiCad (https://www.kicad.org/) and go through these tutorials: https://docs.kicad.org/

ab71e53• 2 years ago

Thanks! Yes I should probably start with something simpler, I have a tendency to want to skip ahead to the good stuff

dragontamer3• 2 years ago

This is advanced level PCB design.

You need to know proper grounding and differential pairs and length matching to accomplish what is listed in this article.

These are very difficult topics, but this 2 hour video should give you an idea of what's going on here: https://www.youtube.com/live/ySuUZEjARPY?si=xfpg8XdjtXa1Uixn

--------

If this is too much for you, then instead stick to microcontrollers and other sub-100Mhz or sub-20Mhz projects.

But 500MHz and above that requires length matching with controlled impedances is waaaayyyyy outside the scope for a beginner designer.

---------

Beginner design is easier at 20MHz or so. The faster your signals, the more susceptible to noise they are. Those 500MHz+ Linux MPUs may fail to boot or fail spuriously if you make grounding / noise / EMC errors.

And it's pretty difficult to explain to a beginner what is going on physically. The PCB board itself needs to be seen as parasitic capacitors, inductors and resistors (especially around Vias, between layers and other elements).

But these 500Mhz or so processors here in this blogpost are still easier than Ghz+ chips.

------

The example I always like to point out: 2.4GHz wavelength is just a couple of cm long. That means any trace of ~centimeters can accidentally turn into a Bluetooth / WiFi antenna, which would inject a ton of noise into your circuit (or conversely: 'Throw' your electrons out like a Bluetooth / WiFi antenna).

That's why faster speeds are harder. 900Mhz and slower needs an antenna over 30cm long, so staying below this means every PCB trace is too short to be an effective antenna.

Going even slower at 20Mhz means many meters before you accidentally make an antenna.

-------

I'd summarize that high speed circuit design is 60%+ just making sure you didn't accidentally make antennas in the frequencies you're working with. And the other 40% is decoupling / turning the PCB into a distributed physical capacitor.

dragontamer3• 2 years ago

That's why faster speeds are harder. 900Mhz and slower needs an antenna over 30cm long, so staying below this means every PCB trace is too short to be an effective antenna.

Oh I forgot to mention, the issue is the 5th or 7th harmonic of your digital signals.

So... a 20Mhz digital circuit will realistically have 10ns rise/fall times or aka have ACTUALLY 100Mhz information in your electricity.

Similarly, a 100MHz clock digital circuit will need rise/fall times of 2ns. That's 500Mhz information inside of your signals (2ns == 500MHz).

Following the pattern: 500Mhz digital clock will easily have 2.5Ghz+ information in them to keep those digital signals square. Aka rise / fall times of 0.4ns.

-----------

If you've accidentally built a 2.5 Ghz antenna twice in your circuit, guess what? That 0.4ns rise time square wave is going to start teleporting between your PCB antennas.

Electricity at high frequencies moves in very complex fashions. In ways that beginners cannot comprehend.

We call it crosstalk, EMC and other ways they move around. But it's all antenna theory, EMI fields and the like. I can't say I'm a high wizard of this magic but... Hopefully I've got it close enough for you to understand the issues at play here.

It's best to just slow down to 20MHz where electricity still makes sense.

davidhyde• 2 years ago

I found this to be a great resource when I was learning PCB design: https://www.youtube.com/@PhilsLab

I watched all his KiCad videos and tried to build them as I went along. Then you can use a cheap PCB manufacturer like JLCPCB to get them made. It's a lot cheaper than you think (tens of dollars delivered in two weeks for simple stuff). There is a lot of know-how to learn in this field but I found it to be very accessible as a SWE and it all made sense. It just took a long time to pick it all up.

phlipski3• 2 years ago

If you're looking to build a product using an embedded linux system you're better off just using an off the shelf SOM from an embedded board vendor like Toradex or Phytec (to name 2 - there are dozens out there). There are some quasi standards around compute modules now like SMARC and SODIMM where the vendors build little pcb's that contain the SOC, DRAM, PMIC and optional EMMC and WIFI. Then you just built a carrier board to breakout the IO like USB, UART, HDMI, MIPI DSI/CSI-2, etc.... The design/layout/validation of the SOC/PMIC/DRAM is the hardest part and unless you're shipping in volumes > 100ku annually you're probably better off outsourcing the SOM.

If you're just looking to learn then by all means give it a shot! Jay's article is a great primer in the basics but just know that all of those parts are pretty slow/old by today's standards. That article is 4 years old and even 4 years ago those parts were fairly slow/cheap/low-power/low-pincount relative to the SOT at the time.

AtlasBarfed• 2 years ago

Isn't the punchline in most of these articles always "Hey so you want to get paid 50% of other people in IT in order to do this field that is 200% harder?"

fleventynine• 2 years ago

Because of this mismatch there aren't many young folks with these skills; expect these salaries to flip around once most of the experts retire.

So you want to build an embedded Linux system?

Related Stories