Open Hardware Ethernet Switch project, part 1

https://serd.es/2025/05/08/Switch-project-pt1.html

by luu • 1 year ago

275 36 1 year ago

This is a great project I've been following along. My personal desire is to see this eventually support https://docs.kernel.org/networking/switchdev.html (although it will likely be someone other than Andrew that implements this) since mainstream switching where it works just like regular Linux networking would change our world for the better!

We need fewer proprietary interfaces for such an important part of networking.

Aurornis• 1 year ago

This is a seriously impressive project and it's been fun to follow along.

I almost hesitate to say anything, but the constant scope creep (that he readily admits in the blog) is starting to feel like a real obstacle to getting anything done. It's fun to see all of the theorizing about 96-port switches with $11K of FPGAs, but it would be fantastic if something like the simple 4-port switch design could be spun out as an open source project that people could iterate on.

I've also been following his solder-in probe projects for years ( https://www.antikernel.net/products.html ). There was a small Kickstarter and some reviews, but now the one distributor has 404s on the product pages and there are a lot of "coming soon" labels attached to all the different products and variations. Again, it would be fantastic if just one of those projects was available in distribution semi-reliably.

azonenberg11• 1 year ago

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand.

(EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that time was deciding that 2x 48 port switches with a bigger FPGA would be a more power efficient, easier to build, and generally superior design to 4x 24 port)

All that's left hardware wise is to design the main logic board with the FPGA, processor, and SFPs, plus the mechanical and thermal design of the 1U chassis. I've been focusing on firmware due to the tariff situation and wasn't planning on spinning the production-ish hardware for quite a while.

I don't plan any further scope changes now that I've e.g. already put in the production line card PCB order and mostly finalized the fabric architecture. I'm just holding off on the final logic board design until I'm sure about, for example, whether I'll want external packet buffer SRAM like LATENTPINK had or if I'll be fine with the KU5P's internal memory resourecs.

As far as the probes go, yes the antikernel.net site is way out of date and I've been too busy to refresh it.

* The AKL-PT1 did a kickstarter and I shipped probes to backers, but I stopped selling them because the fixed tip/ground spacing was too awkward and made it difficult to use.

* The AKL-PT2 (first gen solder in probe) was for sale for a while but extremely fragile, only 4 GHz bandwidth, and also had fixed signal/ground spacing.

* A few in-between designs were total flops and never reached the point of shipping a single unit

* The AKL-PT5 (second gen solder in probe) is >16 GHz bandwidth and has a flexible resistor based solder-in tip and a much nicer form factor. R&D is done but I ran into supply chain problems with the specialized resistor it needs (9 month lead time, poor yield, etc). I think I've mostly worked around those issues and I have a PVT run of a hundred units in the pipeline now. Fingers crossed they'll be hitting Digikey in the next 6-12 months.

jmcguckin• 1 year ago

If i wanted to do a motherboard that held 16 or 24 raspberry-pi compute modules, could i connect them together using the onboard gig ethernet without using any magnetics? Just by using the gige output from the module going to a commercial switch chip.

Bluecobra• 1 year ago

It might help to take a look at Arista/Metamako 10G L1 switches for some inspiration. These are 48x ports and use the FPGA for specific network applications (including switching). You might be able to find some of the older models on the cheap on eBay.

I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.

userbinator• 1 year ago

but it would be fantastic if something like the simple 4-port switch design could be spun out as an open source project that people could iterate on.

You can just get something like a Realtek single-chip switch IC and put it on a PCB. Low-port-count unmanaged switches are cheap commodity products already.

superb_dev2• 1 year ago

But not a cheap open source commodity

transpute• 1 year ago

For the other end of the wire, sometimes you can find NetFPGA academic research devices on eBay. The project has been running for 15 years, with 5 generations of NICs, https://netfpga.org/

A line-rate, flexible, and open platform for research, and classroom experimentation. More than 3,500 NetFPGA systems have been deployed at over 300 institutions in over 60 countries around the world.

zokier• 1 year ago

I wonder if the author considered at any point using Sparx-5 switch asic from Microchip? Those are available in single quantities for not too crazy price ($121 for 128G variant), and they are Linux based.

Of course I understand that having custom switch engine is far more satisfying to do.

azonenberg11• 1 year ago

This project dates back to circa 2013 when at the time, there was nothing available in that class without NDAs. Once I got set on the path of going custom I didn't want to back off from the challenge even if an easier path became available.

Also, I explicitly do not want to run embedded Linux. I much prefer bare metal on the control plane (I ended up writing a bare metal sshd because I couldn't find one that supported no-malloc, no-OS operation)

And one of the architectural plans of this project is a completely separate control/data plane where the processor can't see fabric packets and has a physically separate management interface.

minetest2048• 1 year ago

(I ended up writing a bare metal sshd because I couldn't find one that supported no-malloc, no-OS operation)

Is there a write up for this? This sounds interesting

Palomides• 1 year ago

I really, really would love to see someone make Sparx-5 switch. It's weird that I can't even find any commercially available/closed source switches with them (anyone know of any?).

kijiki• 1 year ago

There was this: https://www.servethehome.com/insane-48-port-2-5gbe-2x-25gbe-...

But I have not been able to find a place to buy one.

teleforce• 1 year ago

Yep, really strange indeed.

I posted the link for the ServeTheHome article last year back in 2024 although it's really year 2023 article [1].

Layer 3 switch for examples from Cisco or Juniper used to cost more than USD100K only ten years ago but now they can be dirt cheap with advent of the networking switch SoC like SparX-5.

High end SparX-5-200 (VSC7558) supports up to 200 Gbps of switch core bandwidth, with flexible interface options including 1/2.5/5/10/25 GbE, and the unit cost is only USD165 [2].

Imagine having this kind of really affordable powerful enterprise layer 3 switch with Linux Foundation SONiC or DENT OS [4], [5].

Someone just need to crowd source this SparX-5 SoC based layer 3 switch and will be selling like hot cakes.

[1] Managed Chinese E-Sports Cafe and Hotel Switch: 48-Port 2.5GbE 2x 25GbE 2x 10GbE:

https://news.ycombinator.com/item?id=40883384

[2] SparX-5:

https://ww1.microchip.com/downloads/en/DeviceDoc/SparX-5_sel...

[3] Sparx5:

https://microchip-ung.github.io/bsp-doc/bsp/2024.03/supporte...

[4] SONiC: Software for open networking in the cloud.

https://sonicfoundation.dev/

[5] DENT: SwitchDev based NOS:

https://dent.dev/

westurner2• 1 year ago

There are 48+2 port switches with OpenWRT support.

Re: initial specs for the (4 port) OpenWRT One, which is built on Banana Pi's, which supports U-boot: https://www.cnx-software.com/2024/01/12/openwrt-one-ap-24-xy... .. https://openwrt.org/toh/openwrt/one:

The non-open-source components include the 2.5GbE PHY and WiFi firmware with blobs running on separate cores that are independent of the main SoC where OpenWrt is running. The DRAM calibration routines are closed-source binaries as well.

Software for FPGA switch, probe, and GHz oscilloscope projects?

/? inurl:awesome vivado https://www.google.com/search?q=inurl%3Aawesome+vivado :

awesome-hdl: https://github.com/drom/awesome-hdl :

sphinx-hwt:

d3-wave probably won't do GHz in realtime. https://github.com/Nic30/d3-wave

Pyqtgraph probably can't realtime plot GHz probe data without resampling either?

pyqtgraph: https://github.com/pyqtgraph/pyqtgraph

The hwtLib README says Vivado supports IP-XACT format.

hwtLib: https://github.com/Nic30/hwtLib :

hwtLib is the library of hardware components writen using hwt library. Any component can be exported as Xilinx Vivado (IP-exact) or Quartus IPcore using IpPackager or as raw Verilog / VHDL / SystemC code and constraints by to_rtl() function. Target language is specified by keyword parameter serializer.

IP-XACT: https://en.wikipedia.org/wiki/IP-XACT

hwtlib docs > hwtLib.peripheral.ethernet package: https://hwtlib.readthedocs.io/en/latest/hwtLib.peripheral.et...

hwtLib.peripheral.uart package: https://hwtlib.readthedocs.io/en/latest/hwtLib.peripheral.ua...

It looks like there are CRC implementations in hwtlib. Which CRC or hash does U-boot use for firmware flashing? https://www.google.com/search?q=Which+CRC+or+hash+does+U-boo... ... Looks like CRC32 like .zip files but not .tar.gz files.

U-boot: https://github.com/u-boot/u-boot

OpenWRT docs > "Failsafe mode, factory reset, and recovery mode": https://openwrt.org/docs/guide-user/troubleshooting/failsafe...

Open vSwitch: https://en.wikipedia.org/wiki/Open_vSwitch :

Open vSwitch can operate both as a software-based network switch running within a virtual machine (VM) hypervisor, and as the control stack for dedicated switching hardware; as a result, it has been ported to multiple virtualization platforms, switching chipsets, and networking hardware accelerators.[7]

"Porting Open vSwitch to New Software or Hardware": https://docs.openvswitch.org/en/latest/topics/porting/

awesome-open-source-hardware: https://github.com/aolofsson/awesome-opensource-hardware

awesome-open-hardware: https://github.com/delftopenhardware/awesome-open-hardware :

Journal of Open Hardware (JOH), HardwareX Journal,

westurner2• 1 year ago

There are also xilinx (now AMD) FPGA modules in hwtlib:

hwtLib.xilinx package: https://hwtlib.readthedocs.io/en/latest/hwtLib.xilinx.html#

acyou• 1 year ago

"I was tipped off by a friend to a batch of Kintex UltraScale+’s, specifically the XCKU5P, on AliExpress for a mere $55 each. He had tested one from the seller and they appeared to be legitimate, although likely salvaged/reballed from some scrapped equipment."

Must be nice to know enough to be sure that your sketchy hardware isn't backdoored. I cannot imagine buying random network hardware from a sketchy source. Though I'm sure that if I had the author's chops, I wouldn't be too worried.

More broadly, network hardware is where the keyest vulnerabilities are. Open source network hardware is interesting from that perspective. You do not want random bad actor open source contributors adding backdoors.

luma2• 1 year ago

In this space, it's the closed source firmware blobs that nobody can inspect that I'd be a lot more worried about.

azonenberg11• 1 year ago

There are no blobs I'm aware of anywhere that are actually running in the system.

The STM32s have a small boot "ROM" burned into a write-protected region of flash but I have it jumpered so I boot from main user flash, not the bootloader.

I did a quick silicon overview of them (just out of curiosity... they came new from Digikey so I have no reason to believe they're fishy).

STM32H735: top metal only https://siliconpr0n.org/map/st/stm32h735/azonenberg_mz_mit20..., deeper dive planned but haven't had a chance

STM32L431: did a full teardown https://serd.es/2025/01/02/STM32L431-teardown.html

The 12-port PHYs are a fused-down version of a switch chip so there is a MIPS core on there, but to the best of my knowledge in the switch SKU it doesn't actually run (e.g. its RAM bus is NC and the IO power rail is grounded).

The FPGA is completely programmed from the bitstream I control. Tampering with a random resold FPGA to add some kind of backdoor is extraordinarily difficult and unlikely, it's the kind of thing you would see done as a targeted attack rather than just dumping FPGAs into the secondary market and hoping that a juicy intelligence target buys them rather than some guy tinkering around with open source projects.

And, if I ever get the slightest hint that there is something fishy about silicon I bought from a sketchy overseas source, I'm gonna take it into the lab at work and go to town. I do semiconductor reverse engineering at my day job and am the absolute last person anyone should try to sneak backdoored chips past. It's going to end up getting dissected inside a SEM and turn into an awesome talk at REcon or something.

While it would be slightly annoying to have my side projects disrupted, having a living breathing nation-state silicon backdoor show up in my lab would be a dream come true. I'd be ordering as many more chips as I could from the same seller and instrumenting it in every way possible, practicing on non-backdoored chips to make absolutely certain I didn't damage any of the spicy ones while studying the altered circuits, etc.

yjftsjthsd-h• 1 year ago

You do not want random bad actor open source contributors adding backdoors.

I think the proprietary network hardware companies have thoroughly demonstrated that it doesn't need to be open source to get back doors.

kristel100• 1 year ago

Love seeing open hardware get traction. Curious how they’re handling cooling and power draw—those are usually the dealbreakers in DIY networking gear.

azonenberg11• 1 year ago

The overall system will be powered by 48V DC on a 6-pin Molex Mini-Fit Jr connector, then stepped down to 12V by a 48->12V intermediate bus converter I've previously designed and characterized (https://serd.es/2024/10/15/Intermediate-bus-converter.html)

The 24-port line card has a 15W TDP overall (6.75W datasheet worst-case power consumption for each of the two 12-port PHYs, plus 1.5W added for power supply conversion losses and such). With my current bench setup (11/12 links up on PHY 0 and 2/12 on PHY 1, all linked up at gigabit but with PHY 1 not talking to the FPGA yet), they're happy but warm - 63.8 and 49.5C die temperature respectively, pulling a combined 7.173W (not counting power conversion losses) according to the internal sensors. PCB temperature is 37.2 to 49C at various measurement points.

The entire setup including the FPGA board, IBC, one of the two planned line cards, and some other glue components that won't be in the final switch is pulling 18W and change right now although the FPGA power consumption will go up as I build out more logic. I don't have measurements of just the FPGA's power consumption currently (I could put a current clamp on the 12V cable from the IBC I guess) but the PDU board does have I2C sensors that will measure consumption of the logic board and each line card separately; I just haven't written the firmware to read them yet. I also don't have FPGA temperature readings in the current firmware although last time I checked them via JTAG I had plenty of margin.

As of right now everything is happy with low-profile heatsinks (Wakefield-Vette 960-27-12-D-AB-0 on both the PHYs and FPGA) and passively cooled just sitting on my bench with no fans. I can go taller if increased power consumption or worse airflow in the future dictates, they're nowhere near the point of hitting the top of a 1U chassis.

My plan for the final system is to have air intakes somewhere on the sides and/or the front around the RJ45s and one or more fans exhausting out the back, details TBD based on more design and testing once I have better projections of the overall thermal load.

Very roughly my overall thermal/power budget is 15W per line card = 30W combined, plus no more than 20W for the logic board (probably quite a bit less) with all ports lit up and passing packets, for a total of 50W plus whatever the losses in the IBC are (it's roughly 94% efficient at this load based on previous testing). This is comfortably below the 72W output limit of the IBC, and should be very reasonable to reject from a 1U chassis with fairly basic air cooling, especially since it's not all coming from a single point load like a single-socket server.

bgnn2• 1 year ago

Wow, this is very impressive. This reminds me of my years of designing the analog front end of ethernet PHYs (5 generations of 1G base-T, 3 geberations of 2.5G/5G/10G base-T, 1 experinental 7.5G, the first gen 25G base-T, 2 genetation of 1G-baseT1 and the first generation of 10G base-T1 over single shielded pair for automotive). I got my hands dirty with board level testing and a lot of the firmeare!

I'm curious if there are any 10G base-T or 25G base-T PHYs out there that one can buy at lower quantities.

azonenberg11• 1 year ago

No idea, the high power consumption and latency of >1000baseT were such that I've never had any interest in anything newer in baseT land.

At home I run 10/100/1000baseT, 10Gbase-SR, 40Gbase-SR4, and am just beginning to prepare for 25Gbase-SR and 100Gbase-SR4 deployments (I have some NICs and optics fielded but no switching yet).

I've worked with 100baseT1 and 1000baseT1 for automotive projects at work and am familiar with the line coding, and have written a protocol decode for 100baseT1 among others, but I don't use them in my own projects.

When I last looked into it, which was admittedly several years ago, 10GbaseT PHYs were unobtainium in the "I want to buy just one, without an NDA" space. Your best option for putting 10Gbase-T on a DIY design is to put a SFP+ cage on there and slap in a baseT SFP+ module that somebody else designed.

bgnn2• 1 year ago

When I last looked into it, which was admittedly several years ago, 10GbaseT PHYs were unobtainium in the "I want to buy just one, without an NDA" space. Your best option for putting 10Gbase-T on a DIY design is to put a SFP+ cage on there and slap in a baseT SFP+ module that somebody else designed.

Indeed this seems to be the case. I hate this. There's no reason to make hardware inaccessible.

Open Hardware Ethernet Switch project, part 1

Related Stories