Network offloading in OpenWrt (2018) [pdf]

https://openwrtsummit.files.wordpress.com/2018/11/hauke-mehrtens.pdf

by workrockin • 7 years ago

157 37 7 years ago

openwrtsummit.files.wordpress.com

cbluth• 7 years ago

Is there a video of the presentation?

workrockin5• 7 years ago

Not of this particular presentation. But there are quite a few other videos talking about openwrt on the website (you'll find them under previous summits menu)

http://openwrtsummit.org/

dpwm2• 7 years ago

Having just been the victim of the Intel I219 on linux, and a heavy user of OpenWRT, my initial response when I saw the logo was pure horror.

For those that don't know – as I didn't – the hardware behind Intel's more recent ethernet chipsets seem only to be tested and supported on the most basic of networking configurations. As soon as you add something like a macvlan, it is apparently normal to expect problems like dropped packets and module unloading every few GB. One of the solutions seems to be to disable offloading [0], but for me that just bought me an order of magnitude more time before the hang.

However, this does look like good work – and I have to celebrate the contribution from the devs on this one. There are huge gains to be had by network offloading – and perhaps in time Intel's hardware division will iron out the problems with their chipsets.

[0] https://sourceforge.net/p/e1000/bugs/571/

voltagex_• 7 years ago

That bug is from 2017 on an older driver. Have you tried the same thing with igb on a recent kernel?

    $ ethtool -i enp0s20f1
    driver: igb
    version: 5.4.0-k
    firmware-version: 0.0.0
    expansion-rom-version:
    bus-info: 0000:00:14.1
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

dpwm2• 7 years ago

No. If it was my own machine I'd certainly try it out – but it's work's machine running Debian stretch, so we switched to a difference NIC with a Realtek interface which has given no problems and meets our performance requirements.

To be clear, this appears to be a problem with the newness of the chipset and the time it takes to get drivers into upstream – the chipset we switched to had its fair share of teething problems when it first came out.

Intel had a long-standing reputation for rock-solid network cards with great open-source drivers ready the day of release – a reputation Realtek has never really had.

philjohn3• 7 years ago

There's a push at the moment to get the Qualcomm QSDK hardware offloading ported to modern kernels for IPQ8064 (Dual Core Arm SoC with an additional 2 Packet Processing cores) - sadly, Qualcomm haven't kept pace with upstream Kernel dev so it's a mammoth task.

And that's the crux of the problem with these SoC's - they can have fantastic packet processing performance (and in the case of the IPQ8064 NSS cores accelerated PPPoE, qdisc and crypto) but it's all tied up in vendor only repos, doesn't track upstream and so you're stuck with buggy vendor firmware.

oneowl• 7 years ago

I was reading QSDK page [1] and it seems to me that they are doing something different from openwrt.

The QCA Software Development Kit (QSDK) project allows users to build an OpenWrt based platform containing additional enhancements for Qualcomm Atheros chipsets that have not yet made it into the public OpenWrt repository.

I'm not aware of the project goals so I may be wrong.

[1] https://wiki.codeaurora.org/xwiki/bin/QSDK/

philjohn3• 7 years ago

Sort of - QSDK uses an old version of OpenWRT and a 3.x Linux Kernel to allow board partners (e.g. Netgear et al) to use their reference designs and spin up a working home router firmware quickly and easily.

An awful lot of devices these days ship with firmware that is actually OpenWRT (often v10-v15) based.

The actual NSS kernel modules have source available, and this is pulled into QSDK OpenWRT builds, but they've not had much luck getting stuff upstreamed[1] and getting them working on a recent 4.x kernel is non trivial.

This was also before the netfilter flow offloading framework, so the work is further compounded because they used their own offloading system.

[1] https://lkml.org/lkml/2015/1/8/534

neilv4• 7 years ago

This topic of hardware offloading is interesting, and, for some people's requirements, is there an open source consideration to keep in mind: are you offloading to effectively "closed source"?

I'm a long-time OpenWrt user on various reflashed SOHO WiFi router SoC boxes, and I recently want to move my home router(s) to somewhat more open PC hardware (with Pfsense, OpenWrt, or DIY atop a GNU/Linux distro), and obtained some Intel gigabit NICs for this purpose.

Since I want to use fanless PC hardware (although even the first Intel card alone runs quite warm at idle), I'm wondering how the performance will be, running all packets through Linux+CPU+I/O, compared to whatever hardware optimizations OpenWrt is doing on the SoC boxes. If the performance on PC hardware falls short, offloading could make the difference, but that would be starting to look more like the closed SoC I was going to a lot of effort to move away from.

(Ultimately, I would want to do this on a fully-open RISC-V board, with open hardware NICs, open hardware AES, etc., but I don't know that will ever be viable. For now, PC hardware, as open and trustworthy as I can get it, might be closest.)

Fnoord2• 7 years ago

The two advancements I'm following in the router field (which do not fit your requirements though) are 1) Router 7 [a router implementation fully written in Go] [1] 2) Turris MOX [a modular, extensible router] [2].

You could run Router 7 on PC Engines [3] who at least use Coreboot.

[1] https://github.com/rtr7/router7

[2] https://www.turris.cz/en/overview/

[3] https://www.pcengines.ch/

neilv4• 7 years ago

These are great links. Router in Go is impressive (and making me wonder whether one could do the same in Racket/Scheme, and work around GC). I like to see parties acquiring/regaining the ability to make their own hardware. The PC Engines apu2 looks tempting, and a couple more NICs would be even better, for an all-in-one device supporting more LANs (that are a bit more compartmentalized), and avoiding the need for separate Ethernet switches/hubs in some cases.

workrockin5• 7 years ago

I just found out about Turris Omnia. Its not open source but its modular like a PC. But its a router. I know it does not make sense so please take a look yourself

https://www.turris.cz/en/overview/

They also launched an indiegogo campaign last year that you can check out here

https://www.indiegogo.com/projects/turris-mox-modular-open-s...

def_true_false3• 7 years ago

Omnia is their previous router, not the modular one.

zamadatix• 7 years ago

If open is what you are interested in try OPNsense before pfSense.

I'd recommend starting off with keeping your LAN on a dumb pure L2 switch (no management interface or anything) with your box acting as the upstream swouterwall. It's a lot easier to start with that side of things and see how much load it generates there than it is to throw the whole thing at it and troubleshoot what parts are making it run slow.

nominated12• 7 years ago

The MediaTek MT7621 SoC has support for hardware NAT in OpenWrt. The driver is being developed by Felix Fietkau of ath9k fame who has stated that it’s the most open (blob free) 5Ghz chip on the market. I’ve been using an mt7621 device since before the Lede split and the progress has been amazing. If you’re reading this Felix, thank you!!

I haven’t been following the development branch but the current stable branch says “Experimental feature. Not fully compatible with QoS/SQM”. For this reason I haven’t enabled it.

EDIT: My decision to go with the mt7621 came from watching the rant by Felix on wireless drivers. https://youtu.be/hiUosbhR0Wo

neilv4• 7 years ago

Is this "hardware NAT" something that's closed in the hardware/firmware, or is it specialized hardware that is programmable with open source to implement NAT in some kind of accelerated way?

(I don't care as much as the FSF does, about where closed behavior is represented -- downloadable blob from host, on-device storage, or burnt into hardware. What's more important to me is getting more of the behavior represented in open source/hardware.)

(BTW, I love the Linux ath9k work. Besides various routers and PCIe WiFi cards, I have a stockpile of Corebooted ThinkPads in which I've replaced the mini-PCIe cards despite the original whitelisting.)

baybal2• 7 years ago

We once tried to develop a repeater at 7628, and it turned out to be a disaster. Mediatek's platform support just does not exist for anybody buying less than 1m chips.

Everybody else are footballed to regional partners who can't do a thing. Their official BSP is a fork of 2.4... No userspace tools can work with their drivers that essentially run their own IP and MAC stack. But even their own tools are not sufficient to config things, communications with them often went like "put byte A1 into secret register B and arm watchdog beforehand just in case it crashes."

devereaux• 7 years ago

The UniElec U7621-06 is interesting: https://openwrt.org/toh/unielec/u7621-06

I would be interested in a 5G "as open as possible" chip. I was considering the RTL-881x family before (https://github.com/nazar-pc/rtl8814AU and https://github.com/astsam/rtl8812au) but a standalone ath9k like the AIRETOS AEX-AR9590-NI could be even better.

Where did you buy your device? It's not on ebay or amazon.

simmons• 7 years ago

I'll chime in, since I moved to PC-based fanless router hardware about 5 years ago, and for my needs it's been fantastic.

One of my WAN connections is 100Mbps down (25 up), and I don't have any problem saturating it. I'm only using whatever hardware offloading Linux supports out of the box, which from the parent article I guess is pretty minimal (e.g. stuff like checksum offloading, I suppose). I haven't noticed any bottlenecks when routing between two gigabit LANs (I just ran a quick dd|nc test that measures 940Mbps). However, my needs may be relatively simple and I'm sure there must be more complex scenarios that would benefit from the offloading described in the parent article.

I bought a 4-port Intel NIC because I wanted discrete network interfaces for 2 LANs and 2 WANs. I ended up having to saw off some of the PCI-e edge connector segments to make it fit in my motherboard, so I'm presumably not getting the benefit of all the lanes the NIC could otherwise take advantage of, but nonetheless it seems satisfactory for my needs.

josteink• 7 years ago

I’ll chime in too.

My experience with OpenWrt and hardware acceleration is that it matters once speeds gets high enough.

I’ve had some TP-Link Archer c7’s I’ve used as a main router in the past.

When all it has to do is switch packets on my local network there’s literally no load (as inspected by using htop). Gigabits ahoy.

However when routing stuff out to the internet, it needs not only to switch but also to do software NAT, and then the SoCs performance starts saturating between 350 and 380 mbps. Doing 500mbps is not happening.

For that reason I changed my main router to a Linksys WRT 1900 ACS (with a Marvel-based SOC) which has hardware NAT.

It runs symmetric 500mbps without seeing the gauges in htop even move. It’s a night and day difference.

Once speeds get high enough, HW offloading isn’t just nice to have, it’s a requirement.

And when you do have that, almost all those other tasks the router has to do becomes trivial because you have wads and wads of CPU to spare.

Imo PC-based solutions are overkill, but yes, you may end up with a more open solution that way.

atomt• 7 years ago

I have no idea about the economics in this area, but it kind of baffles me that they add these propietary, closed source and buggy "accelerators" instead of improving the cores a bit. A bit more L1 cache would go a long way for networking.

Many switched from MIPS to ARM in the past 10? years, but the cores remain mostly just as anaemic as they were.

workrockin5• 7 years ago

Imo PC-based solutions are overkill, but yes, you may end up with a more open solution that way.

Yes exactly! If you look at it from the point of view of functionality to price ratio a pc might just be the best router there is.

- Its modular

- Its easily repairable

- Its portable (in the form of mini pc or a laptop)

- You can buy a second hand PC

Finally if you're taking the trouble of installing openwrt on a router it is a reasonable assumption that you want to do more with it. And in that case you're severely limited by the hardware that you choose.

To get the best of both worlds we can have a dedicated PC as a main router. And other cheap routers as your network extenders.

ausjke• 7 years ago

Openwrt is such a gem,not just for home routers but also for IoT gateways etc. Its installation base is probably on par with Debian. I have long been hoping that Linux Foundation or somebody else to endorse this project and give it a huge boost on top of this already great project.

workrockin5• 7 years ago

Just the other day I tried the build system[1] for openwrt and I was amazed to find how easy it is to create a custom image for a platform of your choice. And not just the kernel you can also build specific packages ,if they are not already supported by the package repository or are outdated.

Everything can be done from a graphical interface. Just a few stokes of keyboard and you have your own custom build. Really impressive stuff.

[1] https://openwrt.org/docs/guide-developer/build-system/use-bu...

Network offloading in OpenWrt (2018) [pdf]

Related Stories