5Gbps Ethernet on the Raspberry Pi Compute Module 4

https://www.jeffgeerling.com/blog/2020/5-gbps-ethernet-on-raspberry-pi-compute-module-4

by geerlingguy • 6 years ago

222 83 6 years ago

geerlingguy20• 6 years ago

Sorry about the slightly-clickbaity title. I actually have at least a 10 GbE card (and switch) on the way to test those and see if I can get more out of it, but for _this_ test, I had a 4-interface Intel I340-T4, and I managed to get a maximum throughput of 3.06 Gbps when pumping bits through all 4 of those plus the built-in Gigabit interface on the Compute Module.

For some reason I couldn't break that barrier, even though all the interfaces can do ~940 Mbps on their own, and any three on the PCIe card can do ~2.8 Gbps. It seems like there's some sort of upper limit around 3 Gbps on the Pi CM4 (even when combining the internal interface) :-/

But maybe I'm missing something in the Pi OS / Debian/Linux kernel stack that is holding me back? Or is it a limitation on the SoC? I though the ethernet chip was separate from the PCIe lanes on it, but maybe there's something internal to the BCM2711 that's bottlenecking it.

Also... tons more detail here: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

mmastrac2• 6 years ago

Awesome work. Been watching your videos on these (the video card one was especially interesting).

At what point are you saturating the poor little ARM CPU (or its tiny PCIe interface)?

geerlingguy20• 6 years ago

Heh, I know that ~3 Gbps is the maximum you can get through the PCIe interface (x1, PCI 2.0), so that is expected. But I was hoping the internal ethernet interface was separate and could add one 1 Gbps more... the CPU didn't seem to be maxed out and was also not overheating at the time (especially not with my 12" fan blasting on it).

wil421• 6 years ago

Do you think an SFP+ nic would work? It would be cool to try out fiber.

baybal22• 6 years ago

There are no SFP option on 5gbps NICs as i understand as per standard

monocasa5• 6 years ago

You might be hitting the limits of the RAM. I think LPDDR3 maxes out at ~4.2Gbps, and running other bus masters like the HDMI and OS itself would be cutting into that.

wmf4• 6 years ago

32-bit LPDDR4-3200 should give 12.8 Gbytes/s which is 102 Gbits/s.

ksec• 6 years ago

Sorry about the slightly-clickbaity title.

Well yes because 5Gbps Ethernet is actually a thing ( NBase-T or 5GBASE-T). So 1Gbps x 5 would be more accurate.

Cant wait to see results on 10GbE though :)

P.S I really wish 5Gbps Ethernet is more common.

ncrmro• 6 years ago

My ATT router made by Nokia has one 5gbe and the fiber plugs in directly with SFP!

geerlingguy20• 6 years ago

True true... though in my work trying to get a flexible 10 GbE network set up in my house, I've found that the support for 2.5 and 5 GbE are iffy at best on many devices :(

StillBored2• 6 years ago

Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe tends to be a rough upper limit on most PCIe interfaces, so the 3Gbit your seeing is pretty close to what one would expect.

edit: Oh its 3Gbit across 5 interfaces, one of which isn't PCIe, so the PCIe side is probably only running at about 50%. It might be interesting to see if the CPUs are pegged (or just one of them). Even so, PCIe on the rpi isn't coherent so that is going to slow things down too.

geerlingguy20• 6 years ago

It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

leptons• 6 years ago

It might be interesting to see if the CPUs are pegged (or just one of them).

This is very likely the answer. I see a lot of people who think of the Pi as some kind of workhorse and are trying to use it for things that it simply can't do. The Pi is a great little piece of hardware, but it's not really made for this kind of thing. I'd never think about using a Raspberry Pi if I had to think about "saturating a NIC".

stratosmacker• 6 years ago

Jeff,

First off, thank you for doing this kind of 'r&d', it is really exciting to see what the Pi is capable of after less than a decade.

Would you be interested in someone testing a SAS PCI card? I'm going to pick up one of these as soon as they're not backordered...

CyberDildonics3• 6 years ago

Instead of lying and then apologizing once you get what you want, it would be better to just not lie in the first place.

geerlingguy20• 6 years ago

Technically it's not a lie—there are 5x1 Gbps of interfaces here. But I wanted to acknowledge that I used a technicality to get the title how I wanted it, because if I didn't do that, a lot of people wouldn't read it, and then we wouldn't get to have this enlightening discussion ;)

soneil• 6 years ago

I assume you saw the video with Plunkett the RPF put out ( https://youtu.be/yiHgmNBOzkc specifically interesting at 10:45 ) - He mentioned he was testing 10GbE fibre and reached 3.2gbit. Now he goes into absolutely zero detail on that, but I find it interesting you've both hit the same ceiling.

(He also mentioned 390MB/sec write speed to nvme, which is suspiciously close to the same ceiling)

geerlingguy20• 6 years ago

Yeah, I think the PCIe link hits a ceiling around there.

Note that combining the internal interface with the 4 NIC interfaces, and overclocking to 2.147 GHz got it up to 3.4 total Gbps. So the IRQ interrupts are the main bottleneck when it comes to total network packet throughout.

baybal22• 6 years ago

A much easier option:

Get a USB 3.0 2.5G or 5G card. With a fully functional DMA on the USB controller it can get quite close to PCIE option.

A setback for all Linux users at the moment:

The only chipmaker making USB NICs doing 2.5G+ is RealTek, and RealTek chose to use USB NCM API for their latest chips.

And as we know Linux support for NCM now is super slow, and buggy.

I barely got 120megs from it. Will welcome any kernel hacker taking on the problem.

vetinari• 6 years ago

The only chipmaker making USB NICs doing 2.5G+ is RealTek, and RealTek chose to use USB NCM API for their latest chips.

QNAP QNA-UC5G1T uses Marvell AQtion AQC111U. Might be worth a try.

ProAm• 6 years ago

That was a fun read. Thanks.

unilynx• 6 years ago

"I need four computers, and they all need gigabit network interfaces... where could I find four computers to do this?"

Why not loop the ports back to themselves? IIRC, 1gbit ports should autodetect when they're cross connected so it wouldn't even need special cables

geerlingguy20• 6 years ago

Would that truly be able to test send / receive of a full (up to) gigabit of data to/from the interface? If it's loopback, it could test either sending 500 + receiving 500, or... sending 500 + receiving 500. It's like sending data through localhost, it doesn't seem to reflect a more real-world scenario (but could be especially helpful just for testing).

nitrogen• 6 years ago

I think maybe they meant linking Port 1 to Port 2, and Port 3 to Port 4? Also I believe gigabit ethernet can be full duplex, so you should be able to send 1000 and receive 1000 on a single interface at the same time if it's in full duplex mode.

jlgaddis2• 6 years ago

It's full-duplex, that's 1000 Mbps in each direction simultaneously.

adrian_b• 6 years ago

When you loop back Ethernet links in the same computer, you need to take care with the configuration, because normally the operating system will not route the Ethernet packets through the external wires but will process them like being for localhost, so you will see a very large speed without any relationship with the Ethernet speed.

How to force the packets through the external wires depends on the operating system. On Linux you must use namespaces and assign the two Ethernet interfaces that are looped on each other to two distinct namespaces, then set appropriate routes.

escardin2• 6 years ago

It's probably outside the scope (and possibly cheating) but could a DPDK stack & supported nic[1] push you past the PCIe limit?

[1] https://core.dpdk.org/supported/

q3k3• 6 years ago

Does DPDK actually let you not have to DMA packet data over to the system memory and back?

escardin2• 6 years ago

No you still have to send the data over the pcie link, but DPDK should basically offload all the network work to the nic, so that you are just streaming data to it. The kernel won't need to deal with IRQs or timing or anything like that.

I might be making things up, but I believe you can also run code on DPDK nics? i.e. beyond straight networking offload. If that's the case you could try compressing the data before you DMA it to the nic. This would make no sense normally, but if your bottleneck is in fact the pcie x1 link and you want to saturate the network, it would be something worth trying.

I mean really the whole thing is at most a fun exercise as the nic costs more than the pi.

drewg1233• 6 years ago

So theoretically, 5 Gbps was possible

No, it is not. That NIC is a PCIe Gen2 NIC. By using only a single lane, you're limiting the bandwidth to ~500MB/sec theoretical. That's 4Gb/s theoretical, and getting 3Gb/s is ~75% of the theoretical bandwidth, which is pretty decent.

geerlingguy20• 6 years ago

I'll take pretty decent, then :)

I mean, before this the most I had tested successfully was a little over 2 Gbps with three NICs on a Pi 4 B.

drewg1233• 6 years ago

Can you run an lspci -vvv on the Intel NIC? I just re-read things, and it seems like 1 of those Gb/s is coming from the on-board NIC. I'm curious if maybe PCIe is running at Gen1

stkdump• 6 years ago

Well, when a LAN is 1Gb/s they are actually not talking about real bits. It actually is 100MB/s max, not 125MB/s as one might expect. Back in the old days they used to call it baud.

wmf4• 6 years ago

This is wrong; 1 Gbps Ethernet is 125 MB/s (including headers/trailer and inter-packet gap so you only get ~117 in practice). Infiniband, SATA, and Fibre Channel cheat but Ethernet doesn't.

geerlingguy20• 6 years ago

I think I've found the bottleneck now that I have the setup up and running again today—ksoftirqd quickly hits 100% CPU and stays that way until the benchmark run completes.

See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

syoc• 6 years ago

I would have a look at sending data with either DPDK (https://doc.dpdk.org/burst-replay/introduction.html) or AF_PACKET and mmap (https://sites.google.com/site/packetmmap/ )

You can also use ethtool -C on the NICs on both ends of the connection to rate limit the irq signal handeling allowing you to optimize for throughput instead of latency.

iscfrc• 6 years ago

You might want to try enabling jumbo frames by setting the MTU to something >1500 bytes. Doing so should reduce the number of IRQs per unit of time since each frame will be carrying more data and therefore there will be fewer of them.

According to the Intel 82580EB datasheet[1] it supports an MTU of "9.5KB." It's unclear if that means 9500 or 9728 bytes.

I looked briefly for a datasheet that includes the ethernet specs. of the Broadcom 2711 but didn't immediately find anything.

Recent versions of iproute2 can output the maximum MTU of an interface via:

  # Look for "maxmtu" in the output
  ip -d link list

Barring that you can try incrementally upping the MTU until you run in to errors.

The MTU of an interface can be set via:

  ip link set $interface mtu $mtu

Note that for symmetrical testing via direct crossover you'll want to have the MTU be the same on each interface pair.

[1] https://www.intel.com/content/www/us/en/embedded/products/ne... (pg. 25, "Size of jumbo frames supported")

geerlingguy20• 6 years ago

I set the MTU to its max (just over 9000 on the intel, heh), but that didn't make a difference. The one thing that did move the needle was overclocking the CPU to 2.147 GHz (from base 1.5 GHz clock), and that got me to 3.4 Gbps. So it seems to be a CPU constraint at this point.

q3k3• 6 years ago

Seems to be in the same ballpark as when I got ~3.09Gbps on the Pi4's PCIe, but on a single 10G link: https://twitter.com/q3k/status/1225588859716632576

geerlingguy20• 6 years ago

Oh, nice! How did I not find your tweets in all my searching around?

q3k3• 6 years ago

Shitposting on Twitter makes for bad SEO :).

nojokes2• 6 years ago

Did you test without main Ethernet connection?

geerlingguy20• 6 years ago

Yes.

nojokes2• 6 years ago

Meaning using on board Ethernet will not increase or decrease the bandwidth?

voltagex_3• 6 years ago

I can get about 1.2-1.7 gigabit on the Pi 4 using a 2.5GBe USB NIC (Realtek). Some other testing shows the vendor driver to be faster, but when I tested it on a much faster ARM board, I can get the full 2.5GBe with the in-tree driver.

magicalhippo• 6 years ago

Thanks, that's useful info as I was just thinking about getting one of those for a project.

classics2• 6 years ago

One mans dull travelogue of copying and pasting things.

5Gbps Ethernet on the Raspberry Pi Compute Module 4

Related Stories