Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug

https://www.phoronix.com/news/Intel-i219-LM-Linux-60p-Fix

by zdw • 3 years ago

105 45 3 years ago

jmclnx• 3 years ago

I wonder if this is an issue with the BSDs too. Sometimes they peek at Linux for these things.

wmf• 3 years ago

This is a hardware bug so I would expect it to affect all OSes. Turning off TSO is counterintuitive so drivers tend to turn it on by default.

AdrianB12• 3 years ago

Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. I am not saying BDS is better, just a few things are quite different vs Linux.

betaby• 3 years ago

a bit more advanced than Linux ones

and then we look at the wifi stack in BSD(s)...

ilyt• 3 years ago

Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring.

for those 3 nics they use.

I doubt some random 2.5Gbit nic is on their list of stuff to fix

KyleSanderson• 3 years ago

It isn't. It's a pure driver bug with all of these. There's a pending patch to clear the MSI registers on disconnect that I'm planning on driving to testing tomorrow. I hounded them for 4 months by sending full stacks with symbols and they don't have any hardware in the lab (which is very common now, unfortunately)...

If you want to recover, set a ifdown script to `rmmod igc && modprobe igc` and you will never panic or hang, just have a longer interface bounce time. I'm running this on a 6 nic system which is completely egregious, but it hasn't crashed / hung / panic'd in the field since doing that.

legulere• 3 years ago

That makes me wonder how much waste there is, just because there is something broken without anyone noticing. Recently a customer had two nearly identical systems behaving completely different in regards to performance. It turned out it simply was some bios setting.

magicalhippo• 3 years ago

That makes me wonder how much waste there is, just because there is something broken without anyone noticing.

I'd say a lot. Not seldom when I go over to a colleague I notice their laptop is breathing hard. "Oh yeah, so annoying! It's been doing that for a while now". Check Task Manager and sure enough, some process is stuck sucking 100% of a core... 5 CPU days worth.

Even I find it difficult to spot this, with my 16 cores. I got an efficient cooler, so no fan spins up noticeably. Randomly check Task Manager, oh explorer.exe is pegging a core, and has been using 15 days worth of CPU... Gee thanks!

LinuxBender5• 3 years ago

It's nice to see them giving this some attention. I hope they also look into the I225-V at the myriad of different negotiation bugs that change with each firmware revision on 2.5gb networks that also vary with different offloading features enabled/disabled including EEE. Each 2.5gb switch also exhibits different behavior which is probably not Intel's fault.

rasz• 3 years ago

is probably not Intel's fault

consensus is Intel went down the shitter quality wise, just one example https://www.youtube.com/watch?v=DXNyHFOWx_k

LinuxBender5• 3 years ago

Could be. Most of the folks I interacted with in the past have retired. I think I am just going to revert back to full tower PC's for anything I do, even firewalls and appliances so that I am not locked into what may be abandonware. I was hoping to use less power and take up less space with the mini-PC's but in hind-sight that may have been a mistake.

snvzz• 3 years ago

I got tired of fighting that NIC, and got a Realtek 2.5g one.

All my problems gone, life is good.

LinuxBender5• 3 years ago

I would totally do that but I have a few mini-PC's that have the I225-V and I can't swap them out. I've used USB3.2->Ethernet dongles as work arounds until I could find a switch that was happy with it. Same with the latest Protectli firewalls, they also have the I225-V. For those I have to use ethtool to force a specific speed which is not ideal. I'm thinking mini-PC's are maybe a mistake in this generation of NICs.

userbinator3• 3 years ago

I've had good experience with Realtek NICs too, not 2.5g yet but their 10/100/1000 models. I suspect that's because they use a far simpler design with not much in the way of fancy features[1]. They also tend to benchmark slower with higher CPU usage, but I'll take slightly slower but stable over fast but intermittent connectivity.

[1] Ironically, in this case the Intel has to have a fancy feature, intended for increasing performance, disabled in order to increase performance.

jeffbee• 3 years ago

The Realtek NIC embedded in my Asus motherboard hits maybe 600mbps with a tailwind. It only has 1 receive queue and doesn't support interrupt coalescing. Junk.

mirker• 3 years ago

When people say you need the third hardware revision to get functional performance, and it still doesn’t work, you should conclude it’s poorly designed. The fact that these chips were being sold with old hardware revisions also seems anti-consumer.

rubatuga2• 3 years ago

Agreed, Intel gave up and released the I226 because their I225 brand was so tarnished. I removed the I225 from my server after I noticed the driver crashing during heavy loads (Ubuntu 22.04). For USB I've been having success with the rtl8156bg from juplink.

malkia• 3 years ago

Not a network guy, but couldn't this be detected (for any other card too) by running some mandatory tests?

andix• 3 years ago

One would expect Intel to do periodic regression tests on Linux kernel releases on their hardware. A lot of Intel customers use Linux, and they may switch to other vendors if Intel hardware has bad Linux support.

arghwhat3• 3 years ago

They do, and they provide some of the most extensive regression testing capabilities provides to kernel development.

But to be fair, this is an old NIC with a hardware bug that requires an odd workaround disabling an otherwise very useful performance feature. Missing that edge case is understandable.

justinclift4• 3 years ago

Hmmm, that doesn't really seem to gel though, unless their extensive regression testing is a recent development?

Saying that because at some point, that NIC was a new thing, not an old one. So extensive regression testing should have picked it up then, yeah?

fulafel• 3 years ago

Interesting that in a era of 100+ Gbit Ethernet we seem to need the offload features working just right to get even 1 Gbit.

arghwhat3• 3 years ago

Well in this case the bug is that the feature doesn't work and needs to be disabled.

But generally speaking you want all the offload you can get. For 1G, it allows pushing power efficiency much further. For 100G, it allows you to spend time on something other than parsing and writing packets.

userbinator3• 3 years ago

you will want to look forward to upgrading your Linux kernel build soon... [...] due to a regression introduced back in 2020.

If anything, that message seems to be implying to not upgrade if everything is already working well.

A bit tangential, but ever since they came out with the '217 I've thought this is one of the worst-named products and could never remember what the actual letter is. Here it's an uppercase I, but even Intel seems to think it's a lowercase L (for LAN?) sometimes:

https://www.intel.com/content/www/us/en/support/articles/000...

dzdt• 3 years ago

I have an Intel I219-V adaptor on a Windows desktop. I had a hell of a time trying to figure out the problem when I upgraded to Verizon FIOS internet; upload speeds were abysmal. The chipset has a hardware bug making it incompatible with the Verizon Optical Network Terminal. The workaround is to disable checksum offloading in the network driver advanced settings. It sounds like this is the linux version of the same thing.

timnetworks• 3 years ago

This dumb chip caused myself and my colleagues some headaches when the driver kept getting deleted as suspected-malicious but reinstalled right away by Windows, causing the whole lot to bounce like on some giant trampoline.

Reading the comments here, we aren't the only ones taken by surprise and/or aback.

herpderperator2• 3 years ago

Doesn't disabling offloading (of anything) result in increased CPU usage when transferring lots of data? Is it now that it's faster but uses more CPU, or is it actually faster with the same CPU usage (a "proper" fix)?

majke• 3 years ago

There are probably a couple of independent things:

(1) HW GRO, most often LRO (TCP)

(2) SW GRO aka GRO

Hardware GRO was typically called LRO for TCP and is often disabled in practice. One reason is XDP, with XDP some offloads need to stay off. Another is device quirks: while idea of waiting for more data, and passing larger packets into kernel soiunds nice, it also means that device must do packet concatenation. Think about ECN or selective-acks. The concatentation is lossy and unless the flow is perfect, it might not be possible to concatenate packets. Furthermore devices often had issues with being too aggrressive and loosing important data.

In Linux there is a software approach that does kindof the same thing, but in software, called GRO, and this is often sufficient. It also supports more than just TCP.

In other words - having LRO disabled is common.

herpderperator2• 3 years ago

Does this mean if someone with an I219-LM did a regular file transfer they would only get 67MiB/s instead of 112MiB/s? Surely this would be immediately noticeable and not take 3 years to fix? Was it just a throughput bug or a packets-per-second bug? Did any other conditions need to be present for it to trigger the slowdown? So many questions... I just don't understand how this can go unnoticed. If you buy a 1Gbps NIC and it performs <1Gbps for a simple file transfer, you would think that's not something Intel would miss, or ignore if reported given how easy it would be to reproduce.

Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug

Related Stories