"Intel Packet of Death" not Intel's problem

http://www.h-online.com/security/news/item/Intel-Packet-of-Death-not-Intel-s-problem-1801537.html

by Maci • 13 years ago

78 32 13 years ago

Maci• 13 years ago

Follow up story on: http://news.ycombinator.com/item?id=5177815

brownbat2• 13 years ago

Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet.

Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Tec...

Charming.

ersii2• 13 years ago

I think you're reading way more into this, than there is to it.

Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.

StavrosK• 13 years ago

The difference is in the people.

alanctgardner2• 13 years ago

Just to clarify, there's a difference between a Special Administrative Region like Hong Kong or Macau, and Taiwan. While Hong Kong is largely self-governing internally, it's still part of the PRC. Meanwhile, Taiwan (the ROC) was founded by people ousted during the revolution. It's like saying North and South Korea are 'basically' their own countries. Politically they aren't even friendly.

brownbat2• 13 years ago

Hmm, thought by sticking to raw facts, I would avoid that... turns out I was wrong, now it just looks thinly veiled.

Sincere apologies, poor judgment on my part.

ajross• 13 years ago

Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc...

And I agree that you don't want to be too paranoid about this stuff. But at the same time, if you were expecting and looking for an "illicit backdoor" in hardware, this is exactly the kind of thing you'd expect. Firmware in a place virtually no one knows about gets modified on a per-product basis to do nefarious things. And this is exactly how you'd expect such a modification to be discovered, by accidentally introducing a bug that distinguishes itself from the clean parent.

I mean, I'm not screaming "spy" here, but if I were to have read this story in a techno-thriller novel I'd be writing a post applauding the author for her excellently researched and eminently plausible plot hook.

samstave• 13 years ago

I have commented about this in the past; when I was at lockheed, in 2006, we had security debriefings about chinese hacking attempts.

There were trojans on the network that were sending little data packets back to china... but more interestingly:

Lockheed employees were not allowed to connect their macines to any foreign network. Even of those which were suppliers.

There was a supplier in Taiwan where employees would go and would transfer some files via sneakernet (USB keys) - the supplier had been hacked and the chinese were using the Taiwanese suppliers machines to attack the lockheed employees via the transfer of the USB sticks.

The point is that don't underestimate Chinese hackers and the potntial vectors they are willing to exploit.

specialist• 13 years ago

Trust, but verify. With everything that we know, it's silly to not be paranoid.

I got pretty excited about election integrity for a while.

The default position of the defenders of the status quo was "I can't believe you don't trust us. Prove there's something wrong. You 'experts' in computers, security, and elections are just a bunch of conspiracy freaks."

My default position is "show me". That skepticism merely makes me an informed consumer.

eps• 13 years ago

If this were a DoS backdoor, it would've not been that much harder to make it less discoverable. Just use two magic bytes, or three. The chance of false positive are virtually zero and yet you'd still be able to use basic ICMP/ping to trigger it if needed.

mark-r• 13 years ago

A backdoor that bricks the device, but only if it's the first packet received, isn't terribly useful. The more plausible explanation is that it's a bug that was introduced by the actual backdoor that still remains undiscovered.

noonespecial4• 13 years ago

Kielhofner's response.

http://blog.krisk.org/2013/02/packets-of-death-update.html

I used to use "Lanner" gear for voip and these had embedded intel ethernets. I don't have any more of them to test, but I swear I've seen it on them as well. We suspected power supply problems because the link lights would just go dark every once in a blue moon and need a power cycle to set right, but then we were never be able to reproduce it.

gonzo• 13 years ago

We use Lanner gear for VoIP, and have never seen a problem.

noonespecial4• 13 years ago

I think we only saw it on FW-7550's at the beginning of the production run (the ones with the "snout" fans on the CPU and no case fan).

jessaustin4• 13 years ago

I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the controller could do that then it would solve the Halting Problem. Improved drivers, his second suggestion, are always a good thing, but drivers only get patched to handle broken hardware in response to the discovery of broken hardware. There is no way to anticipate each particular way a NIC could possibly be broken ahead of time.

The market demands controllers with flexible and expandable functionality. Board manufacturers use the EEPROM to specify exactly what behavior is required. If a particular manufacturer underestimates the importance of correctness and doesn't perform the code review and testing necessary to prevent a PoD, that isn't Intel's fault.

viraptor• 13 years ago

> "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?"

For a device which can be checked from outside (second subsystem for device self-monitoring), this is actually possible to implement and fairly common. Watchdogs are often implemented to restart automatically when the device is completely unresponsive.

mnarayan01• 13 years ago

> That is, if the controller could do that then it would solve the Halting Problem

The halting problem is actually decidable for limited-memory machines, though you need O(2^n) memory beyond the n-memory of the machine to actually decide it.

kkielhofner4• 13 years ago

Kristian Kielhofner here - While I understand your analogy I don't think it's an accurate one. In fact, with the release of the successor to the 82574 Intel has already implemented some of the things I suggested:

http://communities.intel.com/community/wired/blog/2012/10/18...

Clearly they have learned from the various EEPROM issues on previous controllers (including the 82574) and implemented (among other things) EEPROM signing, which addresses some (most?) of my concerns about sane hardware behavior. Software drivers already do some basic EEPROM checks on this hardware (I know because I've had to tweak them); I'm simply suggesting these checks go a little further to verify the various EEPROM settings than could potentially result in a scenario like this one. When the effects are as significant as they are here I hope we can all agree: more sanity checking is a good thing.

fulafel• 13 years ago

Firmware images usually have checksums. Was this an Intel blob suffering from bitrot, or does Intel have some more or less error prone way to build your own FW images for NICs?

noonespecial4• 13 years ago

I suspect NICs these days are tiny computers in their own right. As a motherboard manufacturer, you can probably program them to do all sorts of nifty, with the possible downside of strange things happening if you get it wrong.

mrb• 13 years ago

It is not the firmware that was corrupted. It is the EEPROM that was incorrectly programmed.

The EEPROM is typically 4kB on the 82754. When it is reprogrammed either by the end-user (eg. via ethtool(1)) or by the manufacturer, the programming procedure recomputes a checksum on the first 128 bytes IIRC (when reprogramming via ethtool, the kernel driver e1000e is responisble for automatically updating the checksum.

So all in all, no, the packet of death issue was not caused by bitrot.

GiHe2• 13 years ago

Cross-posted at h-online.com:

I have a plurality of systems with Intel motherboards which demonstrate the same kind of problems. The motherboards in question have two Intel ethernet controllers, one of which is an 82574L.

The systems connect to two different networks. When the systems attach to one of the networks (but not the other) using the 82574L interface (but not the other), that interface dies after some unpredictable amount of time.

I have tried posting comments to the Intel engineer's blog post (and PM-ing the engineer directly), but they do not appear. In fact, there seem to be no comments at Intel's site, despite the post having nearly 6000 views (at my time of writing).

Something is not right here.

kkielhofner4• 13 years ago

This.

As I say in my updated post, this is a complex issue with clear combinatorial factors. More than likely it's not limited to one chip, one packet, or one EEPROM configuration. A quick reading of the web shows various unexplained issues with this family of Intel ethernet controllers randomly exhibiting the exact behavior I've described. Different controllers, different mobo OEMs, different EEPROM settings. Are all of these issues related to some kind of "packet of death"? Certainly not. However, are at least some of them? Almost certainly, even if they're not vulnerable to my (extremely specific) "packet of death". We still don't know exactly why this is happening (even in my extremely specific case).

GiHe2• 13 years ago

I have another interesting (and reproducible) manifestation on Supermicro motherboards with two 82574L controllers. In this case, it is again true that we only experience problems on the first (as ordered by ascending MAC address) of the two interfaces.

That was the case, though I did not clearly state so above, on the Intel motherboard with one 82574L and one 82579LM.

kkielhofner4• 13 years ago

I've been thinking about your situation and I'm reminded of my own words: "there is always a reason".

Would you be able to e-mail me (CAPTCHA here):

http://tinyurl.com/66srzt

I'd like to discuss your issue further. Thanks!

"Intel Packet of Death" not Intel's problem

Related Stories