Ghost in the ethernet optic

https://blog.benjojo.co.uk/post/smart-sfp-linux-inside

by rcarmo • 5 years ago

224 53 5 years ago

blog.benjojo.co.uk

pabs36• 5 years ago

I wonder if it is GPL compliant, including the ability to update the Linux kernel/etc.

iso16314• 5 years ago

There's no requirement in GPL 2 (the one the kernel uses) to be able to update the kernel.

NavinF5• 5 years ago

I was about to say the same thing, but I think he’s talking about how GPL2 requires the company to give OP the kernel source code. OP mentioned that he didn’t have the kernel headers for DKMS.

simiones• 5 years ago

This is debatable - the license is pretty clear about it being a requirement, but it has been patchily enforced:

The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable [emphasis mine].

Even in the infamous TiVo case, customers of TiVo had complete access to modify the Linux kernel or any other GPL binary - the proprietary TiVo software would just refuse to run if any modification had been applied this way.

pabs36• 5 years ago

That is a common misconception. Both the FSF (the license authors) and Software Freedom Conservancy have always held the opposite view, and always enforced that view.

https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://events19.linuxfoundation.org/wp-content/uploads/2017...

ptsneves• 5 years ago

I am surprised they used debian and not Yocto or buildroot, for an embedded device. Would anyone speculate on why debian would be preferred?

benjojo125• 5 years ago

(Author of post)

They advertise this as a network appliance, so it makes sense that they use Debian since being able to install packages via things like apt makes the whole experience much easier. A lot of career network engineers are not that savvy with sysadmin type roles, so something like buildroot would alienate their market a little

imachine1980_• 5 years ago

ARMv7, with 512M of RAM, isn't that embbeded and you win a full os whit ready to use tested packages, package managers, and boot system, it saves a lot of times and expertice, maybe they need to opt out of it if they want to have better termals, or more optimization.

mschuster91• 5 years ago

I've had to work with embedded devices in the past (develop an alternative firmware for $THING), and irgh I hated buildroot which was the original firmware's base. Way too much effort to compile stuff yourself... a simple "debootstrap" and installation of your private packages is way easier to get stuff up and running. And when you run into some package you need, you directly run "apt install" on the device itself, add the install command to the script that creates your firmware image and that's it.

The best thing is, whatever issues you run into will already have been solved a million times before by the wide Debian community and whatever remains can be solved by a quick hop into #debian on whatever IRC network they migrated to.

snickmy• 5 years ago

(random thought) It surprises me how, despite working in the tech industry for over a 15 years now, I struggled in following the details of this blog post (which is well written). It's so impressive how things got very complex over time, and how verticalize the role of an Engineer is becoming.

NavinF5• 5 years ago

I feel like things have gotten a lot simpler over time. Examples:

1. There's only one relevant type of optics (Q)SFP(+)(28) instead of many incompatible ones. Line cards have become uncommon.

2. Everything runs Linux so he can just run tcpdump instead of some vendor specific monitor command.

3. 10G ethernet and 56G infiniband are dirt cheap so now there’s a large online community for homelabs that use data center hardware. Nobody needs to be a network guy to run a data center.

kazen442• 5 years ago

Nobody needs to be a network guy to run a data center.

the complexity from networking at a DC mainly comes from scale.

running enterprsie/dc hardware in a homelab is vastly different from running it in production.

sliken• 5 years ago

I'm looking to upgrade my 1G home network to a mix of 10+2.5G (including PoE) Which "large online communities" where you thinking of?

NavinF5• 5 years ago

Fascinating! Tho I can’t think of that many use cases where a 1G hub and raspberry pi can’t do the job.

phoronixrly• 5 years ago

I find this very interesting. Most of my concerns with using this would be alleviated if I was able to flash my own image onto the SFP, e.g. an OpenWrt installation. (Disclaimer: no offence to the people that produce this product, I'm an equal opportunity closed source firmware avoider:)

The author mentions that 'In a more premium software package for the smart-sfp you can configure ERSPAN sessions with filters'. Selling a more expensive software package for the SFP would be a reason to lock it down and prevent others from offering competing (including open-source) software.

Another interesting aspect is the communication with the programmable logic. What is implemented in the FPGA? Is it purely signal processing? Is there packet inspection and filtering? Could the communication between the CPU and the FPGA be reverse engineered to provide a driver?

Edit: Ben, do you plan on playing around some more with these to find out if they can be hacked to run your own OS?

benjojo125• 5 years ago

Ben, do you plan on playing around some more with these to find out if they can be hacked to run your own OS?

Nope, I'm probably going to use it as a remote monitoring/out of band device when the time comes. The device is useful enough in it's slightly closed source state than as a device for hacking (and maybe a bricking risk)

kloch2• 5 years ago

There are two main uses for this kind of thing.

- Legit: debugging/monitoring. Other legit uses are theoretically possible but the device has severe limitations that likely make them impractical or unwise.

- Surreptitious: This is clearly where the value proposition of this device lies. An optic could be "unknowingly" swapped out on an interesting link to snoop on or infiltrate a network.

Swapping out optics in a large network is not uncommon as they do fail. More often they are swapped out as a troubleshooting step where the original optic may not even be bad. This way log messages indicating link flap and replacement of an optic could likely go unnoticed.

formerly_proven2• 5 years ago

Looking at the pictures this thing isn't exactly the typical SFP form factor, it protrudes ~5 cm or so over a normal SFP. So if you swap one of these out it's going to stick out like the proverbial sore thumb.

benjojo125• 5 years ago

Who says you can see the side they swapped the optic on? For datacenter cross connect links, its not uncommon for the other side of the link to be somewhere you can't see/have access to.

NavinF5• 5 years ago

Ehh this chonker doesn’t look very surreptitious. I can only think of one other device that sticks out even further in front of the switch. Said device is a very old SFP+ to 10G 100m copper module with a huge heatsink and an external power brick!

That aside, I could imagine a gov’t agency programming these things and forcing ISPs to put them on customer ports. No need to wait for a maintenance window or free up rack space.

kloch2• 5 years ago

This is one reason why I am a big proponent of network engineers physically visiting their equipment periodically. In large networks that may never happen, data center techs (often not even your own employees) are the only ones who ever see the racks in person.

Even if the 3rd party tech notices something unusual about the optic the certainly aren't going to touch it without a ticket and will probably not even mention it.

canniballectern• 5 years ago

This is the one they sell publicly to anyone at a relatively affordable price.

If they make ones that run cooler and look the same as a normal optic, they might not be so forthcoming about it.

CountDrewku• 5 years ago

Yeah I feel like this would be mostly used for nefarious purposes since it wouldn't be obvious it was there. Other than that I can't think of any way that it's better than just a raspberry pi.

ale422• 5 years ago

This seems so nice for NSA-like implants...

sneak• 5 years ago

I think those are designed to avoid detection, so they (having near-unlimited budgets for these sorts of covert things) prefer to go with replacement firmwares and suchlike. They've been known to intercept packages of networking equipment and do all-software implants so that you can't even tell a device has been tampered by looking at it, and if they want to exit they can remove all traces that they were there. Leaving physical hardware in place is the opposite of that.

There are so many places to hide malicious code in any computer or networking system these days, using weird nonstandard obvious hardware like this is pretty amateur.

ale422• 5 years ago

Perfectly makes sense! But such hardware can be an additional place for their implants... supposing that the hardware is already there. Not that they are supplying it or replacing the passive one with the (clearly detectable) one in question here.

userbinator• 5 years ago

What caught my eye was the pixelated serial numbers. Also, the fact that the part and serial numbers are plainly visible in a few pictures, but not others.

benjojo125• 5 years ago

Where can you see serial numbers? I thought I got them all

jaywalk• 5 years ago

The first three pics. I only see pic where it's censored.

rytill• 5 years ago

He said

But such a feature could also be used to create a fake 169.254.169.254 (AWS/Cloud metadata IP address endpoint) and serve requests from it.

Wouldn’t such a thing be impossible if the application is using end-to-end encrypted requests to AWS?

optimuspaul• 5 years ago

the metadata service is http and not encrypted.

mmastrac• 5 years ago

While researching EEPROM flashing on AliExpress SFPs, I discovered that many of these do support SSH. I suspect that even the smaller ones might have tiny Linux onboard.

Example:

https://forum.mikrotik.com/viewtopic.php?t=116346

https://github.com/hwti/G-010S-A

w7• 5 years ago

Last year on another transceiver (QSFP28) teardown[0] I was surprised to find out that transceivers not marketed as "smart" also have SOCs inside them to regulate internal temperature. I had always thought the devices were "dumber" and never bothered to look inside.

So programmable CPUs in your transceivers might be more common than one would think.

[0] https://twitter.com/kwf/status/1470508119725805570

winniejinping• 5 years ago

Also there are SFP ONU modules for GPON/EPON, I've been using one (with RTL9601C chipset) for a year and it works great, fiber cable directly to the router or switch, no more shitty ISP ONU & router.

jauer• 5 years ago

Things like this have been around for nearly (at least?) 10 years, but are not well-known outside of spaces that care about telco-style demarc & OAM systems.

An example from ~2012 would be the RAD MiNID which is available in a handy "sleeve" format where you can use your normal SFP with the smart SFP.

Pretty cool to see a writeup of this sort of thing (and increased vendor representation).

Ghost in the ethernet optic

Related Stories