Hacking millions of modems and investigating who hacked my modem

https://samcurry.net/hacking-millions-of-modems

by albinowax_ • 2 years ago

838 270 2 years ago

phs318u• 2 years ago

What a great article. Very easy to follow. The best part was that instead of attacking the messenger and denying any problem, Cox seem to have acted like the very model of responsible security response in this kind of situation. I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs. It's the kind of error that could easily be missed by superficial testing or depending on the reason behind the bug, perhaps not even experienced in the test environment.

p0seidon• 2 years ago

Totally agree, an easy read and a great reaction by Cox. I also like that the discovery and the bug itself were not communicated in a negative or condescending way, which is sometimes the case.

senectus1• 2 years ago

agreed, lets hope they dont bloody sue him into the ground for "hacking"

Its stuff like this that company's should REWARD people for finding.

imadr3• 2 years ago

I assumed they offered a bounty for bug disclosure? You mean to tell me that an internet provider with 11 billion in revenue can't pay someone that found a bug impacting all their clients?

Frankly he could have just sold the vulnerability to the highest bidder

acdha2• 2 years ago

They have a pretty good looking responsible disclosure program which I’m assuming he checked first - it’d be surprising for someone who works in the field not to have that same concern:

https://www.cox.com/aboutus/policies/cox-security-responsibl...

webninja2• 2 years ago

Yeah let’s hope that they don’t prosecute him under the CFAA. He saved the FBI and untold others. He’s a hero.

randomcarbloke4• 2 years ago

it's good but the constant use of "super" was a little off-putting, "super curious", "super interesting", "super interested", etc.

pfdietz• 2 years ago

Super off-putting, you mean.

armada651• 2 years ago

There were 4 occurrences of the word "super" in an article with more than four thousand words in it, there is no need for "etc." you quoted all the occurrences since "super curious" was used twice.

sgerenser• 2 years ago

IMHO, your comment is super nitpicky.

oasisbob2• 2 years ago

Cox seem to have acted like the very model of responsible security response in this kind of situation

It's hard to imagine, but I wish they would have taken advantage of him walking in with the compromised device in the first place.

I once stumbled upon a really bad vulnerability in a traditional telco provider, and the amount of work it took to get them to pay attention when only having the front door available was staggering. Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue.

Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

tw042• 2 years ago

Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

He probably should have gone the responsible disclosure route with the modem too. Do you really expect a minimum wage front desk worker to be able to determine what’s a potential major security flaw, and what’s a random idiot who thinks his modem is broken because “modern warfare is slow”?

vlan0• 2 years ago

Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

I can't really blame them. The number of customers able to qualify that a device has actually been hacked is nearly zero. But do you know how many naive users out there that will call/visit because they think they've been hacked? It's unfortunately larger than the former. And that'll cost the business money. When 99.9% of those cases, the user is wrong. They have not been hacked. I say this as someone who supported home users in the 2000s. Home users that often think they'd been "hacked".

mook• 2 years ago

Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.

They were presented with some random person who wanted to get a new modern on their rental but also keep the old one, for free. They had no way of knowing if they were an actual security professional.

flutas• 2 years ago

the amount of work it took to get them to pay attention when only having the front door available was staggering.

I've seen this across most companies I've tried reporting stuff to, two examples.

Sniffies (NSFW - gay hookup site) was at one point blasting their internal models out over a websocket, this included IP, private photos, salt + password [not plaintext], reports (who reported you, their message, etc), internal data such as your ISP and push notification certs for sending browser notifications. First line support dismissed it. Emails to higher ups got it taken care of in < 24 hours.

Funimation back in ~2019(?) was using Demandware for their shop, and left the API for it basically wide open, allowing you to query orders (with no info required) getting Last 4 of CC, address, email, etc for every order. Again frontline support dismissed it. This one took messaging the CTO over linkedin to get it resolved < a week (thanksgiving week at that).

LeifCarrotson• 2 years ago

Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue.

Sounds to me like their support org was reasonably effective at their real job, which is keeping the crazies away from the engineers.

It's even harder for me to imagine them saying "Oh, gee, thanks for discovering that! Please walk right into the office, our firmware developer Greg is hard at work on the next-gen router but you can interrupt him."

buildbuildbuild• 2 years ago

I’ve often wished I could show an “I know what I’m doing” badge to support to guarantee escalation.

“I’m a three star infosec General, if I’m contacting you it’s not to waste your time.”

ImPostingOnHN• 2 years ago

I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs

I would, too. Not sure we will ever learn. Maybe a load balancer config that inadvertently included "test" backends which didn't check authorization?

thecodemonkey• 2 years ago

It's easy to hate on big companies. But can we just applaud Cox for having patched this within a day? That's incredible.

choilive• 2 years ago

That was the most shocking part of the entire article! Unfortunate this vuln existed but clearly engineers there have enough teeth to get stuff done.

flafla2• 2 years ago

Agreed. Bugs happen, bug fixes don’t always happen (especially quickly)

That being said, we could all do with a bit more input sanitization, and I hope Cox learned their lesson here.

pera• 2 years ago

To be honest I would be very surprised if this was Cox as an organization and not just one or two very passionate workers who understood the severity of the issue and stayed after hours fixing it for free.

nsbk• 2 years ago

Seems more like a configuration error. Load balancer balancing over a few hosts, one of them missconfigured. Most likely over 2 hosts given the 50/50 success ratio of the intruder test. If that’s the case then it’s easy to fix in such timeframe

fellerts• 2 years ago

Great read, I loved following your thought process as you kept digging.

At what point did you inform Cox about your findings? It doesn't sound like you were ever given the green light to poke at their management platform. Isn't work like this legally dubious, even if it is done purely in white-hat fashion?

harisec• 2 years ago

Cox has a vulnerability disclosure program. https://www.cox.com/aboutus/policies/cox-security-responsibl...

mavamaarten• 2 years ago

What sort of authentication system just lets calls through randomly sometimes... The incompetence!

taspeotis• 2 years ago

Discovered this in a vendor’s API. They registered the current user provider as singleton rather than per-request. So periodically you could ride on the coat-tails of an authenticated user.

VBprogrammer2• 2 years ago

I once seen a bug in a Django App which caused similar issues. Basically the app often returned a HTTP no content for successful calls from AJAX requests. So someone had DRYed that by having a global NoContentResponse in the file. The problem was that at some point in the Django middleware the users session token got affixed to the response - effectively logging anyone from that point on in as another user.

internetter2• 2 years ago

This is ridiculously easy to do inside scripting languages like javascript

function foo(token: string) {}

function bar(token: string) {}

function baz(token: string) {}

// hmm, this is annoying

let token;

.get((req) => { token = req.data.headers.token }

function foo() {}

It is even possible to do it by "accident" with only subtly more complicated code! I constantly see secrets leak to the frontend because a company is bundling their backend and frontend together and using their frontend as a proxy. This lack of separation of concerns leads to a very easy exploit:

If I'm using, say, Next.js, and I want access to the request throughout the frontend, I should use context. Next even provides this context for you (though honestly even this is really really scary), but before my code was isomorphic I could just assign it to a variable and access that.

At least in regards to the scaryness of the next provided global context, at least now node has AsyncLocalStorage which properly manages scoping, but plenty of legacy...

The entire ecosystem is awful.

From my distrust in bundlers, I'm now fuzzing within CI for auth issues. Hitting the server 10k times as fast as possible from two different users and ensuring that there is no mixup. Also, scanning the client bundle for secrets. I haven't had an issue yet, but I've watched these things happen regularly and I know that this sort of test is not common

therein• 2 years ago

It could also be that a subset of origin servers that the requests being routed to were misconfigured.

rbut• 2 years ago

The API was reverse proxied. Possibly a caching issue?

acdha2• 2 years ago

That was my thought, as well - it’s easy to imagine this calling some internal APIs which aren’t great, leading someone to toss a cache in front of a few calls but botching the logic so the cache is always populated even on errors. I’ve seen a few cases where people tried to bolt something like that onto Java/Python APIs designed to raise exceptions when validation failed so it was especially easy to overlook the error path because the code was written to focus on the successful flow.

thursley• 2 years ago

In my experience this can be caused by a loadbalancer, for example not being able to route (properly) to servers in the pool or a difference in configuration/patch-level between them.

IAmLiterallyAB• 2 years ago

Found a similar big once. The API would correctly reject unauthenticated requests for exactly 10 minutes. Then, for exactly 1 minute, unauthenticated requests were allowed. The cycle repeated indefinitely. Would love to know what was going on on the backend...

aidenn03• 2 years ago

It's happened with both of the G.hn powerline devices I've used; presumably they are all reskinned versions of the silicon vendor's firmware. You can send commands (including changing encryption keys and updating firmware) and sometimes they just go through.

megous3• 2 years ago

One of the reasons to not be excited about ISP provided cable modems with WiFi functionality and to have good endpoint/service security on your LAN. (TLS, DNS over TLS at least accross the modem/ISP)

I just put it in bridge mode, disable wifi, and all network functionality is served by my own devices.

The last modem I rented from ISP, the ISP didn't bother with any firmware updates for ~10 years. It was rock stable because of that, though. :)

phh2• 2 years ago

Counterpoint: ISP with over 1M customers have the incentives of upgrading their HGW "forever" to reduce Capex. My employer (Free, French ISP also shipping HGW to Italia as Iliad) still upgrade their HGW released in 2011 (though if you have yours dating back from 2011, have it replaced (your oled screen is probably dead ;) to get more recent wifi cards). It runs a modern Linux 6.4. You get modern nifties like airtime QoS, got upgraded mobile apps if you wish, and uh lots of software features.

distances• 2 years ago

In Germany one of the more popular modem/router/wifi devices among ISPs is FritzBox. You can also buy these devices yourself, which gives you both: you're using your own hardware instead of renting, and you benefit from long support thanks to aligning incentives from their big customers.

themoonisachees• 2 years ago

I'd like the record to show that the upgraded mobile apps are significantly worse than the old ones. The old ones are currently still available to download, for now.

bootsmann• 2 years ago

Routers are the most exploited IoT devices on the planet, often vulnerabilities in the router firmware persist for years without getting patched because most endusers don't patch their routers. The ISP having a way to play patches onto router and recall unpatchable ones (because they own them) is a net gain for cyber security.

mrguyorama2• 2 years ago

But ISPs DON'T patch routers. Plenty of spectrum modems still run a decade old firmware

megous3• 2 years ago

Yes, I agree that routers are key and critical.

Otherwise I would not be managing my own high quality one, based on the latest Linux kernel, and a standard, well supported and maintained software and carefully selected wifi HW with active manufacturer provided support.

I also would not be trying to isolate and disable most of the ISP provided HW/FW mess, if I believed otherwise. I don't trust ISP that did not upgrade their modem in 10 years, one bit with security of key entry point to my home network.

matheusmoreira• 2 years ago

I just put it in bridge mode, disable wifi, and all network functionality is served by my own devices.

Same. Somehow I got them to install a simple modem, one without all of the router and access point features. I thought those single purpose devices didn't exist anymore.

Bought a relatively good router, installed OpenWRT on it then bridged it to the ISP's network via their equipment. It's working well. I even have HTTPS in my LAN now.

megous3• 2 years ago

One interesting thing I found was that the newer Vodafone cable modem with 4 ethernet ports, after switching to bridge mode, assigns public IPv4 address to at most 1 network node connected to each wired port. So it's possible to get 4 stable public IPv4 addresses assigned to my home network and use them for whatever.

It's not a great idea to host services (especially if they can be used to identify you) on a home IP address you browse the internet with, and this is one way to get 1 IP address for browsing the net, and different 1 for serving services from home, pretty much for free.

yonatan8070• 2 years ago

I just told my ISP, either I use my own router, or I switch to a different ISP.

jokoon2• 2 years ago

I remember creating some webserver at work years ago, and I saw a router querying it. I warned the company admin.

Also, my wifi firmware occasionally crashes and needs to be restarted.

I don't work in cyber security or on anything sensitive, but if I was told I'm under surveillance by some government or some criminal, I would not be surprised.

jokoon2• 2 years ago

What were those fbi redacted things? Were those backdoors?

phh2• 2 years ago

The above response contained the physical addresses of several FBI field offices who were Cox business customers.

That's literally just FBI's customer account at Cox.

kome• 2 years ago

clients

wouldbecouldbe• 2 years ago

This is seems like a huge vulnerability, are there any legal repercussion that happens in those situations?

uyzstvqs• 2 years ago

I hope not. Companies would close their responsible disclosure programs as a liability issue. Everything would be less secure because of such legal protections.

hifromwork3• 2 years ago

Agreed. On the other hand, there should be legal repercussions if the vulnerability was found exploited in the wild (in Europe this is partially handled by GDPR, but AFAIK only if it can be shown that personal data is affected - not a lawyer obviously).

This aligns incentives nicely:

* Company creates a responsible disclosure program, users/researchers report problems for money/blog post fame, users are secure. Also security team becomes more important, because vulnerabilities cost (more) actual money. * Or company doesn't create a responsible disclosure program, someone exploits the bug in the wild, users are angry and the company is fined.

zaptrem• 2 years ago

Why do y’all think the attacker was replying all of his requests? Could they be probing for unintentionally exposed endpoints themselves?

voidUpdate• 2 years ago

If it was a request to a bank, say, it could have included all the cookies and tokens that would allow the request to go through successfully, and the attacker would gain access to their bank page (though if it was something super high security, you'd hope it would have single use tokens and stuff)

cjbprime3• 2 years ago

A request to a bank that doesn't use TLS would be near-criminal negligence (by the bank) in itself.

If the request does use TLS, then even a compromised router should be unable to decrypt it. TLS is end-to-end encryption.

If the request doesn't use TLS, then the compromised router can already see the request and response that it is relaying. So why does it have to replay the request from somewhere else? It can just exfiltrate the session back to the attacker silently, without replaying it first.

If I had to guess, the attacker isn't sure what they're looking for in the HTTP sessions, so they can't push a detection for interesting sessions down to the compromised routers, and they also don't have the bandwidth to simply receive all unencrypted traffic from their router botnet, so instead they're collecting the URLs and building up a list of detection patterns over time through scanning and using heuristics for which requests are worth investigating, something like that?

walterbell• 2 years ago

Outsource cost/risk of reporting the vulnerability?

rnbrady• 2 years ago

If they didn't have RCE but could push config to the router, they might have pushed a syslog destination and then mined the logs. URLs for uncrypted HTTP requests could end up in the logs due to ALG, parental filtering or any other numbers of features. If you replayed such URLs from a large enough set of victims over a long enough period of time, you'd find something valuable in a response sooner or later.

jetbooster• 2 years ago

Along with the other stated reasons, it could be an attempt to cloak the IP as a normal residential IP by mirroring someone else's traffic

rwmj2• 2 years ago

After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse

Would you trust a thing they say? It seems their whole network is swiss cheese.

fragmede• 2 years ago

this is why everything gets logged to an S3 bucket under an AWS account that has only write permissions and three people are required to break into the account that can do anything else with that bucket. I don't know if that's what Cox has, but that's how it's architect it to be able to claim there's no history of abuse.

rwmj2• 2 years ago

That's how it should be architected, but the article shows that Cox's network gives no thought to security so it's unlikely how it is architected. Even if the Cox answer is correct to the best of their knowledge, we can't rule out that attackers are inside the network wiping out their logs.

randomcarbloke4• 2 years ago

if they say not, does that imply another vector that they may or may not know about given the author had already found a compromised device.

prettyStandard• 2 years ago

That was my first thought. That they didn't even find the original attack vector. But comments above this suggest something even worse they are in Cox's network actively wiping out their own logs.

halJordan• 2 years ago

You can't just refuse to participate, especially if you're the one who started the whole conversation. At some point you say "this is what i have and it's better than before."

wiz21c• 2 years ago

page is now 404 :-/

cjbprime3• 2 years ago

Not from here?

Akronymus• 2 years ago

Seems to have been a temporary hiccup, as I have no trouble accessing it.

biosboiii2• 2 years ago

Holy hell, but how are your laws in the US aligned so doing something like this is okay?

In Germany you would get minimum 3 years in jail for this, people got in front of court for way way way way less.

cjbprime3• 2 years ago

For the researcher? Because the vendor has a responsible disclosure program. Because they'd rather know about the bugs.

(As for the vendor, I'm sympathetic to the argument that there should be vendor liability under some circumstances.)

biosboiii2• 2 years ago

In Germany it is common for vendors to acknowledge the security flaw you send to them, but if you want to publish it (and damage their reputation by doing so) they are going to try you in court, and win.

Sometimes they even try you in court if you don't publish it (yet)

hifromwork3• 2 years ago

Cox has a responsible disclosure program: https://www.cox.com/aboutus/policies/cox-security-responsibl....

In my opinion (as a security engineer) the biggest benefit of such programs is not amoral "hackers will always sell exploits to the highest bidder so companies must provide a high bounty for bugs in their software"[1] but "having a responsible disclosure process makes it totally clear that it's ok to report vulnerabilities without being sued".

Looking at the timeline below the post I can't see anything problematic. The author even waited the usual[2] 90 days before disclosure, even though the vulnerability was hotpatched a day after report (congrats to Cox btw). They also shared a draft blog post with them a month ago.

[1]They certainly should, in the ideal world.

[2]A deadline popularized (or even invented) by Google's project zero.

porbelm• 2 years ago

Yeah when a company says one of their responsible disclosure rules amounts to "just don't ruin our prod system, or reveal or steal data pls" they basically invite you to try and break in - responsibly.

ThePowerOfFuet• 2 years ago

In Germany you would get minimum 3 years in jail for this, people got in front of court for way way way way less.

Great way to make sure researchers don't notify the victim of vulnerabilities, but rather stay quiet or sell it.

You'll note they never tried to change anything but their own equipment; doing otherwise would have been immoral and, yes, likely illegal. Without testing you have no idea whether or not you're actually looking at something that needs to be reported.

PawgerZ• 2 years ago

Germany is an outlier, not the norm, when it comes to security research

MyFedora• 2 years ago

You should advocate for laws that enable security research.

daneel_w• 2 years ago

"...and found no history of abuse..."

Because they didn't have enough logging or auditing to start with, or no logs or audit data left since the hack.

tuetuopay• 2 years ago

from what I can gather from the post, the specific attack vector using "retry unauthorized requests until they are" is very easy to spot in logs. so even the most basic log policy that logs the path, ip, and status code is enough (i.e. default in most web servers and frameworks)

JKCalhoun2• 2 years ago

Or they lied.

I mean, if you think about it from Cox's point of view — why would you disclose to someone outside the company if there had been history of abuse? Why would you disclose anything at all in fact?

redbar0n• 2 years ago

«Absence of evidence is not evidence of absence», seems to apply here.

psd1• 2 years ago

I see arguments in favour of tr069, but it's the mechanism that BT used to reboot my modem every night at 3am. I hate ISPs.

_joel2• 2 years ago

Unplug it then they can't reboot it :) They probably put it under the guise of load-balancing but that does seem excessive.

1023bytes• 2 years ago

Modem FW probably had a memory leak :)

Heidaradar• 2 years ago

I love how well he explains it, even to someone like me who knows p much nothing about cybersecurity.

longsword3• 2 years ago

i'm really glad that i can use my own modem. In germany every ISP is by law required to accept self brought modems. They can't force you to use their often shitty hardware. My current modem/router is up for 3 months without a single interruption to my connection.

nness• 2 years ago

I've noticed it gets quite murky when dealing with fibre-to-the-premises, particularly in the UK. Although I don't think an ISP would disallow BYOD, I imagine they'd just not be as likely to support it.

I recently moved ISP, partly because of cost, but also because they offered a great home router as part of their bundle. The installer could not utilise any of the existing wiring in my house, had to be all drilled a second time...

Conversely, my last ISP used some awful Nokia modem that barely supported any kind of routing or customisation and I picked them specifically because it was a rental and the fibre wiring had already been done.

It's fairly common for ISP's in Australia to also give you a choice of BYOD or buying one of theirs. Usually you pay outright for the modem, however, so its yours to keep. That said, this is changing with the national fibre roll-out. But with ADSL being the de-facto choice, BYOD makes sense.

vladvasiliu• 2 years ago

I've noticed it gets quite murky when dealing with fibre-to-the-premises, particularly in the UK. Although I don't think an ISP would disallow BYOD, I imagine they'd just not be as likely to support it.

In France, I've noticed that some ISPs (Free for FTTH and SFR for FTTC + cable attached to the router) they'll offer the possibility of configuring the provided router in "bridge" mode, where you basically get the external IP to whichever equipment is hooked up to their router.

I've also had FTTH with SFR, which have a separate device which terminates the optical connection (ONT) and speaks ethernet with the main router. I don't remember if the main router was able to work in bridge mode. It was possible to connect your own router to the ONT but you had to jump through hoops [0] to actually receive a working DHCP response.

Bouygues also had the separate device for terminating the optical connection, connected via ethernet to the main router. The only catch was that it talked over vlan 100 for some reason, but other than that it was smooth sailing.

I've never had Orange, but I hear it's a pain to replace the actual router with them.

---

[0] IIRC you had to send some custom DHCP options pretending more or less to be an actual SFR router.

longsword3• 2 years ago

BYOD is really easy here with DSL. With fibre they try to push the legal limits by arguing that their device is required because its part of their network. But because the law explicitly says the network has to end at a passive point, they got fined. If they don't provide the login and vlan id's, you can just report them to the authorities and they will handle it.

edward28• 2 years ago

With Aussie NBN most providers use pppoe or dhcp, which allows byod or ISP router.

tengwar2• 2 years ago

I've got FTTP in the UK (from Andrew's and Arnold). There's a provided bridge which takes the fibre connection and relays PPPoE to my own router. I believe that's a common way to do it.

llm_nerd• 2 years ago

I use the ISP's modem (separating my own choices from having to worry about upgrades to DOCSIS and the like), but then hang my own router off of it. For that matter I have multiple routers inside the cable router for various tiers. With secure DNS and almost universal TLS, and then the firewall that are the internal private networks, the threat profile is negligible.

aidenn03• 2 years ago

Can the ISP load firmware onto your modem? I'm on Cox in the US (same ISP as in TFA) and you can bring your own modem, but Cox will remotely update the firmware.

jeduardo• 2 years ago

Not really as far as I know. Providers in Germany have more or less standardized on Fritz!Box from AVM and the router comes with the admin password available. Updates are then fetched from upstream AVM.

But the key point here is device independence - by law, providers need to give you all information required to establish a connection to them. This allows you to run a Linux or BSD box as a router should you wish to. It somehow makes up for the slow broadband speeds you can get.

*Edit: complaints about slow broadband speeds

longsword3• 2 years ago

No, they can't. They don't have any access at all to your device. But as jeduardo already said, you can fetch updates from the device manufacturer. The mentioned Fritz!Box from AVM has automatic updates and is known for delivering them for a really long time. My 12 year old repeater from that brand is still receiving security updates from time to time.

imzadi• 2 years ago

FWIW, you can use your own modem and router with Cox internet, but most people don't because the provided modem is free and most people don't care to spend money on their own.

matthew-wegner• 2 years ago

Cox can (and does) push firmware to bring-your-own modems, though.

arrty88• 2 years ago

One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd.

Did you determine if POSTs were replayed? As in, logging into accounts and sending payment info and account info?

andrewstuart• 2 years ago

> Authenticate your access patterns.

What does this mean?

taink• 2 years ago

  One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd.

I was thinking about this while reading. My guess is that the vulnerability was limited to reading incoming requests (to the modem) or something along those lines, not full control of the network. Replaying the requests is a good way to get both ends of the traffic if you can only access one. For instance, a login + password being authenticated. Just a thought!

EDIT: I'd be hard-pressed to know how one could exploit this, given TLS would encrypt the requests. Maybe they're counting on using badly encrypted requests, encrypted with e.g. TLSv1.0?

Bluecobra2• 2 years ago

What sucks about this situation is when your ISP forces you to use their modem or router. For example, I have AT&T fiber and it does some kind of 802.1X authentication with certificates to connect to their network. If they didn't do this, I could just plug any arbitrary device into the ONT. There are/were workarounds to this but I don't want to go through all those hoops to get online. Instead, I ended up disabling everything on the AT&T router and have my own router that I keep up to date plugged into that. Unbeknownst to me, the AT&T router could be hacked and I would never notice unless it was adversely affects my service.

Thank god most things use HTTPS these days.

DannyBee• 2 years ago

Fwiw: the hoops are automated these days if you are on xgspon.

It's "plug in sfp+, upload firmware using web interface, enter equipment serial number"

You can even skip step 2 depending on the sfp stick you use.

The 802.1x state is not actually verified server side. The standard says modems should not pass traffic when 802.1x is required but not done. Most do anyway or can be changed to do so. AT&T side does not verify, and always passes traffic. That is what is happening under the covers.

vel0city3• 2 years ago

The CPE AT&T router potentially getting hacked doesn't make much difference if you have your own router between your network and the AT&T network. Even if we removed the AT&T CPE router, you'd still be connecting to a black box you don't control that could be hacked or doing any number of inspections on your traffic.

batch12• 2 years ago

It does matter since it lets an attacker be between your network and the internet. If that black box is a modem- yes it could be hacked, but (maybe luckily for me) the providers I've used don't expose many services from the modem on the public interface so it's much more difficult to compromise. You'd either have to come from the docsis network or the client network.

the_duke• 2 years ago

Always put your own router in-between.

If you really care you can configure a VPN directly on the router, so nothing leaves the network unencrypted.

mh-3• 2 years ago

Fortunately, Cox isn't one of these. Any sufficiently modern DOCSIS modem, appropriate to the speed of service you subscribe to, is accepted.

Unfortunately, my praise of Cox ends there. I've been having intermittent packet loss issues for 2 years, and there doesn't appear to be a support escalation path available to me, so I can't reach anyone that will understand the data I've captured indicating certain nodes are (probably) oversubscribed.

toast02• 2 years ago

If you have the att fiber with the ONT separate from the modem, it's really easy to bypass 802.1X. Plug an unmanaged switch in between the modem and the ONT; let the modem auth; disconnect the modem. You'll likely need to do that again if the ONT reboots, but at least for me, ATT a UPS for the ONT, so reboot frequency should be low.

Personally, I built up a rube goldberg of software and hardware with bypass nics so if my firewall was off (or rebooting), traffic would flow through their modem, and when it was on, my firewall would take the traffic and selectively forward traffic through from the modem, but there's really no need for that when you just use an unmanaged switch. I can find the code if you're interested (requires FreeBSD), but you sound more sensible than that ;)

Bluecobra2• 2 years ago

That's a good idea, I do have an extra UPS/switch I can use for this. In the past when I was a bachelor and had more free time, I used to run my own FreeBSD server with pf and other services running in jails. Now that I am settled down, I just want to make things as idiot proof as possible in case there is an Internet issue at home and another family member needs to fix it.

The XGS-PON workaround that DannyBee looks promising though:

https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500...

I probably could pay to upgrade my speed to 2Gbps and then downgrade it back to 1Gbps and keep the XGS-PON.

gh02t• 2 years ago

If you have a router running PfSense Plus* and at least 3 ports, Netgate actually has pretty detailed instructions for how to do the bypass with their layer 2 routing feature. It sounds a bit complicated, but I followed along exactly as it says and it just worked for me. Has been 100% reliable for almost 2 years, and I get significantly better speed (something like 10-20% vs the built in "passthrough" mode on the gateway, iirc). Plus I managed to cut the suspicious DNS server the gateway tries to interject out of my network.

https://docs.netgate.com/pfsense/en/latest/recipes/authbridg...

There's another method that doesn't require Plus called pfatt, but I'm not sure what the state of it is.

* Plus is the paid version, yeah I know I agree I don't like what they did with the licensing changes but that's a different story

hsbauauvhabzb• 2 years ago

How does that bypass 802.1x? Are the 802.1x packets are responded to by the official modem still? I was under the impression all packets were encrypted or signed with 802.1x, but Ive never had to implement or test it so I could be wrong.

RulerOf• 2 years ago

It was mentioned by a sibling, but there are ways to connect without using one of AT&T's gateway devices. Different methods are catalogued on https://pon.wiki/

ryukoposting• 2 years ago

That's why I'm not an AT&T customer. Spectrum lets me bring my own hardware, and they're the only other option in my area, so Spectrum gets my business. Plain and simple. Unfortunately, not everyone has the palatable solution that I have.

mrguyorama2• 2 years ago

Spectrum remote manages your hardware even if you bring your own modem. This nearly entirely consists of deploying firmware updates once a decade, but they can also command other things like modem reboots.

outworlder• 2 years ago

Like someone else mentioned, at some level you need to rely on your ISP and it is also a good idea to have a router in between anyway.

I would like to bypass the BGW320 because not only it is a large, power hungry box, but it also requires me to jump through hoops to get IPV6 working with VLANs. I need to either use multiple physical links (simulating multiple devices) or simulate that using a VRRP hack, otherwise AT&T will not give out multiple ranges at all (and will not care about what I request). Under Comcast I didn't have to do any of that, I'd just carve out smaller IPV6 ranges, as many as needed.

peter_d_sherman• 2 years ago

Observation: The root of this problem is NOT because Cox's engineering practices lacked a comprehensive enough security review process to find and fix security vulnerabilities prior to them being discovered post deployment ("hindsight is always 20/20" as they say), but rather because there was (and still is) an Information Asymmetry between Cox and Cox's customers, i.e., in terms of complete knowledge of how Cox's devices actually work under the hood...

Although, in fairness to Cox, this Information Asymmetry -- also exists between most companies that produce tech consumer goods and most tech consumers (i.e., is it really a big deal if most other big tech companies engage in the same practices?), with the occasional exception of the truly rare, completely transparent, 100% Open Source Hardware, 100% Open Source Software company...

https://en.wikipedia.org/wiki/Information_asymmetry

Anyway, a very interesting article!

kn100• 2 years ago

Great read, and fantastic investigation. Also nice to see a story of some big corp not going nuclear on a security researcher.

I can't say for certain, and the OP if they're here I'd love for you to validate this - but I'm not convinced requests to the local admin interface on these Nokia routers is properly authenticated. I know this because I recently was provisioned with one and found there were certain settings I could not change as a regular admin, and I was refused the super admin account by the ISP. turns out you could just inspector hack the page to undisable the fields and change the fields yourself, and the API would happily accept them.

if this is the case, and an application can be running inside your network, it wouldn't be hard to compromise the router that way, but seems awfully specific!

JKCalhoun2• 2 years ago

Cox is the largest private broadband provider in the United States, the third-largest cable television provider, and the seventh largest telephone carrier in the country. They have millions of customers and are the most popular ISP in 10 states.

That suggested to me that we shouldn't have ISPs that are this big. Cox is clearly a juicy target and a single vulnerability compromises, as an example from the article, even FBI field offices.

After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse

Feel like author should have written "...they claimed to have investigated...".

mh-3• 2 years ago

I think the author wrote it up factually. Readers can make their own inferences, but Cox did share with him that the service he exploited was only introduced in 2023. Which suggests the security team did do some investigating.

I'm sure* they don't keep raw request logs around for 3+ years. I know what next steps I'd recommend, but even if they undertook those, they're not sharing that in the ticket.

(just based on industry experience; no insider knowledge.)

amluto2• 2 years ago

Some CPE exposes an API on the LAN side, and some of these APIs aren’t protected against CSRF. I wonder whether the modem in question is vulnerable.

mh-3• 2 years ago

Browser security enhancements have made enumerating those a lot more difficult, but a quick google suggests there were still tricks to achieve DNS rebinding as recently as 2023. Very possible.

amluto2• 2 years ago

I can probably guess a cable modem’s IP address and a crappy CPE router’s IP address in one guess each. Enumeration isn’t usually the problem.

mrbluecoat• 2 years ago

Great article, but unfortunately a determined threat actor would just go to the source and get a remote job as a Cox technician to gain access to millions of routers to add to their botnet. A real solution by the ISP would be to implement a software (or, preferably, hardware) setting that prevents remote access by default unless explicitly enabled by the customer. That approach would slow a social engineering campaign and limit the scope of a hack like this.

EligibleDecoy• 2 years ago

My bet on the replays was that the attacker misconfigured their payload or something and it was meant to replay command and control requests to obfuscate where the C2 server was

sammy22552• 2 years ago

No payout?

jfyi4• 2 years ago

It can be inferred that the author is satisfied with that aspect of the transaction by their willingness to list things that they still felt unresolved at the end.

They either were paid and think it's nobody's business or weren't and have no ideological reason for making a stink. For what it's worth, I sympathize with people who feel shafted for their work by large companies, but think it is a little silly to go looking for it.

syngrog66• 2 years ago

WARNING: nerd sniping. lol

mannyv3• 2 years ago

The intermittent auth thing in /profilesearch is a sign that they're round-robinning the servers and misconfigured one.

Also, it looks like he hit a front-end API that drives the TR-069 backend. Changing the WiFi SSID is a long way from being able to "...execute commands on the device"

axoltl• 2 years ago

Is changing the WiFi SSID not executing a command on the device? It isn't _arbitrary_ commands (yet), but it's definitely executing _a_ command.

mannyv3• 2 years ago

That's not the kind of vulnerability that would have installed an exploit on their CPE.

easterncalculus• 2 years ago

Being able to change the SSIDs for thousands or millions of customers remotely within a few hours would definitely be written about as an “outage” for Cox though.

mannyv3• 2 years ago

An open question is still: how were the attackers able to grab his HTTP traffic?

Some CPEs have a cloud Wireshark-like capability for debugging. I'm not sure if those are even on the Cox production firmware images. Usually there's a set of firmware for production and a set for test (which obviously makes it hard to test for problems in production).

I suppose Cox could do a check to see what firmware versions are out there. ISPs can auto-upgrade firmware that doesn't match a specific firmware revision, and this was a Cox modem so they probably have firmware for it. So if it was a debug firmware how did it get there and survive?

ipython4• 2 years ago

also, yet another reason I don't trust (and don't use) any ISP provided equipment. Remote administration from my ISP? No thank you.

whatwhaaaaat2• 2 years ago

Even if you buy your own modem they can push firmware to it (and do). The config file your modem downloads includes a cert that allows the isp to do this. You can flash special firmware (used to be called force ware) to prohibit this.

accrual• 2 years ago

How about putting the ISP supplied modem in a DMZ? Then the ISP could admin it all they want but still never touch the LAN.

moooo99• 2 years ago

I get the perspective, but I also like the fact that ISPs do take over some of the admin burden associated with running a piece of equipment like a router.

You, I and most of the HN crowd may be well capable of maintaining a reasonably secure state of our own hardware and troubleshoot our way through common errors. However, the average internet user isn’t that experienced nor are most people interested in learning those skills.

sammy22552• 2 years ago

Its HTTP not HTTPS, anyone or anything on the wire could see the request

globular-toast• 2 years ago

That's the part I didn't get. The author said there was no other possibility except the modem, but why? It seems like quite a leap. I would have first suspected a compromised router on the internet. Is it possible that changing the modem caused new routes to be used which appeared to fix the problem?

mmsc• 2 years ago

If you create a socket with PF_PACKET you can intercept all the traffic on a Linux system on all interfaces. Think of a low-tech version of tcpdump.

Intercept all data on port 80, parse the http headers, do whatever you need with them, easy.

Not sure why anybody would replay the requests though.

stonks• 2 years ago

Many routers require manual firmware updates. GL.iNet routers had several RCE (Remote Code Execution) vulnerabilities within the last 6 months. I advise you to have a quick look in your own router to ensure its not hacked, and possibly upgrade firmware.

As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and personal devices either don't see it, or struggle to connect. At the same time the router is still connected to the internet - router's internal admin page (192.168.8.1) stopped responding

I imagine many users haven't updated their routers and thus may be hacked. In my case the hacker installed Pawns app from IPRoyal, which makes the router a proxy server and lets hacker and IPRoyal make money. The hacker also stole system logs containing information about who and when they use the device, whether any NAS is attached. They also had a reverse shell.

Solution: 1. Upgrade firmware to ensure these vulnerabilities are patched. 2. Then wipe the router to remove the actual malware. 3. Then disable SSH access, e.g. for GL.iNet routers that's possible within the Luci dashboard. 4. Afterwards disable remote access to the router, e.g. by turning Dynamic DNS off in GL.iNet. If remote access is needed, consider Cloudflare Tunnel or Zero Trust or similar. There is also GoodCloud, ZeroTier, Tailscale, etc. I am not too sure what they all do and which one would be suitable for protected access remotely. If anyone has advice, I would appreciate a comment.

Consider avoiding GL.iNet routers. They do not follow principle of least privilege (PoLP) - router runs processes using root user by default. SSH is also enabled by default (with root access), anyone can try to bruteforce in (10 symbol password consisting of [0-9A-Z] and possibly might be more predictable). I set mine to only allow ssh keys rather than a password to prevent that. Despite running OpenWrt they are actually running their own flavor of OpenWrt. So upgrading from OpenWrt 21.02 to 23.05 is not possible at the moment.

daghamm• 2 years ago

"As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and..."

Could also be the neighbours and their big microwave oven :)

kernal• 2 years ago

Moral of the story - never turn on remote access on your modem.

cdaringe• 2 years ago

Nightmare fuel. Giant tech company, giant vuln. There’s so much to say, but more than anything Im just upset. The article and this dude are amazing. The exploit is not excusable.

goshx• 2 years ago

It's unbelievable that Cox offers no compensation or reward for incredible work like this.

worewood3• 2 years ago

Not trusting the modems we're given is a damn good reason to use a VPN, as opposed to the market bulsshsait the VPN companies usually propagate

underlogic2• 2 years ago

Did they *pay* him? He kind of saved them, tipped them off to a complete compromise of their security infrastructure which was not trivial to discover. Looks like he got nothing in return for "doing the right thing". How insulting is that? What is their perception of someone walking in to their offices with this essential information? I guarantee his self image and their perception are very different. They see an overly caffeinated attention seeking "nerd" just handed them a 300k exploit in exchange for a gold star and then they ran like smeg to cover their asses and take all the credit internally. He feels like superman, goes home to his basement apt, microwaves some noodles and writes a blogpost. This is a perfect example why you never, never report a 0day.

downrightmike• 2 years ago

Its Cox, probably lucky if they don't sue him for fixing their mistake

underlogic2• 2 years ago

It happens. This is the type of revelation where heads roll and a scapegoat is very useful for the CSO, general liability of the company and PR.

Biganon3• 2 years ago

Cox don't pay bounties.

rtev• 2 years ago

Sam is a very famous security researcher, so I would be shocked if he wasn’t making upwards of $350,000 a year. These articles he writes make him a significant amount of money via reputation boost.

codedokode• 2 years ago

Replaying requests might be not a malicious attacker, but simply an ISP wishing to know and sell customer's interests.

_benj• 2 years ago

Wow! I just what a high a security researcher would feel while performing this research and keep finding open doors!

I wonder if it’s a mix of exhilaration and being terrified!

__turbobrew__• 2 years ago

Another reason to not use ISP provided hardware. I have never had issues using my own OpenBSD box as a router.

TeMPOraL• 2 years ago

Great writeup. There's just one thing I don't get: the auth part. It seems the author managed to access protected endpoints without any auth, by just repeating the same request over and over until the endpoint randomly accepted it. The part that confuses me is, how could that possibly happen? What possible architecture could this system have to enable this specific failure mode?

I struggle to think of anything, short of auth handling being a separate service injected between a load balancer and the API servers, and someone somehow forgot to include that in autoscaling config; but surely this is not how you do things, is it?

Affric• 2 years ago

Test server from early development put into production?

definitelyauser• 2 years ago

how could that possibly happen?

Global singleton shared across requests, instead of request scoped.

1. [Client 1/You] Auth/write to variable (failed).

2. [Client 2/ISP] Auth/write to variable (success).

3. Verify what the result was (success)

A race condition combined with a global singleton can easily explain such behavior.

spopejoy• 2 years ago

The article mentions Spring, although I couldn't see anything in the output that would tip me off (like massive java.lang.xxxException traces) ... plus I've seen other mentions of singletons in this discussion --

Are you describing some kind of server-side global object that statefully says a session/api key is "authenticated" and will then allow the request during that time frame? That seems like a bug you could drive container ships through. Yes I know saas s/w sucks out there but this would seem to at least be something an audit could easily flag.

system2• 2 years ago

It was so fun to read this. I am also surprised COX hot patched it within a day.

metadat• 2 years ago

there were about 700 different API calls..

That's more API endpoints than some first tier public clouds, wow. For a modem.

Somebody wanted (and sorta deserves) promo..

But also not, because the whole platform turned out to be incredibly insecure! Egregious!!!

webninja2• 2 years ago

Are there any creation of new laws or removal of hindering laws that would facilitate the fixing of these devastating security vulnerabilities?

gianpaj• 2 years ago

This reminded me to turn off "privacy settings" to "keep your vehicle in good condition and observe the vehicle's health" on my Volvo XC40 after the mechanics asked me to turn it on yesterday during the yearly maintenance. I don't know if they can change some settings remotely, but I prefer to be cautious

Namidairo• 2 years ago

Somehow, someone was intercepting and replaying the web traffic from likely every single device on my home network.

Normally I'd laugh and assume device compromise but...

The largest ISP in Australia (Telstra) got caught doing exactly this over a decade ago. People got extra paranoid when they noticed the originating IP was from Rackspace as opposed to within Telstra. Turned out to be a filter vendor scraping with dubious acceptance from customers. The ToS was quietly and promptly updated.

lanrat• 2 years ago

I observed very similar behavior a few years back when transferring files between two servers under my control on different parts of a large university network.

We also initially thought we were the subject of a breach, but after the investigation we determined that the network's IDS was monitoring all traffic, and upon certain triggers, would make identical requests from external networks.

We found a way to identify all other similar IDSs across the internet and even "weaponize" this behavior. We ended up writing a paper on it: https://ian.ucsd.edu/papers/cset2023_fireye.pdf

Hacking millions of modems and investigating who hacked my modem

Related Stories