Pulling MikroTik into the Limelight Demystifying and Jailbreaking RouterS

https://margin.re/2022/06/pulling-mikrotik-into-the-limelight/

by 882542F3884314B • 4 years ago

135 57 4 years ago

margin.re

aix1• 4 years ago

Nice work. Pity all the images in the article are broken.

dgl2• 4 years ago

and also the GitHub repo with the presentation link is a 404.

sliken6• 4 years ago

Maybe a DRM takedown?

__comedian• 4 years ago

apologies, website migration and resource issue. Images and links are fixed now

sliken6• 4 years ago

Seems like MikroTik and Ubiquiti haven't been particularly good on documenting how their routers work, what each process does, and how to debug/verify what they are doing. I've been particularly worried about Ubiquiti, since they seem to be ignoring the MIPS based EdgeRouter line. The various consumer NAS vendors have been a nightmare security wise, and it's far from clear that the prosumer routers from Ubiquiti and MikroTik are any better.

Should it really require reverse engineering to figure out how a Linux box passes packets?

I gave up on Ubiquiti and bought a tiny $120 router ($140 with a nice metal case) that's a NanoPi R6s. Pretty impressive specifications. 8GB ram, 8 cores (4 fast Ax76 and 4 slow A55s) and no fans. Has 2x2.5GBe and 1x1GBe for networking. I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS. I didn't bother cross compiling, it's plenty fast for native compiles.

I've been impressed so far. It compiles Rust about half as fast as my Quad core Xeon server from years ago, and is SEVEN times faster than a RPi 4 8GB! I have an Apple M1 mini around that manages the same compile in 13 minutes. A nice bonus is neither network or storage is USB attached, unlike the RPi.

Burned it in overnight, running all 8 cores flat out, with no problems. Haven't decided what to use, maybe shorewall or just raw nftables/iptables.

Similarly there's 4 and 6 port 2.5 Gbe boxes with various N5000 and N6000 celerons at around $200. ServerTheHome has reviewed many of them, they seem to be evolving nicely, better cases, better heat sinks, better 2.5Gbe Intel chips, etc. Run *WRT, pfsense, or whatever else floats your boat. STH even demod running the firewall under proxmox on one of these cheap 4-6 port 2.5GBe widgets. Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).

Guess I'm just getting less trusting in my old age.

Arnavion2• 4 years ago

Don't know about the Linux distros, but one problem with the FreeBSD versions (pfSense, OPNsense) is the mostly nonexistent WiFi support because there's no driver support for PCIe cards. You end up needing to hook up a COTS Wifi AP, which has all the same problems that a COTS router would've had.

But other than that, yeah I don't see any reason to use a COTS router when I can just make one. My current router is my old desktop PC that became redundant when I built a new one, sitting as a bare motherboard on a test bench to take up very little space, running OPNsense.

johnklos2• 4 years ago

How does a common wireless AP have the same issues as a common consumer NAT router? Do APs have to worry about NAT state tables? Are they exposed to the Internet? Do they have to be configured to NAT/forward special edge case devices, like Nintendo / Sony games consoles?

Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.

glogla• 4 years ago

I think building it yourself works well for routers and NASes, but there's a gap for WiFi. Lot of the bells and whistles that make modern wifi perform well (beamforming, MIMO, ...) is locked behind proprietary software and your homemade access point will never work all that well.

Interestingly, this is also something MikroTik isn't very good at - their long-range wifi is nice, but their consumer wifi used to be stuck at 2x2 and basically worse than cheap Asus "wifi router" which is somewhat embarrassing.

For switches, if you want fast, for small installation it is hard to beat the Mikrotik 4xSFP+ 10G switch for $150.

jcfhchcjxhx• 4 years ago

you are drinking too much of their marketing.

Not a single wifi consumer vendor have any say on those features. it is one hundred percent done on the closed OS in the radio chip.

your expensive ubiquiti/oanda/cisco have the same mimo/beanforming performance as any opensource/clone using the same wifi radio chipset.

the consumer facing OS just flip a bit somewhere in the configuration flash. granted, knowledge of the right bit to flip mighty be missing on the opensource still, but there's no magic happening in ubiquity owned code.

uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.

pharos92• 4 years ago

Are you using the images available from FriendlyElec? I have limited trust in them.... Been struggling to find a good image except for DietPi.

sliken6• 4 years ago

Heh, it came with FriendlyElec, I just used it long enough to make sure the hardware was working and stable. I always reinstall Linux from something trusthworthy upon receipt.

However all the NanoPi images are pinned to the same 5.10.x kernel, that I don't think it upstreamed, which is pretty ugly.

Armbian just had a new release, supporting a TON of SBCs, even a Risc-V board, which is weird based on the name. In any case, they support another RK3588 board, so I was going to try that. I want something using a mainline linux kernel if at all possible.

dgl2• 4 years ago

I've also replaced a Ubiquiti EdgeRouter with a spare PC I had around. It was a small form factor thing, I don't need WiFi, so I replaced the WiFi card with a M.2 Ethernet card[1], which gives me two network interfaces. The point is pretty much anything is powerful enough to be a router now.

(Actually my setup is more complex, as the machine is powerful enough to run a few VMs, so I pass through one of the PCI devices to a firewall VM, which means the VM host doesn't even see the outside world.)

One thing to watch out for is Intel released some bad 2.5Gbe hardware (e.g. earlier versions of the I225-V). I even have a more recent one that seems to have issues (even when running the latest Linux from kernel.org). You can find Realtek cards though, which seem to work better (yeah, who'd have thought that would be the case).

[1]: It was something like: https://www.amazon.com/Suroene-Ethernet-Adapter-Multi-Gigabi... -- there are others.

sliken6• 4 years ago

Apparently I225 went through 2-3 revisions, and the "b3" is supposedly ok. But the newer $200 PCs are now coming with the I226, which it's claimed is much better.

ThePowerOfFuet• 4 years ago

Re that M.2 Ethernet card, that's a long untwisted ribbon cable. 100Mbit should be fine, but 1G or 2.5G? Sus.

squarefoot2• 4 years ago

There also are dual 2.5G Ethernet (and multiple SATA) adapters for M.2 sockets, which make things even more interesting for those little ARM boards.

https://www.aliexpress.us/item/3256804337466480.html

https://www.aliexpress.us/item/3256804400925663.html

OpnSense/XigmaNAS devs, are you listening?

tibbydudeza2• 4 years ago

I bought one of the Edgerouters thinking it would tie into their Unifi Web controller interface - no hells bells - got a Mikrotik like device where you need Cisco IOS level of skill to configure.

Got a TPLINK Omada ER605 router now - just as powerful but way easier to configure.

core-utility• 4 years ago

got a Mikrotik like device where you need Cisco IOS level of skill to configure

That's giving it too much credit. I have experience with Cisco IOS and NexusOS, and I'll say that Mikrotik's RouterOS was hard.

somat• 4 years ago

The funny thing is that when I was big on ubiquiti I always went the other way edgerouter only. mainly because I did not want to get tied into the whole unifi ecosystem.

They sold two versons of their switches and routers, the black ones(good) which were stand alone and the white ones(bad) which depended on the unifi system.

But then again I am also the idiot who ran openbsd on my edgerouters.

MikusR• 4 years ago

got a Mikrotik like device where you need Cisco IOS level of skill to configure

You have not used a Mikrotik wifi router in the last 10 years.

rjsw• 4 years ago

The 1x1GBe port on your system is the controller built in to the SoC. It supports jumbo frames so you can do full size packets over PPPoE if your ISP can handle that.

RobotToaster• 4 years ago

I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS.

Which one did you decide to stick with? Or are you using both?

The problem I've had with OpenWRT is updates are a pain.

c0l0• 4 years ago

OpenWrt upgrades can be made MUCH more convenient by installing extra packages that enable "Attended Sysupgrade" - check https://openwrt.org/docs/guide-user/installation/attended.sy... for more info, and please spread the word if you like it! :)

luma• 4 years ago

You've thrown out a lot of performance numbers here but most of us aren't compliling rust on our routers. Have you stood up iperf to see how fast it routes? The task is harder than you think if you want any features to go with that and there's a good reason most vendors (Mikrotik and Ubiquiti included) offload most of the task to silicon.

0cf8612b2e1e• 4 years ago

I suspect the majority of consumer gigabyte routers are never saturated. The technology has eclipsed typical demand. Not that there should be deceptive marketing or performance cheating, but I suspect 95%+ of people will never be limited by their network gear.

sliken6• 4 years ago

Yeah, sure, but it's frustrating. Sure casual users who won't saturate the network get fast offloaded performance.

However more aggressive users will end up enabling more complicated firewall rules, QoS, buffer bloat mitigation, or deep packet inspection and disabling the silicon based acceleration or bottlenecking elsewhere.

Thus my interest in the RK3588, that provides 4x the PCIe of the RPi4 and a fair bit more CPU.

I do wish there was a popular benchmark that used a more realistic benchmark. Smaller packets, dozens or 100s of TCP sessions, non-trivial firewall rules, etc. Doesn't seem like you learn much from iperf managing 95% of of line rate with large packets.

But here's the iperf numbers I stumbled across: https://www.cnx-software.com/2022/11/12/nanopi-r6s-review-un...

mindslight• 4 years ago

If you're considering straight nftables, do straight nftables. The syntax isn't bad for editing/reading directly. The userspace tool has quirks but you'll get the hang of it. I run my config through a templating engine (the same ruleset goes to cloud VMs etc).

My main router is virtualized on a Ryzen 5600G that also does other tasks. I have two other wifi APs that are just low power x86 boxes that also run Kodi.

Getting back to the original topic, I've still got Mikrotik for switches. The hardware is decent, the software is janky. I've got three different flavors of config - SwOS, RouterOS, and old RouterOS that requires a different type of config because the older switch chips were never updated to the newer config commands.

Mikrotik continues to hold their software too tightly instead of making the transition to a hardware company. They've built so much routing functionality, that's just pointless on devices that are switches. Yet there is some switch functionality that isn't available on SwOS. And SwOS is web-only with a binary config backup, rather than something conducive to centralized administration.

I'd love to find some low power white box 1Gb+10Gb switches that could run straight open Linux, or even using my existing Mikrotik switches that way if someone were to forcibly open their the environment.

kwesthaus• 4 years ago

Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).

I did this with my (mostly) OpenWrt-based network: https://github.com/kwesthaus/network-config.

I chose the NanoPi R4S instead of the R6S as my router since it is fully supported by vanilla OpenWrt [1] whereas the R6S is still in the works [2].

[1] https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1

[2] https://forum.openwrt.org/t/nanopi-r6s-a-rockchip-rk3588s-ro...

sliken6• 4 years ago

I saw an Armbian release for a similar RK3588 sbc, hopefully that image works and had a mainline kernel.

halfsquatch• 4 years ago

Slide deck from the con's site: https://cfp.recon.cx/media/2022/submissions/TZUFS8/resources...

Slide deck from margin.re: https://margin.re/content/files/2022/11/Pulling_MikroTik_int...

Blurb about the presentation at the con's site: https://cfp.recon.cx/media/2022/submissions/TZUFS8/resources...

Blurb about the presentation at margin.re: https://margin.re/pulling-mikrotik-into-the-limelight-2/

tibbydudeza2• 4 years ago

Thanks.

tapper2• 4 years ago

I never run anything that can't be flashed with OpenWrt!

xx_ns3• 4 years ago

A lot of MikroTik devices run OpenWRT just fine. I never really see the point in it - RouterOS is an excellent.. well, router OS.

zoobab3• 4 years ago

I try to avoid proprietary OSes.

tapper2• 4 years ago

Not going to run that locked down crap on my network.

squarefoot2• 4 years ago

No doubt about the quality, I already have some Routerboards around and will be purchasing some of their ceiling APs and a couple switches at the new house soon. All the worries come from the software being closed; I'm not painting Mikrotik evil, it's just that closed software and firmware are the ultimate place in which place spyware, especially where all software is FOSS. Routers and firewalls are the devices seeing every single piece of data on the network, thus is mandatory for them to be trustworthy, a quality the good reputation of their manufacturer can indeed suggest, but not guarantee.

acd• 4 years ago

Why do we have to jailbreak thins using opensource software? EU right to repair, should be extended to right to patch, hack, tinker.

bestham• 4 years ago

The EU is actively hindering consumers from getting into the radio of wifi-APs with the Radio Equipment Directive.

zoobab3• 4 years ago

Blame the Germans, those were the ones responsible for the FCC Radio Lockdown.

The EU imposed software radio lockdown, there was a public consultation about it where most of the answers from industry and users were negative, but they kept on pushing for it.

The same kind of consultation happened in the US, with the same kind of ignorance.

avidiax• 4 years ago

Same thing with the FCC.

We could have cooperative use of the airwaves and true open-source, open-hardware radios, but there's too much money to be made auctioning spectrum and assuring that most of it is is 99.99% unutilized. Would be horrible if people had cheap 5-10W handhelds that use advanced codecs, or could run 5mw of power on any part of the spectrum.

execveat2• 4 years ago

AFAIK, for a long they were pretty hostile and there's still this webpage which asks a $45 wire transfer in order to receive a CD with open source components used in the routerOS: https://mikrotik.com/downloadterms.html

That might have changed now, but I still can't find an official repo. You could look for older versions like this: https://github.com/robimarko/routeros-GPL or try to contact them directly (or sue if you think you have a standing).

JustFinishedBSG• 4 years ago

I have asked for sources recently and they just sent me a link to a .tar the next day.

xx_ns3• 4 years ago

I recently discovered a vulnerability and jailbroke RouterOS 7.4 (beta). As far as I know, this is the latest jailbreak for RouterOS (as of today):

https://nns.ee/blog/2022/08/05/routeros-container-rce.html

execveat2• 4 years ago

Awesome work! Also, thanks for pointing out that RouterOS supports Docker now, there's no way this is the only vuln in that implementation.

michalhosna• 4 years ago

It is a feature with warning everywhere. You'll need physical access to the device to enable docker.

I think is still a very nice feature to release even if it's no complete and fully secure yet.

your router is as secure as anything you run in container; if you run container, there is no security guarantee of any kind; running a 3rd party container image on your router could open a security hole/attack vector/attack surface;

https://help.mikrotik.com/docs/display/ROS/Container#Contain...

bobleeswagger• 4 years ago

It started with Synology, I'm not sure I like the world where docker runs on anything.

bulbosaur123• 4 years ago

Some guy yesterday was trying to crack this on /g/ board with ChatGPT. Search the threads archive and you might find something related to this code.

zoobab3• 4 years ago

Slides are here, their link on the paghe to the PDF did not work https://margin.re/content/files/2022/11/Pulling_MikroTik_int...

zevlag• 4 years ago

Here is the archive.org link now that the original link is 404'ing.

https://web.archive.org/web/20221208070504/https://margin.re...

withzombies• 4 years ago

It's back

Pulling MikroTik into the Limelight Demystifying and Jailbreaking RouterS

Related Stories