Multipath TCP for Linux (2022)

https://www.mptcp.dev/

by pantalaimon • 2 years ago

232 102 2 years ago

vlovich123• 2 years ago

Why does this require explicit opt in by applications if there’s transparent fallback? Wouldn’t it make most sense for the kernel to do it transparently for every TCP connection so that it can make more global decisions about path aggregation / link preference?

chmod775• 2 years ago

Using this implies that there are multiple IPs per endpoint associated with a single TCP connection. That is going to need explicit support/awarness by the application in many cases.

brenns10• 2 years ago

My understanding is that it was basically a condition enforced by the maintainers of the Linux TCP / networking subsystems. If you look at the initial upstreaming discussions[1], this was setup as a ground rule.

If you look at the older multipath TCP implementation, prior to the upstreaming, it was intended to be fully transparent to the application, which I think makes more sense for the intent of the protocol. Sure, in many cases MPTCP may be better with application-guided logic, but having a standard system approach (e.g. establish sub-flows on an LTE connection for automatic failover, but don't send any data along those sub-flows) would have worked for 95% of cases.

[1] https://lore.kernel.org/all/alpine.OSX.2.21.1707181728570.11...

matttbe6• 2 years ago

Yes, that's correct. In the Linux kernel, it would not be possible to switch to MPTCP by default.

But apps can use it by default. For the server case, it really makes sense: https://www.mptcp.dev/faq.html#why--when-should-mptcp-be-ena...

GNU/Linux distributions could even switch MPTCP on by default (via eBPF).

londons_explore3• 2 years ago

I can imagine new security holes being opened up by allowing multiple IP's to talk over the same TCP connection...

Imagine you have an application which checks the clients IP (eg. against a whitelist) at the time of connection and then assumes it doesn't change...

apitman• 2 years ago

Is this something that middle boxes tend to support in practice?

flounder3• 2 years ago

We found that most proxies/firewalls (90%+ ? I forget) didn't tamper with it. The largest hurdle was working with load balancer vendors to implement it.

iknowstuff• 2 years ago

Apple supports it too and uses it for Siri https://developer.apple.com/documentation/foundation/urlsess...

Arcanum-XIII• 2 years ago

It’s quite easy to use in other app - it’s there in the basics. We were quite amazed to find our voip app to be quite robust in 2011 :D

ohnoesjmr• 2 years ago

I've heard about MPTCP back in 2013.

It made so much sense back then, when mobile apps were not that robust to networks changing, I assumed it's going to get adopted in no time due to how much of a ux improvement it would have been back in the day.

It's incredibly depressing that this gained barely any traction in the last 10 years, and kernel options are appearing just recently, after everyone has wrapped they http calls in multiple retry handlers, and mobile operating systems have abstracted network connectivity to the point where it feels more like you are using zeromq rather than tcp.

zoobab2• 2 years ago

ZeroMQ yeah!

quectophoton• 2 years ago

I've heard about MPTCP back in 2013.
I assumed it's going to get adopted in no time due to how much of a ux improvement it would have been back in the day.

You might also be interested in SCTP[1] from the year 2000, which also hasn't gotten any traction so far.

[1] https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...

gamegod• 2 years ago

WebRTC data channels use SCTP, which ain't nothing! https://datatracker.ietf.org/doc/html/rfc8831

(SCTP over DTLS, that is...)

fanf2• 2 years ago

SCTP is used a lot inside telco networks for carrying switching control metadata for voice connections. https://en.wikipedia.org/wiki/SIGTRAN

throw0101b• 2 years ago

You might also be interested in SCTP[1] from the year 2000, which also hasn't gotten any traction so far.

Probably partly because middleware boxes (e.g., firewalls) either didn't/don't support it and/or rules were written to only support "TCP" (as opposed to 'stream') or "UDP" (as opposed to 'dgram'; see also "DCCP").

toast02• 2 years ago

I wanted to like it, and Apple included it in iOS, but supporting it on real servers was going to be too hard...

When I was deployed on FreeBSD with no load balancers, there weren't recent patches. And even if there were, I'd need to do some serious work to avoid advertising the private network ips as alternates...

When I was on Linux behind a load balancer, it's too complex to get the streams to the right place. And the load balancer doesn't want to do it anyway.

Processing two streams together involves a lot of complexity in a high throughput code path. It's a lot of risk, and you've got to reboot for changes.

And then you do all that work and it only benefits iOS users, who tend to be on better networks anyway.

bbss• 2 years ago

Apple also contributed[1] MPTCP support to Envoy Proxy.

[1]https://github.com/envoyproxy/envoy/pull/18780

RulerOf• 2 years ago

It sounds like this would have taken off if it were added to various managed cloud load balancers based on what you're saying.

The only question I have is if it opens up a different can of worms even if you've got a magic box terminating layer 7 for you or not. Never dug deep enough into mptcp myself to know.

DEADMINCE4• 2 years ago

iOS users, who tend to be on better networks anyway.

I don't think there is any basis to claim that.

ajb2• 2 years ago

I suspect that a lot of innovation energy moved to QUIC, because with TCP your nice new variant can be randomly nobbled by middleboxes. For example, see https://blog.apnic.net/2021/12/08/efficient-multipath-transp...

xvilka• 2 years ago

QUIC is a step back, IMHO. Especially, given how many national networks work poorly with UDP protocols.

karma_pharmer• 2 years ago

QUIC is a step backwards here; it has no multipath support: https://lwn.net/Articles/964377/

Multipath: There are several areas where TCP still has an advantage over QUIC. One of those is multipath support. Multipath TCP connections can send data on different network paths simultaneously — for example, sending via both WiFi and cellular data — to provide better throughput than either path permits individually.

Server connection migration is explicitly forbidden by QUIC:

https://github.com/quicwg/base-drafts/pull/2031

gsich2• 2 years ago

Nobody cares about those middleboxes, those are only relevant in corporate networks.

dheera2• 2 years ago

I was excited about it because we were working on delivery robots and I wanted a good solution for instant failover given 2 cellular modems.

We ended up going with PepLink's SpeedFusion to save engineering time. But the license was costly. I really hope for a free solution in the future for 2 cellular networks and <50ms failover.

Multipath UDP + OpenVPN would also probably be a viable solution.

kariemf3• 2 years ago

I created something like what you're describing with the addition of P2P communication using NAT traversal (https://www.hyperpath.ie)

It will connect your devices in a P2P Mesh VPN and allow them to send and receive data using multiple links (e.g. multiple 5G or 5G + Satellite).

It is significantly cheaper than Peplink's license, less latency and no bandwidth / data limits.

You need to bring your own hardware though. Like a Raspberry Pi with 3 USB 4G/5G dongles.

tubs• 2 years ago

Hehe, I also worked on a delivery robot with exactly the same problem. We ended up licencing phantom auto. Expensive and ... Not particularly amazing.

bennyhill• 2 years ago

I see it as depressing that this is gaining traction it doesn't deserve. TCP doesn't need one hack at a time and then to make us choose combinations that sort of work in half the use cases in the modern world, it needs to be replaced with SCTP.

zoobab2• 2 years ago

Too bad there is no link to projects using mptcp, such as Openwrt derivatives.

PS: I mentored a student for 2 years at GSOC to patch Openwrt with mptcp:

https://blog.freifunk.net/2017/05/29/gsoc-2017-add-mptcp-sup...

atarvaneitor• 2 years ago

one example on openwrt http://www.openmptcprouter.com/

matttbe6• 2 years ago

Good point, the last version (v0.60) is using the upstream kernel by default. I just added it in the list: https://www.mptcp.dev/apps.html#misc

londons_explore3• 2 years ago

Whats the benefit of an openwrt router supporting mptcp?

I'd think support on web servers and mobile devices would be most important...

ranger_danger• 2 years ago

Link aggregation and redundancy

DarrisMackelroy• 2 years ago

I got fiber run to my neighborhood, and for a while, had a 1gb coax connection and a 1gb fiber connection. I used openmptcprouter to aggregate my connections through a droplet and I effectively had a 2 gigabit internet connection. I would have stuck with it, but having a datacenter IP for your home network really doesn’t work.

jburgess777• 2 years ago

It was recently enabled in the Home Assistant ‘HAOS’ kernel. https://github.com/home-assistant/operating-system/pull/3248

matttbe6• 2 years ago

I hope some apps will start using it :)

andrewmackrodt• 2 years ago

I've been looking at this project for a while which may be interesting to you: https://github.com/Ysurac/openmptcprouter.

I recently bought a property where I cannot get a full fibre connection, but I can get 150-400 Mbps using 5G. I've been thinking about using dual 5G connections and tunneling my traffic via mptcp to a VPS to aggregate the connections.

vesinisa• 2 years ago

If any middlebox in between does not support it, the returned SYN+ACK packet will not contain MPTCP options in the TCP option field.

That sounds .. quite restrictive. Is the only requirement on a middlebox to just forward the MPTCP options as-is?

mhandley• 2 years ago

Before we finalized the specs we did a lot of testing to make sure that enabling MPTCP wouldn't break connectivity. Either it would be passed correctly, or it would fall back to single-path TCP safely. Generally, so long as a middlebox passes through unknown options unmodified, and does not try to enforce that the TCP sequence space it sees is contiguous, MPTCP should work through that middlebox.

If you're interested, we wrote a couple of papers on this:

[1] https://www.usenix.org/conference/nsdi12/technical-sessions/...

[2] https://www.researchgate.net/publication/229002024_Is_it_sti...

extraduder_ire• 2 years ago

The examples given on the page seem to focus on multipath to get to a device over the internet, but I can see this being more likely to work properly without needing to fallback on home networks.

txutxu2• 2 years ago

At home/lan we use LACP, VRRP... I mean link aggregation and HA needs are solved time ago.

With multiple ISPs, or on a complex enough LAN, we can use multiple routing tables + weights too.

Also, if the ISP at home can do 10Gbps, 1Gbps, 300 Mbps whatever... I want to be able to use them with a single path, so there is no gain using multiple paths. Eventually, when I have cable+wifi connected at the same time, I use to force one of both, cannot see a reason to prefer using both at the same time.

Maybe the latency thing? Never had that issue at home, but could understand that usage case "just use the network segment with less latency to reach $thing".

matttbe6• 2 years ago

Some ISPs in Europe are using MPTCP for people being too far from the street cabinets. Typically, for people in the countryside, with < 50 Mbps. Thanks to a transparent proxy installed in the home gateway, and servers in the ISP's network, they can combine both the fixed and cellular networks, and use the fixed one in priority.

MPTCP can also be very interesting for mobility use-cases, even when one network is used at a time, e.g. switching from WiFi to cellular, or different cellular networks in the train, etc.

lo0dot0• 2 years ago

For a long time enterprise firewalls (and more recently SD-WAN) allowed load balancing between different links, but unlike MPTCP the traffic of a single TCP connection is not split up. This is in line with the established network admin wisdom saying that reordering packets of a TCP connection hurts performance.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Ho...

semi• 2 years ago

Also, if the ISP at home can do 10Gbps, 1Gbps, 300 Mbps whatever... I want to be able to use them with a single path, so there is no gain using multiple paths. Eventually, when I have cable+wifi connected at the same time, I use to force one of both, cannot see a reason to prefer using both at the same time.

I don't understand why you would want to be able to use them with a single path. the gain would be being able to aggregate them and have individual tcp streams faster than any one IP connection could handle.

Though personally I think the resilience is more appealing. Not having to have a hard cutover when wifi degrades as I walk away would be nice

slt2021• 2 years ago

This can help in security/privacy setting.

for example Great Chinese firewall: if you can split your traffic across multiple uplink channels, the firewall will have a hard time to put them together for enforcement?

Hikikomori2• 2 years ago

They just block or rate limit it severely when they dont know what it is.

EvanAnderson3• 2 years ago

I don't know which makes me sadder-- IPv4 only having a 32-bit address space or TCP using the source and destination IP addresses in the connection tuple. That's one of those "if I had a time machine" of things-- I'd go back and have Cert and Kahn change both of those items.

bobmcnamara• 2 years ago

How would you change TCP?

Do you mean how we have to use 4 fields to track a connection - IP address and port for both ends?

EvanAnderson3• 2 years ago

If TCP had a protocol specific identifier for connections (a couple of 32-bit values, for example-- a client nonce and server nonce) rather than using the source/destination IP addresses multi-homed hosts and seamless transition between different networks would become native features of the protocol. A client could roam between two different IP networks and TCP connections would "survive", for example. (I'm oversimplifying nearly to the point of hyperbole, to be sure...)

(Another fun future would have been one where SCTP got widespread adoption.)

akira2501• 2 years ago

I think they'd point out that they gave you source routing which is half of what you want and is correctly specified as an option.

infogulch• 2 years ago

I found [1] which describes the architectural difference between MPTCP and QUIC, and also introduces the authors' proposed MPQUIC protocol:

QUIC multiplexes application streams on a single UDP ﬂow, whereas MPTCP splits a single stream on multiple TCP subﬂows. MPQUIC combines both features by multiplex- ing application streams on multiple UDP subﬂows.

[1]: "Multipath QUIC: A Deployable Multipath Transport Protocol" https://www.researchgate.net/publication/327122884_Multipath...

Now I'm curious about how these protocols compare in production operation. Anybody have experience with both?

matttbe6• 2 years ago

Note that MPQUIC is still being discussed at the IETF. At the last IETF meeting, more changes have been discussed. Unfortunately, that slows down its adoption. https://lwn.net/Articles/964377/

But both tries to achieve the same goal. Technically, you can have a very similar behaviour. MPTCP is implemented in the Linux kernel, while QUIC is on the userspace side.

bdd8f1df777b• 2 years ago

The only practical use of MPTCP for me is to use mobile and Wi-Fi network together to boost the speed. iOS and WeChat both support this. However, I always turn them off because my mobile network is metered. So in the end, MPTCP is useless for me *personally*.

psaux• 2 years ago

I worked on this. We called it the parking lot bug. WiFi still shows signal but no proper connection. With MPTCP, it will failover to cell.

taken_usernam• 2 years ago

You can even learn to predict these handovers to make it really smooth [0].

[0] https://arxiv.org/pdf/1907.10493.pdf

suprjami2• 2 years ago

I work supporting, debugging, fixing the Linux network stack and drivers. I am amazed how little adoption this has seen.

Like everything which came along and tried to supplant regular TCP, such as SCTP, it seems MPTCP has also been confined to a niche of application developers who will use it forever while the rest of the world forgets about it.

dilyevsky2• 2 years ago

Other folks have commented - Apple Siri uses mptcp so not exactly niche considering how many devices that is

suprjami2• 2 years ago

Similarly, I promise you everything you've ever done over a 4G or 5G phone has used SCTP inside the phone provider. That doesn't mean the general developer population know about it. I bet most developers have never even heard of it.

Multipath TCP for Linux (2022)

Related Stories