Tcpls: Modern Transport Services with TCP and TLS

https://blog.apnic.net/2022/05/24/tcpls-modern-transport-services-with-tcp-and-tls/

by enz • 4 years ago

101 46 4 years ago

blog.apnic.net

simmervigor3• 4 years ago

Some related discussion in this Twitter thread:

https://twitter.com/alagoutte/status/1532013841718120449?t=W...

bediger40002• 4 years ago

Lots of dupes:

https://news.ycombinator.com/item?id=31589575

https://news.ycombinator.com/item?id=31569066

pvg2• 4 years ago

They need to have some discussion to count as HN-dupes and those don't.

bediger40002• 4 years ago

That seems a little wrong - I can just spam uninteresting, yet topical things to HN, and no negative consequences accrue? People will see the headline at least, maybe click on the article. I get the clicks every time, and I get to maybe repeat a Big Lie over and over, which will help my goal of framing the discussion my way.

But thanks for the correction! I appreciate it.

scarmig• 4 years ago

https://dl.acm.org/doi/10.1145/3485983.3494865 seems like a more understandable source.

metadat• 4 years ago

This link was much easier for me to read and parse, thank you scarming.

throwaway7875442• 4 years ago

I would appreciate it if the industry would stop trying to shove everything into user space. Imagine it's 40 years in the future, and we still have to kludge new protocols over an 80 year old TCP/IP stack because nobody wants to add a new protocol to the standard. Every application ships with its own unique protocol stack implementation. Duplicate code abounds. Bugs are fixed haphazardly. Quality and stability varies by application. RCEs are more common because an increase in implementations increases the probability of security bugs.

Compare that to finalizing a new protocol, putting it in the kernel, and letting all applications just connect() and send() and recv(). We lose the ability to constantly mutate the protocol whenever FAANG feels like it, but we also gain the ability for every application to use the new protocol just by changing a flag passed to a syscall.

xxpor2• 4 years ago

Pushing it to the kernel means you have to upgrade your kernel to use it. Instead, you can just include a library that exposes a send() and recv() style API to the main app code, while the library just throws everything into a UDP socket. Seems better to me, IMO.

The real issue is that we can't get away from UDP/TCP because of middleboxes (firewalls, load balancers, etc) and the fact that switches only know how to hash on 5 tuples of protocols they know about. If you're lucky, your router will be able to look inside a GRE header or something, but that's about it.

When was the last time you used SCTP, for example? (If you've used it recently, I bet it was tunneled over UDP)

kazen44• 4 years ago

technically speaking, switches do not hash TCP information but hash based on data in the Ethernet frame. (this is mainly used for correctly balancing load across links in an lacp bundle).

High end routers are able to do pretty conplex lookups into their asics, but this usually only extends to specific use cases. (as far as I am aware for instance, an juniper MX series is able to do a lookup of both vlan tags in a single asic pipeline step when using q-in-q. the same also applies to mpls stacked lable lookups on provider edge routers).

throwaway7875442• 4 years ago

vendors can't get away from middleboxes, because Enterprise customers want middleboxes. But if customers want a completely reinvented underlying technology, everyone is forced to either adopt it or become obsolete themselves.

This whole middlebox obsession is some kind of Stockholm syndrome. Here you are telling me there's people who make shitty devices that don't respect network traffic and because of that you can't change network traffic. It's like saying there's someone breaking into houses to steal stuff, so we can't install stained glass windows because a thief might break them.

boardwaalk• 4 years ago

And it would be great if SCTP support was common across many platforms and supported over networks so we could have something (in ways) better than TCP or UDP.

Unfortunately it's over 20 years old and people just shove it in user space and use it over UDP (WebRTC).

Your sentiment is appreciated but it wouldn't be a way to get anything actually done :/.

a-dub3• 4 years ago

interesting debate whether encryption should occur at the network vs. transport layer. on the one hand, encryption tends to have session state, so the transport layer makes sense but on the other i often feel i'd like to see a new version of IP where encryption is baked in as a primary design principle thereby eliminating all notions of leaks.

zamadatix2• 4 years ago

IPsec tunnel mode delivers that function https://datatracker.ietf.org/doc/html/rfc4303

This contrasts to IPsec transport mode which only encrypts the payload not the IP info https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-es...

Generally there is encryption available at each layer in the network stack but middleware (operating systems, firewalls, NAT) tend to get in the way at lower layers. E.g. IPsec is often now run with NAT traversal which means the ESP header is stuck under a UDP header so it can get through to end devices even though IPsec itself doesn't require this. This, and a few other things, have left IPsec as a thing most people only know for corporate VPNs.

a-dub3• 4 years ago

i'm envisioning more of a sort of ipvx, 64 bit addresses, encryption is non-optional and dos resistance is built-in.

badrabbit5• 4 years ago

TCP is terrible at long distance data transfer. I am sure some people will still prefer it since it can manage congestion but shouldn't newer things use UDP and handle delivery guarantees at the application layer?

gsliepen3• 4 years ago

There is nothing inherently wrong with modern TCP for long distance data transfer. You do need to use the right congestion algorithm and some extensions enabled to handle large window sizes to get high speeds at long distances. But UDP does not magically solve anything; if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. The main benefit is that you are no longer relying on the operating system's implementation of TCP, the drawback is that you won't get any help from the OS or even hardware.

badrabbit5• 4 years ago

I thought you can't keep increasing window sizes, it is a physics limit? The mere fact that you need to wait for an ACK is the problem?

https://en.m.wikipedia.org/wiki/Bandwidth-delay_product

Matthias2472• 4 years ago

The congestion control algorithms that are applied in TCP and QUIC are actually even the same ones (e.g. Cubic, BBR, etc), so the technologies behave not too different over higher RTTs. Both will ramp up a congestion control window, which can get pretty big.

There are differences in behavior regarding what happens if loss is encountered and what exactly is retransmitted, but those still don't make QUIC radically different in terms of behavior and performance.

gsliepen3• 4 years ago

This is quite interesting. The only issue I see is when you want to stream real-time audio and video over TCPLS. That is when you want packets that are dropped, late or reordered to not interfere with the delivery of other packets. Using TCP as the underlying transport makes that problematic, and even having multiple TCP connections cannot help much with that.

ThePhysicist• 4 years ago

QUIC's connection migration feature is really amazing for applications where you switch IPs, e.g. when your phone goes from mobile data to Wifi. Another nice thing is that you can cache sessions and resume them later, so connections can even survive reboots and server failures, or you can move them between different machines.

TCPLS seems like a good proposal but I won't hold my breath for it as the big players like Cloudflare, Google, Akamai, Fastly are already committed to QUIC. There's still significant churn in the proposals and some features like MASQUE are still being finalized, but overall QUIC works really well.

stormbrew4• 4 years ago

I'm not a huge fan of how spdy and then quic were kinda ramrodded into the HTTP standards process, but realistically I think QUIC is basically the future of connection transport wherever tcp or udp alone are failing at this point. I recently discovered the nQuick proposal to do noise-based connection initiation for peer-to-peer connections and I love it[1]. It's an incredibly versatile protocol.

It seems likely that a lot of the issues raised in the OP will be somewhat forced to be resolved due to the aforementioned ramrodding of QUIC into the HTTP3 path, but maybe in the long run the two ideas will co-evolve to the same solution.

[1] https://eprint.iacr.org/2019/028.pdf

unmole3• 4 years ago

quic were kinda ramrodded into the HTTP standards process

QUIC took 4 years and 34 drafts to get standardized. The final IETF version is significantly different from Google initially came up with. So, no it most definitely was not ramrodded through.

1vuio0pswjnm7• 4 years ago

Given that they have had 34 some odd drafts and continuous errors, with some still not fixed, QUIC is not confidence-inspiring. As an ordinary internet user, the problem I see is that QUIC development is being led by companies, not academics,1 that see the internet first and foremost as an advertising and commerce medium, as would be expected since they are companies pursuing business interests. The issues around those commercial uses are prioritised while issues around ordinary internet user privacy and control are secondary. Certainly QUIC is not the first2 and and will not be the only UDP-based reliable transport protocol but what will be interesting is whether firewall administrators will treat all the alternatives equally. It may be that the corporate-sponsored ones like QUIC are allowed through while ones written by academics or other non-commercial authors are not.

1. Many of the people leading development of the older HTTP versions were academics.

2. See, e.g., CurveCP: https://curvecp.org Having played around with CurveCP before QUIC existed, and having already thought CurveCP was amazing, the later commercial effort led by Google has left me underwhelmed.

Matthias2472• 4 years ago

Some thoughts from someone working on QUIC for a couple of years:

There's some good ideas in this one! The idea of just keeping a TCP stream per Tcpls stream while sharing some crypto state means it can be really efficient, since all optimizations that exist for TCP (including hardware offloads) will continue to work. That is also visible in the efficiency benchmarks [1] on the site.

One might even argue that with such a design where the most expensive parts of TLS (handshakes) become no-ops an introduce no additional latency, techniques like HTTP/2 become unnecessary - one could keep just having a single TCP stream per request and enjoy better flow control, fairness and observability than with anything application managed.

However there are some challenges that need to be resolved for running protocols on "real-world infrastructure and datacenters", which change the picture a bit. One is that a single IP address announced by large-scale webservices doesn't necessarily point to a single server, but to e.g. a rack of them or even bigger units. This means that a second TCP connection might (or even should be) established to a different server - which would not be aware about the TLS session state. If the state is kept on a different server, the original one would need to forward all packets either via IP-in-IP encapsulation or by creating a separate upstream TCP connection and proxying all data bidirectionally. All those approaches would be fairly expensive - every time a TLS session would be joined the amount of packets handled by a server would be double of the ideal version.

While a similar challenge exists for QUIC, it's actually cheaper on average: Since multiple streams use the same QUIC connection and usually also the same source/destination IP and port pairs, traffic would usually be directed to the same host by common infrastructure and require no special handling. It would only require effort to "fix" misdirected QUIC connections in the event of connection migrations - which are rather rare. If packet forwarding is required, it can be achieved rather cheap and easily by interpreting the QUIC connection ID fields in all UDP packets and doing raw UDP packet forwarding.

Therefore as far as I understand the proposals so far, it might not be necessarily clear-cut on whether running QUIC or TCPLS would be cheaper and an ingress or load-balancing layer - which are the layers where both protocols have the biggest benefits due to the highest amount of concurrent connections and higher client-facing RTT.

But maybe the challenge for TCPLS could be somehow solved by having a shared place for the TLS state between hosts and making sure streams are operating fully independently - but it requires a bit more thoughts.

I'm wondering how close such a proposal is then to the existing TLS session resumption mechanisms - just extended by allowing more than 1 follow-up session.

[1] Note that the diagrams are really about efficiency and not performance. They tell nothing about what throughput might be available for end-users that are e.g. 100ms away from the server. For local tests the delay and packet introduced by large-scale internet infrastructure doesn't play a role.

frochet• 4 years ago

Very interesting thoughts, thank you for sharing!

Tcpls: Modern Transport Services with TCP and TLS

Related Stories