Tesla’s TTPoE at Hot Chips 2024: Replacing TCP for Low Latency Applications

https://chipsandcheese.com/2024/08/27/teslas-ttpoe-at-hot-chips-2024-replacing-tcp-for-low-latency-applications/

by ksec • 2 years ago

172 91 2 years ago

fragmede• 2 years ago

Interesting! No mention of UDP, or the application being run, or the GPU/TPUs on the nodes, so it'll have to be a mystery as to how much bang for their buck they're getting with this particular bit of work.

What's disappointing is that it's impossible to do a new protocol on the Internet because of all the middleware boxes that drop packets that aren't IMCP or TCP or UDP.

KaiserPro4• 2 years ago

UDP means that upper layers have to figure out if the data is shit or not. to do that in hardware limits your data types, flows and developments. Its far more simple to have a "reliable" data transport system, than it is to deal with a lossy protocol in hardware.

7e• 2 years ago

100Gbps Ethernet cards? The world has moved way past that for training. Their accelerator stack must be really slow if this is good enough for them.

teruakohatu• 2 years ago

It doesn't say they can't aggregate links. I wouldn't say the world has moved way past that yet, but Tesla probably doesn't want to be dependant, like everyone else but Apple, on Nvidia (InfiniBand).

krasin2• 2 years ago

like everyone else but Apple, on Nvidia (InfiniBand)

Google also does not depend on NVIDIA, thx to TPUs. Rents NVIDIA GPUs to external customers - sure, it's a nice side business, but internally TPUs are king and there's no dependency on NVIDIA for that.

stonogo• 2 years ago

You can get 400gbit ethernet from half a dozen vendors.

cout• 2 years ago

This kind of computing must be a different kind of world than the one I work in. 80 microseconds of latency seems high to me when infiniband can do single digit latency with unreliable datagrams, which turn out to be mostly reliable due to the credit system.

exabrial• 2 years ago

whats the credit system?

xtacy4• 2 years ago

OP is referring to "Credit based flow control", which is a way to ensure a sender does not overwhelm a receiver with more data than it can handle.

Usually, this is line-rate, but if the other side is slow for whatever reason (say the consumer is not draining data), you wouldn't want the sender to continue sending data.

If you also have N hosts sending data to 1 host, you would need some way of distributing the bandwidth among the N hosts. That's another scenario where the credit system comes. Think of it as an admission control for packets so as to guarantee that no packets are lost. Congestion control is a looser form of admission control that tolerates lossy networks, by retransmitting packets should they be lost.

yumraj• 2 years ago

infiniband

Or I guess even RoCE

kamikaz1k• 2 years ago

Not really familiar with this space but I think the entire Dojo/DIY strategy was kicked off because Elon wanted to not get cornered on supply or cost by nvidia. And infiniband is an nvidia technology, so they wouldn’t use that simply from strategic POV.

Are there other technologies they could have used?

Also, the 80us is supposed to be the worst case, where typical is supposed to be <10us. Again not knowing anything about infiniband, what’s the typical perf? I tried to google but the people who are talking about it are in the know in ways I’m not.

Thanks!

justahuman74• 2 years ago

Is RoCE no good?

charleshn3• 2 years ago

Indeed, it seems that 80usec is just given as an upper bound based on the 1MB buffer at 100Gbps.

It is definitely possible to go much lower than 80usec on Ethernet. But obviously it depends on the scale, utilisation etc.

At the sizes of GPU clusters we're talking about these days - 32K and up - things get tricky.

The main alternative to Infiniband used in the industry is RoCE - Meta has written a lot about it [0].

There's several reasons to avoid Infiniband, such as cost, availability, vendor lock in, lack of experience etc.

Those are some of the reasons why many players are trying hard to make Ethernet work, see Ultra Ethernet [1].

[0] https://engineering.fb.com/2024/08/05/data-center-engineerin...

[1] https://ultraethernet.org/

namibj2• 2 years ago

Also PCIe is worth mentioning for it's credit-based full reliability (in the absence of hardware failures, which are still signaled).

iforgotpassword• 2 years ago

I assume infiniband is much more expensive, but then again you have to offset all the development cost first.

jabl• 2 years ago

It's not, really. It's been a while since I've checked pricing so my data might be old, but an IB switch is in the same ballpark as an ethernet switch with the same port speed. Same for HCA's.

There's no analogue in the Infiniband world to dirt cheap 1GbE RJ-45 switches though.

latchkey• 2 years ago

At least when I looked into it last, it was less about price and more about availability. NIC's are easy to get, but the switches were a 50+ week lead time. I'm sure it has come down now though at least for my business, we are just focused on ethernet now. We can do 128 NIC's @ 400G into a single Dell Z9864F using ethernet. Few will need proprietary solutions for this as there are lower hanging fruit.

starspangled3• 2 years ago

AFAIKS the protocol can tolerate up to about 80 microseconds of latency.

The graph at the end shows they measured (one way) latency at 1.3 microseconds (compared with 2.0 for IB).

eecc• 2 years ago

Well, whether it matters depends on the workload: IB is basically remote DMA so if you need to pick and poke remote data I guess it'll work as another NUMA tier.

But for AI training, where you're simply shuffling around large stacks of matrices, my guess is latency constraints weaken.

xtacy4• 2 years ago

It's also a bit odd that they do not implement congestion control. Congestion control is fundamental unless you only have point-to-point data transfers, which is rarely the case. All-reduce operation during training requires N to 1 data transfer. In these scenarios the sender needs to control its data transfer rates so as to not overwhelm not just the receiver, but also the network... if this is not done, it will cause congestion collapse (https://en.wikipedia.org/wiki/Network_congestion#:~:text=ser...).

jcims4• 2 years ago

I probably shouldn't be commenting because I don't have any experience at this level, but given it's a closed system where they control supply and demand it seems they could manage away most congestion issues with scheduling/orchestration. They still have a primitive flow control in the protocol and it seems like you could create something akin to a virtual sliding window just by instrumenting the retransmits.

But now I am curious with the distribution of observed window sizes is in the wild.

Edit: I'd bet the simpler protocol is more vulnerable to various spoofing attacks though.

Edit2: Lol I hope the frame IDs are for illustrative purposes only - https://chipsandcheese.com/2024/08/27/teslas-ttpoe-at-hot-ch...

xtacy4• 2 years ago

In principle, with perfect knowledge of flows at any given instant, you can assign credits/rate-of-transmission for each flow to prevent congestion. But, in practice this is somewhat nuanced to build, and there are various tradeoffs to consider: what happens if the flows are so short that coordinating with a centralised scheduler incurs a latency overhead that is comparable to the flow duration? There's been research to demonstrate that one can strike a sweet spot, but I don't think it's practical nor has it been really deployed in the wild. And of course, this scheduler has to be made reliable as it's a single point of failure.

Such ideas are, however, worth revisiting when the workload is unique enough (in this case, it is), and the performance gains are so big enough...

pantalaimon• 2 years ago

I'd bet the simpler protocol is more vulnerable to various spoofing attacks though.

This is a protocol between compute nodes in a data center, it's layer 2 so there is no way to reach this over the internet.

kiratp• 2 years ago

Current public SOTA seems to be “no congestion control”

We proceeded without DCQCN for our 400G deployments. At this time, we have had over a year of experience with just PFC for flow control, without any other transport-level congestion control. We have observed stable performance and lack of persistent congestion for training collectives.

https://engineering.fb.com/2024/08/05/data-center-engineerin...

_zoltan_4• 2 years ago

PFC is congestion control.

choilive• 2 years ago

This is all technically impressive but was it all technically necessary? Was infiniband really just not good enough? All this R&D for a custom protocol and custom NICs seems to just be a massive flex of Tesla's engineering muscle.

2rsf• 2 years ago

It's not only about being "good enough" but also about reliability and maintenance, new protocols and hardware may take time to mature while other solutions are already there. Ah, wait, Tesla doesn't care about those kind of things too much...

karlgkk• 2 years ago

It wasn't technically necessary, but neither was RISC-V. It's a matter of licensing independence.

marcinzm• 2 years ago

It's a matter of licensing independence.

And supply chain independence. I've heard that some GPU clouds are delayed because their Infiniband hardware was delayed due to the Israel–Hamas war. Optimally you probably want to avoid critical hardware that's being manufactured in a high risk of disruption zone.

logicchains• 2 years ago

Was infiniband really just not good enough

Infiniband suppliers charge crazy prices due to having little competition. It might actually be cheaper for them to design their own than to pay the Infiniband tax.

speransky• 2 years ago

Tuned RoCE with udp is really low latency and no need to implement extra layer of silicon. May be there are more motivation then described in article

efitz• 2 years ago

Yeah I never FIN my connections eithRST

jeroenvlek• 2 years ago

Seems like Tesla could really benefit from this about to be released optimizer that reduces intra-GPU communication [0].

[0] https://github.com/NousResearch/DisTrO/blob/main/A_Prelimina...

sgt2• 2 years ago

Isn't this what Dolphin Interconnect have been doing for a couple of decades? https://www.dolphinics.com/

namibj2• 2 years ago

No, they did it the actually good/cheap way: straight PCIe.

sgt2• 2 years ago

You are right, and I realized that a few min after posting. So tell me, why is Tesla not doing straight PCIe then? If it's the actual good/cheap way. It makes sense to me but that's more a feeling.

SilverBirch• 2 years ago

This screams "Not Invented Here" syndrome. Massive yikes at the digram showing TCP in software in the OSI model. There have been hardware accelerated TCP stacks for decades. They called TCP Offload Engines, they work great, have done for ages. Why are you building one and giving it a new name? Seems like a pretty enormous amount of work and you would've gotten 90+% of the gains by just implementing a standard TOE. I guess the only good reason I can think to do this yourself is that they'd left it so late to get to this that all the companies that were good at this got bought (Solareflare, Mellanox etc).

MisterTea3• 2 years ago

I assume they ignore other technologies and research because new shiny things gives them visibility and therefor promotions.

lnsru• 2 years ago

I have built dozen of different FPGA based cameras in the past. There is GigE vision protocol: https://en.m.wikipedia.org/wiki/GigE_Vision TCP is used for “normal” connection and UDP for low latency video data. Such system could be used for other low latency applications as well.

kardianos• 2 years ago

This is a bad take. Here's my take: 1. Standard IP and Ethernet are physically acceptable for their use case. 2. TCP/IP is optimized in a number of areas for un-reliable networks. 3. Their clusters are not unreliable. 4. Their servers already offer hardware acceleration that can be programmed, so remove aspects of TCP/IP that increase latency or might negatively affect throughput. 5. They get to continue to purchase the cheaper IP switches and retain their existing hardware without retooling everything.

As an afterthought, they publish this for marketing/engineering pull for people who like to optimize (do engineering) for specific situations, while supporting the ROI and keeping cost down.

chipdart2• 2 years ago

This is a bad take. Here's my take: (...) 5. They get to continue to purchase the cheaper IP switches and retain their existing hardware without retooling everything.

OP's TCP offload engines do not require any retooling at all, if you consider stuff like buying IP switches not retooling. All you need to do is basically buy a network card.

Also, if you read the article you'll eventually stumble upon the bar chart where they claim that the one-way write latency of TTPoE is only 0.7E-6 seconds faster than existing tech like infiniband, and it's also the only somewhat hard data they show. Does that justify the investment of developing their whole software+hardware stack?

I'm sure the project was fun and should look great on a CV, but overall it doesn't look like it passes the smell test.

lallysingh2• 2 years ago

They don't need the generality of full TCP for their cluster, so they're using a tweaked, incompatible subset. One that's been optimized for better performance on cheaper hardware than you can get with TCP h/w offload. In the offload case you're still paying the latency, wire protocol overhead, and efficiency costs of full TCP.

(Disclaimer: I work at Tesla, not related to this group, opinions on public info only)

throw0101d3• 2 years ago

One that's been optimized for better performance on cheaper hardware than you can get with TCP h/w offload.

How many ≥10 Gbps chipsets that you'd find in a typical server do not have offload nowadays?

Further, once you're in the ≥50 Gbps card range you can often get ROCE, which helps with things like latency.

grumpy_coder• 2 years ago

"Instead of using typical supercomputer networking solutions like Infiniband, Tesla chose to adapt Ethernet to their needs with a modified transport layer."

So, they need to compare it to Infiniband, not TCP, and definitely not software TCP. And they need to explain how/if it works with standard huge capacity switches (which is at least a reason to prefer TCP over Infiniband).

There could be reasons to build this, AWS have something, but for Tesla to build their own stinks of NIH bad.

ai4ever• 2 years ago

agree 100% with this take.

they are purpose building hardware for their specific application. debugging corner cases and making this robust is going to take them a decade. given that nobody else is interested in this non-standard solution, they dont have the benefit of the community debugging it, and improving on it in open-source.

appears to me to be a vanity effort as is the whole dojo project.

darby_nine• 2 years ago

The "Tesla" in the article appears to refer to the car manufacturer, for those as confused as I.

joezydeco• 2 years ago

All this to...train a model? Having worked with automakers that agonize over pennies on a component how does Tesla amortize this much expense, if they even achieve full self-driving at all?

itishappy• 2 years ago

The $1 billion spent on Dojo amortized over 2 million cars sold per year is $500 per car. I'm betting it's viable for more than a year.

bee_rider• 2 years ago

Maybe the point is to just continue to be able to point to FSD as some figure thing that you are working on. That should keep investors happy!

Once they’ve built this cluster, maybe they can make money renting out compute or something.

jcims4• 2 years ago

It's a plant not a product.

throw0101d3• 2 years ago

Having worked with automakers that agonize over pennies on a component how does Tesla amortize this much expense, if they even achieve full self-driving at all?

Or Elon is using the resources of one company to do work for another company (?):

* https://en.wikipedia.org/wiki/XAI_(company)

* https://electrek.co/2024/04/03/elon-musk-xai-poaches-enginee...

MisterTea3• 2 years ago

Congrats. You just reinvented the wheel: http://doc.cat-v.org/plan_9/4th_edition/papers/il/

chronicileiee• 2 years ago

Meanwhile high frequency traders work 1-2 orders of manicure faster in the tens to hundreds of nanoseconds.

gruturo• 2 years ago

If they are so concerned with low latency, how come they are wasting an entire roundtrip (the "OPEN + OPEN/ACK") before sending any data?

I mean in TCP it's not allowed (Even though, super-theoretically, it's not completely forbidden) to carry a payload in the initial TCP SYN. If you're so latency-obsessed to create your own protocol, that's the first thing I'd address.

newsclues• 2 years ago

Do you need low latency right away, or just after the car has started or entered a sport/race mode?

If you don’t need it all the time, why bother?

iamleppert• 2 years ago

There are already high speed, low latency video interfaces that have been around for ages. MIPI and HDMI.

There are ICs you can buy off the shelf for electronic routing and switching of these interfaces.

Tesla’s TTPoE at Hot Chips 2024: Replacing TCP for Low Latency Applications

Related Stories