An Empirical Evaluation of TCP Performance in Online Games

http://www.iis.sinica.edu.tw/~swc/pub/tcp_in_games.html

by cpeterso • 13 years ago

107 37 13 years ago

iis.sinica.edu.tw

e12e• 13 years ago

Reminds me about the last few paragraphs of:

http://250bpm.com/blog:22

Especially:

"Obviously, the right solution would be implement a new transport protocol directly on top of IP protocol, one that would provide desired functionality — failure detection and/or multiplexing — directly, without duplicating the features.

And, as a matter of fact, the above was already done. The protocol built directly on top of IP with both heartbeating and multiplexing is called SCTP and is available out of the box in most operating systems.

And here comes the problem: SCTP is not used, even for projects where those features are needed. Instead, they are lousily re-implemented on top of TCP over and over again."

(Not necessarily saying that SCTP would be the best fit for games, more the general point -- that there's room for more than just TCP and UDP on top of IP -- and also that that might not work, due to firewalls and routers refusing to route anything that's not TCP or UDP).

_stephan• 13 years ago

There's also Google's QUIC:

https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs...

http://en.wikipedia.org/wiki/QUIC

AshleysBrain• 13 years ago

SCTP is used in WebRTC DataChannels, so it will see use in web games hopefully. However I think Chrome and Firefox implement it on top of UDP at the moment (perhaps because not all OSs support it?)

rdtsc• 13 years ago

If it is using native SCTP, I don't see how well it would work with firewalls and routers (both enterprise and home ones). It would have to fallback to user-space over UDP presumably.

fulafel• 13 years ago

The WebRTC approach of using UDP gives NAT resistance and OS independence.

vvhn• 13 years ago

And here comes the problem: SCTP is not used, even for projects where those features are needed.

sctp's achilles heel is its inability to work with NAT. It was designed to not work with NAT. NAT has since become widespread especially for connecting homes and sctp ended up not being so.

selectodude2• 13 years ago

So SCTP will be useful the day after IPv6 has wide adoption.

Well, shit.

tinco• 13 years ago

It's not that SCPT is less known by developers, I think most developers actively looking for a solution to the downsides of TCP will come across SCTP.

There are two real reasons, the first and most important is that Microsoft never implemented SCTP. Apparently there was not enough customer demand, but that's a lousy excuse, I'm not sure what the real reason is. Perhaps they just have no one thinking about networked applications besides the Web at the moment. Sadly lack of forward thinking is something that bugs the Windows team a lot I feel. The second reason can be partly blamed on Microsoft as well, but most network devices operating at the transport layer have difficulty with SCTP. Why implement it when no one uses it right? just like IPv6 support. Anyway, apparently the difficulty lies with checksumming packets, which requires some more memory than is necessary for TCP/UDP.

It's a shame really. I think you can safely say that SCTP is the perfect gaming protocol. It supports almost all features gaming networking libraries usually implement on top of UDP just by configuring your streams.

throwawaykf4• 13 years ago

Perhaps they just have no one thinking about networked applications besides the Web at the moment. Sadly lack of forward thinking is something that bugs the Windows team a lot I feel.

If you found out about the wide range of networking protocols and services they provide in Windows, you wouldn't feel that way. For example, P2P. A few years ago, while working on P2P applications, we discovered Microsoft has a lot of technologies in that area. Things like Teredo (IPv6 transition and NAT traversal) and a complete P2P networking API -- which gives you encryption, self-organizing overlays and DHTs out of the box -- have been around since around 2001, and have been built into Windows since Vista. They also had lower level protocols for things like link-layer topology discovery and automated device (printers, etc) configuration.

Almost all proprietary stuff, of course - except Teredo, which has an RFC and an open source implementation - but quite interesting nonetheless. We talked to the team working on this, and they claimed it was even being used by some customers in 1000+ machine deployments.

For some reason, these things are not widely marketed. Maybe because it's a niche market. However, I wouldn't say they're not forward-thinking, at least when it comes to networking.

jimmaswell• 13 years ago

I remember the forward-thinking present in the NT kernel being been seen in a negative light around here a while ago.

lttlrck• 13 years ago

One reason is quite possibly that SCTP is not supported by most home routers/firewalls.

csense• 13 years ago

due to firewalls and routers refusing to route anything that's not TCP or UDP

If "The world is on fire because we're almost out of addresses" doesn't get residential ISP's to support another protocol for home users (IPv6), somehow I doubt "We can improve the performance of online games that work reasonably well right now" will convince them to support SCTP.

derefr2• 13 years ago

We should probably accept that TCP and UDP are the only "natural" L4 protocols we get, build all new protocols on top of UDP, and then just abstract the difference at the OS level, such that those tunneled-over-UDP protocols appear as native L4 transports. Then when you say "I'm using SCTP", you really mean "I'm using SCTP-over-UDP" -- but since that's the only possible meaning, it's just taken as a given.

Heck, maybe we should even reimplement TCP on top of UDP (stripping out the redundancies, so UDP carries the ports and checksum), and then just consider UDP an L3 protocol--a bit of extra stuff you get with every IP packet, whether you want it or not.

If we can figure out a way to do implicit protocol negotiation, such that we can build both clients and servers that "prefer" TCPv2-over-UDP but will drop back to plain-TCP, this could really happen, and we might get to start writing new transport protocols again. (Heck, for that matter, DTLS might finally get used.)

See also: the comment by eternaleye on the above post http://250bpm.com/blog:22/comments/show#post-1759542

eps• 13 years ago

Re: last point - at the expense of 8 bytes any IP-based protocol can be moved to run over UDP.

Liongadev• 13 years ago

I call that thing bullshit. "Our analysis indicates that TCP is unwieldy and inappropriate for MMORPGs" yet World of Warcraft, Lineage I/II, Guild Wars, Ragnarok Online, Anarchy Online, Mabinogi all use TCP. Extremly successful. I using UDP would have help making World of Warcraft better they would have done it years ago. But they all probably have no idea right?

eropple2• 13 years ago

That TCP is ill-suited for this does not mean it cannot be used. It also does not mean that it is World of Warcraft's particular bottleneck and that it would be a good investment of development time to change it once it is in production.

You should probably resist the urge to call bullshit on something when your best examples rely on legacy concerns.

charonn0• 13 years ago

The premise may be valid, however the authors only examined a network trace for one game. At best, they've proven only that game's use of TCP is unwieldy and inappropriate.

A more appropriate methodology would have been to examine traces of the top 5 MMORPGs in terms of active players.

TillE• 13 years ago

TCP only really breaks down when there's a nontrivial rate of packet loss. Otherwise, excepting very fast-paced action games where you don't want a single noticeable hiccup, the concerns about TCP are largely outdated.

I mean, I was reading this kind of material back when FPSes and even MMOs were played over dialup connections. The world has changed.

derefr2• 13 years ago

Yes, they all manage it--but this is why "MMORPG network middleware engineer" has always been an extremely well-paid senior-level position. If it was as easy as "SCTP.connect(host: 'example.com', port: 3044)" to get something sensible going, I think a lot more games would do the kind of smooth network-latency-compensation-via-action-prediction that you only tend to see right now from the big players.

angersock• 13 years ago

Very commonly TCP was used for messages that had to be reliable, and UDP for the other stuff. A lot, and I mean a lot, of work was put in by devs in the early 00s trying to figure out how to do as well with TCP as they could with UDP.

Also, RO server code was and is a steaming pile of sadness and bugs.

cclogg• 13 years ago

This is a really interesting article, I love reading about game networking. Hard to find many resources on this type of stuff... the 'Age of Empires' Gamasutra one is awesome too.

We used Apple's Game Center API for the online portion of Stratosphere: Multiplayer Defense (iPad), and it was pretty nice that Apple did all the work as far as network packets go... but you do have a choice between reliable and non-reliable sending. It wasn't explicitly said, but we assumed one was TCP and the other UDP. We went with the reliable option for all of our packets and it didn't really prove to be a problem, even simulating under 3G. Granted, Stratosphere is a strategy game. The bigger problem ended up being just having enough people online at one time to even play lol.

plasma• 13 years ago

Check out the Tribes netcode model, http://library.theexiled.pwnageservers.com/file.php?id=2694

throwawaykf4• 13 years ago

Another good read on game networking:

http://gafferongames.com/networking-for-game-programmers/

ericz• 13 years ago

If you're a web game developer reading this you should also know that WebSockets, XHR, and anything built on top of those technologies (socket.io etc.) are TCP based. Only WebRTC DataChannels and flash are able to do UDP based communication in the browser.

salmonellaeater• 13 years ago

HN ought to parse "http://www.iis.sinica.edu.tw" as "sinica.edu.tw" rather than "edu.tw".

saraid216• 13 years ago

Or more broadly, \w+.(com|gov|net|org}.countrycode should really happen. Probably a couple other standard ones.

codexon• 13 years ago

It is also important to consider DDOS when evaluating protocols.

TCP has an advantage over UDP because data cannot be spoofed. There are many ddos protection services that will protect against spoofed TCP connections (SYN floods).

Applications must implement their own 3 way handshake in UDP in order to avoid these problems. Many applications did not do this which is why COD servers ended up being a source of attack traffic themselves.

dedalus• 13 years ago

The problems raised here can be solved with parametrizing TCP itself. A few notes here:

(a) Packet reordering is highly dependent on the timescale that you use. Its far worse in micro/smaller timescales than to be seen as attached in the graph

(b) By using TCP URGent byte effectively you can hack some realtime control stuff onto a data stream

(c) The fast retransmit not kicking in can be resolved by lowering the number of dupacks from 3 to 1 and add a timer that kicks in the fast retransmit after the timer expiry. This isnt a new ide. BTW 50% is the standard number for the whole internet so your stats does not show anything different from the standard internet

(d) TCP slow start after idle can be disabled on server side

(e) Agreed all of these require some co-operation from the client side which might be hard to get (especially on windows) but can easily point to a http proxy for the MMORPG in question and usually the speed benefit will drive the adoption of such proxies before actually having someone sell a Gamer TCP Stack :-)

Long story short all the problems can be mitigated to some degree within the norms if you use the right proxy :-)

scurvy• 13 years ago

Perhaps the developers communicated with their netops teams before designing the game protocols? Maybe they came to the conclusion that it's better to run TCP from a security/DDoS perspective than UDP? In other words, the ability to shut off inbound UDP outweighed any theoretical gains from using it in the game?

TCP SYN floods are handled with ease today. If you're actually using UDP in your product, you might as well paint "DDoS me" on your homepage.

An Empirical Evaluation of TCP Performance in Online Games

Related Stories