Why mobile apps suck when you're mobile (TCP over 3G)

http://blog.davidsingleton.org/mobiletcp

by dps • 15 years ago

298 68 15 years ago

blog.davidsingleton.org

clistctrl• 15 years ago

Not that his advice is bad, but these statistics are a bit biased. Trains make for some pretty unusually difficult channel conditions.

ComSubVie• 15 years ago

That may be, but - at least in my case - the most typical mobile usage is on a train (everywhere else I usually have WiFi).

mooism22• 15 years ago

Trains moreso than cars/buses? Why?

dspillett4• 15 years ago

Faster travel meaning you are passing in and out of cells more often, and you are more likely to be passing through areas with bad coverage either for man-made geographical reasons (a steep embankment between you and most of the towers in range for instance) or because your route is more likely to take a straight-ish line that may pass through a pretty uninhabited area that is ether completely unserviced by the cell network or where the only tower(s) in range are some distance away. Also if you are travelling by bus (and to a certain extent by car) you are more likely travelling a shorter distance than if travelling by train so are less likely to be passing through areas that are badly served because there is no money in the networks serving the two local farmers as well as they serve a small town.

Tsagadai• 15 years ago

Really? I would say trains, buses and cars are really common use cases for mobile data. Every second person I see is watching Youtube or playing around on the internet while they take public transport. It isn't biased data, there is an actual use case that it is presently failing.

aristidb• 15 years ago

I have precisely the same characteristics in my (stationary) garden.

planckscnst• 15 years ago

These are typical results. I make networking equipment that has to deal with wireless connections. Our tests are as ideal as you can get without driving around for a tower and sitting under it (although we have done that, too). We're typically in an office, stationary. This means no tower hand-offs, and stable (relatively) conditions. Round-trip-times in the 20 sec range are not rare. I've even seen RTTs a minute long several times, and once saw even greater than 90sec.

What is really interesting is to watch the RTT while you run a speed test.

justincormack• 15 years ago

Interesting, suggests a quick fix might be for the client to not use keepalive, or to selectively close connections that are very slow so as to start new ones. Potentially a much easier solution than writing a new transport.

dspillett4• 15 years ago

The problem for those of us on capped and/or expensive-per-kbyte mobile connections (in the UK that is everyone who doesn't spend a large chunk on their monthly contract - people on Virgin pay-as-you-go pat £3 for a day's access but IIRC you get cut off after 25Mbytes in that day) with restarting connections early is that the ~20 seconds worth of packets queued up during the blip is going to be sent anyway even though they are now no longer needed. 20 seconds worth of discarded packets could be quite a bit if you were transferring data at decent 3G+ speeds just before the blip.

praptak• 15 years ago

It looks like this should be (at least partially) dealt with at the OS level especially if the OS in question is a mobile one.

kalleboo• 15 years ago

There were plenty of wireless-optimized TCP replacements proposed back in the days when WAP and XHTML Mobile were the hottest things around, but none took root as operators, web servers and browsers needed to adopt them in tandem.

Now that smartphone apps are widespread and someone developing a service can control both sides of the connection, there's definitely room for someone to devise a really good TCP replacement (layered on top of UDP) with an iOS library, an Android library, and an Apache mod.

dps2• 15 years ago

I think this is a really good idea. Startup? PhD thesis?

iqster• 15 years ago

Here is a paper from 03: http://www.usenix.org/event/mobisys03/tech/full_papers/chakr...

As I recall, one of the key problems this group found was that GPRS had a very low incidence of packet loss. This was because the physical layer protocol had some error correction built in. TCP wasn't designed for this and ended up doing the wrong thing. This group proposed a custom reliable protocol between the mobile device and a proxy gateway, and their results showed improved performance.

We're not talking about GPRS but I am curious why there is so much packet loss?

Also, just FYI: MobiSys (the top academic conference on mobile systems) is happening in the DC area this week.

leoc• 15 years ago

I'll bet you thought Vint Cerf was just a trophy hiring. :) http://gigaom.com/2009/11/06/vint-cerf-plugs-his-plucky-spac...

jarek• 15 years ago

Do tell how you reckon a start-up could bring to popularity, let alone monetize a new Internet protocol?

grimoire• 15 years ago

I know of at least one company that tried this and failed. It was quite a long time ago and targeted large companies, but it did work. To give you an idea of how long ago, it was written for PocketPC 3.0.

The company is now defunct.

zokier• 15 years ago

SPDY or SCTP maybe? I'm not sure if they work better specifically in high latency environments, but they are touted as TCP replacements.

ominous_prime• 15 years ago

SPDY isn't a tcp replacement, it's a layer 7 protocol on top of TCP.

adaszko• 15 years ago

Actually, SPDY is an application layer protocol -- as HTTP -- not an transport layer protocol like TCP.

silvestrov• 15 years ago

The mobile OS should be able to fix this.

One method would be to disable the reliable delivery mechanism in the 3G code.

Another method would be to use huge retransmission timeouts at the TCP level for connections going over 3G, so the TCP level seldom retransmits packets.

If the 3G implementation provides completely reliable connections, the OS could fake the TCP implementation and just use that implementation directly instead of sending TCP packets. (not entirely unlike the sshuttle VPN).

jwhite• 15 years ago

The 3G connection does not aim to provide completely reliable "connections" -- it aims to provide an as-reliable-as-it-can data link, given channel conditions and configured transmission parameters (which are to some extent controlled by the RAN). If the error rate seen by the channel decoder is too high, it will drop a frame. There is an ARQ mechanism in the MAC layer to try to compensate so that the higher layers don't see the loss, but it implies some delay and also has limits -- i.e. at some point losses due to bit errors injected by the channel may manifest in lost frames and IP packets.

The L1 and MAC of 3GPP are generally implemented in an ASIC and not necessarily visible to the OS. Many of the parameters are dictated and controlled by the RAN (operators are picky and try to control things so that handsets can't misbehave and crap all over the other users in a cell), so the OS doesn't get a look in -- i.e. I don't think it's possible to turn off the 3G error recovery mechanisms.

Disclaimer: I used to work at L1 so someone with more detailed knowledge the MAC and higher layers of the 3GPP stack could give a better idea of what happens when the MAC ARQ limits are hit.

Summary: 3G was designed more than 10 years ago and is showing its age. User requirements have shifted, but standards can't evolve that quickly. Maybe LTE will help provide the user experience for today's apps... question is whether it will be widely deployed in time to matter.

EDIT: updated for clarity and spelling.

crizCraig• 15 years ago

TCP is end to end. 3G is just carries data to your service provider. So wouldn't faking TCP screw up the communication between the mobile service provider and the server you're communicating with?

PaulHoule3• 15 years ago

The article misses that this is fundamentally an issue with the PHY level of "mobile" broadband systems.

If you've got a very simple radio, say, a ham 2 meter handset, you find there are interference patterns caused by multipath interference that cause the signal to get stronger and weaker as you move half a wavelength this way or that way. This causes a fluttering noise when you're listening to somebody transmitting from a car.

A more advanced radio that uses a spread-spectrum signal finds that at any moment of time, some parts of the signal's frequency range is in-phase and some is out-of-phase.

If your goal is to make a low-bandwidth signal robust, you can spread a low-bandwidth signal over a wide spectral range and you won't notice fading. On the other hand, if you're trying to send as much data as you possibly can in the bandwidth you've got, you're ultimately going to implement something like OFDM. Now, in an OFDM system, you're using frequencies in parallel to transmit more data, not to improve robustness. If you're sitting in one place your system can figure out what the fading is and work around it. If you're moving, the relative performance of the subchannels is always changing too fast for OFDM to work.

This is how you can have a good voice connection on your phone (which is using a PHY truly built for mobile operation) but not have it on for data (which is using a PHY built for high performance operation from a fixed point.)

etherealG5• 15 years ago

I probably missed half of what you were trying to explain there, but I didn't see a good solution. Are you suggesting we should give up speed for reliability at the PHY level?

soult• 15 years ago

I don't think you understood the article. It is not about the physical restrictions and trade-offs of mobile broadband. It is about the inefficiency of TCP on such connections. Current TCP implementations are optimized for wired networks which usually have fixed bandwidth and latency. They use dropped packages for congestion avoidance, interpreting package loss as a signal to slow down transmission.

Wireless networks with their higher packet loss (compared to wired ones) fool the sender into believing that the connection is satured. Combine this with the enormous buffer sizes used by mobile broadband systems and the TCP congestion control algorithm throttles the connection to an unusably slow level.

The author is not making a case for trading speed for reliability by using a different PHY, he is making a point for embracing the package loss and delays that exist in current mobile broadband systems and using those characteristics to create a smarter transport layer protocol to replace TCP on mobile devices.

rubyrescue• 15 years ago

we did this at Visto corp a few years back. we wrote our own UDP based protocol that acknowledged every packet. it worked really well in horrible network conditions and was incredibly slow everywhere else. roughly speaking it was a bit like "TCP with no sliding window" and we had trouble w/proxies in the carrier networks. this was pre-3G.

kabdib• 15 years ago

Yeah, but you don't need to do much to TCP to make it work perfectly well on wireless.

Let me repeat that: Don't fuck with TCP. TCP is fine. It's BUILT to deal with your lousy network, it's DESIGNED to deal with nuclear attacks from the Guys of the Bad Color, and (ironically) the worst thing you can do to TCP is to try to be nice to it.

"What about keep-alives?"

There are no keep-alives in TCP. If two hosts are in agreement about the address and port, they can keep a connection in an open state until the universe goes cold. (There may be keepalives necessary /above/ TCP, but TCP itself doesn't use 'em). Don't yammer something about your internal state every 30 seconds; you don't do it on the bus home, so don't do it on your network, either.

"Don't we need this proprietary fancy-schmancy buffering algorithm from FuppedUck Communications Consultants? They keep telling us we're toast without it."

No. Just run TCP. Spend money on decent routers.

"I'm feeling really nervous and I want to, like, disable Nagle for everything. Isn't that what you do? Turn Nagle off and maybe muck with the --"

You're from AT&T, aren't you?

ori_b• 15 years ago

Not entirely true. TCP is certainly decent, but it's not designed to deal with networks where the delays are extremely transient and the capacity is variable. It's not designed for heavy buffering. And so on.

It can be fixed with congestion control protocols tacked on the side, I think, but it's designed for wired networking with slow transients in the performance profile.

It IS, however, better than anything that I'd be able to hack together quickly.

rowanseymour• 15 years ago

Very interesting and sheds some light on the weird latency issues I see here in Rwanda, where 3G issues aren't limited to being on moving trains. Sometimes pinging shows crazy return times of 30000-60000ms. Other times they're only 200-400ms but every other ping packet times out, i.e. one packet through, next one drops, and so on. Still trying to figure out exactly what's happening then.

jchrisa2• 15 years ago

Bufferbloat means when you refuse to drop packets, you get longer latencies: http://www.bufferbloat.net/

Why you'd have big buffers sometimes and not others, I have no idea.

jb55• 15 years ago

We should probably get these long round-trip protocol issues ironed out before we build our galactic internet

merijnv• 15 years ago

TCP is already out as a protocol for our galactic internet, see relevant comment from Linux TCP implementation:

    /*
     * [...] Note that 120 sec is defined in the protocol as the maximum
     * possible RTT.  I guess we'll have to use something other than TCP
     * to talk to the University of Mars.
     * PAWS allows us longer timeouts and large windows, so once implemented
     * ftp to mars will work nicely.
     */

dave1010uk• 15 years ago

(This is from /usr/src/linux/net/inet/tcp.c)

This is correct. At their furthest apart, Mars and Earth are 22 light minutes apart. Their closest is almost exactly 3 light minutes. For comparison, the Earth and Moon are just over 1 light second apart.

On a more serious note, could tcp.c just be patched on the client to drop packets after ~10sec?

mooism22• 15 years ago

I sincerely hope ftp is a distant memory by the time the University of Mars opens.

jerf• 15 years ago

Unsurprisingly, this is a well-known problem: http://ipnpr.jpl.nasa.gov/index.cfm

While I don't know that we can fairly say the Internet qua Internet has been extended past Earth (and associated environs), NASA certainly uses a Solar System-scale network already, and while they haven't made a big deal about some of the routing they've already done, if you read the press releases carefully they'll sometimes mention how they routed the signal from one probe through another. It's already a network.

vbtemp• 15 years ago

I would encourage you all to look at the site dtnrg.org It's for the DTN research group, which among other things is beginning to architect and implement the so-called Interplanetary Internet. Some early trials have been run among NASA, ESA, and JAXA mission-control centers, as well as one flight test on EPOXI (used to be the Deep Impact mission)

kaeso• 15 years ago

As an historical note, most of these concerns are the same expressed in RFC 3481 (category: BCP). You'll note from there that some of the issues are still open even if almost a decade has passed.

lukego• 15 years ago

Don't worry, our Lisp startup (www.teclo.net) is fixing TCP over mobile networks, it will all be fine soon enough. :-)

etherealG5• 15 years ago

perhaps you could explain a bit more about how you're solving the issue?

hollerith• 15 years ago

With Lisp :)

jsnell2• 15 years ago

As noted elsewhere in this thread, the properties of wireless networks conflict badly with the design principles of TCP. We have a custom TCP stack that replaces some of the standard TCP algorithms with new ones carefully tuned for mobile broadband. It can act as a transparent proxy, presenting a standard stack towards the internet and a optimized one towards the radio network. The packet stream is still standard TCP though, so no client changes are needed.

The product is deployed in some >10Gbps networks, and shows very impressive gains for real live traffic.

wibblenut2• 15 years ago

This is partly why I'm so interested in publishing information at the DNS level (i.e. .tel) - you get to use UDP (or TCP failover), plus other awesome benefits. You can do other innovative things with DNS too.

bartonfink• 15 years ago

Of course, that does depend on DNS having a protocol to run over. What we really need is a DWIW protocol suite at each layer.

wibblenut2• 15 years ago

Yep, but the point I was trying to make is that DNS over UDP works well today.

It's also worth remembering that most of the world has 2G connectivity. Heck, even I'm 2G most of the time (rural UK).

dps2• 15 years ago

Dave Taht points out http://www.bufferbloat.net/ which looks very interesting!

aristus• 15 years ago

Here's animation of the packets of a Facebook page hit over 3G on a moving bus: http://vimeo.com/17248120

warfangle• 15 years ago

Would Vint Cerf's recent work on a high-latency network standard for space[0] apply? Would it make mobile more useful? It's designed for latencies of days (not seconds), so it might be overkill. But something to masticate upon...

0. http://www.technologyreview.com/communications/21601/?a=f

etherealG5• 15 years ago

anyone know what tool I could use to run a similar test?

micheljansen• 15 years ago

Interesting. I cannot stop thinking how cool it would be if Google actually decides to step in and propose an alternative protocol for mobile networks. If they put it in Android, they already have a huge base for adoption.

Ended up writing a piece on Google because of this on my blog: http://micheljansen.org/blog/entry/1060

(shameless plug :P)

jchrisa2• 15 years ago

This is exactly what CouchDB, and Mobile Couchbase for Android and iOS, is designed to fix.

Networks are slow. Mobile networks are slower. The most robust fix to the problem is to "optimistically replicate" your application data to the end user's device, so that the network latency does not become part of the user experience.

This is a strong fit for applications like CRM or geographically constrained apps, as the data sets are small enough to fit completely on your devices. For larger data sets the issue becomes: which subset of the data should be copied to the device ahead of time.

The user should never needs to wait on the network. All data operations are played against the local Couch, which handles asynchronously transmitting changes to and from the remote server, in the background. This pattern makes it much easier for app developers to make responsive applications, where users are never left waiting on multi-second round trip times.

willyt• 15 years ago

Is there any way round this for HTML5 apps? I know you can save an app icon on iphone but when you launch it just launches safari which seems to make a network request to check if the site is up to date? (Sorry, I'm a bit naive about all this HTML5 stuff.) e.g. Gmail in safari on iphone is useless when you get long latency situations like this. Is there a way round that?

sebandr• 15 years ago

I'm in a start up that's developed techniques using UDP to allow someone to roam across wifi - in other words we have managed to reduce the tcp delays and time outs to provide consistent and reliable handoffs between wifi zones and devices - regardless what of the network provider. The technology also allows hot handover between femto and wifi too. Right now we're mostly focused on a mobile app to improve broadband delivery of content to mobile users in shopping malls, commercial zones, etc. but that's low hanging fruit. Eventually we believe that this can be integrated in mobile apps to let others us this for true mobility while running broadband services.

hxf148• 15 years ago

Mobile HTML5 apps, the future is the past. :) Check ours out http://infostripe.net

schiptsov• 15 years ago

The much worse problem is DNS. For big networks that pushes always the same two IPs (even without round-robin) it is a disaster. There are lags of servers, lags of network, dropped packets, useless overhead with EDNS and different packet sizes (timeouts and retransmitions) and above all, the practice by content providers and CDNs to use hundreds of changing in real time hostnames to implement load balancing and/or geoIP based assets loading. They use near zero TTLs which makes caching useless and dynamic sets.

Indian Airtel's network is a live example of that disaster. It is almost unusable, while they still actively promoting 3G and iPhones. ^_^

zobzu• 15 years ago

SCTP anyone?

Why mobile apps suck when you're mobile (TCP over 3G)

Related Stories