TCP is UNreliable

http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/

by LiveTheDream • 13 years ago

26 19 13 years ago

tptacek5• 13 years ago

I voted this up (tried to submit, actually), but is this article correct, or is something in this person's stack not actually programming sockets correctly?

When you call shutdown() to arrange a FIN for a socket, especially after you've sent data and received no application-layer acknowledgement, you're supposed to verify that the connection closed properly. You do this by select()ing on the socket and then read()ing 0 bytes.

The error condition (the RST segment they're getting) is signaled by the "Connection closed by peer" errno --- and -1 from read().

The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.

abofh• 13 years ago

Without looking at the code, it's hard to say, typically errors like the one described won't be caught by return values of read/write, but by return values of close() -- which many many people don't bother to check. (Not saying that's what happened, but that's my bet)

tptacek5• 13 years ago

Did they close() the socket? If you write data to a socket, and receive no signal that it's been processed, shouldn't you shutdown(2), select(2), then read(2) before you close(2) it?

BetaCygni• 13 years ago

after you've sent data and received no application-layer acknowledgement

This would be the most robust fix. Have the server return "OK" or something after it has processed the data. It could have silently crashed but still accept connections.

tptacek5• 13 years ago

I'm not sure I see how this is more robust than simply checking to make sure you had an orderly close. The orderly close check should also catch the case where the application crashes after its TCP stack has received the data but before the application processed it.

I'm not saying there's never any point to application acknowledgements; look at any careful SMTP server implementation for a study of how to handle these problems. TCP can promise a reliable transport but obviously not a reliable disk.

oesmith2• 13 years ago

The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.

An ACK only suggests that a segment has reached the recieve buffer successfully. If the receiving application crashes between buffering incoming data and successfully processing it, that's expressly not TCP's problem. The RST will signal that the receiver has died, but does not imply whether or not the application has successfully processed any buffered data up to that point.

AFAIK, an application-layer acknowledgement is the only way to solve this problem.

tptacek5• 13 years ago

I agree that there are circumstances where you need application-layer acks, SMTP being a canonical example.

However, there are other cases where you don't need lock-step app-layer acknowledgements; for instance, if your messages are idempotent.

I'm really just reacting to the forensic work in this article, which suggests (maybe I'm misreading) a problem that TCP doesn't actually have.

notacoward2• 13 years ago

This doesn't sound like a TCP problem.

Mistake #1: The sender optimistically sends a first packet before getting a SYN reply. That implies that the sender issued a non-blocking connect, and then wrote without waiting for the connect to complete. Otherwise, the sender (kernel) wouldn't have any data to send right away.

Mistake #2: The sender sends a FIN, but doesn't block because all data has already been acknowledged. This is supposedly a problem because the app never receives the data, but a TCP ack has never meant that data got beyond the receiving kernel to an app.

So yes, it's true that neither side ever got an error and unfortunate that it's possible for the receiving app to get no data in that case, but that doesn't make it a TCP problem. Somebody at a higher level - JVM or the app itself - over-optimized their "connection and first data" code without understanding the actual behavior of the TCP protocol and socket API. Then they "fixed" it by working with their own protocol instead of against the standard one.

Nothing to see here. It was just a false alarm. Move on.

gonzo• 13 years ago

It's not a TCP problem. It's a problem with the linux implementation of TCP.

noselasd4• 13 years ago

No, it's not even that. The linked article (http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/) at the bottom tells you what's actually going on.

The only thing slightly fishy is that the peer application didn't receive any connection attempt, but if you're anyway quite overloaded, e.g. running out of file descriptors, that may well happen.

dap2• 13 years ago

I think you're right, but TFA appears inconsistent: it says "no errors of any sort...All standard blocking calls." As you point out, it later says that the failed case optimistically sends data after the first SYN, before the SYN/ACK back, but that's not possible if connect() is being used in blocking mode. (Right?)

songgao• 13 years ago

Interesting! One thing that I don't understand (which might be a little off-topic): If it's an RPC call, shouldn't there be a response for each RPC request to ack that the RPC request is well processed by the application layer? Otherwise how do you make sure the application layer won't screw up? How do you notify the client that the request is not invalid?

noselasd4• 13 years ago

Note that he links to http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-s... which more or less explains what's going on.

TCP is UNreliable

Related Stories