advm

http://typedrummer.com/dj76r0

I was kind of hoping to see Flareon[1,2] included, even if it is not #bigdata per se.

[1] http://www.solarflare.com/Flareon-Server-Adapters

[2] http://bulbapedia.bulbagarden.net/wiki/Flareon

Add refraction, I want to make a laser.

Keep doing what you're doing now: taking a break. You evidenced some initial interest in drawing or modelling, perhaps go back to doing that for a hobby. Reconnect with your friends, as you are. Perhaps go on a trip. You're burned out because you've been pushing yourself too hard.

Give your mind some time to relax. Take all the time that you need. Maybe a few months. Maybe a year or more. The time will come when you know you're ready to get back to changing the world.

http://www.phdcomics.com/comics.php?f=1174

Maybe TCP's issues aren't apparent when you're using it to download page assets from AWS over your home Internet connection, but they become apparent when you're doing large file transfers between systems whose bandwidth-delay products (BDPs) greatly exceed the upper limit of the TCP buffers on the end systems.

This may not be an issue for users of consumer grade Internet service, but it is an issue to organizations who have private, dedicated, high-bandwidth links and need to move a lot of data over large distances (equating to high latency) very quickly and often; CDNs, data centers, research institutions, or, I dunno, maybe someone like Google.

The BDP and the TCP send buffer size impose an upper limit on the window size for the connection. Ideally, in a file transfer scenario, the BDP and the socket's send buffer size should be equal. If your send buffer size is lower than the BDP, you cannot ever transfer at a greater throughput than buffer_size / link_latency, and thus you cannot ever attain maximum bandwidth. I can explain in more detail why that's true if you want, but otherwise here's this: http://www.psc.edu/index.php/networking/641-tcp-tune

Unfortunately for end systems with a high BDP between them, most of the time the maximum send buffer size for a socket is capped by the system to something much lower than the BDP. This is a result of the socket implementation of these systems, not an inherent limitation of TCP.

An accepted user-level solution to this issue is to use multiple sockets in parallel, but that has its own issues, such as breaking fairness and not working well with the stream model. I can explain this more if you want, too, just let me know.

As for zero-copy with sendfile, well, even when you do zero-copy, the above issues still apply, because the socket buffer is still used. Admittedly my research into zero-copy is very cursory, but from what I understand, even when you use sendfile, it still copies from files into the TCP send buffer, so zero-copy is actually less zero than it seems. It just doesn't require the whole user space read-buffer-write-buffer loop, which does yield a noticable performance increase, but that doesn't mean the buffer magically goes away and the BDP issue is solved.

What I was suggesting, by using an mmap'd file as the send buffer, would allow the TCP send buffer issue to be circumvented completely. In a user space protocol specialized for file transfers (not saying QUIC is, but I'm not talking about QUIC), you wouldn't need to use a send buffer, and the window size would become limited only by the BDP; the file contents are always available without needing to buffer them like the socket API does, and the network will never have to be idle just because the send buffer is capped at 10 MB or something.

You mentioned QUIC needing to do the whole userland dance to let the application parse packets and all, and that is a valid point, but, in the case of reading, it's entirely possible to only read in the header data you need from the datagram and let the kernel take care of as much of the actual copying as it can for the rest of the packet. In the case of writing, you can "plug" the socket, write headers in user space, then zero-copy the file segment for added performance.

Certainly there is a performance hit, since it's still going into user space, but there wouldn't be if this was all part of the kernel. Like I was saying, I'm not so much excited over the fact that this is user space or anything, but more so that it has the potential to exert pressure on the Linux community and such to make finally make some broad and well-needed changes to the network stack. Even if QUIC isn't being designed to address the issues I'm mentioning, maybe a user space-configurable socket library with the backing of Google will make experimentation with internals normally obscured by the socket API more accessible to people, and something good will come out of that eventually.

Anyway, all of these issues stem from the fact that BSD sockets are meant to be a general purpose communications protocol. However, some applications (such as file transfers) don't need all of the amenities general purpose sockets offer, such as separate kernel-allocated buffers so applications don't need to keep buffers after calling write(), etc.

There are other problems with TCP, such as slow start being, well, slow to converge on high-BDP networks, bad performance in the face of random packet loss (e.g., TCP over Wi-Fi), congestion control algorithms being too conservative (IMO, not everyone needs to agree on the same congestion control protocol for it to work well, it just needs to converge to network conditions faster, better differentiate types of loss, and yield to fairness more quickly), TCP features such as selective ACKs not being widely used, default TCP socket settings sucking and requiring a lot of tuning to get right, crap with NAT that can't be circumvented at the user level (UDP-based stream protocols can do rendezvous connections to get around NAT), and more. People write whole papers on all these things, though, and I don't want to make this even more of a wall of stupid text than it already is.

Okay, you can call me out for not having citations or backing up my claims again now. Problem is most of the public research exists as a shitty academic papers you wouldn't probably bother reading anyway, and most of the people actually studying this stuff in-depth and coming up with solutions are private researchers and engineers working for companies like Google.

This is more than merely UDP with encryption. Rather, this seems to be a reliable stream protocol that aimed at dethroning TCP. I don't think it intends to replace UDP. If anything, I would describe it as a user space replacement for TCP. It uses UDP, yes, but that's because UDP is about as close as you can get to IP at the user level.

It's about time, too. Certainly user-level substitutes for TCP are not a new idea (I've worked a good deal with UDT), but this the first one to my knowledge that has the weight of something like Google behind it.

There are a lot of advantages to doing this as the user level with UDP as well. Not needing to install a kernel module, for one. But also P2P rendezvous connections through NAT with UDP (we wouldn't even need to wait for router manufacturers to adopt anything), better control over protocol options without going through the kernel, custom congestion control algorithms, finer control over buffers (mmap'd files as buffer for file transfers anyone?), etc.

I'd much rather see this at the kernel level, but TCP as it's implemented on most systems has too many issues on modern networks, and most acceptable user-level solutions (ugh, parallel streams) are really terrible hacks. If Google does it right, maybe it'll force OS developers to wake up and rethink the increasingly clunky and antiquated BSD sockets API that is the source of many of TCP's woes and finally modernize the network stack for the fiber era.