hobohacker

Ex-USDS here. Did 6 months at USDS, still federally employed as a USDS reserve.

Just wanted to chime in on "Who in their right mind would go to work for the government, for minimum wage, unless they were already independently wealthy?" GS-15 pay is nowhere near minimum wage. If you actually think so, I'm sorry to say that you are out of touch with society. And while some USDS employees may be independently wealthy, the majority certainly aren't.

There are also the nice perks of USDS, like working on things that critically matter to the lives of millions of the neediest demographic (both domestically and internationally), and collaborating with a bunch of A+ human beings who, despite the personal sacrifices, passionately want to do what they can to improve the welfare of others.

To answer your specific question about why not in TCP, it's not deployable. Here's why:

* In order to get multiplexing and other features introduced with HTTP/2, you need to change protocol framing. However, this means that the protocol is no longer backwards compatible. There are many ways to roll out non-backwards-compatible changes, but for TCP, they were deemed unacceptable. For example, you could negotiate the protocol change via a TCP extension. However, TCP extensions are known not to be widely deployable (https://www.imperialviolet.org/binary/ecntest.pdf) over the public internet. You could use a specific port, but that doesn't traverse enough middleboxes on the public internet (http://www.ietf.org/mail-archive/web/tls/current/msg05593.ht...). Yada yada.

* More importantly, TCP is generally implemented in the OS. That means, updating a protocol requires OS updates. That means we need both server and client OS updates in order to speak the new protocol. If you look at the very sad numbers on Windows versions uptake, Android version uptake, etc, you'll understand why many people don't want to wait for all OSes to update in order to take advantage of new protocol features.

To quote Roberto (http://www.ietf.org/mail-archive/web/tsvwg/current/msg12184....) in his response to IETF transport protocol folks asking why application protocol folks are hacking around the transport:

""" I understand the points you make, and am sympathetic. I feel the same way when I see people abusing HTTP to provide async notifications, etc. The fact is, however, if it isn't deployed, no matter how nice it would theoretically be when it is, it isn't useful and people WILL work around the problem. That is true regardless of whether the problem is at the application layer (e.g. HTTP), or at the transport layer (e.g. TCP), or elsewhere. The primary motivation is to get things working, and making things that lack redundancy, or are elegant comes in at a distant second.

Deployed is the most important feature, thus things which quickly move protocols and protocol changes from theoretical to deployed are by far the most important things. The longer this takes, the more likely that the work-around becomes standard practice, at which point we've all "lost" the game. -=R """

Basically, this is another instance of Linus's quote (https://lkml.org/lkml/2009/3/25/632): "Theory and practice sometimes clash. And when that happens, theory loses. Every single time." Theoretically, it'd be far more elegant to fix this in the transport layer. But in practice, it doesn't work. Except, maybe if you implement on top of UDP (e.g. QUIC), since that would be in user-space and firewalls don't filter out UDP as much.

For more extensive analysis of HTTP/2's deployability options, amongst other considerations, check out my post on HTTP/2 considerations: https://insouciant.org/tech/http-slash-2-considerations-and-....

Er, I thought I was agreeing with you. What do you think I'm arguing? To be clear, I view these two statements as grounded in the same logic, although perhaps one is more strongly worded than the other:

Yours - "So unless someone has the resources to test and deploy unstable web servers, it effectively means you shouldn't bother with spdy at this point."

Mine - "I think it's a fair assessment that supporting experimental technologies requires more engineering resources. Everyone has to do the cost/benefit analysis themselves."

I think it's a fair assessment that supporting experimental technologies requires more engineering resources. Everyone has to do the cost/benefit analysis themselves.

Speaking as a Chromium SPDY & HTTP/2 developer, we are very much focused on standardization. SPDY is an experimental protocol meant to drive the standards process, not become a de facto standard itself. Therefore, it's critical for us to kill off old SPDY versions.

For more discussion on this topic, please refer to these discussion threads: https://groups.google.com/forum/#!topic/spdy-dev/_uvxTJkeCP0 https://groups.google.com/forum/#!topic/spdy-dev/XDudMZSq3e4

And note that your reference is slightly misleading (I don't think you intended this). It's true, the vast majority of hosts supporting SPDY are running nginx. In practice though, the vast majority of these hosts are Cloudflare or WordPress.com hosted sites. Both run newer versions of nginx with SPDY/3.1 support.

I don't see why you think this is a semantic requirement of HTTP. Perhaps there's some confusion over what HTTP semantics are. Let me refer you to http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-2.... It doesn't discuss exposing all HTTP traffic to network intermediaries. Perhaps you're thinking of the HTTP messaging layer http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-2.... Also, I think your statement about allowing an in-path intermediary to act as a CDN is weird, since a CDN is defined as "a large distributed system of servers deployed in multiple data centers across the Internet. The goal of a CDN is to serve content to end-users with high availability and high performance." [1].

It's true, HTTPS is full of tradeoffs. You've identified some of them.

What do you see in HTTP/2 that "codifies this workaround"? That wasn't immediately obvious to me. Recall that HTTP/2 is basically just multiplexing with prioritized streams. There's no requirement on TLS in the spec, although all current browser deployments (of SPDY) require TLS.

[1] http://en.wikipedia.org/wiki/Content_delivery_network

Only time will tell. It's all still in progress. Of all the major browser vendors (Firefox, Chromium, IE) present at the Zurich HTTP/2 interim meeting, only Patrick McManus (Firefox) has expressed interest. Notably, he's a co-editor of that Alternate-Services internet-draft.

The specification indeed is about proxying http resources, not https ones. So it's not initially as alarming as some other proposals discussing trusting proxies to intercept SSL connections. For more details, you can refer to https://insouciant.org/tech/http-slash-2-considerations-and-....

This specific proposal is interesting because it specifically is related to opportunistic encryption proposals, in particular, the one that allows sending http:// URIs over an unauthenticated TLS connection: http://tools.ietf.org/html/draft-nottingham-httpbis-alt-svc-.... The problem here for proxies is, if you mix http and https (authenticated) traffic on the same TLS connection, the proxy cannot tell if it can safely MITM the connection. The proxy vendor would like to know if it can do so, probably for network management / caching / content modification reasons. Of course, the point of the opportunistic encryption proposal is to increase security (although its actual effective impact is controversial: https://insouciant.org/tech/http-slash-2-considerations-and-...). But if you believe in opportunistic encryption's security purposes, then it doesn't seem to really make sense to make the MITM'able traffic identifiable so proxies on the network path can successfully MITM them without detection.

As Patrick McManus says in http://lists.w3.org/Archives/Public/ietf-http-wg/2013OctDec/...:

If someone can install a root cert onto your computer then you are already owned - there is no end to the other things they can do too. Call it a virus, call it an enterprise, but call it a day - you're owned and there is no in-charter policy this working group can enact to change the security level of that user for good or for bad..

The good news is not everyone is already owned and SSL helps those people today.

Perhaps you should look at the Hacker News comments on that thread: https://news.ycombinator.com/item?id=7249193. Notably, my comments: https://news.ycombinator.com/item?id=7249560 and https://news.ycombinator.com/item?id=7249869. Basically, the author is wrong.

HTTP/2 does not change the meaning of http://. That's the opportunistic encryption proposal: http://tools.ietf.org/html/draft-nottingham-httpbis-alt-svc-.... For more information, you can see https://insouciant.org/tech/http-slash-2-considerations-and-....

I guess I should out myself as a Chromium HTTP stack maintainer (since 2009, so this behavior predates me). One might consider me a domain expert here. I participate in IETF HTTPbis for the HTTP/2 work as the primary Chromium representative. I am not involved with the RFC 2616 revision work as that's tough, thankless work, that thank god we have Julian Reschke and Roy Fielding working on. As far as I'm concerned, I owe them a drink everytime I see them. They do an awful lot of legwork talking to various implementations and trying to build consensus on actually conforming with the standard and all its edge cases. It's really quite unfortunate to see this blog post author treat them so unfairly, although I can see how one might easily jump to his conclusion.

Now, as far as "SHOULD NOT", that's a reasonable thought for people not aware of what popular user agents currently do. The thing is, the majority of major browsers rewrite POST to GET on a 301. Here's my browser's code for it: https://code.google.com/p/chromium/codesearch#chromium/src/n.... Here's Firefox's code for it: http://mxr.mozilla.org/mozilla-central/source/netwerk/protoc.... To my knowledge, all browsers implement this behavior. We basically copied IE's behavior, because, IE did it and websites expected all user agents to do what IE did. Story of the web, sound familiar? :P

So, as you can see, Julian was merely acknowledging the pragmatic reality of the situation when he updated the httpbis specs to reflect this behavior: http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1428. And this reasoning behind it is covered in the introduction to the relevant section in the httpbis docs: http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-2....

"SHOULD NOT" implies that our implementations are behaving badly. Now, it's true, our implementations may not be behaving ideally from a spec cleanliness point of view, but interop trumps spec cleanliness, at least from the perspective of anyone who actually deploys real software on the internet. So it's probably best for the spec to acknowledge this and officially allow this. Specs that don't mirror reality are...probably not just useless, but actively harmful.

I hope that explains things, cheers.

I think the author of this blogpost has a few things off:

- HTTP2 != httpbis. Both work is being done by the same working group "httpbis". http://datatracker.ietf.org/wg/httpbis/charter/ covers this. httpbis (http://stackoverflow.com/questions/9105639/httpbis-what-does...) was originally chartered to revise HTTP/1.1 (RFC2616) The working group will refine RFC2616 to: * Incorporate errata and updates (e.g., references, IANA registries, ABNF) * Fix editorial problems which have led to misunderstandings of the specification * Clarify conformance requirements * Remove known ambiguities where they affect interoperability * Clarify existing methods of extensibility * Remove or deprecate those features that are not widely implemented and also unduly affect interoperability * Where necessary, add implementation advice * Document the security properties of HTTP and its associated mechanisms (e.g., Basic and Digest authentication, cookies, TLS) for common applications

As for the HTTP/2 work, here's a snippet from the charter on that: The Working Group will produce a specification of a new expression of HTTP's current semantics in ordered, bi-directional streams. As with HTTP/1.x, the primary target transport is TCP, but it should be possible to use other transports.

- He seems to think the httpbis folks gratuitously redefined 301. It should be noted that RFC2616 (which, by definition, predates the httpbis work since httpbis is defined to revise RFC2616) had already noted the issue with 301 (http://tools.ietf.org/html/rfc2616#section-10.3.2): Note: When automatically redirecting a POST request after receiving a 301 status code, some existing HTTP/1.0 user agents will erroneously change it into a GET request.

- It's unclear to me whether or not the author acknowledges the existence of buggy implementations as noted in section 10.3.2. It's an open question as to what to do in the presence of buggy implementations. From a server standpoint, if the client is buggy, and you don't want to break the client (willingness to break clients probably depends on how many of the server's users use that client), then you will attempt to work around it, irrespective of what the standard says. Therefore, it's simply pragmatic to ignore the spec if it doesn't mirror reality, and pragmatic spec editors may update the spec to acknowledge this difference.

- As far as current status of the various 308 usages, Julian (author of the 308 draft) is lobbying major user agents to adopt this, and has written up a status update on the Chromium bug tracker: https://code.google.com/p/chromium/issues/detail?id=109012#c....

That's true, which is why I was careful to say in its original form :) Since the original attack was on cookies (request headers). To my knowledge, no other SPDY server defaults response header compression to off. But yeah, if your application does pass secrets in response headers, you should be careful.

Let's be clear that the original CRIME attack was against request header secrets. Therefore, disabling response header compression (as nginx defaults to) does not prevent that. SPDY/3.1 request header compression is a client-side choice, not server-side.

Head of line blocking is when something ahead in the line blocks things behind it. If browsers choose to use connection limits, then that means that things ahead in the line and get to use the connections are blocking things behind in line which have to stay queued up waiting for the connections to free up. So yes, that's head of line blocking.

As for pipelining working great in those browsers, you should note that no major desktop browser uses pipelining. Firefox does not enable it by default. See the Firefox bug thread where it's explained why it's not on by default, and also identifying its head of line blocking issues: https://bugzilla.mozilla.org/show_bug.cgi?id=264354#c30.

Microsoft did test pipelining. In their theoretical lab network, with ideal situations, they can match SPDY performance. But the sad reality is that real websites and networks don't match that. Please see http://www.guypo.com/technical/http-pipelining-not-so-fast-n.... And you have to ask yourself, if Microsoft really believed that result, why don't they enable pipelining in IE?

It's fascinating to me that you still seem to hold onto your claims that HTTP pipelining is feasible in real networks, even though IE, Firefox, and Chrome developers have all clearly tried and given up on it.

To your points about pipelining, there's a reason it doesn't get compared. It's not usable - https://insouciant.org/tech/status-of-http-pipelining-in-chr...

Head of line blocking is so well documented that there are many common web performance techniques to get around it, like domain sharding: http://www.stevesouders.com/blog/2009/05/12/sharding-dominan.... It'd be very interesting to see where you get your data that head of line blocking is not a problem.

I'm not sure where you are getting your info that SPDY was designed largely by one recent college grad. I don't think of Mike Belshe and Roberto Peon as the same person, much less recent college grads :) Maybe 40+ year olds still counts as "recent"?

I'd love to hear what's stopping you from forcing SSL site-wide. Is it cost? If so, what are the specifics if you don't mind sharing?

Also, I don't know if you're using different VIPs for load balancing or lack of SNI support reasons or what not, but if your certificate provides proof of authentication for all your hostnames (probably need to use SubjectAltNames and maybe wildcards too) and the VIPs match, then Chrome & Firefox will send requests for those different hostnames over the same SPDY connection.

https://code.google.com/p/chromium/issues/detail?id=22423 discusses some of the issues with using SRV records in browsers.

SRV records could help clients discover server HTTP/2 support, but it does not mean that all intermediaries along the path support it.

I'm not going to argue whether or not other libraries/protocols (e.g. your example of protobufs) may provide higher value. I was simply questioning your assertion "SPDY has no benefit for mobile APIs."

That said, I've got some comments on your new points:

* "Almost all APIs are transactional. Your API should be returning all the data you need to render a single screen in your mobile app, or it is inefficient." - Addressing this completely would take awhile, and there's no good point in our discussing this exhaustively. I'll simply note that while an API response may return all data necessary to render a single screen in the app, it does seem nice to allow for prioritized out of order responses like SPDY does. There does not seem to be a good reason to have head of line blocking in the responses, since the app should be able to render incrementally.

* "Pings are not the correct way to deal with hangs. I don't want to spill any secret sauce here, but it should be obvious to anyone with low level TCP experience." - I think you must be misunderstanding this, or it must not be obvious Google's TCP team, since they agree with the usage of SPDY PINGs. Perhaps you think SPDY PINGs are a keep-alive mechanism? Note that the article I linked to identifies them as a liveness detection mechanism.

* "Pipelining also uses a single connection. Opportunistic FINs are not unique to SPDY." - I don't know why you bring up pipelining again when I've pointed out that multiplexing is superior. Why put up with response head of line blocking? And I don't know what you're exactly referring to with opportunistic FINs, perhaps you can explain in further detail?

And I'll throw in extra data points. Despite the fact that you don't feel like it's useful for mobile APIs, other non-Google parties clearly do:

Square (https://github.com/square/okhttp has their Android SPDY implementation)

Twitter (https://github.com/jpinner has their SPDY developer's work on Netty, which they use in their mobile APIs)

http://www.chromium.org/spdy/spdy-whitepaper has data on improvements in lab tests. http://googlecode.blogspot.com/2012/01/making-web-speedier-a... has a blurb where Google announces that they've made search (already highly optimized) faster with SPDY. This is a fascinating result, because this result was obtained after Google Search switched to using HTTPS, which typically makes websites slower, but in Google's case, made it faster because of SPDY. Note that Twitter and Facebook have also adopted SPDY. I'd be rather skeptical that Google, Twitter, and Facebook would all switch to SPDY if there weren't real world benefits.

Why do you assert it has no benefit for mobile APIs? Here are a few that I can think of off the top of my head:

* SPDY Multiplexing is superior to HTTP pipelining. Pipelining requires in-order responses, which leads to head of line blocking.

* SPDY header compression is a win for mobile, since mobile uplink bandwidth is often a bottleneck. Request header compression allows for fitting more requests into fewer packets.

* Mobile connections are more likely to hang, so using SPDY PINGs (https://insouciant.org/tech/connection-management-in-chromiu...) will help fail fast.

* Multiplexing more requests over fewer connections will help with mobile power usage (https://insouciant.org/tech/connection-management-in-chromiu...).

Can you clarify this statement? Perhaps there's missing context, but as it reads, it seems like a false statement. When I look at chrome://view-http-cache/, I see a bunch of https:// resources for google.com (which are served over SPDY).

It depends on what you mean by atomic. Please refer to http://www.chromium.org/developers/design-documents/indexedd... where Chromium devs state: """ LocalStorage is inherently racy or a parallelism disaster, depending on whether or not you're willing to implement the "storage mutex" described in the spec. Chromium has decided not to implement it. WebKit itself is single thread/process (i.e. no parallelism period). """

Note that the way Chromium implements localStorage, it uses a disk backing store of sqlite, a caching layer (a std::map) in the browser process, and a caching layer (also a std::map) in the renderer processes (where WebKit runs, and may handle 1-X tabs per process). Changes are propagated via IPC, but as the API does not provide any locking guarantees beyond the optional storage mutex, which Chromium does not implement, there are obviously race conditions.

This probably explains the issues the author of that post encountered with Chrome.

The primary caveat with this approach is race conditions due to lack of locking. Note the "Issues" section in http://www.w3.org/TR/webstorage/ which says:

""" The use of the storage mutex to avoid race conditions is currently considered by certain implementors to be too high a performance burden, to the point where allowing data corruption is considered preferable. Alternatives that do not require a user-agent-wide per-origin script lock are eagerly sought after. If reviewers have any suggestions, they are urged to send them to the addresses given in the previous section.

More details regarding this issue are available in these e-mails (as well as numerous others):

http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-Sep... http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-Dec... """

So as long as you don't need transactions/locks across the shared state, this might work. Of course, transactions/locks are often necessary.

No, we have never had an infinite cache. What is more likely is that your friend may have encountered a bug. If he has information on this issue, please file a bug at crbug.com/new and I will be happy to triage it.

I am said Chromium developer.

Let me respond to this comment from the article: """ This is not the case for Chrome: the browser keeps all the cached information indefinitely; perhaps this is driven by some hypothetical assumptions about browsing performance, and perhaps it simply is driven by the desire to collect more information to provide you with more relevant ads. Whatever the reason, the outcome is simple: over time, cache lookups get progressively more expensive; some of this is unavoidable, and some may be made worse by a faulty hash map implementation in infinite_cache.cc. """

Chromium (and thereby, Google Chrome) does not cache forever. The author is clearly misled by the infinite_cache.cc file he referenced. That is our experiment file, designed to examine a theoretical "infinite" cache's performance for data gathering purposes. It doesn't actually cache the resources, but just records the keys (basically, the URL). It only runs on a small set of user browser sessions (only for users who opt-in to helping make Google Chrome better and a subset of their browsing sessions).

As my previous Google+ post mentions (thanks for the parent for linking it), we cap the cache size at 320MB. The author is simply factually incorrect about the aforementioned claim.

As for cache performance as the cache gets larger, I fully believe that it gets slower. We have data that backs up this assertion. Of course, larger caches means that more gets cached. And there are ways to restructure the cache implementation to avoid the painful latency on cache misses. While cache misses are indeed a large percent of resource requests, it is misguided to analyze the cost of cache misses in isolation. For the opposite argument about how we should be increasing cache sizes, see Steve Souders' posts: http://www.stevesouders.com/blog/2012/10/11/cache-is-king/, http://www.stevesouders.com/blog/2012/03/22/cache-them-if-yo..., etc.

The caching issues are far more complicated than described in the original post. The data is much appreciated, and we have similar data that we're looking at as we're making our decisions about caching.

Sorry, I'm a Chromium dev, and in particular I'm the net maintainer who authored the majority of the current connection management code you are discussing. Our results are not public. The web compat concerns are with more exotic, enterprise configurations, not public ones. They are not the most compelling concerns though.

Chromium builds predictive models which are stored client-side. They are not uploaded to a third party server, like Google.

We've considered this before, and the performance gains do not appear worthwhile (we've run much more extensive tests than you have on top websites to evaluate potential gains), it has extra complexity, it is more of a Hampering Eyeballs approach rather than a Happy Eyeballs approach, it has potential web compat issues (many sites will not expect you to connect to the IPs other than the first), etc etc.