Why Turning on HTTP/2 Was a Mistake

https://www.lucidchart.com/techblog/2019/04/10/why-turning-on-http2-was-a-mistake/

by sciurus • 7 years ago

277 151 7 years ago

This is a neat little writeup. Although the issues were not fundamental and relatively easy to spot and fix, it's valuable input especially since http/2 advocates seem to insist that you just need to put your webapp behind a http/2 capable proxy and you won't even notice a difference. We didn't enable it yet on our servers and now there's definitely something to test first before we roll out.

dstaley3• 7 years ago

There's a lot of reasons I expected to see as their justification, but "our application can't handle concurrent requests" wasn't exactly one of them.

macspoofing8• 7 years ago

That's an unfair assessment. HTTP/2 fundamentally changes how requests are handled. With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years (from when ajax became a thing) so this wasn't something on anybody's mind. HTTP/2 all of sudden removes this constraints and for lucidchart, it led to a number of unintended consequences. This is an important consideration because the mantra has been that HTTP/2 can simply be turned on and everything will simply work as before.

dstaley3• 7 years ago

If I'm reading this article correctly, they're claiming their application couldn't handle the load of a single-user loading their webpage. They didn't talk about load spikes during certain times, so it certainly sounds like they just have an inadequate backend.

Aaargh203184• 7 years ago

With HTTP/1.1 there is a defacto connection pool inside the browser and this throttling has been a feature of front-end development for 15+ years

This is only true when you look at a single client. If you look at a larger number of clients accessing the service at the same time, you would expect similar numbers of concurrent requests on HTTP/2 as on HTTP/1.1. Clients send larger numbers of requests at the same time, but they are done sending requests earlier so there are requests from fewer clients being processed concurrently. It should average out.

If you have, say, a 1000 clients accessing your service in one minute, I doubt the number of requests/second would be very different between both protocol versions. It would only be an issue if the service was built with a small number of concurrent users in mind.

jfoutz• 7 years ago

I have a lot of sympathy for their situation. I can imagine humming along just fine at 5 concurrent requests over 10 seconds, having busy days where you hit 10/10

But to suddenly hit 50 for 1 second, then nothing for 9 seconds, well, that’s a tough spot to be in.

There must be some hard to find sequencing happening there, that they were not really exposed to before.

floatingatoll9• 7 years ago

The reverse is also quite stellar, because in some scenarios if you can lower peak burst demand per node from N to (0.1 x N) you can often reduce allocated capacity by a factor greater than 0.1. (This is more likely the case when N exceeds SOMAXCONN, for example.)

theandrewbailey• 7 years ago

There's also no indication on what kinds of requests were timing out, nor if it was possible to send fewer requests, or minimize static assets (if those were the problem).

da_chicken3• 7 years ago

Do you think your local pizza joint could handle every order that they normally get between 4pm and 6pm if they got them all between 5pm and 5:30pm? At the same level of service, with no late orders, and with no additional equipment or labor?

dstaley3• 7 years ago

This is more like if a pizza joint that could seat 10 people at once moved into a new location that could seat 100 people but still kept the same wait and kitchen staff and is suddenly surprised that wait times have increased.

em-bee• 7 years ago

they are not getting them at 5 but at 4. they should be able to handle all orders in the next two hours.

the problem is not making the pizzas in time but trying to get all pizzas started at once when there is not enough table space to even roll out that much dough, and then trying to squeeze all the pizzas into the oven at once, whereby several of them got messed up.

ummonk2• 7 years ago

Well, with every pizza joint I’ve ordered from I can order a bunch of pizzas in just one phone call instead of having to make a separate phone call in serial for each pizza. And I certainly don’t have to wait for each pizza to be delivered before ordering another.

nine_k• 7 years ago

Rather, maybe, "our application can't handle short intense busts, and we haven't built a queuing proxy in front of it"?

That speed observed by clients should come from somewhere. In their case, they did not have a large reserve of performance to tap.

coldtea• 7 years ago

Nice, because that wasn't the justification.

The change in traffic patterns http/2 imposes was.

thayne9• 7 years ago

Author here. Our application can handle concurrent requests just fine. The problem was actually that our application was partly that our application was trying to handle too many requests in parallel instead of queueing them, and partly that later requests were timing out because our load balancers were configured to expect clients to make a request, wait for the response then send the next request, not send all the requests, then wait for all the responses (which means the last response will take longer to complete from when the request was first sent).

michaelt• 7 years ago

Most places I've worked, the timeouts have been calibrated for people on 1990s dial-up, and the 99th-percentile response time targets have been 2 orders of magnitude less.

Presumably that's not the case for you?

mholt• 7 years ago

So, your application can handle concurrent requests just fine, as long as clients make one request at a time?

zikzak• 7 years ago

I don't want to get into it but I've dealt with issues like this and I don't think you are terrible or bad at your job. I'm seeing a lot of shade going your way. Sometimes a particular configuration works well until something changes. Just wanted to offer my two cents.

jrochkind13• 7 years ago

I'm curious what platform your app is on, if you are willing to share?

A typical non-tuned Rails deployment, for instance, is gonna have queueing built in, with really not as much concurrency as one would want (enough to actually fully utilize the host; the opposite problem). So I'm guessing you aren't on Rails. :)

Curious what you are on, if you're willing to share, for how it effects concurrency defaults and options and affordances.

(I know full well that properly configuring/tuning this kind of concurrency is not trivial or one-size-fits all. And I am not at all surprised that http/2 changed the parameters disastrously, and appreciate your warning to pay attention to it. I think those who are thinking "it shouldn't matter" are under-experienced or misinformed.)

unscaled4• 7 years ago

I assume your application is using a dedicated thread for each request?

Even then you would be handling more requests in parallel than the number of cores you have, but your concurrency would be limited by the cost of context switching and your memory capacity (having to allocate a sizable stack for each thread in most threading implementations).

Queueing is usually required for a stable multi-threaded server, but if you were doing async I/O you wouldn't need it. The extra memory overhead for each extra concurrent request (by means of lightweight coroutine stacks, callbacks or state machines) is not much different from the size it would take on the queue, and there is no preemptive context switching.

In most cases, you'll get the same behavior as having a queue here. Cooperative task-switching happens only on async I/O boundaries, so if you're processing a request that requires some CPU-heavy work, your application would just hog a core until it completes the request and then move to the next one.

geofft7• 7 years ago

Whether or not that's a fair characterization of their justification, that seems like a perfectly defensible justification to me. Most people out there are dealing with legacy code (as in "code I didn't write") and business logic they can't safely rewrite, and it's more important to deliver business value than to build an emotionally-satisfying architecture. And even so, why design for concurrency when the high-level task (loading a web page in a bounded amount of time) doesn't require it? Personally, having a system that scales very well that doesn't need to scale isn't even an emotionally-satisfying architecture.

Waterluvian• 7 years ago

This is so true and is why I use "pure" functions where possible. Make it super obvious that a function can be replaced or reused safely and your future counterpart will praise your name to the code Gods.

rmrfrmrf• 7 years ago

Designing for concurrent requests is pretty much a given for web servers.

29athrowaway• 7 years ago

Handling concurrent requests delivers business value.

Migrating to HTTP/2 delivers business value.

Responsive servers are more satisfying to customers than slow servers. That has business value.

Improving tangible metrics is not done to satisfy your emotions, it's done because of engineering rigor.

paulddraper• 7 years ago

There's a lot of reasons I expected to see but "our application can't handle concurrent requests" wasn't exactly one of them.

Hence the blog post.

Most would not think about the fact that your spikiness could increase 20x.

manigandham• 7 years ago

Unless they're making hundreds of requests per visitor I fail to see how the load is shifted so drastically to make such an impact.

layoutIfNeeded2• 7 years ago

Spoiler: they are making hundreds of requests per visitor. Welcome to “modern” webdev!

macspoofing8• 7 years ago

What's wrong with that? Lucidchart is a fully-featured application that runs in a web browser with a cloud-backend. I don't understand what you're trying to argue. That you have to optimize for request count? Why?

stuff4ben• 7 years ago

We experienced issues when we enabled H2 on our HAProxy 1.8 reverse proxies into our K8s cluster. Didn't anticipate the increased memory consumption and we ran into a few memory-related defects with older versions of HAProxy that were fixed with more recent versions. We'll re-enable it at some point, but we've upgraded our reverse proxies in anticipation of it.

StreamBright• 7 years ago

Isn’t it happening because the load balancer does not distribute the http requests evenly? We used to use advanced loadbalancers that took the actual http req from a client and used fix number or tcp connections to backend and distributed the http requests through those. Maybe http/2 does not allow this style of load balancing?

jrockway3• 7 years ago

HTTP/2 does allow this style of load balancing. Whether or not your load balancer does it, however, is another thing completely.

jrockway3• 7 years ago

This is why Envoy exists. It will take HTTP/2 requests from the user and shard the actual requests out for backends to handle. It appears that what happened to the author is that their web server only balanced TCP connections, which indeed no longer works.

thayne9• 7 years ago

We were using AWS ALBs which load balance requests not connections (although it should be noted they do not handle HTTP/2 prioritization). j

jrockway3• 7 years ago

I see. So it sounds like the issue was one of timing, where a bunch of converted HTTP/1.1 requests all arrived at your application at the exact same instant.

Did the ALB open a new TCP connection for each request, or does it use a pool of connections?

zxcvbn4038• 7 years ago

The article is short on detail but it sounds to me like they are balancing their traffic by connection instead of by request. Either nginx or haproxy should be able to spread those multiplexed requests across a number of servers and give more the desired backend behavior.

lxe• 7 years ago

Unless the application/request/session state is pinned to a host.

tantalic• 7 years ago

The underlying lesson that I have learned (the hard way) repeatedly: anything that may change the traffic pattern can result in difficult to predict infrastructure issues. This can be as related as a changing protocol (as shown here) or a seemingly unrelated like a UI change.

deathanatos• 7 years ago

Really, this is an issue in the library/server: the library/server needs to expose HTTP/2's controls on maximum permitted streams.

And secondly, because with HTTP/2, the requests were all sent together—instead of staggered like they were with HTTP/1.1—so their start times were closer together, which meant they were all likely to time out.

No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking.

Honestly, over the long run, this is a feature, not a bug. The server and client can make better use of resources by not having a trivial CSS or JS request waiting on a request that's blocked on a slow DB call. Yes, you shouldn't overload your own server, but that's a matter of not trying to process a large flood all simultaneously. (Or, IDK, maybe do, and just let the OS scheduler deal with it.)

Also, if you don't want a ton of requests… don't have a gajillion CSS/JS/webfont for privacy devouring ad networks? It takes 99 requests and 3.1 MB (before decompression) to load lucidchart.com.

If you do queue requests, you should be careful not to process requests after the client has timed out waiting for a response

This is a real problem, but I've suffered through that plenty with synchronous HTTP/1.1 servers; a thread blocks, but it's still got other requests buffered, sometimes from that connection, sometimes from others. Good async frameworks can handle these better, but they typically require some form of cancellation, and my understanding is that that's notably absent from JavaScript & Go's async primitives.

toast03• 7 years ago

No, browsers can pipeline requests (send the requests back-to-back, without first waiting for a response) in HTTP/1.1. The server has to send the responses in order, but it doesn't have to process them in that order if it is willing to buffer the later responses in the case of head-of-line blocking.

Browsers can pipeline requests on http/1.1, but I don't think any of them actually do in today's world, at least that's what MDN says. [1] And from my recollection, very few browsers did pipelining prior to http/2 either -- the chances of running into something broken were much too high.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection...

TazeTSchnitzel• 7 years ago

cURL recently removed its 1.1 pipelining support, it was rarely used and pretty broken in practice because few clients had been using it: https://news.ycombinator.com/item?id=19595375

Dylan168072• 7 years ago

In my experience firefox used to pipeline a lot of the time, and only disabled it after http/2 came out.

floatingatoll9• 7 years ago

When we first tried to enable HTTP/2 on our load balancers a few years ago, we ended up breaking several applications built on (iirc) gunicorn. We eventually determined the root cause to be:

1) The browser was sending a "streaming data follows" header flag followed by a 0-byte DATA packet in the HTTP/2 stream to work around an ancient SPDY/3 bug.

2) The load balancer was responding to the HTTP/2 "streaming data follows" header packet by activating pipelining to the HTTP/1.1 backend.

3) The backend was terminating the HTTP/1.1 connection from the load balancer with a pipelining-unsupported error.

The browser removed the workaround, the load balancer vendor removed the HTTP/2 frontend's ability to activate HTTP/1.1 pipelining, and after a few months we were able to proceed.

Diagnosing this took weeks of wireshark, source code browsing, and experimental testing. We were lucky that it broke so obviously that the proximity to enabling HTTP/2 was obvious.

unscaled4• 7 years ago

Good async frameworks can handle these better, but they typically require some form of cancellation, and my understanding is that that's notably absent from JavaScript & Go's async primitives.

Go added cancellation support to the standard library at 1.7. I don't like its coupling with contexts, but the implementation is solid and supported throughout most blocking operations, so this statement is patently untrue for Go.

JavaScript really doesn't have a standard way for doing cancellation, which is a shame.

bastawhiz3• 7 years ago

How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit.

It's also curious to me that the load balancer doesn't smooth this out. If you have ten application servers and a client makes ten requests over a single HTTP/2 connection, I'd expect each server to respond to one request each. The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong.

High CPU load should be fine, really, if your application servers are processing requests. If the load is unbalanced, then by definition you need a load balancer to balance the load. If you have one and the load is unbalanced, something is misconfigured.

adventured• 7 years ago

The details are a little fuzzy, but it sounds like the load balancer is only distributing connections, not requests. That seems wrong.

They may be using sticky sessions or affinity in some regard, having the load balancer hold each client connection intentionally to a server. It's not necessarily wrong, entirely depends on what you need to accomplish.

bastawhiz3• 7 years ago

If they had been using sticky sessions, this should have been a problem before, but to a lesser extent. The same server would have needed to process all of the requests for a single client. You'd still have spikey metrics.

This might not be such a problem with one client artificially limited to a single application server. But in practice, it means that individual servers will be overloaded when they are chosen to handle multiple clients concurrently (while other servers are idle).

thayne9• 7 years ago

Author here.

How many requests does your page make on initial load (that can't be handled by a CDN)? If you're making more than six XHRs to your application servers concurrently, this sounds like a problem that would have existed anyway had it not been for the browser's (rather arbitrary) connection limit.

I don't know the exact number, but definitely higher than six. And I certainly agree that is a failing in our application. The point is that the browserd _did_ arbitrarily limit connections, and our application (unknowingly) depended on that.

bastawhiz3• 7 years ago

It seems odd to pin the blame on HTTP/2, no? Nobody made the intentional decision to lean on the browser's artificial connection limit. It sounds like nobody also made the intentional decision to make lots of XHRs on page load, or configure the load balancer to work the way that it's working, etc. Had you been using HTTP/2 all along and made a change to your load balancer (that resulted in its current behavior), your blog post would likely not have blamed HTTP/2 but rather haproxy/ELB/etc.

twblalock2• 7 years ago

I've seen the same kind of thing happen when people switched from one load balancer to another: people were unaware they were dependent on a particular kind of queueing or rate limiting to protect their backend services, and they got hit hard when their new load balancer did not protect them in the same way.

Shorn• 7 years ago

our application (unknowingly) depended on that

Ah, the good old "unrealised infrastructure dependency" - nice to see you my old friend. People that have never been bitten by one of these never built anything worth talking about :)

It's worth observing that Gatling (load testing tool) supports HTTP/2. Once you've got the hang of it, you can fairly easily build load profiles to simulate situations like these. Probably wouldn't have helped you prevent the situation - unknown-unknowns being what they are; but you might find it helpful during remediation.

stingraycharles• 7 years ago

It's also curious to me that the load balancer doesn't smooth this out. If you have ten application servers and a client makes ten requests over a single HTTP/2 connection, I'd expect each server to respond to one request each.

A lot of people set up their load balancers with session pinning (i.e. always choose the same backend based on the session id). This can improve things like cache performance.

Not sure if this is the case here, but it sounds like it.

gregoriol• 7 years ago

What I understand is that their setup/app was broken but mitigated by HTTP/1 which is not that efficient?

schmichael• 7 years ago

I think a glib attempt at a tl;dr would be:

Sometimes "thundering herd" is a feature, not a bug.

To expand: HTTP/1.1 naturally caused the latency of the Internet to pace requests. A sort of implicit intrinsic rate limiting. HTTP/2 intentionally avoids that "problem" by batching/pipelining requests.

j16sdiz4• 7 years ago

There is a client-imposed limit. In chrome, it is 8 for http/1.1 and 1000 for http/2

thayne9• 7 years ago

Author here.

Let me clarify a couple things I've seen in a lot of the comments.

First of all, we do load balance by request, not connections and we do not use sticky sessions.

Secondly, we are aware that our application has underlying problems with large numbers of concurrent requests, which was exposed by using HTTP/2 and we are working on fixing them. The point of the post is not "HTTP/2 sucks because it broke our app". It is "moving to HTTP/2 isn't necessarily as easy as just flipping switch".

otterley11• 7 years ago

Perhaps the title, then, should have been "Why Turning on HTTP/2 Was a Mistake (For Us)" so as not to imply that doing so is a universally bad idea.

geofft7• 7 years ago

I think it was clear enough - it didn't say "Why Turning On HTTP/2 Is A Mistake" or "Why HTTP/2 Was A Mistake," either which would have implied universal badness. The phrasing had me expecting a specific war story, which is what it ended up being.

Besides, I think it's quite reasonable to argue that it's a majority bad idea, even if it's not a universally bad idea. I think many people are probably in the same boat.

macspoofing8• 7 years ago

It's a good thing people don't just read the headline.. Right?

floatingatoll9• 7 years ago

That’s implied by the post context, and doesn’t resolve the clickbait nature of the missing “Why”. A more correct title would be:

“Turning on HTTP/2 increased request burstiness, breaking our application”

da_chicken3• 7 years ago

Titles aren't supposed to be comprehensive, and rhetoric in general is significantly less effective if you spend all your time carving out exceptions for differing opinions. Titles are supposed to get you to read the article and give you an idea of the topic.

If you read the article, it's clearly discussing the problems that they experienced. I don't see why they need to qualify everything they said with IMO, IMX, for us, in our case, etc. At no point in the article itself does the author assert that their experience will match yours. Quite the opposite; the author goes out of their way to describe their environment and why they were impacted in particular. At some point, the reader needs to assume that the author is only talking about what they're talking about -- their specific situation -- and not asserting some omnipotent will to obliterate your opinion or disagreement because they didn't add "(For Us)" to the title.

In other words, complaining that the title isn't explicitly subjective is probably the weakest criticism you could make because, at best, it's a criticism of a rhetorical style rather than any criticism of the substance of the piece. Not only is the reader more than capable of coming to that decision, not only can they be expected to do so if they actually read it, you're not actually disagreeing with anything the article says nor are you presenting any alternative opinion and you're certainly not providing any contrary evidence. You're disagreeing, but there's no substance to your disagreement. "I had to read the article to understand what it was actually about," is not really a criticism!

manojlds• 7 years ago

Isn't it always in some context and scope?

bryanrasmussen• 7 years ago

Was a mistake, given common English usage, implies a mistake in the individual case - that is to say "Was a mistake" means "Was a mistake for us". On the other hand would be a mistake implies a mistake for at least the reader, and commonly is interpreted in the most general application available.

bmm6o• 7 years ago

I'm familiar with your app, but only as a casual user. It's quite nice, btw.

I'm having some trouble picturing this. Can you add some numbers? Like, how many nodes is the load balancer spreading the load over, and how many simultaneous requests were you seeing from a browser?

iamleppert• 7 years ago

It’s hard to tell because there isn’t anything concrete provided.

That said, if this is a traditional web app, this smells to me of a poorly designed application. It sounds like they’re doing on the fly compilation of static assets or something crazy like that, and in any event need to reduce the total number of requests per page or resource and look for opportunities to make things static or cached?

xmichael999• 7 years ago

Not sure what the authors application is, but we run a dozen servers behind http/2 load balanced and dozens of sites and haven't seen anything similar to what he is describing.

thayne9• 7 years ago

Author here.

Our application is Lucidchart (www.lucidchart.com). It is a very sophisticated web-app with significant amounts of dynamic data,running on hundreds of servers. I would imagine applications with less dynamic data and requests that require substantial amount of compute wouldn't run into this problem.

thecompilr• 7 years ago

There is a setting in HTTP/2 called SETTINGS_MAX_CONCURRENT_STREAMS, if set to 1 it works like HTTP/1.1, with no multiplexing. Setting it to 4~8 would make it behave in a similar way a browser actually does with HTTP/1 (creating multiple connections in parallel).

j16sdiz4• 7 years ago

This is a client side setting and default to 1000 in chrome. The http/1.1 equivalent was 8 or something like that.

floatingatoll9• 7 years ago

While it is correct to say that it is a client setting, it is also a server setting. The HTTP/2 specification uses the word "peer" since SETTINGS_MAX_CONCURRENT_STREAMS (0x3) is negotiated in both directions as part of the client/server handshakes.

  $ nghttp -v https://www.google.com | grep -C5 SETTINGS_MAX_CONCURRENT_STREAMS
  [  0.076] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
            [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
  [  0.091] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
            [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]

ec1096854• 7 years ago

ALB clips this at 128: https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...

Izmaki• 7 years ago

I was hoping to experience this as a lucidchart visualisation of "sweaty spikes" and "work spreaders" because of too many "Internet tubes". Oh well. Another time :D

melan132• 7 years ago

This is where micro-services shine.

macspoofing8• 7 years ago

Why? This has nothing to do with microservices.

melan132• 7 years ago

It does, think twice about the CPU flow.

emmelaich• 7 years ago

It's actually a somewhat difficult thing to predict spikes. Say you have n clients, d timeslices. When does the probability exceed 0.5 that you get more than k requests concurrently? Unfortunately the solution is exponential in k.

People might recognise the birthday problem here; for d=365, k=2 (days a year, single share) the well known answer is 23.

Wikipedia gives a formula for a rough approximation for n for p=0.5 and k < 20.

nhumrich• 7 years ago

Do you terminate http/2 at the load balancer and convert it to http1.1? Or do you support http/2 all the way to the end service? I would imagine the former would solve these issues.

unscaled4• 7 years ago

If your service can't handle well a bunch of requests coming at once, it doesn't matter if it gets them as individual HTTP/1.1 requests or as multiplexed requests in HTTP/2 coming directly. It only makes a difference if the bottleneck is HTTP/1.1 parsing logic.

stevefan1999• 7 years ago

First of all, turning on HTTP/2 was not a mistake.

Second. turning it on when it is not mature yet is.

floatingatoll9• 7 years ago

The maturity of HTTP/2 is not a causative factor here. They removed a previously-unaware limit on the number of concurrent backend requests, which overflowed their backends. They could have experienced this same outage by simply removing that limit without enabling HTTP/2, and then hitting a peak demand period that was sufficient to cause the outage. Yes, HTTP/2 changes traffic patterns, but the issue could easily have occurred with HTTP/1 as well.

StopHammoTime• 7 years ago

Great article on approaching massive technical change. Honestly, I think a lot of people general think that most things are just a "switch flip". Even something has implementing SSL on internal apps can cause a big change, let alone the underlying protocol for managing your requests.

Thanks for this, because honestly I hadn't thought about the implications myself and it'd be good not to accidentally walk into this problem.

tie_• 7 years ago

Surely your application/serving process is able to handle the request burst coming from any single user ? (If not, you have a bigger problem to solver first).

If so, I don't quite see why queueing is discussed as an option at all. Queueing means extra latency and worse user experience (not to mention DoS potential).

What you should be discussing instead is how to (auto-)scale your app and infrastructure to handle your users' requests.

KaiserPro• 7 years ago

decreases latency by multiplexing requests on the same TCP connection

On a decent connection, kinda. Anything mobile or worse mobile and moving will suffer terribly.

EugeneOZ• 7 years ago

Pathetic attempt to get some users from HN.

Why Turning on HTTP/2 Was a Mistake

Related Stories