HTTP/2 Denial of Service Advisory

https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-002.md

by rdli • 7 years ago

191 39 7 years ago

judge2020• 7 years ago

Cloudflare post on this: https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks...

Entirely unsurprising. With all this complexity, HTTP2 is on par with a full TCP/IP stack. All major operating systems had decades to optimize and bulletproof these, and still to this day we find issues with them every now and then. What did people expect would happen when we start reinventing the wheel yet again, on top of what we already have?

And this is just the tip of the iceberg. Consider this a warm-up exercise.

JoshTriplett• 7 years ago

Also, issues like CVE-2019-9517 ("Internal Data Buffering") provide strong motivation for integrating those layers, as HTTP/3 is doing.

the_duke• 7 years ago

We can happily look forward to HTTP/3 then.

It actually IS a full stack...

persistent• 7 years ago

Putting HTTP/3 on a userspace QUIC stack solves a number of these problems.

mey• 7 years ago

I'm curious about how practical UDP amplification/reflection attacks will be in the wild with QUIC. A cloudflare blog mentions it in passing ( https://blog.cloudflare.com/head-start-with-quic/ ). I haven't dug into the protocol, but since what replaces the TCP SYN/ACK is now also setting up encryption, a lot more data is flying back and forth.

SCHiM• 7 years ago

Indeed. Looking at the descriptions of the attacks all these are simple. They probably would have occurred to any halfway competent attacker looking at ways to DOS your sever in the first several hours of playing around with it.

Consider that one of the attacks is described as a "ping flood", remember when we first dealt with that? Decades ago. And the "data dribble" looks like a re-heated version of the HTTP "Slowloris" attack.

It's extremely regrettable that the creators of the vulnerable software didn't take a look at any of the plethora of existing attacks and imagine how they might be adapted to attack their implementations.

zxcmx• 7 years ago

The other way to look at it is that the only people who deeply need to care about things like this are cloud providers.

Basically anyone who cares about DoS resistance is using a CDN and/or cloud load balancers, or has enough scale to build out their own CDN.

Otherwise you'll get soaked at layer 3 anyway, L7 DoS resistance is "nice to have" but not enough.

cmroanirgo• 7 years ago

I am very interested in my little vps not being reduced to a smoking heap just because some script kiddie wanted some cheap lulz. Of course, it's not hard to dos a tiny vps, but I think it very important not to court unnecessary trouble.

drawkbox3• 7 years ago

Version 2 syndrome on full display aka the "second-system effect" [1].

The second-system effect (also known as second-system syndrome) is the tendency of small, elegant, and successful systems, to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence.

They "simplified" it by making it more complex and layers of binary/connection complexity... failing to address many common attack vectors and adding fully new attack vectors that should have been expected. Now debugging those in HTTP/2 is harder and in a big binary blob.

I miss text based/MIME HTTP 1.1, simplicity should be the goal always, not more complexity that solves almost nothing, but HTTP/2 did give some big orgs control of the layer/protocol/standard which was most of the driving force.

Much of this could have been fixed lower level in SCTP or something similar on the transport layer, but instead we get another abstraction on top of it all, with more holes and tougher to debug.

Solution: let's do it again in HTTP/3... new standard to fix all the old standards, more complexity that leads to misunderstanding in implementations that leads to more attack vectors. Obligatory xkcd [2]

[1] https://en.wikipedia.org/wiki/Second-system_effect

[2] https://xkcd.com/927/

lucaspardue2• 7 years ago

Versions prior to HTTP/2 give the appearance of being simple but in actuality there are many many edge cases that also are the cause of vulnerabilities.

HTTP/2's binary syntax addresses some of those edge cases mainly through better-formed header fields and a common chunking of body data.

SCTP is undeployable on the Internet. HTTP/2 is deployed on the Internet. It brings multiplexing and of course that brings some additional complexity and constraints. But these have proven more deployable than HTTP/1.x-style solutions such a pipe lining.

HTTP/3 is actually a simpler application mapping than HTTP/2. This is due to the complexity getting pulled into the transport. But it's a zero sum game, just moving the concerns around doesn't mean they are automatically fixed.

dfeojm-zlib• 7 years ago

Yeap. Emperor's new clothes / Not Invented Here / reinventing the wheel / new is "better" than old - churn / kitchen-sink design-by-committee (cough OpenSSL cough). This happens more and more. Instead of fixing existing infrastructure that has a well-established/proven history, throwing everything away and starting over like what came before never existed. And let's over-engineer it and add every possible feature that we'll never use! It's so new and broken, isn't it awesome!?!

rumanator• 7 years ago

Instead of fixing existing infrastructure that has a well-established/proven history, throwing everything away and starting over like what came before never existed.

Isn't HTTP/2 just that, an attempt to fix existing infrastructure? I mean, HTTP/2 was a revision of HTTP/1.* that aimed to fix some problems such as the inability to address latency issues with basic techniques such as pipelining and multiplexing requests.

manigandham• 7 years ago

Of course a new protocol might have new bugs, but what's your alternative? The web needed to evolve. HTTP2 offers a lot of new benefits.

Maybe if the TCP standards could actually update in time frames other than decades then that's how it would've went. And if did get TCP2, you could say exactly the same thing about it being new and more open to bugs.

mjevans• 7 years ago

Is there a better list of fixed versions for E.G. Apache / Lighttpd (n/a No http/2 support) / Nginx?

m11r• 7 years ago

For nginx, versions 1.16.1+ and 1.17.3+ from upstream fix at least three of the vulnerabilities (CVE-2019-9511, CVE-2019-9513, CVE-2019-9516), however if you use a version provided by a distribution’s repositories (e.g., nginx 1.16 included with Ubuntu 18.04 LTS) you’ll need to watch for those security advisories and fixes separately which may have different version numbers due to the backporting.

rdli• 7 years ago

This is addressed in Envoy 1.11.1 (https://www.envoyproxy.io/docs/envoy/v1.11.1/intro/version_h...) and Ambassador 0.75 (https://blog.getambassador.io/multiple-http-2-vulnerabilitie...).

iampims• 7 years ago

I somewhat wish there was a way to test if any http2 server is vulnerable to these issues:

* https://godoc.org/golang.org/x/net/http2

* https://www.haproxy.com/blog/haproxy-1-9-has-arrived/

* https://repo1.maven.org/maven2/org/eclipse/jetty/http2/

* etc…

Larger list at https://en.wikipedia.org/wiki/HTTP/2#Server_software

Exuma• 7 years ago

I'm also wondering about golang http2

140am• 7 years ago

https://github.com/golang/go/issues/33631 (this is part of go 1.12.8 and 1.11.13 which got released because of this today)

LinuxBender• 7 years ago

This was the response from Willy for haproxy:

    Yes, just discussed them with Piotr. Not really concerned in practice,
    maybe only the 1st one (1-byte window increments) might have a measurable CPU
    impact but the rest is irrelevant to haproxy but could 
    possibly harm some implementations depending how they are implemented of course.

I use haproxy to provide http/2 and ALPN for my site, so if there is a PoC I will test it.

kodablah2• 7 years ago

At least for the PING flood I think you can just send HTTP/2 PINGs over and over (that's what the Go test case does) and confirm it uses more and more resources as it queues the control messages.

jedisct1• 7 years ago

Of course this affects DoH servers, too.

jrockway• 7 years ago

Envoy appears to have been updated today to 1.11.1 to mitigate some of these issues. I upgraded and have not experienced any problems yet.

netsectoday• 7 years ago

Here is a server vulnerability matrix... pretty much if you are running HTTP/2 you are exposed and your vendor has a patch waiting for you.

https://vuls.cert.org/confluence/pages/viewpage.action?pageI...

Matthias247• 7 years ago

A little bit disappointed they didn't test my implementation (https://github.com/Matthias247/http2dotnet). It's more or less the only feature complete standalone HTTP/2 implementation for .NET - but somehow that ecosystem seams to be too niche so that nobody ever got interested in using it.

Actually I feel like it avoids most if not all of the described issues. async/await in combination with the design principle of always exercising backpressure on the client and having no internal queues goes a long way in this.

dsign• 7 years ago

It's a nice write-up, really, but any HTTP/2 implementation should be tested with a nice packet fuzzer. Indeed, server providers should compete in the square miles of the datacenter they use to run the fuzzer. Also, the best servers should come with several defense perimeters, including one with geo-ip-directed tactic missiles. Nothing less will do.

kodablah2• 7 years ago

For some of the DOS things, I'm not sure fuzzers would be that effective. For the flooding issues, it's about quantity and frequency instead of how the packet is crafted.

mholt• 7 years ago

Caddy is patched. v1.0.2. https://github.com/caddyserver/caddy/releases/tag/v1.0.2

cryptonector2• 7 years ago

Flow control in application protocols over TCP has been tried (in SSHv2), and it's failed. In SSHv2 flow control acts as a handbrake on all channels -- not good, though it does fix the starving of non-bulk channels by bulk channels. It's bound to fail in HTTP/2 as well.

ryacko2• 7 years ago

I wish application-level flow control would be paired with a database of internet speeds per subnet. They already track everything else, my connection shouldn’t repeatedly slow because the congestion algorithm doesn’t know my maximum download speed.

cryptonector2• 7 years ago

Not subnet but link. Anyways, that can't quite work because even if you could make that accurate there's still congestion to worry about. What you want is something like explicit congestion control with IP options for reporting max path bandwidth or whatever, but that too is susceptible to spoofing and DoSing.

HTTP/2 Denial of Service Advisory

Related Stories