We found a bug in the hyper HTTP library

https://blog.cloudflare.com/hyper-bug/

by Pop_- • 28 days ago

163 76 28 days ago

blog.cloudflare.com

edelbitter2• 24 days ago

Cloudflare does not notice (until a customer complains) that they are sending broken responses at scale? I would have thought they would notice this from sampling and linting a few replies.. just in case they did something like Cloudbleed again.

ramon156• 24 days ago

Can you get reasonable results without exposing sensitive info? I'm asking because I genuinely have no idea what it's like at their scale

100ms• 24 days ago

The failure was caused by a timing-dependent race condition in hyper’s HTTP/1 connection handling. When the reader was slower and the socket buffer filled, poll_flush returned Poll::Pending, but the dispatch loop discarded that result. Hyper then treated the response as complete and shut down the socket while data remained buffered internally, causing the client to receive an EOF before the full body arrived.

https://github.com/hyperium/hyper/issues/4022

Saved you 3000 words

michalc• 24 days ago

Reminds me of another “slow client”-related bug in gunicorn: https://github.com/benoitc/gunicorn/issues/3334

microgpt9• 24 days ago

That's not even a bug. That's how TCP works. If you keep sending data to a socket the other side has closed, you get RST.

moralestapia• 23 days ago

Hey, you have to justify three engineers full time's worth of salary.

algoth1• 24 days ago

I wonder if this bug was found via project glasswing

re-thc2• 24 days ago

I wonder if this bug was found via project glasswing

Did you read how they said it took weeks? Would run out of tokens at that rate...

worldsavior• 24 days ago

We spent six weeks chasing a nearly invisible bug — a race condition that occurred only under specific conditions — in the hyper library that impacted how the Images binding returned processed image data back to the client. In the end, it took four lines of code to fix it.

That's a long time, must be frustrating.

gmueckl• 23 days ago

It is a long time and it gets frustrating when there is significant time where there is flailing with no visible progress.

I have had long bug hunts (~a month each) and witnessed ones that took much, much longer. But the longest one I witnessed was drawn out because reproduction was initially unreliable and could take weeks to months. Thankfully, reproduction was by letting a box sit in a corner while tje people involved moved on to other tasks. This kept everybody sane.

Atotalnoob• 23 days ago

Sometimes the best option is to enhance observability and see when it happens again

microgpt9• 24 days ago

Would using Rust have prevented this?

Cthulhu_• 24 days ago

The Hyper library in question is a Rust library.

Did you read the article, or are you a "use rust" parrot / bot based on titles?

waysa• 24 days ago

Sarcasm. (I guess)

re-thc2• 24 days ago

Isn't this already Rust?

pjmlp4• 24 days ago

That was obviously a joke question, pointing that Rust isn't the solution for everything.

lelanthran• 23 days ago

Woosh :-)

geodel• 23 days ago

Agree. This is warning to people who thought Rust is optional at cloud scale.

Ygg23• 23 days ago

No. Anyone expecting that hasn't read No Silver Bullet essay.

tialaramex4• 23 days ago

Actually I suspect that Rust is a Silver Bullet in that sense. That essay seems to be a case where people know of the essay but haven't read it. Normally in English a "Silver Bullet" is something much bigger, a panacea or cure all which entirely solves a problem but in his essay Brooks is talking about order-of-magnitude improvements, and that looks a lot like Rust.

Brooks was expecting such "Silver Bullet" improvements as often as every few decades, we're arguably overdue significantly. He cites Ada as an example of where such an improvement might come from, well, Rust isn't Ada but a lot of the same ideas about correctness are present.

Google reports order of magnitude changes from their Rust work for example.

testdelacc1• 23 days ago

I get that it’s fun to dunk on Rust when a Rust bug surfaces. But is it a bit petty to bring this out when there’s any type of bug of any severity in any Rust software?

In this case a small minority of requests were getting truncated responses.

No one said Rust software is bug free. If someone thinks that they’ve been seriously misled.

Thaxll• 24 days ago

So much for Rust forcing you to handle errors.

atoav2• 24 days ago

You could say the exact same thing about safety belts and airbags in cars after someone has died in a crash.

Why even bother with measures that prevent many problems if they won't prevent all of them, right?

chlorion• 23 days ago

This is the argument I like too.

It's the same argument anti-vaxers love to make. "Well you can still get covid after getting the shot", which is something I read and heard quite a lot. That doesn't make the thing useless.

Humans are really dumb.

wongarsu• 24 days ago

You could argue the bug happened exactly because hyper's poll_flush treats flushing some but not all data as a successful return, not an error case.

Matl• 23 days ago

Go does force you too, but it also supports _ as a bypass - because sometimes you do know better. Just not in this case.

Rust never promised it'll let programmers turn off their brain, that's what LLMs are for.

jerf• 23 days ago

There's a hidden equivocation there. "Handling" errors, as far as the language is concerned, mean you do something with them, but explicitly discarding them is most definitely a "something".

From a human perspective we can consider that not handling the error.

But the language has no mechanism for "knowing" that discarding the error is wrong. Discarding errors is a fully valid mechanism that we must be able to do in a program because it is sometimes correct. There really isn't even a sensible way to define a way to "force" a user to "handle" errors. The language can only be designed to make it hard to forget to "handle" them somehow in the way the language sees, but it is always possible for the user to incorrectly handle them, of which discarding them when they shouldn't have is only one particularly cognitively-available option but is hardly the full scope of possibilities. Probably isn't even the most common mistake to make, I would imagine there are far more errors that are not handled "correctly" than ones that are spuriously discarded.

Note I keep saying "language" rather than Rust. All a language can do is surface the issue, and Rust does that. It can't force good code. No language can.

Twey8• 24 days ago

This would have been flagged by Clippy lints `let_underscore_untyped` or `let_underscore_must_use`, which sadly are not enabled by default.

pwdisswordfishq3• 24 days ago

Ehh, easy fix

    #[allow(clippy::let_underscore_untyped,clippy::let_underscore_must_use)]
    let _ = self.poll_flush(cx)?;

nesarkvechnep• 23 days ago

Yeah, but you must know about them and the possible bug first in order to allow them...

lunar_mycroft• 23 days ago

You can set the lints to `forbid` instead of `deny`, which means they can't be `allowed` like that.

Twey8• 23 days ago

I said ‘flagged’, not ‘fixed’ :)

You can always write the wrong code if you want it enough. But hopefully a warning would have prompted someone to think harder about this flow.

PoignardAzur• 23 days ago

And this is why you should warn on `clippy::allow_attributes_without_reason` in your projects.

microgpt9• 23 days ago

Or just by not writing let _ =

Twey8• 23 days ago

All recurrent people problems are system problems.

pseudony2• 24 days ago

So “fearless concurrency” still only happens when one just decides to not be afraid… :)

c0balt• 24 days ago

This does not appear to be a concurrency bug though?

pseudony2• 23 days ago

“ a race condition that occurred only under specific conditions — in the hyper library”

microgpt9• 23 days ago

Of course it's a concurrency bug. It races sending data to the kernel against the kernel sending data to the network. If the wrong one wins the bug occurs.

nopurpose• 23 days ago

Nice writeup, but I don't understand how `curl` didn't trigger bug for them (or any other hyper HTTP server out there), given the explanation in the article.

`curl --http1.1` sends `Connection: Close` so sender (hyper) must attempt to shutdown connection after sending whole body. Surely any network is slower than memory copy into socket kernel buffers, so it must reliably trigger condition "buffer flush can't be done in one go" and thus trigger early TCP shutdown.

xacky• 23 days ago

Yet Cloudflare relies on bugs in browsers to "verify" you.

We found a bug in the hyper HTTP library

Related Stories