Quantifying the Impact of “Cloudbleed”

https://blog.cloudflare.com/quantifying-the-impact-of-cloudbleed/

by eastdakota • 9 years ago

287 128 9 years ago

blog.cloudflare.com

advisedwang• 9 years ago

Disappointing that their response is "damage control" and downplaying impact.

AwesomeBean• 9 years ago

Why? As long as they come along with facts and data, then that's a good thing.

79d697i6fdif13• 9 years ago

they only sampled a few thousand leaked responses out of over a million. The margin of error is 2.5% on conclusions because they didn't use enough data, not even close.

geforce2• 9 years ago

Was talking with colleagues about cloudbleed and S3 problems yesterday.

I don't feel like many people are actually concerned of the implication of having an internet that isn't an internet anymore, but merely a handful of big companies hosting everyone.

Or maybe it's me who don't understand.

Florin_Andrei2• 9 years ago

It's almost as if self-reinforcing feedback loops are all over society these days. Being at the top makes it that much easier to stay at the top and crush everything else. Internet companies are just one small slice of this trend.

79d697i6fdif13• 9 years ago

antitrust is all bark with no bite these days. Most industries having 3-4 "competitors" that don't actually compete is the new norm. I wouldn't be amazed if a link is shown between this situation and the increasing wealth inequality

stickfigure• 9 years ago

What are you proposing? We should all serve off of our DSL lines?

ezequiel-garzon• 9 years ago

Collocation, dedicated servers, even VPS. There's plenty of a middle ground between serving off your home server and AWS, Cloudflare et al.

RodericDay• 9 years ago

that would be cool

quasse• 9 years ago

I wish. Too bad there isn't any meaningful competition in the ISP space for most people.

geforce2• 9 years ago

Ah! Maybe that would encourage people not to make 15mb pages doing dozens of requests!

theseatoms• 9 years ago

http://urbit.org/

mseebach• 9 years ago

20 years ago, sites got "slashdotted" left, right and centre. Cloudflare and friends put an end to that, forever. Sites went offline for days if not weeks because of hardware failure -- some never came back because they didn't back up correctly. S3 went offline for five hours, but didn't lose a single bit of data.

You're looking at the past through rose-tinted glasses. We learn all the time, and we will probably also learn to build some resilience into our systems against these issues. But as someone who's been through a thing or two (see above) on the Internet of yesterday, I like the one we have today.

romaniv• 9 years ago

Some website being Slashdotted is not in the same class of issues as users' (possibly encrypted) page content being spread all over the web. Also, the notion that you either need to use some super-centralized platform or be susceptible to Slashdot effect or loosing backups is a false dichotomy.

staunch3• 9 years ago

Cloudflare had nothing to do with ending slashdotting. This stopped being a problem years before they existed.

Slashdotting was mostly a problem caused by Apache's incredibly inefficient design. It consumed huge amounts of memory per connection at a time when most of us had very slow connections. A link from Slashdot was, in effect, a Slowloris attack on your server.

The big change was moving from a fork/thread-based webserver (Apache) to an event-based webserver (nginx), which was made even more efficient by kernel features like epoll.

tedunangst4• 9 years ago

Wait. How does Clouldflare prevent a site from going offline for weeks because of hardware failure?

enraged_camel• 9 years ago

>Sites went offline for days if not weeks because of hardware failure -- some never came back because they didn't back up correctly. S3 went offline for five hours, but didn't lose a single bit of data.

S3 going offline for five hours probably caused orders of magnitude more damage. The reason is simple: the stakes are higher today than they were 20 years ago. Back then the overwhelming majority of the business in most companies was conducted on paper and in person. Losing a website for a few days, or even a few weeks, wasn't a big deal. Today though? There are so many companies that host business-critical operations and infrastructure in the cloud that it's hard to fathom how they are coping with being taken offline for several hours in the middle of the week.

thehardsphere3• 9 years ago

I'm sorry, but I disagree that we have an internet that isn't an internet.

Even though two particular problems affected a great many sites, they didn't affect every site or even a majority of sites. That's because we don't actually have a "handful of companies hosting everyone."

Maybe I'm not a typical user, but the only site that was affected by S3 that I use was HN. The only three sites affected by Cloudbleed that I use were HN, Discord, and another smaller site elsewhere. The rest of the internet worked perfectly fine for me.

I'm pretty sure 90% of the sites typical users were unaffected also. I mean, facebook didn’t go down, did it? People could still search on The Google. I don't know of anyone whose job was affected by this aside from people who run websites. Neither of these were as bad as the Dyn outage, which wasn't the apocalypse either.

I agree that over-centralization of the Internet to be concerned about actually creating, and it's definitely something you should consider before choosing services like Cloudfare and S3. But I don't think it's so bad today that the Internet isn't the Internet anymore.

olalonde• 9 years ago

I don't feel like many people are actually concerned

Judging by how frequently I read this sort of comment on HN, there are definitely many people who are concerned here.

cortesoft• 9 years ago

Man, back in my day the internet was even LESS of an internet than it is now. Very few companies had websites, and very few hobbyists could afford to run a website (there were few hosting companies, and it cost $100 to register a domain name with the only registrar around, Network Solutions). Running an internet presence was EXPENSIVE.

Sure, there has been consolidation in the industry, but it is much easier to exist on the internet now than ever before.

79d697i6fdif13• 9 years ago

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

Wow, so just as bad as we thought.

We did not find any passwords, credit cards, health records, social security numbers, or customer encryption keys in the sample set.

BUT WAIT, THERE'S MORE

The sample included thousands of pages and was statistically significant to a confidence level of 99% with a margin of error of 2.5%.

Oh, so it could actually be as a high as 2.5% leaking encryption credentials. And if none of the data was found to leak anything sensitive where the fuck is the dataset? I've been around way too long to take a "study" like this at face value without third party verification.

I also enjoy the straight up lie at the end:

We are continuing to work with third party caches to expunge leaked data and will not let up until every bit has been removed.

That sounds great right? Well, its too bad that a lot of 'third parties' are a box sitting on the corporate network edge that hasn't been touched in 5 years. Deleting all of this data from third party caches is not physically possible. In fact it might actually make things worse because it's destroying evidence of which credentials were leaked.

dahdum3• 9 years ago

1.2 million page views over 5 months is almost nothing for the amount of traffic going through Cloudflare.

79d697i6fdif13• 9 years ago

And 1.2 million people dying is nothing compared to the number of people on the earth.

IMO a leak this bad should be enough to sink cloudflare. A provider of SSL was randomly spitting out private data onto public websites. OVER A MILLION TIMES. Entire CA's have been shut down for leaking a couple hundred certificates. This has leaked private data over a million times, cloudflare is a joke

mikeash• 9 years ago

It seems to me that the absolute number is what's relevant, not how it compares to the total amount of traffic. That's 1.2 million potential data leaks. That's it "out of a bazillion" doesn't change it.

Artemis2• 9 years ago

For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot.

0: https://youtu.be/LA-gNoxSLCE?t=2m33s

ProAm2• 9 years ago

If you are this concerned, change your credentials.

79d697i6fdif13• 9 years ago

So every website that uses cloudflare should ask their users to change all of their passwords, credit card numbers, and SSN's?

This leak is being downplayed by webmasters because it's so incredibly bad that there's no way of handling it. The credentials of practically any internet user could have been leaked. The only "safe" way to handle this is to give everyone in the US new credit cards and SSN's and to reset accounts and security questions for every user on a site with cloudflare

zulln• 9 years ago

Password would not help if session cookie is leaked. In many instances you cannot do anything towards that, as many services do not have any "logout all"-feature.

bifrost2• 9 years ago

1.3m pages served with "Extra" data is miniscule considering the number of actual pages served. Ideally you'd have no pages served but when you've got a bugproof technology the world will be your oyster.

dsl• 9 years ago

We are continuing to work with third party caches

One of the caches they worked with was Baidu, which has direct ties to Chinese intelligence. Just because it isn't publicly available doesn't mean people aren't still pouring over it looking for useful data.

79d697i6fdif13• 9 years ago

Also, a lot of web spiders are not benign. All sorts of bots troll the internet looking specifically for data leaks like publicly visible email addresses and SSN's. I'm sure they're having a field day.

bifrost2• 9 years ago

I'm glad to see an article that isn't all puppies and kittens, this wasn't a fun thing to deal with and certainly wasn't anticipated. The CloudFlare team did a great job responding to it IMHO.

"Of the 1,242,071 requests that triggered the bug, we estimate more than half came from search engine crawlers."

This is very important to sort out, most people don't think about security much less the storing of credential or identifying data by search engines, this is a huge part of the incident response.

I think that what we should take away from this is that even though the bug existed it was responded to in a reasonable manner.

jlg23• 9 years ago

TL;DR: "Since this is just a sample, it is not correct to conclude that no passwords, credit cards, health records, social security numbers, or customer encryption keys were ever exposed."

koolba3• 9 years ago

3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and

Downplaying how bad it was that customer cookie's were leaked is what I found particularly egregious in the original response. A session cookie is almost as bad as a password leak. Usually it'll let you everything except change your password (as most sites require your current one as an extra validation).

chipperyman573• 9 years ago

Session cookies can be reset much more easily than asking a user to reset their password. Session cookies also aren't used on multiple sites. It's still a big deal, but leaking a session cookie is much less dangerous than leaking a password.

koolba3• 9 years ago

Lots of sites use session cookies. Many don't automatically expire them or rely on browsers to expire them (ex: signed cookies).

Plus rotating the master session key (to force the issue of all users being reset) requires knowing you should do it. By downplaying the issue, CloudFlare is sending the message that customers don't have to do anything.

hueving2• 9 years ago

Session cookies can be reset much more easily than asking a user to reset their password.

This statement is making an assertion about the behavior of vastly different software backends and the skill-set of the people administering the systems. Things that may appear trivial to you may not be for the sysadmin in charge of servers running healthcare applications that have an opaque internal state.

saurik2• 9 years ago

It was my understanding that people were finding numerous examples of leaked data that included Authorization headers, which don't always leak passwords but which still leak session keys. Maybe if espadrine from this linked comment reads this, he could explain more?

https://news.ycombinator.com/item?id=13719455

FWIW, as an example, Grindr uses CloudFlare: do a Google search for "authorization: grindr3" and you will find a URL which (no longer cached but you can still get snippets) contained an authenticated grindr request, which would be enough to have had temporary access to that person's account.

https://costumla.com/wild-west-costumes-for-men.html

"... U Edge Certificate Authority1 0 U San Francisco1 0 U California�1p ��U� N��U � � �f"�0! %��U@ @GET /v3/profiles/[REDACTED] HTTP/1.1 CF-RAY: 33282514b8d957d7 FL-Server: 15f76 Host: grindr.mobi X-Real-IP: [REDACTED] Accept-Encoding: gzip Client-Accept-Encoding: gzip X-Forwarded-Proto: https Connect-Via-Https : on Connect-Via-Port: 443 Connect-Via-IP: 104.16.85.62 Connect-Via-Host: grindr.mobi CF-Visitor: {"scheme":"https"} CF-Host-Origin-IP: [REDACTED] Zone-ID: 22252132 Owner-ID: 2607399 CF-Int-Brand-ID: 100 Zone-Name: grindr.mobi Connection: Keep-Alive X-SSL-Protocol: TLSv1.2 X-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256 X-SSL-Server-Name: grindr.mobi X-SSL-Session-Reused: . SSL-Server-IP: 104.16.85.62 X-SSL-Connection-ID: 15d1b8de6025864d-DFW X-SPDY-Protocol: 3.1 authorization: Grindr3 ... accept: application/json user-agent: grindr3/3.0.13.16790;16790;Free;Android 6.0 CF-Use-OB: 0 Set-Expires-TTL: 14400 CF-Cache-Max-File-Size: 512m Set-SSL-Name: grindr.mobi CF-Cache-Level: byc CF-Unbuffered-Upload: 0 Set-SSL-Client-Cert: 0 Set-Limit-Conn-Cache-Host: 50000 CF-WAN-RG5: 0 CF-Brand-Name: cloudflare CF-Age-Header-Enabled: 0 CF-Respect-Strong-Etag: 0 Set-Proxy-Read-Timeout: 100 Set-Proxy-Send-Timeout: 30 CF-Connecting-IP: [REDACTED] Set-Proxy-Connect-Timeout: 90 Set-Cache-Bypass: 0 Set-SSL-Verify: 0 CF-Force-Miss-TS: 0 Set-Buffering: 0 CF-Pref-OB: 1 Set-Keepalive: 1 CF-Pref-Geoloc: 1 CF-Use-BYC: 0 CF-IPCountry: [REDACTED] CF-IPType ..."

edit: I have spent the last fifteen minutes pulling the search snippet (edit: and now an hour; but all on my phone, so this is harder than it should be, and I also am distracted by other stuff). The way you do this is by walking through the parts you can see to get nearby context. (I do this a lot to pull content purged from or inaccessible to or simply updated in Google's cache.)

In so doing, while I haven't been able to still have access to the session key (likely too long and unique for a snippet), I have pulled the X-Real-IP address field of this Grindr user and a profile identifier they were checking out (both of which I redacted above, but you could trivially get yourself now using that context).

CloudFlare: if you think there isn't private data that was leaked, OR EVEN PRIVATE DATA STILL ACCESSIBLE, you are a bunch of fucking idiots. 1) Clearing the cache isn't sufficient, as for anything built from short plain text words we can pull the snippet. 2) IP addresses and session keys count as "private data". 3) GET requests actually are often sensitive information :/.

PuffinBlue3• 9 years ago

I am the second responder in that thread.

I personally saw an oAuth bearer token for a user of Fitbit. It was there clear as day.

Maybe I saw the only one that leaked, but that seems...unlikely.

dunham• 9 years ago

OAuth2 does indeed send a secret bearer as an "Authorization: Bearer" header, but OAuth1 does not.

The FitBit stuff I recall seeing other day was OAuth1. It would look like oauth_token=XXX and oauth_consumer_key=YYY, along with oauth_signature=ZZZ. If that's what you're seeing, it's not a bearer token and that data isn't secret. Those values are essentially record locators.

OAuth1 has an undisclosed "consumer secret" (embedded in the app) and "token secret" (acquired at login) that are used to sign the request. So these requests don't leak any usable tokens. If you also see oauth_nonce and oauth_timestamp values, then the signature is also protected from a replay attack.

The response body of the request that originally acquired the token/secret would leak these secrets.

I'm sure there are bearer tokens out there for other services, but if fitbit is using OAuth1, you'd have to snag that original token acquisition to call the api as that user.

Edit - I hope this doesn't come off as argumentative. I just want to point out the good design choices made by the guys that weren't actually compromised by this. OAuth2 is really easy to implement, but it kinda feels like a step backwards security-wise.

eridius6• 9 years ago

I don't see anywhere in the post where they claim private data wasn't leaked. In fact, they explicitly state that authorization cookies were leaked. What they said they didn't find was passwords, CC info, health records, SSNs, and customer encryption keys.

They even have a paragraph talking specifically about cookies in GET requests:

This is not to downplay the seriousness of the bug. For instance, depending on how a Cloudflare customer’s systems are implemented, cookie data, which would be present in GET requests, could be used to impersonate another user’s session. We’ve seen approximately 150 Cloudflare customers’ data in the more than 80,000 cached pages we’ve purged from search engine caches. When data for a customer is present, we’ve reached out to the customer proactively to share the data that we’ve discovered and help them work to mitigate any impact. Generally, if customer data was exposed, invalidating session cookies and rolling any internal authorization tokens is the best advice to mitigate the largest potential risk based on our investigation so far.

tedunangst4• 9 years ago

It's still weird to enumerate things not found, but no mention of examples like dating site messages that were found. They're obscuring that by focusing on internal headers, etc. Why does the table with "0 health records" not have an entry for "X private correspondence"?

79d697i6fdif13• 9 years ago

Cloudflare mentioned they only went through about 2000 leaked responses out of over a million. That combined with random people on hackernews STILL finding private data after the purge points towards this leak being a lot worse than they're letting on.

Thankfully CloudFlare spent a week cleaning up the leak in search engines and caches before publicly announcing the issue, so a lot of the evidence is gone.

rattray• 9 years ago

tl;dr excerpted:

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched;

2) the vast majority of Cloudflare customers had no data leaked;

3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records;

and 4) our review is ongoing.

tyingq2• 9 years ago

we now estimate based on our logs the bug was triggered 1,242,071 times

Based on looking at 1% of their log entries for 10 days of the total ~150 day period.

ryanlol• 9 years ago

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times.

So uh, Cloudflare has logs of all page loads since september at the very least? And I guess with response sizes, since that seems like the most reasonable way for them to come up with such a number.

Awesome.

user59944614• 9 years ago

Doubt that they have any log, they have explained before that they only keep 4hours of logs because it's too much volume of data. FYI: Traffic is over 4M requests per second.

Guess they only kept a counter of requests. A bit of maths and you get an estimate for the amount of leaks.

tptacek• 9 years ago

Not everything Cloudflare has written about this breach has been good, but this is solid. In particular, I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug.

What remains now is an accounting of how some of the most sensitive C code on the Internet was tested prior to the discovery of this bug (by Cloudflare's own accounting, the underlying error seems to have been present long before it became symptomatic), and what they're doing about that now.

hueving2• 9 years ago

I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug.

How would you classify people using all of the crawled data though? It seems sort of hand-wavy to claim nobody deliberately exploited the bug if they used leaked session tokens in crawled data to get access to user accounts.

camel_gopher• 9 years ago

The wording of the post seems to be tilted towards limiting Cloudflare's potential liability as opposed to objectively quantifying the impact. For example, the author mentions that the 100th largest website handles less than 10B requests per month - which positions near the median of the table of likelihood anticipated leaks.

I'm not surprised at this approach as the CEO has a background in law. I think that, combined with his hubris, shaped the tone of this post from an objective post mortem to a goal of minimizing the damage to Cloudflare itself.

I met Mr. Prince once at a tech conference. I mentioned that I followed him on Twitter, to which he freaked out a little bit and said "Oh I hate that when I run across people who tell me that". When I heard him tell his name to someone else there, he said his last name was "Prince, like King". I suspect that his attitude is prevalent at Cloudflare, and may have shaped not only the source of this issue, but the response to it as well, since CEO's generally influence the culture.

bthallplz• 9 years ago

his attitude

I don't quite follow from your description of his reaction. What sort of attitude were you attempting to describe/how'd you perceive his attitude to be?

fredsir• 9 years ago

I think his attitude, as described, was that he manipulates for his own perceived self importantness and perceived gain. If you mention you follow him on twitter the response is "oh that's so beneath me and I hate it", if you don't know him, he is "prince, like a king, and an important one of that so remember me" kind of attitude. Like a person you wouldn't want to meet in a bar.

That is how I understood it anyways.

tvst• 9 years ago

Who would say "Prince, like King" when you can say "Prince, like 'the artist formerly known as'"?

remus• 9 years ago

It seems to me like the point of this post was to clarify whether the vulnerability was being actively exploited prior to the announcement, and I think it does that well.

Just think of how easy it would have been to exploit this bug: buy a load of random domains, host some plausible looking but malformed content on it then setup a few thousand bots to hit those sites and harvest the responses. Keep that plugging away for a few months and you'd easily collect a lot of supposed-to-be-private info.

That would have been an order of magnitude worse than having to sift through caches for scraps. Not to say that what's happened is 'good', but it could have been a whole lot worse.

79d697i6fdif13• 9 years ago

Tagging along on the top reply here, but does anyone else notice some serious HN gaming going on in this thread?

Accounts commenting that have made 10 comments in 5 years, people defending CF that, after going through their profiles, are clearly linked to CloudFlare or maybe even employees. Top relies suddenly being near the bottom and replaced by posts supporting CloudFlare with statements that don't match what was actually in the blog post.

Just seems really really fishy to me

komali2• 9 years ago

I doubt anything more nefarious is happening than a flood of CF employees hitting the thread. Is that gaming?

JumpCrisscross• 9 years ago

It's, at the very least, disingenuous.

SilasX• 9 years ago

It's not gaming for a CF employee to offer the own insights, but it would be if they're doing it as part of a coordinated effort. Also, they should be upfront about the affiliation.

dahdum3• 9 years ago

I'm defending them because they're an incredible value for the $200/month we pay them, they've handled it responsibly, and the real impact looks vanishingly small.

uladzislau• 9 years ago

HN is very clearly gamed and I see more and more of this, not only in this thread.

dang2• 9 years ago

We take this problem seriously and have years of experience with it. In our experience, "very clearly gamed" nearly always reduces to "seems to me, based on personal biases and incomplete information". When abuse really is going on, that's extremely important, but it is certainly fewer than 10%, probably fewer than 5%, and maybe fewer than 1% of the times we hear this. So we actually have two problems, unfortunately intertwined: (1) abuse and (2) this tendency to imagine it.

For this reason, please don't post unsubstantive allegations to HN. If you have evidence, or significant suspicious, email them to hn@ycombinator.com instead.

prdonahue• 9 years ago

This is rich coming from an anonymous account created 69 days ago with 17 total comments (and 7 of them in this post).

79d697i6fdif13• 9 years ago

Hey at least mine is an obvious throwaway. And I don't think 17 comments in two months is that bad compared to ten in 5 years

dang2• 9 years ago

Tagging along on the top reply here

First, no, please don't do that. It's one of the worst patterns in active discussions.

Second, if anyone thinks they see gaming or abuse, they should let us know right away at hn@ycombinator.com so we can investigate. Please don't post about it in comments, for a couple reasons: (1) if there really is abuse going on, we need to know, and we don't see most comments; (2) most of us internet readers are orders of magnitude too quick to interpret our own cognitive biases as abuse by others (e.g. X seems obvious to me so anyone arguing ~X must be a shill). This places the threshold for useless, nasty arguments ('you're a shill. no, you're the shill') dangerously low. Combine that with the evil catnip power of all things meta and you get the most malignant strain of offtopicness there is, so we all have to be careful with it. And yes, genuine abuse does also exist. It's complicated that way.

We detached this subthread from https://news.ycombinator.com/item?id=13767122 and marked it off-topic.

79d697i6fdif13• 9 years ago

:( My bad.

HalfwayToDice• 9 years ago

My website gets 75,000,000 total requests per month through CloudFlare, so they estimate 6-11 leaks.

I am happy with the way CloudFlare have responded to this.

tyingq2• 9 years ago

These exact looking numbers are based on extrapolating a look at 1% of all requests, for the sites they think were leaking, for a 10 day period.

They don't get into the full detail of how they knew which sites to look for. For example, if I created a site with these settings, used it for a week, and then deleted it...is my site counted? What if I changed the settings back to normal? Does it slide under their radar? Did they really look for every site that had those settings at any time, or just any site that had those settings at the specific time they looked?

To me, it still feels like there's a window where a bad actor could have created a site that triggers this behavior, and then pumped it with a scraper, over and over.

caf• 9 years ago

Yeah, exactly - this "look at a 10 day period and extrapolate" idea is based on the assumption that the rate of hits to the triggering pages was constant over the whole time, but in the case of looking for a malicious actor that's essentially begging the question. If there was unnatural traffic in there you can't assume its rate would have stayed constant over the whole period.

matt2000• 9 years ago

But also, we gotta start writing stuff like this parser in safer languages where this class of bug simply can't happen, right? Every time one of these breaches occurs, it's basically the same thing. "Data wasn't what we expected, we read a bunch of extra junk that had nothing to do with the input." It's kind of crazy it's acceptable at all.

At this point, these kind of problems are on us as a community that we keep using unsafe tools. Every time we choose one of these languages we are implicitly trading security for performance (a.k.a. money).

tedivm• 9 years ago

If things aren't written in a "safe language" then they should at least be tested. Valgrind should have picked this issue up.

nikisweeting2• 9 years ago

The lesson I really hope Cloudflare learns from this is to do thorough fuzz testing before deploying anything (and to write their parsers in a memory-safe language like Rust).

mixedbit2• 9 years ago

The problem is that CloudFlare encourages a fundamentally insecure architecture. CDN should be used to host static files on a separate, cookie-less subdomain. With such architecture consequences of any data leak from a shared CDN infrastructure are limited.

With CloudFlare all the requests are passed through the CDN. With dynamic, authenticated requests the benefits of this are dubious. The responses can't be cached anyway, an additional HTTP(S)-level hop is required for each request (and many additional IP-level hops). The drawbacks are now obvious: sensitive, unencrypted data of thousands of different sites resides in the same, not isolated, shared process memory. With such architecture we will always be one C pointer manipulation bug away from another leak.

problems2• 9 years ago

The reason CloudFlare encourages this is because it isn't meant for use as a traditional CDN - they're not a CDN at all really. You don't upload content to them. They're meant as a cache and anti-DDoS proxy more than anything else.

Look at what happens when anyone gets hit with a large scale DDoS attack these days, the first thing people will tell them, myself included is "CloudFlare". How do you handle a massive spike in load on a tiny, usually empty server? CloudFlare is the go-to tool for this purpose, especially for people who can't afford to pay Akamai, BlackLotus, Amazon or others. And they're impressively cheap for small businesses too. You can't hide from a DDoS attack or save your tiny DigitalOcean box if your web server isn't hidden away behind a service like this.

I have nothing against CloudFlare, they've proven themselves extremely trustworthy in many ways and even in handling of past security issues they've been very responsible. But this time that's not the case - the downplaying of the issue is really damaging their reputation in my eyes. I'm a paying customer and I'm quite disappointed to see this.

mixedbit2• 9 years ago

They're meant as a cache and anti-DDoS proxy more than anything else

You can't cache responses to authenticated requests, so in case of DDoS attack against an HTTP end point that requires authentication the best CloudFlare can do to safe the back-end is to drop requests, which makes the attack successful. I really can't see benefits of passing authenticated traffic through CloudFlare.

lima• 9 years ago

Just consider all the private crawlers whose caches weren't purged.

Anything short of invalidating any and all tokens, cookies and sessions is irresponsible and CloudFlare should communicate is as such.

markonen2• 9 years ago

One aspect I haven't seen mentioned yet is that there is a geographic dimension to what ends up in search engine caches; their crawlers end up talking to the nearest Cloudflare point-of-presence so any leaked data is also going to be from requests served in that same general area.

For example, my Cloudflare-using site is in the 1B to 10B requests a month bracket, meaning 112–1,118 anticipated leaks per Cloudflare's calculations. It's enough that you'd expect to possibly find some in the search engines, but we haven't.

One potential explanation is that our usage is heavily concentrated within a geographic area (the Nordics), and this just isn't where the big search engines are doing their web crawling from.

eastdakota• 9 years ago

Yes, that's correct and not something our analysis has taken into account. Search crawlers are more likely to hit certain PoPs. If your content was less likely to be in one of those PoPs then you're less likely to have leaked data to one of them.

We also didn't take into account that the distribution of the ~6,500 sites that triggered the bug weren't even across our infrastructure. We distribute load for any site across some fraction of all the servers in any given PoP. Based on whether you're on a cluster of servers with more or fewer vulnerable sites you'd have been more or less likely to leak data.

For the general case the stats are directionally correct and, if anything, conservative (we believe, if anything, overstate the risk). But a particular customer's situation could deviate from the expected probabilities for a number of reasons.

nikisweeting2• 9 years ago

I've linked to this post from the top of the pirate/sites-using-cloudflare README. I'm glad that it's a refreshingly level-headed response that isn't placing the blame on Google or other caches. It's still down-playing the potential impact slightly, but at least it's a more thorough response than some of the HN comments we've been seeing from CF employees.

My advice to friends and family currently is to use their "log out all active sessions" buttons, and to reset only crucial passwords. As Techcrunch mentioned in their coverage, many sites will likely pay for identity theft insurance to cover the change of a leak, rather than force password resets and lose their users trust. It's a risk analysis that each site needs to make individually, and it may be the right call to eat the losses of ~10 possibly compromised users instead of forcing millions to reset their passwords.

_kst_• 9 years ago

A theoretical question.

There seems to be some question about whether any passwords were actually leaked, but in principle some password data could have gotten out.

What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)?

I have accounts on a number of sites that were affected by Cloudbleed, as indicated by http://www.doesitusecloudflare.com/, and I've already changed my passwords on those sites. But the passwords I use are randomly generated and should be practically unguessable given a hashed version. Would I have been at risk if I had left my passwords alone?

PuffinBlue3• 9 years ago

I forced user password resets on a forum I run. It is powered by myBB.

This is towards the edge of my knowledge circle but...

It doesn't appear that any hashing takes place client side[0] with myBB before the password is transmitted to the server, but I may have missed it when rechecking the code.

Presumably that would mean in my case users who logged in potentially had their cleartext passwords pass through Cloudflare's network when they submitted their login details.

Despite being behind https, when https is terminated at Cloudflare's edge and re-encrypted to my origin server (I use full/strict crypto setting), there was the possibility for decrypted data to be held in memory and then the possibility for it to be injected into the html pages of other sites when the bug was triggered.

It would be a very very small probability of actually happening, but not zero.

No details from the server should have leaked because only the content of transmitted data that passed over Cloudflare could have been leaked and the comparison of the provided password to the hashed/salted one in the database would have occurred on the server and not passed over the network.

So the answer is it would depend on the way the sites you use handled password transmission and comparison. Personally I'd change them, and actually have done so for sites I use.

[0] https://community.mybb.com/archive/index.php?thread-88668.ht...

orthecreedence• 9 years ago

What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)?

Depends on the app. The apps I write will do a PBKDF2 w/ 20K rounds on a password before sending to the server, which then hashes that hash again before storing/comparing. I suspect most apps just sent the username/password plaintext over SSL, meaning the plaintext passwords would have been leaked if the app used Cloudflare to terminate SSL.

That said, if someone got the PBKDF2ed password hash from the SSL payload, they could just as easily use that to log into an account, but they'd have to at least do a little bit of work to break the hash if they wanted to actually crack the password.

In other words, you were absolutely right to change all your passwords.

robocat• 9 years ago

Passwords are in HTTPS requests, and should never be in HTTPS responses (especially not cacheable responses).

Only GET responses should be cached (not requests), so it seems unlikely a password would leak.

gwu78• 9 years ago

If users were discerning based on the inner workings of this organization, they should be switching their nameservers away from Cloudflare. Quantifying the number of domainnames CF has in their zones before and after this reported incident would be interesting. Sadly, I doubt there will be much change.

pron• 9 years ago

Their messaging is a bit strange. They write:

For the last twelve days we've been reviewing our logs to see if there's any evidence to indicate that a hacker was exploiting the bug before it was patched. We’ve found nothing so far to indicate that was the case.

right after they explain:

Cloudbleed is different. It's more akin to learning that a stranger may have listened in on two employees at your company talking over lunch. ... you can't know exactly what the stranger may have heard, including potentially sensitive information about your company.

So they say they've found no evidence after explaining that the situation is analogous to a case where no evidence can be found even if information were stolen.

Karupan2• 9 years ago

I'm not going to comment on the methodology of CloudFlare's analysis, but just the fact that they are logging millions of unencrypted HTTP requests (cookies, headers and all) is just scary! Wasn't the point of SSL/TLS that intermediate proxies cannot peek into requests?

jgrahamc• 9 years ago

We're not logging "cookies, headers and all".

Karupan2• 9 years ago

My bad! Just re-read the article and it did say information about requests and responses. Thanks for the clarification.

Quantifying the Impact of “Cloudbleed”

Related Stories