Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

by tptacek • 9 years ago

3238 986 9 years ago

bugs.chromium.org

jgrahamc16• 9 years ago

Full details from Cloudflare: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

jlgaddis5• 9 years ago

... and the HN link/discussion (submitted by jgrahamc): https://news.ycombinator.com/item?id=13718720

snaky3• 9 years ago

"enable AMP, and more" is a certainly nicer than ""ScrapeShield" feature which parses and obfuscates html".

loeg2• 9 years ago

Incident report on memory leak caused by Cloudflare parser bug

This title sounds like Cloudflare doesn't know what a memory leak is or are intentionally trying to downplay information disclosure. Neither option is comforting.

justin66• 9 years ago

They know what a memory leak is, so...

jjoe3• 9 years ago

From the blog post: "For the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug."

Is this statement accurate considering Tavis said in his report that: "We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users."

snug2• 9 years ago

Not the TLS Private key, this would pertain to the ClientKeyExchange. The TLS Private Key, should NEVER leave the server. The buffer overruns was only what a client/server exchange would see.

HappyTypist2• 9 years ago

Why is your company severely downplaying it?

Honestly, this is the biggest security incident in a long time, and proper mitigation would probably warrant:

- forcefully terminating all cookies on CloudFlare sites, cloudflare already injects JS onto the page anyway

- MITMing all CloudFlare sites with a warning for users to change their passwords

xuki2• 9 years ago

MITMing all CloudFlare sites with a warning for users to change their passwords

REALLY?

tptacek11• 9 years ago

Oh, my god.

Read the whole event log.

If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results.

The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep everyone at work late today. And, this morning, Google announced the SHA-1 collision, which everyone (including the insiders who leaked that the SHA-1 collision was coming) thought was the big announcement.

Nope. A SHA-1 collision, it turns out, is the minor security news of the day.

This is approximately as bad as it ever gets. A significant number of companies probably need to compose customer notifications; it's, at this point, very difficult to rule out unauthorized disclosure of anything that traversed Cloudflare.

travmatt• 9 years ago

I remember Tavis tweeted Friday night asking for a cloudflare engineer to contact him, and everyone joked that the last thing you want on a Friday evening is an urgent message from tavis ormandy.

Magicstatic• 9 years ago

That was my tweet believe it or not. I had to turn notifications off on my phone because out of nowhere it was getting bombarded with shares/likes...

fictioncircle5• 9 years ago

I would say the crazy thing is a mere t-shirt as their "bug bounty" top tier award given how they've pitched themselves as an extremely secure service.

https://hackerone.com/cloudflare

I'm sorry but when the reward for breaking into you is basically a massive pinata of personal information...that simply is a bad joke. Security flaws are going to happen and if you aren't going to even offer a reasonable financial reward to report them to you, well, that is just begging to be exploited with a pinata that size.

brians2• 9 years ago

Nah. Bug bounties don't work for services like CDNs. Maybe they do elsewhere. But for enterprise services, the noise rate is too high, and the very good bug finders are either salaried, free, or working for the adversary.

vacri• 9 years ago

The reward includes a t-shirt, it isn't a mere t-shirt. You also get "12 months of CloudFlare's Pro or 1 month of Business service on us" (~$200). The reward is also not tiered.

The award may still not be all that much, but let's not make things up about them.

deckiedan• 9 years ago

What would make sense (to me, not a business/marketing guy, nor a lawyer, at all) would be a t-shirt and free subscription as the offered thing, something which costs the company nothing.

Then for anything like this, give publically a bonus gift which makes it worth people reporting to them and not blackmarket selling it. Once it's gone through the legal dept. and so on.

Then they can be very quick with handing out tshirts and so on to any and every microissue report, without the people running triage having to care about amounts or tax or whatever.

Having any kind of publically offered payment for service (beyond a tshirt bounty or services in kind) is just begging for legal issues, right?

tptacek11• 9 years ago

In case you're wondering how this could be worse than Heartbleed:

Yes, apparently the allocation patterns inside Cloudflare mean TLS keys aren't exposed to this vulnerability.

But Heartbleed happened at the TLS layer. To get secrets from Heartbleed, you had to make a particular TLS request that nobody normally makes.

Cloudbleed is a bug in Cloudflare's HTML parser, and the secrets it discloses are mixed in with, apparently, HTTP response data. The modern web is designed to cache HTTP responses aggressively, so whatever secrets Cloudflare revealed could be saved in random caches indefinitely.

You really want to see Cloudflare spend more time discussing how they've quantified the leak here.

_wmd4• 9 years ago

It shouldn't be too difficult to feed an instrumented copy of the parser some fraction of their cached pages (after all, that's what they're for.. right?) and calculate a percentage of how many triggered e.g. valgrind, or just some magic string tacked on the end of the input appearing in the output or similar

I prefer CloudScare to Cloudbleed :)

jgrahamc16• 9 years ago

You really want to see Cloudflare spend more time discussing how they've quantified the leak here.

What would you like to see? The SAFE_CHAR logging allowed us to get data on the rate which is how I got the % of requests figure.

carterehsmith• 9 years ago

What a mess.

On the plus side, all those booter services hiding behind the Cloudflare are probably being probed and classified/identified/disabled by competitors and probably FBI. That is good.

AznHisoka• 9 years ago

Can someone tell me the implications of this in laymen terms?

For instance what does it mean "sprayed into caches"? what cache? dns cache? browser cache? if the latter, does it mean you are safe if the person who owns that cache is an innocent non technical iser?

nicoburns• 9 years ago

Far worse than this. Yes, browser caches, but also web crawlers (like google)'s caches. This means that anyone who requested certain public content could have instead received secret content from completely unrelated websites.

hannob• 9 years ago

There are all kinds of places were things are cached, both on- and offline. Your data may end up in:

* Browser caches.

* Sites like wayback machine or search engines that make copies of webpages and save them.

* Tools that store data downloaded from the web, e.g. RSS readers.

* Caching proxies.

* the list goes on and on.

I think what tptacek wanted to say: It's just so common that people download things from the web and store them without even thinking much about it. And all those places where this happens now potentially can contain sensitive data.

tptacek11• 9 years ago

There are caches all over the Internet; Google and Microsoft run some of them, but so do virtually every Fortune 500 company, most universities, and governments all over the world.

The best way to understand the bug is this: if a particular HTTP response happened to be generated in response to a request, the response would be intermingled with random memory contents from Cloudflare's proxies. If that request/response happened through someone else's HTTP proxy --- for instance, because it was initiated by someone at a big company that routes all its traffic through a Bluecoat appliance --- then that appliance might still have that improperly disclosed memory saved.

tr44564• 9 years ago

As for the SHA-1 collision mentioned by jgrahamc[1] earlier today:

How am I going to explain this to my wife?

Actually a serious question. How do we communicate something like this to the general public?

[1] https://news.ycombinator.com/item?id=13713826

emmelaich6• 9 years ago

It's reminiscent of the earlier days of the Squid cache.

When it had bugs and devivered up cached files the typical symptom was that everyone in the company got unwanted porn.

Because the biggest user (by far) of the 'net was the person into porn and so 90% of the Squid cache was porn.

yojo• 9 years ago

Many services on the internet keep a copy of a page they have loaded in the past. Google does this, for example. It lets them do things like search across websites quickly.

Many of these caches are available online, to anyone who wants to look at them.

This bug meant that any time a page was sent through Cloudflare, the requester might receive the page plus some sensitive personal information, or credentials that could be used to log in to a stranger's account. Some of these credentials might let a bad actor pretend to be a service like Uber or Fitbit.

This very sensitive information might end up saved in a public cache, where anyone could find it and use it to do harm.

espadrine• 9 years ago

It is far from over, too! Google Cache still has loads of sensitive information, a link away!

Look at this, click on the downward arrow, "Cached": https://www.google.com/search?q="CF-Host-Origin-IP:"+"author...

(And then, in Google Cache, "view source", search for "authorization".)

(Various combinations of HTTP headers to search for yield more results.)

kristianp• 9 years ago

Wow, I just tried this, the first result with a google cache copy has a bunch of the kind of data described. Although there was only one result with a cache.

PuffinBlue4• 9 years ago

The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines.

So I tried it too, and there's still data cached there.

Am I misunderstanding something - that above statement must be wrong, surely?

They can't have found everything even in the big search engines if it's still showing up in Google's cache, let alone the infinity other caches around the place.

EDIT: If the cloudflare team sees I see leaked credentials for these domains:

android-cdn-api.fitbit.com

iphone-cdn-client.fitbit.com

api-v2launch.trakt.tv

Retr0spectrum• 9 years ago

https://webcache.googleusercontent.com/search?q=cache:lw4K9G...

    Internal Upstream Server Certificate
    ...
    /C=US/ST=California/L=San Francisco/O=Cloudflare Inc./OU=Cloudflare Services - nginx-cache/CN=Internal Upstream Server Certificate

That really doesn't look good.

tlrobinson3• 9 years ago

The first couple I looked at were requests to Uber and Fitbit...

toyg5• 9 years ago

Lol, Google just purged that search.

EDIT: but there's still plenty of fish: http://webcache.googleusercontent.com/search?q=cache:lw4K9G2...

This will take weeks to clean, and that's just for Google.

EDIT2: found other oauth tokens, lots of fitbit calls... And this just by searching for typical CF internal headers on Google and Bing. There is no way to know what else is out there. What a mess.

mintplant3• 9 years ago

The "well-known chat service" mentioned by Tavis appears to be Discord, for the record.

edit: Uber also seems to be affected.

Edmond• 9 years ago

more results from duckduckgo: https://duckduckgo.com/?q=+%7B%22scheme%22%3A%22http%22%7D+C...

alinspired• 9 years ago

this is quite bad. i hope google can put some effort in clearing it's cache too

djKianoosh• 9 years ago

I searched for

"CF-Host-Origin-IP:" token

.... uhm is that what I think I'm seeing???

chrissnell2• 9 years ago

It seems like the reasonable thing for Google to do is to clear their entire cache. The whole thing. This is the one thing that they could do to be certain that they aren't caching any of this.

ikeboy• 9 years ago

It is a snapshot of the page as it appeared on Feb 21, 2017 20:20:45 GMT

So the issue wasn't fully fixed on Feb 19, or Google's cache date isn't accurate?

scurvy• 9 years ago

Time to find out where various "booter" sites are actually hiding.

patio112• 9 years ago

If anyone here is HIPAA-regulated or you have a customer who is, and you used Cloudflare during those dates, it is Big Red Button time. You've almost certainly got a reportable breach; depending on how tightly you're able to scope it maybe it won't be company-ending.

spydum2• 9 years ago

Isn't it worse than that? Even if you are not a CF user, if your apps make calls to a third party site protected by CF, you could be at risk (stolen credentials, API keys), and could be attacked using those now.

chimeracoder3• 9 years ago

If anyone here is HIPAA-regulated or you have a customer who is

Cloudflare certainly does; I founded a health tech company, and Cloudflare was the recommended go-to for health tech startups who needed a CDN while serving PHI.

And this is definitely a reportable breach. Technically any breach is supposed to be reported to HHS, but in reality, a lot of covered entities (e.g. insurers) fail to report smaller breaches (which, as a patient, should terrify you). The big ones, though, are really, really bad, and when reported, the consequences can be very serious and potentially even include serving time, depending on the circumstances.

The reason I can be so confident that this is a reportable breach is that the definition of PHI is so broad that even revealing the existence of information between two known entities can be considered protected information. Anything more specific, like a phone number or DOB, or time of an appointment (even if you don't know who the appointment corresponds to) - that's always protected. And Cloudflare certainly has many of those.

chasb2• 9 years ago

If you are/were using Cloudflare to cache PHI though their CDN without a BAA, you were likely in breach before this.

Some have suggested that Cloudflare might not be a business associate because of an exception to the definition of business associate known as the "conduit" exception.

Cloudflare is almost certainly not a conduit. HHS's recent guidance on cloud computing takes a very narrow view[0]:

"The conduit exception applies where the only services provided to a covered entity or business associate customer are for transmission of ePHI that do not involve any storage of the information other than on a temporary basis incident to the transmission service."

OCR hasn't clarified what "temporary" means or whether a CDN would qualify, but again, almost certainly not. ISPs qualify, but your data just sits on the CDN indefinitely.

p.s. Hi Patrick and Aditya!

[0] https://www.hhs.gov/hipaa/for-professionals/special-topics/c...

askvictor2• 9 years ago

This is approximately as bad as it ever gets.

*as bad as it has ever gotten so far.

porker• 9 years ago

A significant number of companies probably need to compose customer notifications;

As a one-man company who has never done this before (and to the best of my knowledge never needed to): Any guides/examples to writing a customer notification for security ups like this? Or just recommendations? Thanks.

mochizuki• 9 years ago

It's as easy as throwing a red banner on your website that explains the situation briefly and recommends that users change their passwords, if you take this more seriously you can force a password reset for all users. Depends on how sensitive the information that your users trust your site to hold is.

azna• 9 years ago

Email your customers, telling them to change their passwords, and link to some info about the leak. (in case they don't visit your website and miss seeing the security alert banner)

Advise them to change passwords for other services too, list sites possibly affected: https://github.com/pirate/sites-using-cloudflare/blob/master...

omegaworks• 9 years ago

Tavis found it by accident just looking through Google search results.

Curious whether there could be some automated way of preventing such a widespread cache poisoning in the future. Some ML trained on valid pages from a given domain?

Is it even possible to recover the original content of the documents or was the data randomly inserted into different parts?

CapacitorSet• 9 years ago

Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.

Considering the amount and sensitivity of the data they handle, I'm not sure a t-shirt is an appropriate top-tier reward.

summarity• 9 years ago

Not only that, but the "reward" in the program is laughable and frankly insulting to any serious researcher considering the scope of CF. Bug bounty platforms are already becoming the fiverr of ITSEC (that's not a good thing), CF just made an extra effort do diminish the value for researchers.

Management: "Why do we offer $5k for a small bug again? Look at CF, they don't offer any money!"

eli3• 9 years ago

If serious researchers are looking to get paid, I think bug bounties are the wrong approach entirely

p0rkbelly• 9 years ago

I don't disagree.

But, Taviso is probably contractually prohibited from accepting money from CF as a Google employee. Many large companies have 'outside activity' clauses and Google seems to be paying him already for that.

However, it will affect others whom are fully freelance.

ifdefdebug2• 9 years ago

"Why do we offer $5k for a small bug again? Look at CF, they don't offer any money!"

Answer: "Because if they had set up a bounty of $50k for security issues, they'd had thousands of researchers/students/white hats etc. watching the output of their servers."

eli3• 9 years ago

I never really got this argument. Is it not much better than the majority of companies that have no bug bounty and where the reporter needs to worry they will be met with legal threats instead of a t-shirt?

mdekkers• 9 years ago

I got a t-shirt from cloudflare, and all i did was tell them "please send me a t-shirt" - they shipped it halfway across the world as well, for free! (it didn't fit...)

raverbashing• 9 years ago

Good to know the security of their users is worth a t-shirt

xt00• 9 years ago

Wow apparently they never fuzzed their input and looked at the output. A malformed html input should be about the easiest possible thing to try... yeouch...

bovermyer• 9 years ago

Well, my day tomorrow is going to be busy. So's my evening tonight, I guess.

rdl10• 9 years ago

Neither this thread nor the Cloudflare blog post include concise steps for customers who were exposed.

There's an argument for changing secrets (user passwords, API keys, etc.) for potentially affected sites, plus of course investigating logs for any anomalous activity. It would be nice if there were a guide for affected users, maybe a supplemental blog post.

(and yet again: thank you Google for Project Zero!)

meowface7• 9 years ago

What can they even say? "Change everything" doesn't really work. Any potentially secret data to or from a server could have been exposed.

rdl10• 9 years ago

Yes, but in general keys provide ongoing access; sensitive data itself is more limited in scope. Keys, auth tokens, etc. would be what I'd focus on.

Kalium11• 9 years ago

Right there with you. I'm currently scrambling for remediation ideas. "Change everything" isn't tractable.

garblegarble• 9 years ago

I'm currently scrambling for remediation ideas. "Change everything" isn't tractable.

It's not easy to deal with but it is the best remediation available to you, given the exceptionally broad scope and months-long period where data was apparently leaking (the cloudflare blog post lists 2016-09-22 as the first date when leaks were possible)

nikisweeting21• 9 years ago

I'm scrambling to compile an list of impacted domains and some easy-to-understand instructions that I can send friends and family:

https://github.com/pirate/sites-using-cloudflare

_wmd4• 9 years ago

Step 1) MITM the entire Internet, undermining its SSL infrastructure, build a business around it

Step 2) leak cleartext from said MITM'd connections to the entire Internet

I recently noted that in some ways Cloudflare are probably the only entity to have ever managed to cause more damage to popular cryptography since the 2008 Debian OpenSSL bug (thanks to their "flexible" ""SSL"" """feature"""), but now I'm certain of it.

"Trust us" doesn't fly any more, this simply isn't good enough. Sorry, you lost my vote. Not even once

edit: why the revulsion? This bug would have been caught with valgrind, and by the sounds of it, using nothing more complex than feeding their httpd a random sampling of live inputs for an hour or two

revelation• 9 years ago

Step "What does secure mean anyway") SSL terminate even sites that are not sending data to Cloudflare securely

gkop4• 9 years ago

Yup, this made it crystal clear, years ago, that Cloudflare's business incentives were and are at odds with a secure web.

hoorayimhelping• 9 years ago

edit: why the revulsion

I'd guess it's because of the crude and reductive way you describe the service cloudflare provides. I don't know what type of programming you do, but many small services don't have the infrastructure to mitigate the kind of attacks cloudflare deals with and they wouldn't be around without services like this.

I don't like the internet becoming centralized into a few small places that mitigate DDOS attacks like this, but I like the alternative (being held ransom by anyone with access to a botnet) even less.

I'm going to take a more even handed approach than what you're suggesting. Any time you work with a service like this you risk these kinds of things - it's part of the implicit cost/benefit analysis humans do every day. I'm not ready to throw out the baby with the bathwater because of one issue. I'm not sure what alternative you're suggesting (I didn't see any suggestions, just a lot of ranting, which might also contribute to the 'revulsion') but it doesn't sound any better than what we have.

hueving3• 9 years ago

So rather than demand fixes for the fundamental issues that enable ddos attacks (preventing IP spoofing, allowing infected computers to remain connected, etc), we just continue down this path of massive centralization of services into a few big players that can afford the arms race against bonnets.

Using services like Cloudflare as a 'fix' is wrecking the decentralized principles of the Internet. At that point we might as well just write all apps as Facebook widgets.

gambler3• 9 years ago

I'm not ready to throw out the baby with the bathwater because of one issue.

Extreme centralization of the Internet is not a "baby", except maybe in the sense of a cuckoo's egg.

But I'm willing to bet the mentality of this comment is highly representative of many web developers and service providers. They will not seek to fix anything, because they don't see this state of things as a problem in the first place.

EGreg3• 9 years ago

How about... stop CLOUD THIS and CLOUD THAT.

Cloud means extreme centralization.

It means giving your data to a third party you don't control.

Why?

Why does our networked software have to assume a centralized topology?

In the days when developed countries had dialup, protocols (IRC, Email, etc.) were all decentralized. Today, all the famous developers live with fancy broadband internet connections and forgot what it's like to have to think about netsplits.

The result... all the software is either "online" or broken.

There shouldn't be an "online" or "offline". There should be "do I have access to server X currently?"

Why do we need Google Docs to collaborate on a document if we are all in the same classroom?

Why do we need centralized facebook server farms whose engineers post on highscalability how they enable us all to post petabytes of photos and comment to our friends?

Why do we need centralized sites to comment at all? Each thread is local to its parent.

Why does India need internet.org from facebook?

If communities could have a network that survives without an uplink to the outside world then DDOS from the global internet would just cut off that network's hosting of documents to outsiders. They'd still be able to do EVERYTHING locally - plan dinners, book a local appointment, send an email etc. and even post things out to the greater internet.

This is a future I want to see.

We already have mesh networks. We need more web based software to run these things.

That's what we are building at qbix.com btw.

bigiain• 9 years ago

Step 0) Obtain black funding from NSA budget to start and "VC invest" in a global CDN company...

(Now I'm trawling Crunchbase to see if I can work out which investors are NSA front companies, then I'm gonna look to see what _else_ them and their partners have invested in...)

retox• 9 years ago

Long been rumoured in the more paranoid corners of the web they are intelligence front/partners.

nine_k2• 9 years ago

Covertly get into a company that terminates ssl for half the internet, and... spill your precious secrets everywhere, instead of siphoning them off silently?

nickpsecurity• 9 years ago

"Step 0) Obtain black funding from NSA budget to start and "VC invest" in a global CDN company..."

I once came up with that exact concept for a nation-state subversion. It would even pay for itself over time. I kept thinking back to it seeing the rise of the CDN's and the security approaches that trust them.

traek• 9 years ago

Not NSA, but the CIA funds and operates In-Q-Tel[1]. They've funded companies like Palantir and Keyhole (which became Google Earth).

[1] https://www.crunchbase.com/organization/in-q-tel

rz2k2• 9 years ago

Am I misunderstanding that this would be useful for parallel construction, but that the public failure actually subverts the usefulness of Cloudflare as a MTIM partnering with someone?

zzzcpan4• 9 years ago

"This bug would have been caught with valgrind, and by the sounds of it, using nothing more complex than feeding their httpd a random sampling of live inputs for an hour or two"

Or prevented using abstraction that do bounds checking. Or even just used ragel with a memory safe language and prevented all issues like that from ever happening. Probably would have been less work even with the reimplementation of an http proxy from scratch.

jackmott2• 9 years ago

with a memory safe language and prevented all issues like that from ever happening.

drastically reduced, but not quite ever. For instance, use a GC language, especially in this domain, you might do some data pooling to reduce GC overhead. Maybe you forget to clear data in the pool. Same kind of error can result.

But yes, I feel like security sensitive stuff like this shouldn't be done in C / C++ any more.

tootie• 9 years ago

CloudFlare is neither the first nor the biggest CDN. I can't recall Akamai having a hole this big. They're either more secure or better at keeping things quiet.

_wmd4• 9 years ago

To be fair to CloudFlare, Google had a heap issue a few years back (maybe like 7 now) where internal flags and copies of argv (which Google use heavily for config) were clearly present in output from their HTTP frontends, including references to Borg before Borg was ever documented publicly.

Over in App Engine land, someone bypassed their JVM sandbox and managed to extract a copy of their JVM image, which included much of their revered base system statically linked into something like a 500mb binary.

Sorry, I'd have to go digging to find references to either of these incidents. At least in either case customer data wasn't leaking, but suffice to say it's a little bit of the pot calling the kettle black

And finally let's not forget the China incident, which rumour has it, resulted in a system compromise at Google right to the heart of their engineering organization. Of course they didn't get roasted like Yahoo recently did over their password leak

Oxitendwe• 9 years ago

You're absolutely right. Cloudflare is a "global active adversary"[1] and has done irreparable harm to the internet at large. This is just a small taste of what's surely to come from CloudFlare's massive influence. They've shown that they cannot be trusted with everyone's data.

[1] https://trac.torproject.org/projects/tor/ticket/18361

JumpCrisscross• 9 years ago

They also actively deter Tor use. I've cancelled subscriptions with Cloudflare-hosted sites because they make securely and anonymously browsing their sites a pain.

rz2k2• 9 years ago

It makes sense that they treat Tor like a probable adversary, but the cost analysis seems really flawed.

Sure, the proportion of requests passing through Tor are more likely to be malicious, but given the bandwidth constraints the adversary seems limited.

The costs aren't only the lost business from people like you, but people who should use Tor giving in. There's some wisdom to people even researching something as mundane as what their dog ingested using anonymized services, much less other medical questions.

Udo2• 9 years ago

I'm running a side-project on Cloudflare and it's accessible through Tor without problems. I suspect this comes down to the settings a site owner sets up in their Cloudflare interface. It would stand to reason if for example you applied the highest security setting across the board, Tor and VPN users would get presented with a captcha.

orasis• 9 years ago

Our CNAME pointing to github pages was down on Cloudflare today with a 1014 error. I'm guessing they broke some other stuff while scrambling to fix this privacy issue? Not a good day for them.

jgrahamc16• 9 years ago

Unrelated.

golf1052• 9 years ago

Oh that happened to my site as well. Was fun trying to figure out what was going on in the middle of the day.

Nothing on their status page about it though :|

the_common_man2• 9 years ago

How does such a simple bug not get picked by auto tests, ci or end to end tests? I am baffled. Since we are behind cloudflare, I am not sure what I should tell my manager now. I lack the technical know how to parse that extremely technical article. Are we supposed to just assume all our traffic that passed via cloudflare is possibly compromised?

It's also a bit sad that travis has to contact cloudflare by twitter. Seriousy?

Edit: https://twitter.com/taviso/status/832744397800214528 is the tweet in question

rdl10• 9 years ago

I don't think he had to, but he got an answer in minutes. I don't think that's the part to be worried about.

As for what you should do: it sounds like the impact is relatively low. I'd personally change easily-changed secrets which go over the session, and potentially externally facing customer passwords (yes in enterprise, maybe not in consumer).

(I don't have any insider info on this breach, though, but I read both posts and know how the system works.)

tyingq• 9 years ago

Sounds bad to me...

"We've discovered (and purged) cached pages that contain private messages from well-known services, PII from major sites that use cloudflare, and even plaintext API requests from a popular password manager that were sent over https (!!)."

The trouble is you have no way to know if someone discovered this earlier, and harvested info for a long time.

Or, how much harvested info from your site might be in a Google cache for someone else's site.

jlgaddis5• 9 years ago

Read Tavis' comments. He disagrees with you regarding the severity. This is a big f--king deal!

Xorlev2• 9 years ago

One of the advantages of being a service is that bugs can go from reported to fixed in minutes to hours instead of months. The industry standard time allowed to deploy a fix for a bug like this is usually three months; we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes.

Great, that makes me feel so much better! I'm sorry, don't try to put a cherry on the top when you've just leaked PII and encrypted communications.

Additionally, most vendors in the industry aren't deployed in front of quite as much traffic as CloudFlare is. It's a miracle that ProjectZero managed to find the issue.

Tiksi2• 9 years ago

Many of the logged urls contained query strings from https requests that I don't think they intended to share.

I guess this confirms a few things.

- The complete query strings are logged,

- They don't appear to be too concerned with who accesses the logs internally or have a process that limits the access, and

- They're willing to send those logs out to a random person.

jgrahamc16• 9 years ago

This has nothing to do with logging.

Tiksi2• 9 years ago

The quoted part that specifically mentions logged urls containing query strings has nothing to do with logging?

geofft2• 9 years ago

During debugging, that seems fine. The previous sentence makes it clear that they added specific logging to track down the problem. I'd rather have a process that allows engineers debugging memory corruption to see the data that's in the process they're debugging, than a process that prohibits them from seeing it.

hendzen• 9 years ago

I think this bug is kind of an indictment of Ragel. It has some great ideas, but since the generated code is so low level - and allows arbitrary blocks of code to be executed in the guts of the parser, bugs like these can result in this horrible memory issues - particularly since the generated code is often used to parse untrusted user input.

jgrahamc16• 9 years ago

I don't blame Ragel.

eridius21• 9 years ago

Ragel shares part of the blame. Why did it use a strict equality check when it could have trivially done a >=?

chm• 9 years ago

Some important parts:

    The examples we're finding are so bad, I cancelled some
    weekend plans to go into the office on Sunday to help
    build some tools to cleanup. I've informed cloudflare
    what I'm working on. I'm finding private messages from
    major dating sites, full messages from a well-known
    chat service, online password manager data, frames from
    adult video sites, hotel bookings. We're talking full
    https requests, client IP addresses, full responses,
    cookies, passwords, keys, data, everything.

    Cloudflare pointed out their bug bounty program, but I
    noticed it has a top-tier reward of a t-shirt.

    Cloudflare did finally send me a draft. It contains an  
    excellent postmortem, but severely downplays the risk
    to customers.

rrdharan• 9 years ago

Connecting some dots, I'm wondering if the "well-known chat service" is Slack:

http://www.computing.co.uk/ctg/news/2462266/whatsapp-reddit-...

Deimorz• 9 years ago

I'm fairly sure that it's Discord.

meowface7• 9 years ago

Never been so relieved my company uses a different CDN...

snaky3• 9 years ago

Do they have a t-shirt awards in their bug bounty programs?

dmitrygr• 9 years ago

Cloudflare's announcement, as it is currently worded, deserves the understatement-of-the-centry award.

jlgaddis5• 9 years ago

Yep, even Tavis said it "severely downplays the risk".

masterleep• 9 years ago

Is that because, even though a very small number of pages (they claim) triggered the bug, any adjacent traffic in memory could be disclosed?

That traffic could be basically anything sent through Cloudflare, it would seem.

danielweber5• 9 years ago

"Don't worry, the keys weren't compromised."

I know how to replace my TLS keys. I have no idea how to replace everything else.

It's like people who think losing my credit card number is the worst thing. No, it can be a hassle, but once I replace it I'm okay. It's everything else.

fulafel2• 9 years ago

The implied comparison to Heartbleed problem is that everyone's old encrypted traffic was suddenly in the open, key change didn't help.

(except for the enlightened few who used PFS before Heartbleed)

sneak5• 9 years ago

Only three thousand something sites [were potentially serving private data from all 7 million customer domains we host]

creshal2• 9 years ago

If the bug exposes random uninitialized memory, can't it affect a lot more sites?

And if it truly is only ~3000 sites, where's the list?

jkells2• 9 years ago

My first thought was relief, thank god I'm not using Cloudflare.

Where would you even start to address this? Everything you've been serving is potentially compromised, API keys, sessions, personal information, user passwords, the works.

You've got no idea what has been leaked. Should you reset all your user passwords, cycle all or your keys, notify all your customers that there data may have been stolen?

My second thought after relief was the realization that even as a consumer I'm affected by this, my password manager has > 100 entries what percentage of them are using CloudFlare? Should I change all my passwords?

What an epic mess. This is the problem with centralization, the system is broken.

joepie91_3• 9 years ago

My second thought after relief was the realization that even as a consumer I'm affected by this, my password manager has > 100 entries what percentage of them are using CloudFlare? Should I change all my passwords?

Yes. Right now. Don't wait for the vendor to notify you.

What an epic mess. This is the problem with centralization, the system is broken.

Yep.

SomeCallMeTim5• 9 years ago

My password manager has > 500 entries. Changing all the passwords....isn't going to happen any time soon.

If it only took 60 seconds per site, it would still take eight hours to change them all.

Might change a few key passwords, though. Couldn't hurt. I only have a couple of bank/financial passwords at this point. And my various hosting service access passwords.

Anything else is not worth the hassle -- and mostly would have 2FA anyway.

bubblethink2• 9 years ago

How do you check if a website uses cloudflare ? Any scripts that do that ?

adiabatty2• 9 years ago

So it's fixed, then? (I haven't read the article yet.)

tedunangst3• 9 years ago

Note that for sites like HN, changing your password doesn't expire other sessions. You have to go find every browser with an HN cookie and logout.

(Where I mean some other sites that are not at all HN, but might plausibly exist.)

nikisweeting21• 9 years ago

We're compiling a list of domains using several scrapers and updating it here: https://github.com/pirate/sites-using-cloudflare

You can start by cross referencing your password manager with this list, and working your way out from there.

jwcrux2• 9 years ago

You don't have to use scrapers, just use copies of the TLD zone files looking for cloudflare nameservers.

AndrewKemendo2• 9 years ago

As an aside, I found this really interesting:

ashleymadison.com

ashleyrnadison.com

I find it really interesting that they registered that particular misspelling and they both point to the same servers. I can see doing this for some obvious domains like gogle.com, but the distinction there is simply that r+n looks like m.

Probably a really obvious answer here, but my guess is that they are trying to help people throw off the scent of someone browsing a history.

corobo3• 9 years ago

You've got no idea what has been leaked

If your site is served through Cloudflare, assume it's all out there because it might be. Standard Big Red Button(tm) procedure.

I don't run any particularly impressive sites but I'll be resetting passwords today. Also cycling things I use behind Cloudflare like DigitalOcean passwords/API keys.

It's supposed to be read-only Friday, Cloudflare :(

a3_nm• 9 years ago

I won't take the initiative of changing passwords, and I will only be doing it for services that ask me to do it.

In my opinion, if my accounts get compromised because the provider uses Cloudflare and leaks my data all over, it's their fault, not mine... It's not my job to guess which services are using Cloudflare, which ones were affected... and further, if my account gets compromised, others presumably will.

(PS: Of course you may need to change passwords if you reuse passwords from one service to the other, but obviously you shouldn't be doing that in the first place.)

adrianN• 9 years ago

The damage is still yours even if its not your fault

sbuttgereit• 9 years ago

If someone runs a red light, broadsides you while you're in the intersection, and leaves you paralyzed... it is their fault both morally and legally... but it still sucks to be you since you bear the consequences regardless of fault.

While this event is orders of magnitude less severe than my example, depending on the service that could be compromised there can be sufficient repercussions that you could not be made whole or avoid on-going inconvenience through the legal system or other acts of the genuinely responsible party.

I absolutely get and sympathize with where you're coming from... but you may want to check a few of your more important accounts none-the-less :-)

martinvol• 9 years ago

They said https never broke, so if you were doing things right way you should not be affected at all. Do not overreact.

Justin_K• 9 years ago

Wrong. You need to re-read the disclosure.

no_protocol2• 9 years ago

Is there a list of sites potentially affected?

I'm assuming I need to change my passwords on a significant number of sites. So far none of them have alerted me to a potential breach. Would love to have a head start.

nikisweeting21• 9 years ago

Compiling a list here: https://github.com/pirate/sites-using-cloudflare

It will be updated as we find more.

loeg2• 9 years ago

Anyone know which password manager uses Cloudflare? Just trying to figure out if I'm affected.

davb• 9 years ago

Thankfully it looks like it's not 1Password, who seem to use AWS CloudFront.

silencio• 9 years ago

taviso called out 1Password in the initial tweet, although that still leaves room for other password managers too: https://twitter.com/taviso/status/834900838837411840

jeremygeros• 9 years ago

According to Tavis Ormandy's twitter post it was 1Password, https://twitter.com/taviso/status/834900838837411840

Trundle3• 9 years ago

Tavis specifically named 1Password, was that wrong?

https://twitter.com/taviso/status/834900838837411840

artursapek5• 9 years ago

I just checked them too, but they may have just switched over.

chris_7• 9 years ago

I am confused - why is 1Password is using anything but iCloud or Dropbox? Those are the only options I see (and "Folder" which is presumably just local).

travem2• 9 years ago

It looks like it was 1Password who have blogged their take on this Cloudflare vulnerability here - https://blog.agilebits.com/2017/02/23/three-layers-of-encryp...

joepie91_3• 9 years ago

This is probably a good moment to recall the article I published a while ago about how CloudFlare is actively putting the web at risk: http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-hav...

This is precisely why. The only thing that surprises me about this, is that it was an accidental disclosure rather than a breach. Other than that, this was completely to be expected.

EDIT: Also, this can't be repeated enough: EVERYBODY IS AFFECTED. Change your passwords, everywhere, right now. Don't wait for vendors to notify you.

Anything could have irrevocably leaked, and you have no way of knowing for sure, so assume the worst.

sparkling6• 9 years ago

Related: http://crimeflare.com/

WillyOnWheels2• 9 years ago

Just looking at that site... what's so bad about Wikipedia? There's a lot to criticize about Wikipedia, but I've never heard of them violating someone's privacy.

mordant3• 9 years ago

I can't change my Uber password - first, the only way to do so is via the 'Forgot your password' dialogue, and second, that now produces a 500 error from NGINX.

Lots of services are going to crumple under the weight of frantic password-reset requests.

jlgaddis5• 9 years ago

Anyone wrote a script yet that checks the top 1M (or so) web sites to find out which use Cloudflare? It would help with knowing what secrets I need to change (as an end user -- I'm not a Cloudflare customer, thank $deity).

sergiotapia• 9 years ago

Popular companies that use CloudFlare can be found here: https://stackshare.io/cloudflare/in-stacks

snug2• 9 years ago

You don't even have to write a script, all that info is already out there.

jlgaddis5• 9 years ago

I know I could find out by checking DNS, etc., but I'd rather not have to do that for every web site I use... and I'm guessing Cloudflare doesn't publish a list of every domain name that they serve.

nikisweeting21• 9 years ago

Yup, running it now. Results are being posted as fast as I can here:

https://github.com/pirate/sites-using-cloudflare

leoh• 9 years ago

Looks cool, you have a lot of duplicates, though.

   $ cat sorted_unique_cf.txt | wc -l
    7385121

   $ cat sorted_unique_cf.txt | uniq |wc -l
    4287625

pkcsecurity• 9 years ago

Not as epic, but I wrote a small node app that checks a list of URLs (made it to check my 1password sites, but it can be used on stuff in general). Not very sophisticated, just checks DNS and setCookie strings...

https://github.com/weltan/cloudbleed-1password

kogir• 9 years ago

I'm not 100% clear: Only three features were affected, and only sites with one or more of those features enabled leaked data into their pages.

But was the leaked data similarly limited to only the sites with the features enabled? Or could it have come from any request - even an entirely unrelated site?

no_protocol2• 9 years ago

It appears that it could be from any site that passed through that server, not just those with these features enabled.

That's just how I read it.

toyg5• 9 years ago

only sites with one or more of those features enabled leaked data

No. From what he says, enabling that feature on a CF proxy basically triggered the bug on any site that happened to go through that proxy, regardless of whether it used the feature or not.

richardwhiuk2• 9 years ago

It only triggered the bug on sites that were using those features, but any other CF site was vulnerable to getting dumped out.

fagnerbrack2• 9 years ago

TL;DR for the lazy ones:

The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

This is huge.

I mean, seriously, this is REALLY HUGE.

sparkling6• 9 years ago

I don't get it. How is this info leaked? From the blog posts, it seems that "only" the HTTP Headers are being leaked and somehow being crawled by Google? But since when does Google store HTTP request info? Can someone explain?

kalmi10• 9 years ago

Headers (among other sensitive stuff) were being leaked inside document bodies.

beachstartup2• 9 years ago

requesting a page with a specific combination of broken tags, when done through cloudflare, will cause neighboring memory to be dumped into the response. op suspects this is due to a bounds checking bug on a read or copy. one can imagine this can be potentially kilobytes of data in one go.

since anyone can put a broken page behind cloudflare, all you need to do is request your own broken page through cloudflare, and start collecting the random "secure" data that comes back.

HappyTypist2• 9 years ago

Everything is leaked, headers are common but full plaintext is leaked in some instances.

dunham3• 9 years ago

Cloudflare handles SSL for a lot of sites. It decrypts everything and passes it along.

For certain other sites, with malformed html, there is a bug that caused it to grab random data (headers and body) from memory and include it in the body of the response HTML. (Some html rewriting product that cloudflare offered was broken and it ran on the same servers.)

This stuff got sent to peoples browsers and also to web indexers like Google or Bing.

Google lets you search for stuff and will also show you the original page that it scraped, making it easy to find this data.

Edit: Also you may be seeing more headers in examples because headers are easier to search for.

rmc5• 9 years ago

HTTP Headers were being including the http response bodies of other random websites. Those websites were being crawled and cached.

edwinyzh• 9 years ago

Just deleted my LastPass account - have been converted to KeePass for over a month.

uladzislau• 9 years ago

Did Lastpass use Cloudflare? That would be a disaster.

jjoe3• 9 years ago

Everyone: change your HN password asap!

StavrosK• 9 years ago

Everyone: HN should implement U2F or TOTP so we don't need to only rely on passwords :(

simcop2387• 9 years ago

How do i get the key information to HN without passing through cloudflare so that someone could sniff it?

SandB0x• 9 years ago

Can someone provide a lay-person's explanation of the issue and its implications?

arestor4• 9 years ago

See https://news.ycombinator.com/item?id=13719437

packetized• 9 years ago

Salient question at this point: Did Cloudflare have any systems in place that would allow themselves to identify queries that were abusing this defect?

coindork• 9 years ago

And it shall be called Cloudbleed.

Natanael_L2• 9 years ago

Still hoping for Downpour

dkarapetyan• 9 years ago

Every piece of dependency in your stack is a vulnerability vector. I feel like this is the only sane assumption to make these days. Yesterday I was thinking of doing some stuff with cloudflare and today I'm reading this report.

The modern web requires a paranoid attitude.

nine_k2• 9 years ago

Only the paranoid survive, Andy Groves famously said.

sparkling6• 9 years ago

I know what Cloudflare is but i don't quite understand the underlying issue.

Can someone explain in simpler terms what happened here and how it a) affects sites using Cloudflare and b) Users accessing sites with Cloudflare?

arestor4• 9 years ago

Some features had a bug which lead to uninitialized memory (AKA previous memory contents) in the output of a malformed HTML page was requested.

As one such server handles many sites, everything that the server handled before that request may be compromised. This includes all HTTP-GET/POST data (credentials, direct messages to other users, ...), Headers (API tokens, Login-Cookies) and contents.

So, you have to assume that everything you did on a CF "protected" website in the last months (especially between 2017-02-13 and 2017-02-17) is potentially compromised.

zaatar2• 9 years ago

Where is a reliable list of CF-protected websites so I may identify which ones I have interacted with?

fagnerbrack2• 9 years ago

Cloudflare blog post related to this incident: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

ers353• 9 years ago

Hacker News uses Cloudflare: http://bgp.he.net/dns/news.ycombinator.com#_ipinfo

Add the following to your hosts file to bypass Cloudflare and access HN directly:

  50.22.90.248 news.ycombinator.com

misframer• 9 years ago

How did you find that IP?

ers353• 9 years ago

I used Censys to search for the IPv4 addresses of servers serving matching TLS certificates: https://censys.io/ipv4?q=443.https.tls.certificate.parsed.na...

belovedeagle2• 9 years ago

(not op) Well as it happens, Cloudflare has had this massive data leak you might have heard of...

sparkling6• 9 years ago

This might be the time to point out the CloudFlare watch blog: http://crimeflare.com/

sfeng3• 9 years ago

The thesis of that blog seems to be that Cloudflare should be censoring content and deciding who gets to have websites on the internet.

mark_edward• 9 years ago

So you think businesses have no responsibility to police themselves and their users? If a shop was caught knowingly facilitating say welfare fraud they'd get fined up the wazoo but for some reason being ~digital~ makes it OK?

kmfrk2• 9 years ago

So how does one find or generate a list of companies using CloudFlare to figure out how you're affected - kinda like HaveIBeenPwned.com?

swang• 9 years ago

it doesn't seem to matter since _ANY_ service that CF gave data to could possibly have cached it and thus have a random site's secrets.

nikisweeting21• 9 years ago

While theoretically many non-cloudflare sites can be affected, I'm compiling a list of known cloudflare-using domains here: https://github.com/pirate/sites-using-cloudflare

It can serve as a starting point, just Ctrl+F for domains you use.

helper• 9 years ago

I was able to get a few hits from a quick google search that are still in google's webcache.

planetix• 9 years ago

So time to reset password and logout of all mobile apps to get new authorization tokens?

tony1014• 9 years ago

Seems like we need to reset/rotate everything. So yes, I believe so.

philip1209• 9 years ago

Interesting. Cloudflare uses a lot of Go, which should hypothetically be memory safe. Was this system in Go? If so, I would be interested in seeing proof of concept code for a vulnerability like this.

detaro2• 9 years ago

It was a bug in C code (automatically generated by the Ragel state-machine compiler), some details are in CFs blog post: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

daenney• 9 years ago

Their old Ragel-based parser was affected. According to their post mortem both the old parser and the new cf-html one are compiled as nginx modules so I'd venture a guess that this is probably C/C++ code since afaik you can't extend nginx through modules written in Go.

It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.

https://blog.cloudflare.com/incident-report-on-memory-leak-c...

There's a section in that blog post titled "Root cause of the bug" that goes in further detail.

artursapek5• 9 years ago

Holy shit, this could be a company-ending event. For CloudFlare or any of its clients.

chx• 9 years ago

Their clients, sure, especially they are HIPAA regulated (let's pour one out for the poor sods) but CF only if everyone abandons them and many won't. Gross negligence does not even exist online AFAIK and so criminally you can't even start because there's nothing to work with, perhaps negligence but that's a slap on the wrist. A civil suit ... sure you can sue anyone in civil court for whatever but you need to prove damages here and that'll be bloody hard.

spydum2• 9 years ago

People are going to lambast CF for downplaying the impact, and there could be merit in that.

However, I really want to say I am absolutely impressed with both Project Zero AND Cloudflare on so many fronts, from clarity of communication, to collaboration, and rapid response. So many other organizations would have absolutely tanked when presented with this problem. Huge kudos for CF guys understanding the severity and aligning resources to make the fixes.

In terms of P0 and Tavis though, holy crap. Where the heck would we be without these guys? Truly inspiring !

Kalium11• 9 years ago

CF's infosec team is very, very good at their jobs.

jrcii• 9 years ago

Obviously not, right?

sneak5• 9 years ago

Then why are they talking about that 3000-ish number instead of the 7 million number?

rasz_pl• 9 years ago

Good at lying for sure https://twitter.com/taviso/status/834918182640996353

mattbee7• 9 years ago

Cloudflare isn't just a security hole in the middle of the internet, they're a protection racket.

If you wanted to pay to DDoS a site, search for "booter" and you'll get a list of sites that will take another site off the internet for money with a flood of traffic.

quezstresser.com webstresser.co topbooter.co instabooter.com booter.xyz critical-boot.com top10booters.com betabooter.com databooter.com

etc. etc. - from the first 30 results I could find 2 booter sites that weren't hosted by Cloudflare.

But hey, pay Cloudflare and your site too can be safe from DDoS attacks...

sfeng3• 9 years ago

You are essentially arguing against freedom of speech. Cloudflare will protect any site that doesn't host child porn. Yes that includes things which you don't like, but it also includes all the things you do.

mattbee7• 9 years ago

I ... didn't say any of that.

abeyer3• 9 years ago

Cloudflare will protect any site that doesn't host child porn

Doesn't that make it worse? They aren't saying they don't or won't police the content they protect. They are obviously capable and willing to draw a line on ethical or legal grounds, if they have done so in that case. They have just chosen to draw that line on one side of porn but another side of DDoS services.

Ultimately it is their decision to make, but I don't think it's unfair for people to question possible conflicts of interest in how that decision is made.

trowawee• 9 years ago

You don't understand what freedom of speech actually means.

mordant3• 9 years ago

DDoS attacks are the ultimate form of censorship.

mattbee7• 9 years ago

Another implication: they could be using their access to these sites' traffic to prepare their own infrastructure for attacks before they happen.

There's nothing about their hosting of these sites that doesn't reek.

manigandham5• 9 years ago

That is how DDOS protection works, learning from data and scale to better defend future attacks. Every large network and security operator does this. What is your issue with that exactly?

eridius21• 9 years ago

In what way is this a protection racket? That's sort of like complaining that mob-owned businesses enjoy the same police & fire protection that all other businesses have.

mattbee7• 9 years ago

Cloudflare sells protection from the internet attacks through its network. The same company and network facilitates the organisation of those same attacks, and helps keep them anonymous.

That's a high-tech protection racket.

toyg5• 9 years ago

It's the same stance that antivirus developers have always had, more or less. As usual, the difference between blackhat and whitehat is very, very thin - if there is a difference at all.

mcbits2• 9 years ago

By the same logic, the search engine you used to find those sites is also a "protection racket".

caf2• 9 years ago

Really? That search engine sells DDOS protection?

aacook• 9 years ago

I don't really understand your point. In 2012 I was working on a startup that was DDoS'd and it was not fun. This was back before Cloudflare offered a DDoS service and we ended up having to hire a random company in Canada to help get us back online. At the time there were surprisingly few people out there offering DDoS mitigation. Cloudflare wanted to help us but they were still in early development for their service, but I remember them being good guys. What's wrong with providing a service to help fight the bad guys?

kalleboo• 9 years ago

The problem comes from the conflict of interest when you're also hosting the bad guys

rsync• 9 years ago

He is pointing out that cloudflare is also hosting the DDOS sites.

Those are sites that you can go to to pay for a DDOS.

So they are taking money from the people who you are paying them to defend you against.

lsmod2• 9 years ago

Be careful posting random domains.HN might flag/throttle your account for spamming.happened to one of my accounts.

dang4• 9 years ago

That's rare but possible. If you weren't spamming, I'm sorry. Let us know at hn@ycombinator.com and we'll fix it.

mook2• 9 years ago

Given that the whole class of operators seem somewhat shady, I imagine that sometimes they would need to fend off attacks from competition services. In that case, being on a free DDoS protection plan seems like a reasonable thing to do (from their point of view). As long as they're not initiating the DDoS via Cloudflare, I'm not sure how that would be unreasonable of CF's part, given that I assume it's all automated and nobody ever looks at what sites have signed up.

I don't like CF for their fishy SSL architecture, the increased centralization of internet traffic, and the constant captchas when using tor, but the DDoS protection part (regardless to what sort of people they're providing service for) seems fine.

user59944615• 9 years ago

The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

1) From the metrics I recalled when I interviewed there, and assuming the given probability is correct, that means a potential of 100k-200k paged with private data leaked every day.

2) What's the probably that a page is served to a cache engine? Not a clue. Let's assume 1/1000.

3) That puts a bound around a hundred leaked pages saved per day into caches.

4) Do the cache only provide the latest version of a page? I think most do but not all. Let's ignore that aspect.

5) What's the probably that a page contains private user information like auth tokens? Maybe 1/10?

6) So, that's 10 pages saved per day into the internet search caches.

7) That's on par with their announcement: "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains." Well, not that we know for how long this was running.

8) Now, I don't want to downplay the issue, but leaking an dozen tokens per day is not that much of a disaster. Sure it's bad, but it's not remotely close to the leak of the millennia and it's certainly not internet scale leak.

9) For the record, CloudFlare serves over one BILLION human beings. Given the tone and the drama I expected way more data from this leak. This is a huge disappointment.

Happy Ending: You were probably not affected.

nathancahill2• 9 years ago

I think your estimates fell apart at step 2, 1/1000 pages being cached. HTTP is aggressively cached, on many different layers. I'd put it closer to 1/10.

user59944615• 9 years ago

I meant cached by a public service like Google Cache Bing Archive.org that expose the pages.

A browser cache might be 1/10 but that's not open.

mattbee7• 9 years ago

This assumes that the Bad Guys hadn't noticed the bug before Tavis, and hadn't started intensively mining Cloudflare for data.

user59944615• 9 years ago

Intensive mining indeed, if it's true that it requires 3.3M requests to get a page leak.

With a fixed 100Mbps connection and assuming 2kB per HTTP request-response, you can hope to get one leak every 11 minutes and 6.6GB of traffic, which is a constant 5k requests/s.

Maybe if Google reassigns all its SHAterred ressources to doing that...

... and then I realize that we were talking about cloudflare and my mining bot a capcha.

---

edit: correction. The bug was affecting only some pages with some content filtering options enabled, and was more prominent under some specific circumstances.

Hence why it only happens 1/3.3M in average. An attacker could allegedly leak data much more reliably if he was able to identify the patterns that are more likely to trigger leaks.

gukov• 9 years ago

I hope something like intense mining would've caused a detectable spike in turn alerting CF sooner. Looks like it didn't...

tr32q423• 9 years ago

The root cause is apparently coming from auto-generated code that causes buffer overrun:

    /* generated code */
    if ( ++p == pe )
        goto _test_eof;

With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains.

The examples in the report shows Uber, okcupid , etc. It would be good to know the full list, to know what password might have been compromised.

https://blog.cloudflare.com/incident-report-on-memory-leak-c...

nikisweeting21• 9 years ago

We're working on getting a full list up here: https://github.com/pirate/sites-using-cloudflare

I'm currently just searching the Alexa top 10,000 doing DNS scraping, but I'll updating it with reverse resolves from cloudflare.com/ips/ next.

DangerousPie• 9 years ago

Has anybody else actually received an email from Cloudflare about this? I'm a paying customer, but haven't heard anything from them yet. I hope they don't expect they can leave it at a random blog post that will go by unnoticed?

tony1014• 9 years ago

Another paying customer here. No email communication from CloudFlare. Found out about this on HN.

ndesaulniers3• 9 years ago

You and OP should contact them, as customers.

MattBearman• 9 years ago

I'm a free customer, also no email...

jasonlingx• 9 years ago

Just got an email:

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-c...

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince Cloudflare, Inc. Co-founder and CEO

kowdermeister• 9 years ago

I have received 2 emails mentioning many things listed here and also letting me know that my domain was not affected.

---

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-c...

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince Cloudflare, Inc. Co-founder and CEO

shthed• 9 years ago

Has anybody received an email to say that their data WAS exposed?

dantiberian2• 9 years ago

From Twitter:

"@taviso their post-mortem indicates this would've been exploitable only 4 days prior to your initial contact. Is that info invalid?" - https://twitter.com/pmoust/status/834916647873961984

"@pmoust Yes, they worded it confusingly. It was exploitable for months, we have the cached data." - https://twitter.com/taviso/status/834918182640996353

jgrahamc16• 9 years ago

From my blog on this:

    The three features implicated were rolled out as follows. 
    The earliest date memory could have leaked is 2016-09-22.

    2016-09-22 Automatic HTTP Rewrites enabled 
    2017-01-30 Server-Side Excludes migrated to new parser 
    2017-02-13 Email Obfuscation partially migrated to new parser 
    2017-02-18 Google reports problem to Cloudflare and leak is stopped

benwilber0• 9 years ago

Well fuck. I have no idea what (if any, or all) of my authenticated web sessions have been going through CloudFlare in the last 6 months. How do I even start to protect myself from this?

dantiberian2• 9 years ago

With respect, the blog post buries the user with details. In my opinion, there should have been in bold at the top something like:

Title: Security report on memory disclosure caused by Cloudflare parser bug

(This is a security report, "incident" underplays this. Memory leak sounds a lot more innocuous than memory disclosure).

Data from any website that was proxied via Cloudflare since September 22, 2016 may have been leaked to third parties via a bug in Cloudflare's HTML parser. Operators using Cloudflare should:

* Invalidate session cookies

* Reset user passwords

* Rotate secrets

* Inform users that private data (chats, pictures, passwords, ...) may have been inadvertently leaked by Cloudflare.

* ...

Users using websites proxied by Cloudflare should:

* Reset their passwords

* Log in/out of sessions to remove session tokens

(Begin rest of post)

Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare. ...

clebio2• 9 years ago

not trolling, I followed your HN profile link: what blog post? http://blog.jgc.org/

cwisecarver2• 9 years ago

This sounds to me like an object lesson in "Why you shouldn't write your own HTML parser."

Every time I see a dev trying to parse HTML with a custom solution or regex or anything other than a proven OSS library designed to parse HTML I recoil reflexively. Sure, maybe you don't need a parser to see if that strong tag is properly closed but the alternative is ...

abeyer3• 9 years ago

You're right in 99+% of cases. But I suspect that the needs of cloudflare for this use case aren't typical of what's expected of an html parser. I'm not certain that there isn't an existing parser that would work for them, but I'm equally not certain that there is.

cwisecarver2• 9 years ago

I can see the argument but 99+% of this audience isn't cloudflare. My comment was more directed at those who aren't. Special use-cases are all over the place. It's just making sure you're choosing because your use-case really is special and that when you re-implement something that you're doing it because it's different and better, not because you'd rather write something than integrate.

alkonaut5• 9 years ago

Even so, if the parser handles security or human safety then it shouldn't be written in C, or even using a parser generator that generates C.

Just use ML, or Rust, or bloody JavaScript for all I care. I don't care if they add a ton to response time, or add 100% perf overhead costs for running the thing.

Having an OS, ssl library, web server etc written in C is bad enough but at least that code has many eyes on it. Companies shouldnt throw their custom made tyres on top of that fire.

implr• 9 years ago

This seems relevant https://i.imgur.com/l4kjNba.png

yclept• 9 years ago

for easy firewalling and i'm sure a fun internet experience https://www.cloudflare.com/ips-v4

secfirstmd• 9 years ago

"and even plaintext API requests from a popular password [1Password] manager that were sent over https"

Plaintext?

matthewowen• 9 years ago

So, they know which sites leaked data in responses. It sounds like they can also say categorically that some sites won't have been affected (if they don't share any infrastructure with the sites that could have leaked data).

Will Cloudflare be explicitly notifying customers about whether data from their site could have been leaked by this bug?

jacquesm2• 9 years ago

Hm. Not so good. The main website that I log in to that uses CloudFlare is this one.

curuinor5• 9 years ago

Can we start a list of affected right now? I found:

OKCupid

Uber

people claiming 1Password, can't find

Lyft

Yelp

Pingdom

Digital Ocean

Montecito Bank and Trust

curuinor5• 9 years ago

I found:

FitBit

Hacker News

Stack Overflow

Zendesk

Discord

FastMail (not really see below)

nmjenkins• 9 years ago

We, FastMail, are not affected by this. We do not proxy TLS connections via any third party. We use CloudFlare for DNS distribution only, which is not part of this issue.

fabian2k• 9 years ago

Stack Overflow is not directly affected (see http://meta.stackexchange.com/a/291482/151385). They stopped using CloudFlare before this issue was introduced.

curuinor5• 9 years ago

https://stackshare.io/cloudflare

RapGenius

Coinbase

Bitpay

Product Hunt

Udemy

Crunchyroll

curuinor5• 9 years ago

Patreon

4chan used to use it apparently, don't know if affected

kik

Zoho CRM

change.org

Cloudflare itself, of course

Feedly

portref2• 9 years ago

Anyone know if Zoho mail is vulnerable too?

According to doesitusecloudflare.com Zoho isn't using Cloudflare, was it previously?

nikisweeting21• 9 years ago

I'm compiling a list of affected domains here, please submit PRs: https://github.com/pirate/sites-using-cloudflare

I'm currently running a DNS scraper to find more.

curuinor5• 9 years ago

You should probably keep the porn sites on the list, folks have accounts at porn sites too

andrenotgiant2• 9 years ago

Is your list "customers of cloudflare" or "customers of cloudflare that could have sensitive data cached by search engines"

For example, Digital Ocean uses cloudflare, but the domain with sensitive data (cloud.digitalocean.com) is entirely blocked from Search Engines https://cloud.digitalocean.com/robots.txt

xuki2• 9 years ago

It doesn't matter, your info could have leaked via other sites.

woodrow• 9 years ago

Lyft is not a Cloudflare customer (I work at Lyft).

gooeyblob5• 9 years ago

Reddit is not affected.

piker• 9 years ago

Given that the plaintext is cached (or feared to be), is googling/binging one's passwords a bad way to check for pwnage?

artursapek5• 9 years ago

Sounds like a bad idea. Maybe search for a small part of the password?

manigandham5• 9 years ago

Side note: HackerNews uses CloudFlare.

ta2987• 9 years ago

We need an official and comprehensive list of domains served by Cloudflare throughout the affected period.

nikisweeting21• 9 years ago

I'm compiling an unofficial list, hopefully they'll release an official one though:

https://github.com/pirate/sites-using-cloudflare

The issue is they have to get permission from their customers before releasing affected domains.

throwaway7959• 9 years ago

Can anyone ELI5 what's going on?

nikisweeting21• 9 years ago

Change all your passwords. Many sites use Cloudflare to serve secure content, and it was recently discovered that Cloudflare has been leaking secure content, including passwords, API tokens, etc. to unintended viewers.

ThrustVectoring• 9 years ago

Maybe I'm being a bit too paranoid, but shouldn't your services be set up in a way that doesn't let Cloudflare touch that sort of sensitive data in the first place? You can't distrust everything, of course, but "compromised reverse-proxy acts as a MITM by logging and exfiltrating sensitive information" seems like it ought to be in the threat model of service providers.

AYBABTME4• 9 years ago

This comes around to me as something that just shouldn't have happened. CloudFlare are pretty big on Go, as far as I can tell (and I guess Lua for scripting nginx). Why was this parsing package written in a non memory-safe language? Parsing is one of those "obvious" things easy to mess up; the likelihood of a custom, hand written parser being buggy is pretty high. If it's somehow understood that your library is likely to have bugs, why do it in C/C++, where bugs often lead to bleeding memory? In a shop that's already fluent in Go, where they have the institutional knowledge to do it safely? Sure performance is not going to be the same, but with some care it'll get pretty close.

Sorry I hate to just be a coach commentator. Obviously hindsight is 20/20. Still I think there's a lesson here.

belovedeagle2• 9 years ago

Memory safe languages aren't a panacea. There could just as easily have been a bug in the compiler or standard library with the same result.

AYBABTME4• 9 years ago

Sure... but that probability is equally present in the non memory-safe language, so that doesn't change anything.

hkjgkjy4• 9 years ago

True. I don't understand why many of us programmers are not interested in tools that eliminate the possibility of errors?

* Why do we use memoy-unsafe languages (except when Rust or GC is unusable)? * Why do we use type-unsafe languages, at all? * Why do we use state-unsafe (mutable) languages, at all?

Of course there are exceptions to these - but they are few.

ddebernardy• 9 years ago

There aren't so many languages that are a) memory safe, b) type safe, and c) thread safe, that additionally offer d) a large enough pool of developers to recruit from.

aleksei• 9 years ago

It is not the fault of the language if you use it wrong.

CloudFlare is to blame here, nothing else.

As for the reason why C, I'm pretty confident they knew what they were doing, and had considered other tools that did not meet all requirements.

MrBuddyCasino2• 9 years ago

Security is a requirement. They must have been extremely confident indeed to write something like this in C, where a single mistake can make your program fail in catastrophic ways, with no help whatsoever from the compiler.

If some code has bugs, did the author just "use the language wrong"? People make mistakes, and we can prevent some of them by using better tools.

pmontra• 9 years ago

Allow me an analogy. It's not the fault of a rope if you use it to cross between two skyscrapers and slip and fall to the ground. But if if you mind your life you use at least a safety cable to tie you to the rope, or use the lifts and cross at road level. There are still cars to watch for, but...

C is simply too dangerous, even the best developer can slip without noticing. There are safer alternatives now, we should start using them at least for new projects.

alkonaut5• 9 years ago

No, the language is bad if using it wrong can leak sensitive data.

The choice of language is wrong if you pick such a language in a situation where mistakes can lead to safety or security problems.

The first requirement is security.

mook2• 9 years ago

The blog post makes it seem like the problem was in an nginx module. Looking at the docs [1] it looks like that's a C API; as far as I know writing shared libraries in golang for a C caller isn't really a thing (because the runtime needs to exist). Rust might have better luck here (I _think_ there have been attempts to get rust code loaded by not-rust code), but I haven't kept track.

[1] https://www.nginx.com/resources/wiki/extending/api/main/

MrBuddyCasino2• 9 years ago

Calling Rust from C is easy. Details: http://siciarz.net/24-days-of-rust-calling-rust-from-other-l...

ars2• 9 years ago

This could easily happen in Go as well. All that would be needed is to reuse the buffer in between requests, and rely on the buffer length instead of clearing it.

To make it safer you would need to deallocate and reallocate the buffer for each request, but that might be slow. Doing that would fix it for Go, or for C, it would be the same either way.

So I'm not convinced that using Go would have helped here.

duneroadrunner4• 9 years ago

It's a good point, but at least with Go the leak would be limited to the allocated buffer. This is probably a case where Rust or C++ might be more helpful. Presumably you wouldn't want to allocate a new (variable sized) buffer each time (particularly in a GC language), but you could create a new (bounds checked) slice[1] / array_view[2] / gsl::span / RandomAccessSection[3] each time.

[1] https://doc.rust-lang.org/nightly/std/slice/

[2] https://github.com/rhysd/array_view

[3] https://github.com/duneroadrunner/SaferCPlusPlus#txscoperand...

drvdevd2• 9 years ago

I agree. I keep seeing comments about C being the culprit, but in my mind, this is more of a policy issue regarding how any given language initializes and allocates memory.

Sure, in this case, we may see a C-specific bug in play, but I think this sort of bug is more effectively mitigated by forcing buffers to be zero-filled upon allocation and/or deallocation, and perhaps system-wide at the OS level, rather than relying upon language features to cover it.

So - I'm not explicitly defending C here - I just don't think a similar bug could never occur in a "memory safe" language as well.

zzzcpan4• 9 years ago

"This could easily happen in Go as well."

Not really true. Go operates on slices that panic on out-of-bounds accesses. So, for this to happen in Go you would have to reinvent slices and use a lot of manual C-style code to operate on them, which literally nobody does in Go, because it's too hard.

AYBABTME4• 9 years ago

That is true, but reusing buffers in Go is a lot more deliberate an action than in C. The possibility is still there, but I think it's way harder to mess up.

Soarnrobertson3• 9 years ago

This article is beginning to look like a whole bunch of people talking about a leak and not saying that they would use that data for vicious things.

Soarnrobertson3• 9 years ago

Welp, time to move and get a different IP again :\

chiefalchemist2• 9 years ago

Let's be honest. There are holes. More than we care to admit. The truth, if embrassed, could undermine the world's economy. It's just question of when.

chiefalchemist2• 9 years ago

Down voted? God bless the naive.

hkjgkjy4• 9 years ago

HaveIBeenPwnd must be having a great day today!

cypherpunks01• 9 years ago

"We also undertook other search expeditions looking for potentially leaked information on sites like Pastebin and did not find anything."

How comforting!

Soarnrobertson3• 9 years ago

So, would LastPass be involved in this at all? Do tey use CloudFlare?

bartread• 9 years ago

Not according to the lists others have posted, but 1password are.

soheil2• 9 years ago

What's the rationale behind sending user PII through a CDN? Presumably that is useful to that one user only so a CDN wouldn't be super useful in distributing the load across its edges. Also doesn't CDN caching kinda defeat the purpose of having SSL?

orthecreedence3• 9 years ago

Cloudflare terminates SSL and then forwards the request to your servers as one of their services. This isn't about the CDN, but about them terminating SSL, then leaking the plaintext data back through other requests.

soheil2• 9 years ago

What are the benefits of terminating SSL early at the CDN level? It seems to me the risks associated with not having SSL still remain they're just shifted to between the CDN and the backend. Is it much more than just giving lip service to SSL and getting away with things like browser restrictions, etc.?

techolic• 9 years ago

Is there an International Day of Internet Security? I think we should make today that day.

hehheh• 9 years ago

I'm a little drunk so please forgive me if I'm way off base here or if I'm ultimately describing a service that already exists.

Unless I'm mistaken, CloudFlare's services necessarily require they act as a MITM. Would it be possible or practical change the DDoS protection service such that it uses an agent on the customer's end (the CF customer) that relays relevant data to CF, instead of having CF MITM all data?

As it is now, we have:

  End user <-> CF MITM to inspect packet data <-> CF Customer site

where CF uses the data discovered through MITM (and other metadata such as IP) to determine if the end user is a bad actor.

What if we, instead, had something like:

  End user <-> CF TCP proxy <-> CF Customer site
                   ^                    |
                   |                    v
             CF decision agent <-- CF metadata ingest

The CF captive portal would not work with this but they could still shut down regular ol boring TCP DDoSes.

hkjgkjy4• 9 years ago

I think you have yourself a solid company idea right there. Go for it.

manigandham5• 9 years ago

You wouldn't be able to have any CDN caching, only transit of encrypted traffic. Which is fine, but all the major clouds have load balancers that already do this and have varying levels of included and paid DDoS protection.

pmahoney2• 9 years ago

I haven't found a clear answer to this:

CloudFlare has multiple SSL configurations:

Flexible SSL: There is an encrypted connection between your website visitors and Cloudflare, but not from Cloudflare to your server.
Full SSL: Encrypts the connection between your website visitors and Cloudflare, and from Cloudflare to your server

(I'll add Full SSL mode still involves CloudFlare terminating SSL (decrypting) before re-encrypting to communicate to your server)

If I am running in Full SSL mode, is (or was) my data vulnerable to being leaked?

maknz• 9 years ago

Full SSL requests still terminate at CloudFlare, and would still be vulnerable. It's just that CloudFlare's connection to your origin is also encrypted.

pmahoney2• 9 years ago

Thanks. Wish they had explicitly stated that all SSL modes were affected (unless I missed it...)

mabbo3• 9 years ago

Signs you are about to have a bad time: Tavis Ormandy publicly tweets that he urgently needs someone from your security team to contact him, and no, the public disclosure form won't do.

daxfohl• 9 years ago

Anyone know of a way to google for your passwords (assuming you have strong, unique passwords) to see if they've been exposed anywhere, without exposing them?

ceejayoz4• 9 years ago

Change them, then Google them?

joeyh• 9 years ago

The bug was in cloudflare servers, not code run on customer's own web servers, right?

nikisweeting21• 9 years ago

I've compiled a list of 7,385,121 domains served through cloudflare using several scrapers. https://github.com/pirate/sites-using-cloudflare

The full list is available for download here (23mb) https://github.com/pirate/sites-using-cloudflare/raw/master/...

I will be updating it as I find more domains.

jkells2• 9 years ago

More than 7 million domains... Letting that sink in...

I'm assuming this list is based on DNS records? I wonder what proportion of those offloaded their SSL to Cloudflare.

nikisweeting21• 9 years ago

I had duplicates, it's actually only 4,287,625 (still a lot though).

Fixed the duplicates: https://github.com/pirate/sites-using-cloudflare/raw/master/...

fulafel2• 9 years ago

Yet another strong argument for end-to-end security. Terminate in the middle, and you risk things like this.

Hopefully people will learn something from today.

enraged_camel• 9 years ago

Your comment got flagged and killed, which I thought was bullshit so I vouched for it.

Because you're correct: if CF's info sec team is "very very good at their jobs", how did this incident happen?

dang4• 9 years ago

Vouching for a good comment that happens to be dead is fine—that's what vouching is for. Posting commentary about it is not fine. There are plenty of non-bullshit reasons for comments to be dead, including that an account is banned for having abused the site.

We created vouching because some banned accounts sometimes post ok comments; indeed some banned accounts only do that when they're banned, and immediately start posting abusively the moment they're unbanned. Life is complicated.

We detached this comment from https://news.ycombinator.com/item?id=13720437 and marked it off-topic.

hkjgkjy4• 9 years ago

One of the reasons I prefer paying with Bitcoin over credit card, is that when using cryptocurrency I don't have to give them they key to my account - instead they give me an inbox that I send the value into.

Guessing a lot of credit card details are ripe for picking in the data they leaked.

ceejayoz4• 9 years ago

Guessing a lot of credit card details are ripe for picking in the data they leaked.

Sure, but that's where credit cards shine in comparison to Bitcoin. In the US, you're protected by Federal law in that scenario. A brief pain in the ass - reporting the fraud and getting a new card number - and you're out $0.

Meanwhile, a bit of malware can drain millions of dollars of Bitcoin with zero recourse. It's gone. This isn't theoretical, it has happened. https://www.theguardian.com/technology/2016/aug/03/bitcoin-s...

aerovistae• 9 years ago

I noticed StackOverflow is on the list of compromised sites. I sign into that via my google account. Does this mean I need to change Google credentials?

bichiliad2• 9 years ago

In StackOverflow's case, login happens using Oauth2 by way of accounts.google.com, meaning your google login creds don't go through StackOverflow's servers - Google acts as a trusted 3rd party and verifies that you are who you say you are, and tells StackOverflow.

ceejayoz4• 9 years ago

You won't need to change your Google password, that's never given to websites as part of Google's OAuth process.

You may want to revoke access at https://myaccount.google.com/permissions and reconnect to SO.

niftich4• 9 years ago

Most of these 'Sign in with [Social Identity Provider]' implementations, including Google [1] use OIDC ("OpenID Connect"), which in turn itself is built on top of OAuth 2.0. From the OAuth 2.0 side, the site into which you wish to gain access into -- in this case, StackOverflow -- only sees opaque tokens that are usually short-lived.

However, OIDC then typically delivers some choice personal info -- no more than you agreed to when first consented to the integration, but usually account name and/or email, and maybe real name and some demographic data -- to the requesting service so that they can both find you in their datastore, and sync up these attributes. In the case of a service whose OAuth/OIDC callback url's SSL is terminated with CloudFlare, which we (as of writing) don't yet know if applies to StackOverflow, this info will touch CloudFlare servers and could have been contents of memory that was exposed. However, your password would not be, as in your case, the password was supplied to the Social Identity Provider (Google) who didn't use CloudFlare to terminate that connection, and the password never left Google, which was the precise usecase and requirement that the OAuth/OIDC specs were authored to support.

[1] https://developers.google.com/identity/protocols/OpenIDConne...

kyledrake5• 9 years ago

Friendly reminder that Cloudflare willingly hosts the top DDoS-for-hire attack sites, and refuses to take them down when they are reported.

Run WHOIS on them, it's almost 100% behind Cloudflare: https://www.google.com/#q=ddos+booter

I would be less concerned about the fact that Cloudflare is spraying private data all over the internet if people weren't being coerced into it by a racket.

We won't have a decentralized web anymore if this keeps going. The entire internet will sit behind a few big CDNs and spray private data through bugs and FISA court wire taps. God help us all if this happens.

chipperyman573• 9 years ago

Friendly reminder that Cloudflare willingly hosts the top DDoS-for-hire attack sites, and refuses to take them down when they are reported.

Why should CF be required to police the internet? CF doesn't even host them, they just protect their sites from DDoS and DNS.

kyledrake5• 9 years ago

Cloudflare has spent a lot of time gaslighting people into believing this, but it physically, scientifically, OSI model-y isn't true. Cloudflare hosts web sites. When Cloudflare CDN edges that content, that content exists on their servers. Just because the canonical store is on another machine doesn't mean they don't host the site. If I mirror a site from some other server, and you're loading that site from my server, I'm the one hosting that site. That's how HTTP works.

The argument that they don't know what's hosted on their network has also been demonstrated by evidence as nonsense. The reason the Pirate Bay got blackholed by Cogent last week was because Cloudflare was grouping all of the BitTorrent sites on their network onto a single IP address, and a Spanish court order related to a different site ended up BGP blackholing over two dozen torrent-related sites as collateral damage.

http://seclists.org/nanog/2016/Jul/400 https://mailman.nanog.org/pipermail/nanog/2017-February/thre...

Cloudflare is completely capable of enforcing this, yet they don't do anything about it. It benefits them financially to not do anything, because they get business from these DDoS attackers trashing other networks on the internet, making it so you can only have sites stay up if they are hosted by Cloudflare's broken, bleeding servers.

This is fundamentally an extortion racket. Frankly, it should be a crime. This is exactly the kind of problem laws exist for.

tfar• 9 years ago

"CF doesn't even host them, they just protect their sites from DDoS and DNS."

The #1 excuse people use. They do more than just DNS, they deliver the actual data, that would have been delivered by the original host, to visitors. So I'd consider them hosting an automatically updated mirror, and as bad as the original host.

mcintyre19943• 9 years ago

Note that if Cloudflare didn't have the content of those sites and their requests in memory this couldn't have happened.

Coincoin• 9 years ago

They are charging us money to protect us from the same people they are protecting? Genius.

stephenr• 9 years ago

Who wants to bet this won't change a lot of developer's making statements like "I use <Insert HTTPS offering CDN> so my site is secure"

symlinkk• 9 years ago

I've been going through Google's and Bing's caches for about 2 hours looking for leaked credentials and I don't see much - many results don't have an option to view a cached copy. I think Google and Bing are wiping any cache entry that are affected by this vulnerability.

aioprisan3• 9 years ago

CloudFlare's disclosure severely downplays the impact that this can have on their customers. We're going to close our account shortly.

DanielDent• 9 years ago

There are still a lot of results with leaked data in Google's Cache and they are pretty easy to find..

Some possible queries: "CF-Int-Brand-ID", nginx-cache "Certisign Certificadora Digital",

Once you find one, you can look through the results for unusual strings/headers which you can use to find more results.

Many results have clearly been removed from Google's cache, but.. many also have not.

bcl• 9 years ago

Here's a simple little Rust app to check a list of domains for CF usage -- https://github.com/bcl/uses-cf

jcwayne• 9 years ago

Makes me wonder if the Great Firewall has a caching layer.

jononor2• 9 years ago

It would surprise me immensely if they did not. Many mobile operators also operate caches.

actuator• 9 years ago

I wrote a script which checks the domains you have visited from your chrome history to see if they use Cloudflare by checking if the header `cf-ray` is present in their response headers: https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d...

Found my bank's site on it. :(

mordant3• 9 years ago

Apparently, the only way to change one's Uber password is to use the 'Forgot password' path on their login page.

So, I clicked on that - and I get a 500 error from NGINX.

My guess is that a lot of services are going to be overwhelmed by the sheer volume of password reset requests, thus preventing users from resetting their passwords.

abalone• 9 years ago

Cloudfare's bug bounty maximum reward[1]:

1. Recognition on our Hall of Fame.

2. A limited edition CloudFlare bug hunter t-shirt. CloudFlare employees don't even have this shirt. It's only for you all. Wear it with pride: you're part of an exclusive group.

3. 12 months of CloudFlare's Pro or 1 month of Business service on us.

4. Monetary compensation is not currently offered under this program.

Guessing they're gonna reconsider #4 at this point.

[1] https://hackerone.com/cloudflare

kbody• 9 years ago

Indeed, I've heard the issue of low signal-to-noise ratio, but it's pretty irresponsible not to offer any real reward.

stevenhubertron• 9 years ago

Well now I have a great response to the sales guy that bugs me everyday.

Blackthorn4• 9 years ago

This is probably gonna get buried at this point, but one thing I'm surprised about is this seems like yet another parser bug. Why are we still using hand-written parsers? Even if you're Very Smart, you'll probably get it wrong. We have parser generators for a lot of things. Even for mostly unparseable garbage like wild-type HTML we have pretty good libraries for handling it. Fresh hand-written parsers are just bombs waiting to explode.

chrisper5• 9 years ago

https://news.ycombinator.com/item?id=13719530

Doesn't seem to be hand-written.

niftich4• 9 years ago

Your comment doesn't apply for this particular case, because the submission goes into great detail that the parser in question was written with Ragel, a parser generator. The code written by them in Ragel contained a bug, which lay uncaught and dormant for years, and manifested only when calling/wrapping code was altered.

Blackthorn4• 9 years ago

It still seems like a gross mismatch of power though. Correct me if I'm wrong but Ragel only can output parsers for regular languages, yes? You can't call their Ragel code an HTML parser because Ragel can't output a parser powerful enough to parse HTML.

jononor2• 9 years ago

I would also expect that regularly performed fuzz testing would be able to catch such a bug? Especially if ran in combination with dynamic memory analysis like Valgrind. So were they not doing this?

xenadu02• 9 years ago

Another day, another C memory safety bug that completely breaks all security everywhere.

We're definitely doomed to repeat the same mistakes over and over.

sneak5• 9 years ago

Probably only for 30-50 more years, honestly.

mshenfield• 9 years ago

Most honest comment in this entire thread.

wereHamster• 9 years ago

So.. when are we going to stop using unsafe languages which allows these kinds of memory corruption or leaks? If this is not reason enough, what else needs to happen before people realise that whatever language the cloudflare proxy is written in is a really bad one?

mrep• 9 years ago

What is the optimal balance between centralization and decentralization? Most people in this thread are complaining about how using a big centralized service (cloudfare) causes so much damage when security issues come up, and yet I have seen many people advocate using a single password manager (like 1password) to which this exact type of huge security problem can happen (your password manager is the single point of security failure which can comprimise all of your accounts!!!).

What is the optimal solution???

floatboth5• 9 years ago

There's a difference between a MITM proxy in front of a huge portion of the web and a password manager that's running locally on a personal machine.

Also there's the 2-factor stuff to protect you when you somehow lose your manager's master password. What protects you when the proxy in front of you misbehaves and exposes your shit?

homakov5• 9 years ago

Chrome marking Cloudflare HTTPS as "Secure" must be turned into something different, like "Not So Secure" or whatever. Secure = end to end.

Cloudflare is MitM by design. Chrome and others must not tolerate it. This vulnerability is just another reason to do it asap.

bubblethink2• 9 years ago

That's because HTTPS allows that. Whether it's cloudflare, or your own servers and load balancers, it's all legal. So it would be unfair to single cloudflare out. You could take some measures to identify their flexible-ssl traffic, and that's a grey area, but their regular ssl is fine. If it weren't for them, you would roll your own solution, which wouldn't be very different.

lifthrasiir2• 9 years ago

Ultimately I believe CF is sustaining its business by filling a gap in the Internet, namely DDoS protection. Until somehow the gap is closed we will see CF-like services continue to be popular even after this incident.

pat2man• 9 years ago

So is any CDN.

homakov5• 9 years ago

Forgot about that, thanks. However we can use "integrity" attribute

stevebmark• 9 years ago

Is there a government body that can enforce fines over this? Or is a class action lawsuit the only way to seek damages?

steven_pack• 9 years ago

If only there were a systems programming language, offering c-like performance with memory guarantees and well suited to high throughput network servers that would catch this class of bugs at compile-time [1] [2]

[1] https://www.rust-lang.org/en-US/ [2] Self declared rust fanboy

ComputerGuru2• 9 years ago

Some day, the world will wake up to the fact that we've taken the beauty of a decentralized internet and willingly traded it in for a single-point-of-failure design.

I will refrain from any criticism of Cloudflare and what I think about this because they're going through hell as it is. But everyone else is fair game. The higher a level of service you centralize, the more you stand to lose.

DannyBee• 9 years ago

So, does the t-shirt say: "I found a zero-day bug in cloudflare and all i got was this lousy

X-Uber-token:

X-Uber-latitude:

... "

leni536• 9 years ago

"GO EASY ON THE BUGS, leave some for the next person."

jtchang• 9 years ago

This is scary stuff. Any key/password that you used on a cloudflare site should be considered compromised.

That's a crapton of keys.

ReedJessen• 9 years ago

Reddit just told me my account was compromised

gooeyblob5• 9 years ago

Unrelated.

snek• 9 years ago

gg to project0

thurston2• 9 years ago

Author of Ragel here.

An experienced Ragel programmer would know that when you start setting the EOF pointer you are enabling code paths that never executed before. Like, potentially buggy ones. Eek!

duneroadrunner4• 9 years ago

I'm not familiar with Ragel, but you might consider adding support for SaferCPlusPlus[1] as an output target. It should be a fairly simple modification of your existing C or C++ output generator. SaferCPlusPlus was, in part, designed for this (I mean, CloudFlare's) sort of situation - using C/C++ in internet facing applications.

[1] https://github.com/duneroadrunner/SaferCPlusPlus

duneroadrunner4• 9 years ago

Actually, the Ragel generated code (fragment) looks like it's already pretty SaferCPlusPlus compliant, since it itself doesn't declare any buffers or pointer/iterators. So I guess the recommendation instead would be for CloudFlare, and anyone else writing nginx modules in C/C++, to wrap any nginx-supplied buffers in a bounds-checked wrapper (like an array_view[1], gsl::span, or RandomAccessSection[2]).

[1] https://github.com/rhysd/array_view

[2] https://github.com/duneroadrunner/SaferCPlusPlus#txscoperand...

mcintyre19943• 9 years ago

I'm not sure if you've had a chance to look at the Cloudflare blog post yet (https://blog.cloudflare.com/incident-report-on-memory-leak-c...), but while they take full responsibility they do point out (under root cause of the bug) that a generated equality check could be >= instead which would avoid the bug. Obviously GIGO applies and it's their bug, but it might be worth seeing if there's anything you can do on Ragel side?

thurston2• 9 years ago

Well doing that would mean ragel would incorrectly read one character, rather than run off forever. Personally I'd rather have the latter. Much easier to catch with memory checkers. Eventually you try to read some thing you're not allowed to read, or blow something else up, instead of just read the first byte of the int following the buffer, or whatever.

There would have to be an additional bounds check when issuing a goto in an error action, but doing that is contrary to the simple execution model that ragel users have come to rely on.

Gotta ask the question, where was the testing when they altered 7 year old code without the involvement of the original developer?

benevol• 9 years ago

Well, keep centralizing and this is what you get, sooner or later.

Also, mono-cultures have always been a very bad idea, not just in agriculture.

borplk• 9 years ago

It says their bug bounty program has a top-tier reward of a t-shirt? Wow ... don't go bankrupt Cloudflare.

acd• 9 years ago

Cloudflare is also breaking anonymous surfing by throwing captchas at you. Security wise they do DDoS ok but not WAF which Incapsula does a lot better. When I mean better I mean protection against exploits.

Rican7• 9 years ago

Yeaaaaa, this isn't good.

This is what CloudBleed looks like, in the wild: https://gfycat.com/ElatedJoyousDanishswedishfarmdog

A random HTTP request's data and other data injected into an HTTP response from Cloudflare.

Sick.

jeromenerf• 9 years ago

This is what CloudBleed looks like

Ironically, gfycat seems down now.

willtim• 9 years ago

If you must write your HTML parser in C/C++, then you should expect buffer overruns.

codezero• 9 years ago

Why do they need to add google analytics to random people's web pages?

ffjffsfr• 9 years ago

Does anyone know answer to this question someone is asking there at the end? Is it related?

could you tell us why a lot of people had to re-authenticate their Google accounts on their devices all of the sudden? It may not have been related, but Google definitely did something that had us all re-authenticate.

I too had to reauthenticate and was very worried because it was first time I had to do this, I thought something bad happened with my account and it was very suspicious.

dreamcompiler• 9 years ago

Only inherently unsafe languages like C make it possible for an amateur-hour HTML parsing blunder to spew secrets all over the Internet. If you can't be bothered to check your return codes, at least use a language that doesn't multiply the damage from that mistake a million-fold.

danvdragos• 9 years ago

How was https traffic leaked? Cloudflare, in order to offer its services, acts like a man in the middle and internally decrypts https traffic [0]

[0] https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflar...

johnhenry• 9 years ago

"Cloudbleed".

omgtehlion• 9 years ago

What bothers me is not the bug itself, but the fact that so much sites and apps terminate SSL at cloudflare that NSA/FBI/other-3-letter-agency does not need to come after any separate company, but just needs to tap cloudflare and call it a day.

27182818284• 9 years ago

Everyone has heard the expression that we need to diversify portfolios. Perhaps it is true of CDNs

tannhaeuser3• 9 years ago

Holy sh*t. Is this the end of Cloudflare with the trust being absolutely destroyed and lawsuits coming in? Can't say I'm sad for them. Cloudflare sells you DDOS protection, and hosts (eg. masks the IP of) the very DDOSers to protect against themselves, which I find bordering on the criminal.

Hosters like Hetzner, OVH have for a year now offered DDOS protection (I'm guessing it's heuristic rate limiting, but they won't tell details b/c that would make it trivial to workaround it, so they say). Could someone characterize their offering and tell me if it's any good?

To those spinning a story against C programming here: it is entirely possible (trivial, even) to isolate address spaces between requests, and has been for like 25 years (CGI programming) and more. When you absolutely must use a long running, single-address space service container, OpenBSD's httpd shows how to do it right (goes to great lengths to randomize/re-initialize memory etc.). I agree, though, that using straight C isn't a good choice for the latter.

efdee• 9 years ago

Last time I checked, Hezner's DDoS "protection" basically meant they disconnected you from the network if you got an incoming DDoS. Has this changed?

tannhaeuser3• 9 years ago

According to https://wiki.hetzner.de/index.php/DDoS-Schutz/en, they're using Arbor and Juniper kit

tannhaeuser3• 9 years ago

From https://arstechnica.com/security/2017/02/serious-cloudflare-...:

    A while later, we figured out how to reproduce 
    the problem. It looked like that if an html page
    hosted behind cloudflare had a specific
    combination of unbalanced tags,
    [...]
    The leakage was the result of a bug in an HTML
    parser chain Cloudflare uses to modify Web pages
    as they pass through the service's edge servers.

Ahem, at the risk of sounding pedantic, but this wouldn't have happened when using a proper HTML/SGML parser ([1]).

[1] http://sgmljs.net/blog/blog1701.html

Karupan• 9 years ago

This is huge and CF is certainly downplaying the issue. To be clear, I think the kind of tech that they deal with is extremely complex, which makes it ever harder to test or uncover them easily. And they have been reasonably good with disclosures (prior to this incident).

When I was evaluating CF for a small personal app, I really thought hard about using a public reverse proxy and decided that it wasn't worth it for the scale I was dealing with. No one can predict these security issues, but I sure am glad I didn't go with them!

XorNot• 9 years ago

Incidents like this remind me that the password problem is only partially solved by password managers: most of the internet (i.e. if you're not my bank) needs a simple, easy to script protocol that allows me to automate the process of rolling a lot of passwords.

mixedbit• 9 years ago

There is a huge fleet of compromised machines out there that belong to botnets. Soon we will see the botnets operators extracting content from these compromised machines browser caches to hunt for data leaked in this incident. Clearing search engines caches is just not enough, all secrets need to be replaced.

askvictor2• 9 years ago

Could this be the reason behind having to reauth my Google accounts in the past couple of days? I.e. did Google invalidate all auth tokens in case they leaked via a third party website via CF?

orian• 9 years ago

It seems so.

frankmoodie• 9 years ago

Question: what about the %99 of the internet users who have no idea what SSL/HTTP/any other web tech is ? How are they even going to be notified?

a3n2• 9 years ago

When they receive their free one year credit report.

fatal510• 9 years ago

Nothing will come of this. Just another hype event. You will get your usual change your password PR emails from a few companies.

bartkappenburg• 9 years ago

From a cloudflare employee:

"We were working to disclose the bug as quickly as possible, but wanted to clean up search engine caches before it became public because we felt we had a duty of care to ensure that this private information was removed from public view. We were comfortable that we had time as Google Project Zero initially gave us a 90 day disclosure window (as can still be seen in their incident tracker), however after a couple of days, they informed us that they felt that 7 days was more appropriate. Google Project Zero ended up disclosing this information after only 6 days."

ricardobeat• 9 years ago

Straight from the issue tracker:

    They then told me Wednesday, but in a later reply started saying Thursday
    [...] If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks.

    https://security.googleblog.com/2013/05/disclosure-timeline-for-vulnerabilities.html

HugoDaniel• 9 years ago

So its not only the tor browser experience that sucks with cloudflare.

alkonaut5• 9 years ago

Just stop using pointer arithmetic and manually managed buffers for anything security/safety related already.

Had this proxy been written in nearly any other language it wouldn't have had this vulnerability, like so many similar vulnerabilities.

Using ML or Rust or Java or whatever doesn't magically make all vulnerabilities disappear but it sure makes those that are intrinsic to C disappear. And that's not just a few.

There is just no excuse.

_pmf_• 9 years ago

From the incident report at https://blog.cloudflare.com/incident-report-on-memory-leak-c... (not the article):

About a year ago we decided that the Ragel-based parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain.

I'd assume that at this point, customers would like to have a little more than a vague promise.

tete• 9 years ago

Wow, I only recently had a discussion about "What if this happens?". Great timing to make a point. Unique "told you so" opportunity, but I actually am sad that this happened. Millions of people wasting time on password changes and related things again. :(

And now off to resetting a lots of password and checking where OTPs are possible.

dorianm2• 9 years ago

Here is a list of domains where I found public leaked data: http://doma.io/2017/02/24/list-of-affected-cloudbleed-domain...

afandian• 9 years ago

In addition to comments here calling the words 'memory leak' disingenuous because it's technically correct but underplays the problem, I'm now seeing articles in non-technical publications referring to the incident as a "leak".

In the wider world the word "leak" doesn't mean memory access patterns, it means deliberate sabotage.

The headline in "The Verge" is "Password and dating site messages leaked by internet giant Cloudflare". That's technically correct too, but also gives completely the wrong message.

Simpler, proactive messaging from Cloudfront might have helped here.

a3n2• 9 years ago

Password managers are mentioned.

I looked on the lastpass blog (s/www/blog/), nothing about this. Is it just too early?

sleepychu• 9 years ago

Lastpass does the crypto on the client side, your encrypted password database could have been leaked but if your master password is sufficiently strong then it will be hard to break.

That said if you reset all your current lastpass passwords with newly generated ones after changing your master password you'll protect yourself from any attack.

Sami_Lehtinen• 9 years ago

I made similar site too, but with geolocation, tags, and fully threaded replies and private messages. Like & Dislike - As well as machine learning which will dig most interesting posts for you. As well as score near by posts higher etc. But nobody cared. So I'll be shutting it down in 6 months. (Domain expires)

rickdmer• 9 years ago

I created a Chrome extension that searchs your bookmarks for sites that use Cloudflare: https://chrome.google.com/webstore/detail/cloudbleed-bookmar...

csomar• 9 years ago

Oh boy, this is bad as fuck. Major bitcoin exchanges were affected and these are exchanges where if you can login, you might be able to withdraw the cash irreversibly for ever.

I'm trying to figure out how bad this is; and a part from the exchanges I'm using which other sensitive sites are concerned.

clebio2• 9 years ago

So, two of the three hard problems in computer science (fencepost and cache invalidation)?

vermontdevil• 9 years ago

I got an email from Cloudflare and here's an excerpt about the # of sites affected by this.

Not sure what to make of it - the low number of domains affected.

====================================

problems6• 9 years ago

Yeah, I got this this morning too. It seems to be a pretty big downplay - it should be closer to "change all your passwords, have all your customers change all their passwords". They're busy shredding data from caches, but anyone scraping cloudflare sites in recent days might have data around that they'll never know about.

But I don't blame them entirely - it's unlikely this will have been used and unlikely a given customer's data would be present, so it'd induce panic which would probably never have resulted in an attack.

nkkollaw• 9 years ago

Could some kind soul do an ELI5?

I'm not lazy, it's just overwhelming trying to figure out what's actually going on with all these comments...

faragon• 9 years ago

Does anyone know if there is a way for mapping virtual addresses to areas with zeroes and replacing it with the memset to 0 on write access, so software could be still efficient without calling calloc() instead of malloc()? (i.e. memset to 0 only for actually written zones)

jitbit• 9 years ago

Webmasters and App-devs running on CloudFlare. You (at least) have to "force-logout" your users that have "remember me" cookie set.

At least change the cookie name so the token stops working. For example, in ASP.NET - change the "forms-auth" name in the web.config file. etc etc.

sambull• 9 years ago

RIP Cloudflare 2017.. took you long enough

hatsunearu• 9 years ago

(It took every ounce of strength not to call this issue "cloudbleed")

and some chap did it anyways. yay, i guess.

jhgjklj• 9 years ago

Cloudflare please stop asking me if i am a robot and then ask to pick the store board posts for ever. What kind of idiot coded that, asking me always.

Globz• 9 years ago

I was planning on moving my website over DigitalOcean and I now http://www.doesitusecloudflare.com/?url=www.digitalocean.com is telling me that they are affected by cloudbleed, I guess I should wait it out...

snikeris• 9 years ago

Anyone have any additional information about this bit from the comments:

and even plaintext API requests from a popular password manager that were sent over https (!!).

chousuke• 9 years ago

I used the lastpass CLI tool and some UNIX tools to do a tentative check of which of my domains might be affected. Something like the following should work okay:

    lpass ls | egrep -o '[a-z]+\.[a-z]+' | sort > mydomains.sorted
    sort sorted_unique_cf.txt > cf_really_sorted
    comm -12 mydomains.sorted cf_really_sorted

It's not perfect (since it will only look at the lastpass item description, not the actual URL, and will only match foo.tld type domains), but it still found a number of domains for me

brilliantcode• 9 years ago

how far back does this affect websites on cloudflare? I removed mine a year ago because I was using it for the SSL.

This will put the final lid on cloudflare anyhow. Sticking with AWS.

djhworld• 9 years ago

Can someone explain to me why they were parsing HTML in the first place? That's the bit I don't fully understand, but I've not got experience of what Cloudflare does, I thought they were a CDN

SadWebDeveloper• 9 years ago

Oh boy what a great week... first we have SHA-1 getting a fast-track to the obsolete hashes and now cloudflare is f*cking everyone because they tried to obfuscate emails from websites and fail to "test every edge" case... whats next is the question.

apple4ever• 9 years ago

I have yet to receive an email about this. Very disappointed that I had to find out via another source 12 hours after the blog post was up.

tkachenko• 9 years ago

Small service to check if your site is POSSIBLE affected to CloudFlare data leaks https://cloudflareleaks.webtls.com/

hacknat• 9 years ago

I'm surprised to learn that people with real security concerns are using Cloudflare. I put it in front of my blog, but I would never use it in front of something that has sensitive data. I just don't get how companies like Zendesk could be so stupid. I barely blame Cloudflare. If you think terminating SSL with a CDN is a good idea you get what you deserve.

mderazon• 9 years ago

So did anyone find out why so many Google accounts got "action required" alerts yesterday ?

pjmlp2• 9 years ago

Time for the C. A. R. Hoare's weekly quote, taking time to reflect on what happened since 1981 regarding computer security on system languages.

The first principle was security: The principle that every syntactically incorrect program should be rejected by the compiler and that every syntactically correct program should give a result or an error message that was predictable and comprehensible in terms of the source language program itself. Thus no core dumps should ever be necessary. It was logically impossible for any source language program to cause the computer to run wild, either at compile time or at run time. A consequence of this principle is that every occurrence of every subscript of every subscripted variable was on every occasion checked at run time against both the upper and the lower declared bounds of the array. Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to - they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law.

-- Turing Award lecture 1981

Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

Related Stories