Ask HN: Does Cloudflare block HN comments if you have code blocks in a reply?

https://news.ycombinator.com/item?id=38986187

by zikduruqe • 3 years ago

311 139 3 years ago

news.ycombinator.com

I was going to make a comment about using netcat to send files, and Cloudflare blocks the submission. I have never seen this happen to me on HN.

https://imgur.com/a/Pyhcssl

Has anyone run into this before?

bhaney• 3 years ago

Any code including netcat (for it's tendency to be used in reverse shells) or SQL (for it's tendency to be used in SQL injections) tends to be blocked across the entire cloudflare-net these days.

zikduruqe• 3 years ago

Interesting. That's the first time I have run across this. Makes sense.

crotchfire• 3 years ago

That does not make sense at all; it is batshit-crazy broken.

Jabrov• 3 years ago

How does that make sense?

tambourine_man• 3 years ago

In a site like HN, that’s ridiculous.

goles• 3 years ago

See: https://news.ycombinator.com/item?id=38939668

renonce• 3 years ago

Sites like HN could have disabled WAF. It's entirely configurable on HN's side. Let's just wait until dang wakes up and implement the required changes.

creatonez2• 3 years ago

AFAIK most of these filters are disabled by default when setting up your website on Cloudflare, so most websites using the Cloudflare network likely have this turned off.

stingraycharles2• 3 years ago

This is correct, it’s part of their WAF offering where there is rule-based blocking of content.

dvh• 3 years ago

drop table users where id=id

kristianp• 3 years ago

Select id, email from users;

Works. Any sql thats actually blocked?

gus_massa2• 3 years ago

To get a fast answer, it's beter to send an email to the mods hn@ycombinator.com

LeoPanthera• 3 years ago

In this case, I appreciate that OP made a public post, in case others encounter the same problem.

gus_massa2• 3 years ago

dang is not reading every single post in real tieme, so posts may get unoticed and the guidelines ask to send an email instead.

In this case, it looks like some comments are giving good advice about the tradeoff and how to fix it, so I have to agree with you.

kaimac• 3 years ago

Cloudflare has access to everyone's cleartext? I was unaware of this. NSA must love that

strombofulous• 3 years ago

Yes, although it requires configuration https://developers.cloudflare.com/ssl/get-started/

arch-choot• 3 years ago

I'm pretty sure the default is they can see all the cleartext, since their product is based on TLS interception, for example to evaluate page rules.

This is also how they insert extra headers in both the request and response.

creatonez2• 3 years ago

When you add a DNS rule, it's configured as proxied by default. Here is what it looks like in the UI:

https://i.imgur.com/TO2Tfk3.png

https://i.imgur.com/jVW5db4.png

midasuni• 3 years ago

If cloudflare have thr certificate’s private key and are advertising the A record they have access to everything you send, from emails to credit card numbers.

ytch• 3 years ago

It's a TLS termination proxy that decrypt and re-encrypt your TLS packet. Technically Cloudflare can read anything unless you add your own crypt layer on top of TLS.

judge20207• 3 years ago

Same for Akamai, Cloudfront, Fastly, etc. Pretty much every business that wants to offload DDOS protection, caching,and some level of frontline security uses a proxying CDN.

An alternative is to keep all of your CDN assets on a CDN bucket on its own hostname, with your main secret-containing business apps on your own servers, but it costs a lot to manage this level of separation and the payoff is only protection against the theoretical attack of "NSA can't attack our users/spy on them". If the NSA ever did do this on a large enough scale or to target a particularly notable person, it's very unlikely it would be kept a secret for long, and the end-business that used Cloudflare et al. wouldn't be implicated whatsoever since every business uses one of the big CDN providers.

bejk22• 3 years ago

That makes using https instead of http a lot less relevant.

bomewish• 3 years ago

They kept the other spying secret for a long time and it was only due to pretty heroic actions by one person that it got exposed. So I duno.

stingraycharles2• 3 years ago

It’s a CDN that caches content and it’s able to inject “are you human?” verification pages, it can rewrite content on demand (e.g. serve optimized images / html / JavaScript). It seems obvious to me that they have access and ability to modify all cleartext content in-flight.

mlindner• 3 years ago

Yes that's how Cloudfare works. The TLS certificate for basically any website using Cloudflare "ends" at Cloudflare's servers. It's then either forwarded on to the actual servers in cleartext or re-encrypted with an internal company certificate (maybe signed internally as well) to pass the connection on to the actual servers. It was the easy way many companies who didn't have the expertise to do their own certificate management moved from the http world to the https world. They just handed it off to cloudflare and kept their servers running http.

F5 Networks, my former employer, sells something similar, but it's a box (or virtual appliance) you put in your own data centers somewhere that dead-ends the connection instead.

jpc0• 3 years ago

Btw the same is possible for phishing sites.

It's entirely possible to have a proper SSL connection to a bogus hostname, that is showing the correct website and even interacts correctly.

Bogus MITM decrypts the traffic, logs it, then forwards the traffic once again encrypted to the destination server. Then does the reverse for the resonse.

"Look for the padlock" is only useful if the actual hostname is correct in the browser.

If I hosted news.ycombnator.com using this and you didn't notice that I could be proxying just like that. It's possible cloudflare has protections against this in place but doesn't every website on earth?

Look at the damned hostname people.

cheekibreeki23• 3 years ago

And every big company can decrypt your tls web traffic with their special CA keys.

fragmede3• 3 years ago

Can you explain what you mean a bit more? My connection to eg my bank isn't decryptable by anybody but me and my bank (and their CDN which is serving their certificate). That is, eg, Verisign has root CA keys to sign the cert, and they could give me a cert that says they're my bank and I could make a new connection that they could decrypt, but the original connection to my bank can't be decrypted by their keys.

furyofantares• 3 years ago

Huh. Just tried submitting the same comment with the same result.

Minimal test, if I try to edit this post removing the asterisk, I get the "banned" page

  nc* 192.168.2.100

neilv• 3 years ago

Probably not related, but I've been getting lots of throttling-like huge page load delays on HN the last couple days, only when logged in.

Any idea whether that's just an overloaded application server, or something Cloudflare is doing?

pixelesque• 3 years ago

That's often the case with HN I think from past experience when there are large threads on HN, and dang has in the past said that's due to the application server.

keepamovin2• 3 years ago

Single core application server no less

djha-skin• 3 years ago

hyperhello• 3 years ago

What if you Base64 encode this? Pretty trivial to add to the form logic.

dylan604• 3 years ago

yeah, i'd expect dang to just jump right on that. just because you feel it is trivial does not mean that it should be done.

kevincox3• 3 years ago

That's how one of my past employers resolved this. Basically base64 encoded every field in the JSON as someone reported a bug where the WAF blocked it. Not only was this done inconsistently and was super tedious but completely defeated the purpose of the WAF. (Except of course to check the checkbox that we had a WAF.)

usr11064• 3 years ago

So HN uses Cloudflare? That surprises me because typically I notice sites using Cloudflare because my mobile running GNU Linux cannot pass their dreaded Turnstyle. Luckily that does not happen for HN.

judge20207• 3 years ago

They used to run it but stopped (I want to say) around 2016 or 2017. Another poster here linked[0] to how dang confirmed it is to protect against a DDOS attack.

0: https://news.ycombinator.com/item?id=38939668

usr11064• 3 years ago

The cert is Cloudflare today.

cj• 3 years ago

The aggressiveness of the "dreaded Turnstyle" is 100% configurable.

It's very easy to disable it completely via Cloudflare settings. Using cloudflare doesn't require you to use all of its features, and almost every feature can be turned off.

viraptor3• 3 years ago

While this is true and worth reminding the ops about, it still sucks because many people don't understand the issues they cause by turning WAF on. CloudFlare should have a big "I understand I'll block many legit clients when I enable this" checkbox. Or you know... fix it in general. Or at least have a "report this block as invalid" link on the page.

helsinkiandrew• 3 years ago

I always feel the turnstyle makes a website feel a bit condescending - you need to test MY connection before I get to your crappy site?

Is there a reason it needs to be visible whilst performing checks? or is it just security theatre?

a-dub• 3 years ago

bummer. i used to like the legend that it was all on one commodity linux pc implemented in some nice concise lisp running on sbcl.

edit: my memory is crap. it was a single machine, but the codebase was written in a custom experimental language that i think was a lisp derivative. (which would make sense!). the source was online at some time, can't find it now.

usr11064• 3 years ago

I understand the backend is still somewhat limited. After Altman was fired (IIRC) when HN got 1000s of comments within no time, dang asked everyone to logout and login again, to make life for the poor machine a bit easier. So besides limited HW resources, also some non-optimal implementation details ;)

lolinder• 3 years ago

It still is a single single-core server, dang references it frequently when there's unusually high traffic [0]. And the language you're referring to is Arc [1]. They do have caching for not-logged-in users, historically done through nginx [2]. From other comments in this thread, it sounds like they just temporarily put Cloudflare in front of that single server to block a DDoS.

[0] https://news.ycombinator.com/item?id=38310213

[1] https://arclanguage.github.io/

[2] https://news.ycombinator.com/item?id=26473226

dividuum• 3 years ago

Yep. Without leaving the browser, https://news.ycombinator.com/cdn-cgi/trace confirms that.

usr11064• 3 years ago

Oh, how did you find that? Some inside knowledge about HN or is that a well-known path published by Cloudflare sites?

Edit: Seems to work also for at least some other Cloudflare sites. Interestingly HN is served from Stockholm (behind a sea cable) while others are served from Helsinki (should be closer). Not enough hackers here in Finland?

Edit 2: Works also on sites where Turnstyle keeps me out.

LinuxBender• 3 years ago

It is at the moment.

    news.ycombinator.com. 3710 IN CNAME news.ycombinator.com.cdn.cloudflare.net.
    news.ycombinator.com.cdn.cloudflare.net. 3710 IN A 104.22.6.236
    news.ycombinator.com.cdn.cloudflare.net. 3710 IN A 172.67.5.232
    news.ycombinator.com.cdn.cloudflare.net. 3710 IN A 104.22.7.236

nneonneo• 3 years ago

I ran into this when trying to post a comment with

  ../ ../ ../ etc/ passwd

(remove the spaces)

M4v3R• 3 years ago

Yep, confirmed on my side too that it does indeed get blocked.

Jamie99122• 3 years ago

I don't remember HN being on Cloudflare. Have they recently added it?

ignoramous• 3 years ago

Yes, I see news.yc proxied through Cloudflare.

Also: https://www.nslookup.io/domains/news.ycombinator.com/dns-rec... / https://archive.is/R9BE8

jrockway• 3 years ago

This line works:

   nc -l -p 1234 -q 1 > testfile.txt < /dev/null

The other one doesn't.

   alias foobar=nc
   cat testfile.txt | foobar 192.168.2.100 1234

I was hoping that it was a "useless use of cat" filter, but nope. It just doesn't like the bytes nc next to an IPv4 address.

This is also fine, but blocked if you change the slash to a dot:

   nc 192/168.2.100 1234

This works too:

   nc \
   192.168.2.100 1234

OK, that's all for now. Can you believe people pay money for "web application firewalls"?

david_draco• 3 years ago

Good that there are open source alternatives:

  socat - TCP4:192.168.2.100:1234

  socat TCP4-LISTEN:1234,fork,reuseaddr -

https://www.redhat.com/sysadmin/getting-started-socat

gnfargbl• 3 years ago

Can you believe people pay money for "web application firewalls"?

I think it's like a lot of things in computer security, in that system owners just don't want to be the slowest gazelle in the herd. If an attacker is mass-exploiting some new remote vulnerability, then maybe the WAF means that you're one of the lucky ones who doesn't get hit. And yes, that's a very big maybe there.

WAFs don't do much to prevent targeted attacks except to require the attacker to craft a WAF bypass. As you've shown.

H8crilA• 3 years ago

They also have an organizational purpose. Once an attack happens you can shift the blame onto the WAF. And the WAF provider can, if needed, claim this to be a novel attack against which they have prepared for the future. Even issue an emergency patch that detects and blocks the novel backslash-newline line break technique.

(I'm exaggerating here, but only a bit)

kevincox3• 3 years ago

I think that important difference though. WAFs are very useful when they are used as a quick fix for a zero-day vulnerability. Especially if you are running some generic software like WordPress and can subscribe to a managed WordPress rule set.

The key is that they should mitigate specific vulnerabilities and ideally once the proper fix is deployed the rules are then removed from the WAF.

WAFs have near-zero value for things like trying to detect shell-code or generic SQL injections as they just turn into fuzzy bug injectors. Any real attacker will very quickly find a way to structure their exploit to avoid the heuristics.

WAFs also have minimal value for custom software. Because even if you use it to deploy a quick block for an exploit unless you are just blocking a whole endpoint the attacker will also likely find a way to work around it.

teeray• 3 years ago

Also, never underestimate the power of FIPS999999999 or whatever compliance. If it’s a checklist item to have a bowl full of M&Ms without brown ones in your data center, your security people will make sure that box gets checked. It doesn’t matter how outdated the requirement is.

mac-chaffee• 3 years ago

WAFs are 2000s-era software that have long overstayed their welcome: https://www.macchaffee.com/blog/2023/wafs/

mr_mitm• 3 years ago

Nice summary.

I can add to the list of attack vectors a case where the WAF introduced a reflected cross-site scripting vulnerability. The site it was supposedly protecting was blank, i.e. it just returned a 404 error or something. But just by appending a URL parameter with JS in it, the WAF would trigger and reflect the code. So I was able to build an outlook web app lookalike for phishing on a site with the domain of the company.

leftnode• 3 years ago

Excellent article, thanks for writing and posting it.

kazinator• 3 years ago

Summary:

1. WAFs require entire requests to be buffered in order to be scanned before the server sees them. This can require lots of RAM.

2. WAFs scan requests with all sorts of hacky rules, which takes gobs of CPU time.

3. The hacky rules look for programming language syntax, for which the attackers can easily find alternative expressions to get around the rules.

4. ... yet, WAFs have high false positive rates.

5. All that kludgly processing is a security weakness. WAFs tend to be closed source behemoths written in low-level languages.

d0mine• 3 years ago

There are many valid points. Though it doesn't cover many things modern wafs systems do in addition to the regex rules.

The question what is the alternative and the suggested alternative is that everyone become perfect security expert. It is even less likely to succeed than creating security-aimed software by professionals.

Consider what is the proportion of sites that are created by people who knows zero about security. Wordpress is like on 40%+ web sites (ridiculous).

WAFs unlikely to prevent targeted attacks, they don't have to to be useful. In practice, simple measures can prevent many common attacks.

oars• 3 years ago

Interesting article about whether WAFs are actually useful or not in modern times. Thanks!

whodev• 3 years ago

I was kinda with you until you made this statement: > No, "defense in depth" is not a valid excuse to use a WAF anyway, because it provides no real defense!

I have to disagree here. You are making assumptions that every developer in an org will always do the correct thing and deploy code that won't be exploitable to SQL injections, XSS, file inclusion, etc... That's just not the case. I'm all for doing the correct thing, and not just performing security theater, but WAFs do offer some protection. You need multiple layers of security covering the holes that may left in other layers. And a WAF can be one of those layers of protection.

cheekibreeki23• 3 years ago

Have you ever tried to run mod_security? You sound like you've never been in the trenches.

donatj• 3 years ago

Corporate recently pushed us into using a WAF. It has been nothing but a PITA

terom• 3 years ago

Easy: just set up the WAF but with an empty ruleset.

andrewaylett• 3 years ago

Honestly, so many of these complaints about web application firewalls fly so far past the point that I'm not surprised folk don't understand them.

Yes, they can be trivial to bypass. And yes, they don't block everything. But also, yes, they can be very useful in some situations.

My employer's security team maintains a WAF, and while it may be frustrating at times (like when anti-directory-traversal rules broke page names with '...' in them) I mostly prefer that they continue to do so for two big reasons: script kiddies and botnets.

It doesn't matter that a bypass is trivial when in practice your attacker won't mutate their attack -- if the attacker was more sophisticated, the defence could be too, but if the attack is dumb then there's no point in a sophisticated defence.

Botnets mean that purely reputation-based defence is insufficient. The best defence to a distributed attack is one that's really cheap to evaluate. If all an attacker ever tries is to hit our homepage with a fixed user agent string, then all we need to do is block that UA from hitting our home page. A simple WAF entry is sufficient to block that particular attacker.

This precise example is indeed poorly-applied, as the system is intended to receive arbitrary text of arbitrary technical complexity. But I wouldn't mind the rule being applied to my team's endpoints, as we can be confident that anyone sending shell has malicious intent regardless of whether there's any chance that my services would try to execute the code (they won't).

So long as it is possible to bring down services without any effort, skiddies will keep trying to do that. And so long as we've people trying dumb attacks in infrastructure, dumb defences can have a worthwhile effect. And if the dumb defences start catching stuff they're not supposed to catch, like the example with '...', they're dumb enough that we can understand why they're doing that and if we can safely turn them off.

sixhobbits• 3 years ago

Test

`nc 192.168.2.100`

rstuart4133• 3 years ago

There is no need to do that here. Putting the text into the 'About:' box of your HN profile works just as well while not cluttering up the thread.

buro9• 3 years ago

Probably, the WAF, specifically Cloudflare specials, matches a number of things. And as a lot of it is just regex matching the context of where the match occurs isn't precise.

Additionally cloudflare doesn't know what is safe for a given site, so it has to be a little conservative. The sites that can handle malicious input, or are tech sites that expect things that are SQL or commands that may contain directory traversal, these are in the minority.

Essentially these are false positives, which are typically viewed as more acceptable than false negatives as those would allow attacks through.

These things are configurable by the site owners, but the issue here is that the site owners are not shown the code of the rules, so have to guess from the names and descriptions whether something is safe to disable, meaning everyone just leaves everything enabled. Usually reporting this to a site owner with the cloudflare trace id is sufficient to enable the site owner to disable a rule that is causing false positives, as the site owner can use the cloudflare dashboard to search the trace id.

I do not work there any longer (left 3 years ago), but did write significant parts of the firewall and also manage the firewall, WAF, and DDoS protection teams.

kxrm• 3 years ago

Apparently you can get around it using other white-space characters, here I am using a horizontal tab between nc and the IP.

    nc 192.168.1.100 8000

cwillu2• 3 years ago

For reference, the id in the block message is “Cloudflare Ray ID: 845543eb88d461ee”

ifeja• 3 years ago

nc domain.com

6865• 3 years ago

aoeaoe

kevincox3• 3 years ago

I'm sure that there is a huge chart on their Cloudflare dashboard about how many attacks were blocked! This is one thing that gets me, all of the reporting Cloudflare provides treats every block as a huge success. Nothing to help identify actually attacks vs false positives. Let also false positives that would have actually has a negative effect on the application behind the WAF.

deno• 3 years ago

Cloudflare is just dumb, they block XHR requests randomly in the same session they’ve already challenged breaking websites in not quite obvious ways and have been doing that for as long as I can remember. Trying to do anything on for example Montana's SOS BIZ portal takes a lot of patience. They’re like TSA of the Internet but at least with TSA you can pay for a fast pass.

ForestCritter• 3 years ago

cloudflare blocked me from signing in to my petflow account to buy cat food. It was in an endless verification loop. Awhile back it did the same with my paid crunchyroll subscription. I don't code, I have a very ordinary setup with a well known browser. Apparently cloudflare now owns our access to the internet and can block whom it pleases, when it pleases, no recourse. The internet is soon to be available only to those who fit cloudflare's criteria, whatever that may be, as long as companies keep buying in to the third party control.

1vuio0pswjnm7• 3 years ago

"If you're seeing this message, that means JavaScript has been disabled on your browser, please enable JS to make Imgur work."

Well, it's a client that has no Javascript engine.

There is no need to use Javascript. This works for me just fine:

https://i.imgur.com/YtepoDbh.jpg

EasyMark3• 3 years ago

ever since imgur started hating VPN users, I hate it back.

flexagoon• 3 years ago

You can use Rimgo, an alternative frontend for Imgur

Just replace "i.imgur.com" with one of the Rimgo instances:

https://rimgo.codeberg.page/

(The most popular ones are usually rate-limited though)

Am4TIfIsER0ppos• 3 years ago

How can Cloudflare be reading anything you send? Your connection is encrypted to HN's server, is it not? They don't MITM everyone's connection.

Ask HN: Does Cloudflare block HN comments if you have code blocks in a reply?

Related Stories