Nginx doesn't suck at SSL after all

http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/ur

by seiji • 15 years ago

288 107 15 years ago

thirsteh5• 15 years ago

So Nginx got unwarranted hate for having the most secure defaults. That sucks. I hope the user nginxorg -- whom I assume is Igor Sysoev -- who dropped by the previous thread (http://news.ycombinator.com/item?id=2752136), sees this post too.

Either way -- based on his attitude in the first post, I'm really surprised by how Matt owned up and did his homework for this one. (He should have done it from the beginning, of course, but none of the people bashing him in the previous thread actually provided anything to support what they were saying.)

tptacek23• 15 years ago

It's not the "best/most secure". All the other servers can trivially enable EDH as well; it's OpenSSL that implements it, not nginx. Reasonable people can disagree on what the right default is, but plenty of financial institutions have made the studied choice not to enable it.

thirsteh5• 15 years ago

Fair point. Clarifying...

schleyfox• 15 years ago

Plenty of financial institutions have also made the studied choice to limit the length and types of characters I can use in my password.

masklinn• 15 years ago

> All the other servers can trivially enable EDH as well

Unless I'm reading OP wrong, that's not the case for the servers he uses for his tests: Stud can't enable it at all:

> stud doesn't have at all.

and in stunnel you have to compile it in for support, it's not just "not enabled by default", it's not compiled in:

> stunnel has it as a compile time/certificate configurable option.

alnayyir• 15 years ago

Well at least he followed up. Most people don't bother to correct their mistakes.

cbetz• 15 years ago

I can't tell if this is an apology or a non-apology. It seems to have elements of both.

Clearly the moral of the story is: "Don't claim that X sucks unless you are are damn sure".

Saying something sucks is fightin' words. Don't expect to people be nice if you are wrong.

tptacek23• 15 years ago

Why would someone apologize for writing an informative blog post? I'm glad he wrote both, even if he had to walk the first one back a bit.

SnowLprd4• 15 years ago

I found both posts informative, and yet I'm with cbetz on this one. The issue isn't whether Matt's first post was informative but is instead whether it was fair|wise|necessary to say "Nginx sucks at SSL" instead of, say, "My initial testing, which needs to be investigated further, is showing Nginx SSL performance lower than other alternatives." The first headline is more likely to grab folks' attention, which is probably why Matt chose that headline. It is probably that choice that cbetz finds objectionable, and if so, I agree with him.

clintjhill2• 15 years ago

I don't know that it required an apology or if he was even really wrong in the first post.

The result was his digging into it to figure out the default config included a computationally expensive setup.

That's a debatable thing. Not necessarily wrong.

SnowLprd4• 15 years ago

As far as I can tell, saying "Nginx sucks at SSL" is actually not debatable and is indeed just plain wrong. Yes, the default SSL config uses a cipher that is, relatively speaking, more computationally expensive than others. That doesn't mean Nginx "sucks at SSL." Don't like the default cipher? Great -- change it.

I found the posts to be informative, but I get the impression that the "Nginx sucks at SSL" linkbait headline is what's rubbing folks the wrong way.

pyre3• 15 years ago

  > Clearly the moral of the story is: "Don't claim that
  > X sucks unless you are are damn sure".

More like: "If you say something that upsets people, they will spend a lot of time and effort to display their ignorance in an effort to prove you wrong -- as if you had insulted their very being -- even though you just made and honest mistake." I read the original article, and I didn't see it as an attack on Nginx, or Nginx's SSL support. He ran some benchmarks and said, "Wow, those numbers suck."

  > Don't expect to people be nice if you are wrong.

I read the HN comments here and on the original post, and I don't remember anyone saying anything about cyphers. The people that were 'not nice' were also wrong. Would that excuse the author of the post, from turning around and being 'not nice' to those people because they are wrong?

beachaccount3• 15 years ago

I ran this against the slowest SSL website I know of. This site absolutely kills my phone web browser and I've been wondering about this problem for years. The site: manager.skype.com. The result? DHE-RSA-AES256-SHA. No wonder! Fix this guys!

In regards to this post, if this is the default configuration of Nginx then I agree that Nginx sucks. This is not a good default configuration for the Internet.

bdonlan• 15 years ago

There's nothing wrong with using DHE algorithms, particularly if you're going to be transferring financial secrets around. If your phone can't keep up, well, then it probably should have a better entropy generator.

tptacek23• 15 years ago

Are you sure the problem is entropy generation and not just extra bignum math? Also: there are plenty of major financial apps that are not allowed to use DHE, because DHE makes it impossible for the provider to monitor its own connections ("conventional" SSL/TLS allows for middleboxes that monitor and archive sessions by holding a copy of the server's RSA key).

beachaccount3• 15 years ago

It maxes has been maxing out my laptop CPU for an inordinate amount of time for years as well. These "financial secrets" are emailed in the clear after every transaction, so your point is moot.

Burgerbrain: PayPal.com, CitiCards.com, 2checkout.com, and many payment processors all use RC4-SHA or AES256-SHA. You are wrong.

tptacek23• 15 years ago

In case you're wondering what "Perfect Forward Secrecy" is: SSL/TLS, like most protocols, uses (expensive, dangerous) RSA to exchange (cheap, simple) AES or RC4 session keys; bulk data is encrypted with session key.

In the normal protocol, if you lose the RSA key, an attacker can retroactively decrypt the session keys, which are protected only by that same RSA key.

In ephemeral DH mode, instead of encrypting a session key with RSA, both sides run the Diffie Hellman protocol to exchange a key†. DH allows two unrelated parties who share no secrets to exchange a secret in public; it's kind of magical. But it's also trivial to man-in-the-middle. To get around that problem, ephemeral Diffie Hellman mode in SSL/TLS signs the DH exchange with the RSA key.

The win here is that losing the RSA key now only allows you to MITM future SSL/TLS connections. This is still a disaster, but it does not allow you to retroactively unwind previous DH exchanges and decrypt earlier captured sessions.

† DH is unbelievably simple; go read the Wikipedia page.

thirsteh5• 15 years ago

"Ephemeral Diffie Hellman"

It sounds magical too.

shrikant• 15 years ago

Diffie-Hellman and Boyer-Moore are two algos that completely blew me away with their simplicity when I first encountered some theory on them.

http://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exch...

http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_sear...

alecco• 15 years ago

It could be a really nice blog post. A lot of people get lost on wikipedia pages like that.

clintjhill2• 15 years ago

It's great to follow up. Especially with such a particular detail.

It's also good to have thick skin. HN can be aggressive. But for good reason. I'd be willing to bet this tiny sting will result in more rigor in the future. I know it has worked that way for me.

ccollins• 15 years ago

From the article, to find out what your website is doing:

openssl s_client -host HOSTNAME -port 443

I ran this for my own website and a few bigger websites

  openssl s_client -host www.gusta.com -port 443 (My site, hosted on Heroku)
  Cipher    : DHE-RSA-AES256-SHA
  
  openssl s_client -host www.google.com -port 443  
  Cipher    : RC4-SHA 
  
  openssl s_client -host www.airbnb.com -port 443  
  Cipher    : AES256-SHA
  
  openssl s_client -host www.facebook.com -port 443
  Cipher    : RC4-MD5
  
  openssl s_client -host www.paypal.com -port 443
  Cipher    : AES256-SHA

  openssl s_client -host www.amazon.com -port 443
  Cipher    : RC4-MD5

rythie• 15 years ago

Presumably Amazon, Facebook and Google are using RC4 for speed reasons, though it's not really thought to be secure anymore.

paulitex• 15 years ago

Is that so?

openssl s_client -host online.citibank.com -port 443

Cipher : RC4-MD5

openssl s_client -host www.bankofamerica.com -port 443

Cipher : RC4-MD5

tptacek23• 15 years ago

RC4 is fine in SSL/TLS.

Nobody likes it, but until relatively recently, the AES ciphersuites were all CBC mode, which means they burned a couple bytes of padding for every record.

RC4 is also faster than AES, which was, until very recently, an issue for server performance.

We have AES-CTR ciphersuites now, but I'm not sure how widely deployed they are.

benblack8• 15 years ago

While I would certainly prefer to see the major players leading the way in adoption of AES by default, RC4-MD5 persists for at least 2 reasons:

1) Habit

2) As implemented/deployed in SSL, it still provides some security

RC4 has gotten a bad reputation in large part because of its poor application in WEP that resulted in keys being rapidly recovered by sniffing traffic. The Wikipedia entry is a good place to start http://en.wikipedia.org/wiki/RC4#Security (& numerous references for the original papers/pubs cracking various bits of RC4). The RSA response to RC4 concerns (from WEP) is worth reading, as well http://www.rsa.com/rsalabs/node.asp?id=2009 .

kronusaturn• 15 years ago

RC4 has known weaknesses, but it's still extremely difficult to crack when implemented correctly. Attackers want to go for your weakest point, which is almost certainly not RC4; typically it's far more difficult to crack RC4 than to steal your data by finding a SQL injection or buffer overflow, or simply breaking into your building.

WestCoastJustin• 15 years ago

changes: slow ssl encryption ciphers on by default, keepalive

before: nginx (ssl) -> haproxy: 90 requests per second

after: nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second

tlrobinson• 15 years ago

"Final feeling: Twitter is better than HN in all social dimensions of engagement, kindness, and authenticity."

Ouch.

thirsteh5• 15 years ago

Yeah, well, arrogantly complaining about people calling you out on unfounded speculation doesn't really work when you were originally quite arrogant in that speculation.

rfugger• 15 years ago

This is interesting to me. Maybe it's because twitter places much more emphasis on identity than HN comments? Twitter users are also more likely to be using that service to market their personal brand, rather than just to share whatever they're thinking, so are probably much more careful about what they say.

nandemo• 15 years ago

Twitter is better than HN? That's like comparing apples to a fish market.

michaelbuckbee• 15 years ago

There is some selection bias and context that the OP had via Twitter (people following already kind of know him and his work, etc.) whereas HN evaluates his work solely on this one post.

AltIvan• 15 years ago

Yeah, the correct stament should be something like:

Final feeling: My long time Twitter Followers & Friends are more engagement and kind than the anonymous programmers on HN.

ynniv• 15 years ago

Ah, if only we had an occasional second upvote! This is far more useful than the original post, which was already well above average. If you are deploying nginx with SSL, you need to know about the configuration details in the article.

grandalf• 15 years ago

Any serious article ripping on the performance of something should at least link to the config file used.

jeremyw• 15 years ago

Unlike the above post, this fellow actually did some broad cipher testing (http://zombe.es/post/4078724716/openssl-cipher-selection), particularly around AESNI instructions in recent Intel chips.

With AESNI, use AES-128, AES-256, RC4-SHA, CAMELLIA-128. Without AESNI, use RC4-SHA, AES-128, AES-256, CAMELLIA-128.

In nginx, this looks like:

  # (wo/AESNI): ssl_ciphers RC4:AES128-SHA:AES:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2
  # (w/AESNI):  ssl_ciphers AES128-SHA:AES:RC4:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2

You eliminate weak ciphers. You retain RC4 for compatibility and speed. You order by performance. (Note that AES-128 is still ranked as secure through 2030 [at least]. You don't need to prefer AES-256.)

epi0Bauqu• 15 years ago

So what should ssl_ciphers be? Can't you just move that one to the end somehow?

benblack8• 15 years ago

An article about configuring SSL that doesn't 1) discuss trade-offs of security vs. resource consumption, 2) how to figure out your performance requirements, and 3) indicate the author really understands implications of decisions about crypto is an article you should probably disregard. Modern CPUs are so ridiculously good at crypto, and most sites have such ridiculously low connection rates, that optimizing for maximum performance at the expense of security is a fool's game in most cases. Instead, focus on measuring your real performance requirements first, and things like sane configuration of SSL, for example by explicitly listing ciphers instead of using the impenetrable +aNULL:-yourMom syntax.

Here's my vintage code for scanning SSL configs: https://github.com/b/tlscollect

Here are a couple of must read posts from someone who really knows his SSL business:

http://www.imperialviolet.org/2010/06/25/overclocking-ssl.ht...

http://www.imperialviolet.org/2011/02/06/stillinexpensive.ht...

It's great to learn.

Lil' B

tptacek23• 15 years ago

Does any of this have anything to do with Matt's post? Adam's first post says the same thing Matt's does: DHE is expensive.

The "tradeoff" in security vs. performance you're referring to irrelevant to almost everyone building on nginx. If you've lost your RSA key, you are well and truly fucked. DHE is interesting, but sniping at people for not using it (in your case, implicitly) is unfair.

benblack8• 15 years ago

Adam's post is rather more thorough and nuanced, which makes sense since he actually understands SSL and benchmarking. While you might summarize them both as "DHE is expensive", I don't know why you would. Here is each post on DHE:

Adam - "However, with a pure RSA ciphersuite, an attacker can record traffic, crack (or steal) your private key at will and decrypt the traffic retrospectively, so consider your needs."

Matt - "Unfortunately, it also includes a very computationally intensive cipher using an ephemeral Diffie-Hellman exchange for PFS. Sounds scary already, doesn't it? ... The problem cipher is DHE-RSA-AES256-SHA [b]."

The first is factual and straightforward. The second is muddled and clearly skewed towards blindly disabling DHE. I believe we are in agreement that it is irrelevant to almost everyone building on nginx: their connection rates are so low they will not notice the overhead introduced by DHE.

I am sniping at enthusiastic ignorance and encouraging others to behave similarly. I hope that is all quite clear now.

Hugs and kisses, Lil' B

benblack8• 15 years ago

On a related topic, these are both your comments on this post:

"The win here is that losing the RSA key now only allows you to MITM future SSL/TLS connections. This is still a disaster, but it does not allow you to retroactively unwind previous DH exchanges and decrypt earlier captured sessions."

"If you've lost your RSA key, you are well and truly fucked."

Thanks for clearing that up!

Make love not war,

Lil' B

SnowLprd4• 15 years ago

Speaking of fairness, how do you conclude that Ben is sniping at people for not using DHE? I've re-read his comment multiple times, and I don't see that in there at all -- explicitly or implicitly.

lanstein• 15 years ago

keywords in footer: nginx, openssl, ciphers, DHE-RSA-AES256-SHA bad, AES256-SHA good, hn, twitter, glee soundtrack

getsat• 15 years ago

Possible SEO play? Maybe not, considering his site is almost completely lacking on-page SEO. Perhaps an accidental copy/paste?

sthatipamala• 15 years ago

Look at his other posts. He just has joke keywords in all his posts.

newman3144• 15 years ago

Guess I was right about the cipher used.

http://news.ycombinator.com/item?id=2753903

ldar15• 15 years ago

So, are these the new numbers? I copied the numbers from the original and the new post:

  haproxy direct: 6,000 requests per second
  stunnel -> haproxy: 430 requests per second
  (OLD) nginx (ssl) -> haproxy: 90 requests per second
  nginx (AES256-SHA) -> haproxy: 1300 requests per second
  nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second

Did other things change or is nginx more than twice as fast as the next best solution?

giberson• 15 years ago

I use a similar directive in my apache2 configuration. Would I see an performance improvement in removing the DH option from the cipher suite? Or is this only directly related to ngix and how it implements its ssl protocol?

Secondly, by removing the DH method do I restrict any browsers from connecting my site? Ie, are their any browsers, or security settings on browsers that prevent the site from being trusted if DH isn't available?

blumentopf• 15 years ago

It's an openssl configuration option, so it does affect performance for Apache with mod_ssl. It's not specific to nginx.

If you do not allow DH ciphers, you'll probably just lose users who deliberately configure their software to use only strong ciphers.

What you can do is put DH ciphers at the end of the list. That way, weaker ciphers are preferred but you're still supporting strong ciphers.

dfc• 15 years ago

This is why security is such a wierd/nice/confusing/irratating line of work to be in. Newsflash SSL is not a one size fits all secure you against anything technology. I did not see the original article so I won't pretend that I knew the answer ahead of time. I just hope that I did not accept SSL as being a onesize fits all completely uniform technical conmponent.

There is a Dave Chapelle joke about cops sprinkling crack-cocaine over a crime scene in order to make the case quick and easy. Too many developers trest SSL like magic pixie dust for security.

Or as ptacek says "thanks in advance for putting my kids through college."

alexkon• 15 years ago

If you are exploring and testing SSL, the SSL Labs tools come in handy. For instance, see what Gmail and Github are doing:

https://www.ssllabs.com/ssldb/analyze.html?d=mail.google.com

https://www.ssllabs.com/ssldb/analyze.html?d=github.com

mmaunder• 15 years ago

Thanks to Matt and everyone who contributed to figuring this out - and then figuring it out again. Comments on the nginx mailing list (which I encourage you to subscribe to if you're a user):

http://forum.nginx.org/read.php?2,212229

Nginx doesn't suck at SSL after all

Related Stories