Fighting Back After Hacker News Took Down My Site

http://www.plainlystated.com/2011/07/fighting-back-after-hacker-news-took-down-my-site/

by rellik • 15 years ago

45 51 15 years ago

js2• 15 years ago

http://www.google.com/search?q=site:news.ycombinator.com+pat...

I will also quote myself: "I first learned to disable Apache KeepAlives in 1998. Yes, 1998. It's disheartening that Apache still ships with it enabled by default. It has always allowed a relatively small number of lingering clients to completely DoS your server."

tluyben2• 15 years ago

Yeah I hear tons of people saying 'check if KeepAlive if on, your site will be much faster if it is'. Not the smartest general remark about configuration of web servers to make :)

spydum3• 15 years ago

There is nothing wrong with keepalives if you give them proper timeout and request limit values. They definitely cut down tcp/ip setup overhead, and improve page render times. Just don't let the client be too greedy about how long they hold that connection.

pornel• 15 years ago

It's not problem of web servers in general, but specifically Apache.

HTTP keep-alive connections are faster (saves 3-way handshake on subsequent requests and allows pipelining).

The problem is that Apache implements them in a horribly inefficient manner (thread/process and all its memory kept in use just to hold on to a socket).

If you use nginx, lighttpd or shield Apache with haproxy/varnish, then you can easily have keep-alive enabled and clients will see better performance.

wheels4• 15 years ago

This is the most critical bit. We run our website (totally disconnected from our webservices, naturally) on a 256 MB VPS and frequently handle top HN stories on our wordpress blog (which runs on a host with several of our other PHP / static subsites).

Key points:

• Make sure that wordpress supercaching is on. You can verify this by looking at the last line of the HTML that comes back which has a timestamp for when it was generated.

• Turn off KeepAlives.

• Set MaxClients to 8.

• Use monit to check to make sure it can connect and restart apache with a kill -9 if it can't. (This is optional, but helps if you have some random thing that ends up taking a very long time to execute and eats up connections.)

With that in place even a much smaller server can easily handle a HN top story without breaking a sweat.

rellik8• 15 years ago

Do you use something automated to ensure things are cached? I use a different page caching plugin (W3 Total Cache), but it also puts the comment in the HTML.

peterwwillis2• 15 years ago

> Set MaxClients to 8.

LOL what? what's your mpm? did you enable this to keep the backend from blowing up from too many queries? surely you can handle more than 8 connections at a time. is this the proxy layer, and if so are you using web caching on top of wordpress caching?

if monit is restarting apache every time it can't connect (i hope you have a long timeout) you're denying service to a lot of people. connections are supposed to queue so they don't get dropped.

static_cast• 15 years ago

If you use ngix + php-fpm: the nginx fastcgi_cache module http://wiki.nginx.org/HttpFcgiModule#fastcgi_cache is really neat and a lot faster than the full-page caching plugins. I also found that an object cache does improve performance a lot for pages that are generated from wordpress, saving tons of database queries.

rs• 15 years ago

Nginx has proved extremely useful and capable especially in memory constrained virtual environment. It effectively gives an easy way to move from a synchronous web server to an asynchronous one.

teadrinker3• 15 years ago

BETTER SOLUTION: Get on the cloud. 15 minutes to upgrade your server when load gets high and billed by the hour at the ram level you're 'using. Theres very little reason to stay on traditional hosting if you're running anything close to a serious startup.

rellik8• 15 years ago

While the cloud does enable you to scale your resources on demand, you still have to address the same issues I had to deal with (how to efficiently use those resources). Plus, moving to a mutli-server setup involves additional complexity in the architecture of the system, which I'd rather not mess with unless I have to :)

teadrinker3• 15 years ago

Multi-server? Nope, single. Same as you have now, just with added flexibility on ram with 2 clicks I can handle a huge server load with ease, then drop back when done. That way I don't waste time on low level optimization tasks and can keep to what I do best. Building things.

Seriously, there's no reason to stay on a traditional style server setup and you'll never go back once you've tried it. It can be pricey once you ramp up, but give it a shot as you can literally pay by the hour while you play.

sc68cal2• 15 years ago

What about isolating the popular URL and temporarily serving a static version of the page?

rellik8• 15 years ago

That's what the page caching plugin does. It puts a static file in nginx/apache's search path, so it never hits PHP

sc68cal2• 15 years ago

Which plugin did you use? WP Super Cache? I'd like to take a look through the code.

bradleyland3• 15 years ago

I cringed a little bit when I read this: "...none get enough traffic for me to have made caching or performance tuning a huge priority".

I don't mean to level criticism solely against the post author for this, because he is not alone, but I find this mentality inexcusable. This is especially true if you're using WordPress, for which there are a multitude of caching plug-ins. All of which will offer orders of magnitude better performance than allowing the blogging engine to build the page from scratch every time.

In other words, caching is always a priority. It's not an add-on or an after-thought, it should be part of your design.

Consider a scenario where someone asks you to add up three arbitrary numbers:

123 + 456 + 789

Now, imagine you type those in to a calculator to add them up. It takes you a few seconds. The output is 1368. You can easily remember this number, so the next time someone asks you what the result of 123 + 456 + 789 is, you can just say 1368. Not caching is like keying the numbers in to a calculator every time, rather than just relying on your memory.

I know this is a rudimentary example, and I know that most people running a WP blog probably know how caching works, but even if you don't "need" it today, why would you leave your blog set up so that it's constantly re-building content that can be cached by simply installing a WP plug-in?

I implore you. Make caching the second thing you do (after security) when setting up or building your web app.

rellik8• 15 years ago

Definitely agree! For me, the counter-point is that there's tremendous value in getting the Minimal Viable Product out the door, and iteratively improving it (the post in question was only the 2nd post after moving off a hosted blog). Projects that I want to be perfect before release usually don't make it out the door.

Plus, I'd grown lazy from my lack of traffic :)

benmills2• 15 years ago

I agree with you when you're taking about web apps, but what about personal blogs?

If I just want to set up a personal blog that I don't plan on promoting or spending a lot of time on I wouldn't want to spend any extra time setting up things that don't help me with my primary goal, writing blog posts.

bradleyland3• 15 years ago

I totally get that. I'm beyond the phase in my life where I enjoy tinkering with blogging software settings. I'm far more interested in engaging in the conversation. But here's the counter argument: it takes a sum total of 60 seconds to implement. If you're using WordPress to host your blog, installing caching is beneath trivial. You click "Add New" under the Plugins menu, then search for WP Super Cache and click install. The only thing remaining is to turn it on.

If you're not using WP, you might use something like Tumblr or Posterous. In that case, caching isn't your problem. If you rolled your own blogging software, well, you already violated the principle that blogging is your primary goal.

lukeschlather• 15 years ago

That's why I went with static HTML + Jekyll. I only deal with the moving parts when I'm uploading a blog post.

It also increases the barrier of entry to posting something, so I'm less likely to post something inadequate.

voidfiles• 15 years ago

I agree. Also why not use a full page cache, WP super cache allows you to generate the HTML version of your site and then serve the static HTML you don't even need to spin up the PHP.

scott_s4• 15 years ago

If they don't have the load, then it's just not at the top of people's priority stack. This doesn't seem like a terrible thing to me. It's not that caching and performance tuning isn't important, it's just that for people who don't have high load, they have many things that are more important to deal with.

Lewisham• 15 years ago

I feel that Wordpress should be smarter about this. You shouldn't need to download a plugin to make sure your site doesn't choke.

However, lots of people get confused by caching plugins when they're editing templates and things.

Idea: Wordpress caches pages. It checks to see when its template files were last touched. If they were touched after the cache, it regenerates. Otherwise, the cache is served.

You can have that for free, Wordpress.

Vivtek• 15 years ago

* but I find this mentality inexcusable.*

I find that mentality kinda odd.

wheels4• 15 years ago

Note that caching didn't and wouldn't have significantly helped in this case. It was an inappropriate Apache configuration. No amount of caching will stop a server from falling over when it's running far more Apache processes than it has RAM for. While caching is part of the solution, it's not the high bit (lowering MaxClients and properly configuring KeepAlives is).

jules• 15 years ago

Why doesn't WordPress cache by default?

jasonlotito• 15 years ago

The TL;DR: Persistent caching isn’t needed for most sites and is a pain to get working in all hosting environments. Instead WordPress makes it really easy to install a caching plugin if you need it.

http://www.viper007bond.com/2010/08/10/why-wordpress-doesnt-...

ck2• 15 years ago

Who run WordPress without a page cache these days?

Just look at the query count even without plugins.

Add a few plugins and it's a total mess.

sciurus• 15 years ago

An important point that was missed is how was he running the php and rails apps? mod_php? mod_rack? cgi? fastcgi? Proxying to an application server? The answer to that is going to greatly affect how well Apache scales when you're memory-constrained.

rellik8• 15 years ago

apache: rails & sinatra were on passenger. php was mod_php. nodejs was proxied.

nginx: rails & sinatra are on passenger. php is php-fpm. nodejs is proxied

etherealG3• 15 years ago

thanks for this, useful. probably worth amending in the original post :)

anigbrowl• 15 years ago

What a strange headline. Who is he fighting back against - Hacker News? Traitorous Wordpress? Himself? If I needed a plumber I wouldn't want one that talked about fighting back against the water. Nor would I want an architect that talked of fighting back against gravity. The language of confrontation is inappropriate here, since the problem as described stems from a failure to spend time learning or configuring Apache. This might seem trivial, but to me it suggests fundamental flaw in the approach to the problem, which probably increased the time needed to fix it.

Fighting Back After Hacker News Took Down My Site

Related Stories