Ouch

http://www.antipope.org/charlie/blog-static/2010/11/ouch.html

by MaysonL • 16 years ago

123 32 16 years ago

CharlesPal• 16 years ago

I've also experienced similar spikes before. It's important to added that everything will slow way down... that includes SSH access.

Even if you think "I'll just jump in there and correct the issues if/when the time comes" you may not be able to if SSH access slows down to a crawl.

Like the post says, the best course of action is to be prepared ahead of time and not trying to make tweaks while the traffic spike is happening.

rythie• 16 years ago

Or put your blog on posterous or blogger.

cstross2• 16 years ago

There are several reasons my blog isn't on blogger.

First: I like to own my own IP. And I don't want to be vulnerable to a spurious takedown notice if some random griefer decides to take aim at me.

Second: blogger has some hard limits -- official list here: http://www.google.com/support/blogger/bin/answer.py?hl=en... -- and there are anecdotal reports that there is a limit on how many blog entries you can have.

I seriously considered Typepad, but once I worked out how much it would cost it was cheaper to stick with a dedicated colo server (hint: there's more than one blog on the system I run, and there's also a mailman server and some other stuff).

Also: I've been running on a Linux colo since 1996/7, and I really don't feel the need to take a couple of weeks off work to re-jig my entire online presence. This is the first time I've ever had anything like this load level. If it starts happening regularly, I'll reconsider my strategy. Until then, it's a one-off.

patio11• 16 years ago

And, for God's sakes, turn off Apache KeepAlive. It kills more blogs than cancer.

andfarm• 16 years ago

Or switch to nginx, which eats keepalives like candy.

lopezka• 16 years ago

Or use the event MPM.

mhd• 16 years ago

What's the current name for that? Do we still call it "slashdot effect"?

wazoox• 16 years ago

Sometimes it's the reddit effect, or similar. However "slashdotted" should remain understandable to anyone.

RyanMcGreal• 16 years ago

Though in this particular case, the site seems to have been Gaimaned.

metageek• 16 years ago

Isn't the general term "flash crowd"?

RiderOfGiraffes• 16 years ago

Please could you make the submitted title a little more informative? Perhaps:

    Ouch!  (First hand experience of a massive load spike)

Thanks.

xiongchiamiov• 16 years ago

> 4. Tune Apache now — you won't get a chance to do it when the stampede arrives!

Or take the easy route and ditch Apache.

I'm not saying that Apache can't be great (people who know much more than me use it), but if you need to quickly serve up a ton of requests without bothering to tune this and tweak that, there are better choices, in particular for static pages.

Cherokee's my preferred, but lighttpd, Nginx, thttpd, etc. are all good, too.

mechanical_fish• 16 years ago

Ditching Apache is only "easy" if (a) your blog software isn't designed around Apache-isms like .htaccess and mod_rewrite, and (b) you have time to find and comprehend the documentation for your alternative server.

Nginx is great, for example, but the docs are primitive compared to those for Apache, which has been around forever and which has spawned thousands of hints and snippets all over the web.

EDIT: At this point I should probably say out loud that for accelerating static pages it might well be simpler to get a reverse proxy, e.g. Varnish, running in front of Apache than to actually ditch Apache.

wiredfool3• 16 years ago

Is varnish a drop in fix for something like this while everything is going to hell, or is there some monkeying that has to be done to make it play nice with movable type?

ErrantX• 16 years ago

I've moved almost exclusively to Nginx, once you get your head round how to do the "Apache-isms" (great potential neologism that :D) in Nginx configurations it becomes second nature.

Cherokee is a possible alternative too; that has some default configs for Wordpress etc. that work well. But for anything custom I found it a nightmare to get "just so".

Nginx is a neat alternative to Varnish as well; for example I'm having great success sitting it in front of a Cherokee site (serving the http) to provide SSL :)

pronoiac4• 16 years ago

On one memory-constrained server, I have nginx reverse proxying to Apache, so the Apache-isms are not an immediate problem. I have nginx bypassing Apache for some static files. Having a ton of slow connections was a problem for Apache, but not for nginx.

cstross2• 16 years ago

Let me add: I am paid to write books, not paid to ride herd on servers. Apache's what I know from days of yore; switching to a new platform means taking an opportunity cost hit.

Having said that, I'm looking at Varnish (once the stampede dies down) ...

wwortiz• 16 years ago

Cherokee is actually pretty good at following .htaccess and mod_rewrite for the big blogging/cms engines (like drupal, wordpress). It is much easier to configure, though perhaps a bit more tedious, as it is all done through the web gui.

Other than that Apache is pretty much king with compatibility and nginx is pretty hard to get to similar qualities off the bat.

But if you have the space something like a reverse proxy for Apache with sane times set would probably be the better option.

I use cherokee on VPS's though and it works great.

param• 16 years ago

" I don't have an exact breakdown of what just happened, because my logfile analyser is still churning, from 2am last night.That's because the Apache logfile today is over 500Mb it's normally about 150Kb. " The post is timed at 11:31.

I think another lesson learned is that the log analyzer needs to be analyzed.

pronoiac4• 16 years ago

If the server's under really hard load - memory or I/O or both, with the swap thrashing - I'd be unsurprised.

I've never checked to see how my stat analyzers fare on half a gig of logs.

pero• 16 years ago

I understand this is HN and all, but has anyone even bothered to take a peak at the EXTRAORDINARY content that was behind the spike?

In a nutshell, it is a transcript of a British parliamentary session in which the speaker claims a shadowy yet totally legitimate organization--"Foundation X"--is willing to donate billions to the British government no strings attached. It is extremely engrossing.

wiredfool3• 16 years ago

Yep: http://news.ycombinator.com/item?id=1864932

mcantelon• 16 years ago

Here's video of the speech (starting at hour 2, 34 minutes). Exceedingly odd. http://bbc.in/9qS59B

zoomzoom• 16 years ago

I still don't know if this is to be taken seriously. What exactly could be going on?

fredoliveira• 16 years ago

Sky news is reporting on it too, and while it looks like it could be a work of fiction (it sure reads like one - Dan Brown comes to mind), it apparently isn't.

http://news.sky.com/skynews/Home/Politics/Foundation-X-Lord-...

Ouch

Related Stories