Cause of YC/HN outage discovered

http://www.archub.org/ycoutage.txt

by pg • 16 years ago

172 102 16 years ago

archub.org

stuntgoat• 16 years ago

So your network traffic is guilty and sentenced until proven innocent.

a2tech• 16 years ago

Thats some good customer service there Lou. (Said in Chief Wiggims voice)

Seriously-they couldn't just email the account holder?

pg4• 16 years ago

Things should be back to normal now. Trevor just moved www.ycombinator.com to Slicehost, and I just told HN to refer to static stuff there instead of serving it locally.

Interesting how easy it is to move your whole web site. Good customer service is important when users can switch so easily.

othello• 16 years ago

Thank you ! Procrastination becomes much harder when HN does not provide a continuous flow of interesting distraction.

nfnaaron2• 16 years ago

Hacker News: We get shit done, so you don't have to.

axod3• 16 years ago

>> "Things should be back to normal now."

Hopefully that's the normal of months ago when loading comments/threads etc was quick?

Still taking 10-20 seconds to load most pages. One thing I haven't tried yet is just creating a fresh account - maybe mine just has too much associated with it (Maybe I comment too much etc).

edit: ah I see it only affected static content. Shame :/

Locke1689• 16 years ago

I've had some speed issues too. My account isn't tiny, but it is a 12th of your upvote count.

carbocation• 16 years ago

Any particular reason for Slicehost as opposed to any of the other similar options (Linode, prgmr)? I'm not affiliated with any, though I run my site on VPSes from Linode. I was convinced by a Dec 2009 post by an HNer, uggedal: http://journal.uggedal.com/vps-performance-comparison

jyothi• 16 years ago

Me too. That post was detailed & fair. On slicehost I had a 256MB slice which I upgraded to 512MB slice to host 2 small websites. After the post got a 360MB linode VPS instead and am very happy with it.

Edit: Slicehost help articles & resources, well organised & exhaustive is a plus though.

projectileboy• 16 years ago

Heh heh... Totally clueless companies are so cute sometimes; like watching a toddler try to eat chocolate cake.

fletchowns• 16 years ago

This site was on a shared hosting box?

And of course they are going to disable it without warning if it's causing problems for all the other customers on that box...

pg4• 16 years ago

Not this site, www.

There was no sudden spike in traffic. If they'd bothered to check the logs they'd have found that the load, whatever it was, was no higher than it had been.

wvenable• 16 years ago

So were they just jerking you around then?

guelo• 16 years ago

I'd imagine what happened was one of their other customers finally decided to call support to see why their site was so slow.

archon810• 16 years ago

This is almost as bad as the reason for my server's downtime recently: http://beerpla.net/wp-content/uploads/img_2958.JPG

My server literally had its plug pulled.

nkassis• 16 years ago

Are we at the point were we have to deal with robotic janitors too?

Geez, and I thought that watching the janitors while the work was over the top, now I gotta watch the rumba too :(

bshep• 16 years ago

"We just figured out what caused the problem. Apparently Pair Networks' procedure for requesting that users upgrade to a dedicated server is to shut down their site without warning...."

I think you should upgrade to a more "dedicated provider" rather than a "dedicated server"...

But seriously, not even a warning?

eli4• 16 years ago

I think this is pretty common. When Joyent shut down my shared account way back when, I discovered it in a comment in my httpd.conf

swombat• 16 years ago

Shared hosting is generally sucky for anything remotely successful. I'm amazed you got away with storing your static assets there so long! When I got 100k hits in a day on my first blog, Dreamhost promptly shut it down without warning (in the middle of a slashdotting!)

PostOnce• 16 years ago

Dreamhost? Their main advertising point is UNLIMITED TRAFFIC!!! I guess I'll scratch them off the list of potential hosts.

jackowayed• 16 years ago

No, they they're careful to say unlimited bandwith (and storage and such), but not unlimited traffic. When I was looking around dreamhost docs/FAQs/something either right before or right after signing up, it did say somewhere that they might ask you to move up to their VPS if you're using too much CPU, which could happen just from a whole bunch of static hits.

It's nice for your hosting all of your little sites and getting them up in basically 0 time, even if you need wordpress or something. The panel's pretty nice. I'd at least sign up when they have one of their crazy deals (I got a year of hosting with a free domain for $9.something, which basically means you pay for the domain and get the hosting for free. It's a good way to give them a try with the main cost being that they'll suck you in and get you paying $9/month after your cheap price expires.)

axod3• 16 years ago

Nothing is unlimited. It's just sales tactics.

froo• 16 years ago

I actually don't mind Dreamhost for very targeted uses.

I jumped on a deal at the beginning of this year for ~$9 for 1 year of shared hosting. I now use that account purely for running things like scripts through SSH for doing basic data grabs with wget for further analysis elsewhere.

I've more than got my money's worth and I've yet to see any kind of email message asking me to upgrade.

DrewHintz• 16 years ago

Dreamhost also repeatedly shut down my sites, such as mapwow.com, due to too much traffic. One time they even denied they shut it down -- even though my web directory had been chowned to root, which requires root privilege.

I moved to slicehost and appengine[disclaimer,etc] and have been relatively happy with them.

eli4• 16 years ago

And this is why I don't recommend shared hosting for anyone

bwb7• 16 years ago

Because everyone you know has one of the busiest sites on the net?

eli4• 16 years ago

Anyone can get slashdotted. And my time is valuable -- I'd much rather pay more for better service.

samd• 16 years ago

I'm not sure what is more ridiculous: that they disabled the site without warning or that they were too lazy to look at their logs or do a simple Google search to find out that Hacker News is real site with lots of regular traffic.

VBprogrammer• 16 years ago

What I really loved is that they just demonstrated a fair degree of incompetence to a site used pretty much exclusively by people who are very good potential customers! This site is a wet-dream for their marketing department.

What they should have done is upgrade the website to a dedicated server for free and let that news hit the front page.

nfnaaron2• 16 years ago

Well, maybe upgrade for free, with a time limit, and an email saying "if we don't hear from you in x days, we'll shut 'er down."

rrhyne• 16 years ago

Really, we are all guilty. If we'd just get back to work, HN wouldn't have this problem. ;)

sh1mmer2• 16 years ago

Dear Pair Networks,

I'd like to highlight this lesson in how to loose (or not gain) customers by randomly shutting down technology sites that serve the decision makers you wish to influence.

Tom

P.S. Well done Slicehost.

gojomo3• 16 years ago

150K hits in 30 minutes is 83 hits/second. That's a lot to ask for from a shared-hosting account.

When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual.

The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?)

The Pair message suggests end-of-day logs will be the way to tell for sure.

sh1mmer2• 16 years ago

I was being tongue-in-cheek, but there is a real point here.

As you say I doubt today was unusual for HN, it's the way Pair went from zero to shut-down.

One of the things I like about the way that Joyent operate as a cloud host is that they allow you burst on shared boxes because of those times you need it. At the same time they'll let you know you need to think about buying more resources without just slamming on the brakes.

Pair should do a much better job of noticing a soft limit earlier, so if a heavy traffic day had hit HN PG et al should have been already aware they were overusing their shared hosting and were planning a route out.

neurotech1• 16 years ago

Yes and no.

I doubt it is a HN-Focused tool, as this affected the static content from www.ycombinator.com, and images and CSS are not the focus of bots usually.

Assuming it was not a DoS attack, a smart host should have noticed the traffic load increasing over time, and offered to upgrade to a less-loaded server and recommend a dedicated server.

By disabling the site, they have lost a customer, and lost on a up-sell to a dedicated server.

BrandonM2• 16 years ago

So let's see... you're hosting a website that has gotten large; that is, it's grown to the point that it will need higher-cost services in order to meet demand. You have a chance to add a valuable customer to your client base. How best to handle this?

a) anything

b) except

c) killing their service

bigboote2• 16 years ago

Todays's logs are similar to other days. I guess the traffic is high for a shared account, but I've never noticed it being slow so I didn't think about it.

I would have been happy to respond to an email saying I should upgrade & pay more, but even after a few emails with tech support that option didn't come up.

The only limit they advertise for that class of account is data transfer, and because it's mostly just serving uparrow.gif -type files we're well below that limit.

InclinedPlane3• 16 years ago

The "ask" is not that a shared hosting account support 83 hits/second. It's that the hosting company demonstrate the bare minimum of respect and professionalism in contacting the hostee ASAP when their site is being shut down due to excess traffic. This isn't bitbucket or geocities, dollars are changing hands. If you don't like running a business that makes money then by all means, treat your paying customers like they are a burden to you, they will get the hint and go elsewhere.

axod3• 16 years ago

FWIW, It's probably not worth their effort of working out exactly what websites are using cheap shared hosting and overusing resources.

I don't get shared hosting at all. VPSs are dirt cheap these days.

CGamesPlay2• 16 years ago

VPSes are still shared hosting and are not immune from the problem.

noonespecial• 16 years ago

Customers nothing. "I didn't know what that 'hacker news' thing was or if it should have that much traffic so I shut it off". Really?

Please turn in your geek card at the door.

moe2• 16 years ago

And the Darwin award, category marketing, goes to...

dirtyhand2• 16 years ago

The thing I can't believe is that nobody at Pair Networks that was involved with this actually knew about Ycombinator or Hacker News... otherwise they wouldn't have shut down such a popular site. Somebody needs to hire real hackers

bwb7• 16 years ago

do real hackers like working customer service :)? customer service is 80% of a good hosting company.

dirtyhand2• 16 years ago

Somebody had to pull the plug, and it wasn't some customer service guy with a headset...

bwb7• 16 years ago

Guys, shared hosting is designed for the 99% of non busy sites, say under 1,500 unique visitors a day, with the level of traffic HN is doing I'm surprised Pair didn't warn you earlier. A site this busy needs a dedicated server or virtual server.

That is why shared hosting is cheap, you start with it and once you are successful or starting to get slashdotted you buy something bigger that can scale.

bwb7• 16 years ago

Also, just going to point out here that Pair.com has had a great reputation in the hosting industry, and has for the last 10 years.

I've been in the industry for 10 years and worked for quite a number of hosting companies, not Pair though, and when you have 150 shared clients on a machine and 1 client is causing the problems you do your best to deliver the warning before it gets out of hand but it is very hard to do.

rg• 16 years ago

When I emailed Pair.com to tell them that my site was about to get a writeup in the Wall Street Journal, they moved my site to a dedicated server within an hour, left it there until the spike of traffic was over, then moved my site back to a shared server--all at no charge. If you use Pair.com properly, they are more than excellent.

jseliger• 16 years ago

I don't think anyone will disagree. What they will assert, however, is that shutting down a site with zero notice is a jerky thing to do. As the other commenters have said, the nicer/smarter thing to do would've been to send an e-mail that says "Hey, you're using too much bandwidth/whatever." The smartest thing of all probably would've been, "You're using too much. Want to by more?"

jqueryin• 16 years ago

I actually find it quite embarrassing that the guy had no clue what HN or YC were. He had absolutely no idea... and he works for a hosting company. It's quite a shame. I really got a kick out of the fact he questioned the legitimacy of traffic as well.

MikeCapone• 16 years ago

There are probably a lot more hosting company employees out there than there are HN regulars (or even occasional) users.

ramchip• 16 years ago

I don't really see why every hosting company employee out there is supposed to know about a certain small american investment firm started in 2005 and its associated news aggregator... we're probably an order of magnitude or so less popular than Reddit, which is itself rather niche. It's a small dot in the web world.

feverishaaron• 16 years ago

"Biting the hand that feeds you" seems like an appropriate analogy for arbitrarily shutting down this site without warning.

ck2• 16 years ago

To be fair, if I was another customer on that shared host and you made my side crawl through no fault of my own, I'd want your site disabled asap too.

VPS or cloud is dirt cheap these days, no excuses for not planning growth?

CGamesPlay2• 16 years ago

The real WTF here is why HN considered it appropriate to run a production site on a shared host. Shared hosting is a bad idea all around if you are even slightly concerned about reliability. You don't control the server, so any 60 year old woman running a photoblog with a vulnerable WordPress version can bring your site down just by getting hacked.

marcamillion• 16 years ago

Hrmm....HN was hosted on a shared server. That's pretty impressive..if I must say so myself.

RyanMcGreal• 16 years ago

The email sounds pretty generic and policyish, which leads me to suspect it was auto-generated rather than typed out by some hapless sysadmin who's never heard of HN.

InclinedPlane3• 16 years ago

There are too many grammar errors for it to be auto-generated (I hope).

mikestanley2• 16 years ago

Kinda funny and sad that so many people here only see this as some sort of stupid or unfair action against HN, seemingly without even acknowledging that every single other customer on that shared server had as much right individually, and more right collectively, to not have their performance negatively impacted by HN.

Yeah, it sucks that one of our favorite tech news sites was impacted by this, but how impacted were all those other customers?

It is easy to make a smartaleck comment about how Pair was trying to upsell by doing this, which is preposterous. Pair is a well-respected provider with many more years providing good service at a fair price than HN has existed, and I'd be willing to bet will be around after HN has peaked and begins to move back to the traffic load that might make sense on a shared system.

But the fault here ultimately lies with the folks running HN who thought it was wise or appropriate to host any of its content on a shared server that likely cost them less money per month than most of us spend on soft drinks in a week.

euccastro2• 16 years ago

Strawman. Nobody's saying HN/YC should be allowed to overuse resources (then again, I don't know the terms they had agreed to). But not giving a warning is crappy customer service, no matter how many other folks do likewise, how many years Pair has been doing other stuff well, or however else you want to spin it. In this case, it also happens to be a big sales screwup.

mikestanley2• 16 years ago

Well, you didn't identify what I said that you think is a strawman, but I'll point out yours.

Fine, nobody's saying HN/YC should be allowed to overuse resources. I didn't say anybody was saying that.

Not giving a warning probably doesn't count as great customer service, but then again, once the problem had been identified by Pair, and once they knew of the negative impact HN was having on every other paying customer on that server, what kind of customer service to those other customers would it have been for Pair to fire off an email to HN then wait an hour, or thirty minutes, or ten minutes, before shutting it down?

How long should Pair have allowed HN to impact other customers to satisfy folks here? And what makes HN more important than any other paying customer on that server?

Oh right, it's because you read and like HN, which, ironically, so do I.

As for it being a sales screwup, maybe. I kinda doubt there is a great deal of overlap between HN readership and the average potential Pair customer. We could also suggest that Pair taking action to protect all those other customers on the server is an example of how they would provide good service to the many when they're being hammered by one overpowering fellow customer.

anotherpaulg• 16 years ago

This makes me think of a little side project I've been chipping away at called InstaCDN.

It makes it easy to minify, combine, gzip and push your css, js and image assets into the Amazon Cloudfront CDN with far-future expiration headers. It also automagically detects background images referenced in your css, and puts them in the CDN. It rewrites the css to use the new CDN image urls.

It's all done through a trivial REST API.

Would love some feedback, and to find out if/how it's breaking any of your complex css/js.

http://www.instacdn.com/

kordless• 16 years ago

Wordwrap on you page is pretty nasty on the iPad.

papaf• 16 years ago

That looks really cool. I'll be in a position to give it a go in a few months. One thing that would make me hesitate though, is not knowing what the potential pricing would be when you move from non-free.

_pius• 16 years ago

Complete failure on the part of Pair Networks.

ivankirigin• 16 years ago

If I ran biz dev at _any other hosting company_ I would offer to host the site for free in exchange for a little logo on the bottom of the page.

mstevens• 16 years ago

I'd suggest this as a business opportunity, but I suspect non-sucky shared hosting is a hard sell.

patio112• 16 years ago

I typically don't tell other people how to run their businesses, but if a similar issue brought my website down and I were to post about the causes, I might focus more on my failures in capacity planning, vendor selection, and monitoring rather than on my vendor's lackluster customer service. User-visible failures are, ultimately, process failures on my part, regardless of the surface cause. A nice side effect of this philosophy is that improvements to my processes help with all sorts of surface causes whereas if I were to address surface causes individually it would be like playing whack-a-mole. Bad vendor whack, hard drive failed whack, traffic spike whack, poor customer service whack, out of memory exception whack whack whack -- why is everyone conspiring to keep me from getting any work done.

davidw2• 16 years ago

One obvious point is that HN is a peripheral part of YC's business, mostly aimed at recruiting, rather than something that's critical.

pg4• 16 years ago

It didn't bring HN down. HN deliberately didn't rely on that server for anything except hosting static content that was also duplicated on this server. I planned in advance for the possibility that the other server wouldn't be usable, by writing the code so that I could switch to serving the same content off news by changing one variable, which I did. As a result service was barely affected.

In short, Pair flaked, but we had in fact planned the system in a way that protected us against it.

dnsworks• 16 years ago

HN has been pretty sluggish for a little while now. If you rewrote it in Rails you could it host it on Heroku and get back to not worrying about scaling!

alnayyir• 16 years ago

Please tell me you're not serious.

voxio• 16 years ago

No company worth their weight should be using off the shelf shared hosting.

Cause of YC/HN outage discovered

Related Stories