Hacker News down, unwisely returning HTTP 200 for outage message

http://bibwild.wordpress.com/2014/01/06/hackernews_http_200_outage/

by pcvarmint • 13 years ago

399 113 13 years ago

bibwild.wordpress.com

HTTP 200 = "Cloudflare, please cache this status message instead of passing through a million requests to our dead server while it's busy restoring a backup".

PG doesn't care about HN's search listings, so there's no drawbacks to doing that.

Crito3• 13 years ago

"PG doesn't care about HN's search listings"

Yup. re: "Why does HN have a relatively low Google PageRank?"

"Probably because we restrict their crawlers. But this is an excellent side effect, because the last thing I want is traffic from Google searches."

https://news.ycombinator.com/item?id=5808990

If you need to search for something that you know was on HN, HNSearch is a great tool. I use it all the time.

guyinblackshirt• 13 years ago

> HNSearch is a great tool. I use it all the time.

Agreed - so is pinboard's search for #hn tags

rangibaby• 13 years ago

Thanks for pointing out HNSearch. site:news.ycombinator.com (whatever) works nicely with Google too.

6cxs2hd6• 13 years ago

"Probably because we restrict their crawlers. But this is an excellent side effect, because the last thing I want is traffic from Google searches."

Am I the only one who finds that puzzling?

This isn't Fight Club. It's not even Entrepreneur Club. It's a bunch of generally smart people talking about technology, with an emphasis on making money from it. It's one of my favorite sites, and I love it, but it's not an invitation-only club, is it?

(I also find it weird that one of the go-to sites for web-savvy people would be like, "yeah, screw status codes and how the open, linked, web is supposed to work".)

To be clear, I'm not Protesting a Great Evil. I just find it puzzling, as in, "That's odd, I must not understand what this is all about, after all."

ricardobeat2• 13 years ago

I seriously doubt Cloudflare's behaviour would be that stupid, wouldn't it momentarily cache error pages instead of hammering the server? At a minimum it would throttle/prevent concurrent requests.

cbhl• 13 years ago

Unfortunately, it took me 12 hours to find out the site was back up because the outage page had been cached for me. I had to realize I had to do a hard-refresh of the page.

X-Cubed• 13 years ago

Other options would be to return status code 503 Service Unavailable with a Retry-After header, or drop connections while the dead server is busy.

Both of those would be much better than returning 200 OK.

mcgwiz• 13 years ago

pass through a million requests...

How does that make sense? As if CloudFlare would honor status codes but not take advantage of cache headers (which in this case stipulated an absurd 10 year expiration).

elwell• 13 years ago

cache was too long though. i didn't think to try ctl+f5 until i saw that it was on according to twitter.

RobinUS2• 13 years ago

CloudFlare also takes of the load of your dead server if the server is dead. It still has a "Try the original server"-button, but that is more manual.

archildress2• 13 years ago

Welcome back, HN - I had an unusually productive (yet unstimulating) day at the office.

Udo• 13 years ago

Don't take this the wrong way, but after seeing hundreds of identical HN-outage productivity jokes/references on Twitter today, I really hope this is the last one I'll see in a while.

archildress2• 13 years ago

Wasn't really meant as a joke, but I completely understand where you're coming from. Haven't really been on Twitter today.

frou_dh• 13 years ago

I finally added tail-recursion optimisation to my Lisp interpreter while HN was down. Unless I hear otherwise, I'm taking the Onion crown!

NigelTufnel• 13 years ago

Someone should write a twitter bot that monitors HN status and tweets "Hacker News down, productivity up".

What the hell, I'll do it.

foobarian• 13 years ago

Seems like a fair point in principle, except in this case HN is one of the few sites I don't need Google to get to and don't care about any other tools that might rely on returned status codes.

sysop073• 13 years ago

Tools like web browsers?

JetSpiegel• 13 years ago

Of course. foobarian sends an email with the URL to a service and it returns the webpage, which he reads in emacs.

dragon1st• 13 years ago

I just realized I've addicted to HN, have been kept refreshing the page millions time :) thanks HN

eande• 13 years ago

I am not a news junky, but I realized how dependent my newsfeed is from Hacker News. Seeing about 6-8 times the outage message reminded me on why I keep coming back and reading the quality entries of these forums.

ignostic• 13 years ago

He's right that for most sites this would be undesirable, but PG has stated that they aren't looking for a lot of Google traffic. But then, just because it doesn't matter to PG doesn't mean it doesn't matter at all.

I've more than once found myself Googling old HN threads I'd like to find, but can't. Google's search (and site search) is miles better than HN's, but HN intentionally limits Google's crawl rate, thus limiting the amount of content crawled and indexed.

yawboakye• 13 years ago

Browsers tend to cache 200 OK responses. When HN came up (as reported on Twitter) I kept getting the error page until I bust the cache and reloaded. Yup, that's what 200 OK for an error page can cause. A regular reload will still show your _down_ page

ars• 13 years ago

I had the same issue. Checking all day. I saw that it was back up so I tried it - and it was still down.

Finally thought to try reload and it gave the page.

tedd4u• 13 years ago

Well by the letter of the RFC browsers and middle boxes (all the invisible caching proxies out there at your ISP, etc) are only supposed to cache if the cache/expires headers are set for that, but 200 is a bad choice for most people running a site for the other reasons listed above like crawlers/indexers. 5xx is correct for a problem on the server, usually 500, 503 or 504 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html).

At the same time there are many erroneous implementations (intentional and unintentional) that, as you imply, just cache any 200 without checking cache headers. This is also a good practical reason to avoid doing this.

0x0• 13 years ago

I think that was actually caused by the incredibly high cache-control expire header, which claimed cached versions of the page should be valid for 10 years (!)

wiradikusuma• 13 years ago

I know it suppose to make you more productive when it's down (I was), but suddenly I felt clueless during my commuting (which can take 2 hours total).

joshsegall• 13 years ago

This isn't bad for just Google. RSS aggregators were also getting the "everything's fine" message. I thought I had a bug in my aggregator until I went to the site and realized it was down.

tedivm3• 13 years ago

As people mentioned this site is an exception to how to do things, in that PG actively does not care about search engine results. However, for the people who are interested, here's a few ways you can handle a situation like this.

1. If you add the "max-stale" header to your content you can tell your CDN to hold on to things longer in the event of an outage. What happens is that the CDN will look at the cache time you set and check in as normal when that expires, but if it gets back no response or a 503 code (server maintenance status code- if you work with CDNs this is your friend!) it will continue serving the stale content for as long as you tell it.

2. Lets say you're beyond that, or your site is too dynamic. Instead of setting up an error page that responds to everything, setup a redirect with the 302 status code (so crawlers and browsers know it's a temporary thing). Point that redirect at an error page and you're golden. The best part is these types of requests use minimal resources.

What I do is keep a "maintenance" host up at all times that responds to a few domains. It responds to all requests with a redirect that points to an error page that issues the 503 maintenance code. Whenever there's an issue I just point things at that and stop caring while I deal with the problem. I've seen webservers go down for hours without people realizing anything was up, although with dynamic stuff everything is obviously limited. The other benefit to this system is that it makes planned maintenance a hell of a lot easier too.

Oh, another thought- you can use a service like Dyn (their enterprise DNS) or Edgecast to do "failover dns". Basically, if their monitoring systems notice an issue they just change your DNS records for you to point at that maintenance domain. You can also trigger it manually for planned things.

username223• 13 years ago

"actively does not care about search engine results. "

Or data -- they tossed a day or so of submissions and comments when restoring from backup, AFAICT. I'm not saying that's a problem; 4chan loses data and works just fine.

n09n• 13 years ago

That's a poor comparison. Users expect 4chan to lose that same data as part of normal operation anyways, so you can hardly consider it a problem. That's not the case for HN.

ealexhudson• 13 years ago

Search result ranking is an obvious casualty, but it's not the only one. My Firefox start page (as an example) now has a cached but empty version of the HN pages that doesn't really look quite right. Any system that pulls content automatically, like that, or an offline reader, etc. is going to get confused.

It's simply good practice to get the HTTP status codes correct in the ways you outline.

danielweber• 13 years ago

HN was down much longer than I thought it was because the bad pages were stuck in caches. I had to do a force-reload to see that it was back up this morning.

ErsatzVerkehr• 13 years ago

As people mentioned this site is an exception to how to do things, in that PG actively does not care about search engine results.

Is pleasing Google now the only reason to obey the HTTP spec?

vog• 13 years ago

Unfortunately: Yes.

For example, I had a hard time telling people that they should use a proper HTTP redirect from their domain "example.com" to "www.example.com" (or vice versa), instead of serving the content on both domains. All arguments about standards, best practice etc. were unconvincing. But since Google startet to punish "duplicate content", I never had problems convincing people again.

tomkin• 13 years ago

This a strange reaction. I agree that perhaps "this is different", but I am hoping I can assume that the next time I see an article on HN, about another website going down, or losing their data, that the HN community shows as much compassion and understanding. In my experience, when not related to the work of pg, there tends to be a lot of "you should have done X and there is no excuse for not doing Y!!"

If I'm being honest, it's off putting to watch community-wide apologist response to what would normally be outrage of poor execution.

tedivm3• 13 years ago

Eh, as someone who has had to run some large sites I'm a bit more sympathetic regardless of who it is. Shit happens, and they're just websites.

gtirloni• 13 years ago

I think it's an expected response from a community that is largely comprised of people that desire YC-bucks.

But you nail it. Had Gmail, Reddit or any other unpaid service gone down like this, the uproar would be heard from another galaxy.

jwilliams• 13 years ago

Maybe this was a prior error message, but the original CloudFlare "Origin Server" error was returning a 520 - which is a CloudFlare-custom HTTP Status Code.

Edit: CloudFront -> CloudFlare

garindra• 13 years ago

It's CloudFlare, not CloudFront.

creativityland• 13 years ago

@HNStatus helped keep me updated but it wasn't posted on the maintenance page until the end of the day for everyone.

jacalata• 13 years ago

And because of the caching, it wasn't posted on the maintenance page at all if you visited before it was added. (I was seeing it on one machine and not the other).

leoh• 13 years ago

Of hundreds of comments I have made on this site, only one has been snarky. Here's my second: chill your tits.

grandalf5• 13 years ago

By the logic of this blog post, a page like status.heroku.com should return a 503 when heroku is experiencing downtime and a 200 otherwise.

200 means that the page loaded as intended (which it did).It turns out that some of the page's content (the interesting stuff) was unable to be loaded, and the site's content reflected that.

A 503 would be appropriate if there was a server problem, which might have actually been the case, but with Cloudflare's landing page there was not actually a server problem (since cloudflare served the substitute content properly w/o error)

pizzeys2• 13 years ago

I disagree with this. By that logic, 404 pages should also return 200, because the error page stating that the content couldn't be found was indeed rendered successfully.

The difference between your status.heroku.com example and this is that in the former case you are seeking to look at a page that tells you about their status, when in this case you were seeking HN's index but instead got a page about status - because there was a problem preventing you from getting what you wanted.

randallsquared• 13 years ago

At my work, we have had arguments about what searches in an REST API should return in the case that no results (literally, in our parlance, "no documents") were found. Is that a 404 or a 200?

grandalf5• 13 years ago

So by that logic a wide variety of Cloudflare's serving modes ought to return a 503, including the one where it preserves cached content because the underlying server is unavailable.

Suppose HN consisted of two content panes and one content pane was unavailable, and the unavailable pane was replaced by a message indicating a partial outage, should HN return a non-200 response code then?

200 means "this response is being served as intended without the server serving it having an error". With Cloudflare acting as a hybrid caching proxy and static site, it is appropriate for it to return a 200.

If HN were not using Cloudflare, the underlying server would probably just show an automatically generated error message of some kind and return a 503 status.

fmax30• 13 years ago

Now i know why i had to refresh when i first opened HN 10 minutes ago even though it was up at that time.

datawander• 13 years ago

Even if the author is correct in that Google's ranking algorithm for HackerNews will be affected by just 24 hours of downtime, wouldn't the algorithm update itself back to normal over the next 24 hours?

cyanbane• 13 years ago

I was 8% more productive today. Down-vote at will.

anonfunction• 13 years ago

For anyone interested in HTTP statuses this is a great resource:

http://httpstatus.es/

factorizer• 13 years ago

those people certainly don't know their latin. just kidding.

n1ghtmare_• 13 years ago

Anyone knows the reason for the downtime ?

rplnt• 13 years ago

I also noticed loss of data.

For example:

http://webcache.googleusercontent.com/search?q=cache:r53Zl8w...

https://news.ycombinator.com/item?id=7017250

kogir• 13 years ago

This was all me. I probably should have thought about it more, but just wanted to make it clear we knew something was wrong and were working on it. Load was not a concern.

Though the article is correct, with everything else that was going on response codes and cache headers were the least of my worries.

I think the best takeaway is that you will go down at some point, so it's best to have a reasoned plan in place for when you do. Handling it in the heat of the moment means you'll miss things.

thaumaturgy2• 13 years ago

If it becomes a big deal, and you can't get the data from ThriftDB for some reason, I've got a copy of HN data (submission & comment content, user, points, item date) up to id 7018491, a comment by kashkhan, time-stamped 2014-01-05 23:56:34.

edit: Ack, I just realized that item_id got reset back to 7015126 on the reboot. My data matches HN up to 7015125, and then diverges after that.

emillon2• 13 years ago

Can you make it available somewhere? I'd be interested in that data, and sure that others would be too. Thanks!

jrochkind15• 13 years ago

Yep, I think your takeaway is exactly the right one, about planning in advance for outages.

I didn't post to try and make HN look bad, using the wrong http status code isn't a huge deal or anything. I just wanted to take the opportunity to discuss http response codes, an issue near to my heart. (In my day job at an academic library, the fact that most of the vendors we deal with deliver error pages with 200s does interfere with things we'd like to do better).

Thanks for the reply!

lnanek2• 13 years ago

CloudFlare really messes a lot of things up. I've seen CloudFlare refuse to give me error responses from forms before. Enter a bad value, get a cached page of the empty form, lol. Server was trying to return a page explaining the wrong entry, but CloudFlare refused to send it to me because it had a non-200 response.

belorn• 13 years ago

Question: Is the HNsearch still being worked on? It has broken "link" and "parent" for search results.

Gonzih• 13 years ago

Some people where speculating that HN banned goodle and other search engines. But from at least /robots.txt I can't see that. Do they do any ip based filtering? Does any one have information on that? I'm just curios.

TeMPOraL• 13 years ago

I know that pg "actively does not care about search engine results", but HTTP spec has other applications besides Google pagerank. It's hard to build amazing new technologies and improve the Web if people keep ignoring the standards without a good, technical reason. Please, for the sake of the example set to others, send the proper HTTP codes.

nayefc• 13 years ago

who cares...

ck2• 13 years ago

By the way, HN still says it is down at this url

https://news.ycombinator.com/

maxerickson• 13 years ago

The down pages had long cache times, you just need to refresh.

Yuioup3• 13 years ago

BTW I just realised that Chrome has been serving me a cached version of HN the whole time. Didn't realise it was up again. When did it go back on-line? An hour ago? More?

gedrap• 13 years ago

More than 10 hours ago

Yuioup3• 13 years ago

Thanks.

dntrkv• 13 years ago

Were you a member at computer-forums.net like 8 years ago? I know it's a long shot, but I remember someone with the same username as yours. I know it's not to the most unique username, but I thought I would give it a shot.

Yuioup3• 13 years ago

No sorry. I did come up with the name a long time ago but (like 1999 or something). I've used this username on /. for a while.

garthdog• 13 years ago

I didn't know this was up until somebody told me to ctrl-reload. Nice one with the 10yr cached soft-500 page. :P

Now that our brittle forum is back, let's get back to work nitpicking the Android UIs that aren't quite beautiful enough! This is not enough drop shadows!

mountaineer• 13 years ago

Just hit HN this morning and got the downtime message. Then I remembered this post and did a hard refresh to get back to normal. Browsers certainly aggressively cached it.

HNJohnC• 13 years ago

Same here, not sure why that isn't touted as a more important reason to provide the correct status codes rather than simply saying Google search isn't important.

Standards are important.

conanbatt• 13 years ago

Completely agree. I didnt even think of hard refreshing until i saw the "server is back up" status on twitter.

nawitus• 13 years ago

I learned about doing a hard refresh on Twitter (I didn't know about this post). I think it's a big problem, there's lot of users who can't currently access the site.

catshirt• 13 years ago

ah. this explains why HN has been telling me it's down for 2 days now. had to open it in incognito to realize it was a cache issue.

Hacker News down, unwisely returning HTTP 200 for outage message

Related Stories