So what happened to the HN server?

https://news.ycombinator.com/item?id=5237947

by huhtenberg • 13 years ago

170 127 13 years ago

news.ycombinator.com

I'm sure I'm not the only one curious how that Altair is holding up.

There was a post about it recently (advising that there would be downtime) - they've upgraded. It's considerably quicker now!

neeee2• 13 years ago

That was for Saturday morning.

pyre• 13 years ago

Obviously the new server has some growing pains. :P

Just because the switch-over went smoothly doesn't mean that normal operations didn't fall apart later on.

gabriel34• 13 years ago

But saturday morning the site was up Perhaps the server upgrade was delayed or something went wrong with it.

detst• 13 years ago

That was Saturday but appears to be related. PG posted this to Twitter[1]:

  Apologies for the HN outage. We think it may be related
  to the new server. This could take a while to fix.

[1] https://twitter.com/paulg/status/303350671460147200

level092• 13 years ago

isn't a bit long for a server upgrade ? shouldn't be there better strategies for quick switching ?

neilkelty• 13 years ago

Are you paying for this?

viraptor• 13 years ago

Yes they are and they're applied to critical systems. I doubt HN is regarded as such. Especially if it would require splitting backend/frontend servers and replicating both to do step by step upgrade of each component.

Alternatively you know... you can just accept that the service will be down for an hour or two and most people wouldn't even notice without this post.

supercoder4• 13 years ago

Though do wonder if it being quicker has something to do with everyone getting bored of refreshing it.

gabriele• 13 years ago

original pg's announcement:

http://news.ycombinator.com/item?id=5229364

superflit2• 13 years ago

My bad karma tainted the server..

supercoder4• 13 years ago

I'll upvote you so it doesnt happen again.

superflit2• 13 years ago

Thanks Bro. Lived in Colorado for 3 years, Eric cartman was my role model ;)

dutchbrit• 13 years ago

Curious too. I was able to ping the servers when it went down again a few hours ago just fine. Let's just hope it stays up now!

guojh05702• 13 years ago

I think it may need some time to adapt the new server.

dotmanish• 13 years ago

More importantly, why isn't there a status page for something that's equivalent to a lifesaving drug for many?

supercoder4• 13 years ago

there's only one server. if the server is down, then assume the status is not 'green'.

dasil003• 13 years ago

How can Joe-random-user be certain the server is down?

andrewcooke• 13 years ago

use twitter - there were various things there, including an explanation from pg and some people recommending alternatives.

https://twitter.com/search?q=hacker%20news&src=typd

GFischer• 13 years ago

And five free issues from Hacker Monthly :) . That made me happy.

gbraad4• 13 years ago

phew Almost thought HN was blocked here in China...

lucb1e5• 13 years ago

There is no way to know if it's just you for whom it's blocked? Are services like isup.me/domain also blocked?

bbrizzi• 13 years ago

www.viewdns.info has a "Chinese Firewall Test" to check if a particular domain is blocked in China.

gbraad4• 13 years ago

There s a plethora of checks I could do, and a lot of sites which check from different locations within china. The Internet is not same everywhere... But even some of these sites are blocked.

I did visit one of those quickly and the result was HN should be reachable... Hmmmm, therefore I. Checked with another age and this also timed out on my tablet. So either my ADSL or the server was problematic? I went for dinner and upon return it worked.

The check I used was http://www.blockedinchina.net/ First time I used this site...

X47• 13 years ago

phew lmost thought HN was blocked when behind VPNs

gbog• 13 years ago

When blocked browser behave differently. btw I'm in Beijing too, maybe we met in open parties?

seanmcdirmid2• 13 years ago

This looked more like the block you get when you are on the site's black list. How many HN readers are in Beijing? In zhongguancun?

gbraad4• 13 years ago

Was on a tablet and page timed out... Didn't feel anything different than nytimes. And another page also didn't load as if new records were added.

But you could have easily met me. Mostly Linux events... mail me if you want to know

runarb• 13 years ago

Do we get a new server? Please tell me more :)

kaushaldhora• 13 years ago

Thank God Its Up now...:)

lucb1e5• 13 years ago

Thank God it's up now. FTFY

Overcapitalization is annoying, and it's "it's", not "its".

odiroot• 13 years ago

You should thank PG instead.

tferris3• 13 years ago

Saturday, I thought: "This guy (PG) is really brave, everybody is nowadays using virtualized server environments on mulit-core machines and PG just gets some bare metal and puts his app on one core and one thread. WTF, this guy is a genius, single-threaded apps on bare metal FTW, this is so great and I am doing this too."

After seeing this thing down for several hours -- and Google hate downtimes, they punish you instantly -- I think: "Maybe not."

However, happy to hear what went wrong and why we still should go for the bare metal thing.

supercoder4• 13 years ago

the site actually runs very quickly, so can't argue it needs more cores / threads.

seems the issue was more with the migration.

so i reckon you're good to get back to removing all the threads from your apps :)

tferris3• 13 years ago

I totally agree, this new machine is so fricking responsive that I still think bare metal rules but I am just so afraid of the system operations of such a machine. But maybe Paul can give us some hints and why it's still worth to go the bare metal route.

EDIT: since Heroku fooled Rapgenius and us all it would be one more reason to get into system operations again and host on bare metal.

lucb1e5• 13 years ago

I wanted to reply that it's actually relatively fast, not very fast. Much faster than before, but there are many sites which are much quicker.

Timing some pages, the response is around 350ms. That means 95% of the loading time is networking, not generation time. You're right, the server is really fast given the load HN brings down on one server!

stiff• 13 years ago

What would the benefits of using a virtualized server for a relatively small (in terms of server needs) site like HN be? The way I see it so far, is that virtualization is mostly beneficial for hosting companies that can provide cheaper hosting and have better isolation and easier resource allocation. But I can't think of any big benefits for the site owner, assuming they can afford a dedicated server, and I can see downsides of virtualization, so I am genuinely curious if I miss something.

zokier• 13 years ago

(Live) Migration is one nice feature of virtualization.

gmac• 13 years ago

One thing is abstracted failure-proof(ish) disks. I'm considering moving from Linode to Hetzner for the enormous amount more RAM I can afford on a bare metal box there, but the one thing that gives me pause is having only RAID1 redundancy, and having to manage it myself.

zura• 13 years ago

Oh, that explains why this site is really slow occasionally.

nkohari• 13 years ago

Virtualized environments vs. bare metal has never really been about performance. It's about whether you want to spend the time required to tend your servers, or whether you want to treat them as disposable.

guojh05702• 13 years ago

how did you know everyone uses virtualized server environments on mulit-core machines..

jules5• 13 years ago

Virtual servers vs dedicated servers has not much to do with reliability. Both go down. What you want to minimize the risk of downtime is redundancy, e.g. hot fail-over.

tferris3• 13 years ago

what's the best way to have hot fail over?

markyc• 13 years ago

strange productivity peak recorded in the IT sector during the past 16 hours

lucb1e5• 13 years ago

16 hours?! I'm so glad I slept through most of it then!

mattquiros• 13 years ago

strange productivity dip happening in the IT sector right now

carlio• 13 years ago

Soon in the news: "New startup emerges from stealth mode after delivering on promise to improve IT efficiency by 80%"

farious• 13 years ago

I was starting to freak out without HN! (Even created an account just to express my feelings)

purplepilot• 13 years ago

i had to work for heavens sake

jbrooksuk• 13 years ago

Goodbye productivity.

X47• 13 years ago

Goodbye procrastination.

X47• 13 years ago

What else can go wrong on monday?

X47• 13 years ago

Why don't you Ajaxify HN while you're at it??

Yuioup2• 13 years ago

Not trolling but just wondering. What added value does Ajaxifying the site bring to HN?

X47• 13 years ago

:) I'd instantly know when I get downvoted for saying something stupid ;)

seeing submissions in realtime, reducing load on the HN server. Saving F5 addicts.

profquail• 13 years ago

Why don't you implement AJAX support in news.arc? That way PG won't have to spend the time writing it...

https://github.com/nex3/arc/blob/master/lib/news.arc

X47• 13 years ago

Good Point. Thanks for posting the link.

Sorry, I don't know Lisp and Arc seems to be a dialect of it that I also don't know.

youngerdryas3• 13 years ago

Apparently it is not out of the woods yet - the comment tab links to the front page.

samwillis• 13 years ago

Seems to be showing the homepage, not redirecting it. I think PG/RTM must have introduced a bug while fixing the problem. Have they no unit tests ;-D ?

youngerdryas3• 13 years ago

Or maybe they removed the comment page because it was causing the crashes, possibly related to the Unicrud Twitter crashes?

JuDue• 13 years ago

Fast is one thing.

BUT I'd much rather get rid of the "this link has expired" syndrome.

Fast is only impressive if the results are modern.

brudgers4• 13 years ago

The superior results from my browser's back button far outweigh the inconvenience of the expired link. I click on a link from the "Ask" front page, reply to the top comment, and my comment appears in the thread. Next I click "back" on my browser and I'm back at the front page of "ask".

I do that much more often than I hit an expired link.

cincinnatus• 13 years ago

Still, why not link to a page offset and just show what is current? Or include it with the current linking and automatically redirect? Pretty basic ux optimization.

jules5• 13 years ago

That doesn't really have much to do with the expired link issue. That issue exists because the targets of links are stored as closures on the server. What you want is to serialize those closures into the URL itself, instead of letting the URL be a pointer to a closure stored on the server. In traditional web applications, you'd have to do that serialization manually (although if you've never used such a closure based web framework you might not even be aware that you're doing this, just like a C programmer who has never used a high level language might not consciously realize that he is implementing objects or closures or garbage collection manually -- or like an old Fortran programmer who is not aware that he is implementing recursion manually). The problem used to be much worse, but then PG did this transformation manually for the important subset of the site (most importantly for links to the comments section -- these links no longer expire).

Here's an example. Currently if you go to the home page and then click the "More" link at the bottom to go to the next page, and wait long enough, the more link expires. That's because it's currently implemented as something like this (not sure about ARC syntax, I'll use Scheme syntax here, but you get the idea):

    (define (show-list-of-posts page-number)
      ... display the rest of the homepage ...
      (link "More" (lambda () (show-list-of-posts (+ 1 page-number)))))

This stores the closure `(lambda () (show-list-of-posts (+ 1 page-number)))` with the free variable `page-number` in a hash table on the server. The URL becomes an index into that hash table. But the only information needed to reconstruct such a closure, is the function body and the free variables. So if we defined an auxilary function:

    (define (foo page-number)
      (show-list-of-posts (+ 1 page-number)))

then we could represent the closure as the pair ("foo",page-number). If we encode that in the URL, then instead of looking up the closure from the hash table, we can reconstruct the closure on the fly. Hence we do no longer have to store anything on the server, and no links can expire anymore.

There are some challenges when you want to do this transformation automatically, but they can be overcome.

lmm• 13 years ago

There are extensions that help with that.

raldi• 13 years ago

The worst is when you write a big long comment, submit it, and get "this link has expired" -- then you hit back, and the textbox is empty.

This is particularly infuriating on a cellphone.

One quick fix: in the code that prints this message, check to see if the request was a comment-post action. If so, append, "...but for your convenience, here's the text you tried to post, so it's not lost forever: [...]"

(Just watch out for XSS!)

kyro• 13 years ago

PG wasn't aware that PetSmart closes an hour earlier on Sundays, so he was forced to wait till opening this morning to get the additional hamsters needed to keep this place running.

3327• 13 years ago

amazing... Just amazing... Would you happen to know what breed of hamsters he is using?

Filligree• 13 years ago

Cured djungarian hamsters, I believe.

chairmankaga• 13 years ago

I think they are deadly poisonous Zanzibar hamsters. Genetically engineered from rat and hamster DNA.

noonespecial• 13 years ago

All I know is that you never have to clean the cage. I'm told that when they run, they don't have side-effects.

ralph• 13 years ago

I remember when this would have been considered noise here; now it's the top comment.

Justsignedup• 13 years ago

Probably because thats what powers my computer nowdays. Though it's not hamsters, its a plump labradore australian cattle dog mix. Going green.

stcredzero2• 13 years ago

We three have all been hell banned, you see.

EDIT: More proof -- downvoted -- what could be more hellish than to ban humor?

tuke• 13 years ago

I thought the thing was running off of a server on an iPhone; probably just ran out of battery power.

lifeisstillgood2• 13 years ago

I have only one portal onto the Internet - this site. I check it first thing and last thing, and when it was down today I had a weird epiphany - I did not mind.

HN is not a service, its not a startup trying to get mindshare - its a (luvvie alert) community - and one where the downtime had a known and obvious cause and did not matter - rtm would fix it, I would catch up, I knew this thread would exist and I would still get insight into what is happening 12 hours ahead of everyone else.

I thought I would be more bothered - turns out I am emotionally invested, not emotionally dependant.

To PG et al, good luck with the fixes, and thanks.

Cheers

Udo6• 13 years ago

It's curious, isn't it, that we still have "Home" pages. For a while there, Google Reader was my starting point, but nowadays I find myself starting off from HN. This must be what Facebook feels like to normal people, but without the insidious stalking and the dreaded obligations that come from surveillance features like "X is online NOW, you have to interact with him/her!" and "X has seen this (but has neither liked nor commented on it)". Also, I wish my Facebook friends would produce a link and comment stream that is even one tenth as interesting as HN ;)

dfc6• 13 years ago

Normal people? I just vomited a little. Thanks for ruining breakfast.

dfc6• 13 years ago

I am not familiar with "luvvie alert"?

Surio• 13 years ago

Oh British English, how much I love you..... Let me count the ways! [http://en.wikipedia.org/wiki/Let_Me_Count_the_Ways] ;-)

http://oxforddictionaries.com/definition/english/luvvy

http://tvtropes.org/pmwiki/pmwiki.php/Main/Luvvies

lifeisstillgood2• 13 years ago

Imagine you are at a nice party in London. You are chatting to an interesting sounding person who turns out is an actor, and then you hear phrases like this

"just one summer with the RSC back when I was younger".

"Of course Johnny said, ... umm, Gielguld of course dahling, did you see him in the Tempest in 74? Wunderful"

"I was saying to Sam just after that Quantum thing. You really should talk to Barbara about directing the next one. I have her number; charming lady, did I ever tell you about the time she ..."

A small red rotating flashing light will rise up out of the coffee table next to you and the words "luvvie alert" will be flashed across your field of vision.

wglb• 13 years ago

Here is a curious thing--now that its up, checking http://news.ycombinator.com/threads?id=pg times out. And apparently only for PG. Other frequent commenters work fine.

youngerdryas3• 13 years ago

It is back up now as well as the comments page.

barking• 13 years ago

This latest test suggests that HN seeks membership of errorban club. http://www.codinghorror.com/blog/2011/06/suspension-ban-or-h...

So what happened to the HN server?

Related Stories