Ask PG: Postmortem of the outage?

https://news.ycombinator.com/item?id=7015133

by lukeqsee • 13 years ago

330 132 13 years ago

news.ycombinator.com

Clarification: This is not meant with any ill-will towards PG or any of the other individuals who help run HN. It is a simple request for a postmortem eventually. Perhaps it's an unneeded request, but I think a lot of HNers echo the sentiment.

theGimp2• 13 years ago

It seems all activity from the past two days has disappeared -- backup storage is something you never regret paying for.

You've probably all seen it by now, but from @HNStatus: [1]

  Server back up and seemingly stable. Now restoring our latest backup to recover from limited filesystem corruption.

[1] https://twitter.com/HNStatus/status/420179162138021888

t0• 13 years ago

Were they not using raid or performing multiple database writes? A mechanical hard drive failure is pretty common and can be mitigated fairly easily.

dvanduzer2• 13 years ago

RAID arrays fail all the time; the system has famously been one server, and the only visible recent scaling work has been front end caching.

edit: the code has been public for a long time, and there is not a database to replicate. the site ran as a single server for years, and it is unlikely the front end caching has changed anything about the "database" components.

Since RAID failures actually are somewhat common, they are probably looking at a higher level replicated storage system now, a la DRBD, or some kind of distributed file system, a la Gluster.

venus• 13 years ago

Databases? What databases?

HN is persisted to flat files.

Sami_Lehtinen• 13 years ago

I guess he meant having two separate logs. One for production, and secondary with his journal. In this case you could restore from backup the original data, and then replay rest of stuff from the external log. That's the solution I'm using with really important data where I cannot afford any data loss, even if down time is acceptable. On commit, it committed to two separate systems, but the secondary system is only journal which can be replayed.

yaddayadda7• 13 years ago

Several comment threads that I was following when it went down are gone ("No such item."), although their original links are still valid.

theGimp2• 13 years ago

Indeed you are right; the original links still work for me. Though the ones I checked so far look exactly the same as the Google Cache version, so I don't know what happened there.

sehrope• 13 years ago

Yes I noticed this as well from (the lack of) my own comment activity. I don't comment that often but I had written something yesterday that has disappeared.

On a more general note if anybody has backups and they aren't regularly tested restoring them, then you really don't have backups! As an added bonus, regular restoration tests let you practice for the "real deal" and you know how long the entire process will take.

pacaro• 13 years ago

This. Sometime last century I had to restore an old codebase from a tape backup. Step one find the correct drive...

We'll never need that old repo again.

jon-wood• 13 years ago

One of the nicest ways I've experienced of making sure your backups are good is to sync up your development machine with them occasionally. Obviously there are situations like HIPAA where you can't do that, but if you can, do. You'll catch problems with your backups long before you actually need to use them.

pconf• 13 years ago

IMO data loss is less a symptom of untested backups than it is of developer-managed systems. I wonder if ycombinator has a sysadmin (who isn't also a developer)?

aroman• 13 years ago

Yeah, I lost about 200 karma (what was about 15% of my total) in the crash.

Good thing they're just silly internet points :)

yaddayadda7• 13 years ago

Thank you for making my paltry 22 lost karma points look so, well, so paltry. ;-)

edit: to clarify "lost" karma

8ig82• 13 years ago

Just gave you an internet point for having perspective.

allochthon• 13 years ago

I gave you back an Internet point.

bvirkler• 13 years ago

I lost 25% of mine! A whole point!

brianbarker• 13 years ago

Gave you a point so you can feel special again. Here's looking at you, kid.

KC8ZKF• 13 years ago

I hope at some point things reverse, and instead of accruing karma we shed it. When we reach zero--

lucb1e2• 13 years ago

And here I was wondering what I was being downvoted for. Turns out: nothing, a post just disappeared :)

jnsaff2• 13 years ago

Here you go, I gave you one back.

Alterlife• 13 years ago

So, you had about 1333 points?

It looks like you've already exceeded that now... Well done! The internet gods must favour you.

rullopat• 13 years ago

Indeed, the first time I got something on main page and my karma skyrocketed from 5 to 80 something, I lost all.

Just internet points, fortunately :)

elwell• 13 years ago

Well you're number 1 on bestcomments [0]. Apparently, this is the most effective way to rack up karma.

[0] https://news.ycombinator.com/bestcomments

8ig82• 13 years ago

What's odd is that if you look at your submission history, you can up vote your own submissions.

Maybe that's nothing new, but I just noticed it. Seems like a bug.

neurostimulant• 13 years ago

What's odd is that if you look at your submission history, you can up vote your own submissions.

It doesn't seem to do anything though.

mixmastamyk• 13 years ago

That's a shame, I was really looking forward to the comments for the article below. Unfortunately I had it loaded, but hit Ctrl+R (like I sometimes do) and lost it forever. :/

The google cache got a few comments, but very few.

http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

https://news.ycombinator.com/item?id=7015438

http://webcache.googleusercontent.com/search?q=cache:9368SwV...

yaddayadda7• 13 years ago

I missed the original posting of https://news.ycombinator.com/item?id=7015438 and it's right up my alley (now it's tomorrow morning's reading :) )

hga• 13 years ago

"backup storage is something you never regret paying for"

Indeed, but the time and CPU it takes to do backups can be a very nasty trade off.

For that matter, as I write this, due to a mistake I made after a 85 minutes power outage yesterday, I'm just now doing my daily incremental backup of my home machines to an LTO-4 tape drive. Keeping that drive fed fast enough to prevent "shoe-shining" took some effort, Bacula spools up to 100G at a time to a partition on a single 15K disk on a separate controller. But if I had a LTO-5 drive, from what I've heard there's no single disk in existence that can keep up with a drive (not counting SSDs, which are a very poor match for this use case).

sitkack2• 13 years ago

My feed array to the LTO-5 drive is 4 2TB (Hitachi) in RAID-10. Backup strategies, much like build systems are some constant factor harder than they appear.

I'd like to migrate to ZFS but have yet to. Still just running EXT4.

HN should be on a replicated data store like Riak. Losing a node or two shouldn't take the system down, or should at least run in a degraded state (read only) until hardware is restored.

nhangen7• 13 years ago

Because it's not good enough that the site is back, we need to pile on and complain too...

lukeqsee5• 13 years ago

I'm not complaining. I'm asking a legitimate question. A popular site has been down for a significant period of time; I think a postmortem will be insightful.

nhangen7• 13 years ago

If you're asking, it means you know that PG authors them in most instances. The fact you felt the need to start a new thread, just minutes after the site is back, tells me you're anxious and overeager.

yaddayadda7• 13 years ago

Asking a technical site about a technical site outage isn't, in and of itself, complaining. It's an opportunity to learn.

bradly• 13 years ago

Asking for a postmortem is not complaining.

nhangen7• 13 years ago

The site has been up for minutes and already asking for a report. I'm sure that's the last thing on PG's mind at this point. It's just clueless and classless.

EpicEng• 13 years ago

Who's complaining? I only see curiosity.

jader2013• 13 years ago

This is a bit silly. Anytime a (very active) social site goes down for a few minutes, people are interested in what happened, why, and what is being considered to prevent it going further.

There's no doubt that the whole time it was down, these questions were being asked -- both by PG and co. as well as the end users.

Not that we expect immediate and concrete answers, but it's certainly expected that the question will come up.

And it's pretty common for services to post a quick update stating the issue, cause, and postmortem at a high level, once service has been restored.

I know I try to do this every time my site goes down for even a couple minutes, let alone ~24 hours, and it has nowhere near the activity of HN.

Several HNers have the same question, thus the reason it is the top-voted new post.

nhangen7• 13 years ago

I agree that knowing what happened is a legit curiosity, I'm just calling the op out for being a point sniffer.

alan_cx2• 13 years ago

You know.... just to help you in future...

Your post might have done better if you had gone at it this way:

"Hey guys, PG and his minions are probably flat out on a couch somewhere, breathing many sighs if relief, and slinging back a well earned Scotch (might be a cider). Could be a while until we hear form them."

xmonkee• 13 years ago

Social experiment

jader2013• 13 years ago

I considered this as well (seriously), until I saw the lost data from the past day or so.

noblethrasher• 13 years ago

You may jest, but I once suggested something like that: https://news.ycombinator.com/item?id=2403880

geerlingguy2• 13 years ago

Would like to read it too. And it looks like right now is a good time to get just about anything in the front page. Front pretty much == new.

ewoodrich• 13 years ago

Are you actually able to see 'top'? I'm still getting the error.

EDIT: (never mind, it was just cached)

geerlingguy2• 13 years ago

Yeah, the 200 response during the outage is playing with everyone, I think; you have to do a hard refresh on any URL you had visited during the downtime :/

stickhandle• 13 years ago

It bothered me more than it (probably) should

sigvef2• 13 years ago

During the outage, https://twitter.com/HNStatus went from somewhere around 300 to 1163 followers.

lukeqsee5• 13 years ago

Earlier today I saw it had about 45 followers. I think it was a new account. (Please correct me if I'm wrong.)

yaddayadda7• 13 years ago

Their first post is dated 28 July - https://twitter.com/HNStatus/status/361707202123268096

edited: "They're" --> "Their" (there/their/they're will be the death of me!)

MichaelApproved• 13 years ago

here's a tweet from July 28th, so it's not a new account https://twitter.com/HNStatus/status/361707202123268096

cenhyperion2• 13 years ago

It has tweets from as far back as July of 2013.

pairing• 13 years ago

I'm surprised HN is using twitter instead of one one of their investments, Statuspage.io.

lucb1e2• 13 years ago

I actually looked for a twitter account, but didn't know which would be official. Then jgrahamc retweeted hnstatus and I knew which to follow.

yaddayadda7• 13 years ago

I find it interesting that this question is fresher (by a minute), has more points (67 v 42 at snapshot), and has more comments (18 v 10 at snapshot) than "HackerNews down, unwisely returning http 200 for outage message" but is ranked lower (2 v 1 at snapshot).

snapshot - http://oi40.tinypic.com/2mmbv5y.jpg

Kronopath• 13 years ago

Self posts are penalized so they don't clog the front page for long.

http://jacquesmattheij.com/The+Unofficial+HN+FAQ#selfposts

dschiptsov• 13 years ago

Is there any plans to release a new version of Arc, if it exists or server side code (without business-critical stuff)? I guess that there are lots of improvements since last Arc release.)

cenhyperion2• 13 years ago

I'm also interested in what the infrastructure of HN looks like. One of the tweets via @HNStatus seemed to imply that the site runs off of one application server.

sigvef2• 13 years ago

HN is indeed running on a single (10 month old) server, it seems [1].

[1] https://news.ycombinator.com/item?id=5229364

morganherlocker• 13 years ago

I was bummed that the conversation around openstreetmaps got killed in the middle of it, and now I do not see it on the front page. Does anyone have a link to that thread or did it disappear?

eevilspock• 13 years ago

me too. i guess we can start over: https://news.ycombinator.com/item?id=7015502

jpatel3• 13 years ago

Its no longer on the front page anymore..

dbaupp• 13 years ago

It's been resubmitted: https://news.ycombinator.com/item?id=7015502

(But I think the original thread was totally lost, I submitted it and it's not listed in my submission history.)

rhizome2• 13 years ago

Postmortem: it went down last night when people should have been going to sleep before their first day back at the job after holidays. It stayed down until the end of that day, with the last couple of days of vacation insanity erased.

Appreciate the gift of perspective that has been given.

carljoseph• 13 years ago

Interesting that your perspective is locked into one side of the globe. ;) HN was down during the day my time, when we had already slept before returning to work. :)

Appreciate the gift a new perspective gives you.

rhizome2• 13 years ago

I scoped it to a YC framing.

royalghost• 13 years ago

I am sure pg is going to write an essay on this :-)

joshuaheard• 13 years ago

I'm not an expert in internet architecture, but shouldn't a site this important be running on redundant servers? The irony of a tech site going down due to technical issue is making me grin, however. Glad to see it back :)

alan_cx2• 13 years ago

"Important"?

Really?

Obviously Im a fan of the site, etc, etc, but "important"? On what level?

Im not even sure I'd call Facebook or Twitter important. Banking, yes. Weather warnings, yes. Things like that, sure. But, Im also pretty sure "important" is slightly over egging it for dear HN.

(No offence PG xxxx)

acjohnson55• 13 years ago

I definitely find it important, from a career perspective.

kamaal• 13 years ago

>Im not even sure I'd call Facebook or Twitter important.

Imagine Twitter or Facebook being down during Egyptian revolutions.

DonGateley• 13 years ago

If the outage was due to something malicious I don't really expect to see a postmortem.

pg• 13 years ago

I don't know the details. Nick Sivo is in charge of this stuff, and he'll post something about it. I know he thinks the root of the problem was a disk failure. The server got wedged, and when we rebooted, the file system was corrupted. I'm not sure exactly why it took so long to restore. I was out of town the whole time this was happening.

The reason we lost so much data was that we only do nightly backups. That seemed enough when we started. Now that HN is a bigger part of more people's lives, we'll make more of an effort to make it proof against this sort of problem.

saurik• 13 years ago

Another point to make is that, even with a lot of data loss, it at least would be good in the future to skip a bunch of post identifiers (potentially even just saying "well, we certainly didn't use 100,000 of them, so I skipped up to 7100000") in order to try to not cause data that used to be there, and may have been archived by various sources including Google Cache or in the various reader clients people use on various devices , to suddenly have been swapped out by different posts leading to wide-spread cache corruption (and further confusion or loss). I mean, even just for the sake of people who may have posted links to this content in various places: maybe they wrote an article or posted a tweet/comment somewhere referencing how great/horrible a post was, and now suddenly it is saying something entirely different and potentially quite awkward ;P.

As a quick example, for those still not sure what I mean: what used to be an article about Python 2/3...

http://webcache.googleusercontent.com/search?q=cache:9368SwV...

...becomes a comment with numerous references for how to learn arduino hacking.

https://news.ycombinator.com/item?id=7015438

jaakl• 13 years ago

The problem happens also with HN own search, tried to get comments for first entry of https://www.hnsearch.com/search#request/all&q=openstreetmap

kogir2• 13 years ago

I'll post something more detailed tomorrow, but in terms of data loss, we went down at 2014-01-05 16:10:29 PST and restored a backup from 2014-01-05 01:00:00 PST.

kogir2• 13 years ago

I lied about tomorrow. The "post" of post-mortem requires it be over. Still not done changing things.

justinzollars• 13 years ago

Thanks for the update!

swalsh• 13 years ago

Its amazing to think that HN is still someone's "side project"

hnriot• 13 years ago

why? the value of hn isn't the software, but the community.

pbhjpbhj3• 13 years ago

It seems to be valued much more by the community than those who manage it - this is a problem IMO as there's then an inbalance in the desire of the community to have an optimal tool and the desire of the management which appears to be more to provide something akin to an MVP.

Of course the community is diverse and my view is not necessarily representative of any type of majority.

bane• 13 years ago

Thanks for the update. I don't think all that much was lost, and a day to restore from a disk failure ain't bad. Please thank Nick and whoever else worked overtime to get things back up and running!

rcfox• 13 years ago

pg: I don't know how much you care to get back the data that was lost, but it seems like it's at least partially available in the hnsearch.com API: http://api.thriftdb.com/api.hnsearch.com/items/_search?prett...

pearjuice• 13 years ago

Do we get the karma we lost refunded somehow? I am certain I am missing around 30 points.

pbhjpbhj3• 13 years ago

Because?

nmc2• 13 years ago

Despite the website being back online, the root URL still redirects to the error page (at the time of writing this).

So https://news.ycombinator.com/news works, but https://news.ycombinator.com still redirects to "Sorry for the downtime. We hope to be back soon.".

watermel0n• 13 years ago

It's your browser cache.

nmc2• 13 years ago

Yes it was!

rainmaking• 13 years ago

This must have been the most productive time for the tech industry in months.

ithkuil3• 13 years ago

No, I just kept wasting time reloading HN home page or following notifications on twitter!

ithkuil3• 13 years ago

I wonder how much effort would be reasonable to improve the resilience of HN to this kind of issues, given that's a relatively rare issue and HN doesn't really have a money loss in case of a downtime such as this.

ams6110• 13 years ago

Probably little. There's no ad revenue being lost, no business transactions that can't be completed, no life-saving information that can't be accessed. When you boil it down, it's a social/entertainment site, nothing anyone can't live without for a day or two.

ithkuil3• 13 years ago

but afaik there was some dataloss, some users lost their karma and their comments.

pbhjpbhj3• 13 years ago

nothing anyone can't live without for a day or two //

On this basis can't you shutdown pretty much anything the majority use day-to-day?

vanwilder77• 13 years ago

Damn I made it to 1300 karma's!

Now I'm back at 1273.

Ask PG: Postmortem of the outage?

Related Stories