Github is down

https://status.github.com

by jayniz • 14 years ago

123 43 14 years ago

status.github.com

Swizec2• 14 years ago

If only there was some way of using git locally without needing github!

Oh wait, this isn't svn.

Diederich• 14 years ago

My company uses github enterprise, which is a VM that's very easy to make highly available.

stevejalim• 14 years ago

Link: https://enterprise.github.com/

nirvdrum4• 14 years ago

Aside from the fact this argument falls apart when you use submodules or third party repos, a lot of us use GitHub for the services they offer beyond just git.

1gor• 14 years ago

After you have init'ed and updated your submodules, the argument holds fine.

Swizec2• 14 years ago

The outage probably isn't going to last forever. And it's very unlikely you'd need to refresh third party repos in the time github is down.

Now if you're juuuust looking for some library or its README/documentation (I usually use Github for that) ... well I'm sure going for a walk for a few hours won't hurt you :)

tow21• 14 years ago

If only there were some way to mirror submodules or other third party repos to your own servers without needing github.

Oh wait, this isn't svn.

foz• 14 years ago

In my experience most Rails deployments and CI setups depend on Github being up. While it's possible to bundle and copy code on deploy rather than checkout from the server, most projects don't.

So of course you can continue developing locally, but if Github is down, it means the release/ci process is likely broken. I'm not saying this is Github's responsibility, but it's a reality for many people. It's another sales point for Github Enterprise.

tmh88j• 14 years ago

Ever tried using gitX? I really like it.

peterwwillis3• 14 years ago

Question for the Github people: Why not keep serving non-stale cached data while your databases are down?

You can do this with proxies or by modifying your code to always serve out of cache, and the db updates the cache, so if the db is down, the cache is your temporary failover while you fail over to the secondary db. ('cache' is anything memcached-like that's separate from your db)

jeremymcanally• 14 years ago

We do in a lot of cases. We just don't have every detail in the whole app cached. :)

peterwwillis3• 14 years ago

For the stuff you haven't got serving out of cache yet, might I recommend putting a proxy on the frontend layer? It can reverse proxy almost all of your stuff, and cache/proxy the remaining content which isn't dynamic (or even dynamic stuff, given the right voodoo). Cheap and codeless.

benatkin• 14 years ago

You don't have some things cached that should be obvious to cache, like the HTML for the most popular repos. Loading the commit messages from JSON for a page that's being accessed tens of thousands of time a day is less than ideal.

It isn't just github, it seems like a lot of web apps don't use the lessons learned for web sites.

hinathan2• 14 years ago

This feels like a pretty standard pattern for a lot of services — fail, come back up on backup DB, fail again when backup proves to not be capable of handling the surge of load, then eventually come back up on the primary DB once people have gotten bored and stopped hitting 'refresh'.

Is that a function of not prewarming failover DBs, or is there something pathological about the primary-secondary pattern?

kevindication• 14 years ago

Failing over to the secondary only helps if the problem is local to the primary. If you pushed bad code or the system just can not handle the load, the secondary will just fail in the same way.

pinko• 14 years ago

This is why Chaos Monkeys pay dividends.

minikites• 14 years ago

Maybe we just don't hear about situations when the backup/secondary server succeeds in picking up the slack because it would be transparent to the end user?

hinathan2• 14 years ago

Selection bias, good point.

Peroni2• 14 years ago

Frontpaging on HN will certainly help.

Andrex• 14 years ago

The purpose of a "status" page is ostensibly so that it can get the most visibility when the service is down, so that people aren't constantly refreshing the main site.

patrickaljord• 14 years ago

Also the status page is usually hosted on a different server than the service it reports on. Otherwise it would go down with the main service and be pretty useless.

Peroni2• 14 years ago

I know. I forgot my /sarcasm tag. ;-)

anupj• 14 years ago

I can see a github post-downtime analysis blog post coming up (I hope). :)

binarydreams• 14 years ago

Yay!! We made it, frontpage on HN!

drostie• 14 years ago

I've noticed that these are not immediately marked [dead] by the Hacker News admins like some other spammy topics, and I guess this is a sort of gray-area. On the one hand, it is in some sense a "new thing" and it affects the community; on the other hand it does not particularly gratify anyone's intellectual curiosity to be linked to that status page. The later explanation and diagnosis of what went wrong, if it's made public, might be interesting: just knowing "you can't use this now" isn't of that caliber though.

It's also puzzlingly dynamic: In other circumstances the "current status" pages linked during outages have not been blog-type, but have instead just literally reported the current status, leading to links which say "X is down" -- only for you to click it and see "X is functioning normally." This has already happened for the folks who linked the GitHub main page, which loads normally. (And won't it also prevent the same URLs from being submitted at a future date?)

TazeTSchnitzel• 14 years ago

Also, many of these things, by the time some people (such as myself) see them on the front page, are already fixed.

doki_pen• 14 years ago

I remember a time when meta-discussion on HN was looked down upon and never voted to the top of the comments section. I realize the irony of my comment.

lazyjones• 14 years ago

Cool, an auto-refreshing status page that will mostly be looked at when the servers (possibly the network) are already stressed.

tomschlick• 14 years ago

That page is hosted on separate infrastructure than their production app.

orangethirty• 14 years ago

Now that is a good service status page. It made me go from "Github sucks" to "Go github." It is well pieced, informative, and upfront about the whole deal. Their auto refresher seals the whole deal. It shows they are confident on their skills to get the problem fixed. Bravo github. Well done. Now hurry up and finish so you can answer the support email I sent this morning. :)

mylittlepony• 14 years ago

If only there was a better git hosting provider. Oh wait, I use bitbucket.org!

Github is down

Related Stories