Major Service Outage at GitHub

https://status.github.com/?date=now

by acconrad • 10 years ago

155 85 10 years ago

status.github.com

the_duke5• 10 years ago

Again.

Affects both the website and git.

joatmon-snoo3• 10 years ago

Affects push to GitHub*

I really hope that your copy of git isn't broken because GitHub went down :p

the_duke5• 10 years ago

Haha got me. :P

No I just tried both the website and a `git pull`.

synthecypher• 10 years ago

How am I supposed to do any work if I can't copy and paste code.

PerryCox• 10 years ago

StackOverflow is still up.

mkaziz2• 10 years ago

My builds are broken because they have dependencies on github repos.

mjolk• 10 years ago

This is a teaching moment for you then. Minimize external dependencies when you've moved from "thank god it works" to "this is production ready."

dsp12343• 10 years ago

One solution to that problem is to run your own local git server, with an automatic process for pulling github repositories of the dependencies you need. Then you can always know that you will have the access to our dependencies even with a third party is unavailable.

Cafey• 10 years ago

Has anyone heard what might be the cause?

the_duke5• 10 years ago

It just went down about 10 minutes ago, you'll have to wait a bit for an explanation...

andrewstuart22• 10 years ago

09:14 MST Major service outage.

It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe one guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.

joatmon-snoo3• 10 years ago

The last outage they were back up in 4 minutes.

alfalfasprout• 10 years ago

Pretty sure devops at a service like Github have slack alerts and/or something like pagerduty. They'll be drowning in alerts.

TheRealWatson• 10 years ago

Rumors are that they edited some JSON files. Not a good day to touch JSON.

pella• 10 years ago

Hacker News Time! :)

joatmon-snoo3• 10 years ago

I count 4 threads on /new referencing GitHub down :D

messutied• 10 years ago

but we are keeping this one :) I like it

madlynormal• 10 years ago

It's always Hacker News Time!

sahin-boydas• 10 years ago

Long live Gitlab...

hackerboos• 10 years ago

Except. Pushes of new repos are still broken in the CE. No fix for over 3 weeks.

https://gitlab.com/gitlab-com/support-forum/issues/207

https://gitlab.com/gitlab-org/gitlab-ce/issues/26369

https://gitlab.com/gitlab-org/gitlab-ce/issues/21502

Only work around is to create a README on empty repos via the GUI and which then opens up the Protected Branches page. That said the rules around who is able to commit on the repos are broken.

Perihelion• 10 years ago

Hey there, thanks for bringing this up. First, I want to apologize for the inconvenience you've experienced as a result of this issue. Second, I'd like to apologize for the delay in getting back to you on this as well.

You're right, 3+ weeks is a pretty long time for something like this to be affecting people. Unfortunately I don't have an ETA on when this issue will be resolved, but I'll be keeping tabs on it going forward. Please feel free to add additional comments to the issues if you haven't already, or you can reach out to me directly. Again, I'd like to apologize for any trouble this caused you and others affected by this issue.

afshinmeh• 10 years ago

Gitlab can go down. Gitlab might have outages as well unless you are using the self-hosted version (although, in this case there is a self-hosted version of Github as well).

Just because Gitlab is not as famous as Github and hackers don't run their botnets on it, it doesn't mean it is safe or stable.

dagenleg• 10 years ago

Github is down --> check out the unicorn --> refresh hacker news --> refresh again --> shitpost in comments

karlmdavis• 10 years ago

The coding equivalent of accident rubbernecking.

euph0ria• 10 years ago

I wonder how many companies are unable to deploy new updates now because they rely on Github?

steve1011• 10 years ago

mine...

holtalanm2• 10 years ago

mine as well

samueloph• 10 years ago

mine.

Try to deploy -> bad key -> check github website -> go to hackernews

dbg31415• 10 years ago

Friends don't let friends deploy on Fridays... it's always a horrible idea.

smonff• 10 years ago

Especially in France where it's 5:27.

efrafa• 10 years ago

Deploying on Friday is fine if you can rollback quickly :)

yeukhon4• 10 years ago

Meanwhile, today is Friday the 13th.

danielsamuels• 10 years ago

Not if you've got your CI configured properly.

chrisallenlane• 10 years ago

_Friday the 13th_, no less.

yeukhon4• 10 years ago

Mine. Let this tree grow!

GabeIsman• 10 years ago

Mine :)

mkurz• 10 years ago

Someone was faster: https://news.ycombinator.com/item?id=13391883

mariusbutuc2• 10 years ago

…and this is how we remember how much we rely on GitHub, and how much value it provides. Every day.

yeukhon4• 10 years ago

It sort of becomes your Internet and computer. No computer no work. No Internet no VPN no work.

But in this case I guess investing in local mirror is a great investment.

ComputerGuru2• 10 years ago

Obligatory Gogs plug: https://gogs.io

It also has an awesome automatic mirror mode that can remain sync'd to a remkte repository. I don't care what your company does or how you run it, but please don't just rely on Github to host your code without a mirror somewhere. There are drawbacks to this incessant centralization.

VMG• 10 years ago

does it mirror issues, PRs, wiki pages etc?

ComputerGuru2• 10 years ago

I don't think so (we use other apps for those features) but Github's wiki is just a git repo, isn't it?

jonathanatx• 10 years ago

I'll have to check out the source code for their github replacement when github is back up.

mbertschler2• 10 years ago

https://try.gitea.io/gitea/gitea

johnnycarcin• 10 years ago

Just this week I setup gogs on my VPS to mirror all of my repos on github. It's part of my new year goal to avoid my dependence on 3rd party apps. Of course I still am depending on my VPS provider so...

the_duke5• 10 years ago

Just set up a Banana Pi or whatever at home to mirror your mirror.

dsp12343• 10 years ago

The nice thing about git is that it's distributed. You can place as many mirrors as you like. If you make the project and repo names the exact same as the official ones, then switching between them can be as easy as replacing a hostname in your dependency manager.

mbertschler2• 10 years ago

Instead of Gogs it is now recommended to use Gitea, since it has more activity (~400 more commits last time I checked) and is community driven: https://gitea.io/

According to their Blog: Gitea is a community fork of the popular self-hosted Git service Gogs. We’re a growing group of former Gogs users and contributors who found the single-maintainer management model of Gogs frustrating and thus decided to make an effort to build a more open and faster development model. https://blog.gitea.io/2016/12/welcome-to-gitea/

bisby4• 10 years ago

I would love this if gitea had a docker image for arm prebuilt. gogs/gogs-rpi is several versions behind gogs/gogs, but it at least exists.

I'm capable of building my own, but it's nice when there's already an official x86 version to have an arm version too.

almonj• 10 years ago

Friday the 13th...

Rooster61• 10 years ago

The day following a full moon...

ff10• 10 years ago

The day github had a major outage...

holtalanm2• 10 years ago

looks like it is back up now

mbaha2• 10 years ago

It's back. [END OF THREAD ?]

dajohnson892• 10 years ago

Since I can't push to my repo, I'm gonna say no, it's not back.

turshija• 10 years ago

Site came back online, but ssh/git still down.

philipDS• 10 years ago

Github has been down (or very slow) a lot the last few months. They used to have a good engineering reputation (at least with me), but it's been tough times over there.

analogmemory• 10 years ago

"Github is down" is the new compiling.

http://xkcd.com/303/

yeukhon4• 10 years ago

So there was an outage yesterday which required failover to another data center. Wonder if both are related. Either way I want to read their postmortem. GitHub's postmortem is usually very educational.

mariusbutuc2• 10 years ago

Currently 98.9212% app server availability in the last month.

That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last month.

And the trend of that percentage doesn't look that rosy.

[1] https://uptime.is/98.9212

mkaziz2• 10 years ago

I use JSPM on my projects and I have several dependencies on github repos that aren't in NPM. Anyone have any ideas on a fallback solution for when github goes out like this?

shepbook• 10 years ago

Is there a way to be able to point JSPM to a different, non-github repo? A remote clone of the repo might work then.

dsp12343• 10 years ago

Have a local git server, with an automatic process for pulling in the repositories that are your dependencies. Then use the repos from that local git server instead of github directly.

vitro• 10 years ago

I've configured one remote with two pushurls and set this remote to all my branches. This way I always push to BitBucket and GitHub at the same time. Just wondering, how this would work if I was not the single user contributing.

bisby4• 10 years ago

You'd have to either coordinate with the other users to have them do the same, or officially declare which one is "official" and which one is "backup" and set up something so the backup tracks the official (and then you only need to push to the official as well)

webmaven• 10 years ago

GitHub's Twitter status post at the start of the outage:

https://twitter.com/githubstatus/status/819940738452615168

Currently 442 retweets, 266 likes.

⋯

OK, they say it's back up:

https://twitter.com/githubstatus/status/819947685503442945

etimberg• 10 years ago

Still getting 503 First Byte Timeouts when trying to view content on raw.githubusercontent.com so definitely not all back yet

royka118• 10 years ago

Been working on a down detector type app, its like half baked at the mo, hopefully I can build up a data set that allows people to fairly compare the reliability of different SASS products such as github and bitbucket

https://young-forest-39406.herokuapp.com/sites

escapologybb• 10 years ago

I hope everybody on HN is doing okay with this recent outage, I'm here if anybody wants to talk.

Seriously, Github Outage Syndrome (GOS) can be horrible. If you don't talk about your outage feelings, they will eat you from the inside out.

Obviously anything you tell me will be kept in the strictest confidence, all without judgement.

Stay strong HN, Github will be back soon!

Edited to Add: Why they hate guys? Didn't mean to offend, thought the post said it all and could probably do with a little levity. My bad. Maybe.

srik• 10 years ago

Problems at quite a few places actually - http://downdetector.com/

dajohnson892• 10 years ago

OAuth API is still down.

kumarm• 10 years ago

Seems back up now.

achairapart• 10 years ago

A little bit OT, but having colored favicons based on the status is a nice touch!

https://status.github.com/images/status-icon-green.png

https://status.github.com/images/status-icon-orange.png

https://status.github.com/images/status-icon-red.png

mwpmaybe• 10 years ago

I'm envisioning some kind of caching forward proxy for git. Point your clones/fetches at it and it will proxy the connection to the source fork. If the source fork is down, it will serve its cached copy of the fork.

I found this[0] but it's prerelease and looks like it hasn't been touched in five years. Anyone know of any other solutions like this? Good idea? Bad idea?

0. https://github.com/rohanpm/ngitcached

mojitoking2• 10 years ago

GitHub adds value. They have a few engineering issues to solve given the negative attention such a popular code versioning system attracts. Although there are some outages due to this condition, the value added to our organization far outweighs these minor inconveniences.

Major Service Outage at GitHub

Related Stories