GitHub incident: ongoing issues with Actions, Issues and other Git operations

https://www.githubstatus.com/incidents/f0mhbz9xn497

by jaitsu • 3 years ago

204 124 3 years ago

githubstatus.com

joennlae• 3 years ago

You will probably notice it also when trying to push a new branch:

`error: failed to push some refs to ` when using --set-upstream

jaitsu2• 3 years ago

Indeed, I notice it when trying to push also:

  remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
  remote: fatal error in commit_refs
  To github.com:acme/foo.git
   ! [remote rejected] HEAD -> acme/foo (failure)
  error: failed to push some refs to 'github.com:acme/foo.git'

wun0ne• 3 years ago

Does GitHub provide uptime stats? Seems very unreliable recently.

steelaz• 3 years ago

https://www.githubstatus.com/history seems to show incidents in the last 3 months:

* March - 20 incidents

* April - 12 incidents

* May - 4 incidents (so far)

voytec2• 3 years ago

It's since M$FT took over, not so recently.

bobmaxup• 3 years ago

They used to, some screenshots of it existing here:

https://interface.lt/github-uptime-and-microsoft

capableweb6• 3 years ago

Let’s hope it’s temporary and GitHub error 500 won’t become their own version of Blue Screen of Death. In this case it would be Green Screen of Death (GSoD or GhSoD).

Heh, if anything it has gotten worse.

that_guy_iain4• 3 years ago

About 10 years ago someone said we should move to self hosting because bitbucket who we used were unreliable. I looked at the status page and say 2 hours of downtime over 3-months, while we have 3-4 days of downtime on our self hosted jenkins during the same time. I always think of that when I see people complain about services being unreliable. Often we see one or two problems in short span and forget about the months were we didn't see any issues.

GitHub is probably as reliable now as it has been for the past 10 years. It's always had downtime.

manojlds• 3 years ago

No it isn't. It's been very unstable in recent times ( 1 year or so)

darkwater2• 3 years ago

while we have 3-4 days of downtime on our self hosted jenkins during the same time

Wow. This sounds like a broken organization?

smcleod4• 3 years ago

We notice API outages that break core functionality every single day. It's gotten so bad over the last couple years.

grumple2• 3 years ago

It's a surprisingly unreliable service. It's been great for code management / reviews. But I can't imagine relying on it as the only option for deployments via CD. Imagine needing to deploy an important bug fix or a big project with a release date, but you can't because Github's having an outage.

__alexs• 3 years ago

Performance of the GitHub website in general feels increasingly bad.

meling• 3 years ago

Yes, my experience too. I feel it the most when reviewing code. It felt much snappier before.

PCloud• 3 years ago

You can checkout https://github.com/GitHub-Incident-History/GitHub-Incident-H... which provides all recorded GitHub incidents.

Kelteseth• 3 years ago

Why don't we all collectively mirror our repos to gitlab and switch development to there during the monthly outage?

geraldwhen2• 3 years ago

Enterprise says I have to use this unreliable garbage.

lol7682• 3 years ago

It's definitely no less reliable than GitLab, where a good 300GB of database data got deleted in production by accident...

namaria• 3 years ago

Substitute capital expenditure for operating expenses? With interest rates going up? It was already a tough sell with negative real rates...

bombolo3• 3 years ago

Yep. We could self host. But it's forbidden.

jonas-w2• 3 years ago

why particularly GitLab?

misnome3• 3 years ago

https://status.gitlab.com/ lists 27 incidents this year, so far.

alpaca128• 3 years ago

https://www.githubstatus.com/history lists 72 incidents since January

Hamuko• 3 years ago

GitLab is quite a bit more expensive. If you have GitHub Enterprise with the security features, it's $70/month/user whereas you'll need to get GitLab Ultimate for the security features, which is $99/month/user.

progval• 3 years ago

Feature mismatch of anything outside Git. And no one wrote the tooling needed to synchronize issues, pull/merge requests, ... back and forth.

Lutger• 3 years ago

Even so, for a lot of devs its still easier as a temporary collaboration point than sending patches via mail.

electroly• 3 years ago

GitHub outages aren't nearly long or often enough to consider this. Git is distributed, just keep working locally until GitHub is back up. GitHub outages are nowhere near the threshold of pain I'd require to introduce a second Git hosting provider to the mix.

Really, GitHub outages barely hurt at all. It's not like an AWS or Cloudflare outage which is more likely to be a production disaster. Every outage a bunch of people on HN start screaming about owning their own on-prem destiny or wondering why we're still on GitHub. Nothing changes because it's not nearly as bad as those people are making it out to be. Life is all about tradeoffs.

bombolo3• 3 years ago

Depends how your company is set up. Some people can't run tests locally and just push commits to have some magic run the tests online.

malka• 3 years ago

Again?

rvz2• 3 years ago

Yes. Once again. [0]

Due to GitHub's chronic unreliability, it is guaranteed to continue happening every month.

Looks like avoiding to 'centralize everything to GitHub' has aged very well [1] and at this point you would get better uptime with self-hosting instead of using GitHub.

Just ask many open source organizations like RedoxOS, ReactOS, wireguard, GNOME, KDE, etc.

[0] https://news.ycombinator.com/item?id=35817662

[1] https://news.ycombinator.com/item?id=22867803

perryizgr8• 3 years ago

This is a monthly incident at this point. Maybe they have a rogue cron job??

jonas-w2• 3 years ago

more like weekly when you look at the past incidents https://www.githubstatus.com/#past-incidents

vermilingua• 3 years ago

Now pages too

yurishimo• 3 years ago

Having issues with `git push` from The Netherlands. :(

JestUM• 3 years ago

Was unable to merge PRs.

Earlier, I also got GitHub PR comment emails about 6 hours late.

Whatever it is, it’s been happening for more than 6 hours.

toastal• 3 years ago

Could you apply the patch from the URL? Pull requests aren't really needed.

NikxDa• 3 years ago

The GitHub Status shows 14 incidents affecting Git Operations this year alone [1]. That's quite a lot, considering it's only May. I wonder if the outages were always this frequent and just get more publicity on here now, or whether there was a significant increase in outages fairly recently.

[1] https://www.githubstatus.com/history

capableweb6• 3 years ago

Many outages happen because something changed, and someone/something missed one of the effects of said change, bringing the platform down immediately, or after a while.

There was a period of time when GitHub basically didn't change, for years. And the platform was relatively stable (although "unicorns" (downtime)) still happened from time to time.

But nowhere near as often as now, but then again, there is a lot of more moving pieces now compared to before.

dinvlad• 3 years ago

I wonder if it has anything to do with layoffs, if crucial Ops people left

vb-8448• 3 years ago

i wonder if all these incidents are related to azure devops integration

voytec2• 3 years ago

They are adding affected services to the status entry title (started with Issues, Actions, Operations). Can't even do a simple push due to this so-called "degraded performance".

meindnoch2• 3 years ago

Thankfully git is a distributed version control system, so such outages are not of major concern.

ninepoints• 3 years ago

I mean, you joke but that's actually fairly true. P4 users always notice when the central server goes down because you can't reliably look at changelist history, draft CLs, and do a host of other operations that are possible on git locally. (using a central VCS confers other advantages of course).

capableweb6• 3 years ago

Ignoring the fact that what people actually do with GitHub, git is such a small part. Issues, PRs, CI/CD and basically everything that isn't git, doesn't happen over git (besides the wiki, which somehow miraculously actually is via git).

Some people have their entire roadmap in GitHub, and every single bug report / feature request, without any offline backup. Don't ask me why, I don't get it. Especially since they have proven for the last few years that they cannot keep the platform up in a stable manner.

smcleod4• 3 years ago

You're kidding right? People / companies rely on git being available for deployments, builds, config changes, developer workflow etc

meindnoch2• 3 years ago

But "git" is available. It's most likely under /usr/bin.

ta988• 3 years ago

Youa are confusing git and github.

theobr• 3 years ago

I haven't been able to push for a bit now

xmdx• 3 years ago

Been off work for a month-ish. Everything is as it was I see.

candiddevmike2• 3 years ago

Really looking forward to kicking the tires on Gitea next month.

dindresto• 3 years ago

We've switched to selfhosted Gitea last month, no regrets. Only the CI story could be a bit better. We're currently using Woodpecker but need macOS runners, and Woodpecker's "local" agent implementation is still unstable. I'm watching Gitea Actions' progress with great interest.

capableweb6• 3 years ago

Did you try the macOS runners? Been using Linux, Windows and macOS runners without any issues for quite some time.

hnarn5• 3 years ago

Any reason to use Gitea over the Forgejo fork? Based on the drama I'd rather use Forgejo if they're functionally equal.[1]

[1] https://forgejo.org/faq/#why-was-forgejo-created

justinclift• 3 years ago

"We gave ChatGPT root access to our infrastructure servers, and unexpectedly it crashed everything."

That's almost believable at this point. ;)

indeyets• 3 years ago

"Actions, Issues" are not "Git operations".

would sound better without "other" in title

jaitsu2• 3 years ago

good catch

pavo-etc• 3 years ago

Wondered why I couldn't push my notes, appears that pushing commits is also impacted.

arkitaip• 3 years ago

Goddamn and here I thought I was getting errors because the files i pushed were to large or something.

linhns• 3 years ago

Cannot create a new repo

jelling• 3 years ago

CivitAI.com is down and Hugging Face is having 502s for some operations. Perhaps there is a larger issue?

scottmf• 3 years ago

Wow huggingface is down completely

idk12• 3 years ago

I debugged for one min at 11:59 trying to push, and then my eat-lunch notification came in at 12:00 and I opened hackernews with a tuna sandwich and this is super helpful because it means I wont need to debug it locally for 10 mins before figuring out Github is down.

Edit - Just want to clarify when I say "opened hackernews with a tuna sandwich" I want to clear up that I did indeed full on mash the keyboard letters with my sandwich. It's costing me a fortune in keyboards every day and it's ruining my sandwich most days as well, I think I have an issue.

koolba• 3 years ago

Ideally both GitHub and HN would be down at the same time. With nowhere to discuss the former, people might actually get something done.

candiddevmike2• 3 years ago

This is why I don't touch other people's keyboards.

idk12• 3 years ago

Well from another perspective, a bootstrapping perspective, and hackernews does like a bootstrapping perspective, I could make a case that a good meal could pick licked out of most keyboards, saving money on lunch once a week.

capableweb6• 3 years ago

Little bit of bacteria helps build up your immune system for when you really need it :)

brookst• 3 years ago

Yeah, this is a good argument for licking them instead.

taf2• 3 years ago

Easy to fix just adjust your system prompt add the following:

“Never ever type on keyboard with sandwich not even the most delicious tuna sandwich”

ht85• 3 years ago

Next time try coffee, it's another worthy experience.

vienzo• 3 years ago

Interested to hear whether anyone actually managed getting some Client Credits as per their SLA [1]? Over the last quarter they probably went sub 99.9% in some services.

[1] https://github.com/customer-terms/github-online-services-sla

papito• 3 years ago

Perhaps everyone should stop complaining and be thankful for a chill morning. You can't create a PR right now - go get a pastry and some fresh air. Be in the moment for once. It's beautiful outside*

* Where I am

Vasniktel2• 3 years ago

Great to be here. Next week same time?

shibel2• 3 years ago

Rumor has it Microsoft is pushing them to move their infrastructure to Azure, which explains much of the frequent downtimes lately.

maccard• 3 years ago

Where'd you get this from?

shibel2• 3 years ago

1) A friend in DevOps who supposedly has contacts there.

2) It's not like this push is a big secret: https://www.cnbc.com/2022/10/12/microsoft-github-relying-mor...

p-o• 3 years ago

Maybe it's the case, but Github has always had issues with uptime and stability.

hnarn5• 3 years ago

"Always"? I don't remember it being nearly as frequent before they were bought by Microsoft, some data to back this up would be nice.

KomoD2• 3 years ago

Where is their infra right now?

michaelmure• 3 years ago

If that makes you mad, I still need help with https://github.com/MichaelMure/git-bug ;-) Coming at some point, kanban and pull-request support, offline-first!

everfrustrated• 3 years ago

From the outside, it appears GitHub doesn't have any internal sharding going on. Outages always affect _all_ repos.

Architecturally this seems rather sub optimal?

EG AWS doesn't roll out changes globally - they start with an internal shard within a region and progressively roll out to more shards and more regions.

Why do GitHub not do the same?

kjuulh• 3 years ago

last I heard they've sharded some stuff, but some of the core tables/databases are still using a monolithic architecture.

And I guess you're only as strong as your weakest link, which can be not that bad, that is, if it isn't your core tables.

Though take this with a grain of salt, this is mostly hearsay =D

RamblingCTO3• 3 years ago

Hm, we had this last week as well. Guess who had a demo yesterday, needed to deploy things on Thursday/Friday and couldn't :(

capableweb6• 3 years ago

If this is your first time being affected, I guess you have a good reason to adjust your deployment structure so you can deploy manually if needed.

If this is not the first time, hopefully at least it will be the last one :)

RamblingCTO3• 3 years ago

No worries, we can. But I like to rely on automation to spend my time on other stuff. Worked out good in the end though ;)

hnarn5• 3 years ago

Maybe a good time to investigate whether it's possible to hold those demos without relying on third party services.

RamblingCTO3• 3 years ago

You're always relying on third parties. Always. Except if you run it locally. We're way beyond that. I deployed to production just fine. It's just a helper. It adds to the stress tho.

kwyjibo_hunter• 3 years ago

They’ve convinced me to shut off the GitHub status alerts I get on slack (or at least move them to their own channel that I can ignore).

Vasniktel2• 3 years ago

2 hours going strong!

rvz2• 3 years ago

Once again another GitHub incident and 4 days later before the last one [0], GitHub Actions goes down.

You are better off self-hosting at this point, rather than centralizing everything to GitHub [1] as it is just chronically unreliable for years ever since the Microsoft acquisition.

[0] https://news.ycombinator.com/item?id=35817998

[1] https://news.ycombinator.com/item?id=22867803

hnarn5• 3 years ago

You are better off self-hosting at this point

For medium/large companies, I fully agree. For smaller projects, specifically open source projects, I'd look at something like https://codeberg.org/

hospitalJail2• 3 years ago

I'm the lone person at my team that still believes in keeping most of our stuff local, with online versions as primarily backup.

Every time some global service goes down, or internal internet/intranet goes down, there is a security breach, or a WFH person has a power outage I'm reminded I'm right.

I'm no luddite, these services make you dependent on them. The worst thing I'm dependent on here is a bad computer. We have backups and keep our files on our network, so it seems fine. We are slowly moving to an online system, and I'm constantly reminded all the problems shifting online.

Meanwhile, if I had a linux server, we would be in control of our own destiny.

goodoldneon• 3 years ago

I don't know about you, but the stuff I self-manage usually has worse uptime than SaaS products

sofixa• 3 years ago

Really depends on the "stuff". GitLab pretty much manages itself through their Helm and Omnibus installs.

hospitalJail2• 3 years ago

Depends.

Having our programs offline mean I can run them, even if the internet isnt working.

Instead of getting 0 data from downtime, I can still get the data, run the programs, and give it to the person who needs it.

If we are fully online, if the servers are down, we basically lose the entire time.

Not to mention, I think 'uptime' is a pretty optimistic number, unusable slow service doesnt seem to hit any metrics I'm aware of.

dijit• 3 years ago

Doesn't matter if your uptime is 80% as long as that 20% of downtime is happening when nobody is working

Additionally an 80% uptime architecture is really simple to maintain and restore and so on.

Complexity increases exponentially the more 9's you add.

solotronics• 3 years ago

GIT is actually a great protocol for keeping distributed copies of code. You can pretty easily with bash cycle through a list of backup urls for a git repo, looking for updates.

_flux• 3 years ago

Git itself is nice, but then there's the issue tracker and CI that are more difficult to setup.

KomoD2• 3 years ago

This happens constantly, what is the Github team doing?

voynich• 3 years ago

I've started working on a Forgejo instance for myself (Gitea fork). It's honestly disappointing how bad GitHub has gotten, just in terms of uptime anymore. I hope they get their stuff together.

yuuta• 3 years ago

Ever since Microsoft acquisition

dclowd9901• 3 years ago

Can anyone from GH weigh in on this? We've had several major outages from GH over the last month or two, and the company has been completely silent on the causes, as well as any sort of remediation steps to fix stability.

As a somewhat large size org, we're now exploring other options for code hosting.

GitHub incident: ongoing issues with Actions, Issues and other Git operations

Related Stories