Incident with multple GitHub services

https://www.githubstatus.com/incidents/myrbk7jvvs6p

by bwannasek • 3 months ago

267 132 3 months ago

githubstatus.com

supakeen• 3 months ago

I mean; this is the normal mode of operation for GitHub at this point.

napolux• 3 months ago

0 nines.

causal• 3 months ago

9 nines found somewhere after the decimal point if you measure with enough precision

argee• 3 months ago

I moved to Gitlab a while ago. It's a whole new level of freedom not having to pay for self-hosted CI runners.

cjonas4• 3 months ago

It would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.

waiwai933• 3 months ago

https://mrshu.github.io/github-statuses/ suggests that their combined uptime doesn't even meet 1 nine, let alone 2.

cjonas4• 3 months ago

ya i was just doing the math on their chart for the git operations. I added up 14.93 hours combined hours, which puts them WAY lower than the reported 99.7 metric they show right next to it.

So based on their own reporting, the uptime number should be 99.31. Which means only like 6 additional hours and they'd fall below 99.0%

cjonas4• 3 months ago

also I never had considered that breaking your up-time into a bunch of different components is just a strategy to make your SRE look better than it actually is. The combined up-time tells the real story (88%!). Thanks for the link

CamouflagedKiwi3• 3 months ago

The intersection of uptime across every possible service they offer isn't a particularly great metric. I get the point that they are doing badly, but it makes it look worse than I think it really is.

What I would like to see is a combined uptime for "code services", basically Git+Webhooks+API+Issues+PRs, which corresponds to a set of user workflows that really should be their bread & butter, without highlighting things you might not care about (Codespaces, Copilot).

roblh• 3 months ago

GitHub is going for “eight 8’s” at this rate.

fishgoesblub• 3 months ago

At this point it'll be better to have alerts for when GitHub is online, rather than offline.

josefritzishere• 3 months ago

Microslop is destroying Github

AnkerSkallebank• 3 months ago

Some of my jobs are completing, some are failing. Seems to be random. Kind of wish they would just fail outright, instead of running for 10 minutes and then failing.

ossa-ma• 3 months ago

Seems like outages are increasingly more frequent nowadays. Obviously, this is not the best state of affairs, and developers should not be limited by services. In the meantime I've been experimenting with building third spaces for people to chill while they wait for the services they are dependent on to go back up.

The first one I've built is a little ASCII hangout for Claude @ https://clawdpenguin.com but threads like this make me want to build it for Github too.

0xbadcafebee3• 3 months ago

I am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)

pocksuppet• 3 months ago

Codeberg and Sourcehut are doing it for free, for open source. Corporate probably won't ever move off Github, because they need the prestige of using Github - the actual service quality is completely irrelevant. This is an aspect of the enshittocene epoch - I repeat, quality is irrelevant to corporates.

apparatur2• 3 months ago

Corporate will move off GitHub as soon as it loses prestige and they will move on to the next thing.

0xbadcafebee3• 3 months ago

Sourcehut isn't free and has weird UX, Codeberg is free but has poor performance and weirdly over-moderates discussions. I know corporate will always suck, I'm just talking about having something that approximates the "old GitHub" for personal/professional use

apparatur2• 3 months ago

Gitea has a paid plan. On Forgejo forums you can find 3rd party offers of paid hosting as well.

0xbadcafebee3• 3 months ago

I actually tried to use Gitea and their login page wouldn't work, so that told me all I needed to know about them

linhns• 3 months ago

Business as usual.

ChrisArchitect• 3 months ago

Multple 9s

embedding-shape5• 3 months ago

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally.

This is probably the first time I felt vindicated with my self-hosting move literally the day after I finished the migration, very pleasant feeling. Usually it takes a month or two before I get here.

shevy-java2• 3 months ago

Interesting. I speculated not long ago that Microsoft is really taking a dive here, and other companies may look to provide better alternatives to GitHub, as one idea. Today I read your comment about self-hosting here; while that is not quite what I compared or had in mind, it is interesting to read about it, of people who go that route. Microsoft is really putting themselves into trouble in the last year or two. Some things no longer work, so much is clear here.

yakattak• 3 months ago

I moved my forge to my home, outside of a little stress getting all the containers wrangled it was pretty effortless to setup Forgejo.

I do need a good backup solution though, that’s one thing I’m missing.

TranquilMarmot2• 3 months ago

I use https://github.com/garethgeorge/backrest to manage nightly encrypted backups of my Forgejo instance to a Hetzner Storage Box. <$4/mo for 500GB of storage. It's also where I back up my Immich library to.

Immich automatically dumps its DB every day, for Forgejo I have a little script that runs as part of the Backrest backup that does a pgdumb of the database before doing the backup.

It works great, I even had to do disaster recovery on it once and it went smooth.

lone-cloud2• 3 months ago

I use rclone + backblaze. You get 10GB for free which is more than enough for self-hosted stuff.

lisplist• 3 months ago

The only problem I've found with Forgejo is a lack of fine grained permissions and also the lack of an API for pulling action invocations. The actions log api endpoints are present in gitea from what I can tell.

mfenniak• 3 months ago

Forgejo 15 was just released last week with repo-specific access tokens. More to come in the future.

cedws2• 3 months ago

The idea of a homelab is appealing to me, but then I actually start building one and get tired of it quickly. When I’ve been fixing broken systems at work all day I don’t really want to have to be my own sysadmin too.

I’ve got a nice and powerful Minisforum on my desk that I bought at Christmas not even switched on.

embedding-shape5• 3 months ago

I've tried for 15 years to have my homelab, but always get lost in the complexity after a year or so, in the past. About 3 years ago I gave NixOS a try instead for managing everything, which suddenly made everything easier (counter-intuitively perhaps) as now I can come back after months and still understand where everything is and how it works after just reading.

Setting up Forgejo + runners declaratively is probably ~100 lines in total, and doesn't matter I forget how it works, just have to spend five minutes reading to catch up after I come back in 6 months to change/fix something.

I think the trick to avoid getting tired of it is trying to just make it as simple as humanly possible. The less stuff you have, the easier it gets, at least that's intuitive :)

kivle• 3 months ago

With the help of coding agents it's easier than ever. Just get Claude/Codex to create Helm Charts / Docker Compose files for you. Struggle with some command line juggling to fix some obscure error? An agent can mostly help you in no-time.

prmoustache• 3 months ago

There isn't much work or maintenance to do really. When you are the sole user everything is over sized and if it is only accessible at home you can be lazy with updates and security anyway.

VerTiGo_Etrex• 3 months ago

When I’ve been fixing broken systems at work all day I don’t really want to have to be my own sysadmin too.

There’s only one solution to this.

Quit your job.

neilv• 3 months ago

I self-host Forgejo for personal and indie-startup purposes, and like it well enough.

The downside with that is it misses one of the key purposes of GitHub: posturing for job-hunting/hopping. It's another performative checkbox, like memorizing Leetcode and practicing delivery for brogrammer interviews.

If you don't appear active on GitHub specifically (not even Codeberg, GitLab, nor something else), you're going to get dismissed from a lot of job applications, with "do you even lift, bro" style dissing, from people who have very simple conceptions of what software engineers do, and why.

radlad2• 3 months ago

There is a fairly straightforward feature in Forgejo to sync your repos to Github, if that's what you want to do. It's not perfect, of course, but should help to advertise your projects and keep your activity heatmap green.

I mostly use Forgejo for my private repos, which are free at Github, but with many limitations. One month I burned all my private CI tokens on the 1st due to a hung Mac runner. Love not having to worry about this now!

gill-bates• 3 months ago

I get that. To counter it I usually try to have at least one public repo on my Forgejo instance and link to that on my resume/LinkedIn. It helps that I'm angling for security/infra positions so the self-hosting aspect actually helps but even without that I would imagine it signals something. Maybe not ideal for the most mainstream jobs (whatever that even means...), but I suspect some people will be intrigued by the initiative.

Edit: to the "do you even lift bro", the response becomes "yeah man, I've built my own gym - oh, you go to Planet Fitness? Good luck."

8cvor6j844qw_d6• 3 months ago

If you don't appear active on GitHub specifically... you're going to get dismissed from a lot of job applications

Sometimes wonder if my coursemates back in the days, who automated commits to private repos just to keep the green box packed, actually got any mileage out of it.

bmitc• 3 months ago

Fine with me. Not the type of jobs I want anyway.

radlad2• 3 months ago

I recently did this as well and one of the things that has struck me is just how fast Actions are compared to Github!

That said, I've got Linux and macOS setup with a Mac Mini (using a Claude-generated Ansible task file), but configuring a Windows VM seemed a bit painful. You didn't happen to find anything to simplify the deployment process here, did you?

embedding-shape5• 3 months ago

You didn't happen to find anything to simplify the deployment process here, did you?

No, unfortunately not, the Windows VM setup + Forgejo Windows runner was the most painful thing for me to setup, no doubt. It's just such a hassle to reliably set things up, even getting logs out of it was trouble... To be fair, my Mac Mini was manually setup at first, then I have Nix on top of it, while Windows I've 100% automated it, so not entirely fair comparison, automating the Mac Mini setup would be similarly harsh I think. But it's a mix-match of Nix for configuring the VM and booting it, XML files for "autounattend" setup, ps1 bootstrapping scripts and .cmd script for finalizing, a big mess.

colechristensen• 3 months ago

Instability aside I found several things about GitHub awkward, annoying, or missing features so I spent a month building my own. I think we're going to be seeing a lot more of this.

rvz2• 3 months ago

Self hosting was the correct solution.

6 years early [0] and you have better uptime than GitHub.

[0] https://news.ycombinator.com/item?id=22867803

dietr1ch• 3 months ago

My Raspberries (and OrangePi) have better availability than github, and if were to be down I'd be out of power/internet and wouldn't be able to work much anyway.

koyote2• 3 months ago

And once you start self-hosting, you realise how slow the 'modern' web actually is.

I host forgejo on a single NUC with a bunch of other stuff in Proxmox, the page loads in 6ms! Immich is not quite as fast but still a ton faster than Google photos.

oefrha• 3 months ago

That’s not a given. Self-hosted GitLab on my pretty good hardware is still slow. I just opened a very small repo with ~20 files and ~5 commits. The page spun for 5s+ before showing me the directory listing and readme. Subsequent loads are faster (~1s) but still not instant.

snailmailman• 3 months ago

I've been running my own private forgejo instance for a while now. I host all my own private side projects and stuff there. Its a much more pleasant experience than github, if only because it has higher than 90% uptime. The UI is mostly identical otherwise.

The number of consistent issues i've had with anything github-related lately is crazy. Even just browsing their site is difficult sometimes with slow loads that often just hang entirely.

lone-cloud2• 3 months ago

Are you me? Somebody was talking about gitea on here yesterday and I also ended up self-hosting and moving all of my private projects to Ferjero yesterday after a bit of research. I can't bring myself to move public projects due to job prospects + GitHub network effect. Otherwise I'm role playing as a system admin now with 20 local services for various things I need. I think the most important thing is to have regular backups as you're now in charge of keeping your data from getting lost.

estimator7292• 3 months ago

My GitLab server at home is so nice. My last job ran a self-hosted GitLab on AWS with CI runners on-prem, which was a delight to work with.

New job runs on GitHub. I frequently have to stop work and wait for GitHub to recover before I can checkout some dependency or push a commit. It's outrageous.

dankobgd• 3 months ago

Don't worry, status page says that it's 100% working - green color, all good. even though i can't access a static page

surya20062• 3 months ago

what are the good alternatives available for github i find some alternative but as long as widely people use github i cant use other service right like i cant share my alternative to other developer and force him to use this for me. so i feel like i locked in even i want to move i can't

embedding-shape5• 3 months ago

Huh? Why not? Say "My git repository is here $URL" then if they want to visit and/or clone it, they'll do that, otherwise don't, why does it matter?

Sure, if you're out after reaching the most people, gaining stars or otherwise try to attract "popularity" rather than just sharing and collaborate on code, then I'd understand what you mean. But then I'd begin with questioning your motivation first, that'll be a deeper issue than what SCM platform you use.

mghackerlady• 3 months ago

gitlab is about as close as you'll get

pxc• 3 months ago

GitLab annoys me in tons of ways, but I feel it's generally better than GitHub in lots of ways.

shimman• 3 months ago

codeberg.org is a thing, and it's perfectly suited for open source projects. Many neovim plugins and home lab tech I use are hosted on codeberg with no issues. If you just want to github as social media, you will never be happy.

nosioptar• 3 months ago

I'm probably going to use source hut in the future. It allows contributions via email without an account requirement.

https://sourcehut.org/

nerdypepper2• 3 months ago

give tangled.org a go perhaps. its got the self-hostability that cgit/forgejo does and a the social bits that github does.

shevy-java2• 3 months ago

Microsoft again.

I think it is time that Microsoft lets go of GitHub. They are handling it too poorly.

surya20062• 3 months ago

even vercel also have more downtime nowadays

buildbot• 3 months ago

Anyone also seeing Active Directory/Entra issues?

throwatdem123112• 3 months ago

Seems like they just can’t deal with the absolute deluge of AI vomit being uploaded every day.

Good riddance I hope it completely destroys them.

mijoharas• 3 months ago

Are you taking about what they write to run the service? Because looking at the uptime, and considering it's Microslop, I wouldn't be surprised.

throwatdem123112• 3 months ago

What they write and the extra demand from vibe coders.

LorenDB• 3 months ago

https://mrshu.github.io/github-statuses/ says they are down to 88.15% uptime. Even when you consider uptime of individual components, their best is 99.78%, so two nines.

Ygg2• 3 months ago

I see Microsoft mandated AI is doing wonders. For self hosters and Linux enthusiasts.

jpleger• 3 months ago

I wonder if there is any correlation between them moving towards Azure.

https://thenewstack.io/github-will-prioritize-migrating-to-a...

frakkingcylons• 3 months ago

They are dealing with vastly more activity as a result of AI usage. It's that simple.

sharts• 3 months ago

They’re pushing out AI slop as production services

eddyg• 3 months ago

The scale of growth they’re dealing with is insane.

“There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)

GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.”

Source: GitHub COO on April 3, 2026. https://x.com/kdaigle/status/2040164759836778878

CodesInChaos• 3 months ago

What's causing that growth? AI bots spamming commits?

rpmisms• 3 months ago

That really does put it in perspective. Wow.

tomjen3• 3 months ago

At this point it should almost be news when it works.

jasoncartwright3• 3 months ago

Just cancelled my GitHub Copilot Pro+ year subscription. Removal of Opus 4.6 stung, but the repeated continued downtime makes it unusable for me. Very disappointed.

No fuss instant refund of my unused subscription (£160) appreciated.

miltonlaxer• 3 months ago

What will you use now?

jasoncartwright3• 3 months ago

Claude Code

tgrowazay• 3 months ago

Doesn’t GitHub Copilot Pro+ only have month-to-month payment option?

Only Pro (without plus) can be paid annually for some reason.

jasoncartwright3• 3 months ago

I paid 390 USD for a year Pro+ subscription in November 2025.

I used all the 'Premium Requests' every month on (mainly) Opus 4.5 & 4.6. From what I've read on here it seems I was probably a rather unprofitable customer - it felt like a steal.

pkaye• 3 months ago

Pro+ does have a annual plan but recently they paused or dropped the annual plans because they are trying to adjust the pricing model.

bakies• 3 months ago

I definitely have better uptime hosting my own gitea instance. It's faster too. It's basically a knock off GitHub. Plus with privacy concerns, I'm just happier overall. Easy setup, all I did was deploy the helm chart.

CamouflagedKiwi3• 3 months ago

I wondered. We'd seen for most of today that Actions were slow to trigger, I had at least one that was just missed, it felt like something was definitely off but the status was green all day until this.

nhhvhy• 3 months ago

If the day ends in Y…

jonnonz• 3 months ago

Well I suppose they are finding out if you lay off too many people the IP of how the system works goes out the door with them.

poplarsol• 3 months ago

Azure webapp deploys are also trash right now. Microsoft needs to stop slathering h1b copilot slop and get basic things like Windows patches working.

frereubu• 3 months ago

We have pretty basic needs - git repos + actions - and a bit of downtime here and there doesn't really affect us too much because we're not constantly committing and deploying, but even we're looking around for alternatives now.

Also, looks like people might be pummelling the SourceHut servers looking for an alternative: https://sr.ht/ is down. (Edit: was down when I wrote that, back up now).

nerdypepper2• 3 months ago

tangled.org maybe?

delusional• 3 months ago

Once that 10x developer velocity from AI kicks in, I'm sure github stability improves. Did you know AI finally makes it economical to fix all the little bugs?

Groxx• 3 months ago

So... three incidents today, all of them ~1h or longer, and everything's green for the day with "no recorded downtime".

These don't really look any different than past incidents which have red bars on their respective days, except maybe that those tended to be several hours.

What do the green bars even mean? Are they changed to non-green retroactively if people complain enough or something? So far as I can tell, literally none of the previous green days have any incident shown in the mouse-over, but there are multiple for today only, so I kinda have to assume the mouse-overs are conveniently "forgotten" or all incidents become non-green and they just don't bother informing anyone on the same day. Either seems intentionally misleading.

agartner• 3 months ago

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf

We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC.

We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.

nulltrace• 3 months ago

Downtime is one thing. Silently reverting commits on your default branch is something else entirely.

darknavi• 3 months ago

Similar here. Somewhat ironic that a tool that was supposed to be preventing merge conflicts was authoring completely mangled commits to our mainline branch.

robertwt7• 3 months ago

yeah this is crazy we had many PRs reverted as well on many repos. downtime is 1 thing, but reverting PRs is failure on another level

CoastalCoder• 3 months ago

Really makes one appreciate that concept of A.C.I.D. database transactions.

lucasqueiroz• 3 months ago

We've also seen quite a few commits disappear from main, the status of the PRs continued on merged. Was stressful.

olingern• 3 months ago

Same, we got an email with a PDF with the affected commits and how to fix. What an absolute clown show.

hedayet• 3 months ago

Is Github losing any significant business from all these outages?

Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days.

Happy to be corrected about my perception too.

braiamp• 3 months ago

They are entrenched enough that it's wrote off as cost of doing business. Big business have their internal instances so they are "insulated", everyone else isn't as critical and have the resources to do an internal solution or move.

ozarkerD• 3 months ago

Org instances have been affected by these outages too. Including the one today

datadrivenangel• 3 months ago

They have on-prem / dedicated instances? I thought that microsoft only offered that through their Azure DevOps git offering.

bandrami• 3 months ago

And as recently as two or three years ago it was universally agreed that the only way to reliably and securely deliver software was via repeatable builds with an attested chain of custody and auditable bill of materials and everybody just gave up on that completely when the LLMs got somewhat better.

alper• 3 months ago

From Github to Gitlab? Out of the frying pan into the fire.

It'd be nice if there was a good alternative for anybody working at scale.

k4rli• 3 months ago

Codeberg or self-hosted Gitea seems a perfect replacement.

chrisweekly• 3 months ago

mods: title tupo "multple"

x0ruman• 3 months ago

We lost about a day of git history across several repos on Bitbucket a while back-not an outage, a data issue on their side. Local clones saved most of it, but issues and PRs from that window were just gone. That’s roughly why I started building gitbacker as a side project. Turns out the ‘back up the repo’ part is easy; the metadata is where it gets interesting.

BhavdeepSethi• 3 months ago

I'd really like to read the Post Mortem doc for this. I'm still shocked how they allowed this to happen in production.

eshack94• 3 months ago

At this point, I feel like there should be a HN post whenever there ISN'T an issue with some GitHub service. Otherwise, it's business as usual...

redwood• 3 months ago

Guessing the sheer volume of pull requests related to AI code jitter is leading to instability of this Microsoft product.

alansaber• 3 months ago

How much of this github downtime is due to git actions?

rvz2• 3 months ago

If any company had so many incidents like GitHub, they would be out of business by now.

Another outage at GitHub thanks to the AI agents like (Tay.ai, GitHub Copilot) at the helm with no employees maintaining it as I said before [0]

"Centralizing" everything to GitHub in 2026 is starting to look quite unthinkable.

[0] https://news.ycombinator.com/item?id=47592294

Incident with multple GitHub services

Related Stories