Days without GitHub incidents

https://www.dayswithoutgithubincident.com/

by goalieca • 3 months ago

384 163 3 months ago

tayo42• 3 months ago

I wonder what morale is like at github. This is like gamer level hating

jedberg8• 3 months ago

It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.

I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.

But they need to make some changes quickly.

giwook4• 3 months ago

I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.

JamesSwift• 3 months ago

I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.

Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.

jcgrillo4• 3 months ago

The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.

zipy1242• 3 months ago

But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.

Scubabear685• 3 months ago

"AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that....".

What "brunt"? These are not large numbers.

kedihacker• 3 months ago

I don't think aggregating the whole platform into one number is fair. It's like adding the whole aws into one number

blinded• 3 months ago

Github has far less services and regions that AWS.

stevekemp• 3 months ago

On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.

Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.

But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.

tensegrist2• 3 months ago

splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse

there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens

remus• 3 months ago

splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse

This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.

bluetidepro• 3 months ago

It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.

einsteinx22• 3 months ago

Ironically I’ve never found official status pages to be all that accurate either since companies love to exclude all kinds of outages from counting towards uptime. Anthropic is hilariously egregious about that as a recent example I can think of, but I assume GitHub does the same since it’s so common in the industry.

8organicbits2• 3 months ago

I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.

loloquwowndueo• 3 months ago

You’re kind of an outlier - nobody wants AI but Actions are core for tons of workflows and deployment pipelines. Everyone bought into the “only robots can deploy” mantra (correctly IMO, it’s a huge time and friction saver) only to be bit in the ass by the platform being so u reliable they can be stuck for days without deploys.

jasonvorhe• 3 months ago

If S3, EC2, EKS and RDB alone had a similar uptime as all of Github right now, we'd all know.

No one cares that much if repo wikis, commit stats or gist had these issues. It's the combination of inter-dependent services that are used in combination, like PRs, actions, discussions, etc.

If one were to build a single percentage for each of these components of both systems, github would still lose. Maybe it's a few days without outages more but this isn't a comparison.

llbeansandrice• 3 months ago

From a user perspective this makes sense. But if you’re MSFT or GitHub this number is pretty embarrassing.

They would love if everyone on the platform used all of the features and had massive lock-in right? So if some part of that is always broken, it’s not a confidence booster for users to adopt more of the feature set.

Sure the more things you use the more likely it is that one has an issue but clearly stability isn’t a goal for these type of companies anymore.

sharts• 3 months ago

Why embarrassing? This is normal MSFT

timoxley• 3 months ago

end user doesn't care, if it don't work it don't work

mproud• 3 months ago

Supposedly commits on GitHub are up 14x YoY.

dotwaffle• 3 months ago

Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.

reaperducer• 3 months ago

Supposedly commits on GitHub are up 14x YoY.

So?

If Microsoft can't scale, who can?

If it can't provide the service, it should stop selling until it can.

This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.

jh00ker• 3 months ago

If it can't provide the service, it should stop selling until it can.

You literally cannot buy GitHub Copilot right now [1].

1: https://github.com/features/copilot/plans

Scubabear685• 3 months ago

Precisely.

If Microsoft can't scale something like Git 14x, then the problem is with Microsoft.

ex-aws-dude• 3 months ago

They are getting spammed by AI agents?

lwansbrough• 3 months ago

Yes. There’s no other explanation for 14x, that’s nuts.

masklinn2• 3 months ago

Is it spam when they’ve been pushing for this shit and putting AI prompt everywhere fir a year or more?

yonatan8070• 3 months ago

I'd say certianly yes. At my company we've set up an experiment: we've forked our monorepo and gave several agents (OpenClaw, Claude code, GH Copilot reviewer, Codex) full R/W access and a Slack channel where we give them tasks and they carry them out. The fork shows 490 commits ahead, 10 commits behind our real repo (we're only a couple people, and use "Squash and merge" on PRs).

They naturally produce bugs at an astonishing rate, and we don't review the code ourselves, but the project is growing faster than their context windows, and I believe we'll drop it soon.

perrygeo• 3 months ago

14x is insane, especially since the quality and quantity of IRL software has barely budged.

One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?

masklinn2• 3 months ago

The same thing we’ve done with every other productivity increase in a world based on unfettered growth: garbage.

Scubabear685• 3 months ago

I really don't understand people saying that this is due to AI commits and it is all the volume's fault.

A volume increase that is a single order of magnitude (which 14x is) should not result in this level of failures.

When I compare what Github does and the volumes vs social media companies, payment companies, video platforms, etc, it just doesn't make sense that it is just a volume problem.

It looks a lot more like a platform that already has baseline issues that are compounded by increased volume.

tardedmeme3• 3 months ago

What happens at your job if there's suddenly 14 times as much load?

dijit5• 3 months ago

Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.

Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.

Especially if they can't hold up their end of the deal.

Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.

Imustaskforhelp• 3 months ago

I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.

But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.

At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.

EduardoBautista• 3 months ago

Defending a multi-trillion dollar company you mean (Microsoft).

IshKebab2• 3 months ago

Maybe that's good-will doing the work?

Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.

kevmo• 3 months ago

You're right, but that GitHub is dead.

Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.

Any goodwill they earned has been spent.

gordon_freeman• 3 months ago

they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.

collinmanderson• 3 months ago

https://xkcd.com/1150/

otterley18• 3 months ago

Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?

It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.

What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

logicchains• 3 months ago

What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?

There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.

VirusNewbie6• 3 months ago

Executives have made a choice to not pay for top talent at Microsoft Azure and Github.

turtlebits• 3 months ago

#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.

Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).

StableAlkyne• 3 months ago

When did OP blame the people involved personally?

If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.

MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.

ofjcihen2• 3 months ago

OP didn’t blame the staff. His focus is on the company.

Invoking individual workers well-being to defend a billion dollar company is also very strange.

ebiester• 3 months ago

I have all the empathy for people in the world.

A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.

estimator7292• 3 months ago

If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.

GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.

To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.

nimih• 3 months ago

I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.

pluc2• 3 months ago

I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.

saghm2• 3 months ago

As far as I'm concerned, any benefit of the doubt I might have had for Microsoft is gone after this debaucle: https://news.ycombinator.com/item?id=47989883

wilg• 3 months ago

Using "apologia" here is pretty embarrassing.

nout• 3 months ago

I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.

maest• 3 months ago

Does this logic still applies if the company is getting other benefits from having me as a user? (Genuine question, I can see arguments for both sides)

For example, if I am using the free tier of a service and "paying" by seeing ads, should I have similar expectations?

I'm not saying that's how users pay for github - in that case it's more subtle, for example by giving up control of some of their stack and bolstering github already near monopolistic network effect.

mghackerlady2• 3 months ago

there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans

hootz2• 3 months ago

Apple, clothing brands, even some Microsoft.

ryandrake• 3 months ago

Aside from the fact that defending a billion-dollar company is a bit strange

More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.

xp843• 3 months ago

1. Telling that you think the only possible motivations are financial (getting paid, stockholder, or foolish expectations of a gift from Satya).

2. Maybe you know a bunch of people who work there, could be ex-colleagues etc. and you think overall it’s mostly good well-intentioned people there. Therefore you want to see them succeed, and also you might disbelieve that the company is deliberately being awful.

I don’t have any specifically warm feelings about a corporate legal entity, but I know people who work at various companies and partly for that reason I am not rooting for those companies to fail and I also don’t believe the least charitable explanations for all their failings.

reilly3000• 3 months ago

This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.

baby_souffle• 3 months ago

We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.

Make sure you cache all the actions you need locally if you go this route otherwise it's not much of an improvement.

hx8• 3 months ago

Microsoft is causing Github incidents when Azure data-centers are too hot and they need to make room for Palantir's workload.

NBJack• 3 months ago

This wouldn't surprise me at all, but I'd appreciate evidence to that effect.

ProofHouse• 3 months ago

Love. Hope Github is a relic of the past inside 12 months

hootz2• 3 months ago

Won't happen. Stars are money.

badgersnake2• 3 months ago

Becoming a joke is the one think that could end the GitHub monopoly.

samgranieri• 3 months ago

GitHub is not a monopoly. It never has been. You've always been able to self-host or you can use gogs, gitea, gitlab, bitbucket, you get the idea

badgersnake2• 3 months ago

Good luck with that. GitHub stars is literally a criteria VCs use for funding.

SwellJoe• 3 months ago

With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)

zem• 3 months ago

quite literally kicking github while it's down!

nifty_beaks• 3 months ago

Holy bootlickers Batman.

annoyingnoob• 3 months ago

Meanwhile, my local Gitlab install just hums along no issues.

frizlab2• 3 months ago

Should be 0 today AFAIK

EDIT: I’m a moron, lol.

alpb• 3 months ago

it already is. re-read?

frizlab2• 3 months ago

Yes, it is, my bad. I was on my way to delete my comment actually! Oh well, too late now… (:

javier123454321• 3 months ago

That purple to blue gradient is the emdash of css.

steviedotboston• 3 months ago

not joking, is there a github repo for this project?

Jonpro03• 3 months ago

No - I made this as a joke for work. Didn't think anyone would look at it!

thangalin• 3 months ago

https://repo.autonoma.ca/treetrek

My free, open-source, bare-bones, caching-free, dependency-free, authentication- and authorization-free pure PHP raw Git viewer. I developed it because GitList blew out my shared host's drive space and memory (due to a caching bug) and to consolidate my GitHub, BitBucket, and GitLab repos. There's something rewarding about self-hosting and not being beholden to the whims of third parties.

dpe822• 3 months ago

I recently moved all my projects to a self-hosted forgejo instance and have found it quite satisfactory so far. And it's fast! If you're in the market for a github alternative, take a look - there are options.

xp843• 3 months ago

It’s not fashionable anymore, but I feel that Phabricator deserves an honorable mention as a self-hostable GH alternative too. Actually its “dated” UI is kind of a plus considering how bad everything is now.

dijit5• 3 months ago

I unironically love the aesthetics of Phabricator.

I also like stacked PRs (which is mercurials default).. Maybe it's worth a shot tbh.

rpearl• 3 months ago

It's not "unfashionable". Phabricator has been unmaintained since 2021.

SpaceNoodled• 3 months ago

I always balked at GitHub, but was impressed with git very soon after I was first introduced; I migrated from an old Gitea instance to Forgejo for my personal projects and have been very happy with it.

thatxliner• 3 months ago

What about Gitea?

dpe822• 3 months ago

Forgejo is just the one I landed on, but like I said - there are options!

sailfast• 3 months ago

Feature request: can you make it look like a chalk board from an old manufacturing plant in the appropriate Green? :)

spearmint27• 3 months ago

A vibe coded app that most likely contributed to the onslaught of vibe coded apps that are causing Github to go down. I feel bad for the people working at Github who are basically trying to keep a sinking ship afloat and Microsoft doing everything they can to sink their own ship.

tdiff• 3 months ago

Not buying that untill they use a picture from the Simpsons

gyoridavid• 3 months ago

I'm pretty sure we all took down a production enterprise system once or twice. At InVision we had an incident every week, despite all the SOPs and safety nets. And that way waaay before vibe coding..

h14h• 3 months ago

I'm currently setting up a self-hosted "Knot" for use on tangled.org.

Mainly doing it because I think AtProto is cool and self-hosting is fun, but also because owning the infrastructure that hosts my projects is definitely the direction I want to move in.

Tangled's Knot system feels like a really strong abstraction for this. I host the data in an AtProto Repository, but can rely on a third party to host/manage the AtProto Application that presents it to the rest of the world. If Tangled goes under, I can happily take my AtProto login to a different platform and point it at my Knot without changing a thing about my hosting setup.

Much more convenient that hosting an entire, siloed webapp on my own corner of the internet.

robinduckett• 3 months ago

This is embarrassing.

Days without GitHub incidents

Related Stories