GitHub Git Operations Are Down

https://www.githubstatus.com/incidents/qd96yfgvmcf9

by hunkins • 2 years ago

328 260 2 years ago

githubstatus.com

hunkins3• 2 years ago

GitHub git operations are down. Any git operation returns a 500 error response.

localghost30002• 2 years ago

Can't pull anything from my repos which is not great.

hunkins3• 2 years ago

Yeah, this is not good. I've experienced other outages on their other products (e.g. GitHub Actions), but not even being able to do a basic git pull is an entirely different level.

localghost30002• 2 years ago

I'm effectively dead in the water here. I guess I'll go touch some grass. Thank you to the SRE's who are looking at this.

normie3000• 2 years ago

Is it possible to download a zip of a repo through the GUI?

__d• 2 years ago

SSH access appears to be rejecting my key too.

jxi• 2 years ago

GitHub Actions are also failing with 500 (gateway timeout) errors when trying to fetch the repository.

__turbobrew__• 2 years ago

Yep, 100% of git operations are failing. CI/CD and dev operations are at a halt.

lukeh2• 2 years ago

Yup, SSH rejecting my key, thought it was something at my end but no. Can't run any Yocto builds... I guess a good opportunity to take a walk or do some life admin.

lukeh2• 2 years ago

Back up now for me.

remram2• 2 years ago

I hope you didn't wait until now to push your day's work!

l33t_d0nut• 2 years ago

I did :(

vinayan3• 2 years ago

Experiencing this as well. I was a bit concerned something was wrong with my key but alas it is not my key but an outage.

Developer "snow day".

TheRizzler• 2 years ago

GIT is down today, Code rests in snowy silence, Developers play.

itissid• 2 years ago

Spent like 30 mins tryig to figure out why ssh auth was not working. Compared sha256 signatures, doubted my reading abilities and pulled my hair.

guluarte• 2 years ago

same..

jcgrillo• 2 years ago

this coincided with an os upgrade for me.. made for a very confusing 15min until i checked gh status

sc68cal• 2 years ago

same here I was scratching my head trying to figure out if something had corrupted my system or what

parpfish4• 2 years ago

damn, why couldnt it happen during east coast business hours?

unwoven• 2 years ago

lmao i deleted and readded a new key thinking my key was fucked

crest• 2 years ago

It's almost as if only Git is distributed, but people sold out to Github for convenience. Too bad Git lacks a distributed bug tracker and wiki system like Fossil. Guess Github has to fail a lot more for things to change.

sixo2• 2 years ago

people could not be more clear that their preference is for reliable and easy-to-use centralized services maintained by professionals, and not decentralized systems that require a great deal of user expertise

ajross• 2 years ago

Ironically github is sort of the exception that proves the rule: decentralized in the one way that really matters (decoupled development on individual systems), but centralized for easy interaction in the way the market demands.

cyanydeez2• 2 years ago

Thats mostly because of herdmentality not because of a considered approach. Decentralization assumes the majority are anti social.

eru10• 2 years ago

I actually don't care whether it's centralised or decentralised, or who's managing it.

But you are right that I want reliable and easy-to-use services. And centralisation is often one way to go there.

As an interesting counterpoint: Git itself is decentralised and replaced centralised services like Subversion. And that made git easier to use, especially easier to get started with: no need for a server, no need to be online, just do `git init` in any old directory.

A GitHub-clone could be more decentralised, but they'd need to use that decentralisation to drive those other features that people actually care about day to day.

dangoodmanUT• 2 years ago

this, especially if it's a business

__MatrixMan__4• 2 years ago

And I'm sure they'll continue to feel that way right up until the first time they experience "the" internet from a minority partition for more than a few days. I just hope that the distributed stuff is easy enough to limp along with if/when that happens.

ajb• 2 years ago

There is not actually a need to choose between a single centralised monopoly and going full-on techno-prepper and running all your own services in your garage as an individual . We could have intermediate points, such as having services run by professionals but based on portable open standards.

flats• 2 years ago

this should be on a wall somewhere

koito17• 2 years ago

I've been keeping an eye on radicle[1] but the documentation for setting up a peer and web frontend is a bit complex. It seems to offer what you're describing: a "decentralized" Git frontend with issue tracking. Seems to be missing wiki functionality, however.

[1] https://radicle.xyz/

whalesalad• 2 years ago

The GitHub wikis are actually git repos.

remram2• 2 years ago

I forgot about that, I wonder if they were broken too.

joecool1029• 2 years ago

Too bad Git lacks a distributed bug tracker

Email and mailing lists?

Groxx2• 2 years ago

That's the same as not having one.

Otherwise you can claim Facebook is distributed because you can email people links to Facebook pages.

tomnicholas1• 2 years ago

Is there a world in which GitHub used an open protocol for the social network part of their product like BlueSky's AT protocol[0]?

[0] https://docs.bsky.app/docs/advanced-guides/atproto

NetOpWibby• 2 years ago

...this is an interesting thought exercise, thank you.

theultdev3• 2 years ago

not p2p, but federated: https://forgefed.org (ActivityPub extension)

I believe Gitea has support for it, not sure to what extent.

nine_k3• 2 years ago

Yes, conveniences like:

- A canonical name and place on the web;

- Access policy enforcement (who can commit and when);

- The whole pull request thing, with tags, issues, review, discussion, etc linked to it;

- Code review, linked to said policy enforcement;

- Issue tracking, even as basic as what GitHub offers;

- A trusted store for signing keys, so commits are verified;

- CI/CD triggers and runners;

- A page with releases, including binary releases, and a CDN allowing to use the download links without fear.

This is way more than distrusted version tracking. Actually the above is not even married to Git; it could be as valuable with Mercurial, or even Perforce.

This is a large product, actually a combination of many potentially self-contained products. It should not be compared to Git, but rather to Gitea or BitBucket. Not all of this can be reasonably decentralized, though quite a bit can.

eru10• 2 years ago

- Access policy enforcement (who can commit and when);

Interestingly, what GitHub mostly enforces is where your branches point to. Not who can make commits. That's mostly because of how git works, not because of any grand design on GitHub's part.

bluGill3• 2 years ago

The above is why my company left mercurial years ago for git. Mercurial of 10 years ago was better version control than git (git still can't track a branch and how it changes) - but the rest of github is much better.

ncruces• 2 years ago

- CI/CD triggers and runners;

I've used up 17h of CI time these two (slow) January weeks, for free, testing stuff across ~20 different OS/CPU combinations.

That's on just one "personal" project; a bigger dependency of that, of which I'm a maintainer, spends an order of magnitude more.

Can you (GP post, people complaining, not parent) blame us? Should we instead self host everything and beg for donations just to cover the costs?

tokioyoyo• 2 years ago

People said that when GitHub got bought out, and only more people ended up moving there. It's really not fun to manage your own Git servers, and when things go down, they get fixed much slower than in-house hosted version of it.

adamtaylor_13• 2 years ago

I don’t know anyone beyond hobbyist hackers who want to set up and maintain this stuff for themselves.

As a professional software developer, I want tools that just work that I can rely on. GitHub 99.99% uptime is something I can rely on.

TZubiri4• 2 years ago

As a professional software developer, you rely on software written by those hobbyist hackers.

Whenever you do a clone or an npm install or apt get or pip install, etc...

You choose github because your dependencies chose git

fragmede• 2 years ago

It was down for ~45 mins according to the linked page page, which would put it at 99.897% uptime for the month, assuming no other downtime.

gerdesj• 2 years ago

A bloke called Linus Thorvalds created git for Linux development when the commercial service used for that ceased to be useful, for one non technical reason or another.

github basically shoves a webby frontend and workflows on top of someone else's work. That's all fine and good but github is not git.

As a professional IT consultant, I want tools, I use lots of other's and I also create my own and I also generally insist on hosting my own. I'm also a fair carpenter and several other trades. I have tools for all my trades and I look after all of them. That way I can guarantee quality - or at least reproducible and documented quality.

I'm sure you are fine with abrogating responsibility for stuff that you are not able to deal with - that's all good: Your choice.

EDIT: Sorry, forgot to say: "Yay cloud"

Over2Chars11• 2 years ago

four 9's uptime https://uptime.is/99.99

Daily: 8.6s

Weekly: 1m 0.48s

Monthly: 4m 21s

Quarterly: 13m 2.4s

Yearly: 52m 9.8s

If you assume that "uptime" means all tools are available

https://statusgator.com/services/github

this appears to be 45 minutes in just one day

Incident with Git Operations 30m Jan 14, 2025 9:01 AM Down

Incident with Git Operations 10m Jan 14, 2025 8:51 AM Down

Incident with Git Operations 5m Jan 14, 2025 8:46 AM Down

Not much margin to hit four 9's left for the rest of the year.

Dylan168074• 2 years ago

GitHub 99.99% uptime is something I can rely on.

Their enterprise level SLA is only 99.9% (measured quarterly) and the remedy is a 10% credit, increasing to a 25% credit if they drop below 99%.

TZubiri4• 2 years ago

Contrary to popular belief, sarcasm makes you harder to understand and is no longer cool

kspacewalk22• 2 years ago

Counterpoint: it's a 45 minute outage once in a blue moon. Very small price to pay for the convenience of a centralized VCS with many features that aren't easy to reliably set up in standalone installations.

vrosas2• 2 years ago

Once in a blue moon? It’s like, monthly at best.

freewizard2• 2 years ago

Git is distributed. Distributed system does not guarantee 100% uptime or real time consistency. You can take the whole history with you and push to a different remote.

xelxebar• 2 years ago

Too bad Git lacks a distributed bug tracker

Not your point, really, but fortunately, git is easily extensible. This in-repo issue tracker is surprisingly feature complete: https://github.com/git-bug/git-bug. Anyone else given it a whirl?

bastardoperator4• 2 years ago

It's this type of negativity ruining the internet. Nothing thoughtful to say, nothing to add, and hoping for failure. I hope everything is okay over there...

liontwist4• 2 years ago

They are advocating for decentralized tools. That’s actionable and hoping to prevent failure.

Over2Chars11• 2 years ago

You're quite a Mary Poppins for someone with the heavy handle of @bastardoperator.

AFAICT the internet was built on negativity.

Here's the 2nd post from a random USENET group I found:

https://www.usenetarchives.com/view.php?id=comp&mid=PDQ5ajZp...

Dylan168074• 2 years ago

They want people motivated to design systems that can handle github going down. That doesn't strike me as negativity, and especially not negativity ruining the internet. It's not the most thoughtful thing in the world but it's a reasonable opinion, and most comments are also not the most thoughtful things in the world.

nextaccountic• 2 years ago

Too bad Git lacks a distributed bug tracker

Well there is https://github.com/git-bug/git-bug

evbogue• 2 years ago

Makes me wish for the good old days of git-ssb

tracerbulletx• 2 years ago

I mean I could very easily still push to another remote until it comes back? I do not feel locked in at all.

paulddraper3• 2 years ago

Convenience is nice.

XorNot• 2 years ago

git was never really decentralized though. The whole system is aggressively a funnel towards central repositories, it's just that because it deals in whole repos, every git repo has the potential to be promoted to that role.

Nothing is built into git to let it actually run decentralized: there's no server or protocol where someone can register say, an identifying public key and then just have a network of peers all communicate updates to each other. It's even pretty damn unsafe to just run a repo through basic file-sync (I do this for myself with syncthing with bare repos in a special folder, which seems to work fine but I'm hardly loading it up to chase down why it doesn't).

never_inline• 2 years ago

If you have a local development setup, few hours of not being able to push should be a not problem most devs.

Except if you have a release planned but most don't at that time, statistically.

Problem is that people get comfortable with pushing to branch -> deploying in dev and testing from there.

JSTrading2• 2 years ago

Panic for a minute SSH rejected a key!

atulvi• 2 years ago

I just reset my ssh key lol.

JSTrading2• 2 years ago

They are going to get DDoS's by people trying to do this haha.

j45• 2 years ago

I guess it's a good time as any to setup a backup or check out alternatives that could mirror or standalone. Gitlab, Gitea, any others worth checking out?

dboreham• 2 years ago

Gitea works for us.

codetrotter3• 2 years ago

I use a fork of Gitea called Forgejo. It works nicely as well.

https://forgejo.org/

kivikakk• 2 years ago

Consider Forgejo: https://forgejo.org/.

vv_5• 2 years ago

What's the point of having a mirror? It'll only lead to conflicts that'll be difficult to resolve once the main service you use is back up again.

For example, I doubt you would be able to easily merge any pull-requests or use the same CI/CD code for the same services without hacky solutions.

homebrewer3• 2 years ago

JFYI, the next version of gitea (which should come out in April) will have full GitHub mirroring functionality (which will let you set up a mirror and have it pull code/issues/wiki/PRs/etc. every few minutes). The current version can either migrate the full repo once, or mirror the code and nothing else.

itsjustjordan• 2 years ago

I was going insane doubting my SSH knowledge, stopped short of creating new keys thankfully!

donohoe• 2 years ago

Same - but created new keys :) ... which also didn't work, and then I went to check... oh well.

atonse• 2 years ago

I thought my account was hacked and ssh keys removed. Panicked a little bit and then double checked that my keys were still there.

Then went to github status and calmed down.

eru10• 2 years ago

I briefly entertained the notion that I was fired and my ssh keys revoked. But I still had access to slack and email etc, so I banished that thought.

Layvier• 2 years ago

well on my side I'm the proud owner of a new ed25519 key.. The status page didn't update quick enough

smus• 2 years ago

as am I!

G1N• 2 years ago

Was trying to install homebrew on a device and kept wtfing because the clone step kept failing… thought I was crazy

NautilusWave• 2 years ago

I had just upgraded to Windows 11 last week, and for some godforsaken reason earlier today, I could SSH via WSL but not from the host OS even though they were both using keys served from the Windows OpenSSH agent! I'm just going to blame this service outage and hope for the best tomorrow.

sashank_15092• 2 years ago

I for a second thought was I getting fired

shaftway• 2 years ago

As someone who found out that way twice in the last two years, I also thought I had been let go. Anxiety is a hell of a drug.

rasengan• 2 years ago

Just the info I was expecting to see on HN. Unfortunately (or Fortunately?), a better source for these kinds of things than anywhere else.

cdme• 2 years ago

It's just a hub at the moment.

hunkins3• 2 years ago

Update: looks like the problem has been identified.

"We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."

arnath• 2 years ago

It's odd to me that there's nothing indicating an issue is ongoing on the github.com homepage

clintonb• 2 years ago

Unclear if related, but the Terraform Registry is also having issues serving data: https://status.hashicorp.com/incidents/jypswvyh0h3z.

In a wonderful twist, we are relying on a couple modules served from GitHub!

impure• 2 years ago

I thought it was my internet. Oh well, time to finish that game of Civ 6. Edit: and it literally came back up the moment I posted this.

486sx33• 2 years ago

Damn you azure! Or is Microsoft not even running their precious GitHub in azure

bobnamob• 2 years ago

Who said it was an infrastructure failure?

There's innumerable causes of this kind of failure that aren't rooted cloud service provider shenanigans

(Admittedly the duration of the outage does "feel" like an infra outage)

qqqult• 2 years ago

lol I changed my ssh key 2 times until I figured out pubkey permission denied was an issue on their side

james33• 2 years ago

All checkmarks just turned green and my push went through. Snow day over.

tom13372• 2 years ago

Not sure if I am just noticing GitHub's issues more often as I am using their tools pretty much every day but their availability is kinda not great. Be it Actions failing or something "not core business" (read: git operations) but I can't remember a month in the past where I was not annoyed by any outage on GitHub

croemer6• 2 years ago

Agreed, I wonder what their downtime percentage is. If guess it's down on the order of one hour per month, so 1/1000.

Update: They promise >99.9% on a quarterly basis for enterprise customers - https://github.com/github/docs/blob/main/content%2Fsite-poli...

mcdeltat• 2 years ago

99.9% uptime corresponds to about 2 hours downtime per quarter, if my maths is correct. If that is indeed the guarantee, based on the experience at my company, GitHub has failed its promise recently (or is getting damn close). I recall 2 decent outages in the past few weeks alone. It's making me begin to doubt if GitHub's reliability is appropriate for an enterprise service.

FridgeSeal2• 2 years ago

The whole product feels like it’s getting g progressively jankier.

The front-end is glacial nowadays and frequently has issues actually loading the page, actions frequently has some kind of panic attack and breaks or just grinds along a glacial speeds. The UX has gotten worse (why no merge-queue button?).

tom13372• 2 years ago

Agree with the second part. Github actually has merge queues, but only for the enterprise tier…

IshKebab2• 2 years ago

I wouldn't say glacial. If you want glacial try Jira. Mind-bogglingly slow.

burgerrito2• 2 years ago

I'm not sure if it's just me but I swear GitHub UI is one of the slowest web app I've used.

Rexogamer• 2 years ago

the incident has been marked as resolved; I am now able to push

freewizard2• 2 years ago

It was down for ~2 hours. The status website claims "degraded performance", but in reality we get

    git@github.com: Permission denied (publickey).

Either GitHub didn't know how to communicate, or they were not sure about the real impact.

This is bad.

manfre• 2 years ago

Status pages are rarely honest. The company will lie to salvage their SLA. "Degraded performance" or "some customers are experiencing an elevated error rate" should be interpreted as "service unavailable / outage"

croemer6• 2 years ago

I'd be curious by how much they downplay downtime. Wouldn't be too hard to put together an honest status page that pulls & pushes something new to main every 5 minutes, creates an issue, comments on it etc. Very basic high level checks.

croemer6• 2 years ago

Someone else lamenting the delayed status page updates almost 2 years ago: https://news.ycombinator.com/item?id=35887213

enasterosophes• 2 years ago

I'm not convinced it's deliberate dishonesty. Just a communication disconnect. Firstly, it can take time from the first yellow flags to the full realization that there really is an incident underway, secondly it needs someone to decide how to communicate that incident, and thirdly the engineers who are actually working on the incident need to be able to get on with it instead of being pestered for an update every 10 minutes.

deathanatos2• 2 years ago

They used to be. Github's is a prime example of how "useful info" has turned into "PR mouthpiece" — it used to display graphs of a few choice Github system metrics, and those spiking could often usefully indicate "yeah, they're having a problem" well before a human could update the page.

But yeah, also status pages seem to be under the domain of non-engineers who are too concerned with how things look, vs. conveying useful information in a timely manner, and ultimately, fail at both.

spacedcowboy• 2 years ago

Yeah, I've been trying to check out the SDL3 suite for the last hour or so, and it's still failing on SDL_ttf...

fatal: clone of 'https://github.com/libsdl-org/freetype.git' into submodule path '~/src/SDL/SDL_ttf/external/freetype' failed

...

fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

...

fatal: clone of 'https://github.com/libsdl-org/harfbuzz.git' into submodule path '~/src/SDL/SDL_ttf/external/harfbuzz' failed Failed to clone 'external/harfbuzz'. Retry scheduled

...

Failed to clone 'external/freetype' a second time, aborting

For a problem that's supposedly "fixed" that's a whole lot of errors...

eru10• 2 years ago

They were also down a few days ago, but HN flagged my submission as a dupe. :)

burgerrito2• 2 years ago

Thankfully I use Forgejo! No reason to stick with GitHub when Git itself is distributed by nature

simondotau• 2 years ago

True. Also there's no reason not to stick with GitHub when Git itself is distributed by nature. That's the beauty of distribution.

ashishb• 2 years ago

Why does GitHub go down almost once a month?

1propionyl• 2 years ago

Acquisition.

ducks

0xbadcafebee5• 2 years ago

all Git operations were unavailable due to a configuration change

eats popcorn waiting for explanation of why they didn't catch it in non-prod

pnw_throwaway• 2 years ago

Have you never fixed a prod-specific bug before? They’re common enough, and it sounds like they patched this one within a reasonable amount of time.

0xbadcafebee5• 2 years ago

I expect more from billion-dollar companies. If you have that much money, you can pay somebody to ensure the changes are properly tested. (I've worked in billion dollar companies, and usually they are too cheap to do this, despite their massive wealth. They just leave it up to ICs and hope they are testing right, rather than making sure)

noplacelikehome• 2 years ago

I've just ended my Pro plan; enough is enough. I've been debating moving my repositories to a self-hosted forge for a while; it'll be an interesting opportunity to move more CI operations out of Actions and into Nix.

jcims7• 2 years ago

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise.

https://www.federalregister.gov/documents/2023/06/09/2023-12...

On the surface a product like managed git repos would seem to be relatively straightforward to deal with, but these same regulated firms are also under tremendous scrutiny for access management, change management, SDLC, etc etc. They also have a huge software footprint which just increases the impact.

Self-hosting is obviously the traditional answer, but that's not exactly a simple one either.

Just an interesting problem.

dangero• 2 years ago

That's a lot of overhead. Do you know how they are calculating the possible risk of these events? It feels like there are a million rabbit holes like this you could go down when modern infrastructure is so cloud connected.

Is the risk of your git repos higher than a chip shortage causing you to lose access to the infrastructure you need? So many factors to consider. A chip shortage doesn't seem that unlikely with geopolitics.

The list of scenarios you mitigate for seem like they could very easily be an arbitrary list of the of scenarios a single person came up with.

jcims7• 2 years ago

It's an astonishing amount of overhead that slows everything to a veritable crawl. It also creates downstream issues because you need to build in contingencies for your service consumers until you are able to fully abstract the dependency to a point where rehoming doesn't impact them.

The risk calculations are very primitive at the moment, I'm guessing they will be refined over time as industry feedback starts to resonate.

cheeze• 2 years ago

And the ultimate problem that you don't need a solution for, until you _need_ it. Which often leads to "yeah we have backups" but without regular testing, can you trust them?

parpfish4• 2 years ago

but if you self host, how do you anticipate power outages and natural disasters?

you shift it from a problem of software reliability to a problem of physical infrastructure. at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.

0xbadcafebee5• 2 years ago

Basically you rent space and machines in datacenters that are geographically distributed and designed to resist natural disasters

jcims7• 2 years ago

Most of the companies exposed to this are already going to be running a number of geographically diverse datacenters and have been exposed to regulations around basic DR/BCP for decades.

at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.

Yes. This is part of the reason why services like git are being moved outside the datacenter. Most of the product offerings on the market don't scale well, have terrible reliability and are still very expensive to run.

Over2Chars11• 2 years ago

You're just using your own data center instead of Jeff's bit barn.

I mean seriously people, a "cloud" is just someone elses' data center.

Am I missing something?

0xbadcafebee5• 2 years ago

First pay a consultancy a fortune to tell you which SaaS you are able to use based on compliance requirements. Then run your own IdP, use service providers that have a data takeout mechanism, take regular backups, and use standards-based technology. Line up a self-hosted fallback for each non-self-hosted option for if you're forced to exit them. Basically you line up the exit strategy for the auditors but hope you never have to use it.

macintux• 2 years ago

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed.

Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.

jcims7• 2 years ago

I think my point is we are losing the option to worry about it or not. There needs to be an answer with a runbook to restore services within a given window of time should we lose that vendor.

dangoldin2• 2 years ago

Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.

theteapot3• 2 years ago

Git is only a small part of Github these days.

deathanatos2• 2 years ago

The source code, perhaps, but a good many of the orgs I've worked at also use Github for PRs, Actions (CI), and for triggering deployments. Those things take time to setup, and across a whole org, I wouldn't want to have to have an unplanned change to another vendor.

In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML that runs commands, right? Translate the format, right?" but it's really not; differences between how CI systems map commands to machines, external integrations (e.g., in this case, into Action's artifacts system, or output system) etc. all make it more complicated.

st3fan• 2 years ago

GitHub not being available for 45 minutes is not an "extended outage".

jcims7• 2 years ago

Agreed, that's why i said 'tangential at best'.

Over2Chars11• 2 years ago

I don't see how self-hosting solves the problem of 3rd party vendors unless you're standing up a self-hosting solution as a hot/cold backup to your 3rd party vendor "in the event of an extended outage".

Kinda eliminates all those pennies saved (in theory) for outsourcing to "the cloud" if you have to duplicate your infra.

Hybrid has always seemed the most optimal approach, but there's always someone in charge who thinks spending money on safety nets is just wasted money.

theteapot3• 2 years ago

Doesn't it exactly solve "extended outages and forced exits from third party vendors."?

hamandcheese4• 2 years ago

Since git is distributed, I wonder if it's enough to demonstrate the capability to spin up an alternative, but not necessary keep it up as a live backup 24/7.

outworlder• 2 years ago

My company is replacing the entire system we built with Github actions. Our previous DR plan involved running our automation scripts to reprovision the infrastructure that manages the rest of the infrastructure (runs jenkins jobs, etc).

They are replacing everything with Github actions. I wonder what they are going to do when Github is down.

globnomulous• 2 years ago

I work on one part of a huge suite of interconneted services that have extremely strict SLAs for up-time. It really is an interesting problem, and the quantity of engineering resources devoted to ensuring availability and avoiding downtime -- basically fighting to make inherently unstable, complex systems stable and reliable -- is dumbfounding.

When I'm trying to explain to people what it's like to work on this kind of software, I like to use an analogy: it's as though I have my own personal brick, or group of bricks, in the great pyramids of Egypt, just a a tiny piece of a stupefyingly, inconceivably larger whole -- and when I twist my chisel and strike my block just so, at exactly the right (or rather the wrong) angle, I can shake the very foundations of Egypt.

thayne• 2 years ago

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud.

And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable, or even go out of business, and being self hosted would leave you in no better position than if you had used cloud with external backups. You can somewhat mitigate this risk by self hosting an open source solution, so that in the worst case scenario, you can fork the project and maintain it yourself, but there is still risk that the project could be abandoned, or possibly have the license changed in future versions.

To be clear, I'm not saying that you shouldn't self host and SaaS is always better. But it isn't a magic bullet that solves these problems.

bgrainger• 2 years ago

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.)

Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infrequent updates during maintenance windows that are convenient for us (since we don't need 100% availability outside of US working hours).

I don't have the metrics in front of me, but I would say we've easily exceeded github.com's uptime in the last 12 months.

shortrounddev23• 2 years ago

Could have a self hosted git or ci/cd pipeline for deployments and code access during outages. They don't need to be up constantly, just need to be able to keep a backup of the code somewhere and have some way to run critical deployment scripts if github actions are unavailable

the_svd_doctor• 2 years ago

Ha so that's why my git pull were so slow. I should have checked HN faster.

parpfish4• 2 years ago

it'd be great if they set up their CLI tools such that

if there's an error/timeout, they'll do a check of their status page so you don't get the standard 'error' but rather a 'dont worry, youre not doing anything wrong. its borked, our bad' message.

Aperocky• 2 years ago

That's just an indirect manual dependency on the human that gets to decide to update their status page.

croemer6• 2 years ago

*great not break :)

Make an issue in gh CLI repo, I like the idea

xyst• 2 years ago

Yet another reason to self host your VCS. Only thing hosted to “the cloud” these days are my backups - split between S3 and GCS.

~$25/mo

orf• 2 years ago

That’s expensive, I get it for free with GitHub. And a loooot more functionality.

souenzzo• 2 years ago

windows window operations are down

GitHub Git Operations Are Down

Related Stories