GitHub is down

https://www.githubstatus.com/incidents/bnkkbj90yhz6

by enescakir • 2 years ago

73 68 2 years ago

githubstatus.com

cranberryturkey2• 2 years ago

homepage gives me a 500

teruakohatu• 2 years ago

Working for me in NZ, logged in.

cranberryturkey2• 2 years ago

its back

jamil7• 2 years ago

Up in Europe still.

schleck82• 2 years ago

Not for me, in Europe.

_astromonkey_2• 2 years ago

Not working for me in Europe

mandalorianer• 2 years ago

Yea, in Germany it works.

pgeorgi• 2 years ago

The failure has yet to propagate to Germany, given how slow internet is here.

weberer• 2 years ago

Which country? There are multiple data centers across the continent.

_astromonkey_2• 2 years ago

Yeah, experiencing issues with Pull Requests and commits

schleck82• 2 years ago

"Degraded performance"

zuck_vs_musk2• 2 years ago

This happens fairly regularly. Do HNers think that Microsoft azure and/or Ruby has a role in it?

furkansahin2• 2 years ago

Why would Ruby play a role in it? Azure definitely does though. (ex azure employee here )

usrme• 2 years ago

GitHub itself is written in Ruby.

zuck_vs_musk2• 2 years ago

Why would Ruby play a role in it? Azure definitely does though. (ex azure employee here )

Oh yeah. If someone says C++ has poor memory safety than Rust then no one is triggered. But say something about Ruby (the language on a secular decline - https://www.dice.com/binaries/content/gallery/dice/insights/...) and the pitchforks are out.

miningape• 2 years ago

Azure is such dogwater - Even a narcoleptic intern could run a more effective cloud service using raspberry pis and a tin can.

A better question would be "is there anyone who doesn't think Azure played a role here?"

erazor42• 2 years ago

Classic Github for the past few years we are used to it / resilient now.

pythux• 2 years ago

The status page says it's been resolved 8 minutes ago (Apr 05, 2024 - 08:48 UTC): https://www.githubstatus.com/incidents/bnkkbj90yhz6

But it definitely still happens now (500s on refresh on PRs and GitHub actions)

Edit: still ongoing

Edit 2: still ongoing at Apr 05, 2024 - 08:56 UTC (keeping updated for the record since their status page cannot be trusted apparently)

Edit 3: I see they have switched to a different (ongoing) incident ID now: https://www.githubstatus.com/incidents/5ly0psff2s5d

s_dev2• 2 years ago

Status pages can never be trusted.

jll29• 2 years ago

=> Third-party sites can never be trusted.

joelanman• 2 years ago

I always think this should be an incident in itself - why did our status page not reflect the reality of a degraded service? It's so common that they don't, and something user-driven like DownDetector is often more reliable

johnchristopher3• 2 years ago

I recently set up a status page for the services I run on my pi. The idea was to get some insights and apply experience at work.

My experience now tells me what we really need first is a solid alerting system, the status page can't be trusted. It's a PR tool (a useful one), not a sysadmin tool.

compumike• 2 years ago

What about something like https://heiioncall.com/status (disclosure: helped build it) which gives a real-time view into what our monitoring & alerting system sees, both from various HTTP endpoint checks, and cronjob checkins?

But I don’t think most orgs would want a public version of this (too much transparency), which is why we haven’t built that.

enescakir5• 2 years ago

Initially, perhaps by different teams, three separate incidents were created. Since then, two have been resolved, and one continues to be updated.

enescakir5• 2 years ago

They marked this one as resolved. They keep providing updates on another active incident: https://www.githubstatus.com/incidents/5ly0psff2s5d

manfre• 2 years ago

Already listed as being resolved. A 15 minute outage at 4:30 am EST.

Macha• 2 years ago

Or 9:30am GMT, so just as europeans are starting to work. Or middle of the working day in India.

enescakir5• 2 years ago

Apr 05, 2024 - 08:51 UTC: Update > We're seeing connection failures to some databases in two of three sites and are investigating.

drcongo• 2 years ago

This should probably link to the main status page as it seems there's more than one incident ongoing.

enescakir5• 2 years ago

In the past two months, they've had over 10 incidents. Are they so hyped by AI features that they're neglecting their core services?

cqqxo4zV46cp• 2 years ago

Please stop doing catsstrophising tech journalist cosplay.

ta12433• 2 years ago

New features are sexy. Core stability is boring. AI is just the latest new feature.

jq-r• 2 years ago

True, and SRE teams get smaller and smaller and/or tasked to do other non-related things. So when users start leaving for some competitor, the same SRE team suddenly gets more heat, pressure, more belt tightening, more meetings "for transparency and coordination", and directly and indirectly more blame. Ask me how I know... I bet its a story as old as time though.

pm90• 2 years ago

There is little incentive for them to invest in stability. Customers will eat the pain because they are so invested in the platform. They will never move away.

Meanwhile all the Staff engineers trying to get promotions are building AI shit.

pwmtr• 2 years ago

Since last few months, GitHub has become very unreliable. For example, last month, they reported six official incidents in just one week. I've also noticed many shorter, non-official incidents (i.e. the ones where there is no active incident on their status page, but many API calls return a 500 error). I learned about this the hard way (we offer affordable GitHub Runners at Ubicloud, which gives us the opportunity to observe all these failures firsthand)

masklinn2• 2 years ago

I’d say years though it might depend on both your physical location and te services you use.

We’ve had to harden a bunch of automation to be more reliable in case of github going tits up, and while github is still currently the broker (as it’s where users get statuses / information from it made sense at the time to also have it handle broadcasts) we’re definitely planning on direct communication between services eventually.

bushbaba• 2 years ago

Curious if this is related to them adopting azure

ksec• 2 years ago

Did they officially announced they are adopting Azure?

alt227• 2 years ago

What does 'adopting azure' mean?

Am4TIfIsER0ppos• 2 years ago

Thank god (Linus) that git is a distributed "protocol" letting you pull and push to any location on your file system or anywhere you can access over ssh.

ta12433• 2 years ago

99% of git users have no concept of this. It's very much a client/server relationship with github.

masklinn2• 2 years ago

Git itself is only a piece of the puzzle, you still need synchronisation points for collaboration e.g. bug reports, code reviews, discussions, CI, synchronisation, etc…

Those are the services forges provide.

Also git’s support for point to point distribution is bad. I miss `hg serve`.

mr_mitm• 2 years ago

This comes up pretty much everytime Github is down. Github is not a replacement for git. Github is an issue tracker and CI platform just as much as it is a git hosting site, none of which git can provide by itself. It's complementary. The comment really doesn't make any sense.

It's like being snarky about the distributiveness of git when the Linux kernel mailing list goes down.

ta12433• 2 years ago

My on-prem gitlab has now survived 6 cloud-based github outages.

Already__Taken• 2 years ago

how long do you spend upgrading it?

dijit• 2 years ago

Not OP, but for mine I spend about 10 minutes every 2 weeks.

Once: an upgrade went wrong and I had to roll-back, that made me spend 30 minutes instead of 10.

Gitlab instance stats:

Projects 256

Users 216

Groups 17

Disk usage: 877 GiB / 3.38 TiB

_hao• 2 years ago

In London it's dead. I guess I'm taking the day off! Happy Friday everyone!

Whiteshadow12• 2 years ago

Sorry everyone. This was me.

notoverthere• 2 years ago

What did you do?

jjgreen• 2 years ago

Bounder

blippitybleep• 2 years ago

I saw a post on LinkedIn, from GitHub, boasting about how many changes they deploy in a day. I couldn’t believe it. GitHub actions have been broken so many times for me this past month, not able to deploy my code when I want to, so it felt like a slap in the face.

nonrandomstring• 2 years ago

Maybe what is happening in this thread should give everyone cause for concern and a clear heads-up that more serious problems are afoot.

Very emotional reactions here, whether fuming at the unreliability of the service, or zealously defending and identifying with it, indicate people have far too much invested in this single point of service.

People are saying - "That's me done for the day, have a good weekend!" What is the economic cost of this? On a global scale? The pattern is the same as we've seen recently with Office365 and other cloud services.

People talk about setting up elaborate monitoring services, yet with a similar effort one could set up one's own development repositories and tools.

It's not like this crowd is typically "helpless", or unaware of what is being done, yet people remain beholden; Github has diversified into a social network and AI tool, far beyond its initial remit as a code repository. People seem genuinely terrified at the prospect of it going. And rightfully so.

In the wake of the xz backdoor attempt Microsoft will soon move to offer "developer protection" and a system of validation, testament and supply-chain protections. At that point Microsoft will have all of you absolutely in the palm of its hand and at its mercy.

Now would be a good time to seriously invest in long term thinking and strategic protection of your development future, indeed of your fundamental ability and rights to develop software free from disruption, pressure and encumbrance.

GitHub is down

Related Stories