GitHub major issues - repos have "lost" commits and site is erroring

http://support.github.com/discussions/repos/5401-something-went-wrong-error-on-walkerledge-zeppelin-private-repo

by andrewljohnson • 15 years ago

119 86 15 years ago

support.github.com

It isnt just private repos. I'm seeing it on various public repos on github.

cilantro• 15 years ago

Someone had shared a public repo with me on Monday. It started 404ing the next day and I had assumed he took it down. I realize now that it must have been this issue. I just checked for it and it is back.

mcmatterson2• 15 years ago

In future, they keep a very up-to-date status page at

http://status.github.com/

andrewljohnson6• 15 years ago

There is nothing useful on that page. The error is in full effect right now.

mcmatterson2• 15 years ago

Oops, my bad. I mistakenly thought the 'db error' notice was for today.

I retract the 'very' in my original parent comment, replacing it with 'usually'.

rabidsnail• 15 years ago

Is it just me or has guthub's performance and reliability been getting progressively worse over the past month or two? Even putting this issue aside, and the outages, pushing to a private repo takes _minutes_ now, and file lists in the web interface often don't load without a page refresh.

windsurfer4• 15 years ago

At the price and size of Linode instances, it's cheaper and I feel better knowing I'm in control when I run Redmine+Gitosis. When I had my github account and something went wrong, I couldn't just ssh in and fix it. All I could do is wait.

Now, for some people, I can see how waiting is an easier solution. I personally just don't like it.

(of course, this is reffering to private repos. public repos are still king on github)

SpiralLab2• 15 years ago

A major selling point of Git is the distributed part. You are allowed (even encouraged) to have more than one "source of truth". For ~7$/mo (including admin. and cheaper than Linode), GitHub is just another place to have a hosted version of your repo, with a nice UI and social features.

Shit happens, servers go down, that's why you also have a remote repo hosted on Linode, and X, and Y too.

erikpukinskis3• 15 years ago

It's amazing to me that you feel more confident in your own abilities to keep a server running than GitHub... you must be a really great sysadmin! The GitHub folks are pretty hardcore!

I, for one, am a positively mediocre sysadmin. And the amount of time I have to spend waiting for GitHub to fix bugs is almost certainly far, far, FAR less than the amount of time I'd spend scrambling (but not waiting!) to fix my own.

This is also how I feel about Heroku.

For whatever reason, I prefer the helplessness I feel while waiting for them to fix stuff to the helplessness I feel when I'm in "OH SHIT MY SERVER IS DOWN AND I DIDN'T EVEN DO ANYTHING AND WHY IS THE DATABASE NOT RESPONDING IT RESPONDS WHEN I QUERY IT MANUALLY AAAUUGGHHH!!!" mode.

Although I do try to keep myself sharp by doing things by hand in other areas of my engineering activities.

guywithabike• 15 years ago

It's just you.

My team uses dozens of private repos hundreds of times a day and everything's been smooth for the past few months, aside from the disruptions noted on their status blog.

andrewljohnson6• 15 years ago

It's not _just_ him. Right now, it's me too.

jamesbritt5• 15 years ago

I have two repos living in "Repository temporarily unavailable" status for several days now. I understand that shit happens, but what's really frustration is inquiry about what can or will be done about it and getting no response.

Do I wait? Do I just move stuff to my own server? Go back to Gitorious?

It's not that I can't keep working, but the main reason I use Github is to make my OSS projects available. For other things I run my own server and my own git host. Having a project unavailable to people for days, when I'm trying to get bug fixes and such out, is not so good.

erikpukinskis3• 15 years ago

Two questions: 1) did you file a support ticket? that seems like the sort of thing they'd be serious about fixing, and 2) do you have a paid account? In some sense, it doesn't matter, OSS developers are important to their mission too, but... well, I'm curious because I have a paid account and if you do too then that makes me more afraid for myself.

sreitshamer• 15 years ago

Responding is key.

I believe the easiest way to build a successful business (software and/or service business anyway) is to respond promptly. Even if you don't have an answer yet. If you do, your customers will stick with you through almost anything.

oomkiller2• 15 years ago

I haven't had any trouble, but I have seen more interruptions happening recently. I relate it to growth, because as you grow, these things sometimes happen more often; "growing pains" if you will. Overall I am happy with Github, and their downtime has not cost me any time or money.

hsuma• 15 years ago

I'm still on the fence about that whole cloud thing, personally.

citricsquid• 15 years ago

is reliability the price of convenience, or the other way around?

edit: that makes no sense

jameskilton• 15 years ago

Not sure how this is relevant. Github isn't hosted on any cloud.

http://www.anchor.com.au/blog/2009/09/github-designing-succe...

baddox• 15 years ago

Well, every resource is locally on some physical machine somewhere. I think his point is that Github is often treated by its users as a "local" resource, in that it takes the place of what would otherwise be a local server running a Git front end or repository.

Now, Github probably wouldn't encourage total reliance on their service. Git is fundamentally distributed, so this doesn't have to be a problem. Still, the Github service is an integral part of many developers' workflow, and I do agree with the above commenter that I'm not sold on the idea of trusting remote servers with integral steps in my workflow.

__david__• 15 years ago

Well, here's the beauty of Github--if they lose all my repos I just don't care. Since I've got a copy of my repos locally, there's absolutely nothing they can lose... Worse case if they go completely down for a couple days I can still collaborate using ssh or "git send-email".

This is the only reason I decided to use Github after avoiding Sourceforge and their ilk for the past decade and a half.

jamesbritt5• 15 years ago

"This is the only reason I decided to use Github after avoiding Sourceforge and their ilk for the past decade and a half."

But why github? Why not gitorious.org?

If github goes away, I still have my code, but I lose any issues people have filed on the site, and I lose being able to easily check on forks of my projects where people might be doing interesting stuff.

I had been using gitorius.org, but moved to github for all the things other than git that makes one public git host different from another.

davnola• 15 years ago

That's the beauty of a DVCS (Git in this case) _not_ Github.

mtrn• 15 years ago

You mean this cloud? http://cloud.github.com/

jasonervin• 15 years ago

Not good.

ericmsimons2• 15 years ago

I hope this has nothing to do with the people who hacked sourceforge..

jrockway2• 15 years ago

Git has full cryptographic history (and data) authentication, so any changes to the history would be easily detectable. And you can sign known-good commits, so that even if you've never pulled before, you can still verify the part of the history that's been signed.

SVN and CVS are missing this key feature, which is why the sf.net hacking is scary.

ericmsimons2• 15 years ago

I'm a Git n00b, so thanks for pointing that out! No wonder git has killed off SVN

saurik7• 15 years ago

Git's hashing detects data corruption to an existing user, but to someone who hasn't downloaded the repository yet and is getting it for the first time you really have to remember that SHA1 is currently considered "broken". It is increasingly feasible for someone to generate a collision to an existing file, allowing them to forge a commit. This has been discussed on the git mailing list, for the record, and the response is generally "don't care, we don't claim this to be a security mechanism". This is even a problem if you use signed tags, as you are only signing the result of a broken hash function.

ciaogiorno• 15 years ago

Why can't we have a web interface for GitHub?

I should be push a new release right from my browser.

wizard_2• 15 years ago

Just posted to the support page.

"Sorry guys, there was a Javascript error displaying an error when there was none. We've deployed a fix for this and everything should be good to go."

andrewljohnson6• 15 years ago

This seems to be fixed now - for my repo at least.

The GitHub guy saying it was a "JavaScript display error" is incorrect though. I was pulling a repo and getting an old version.

EDIT: I'm not getting the javaScript display error, but I'm still missing my commit, so I guess these errors are unrelated :/

mojombo• 15 years ago

Did you file a support ticket for this? It sounds like an unrelated problem. We can investigate if you let us know what repo is causing the problem. Visit http://support.github.com or email support@github.com.

andrewljohnson6• 15 years ago

This was my fault. GitHub was never messed up.

mfukar• 15 years ago

For what it's worth, I was pulling & pushing just fine while the error was displayed on my repositories, but other people reported commits lost or mangled. Maybe there was something more to the Javascript display error?

btipling• 15 years ago

Well luckily you have the entire repository where ever it exists on your dev machines. So I don't expect there would be any data loss. In addition all the commits, tags, trees and files are hashed. The content is hashed, if there is a problem you'd know about it.

antirez• 15 years ago

Suggestion for creating a startup:

For private repositories gitolite is really awesome, but I think there should be a company about it. What I mean is that github or some other company should offer hosted gitolite instances, in the following form:

You pay something like price_of_small_linode_instace+price_of_service, to get:

A virtual machine with gitolite up and running, with a good web interface and online documentation. I think that the web interface part is the piece this company should not opensource. With this web interface you can create repositories and users, and modify the gitolite settings in general.

A backup service, in some other different VM and network, so that if something goes wrong with your gitolite VM you can ask a free (and automated) restore, possibly with hour granularity for the latest 24 hours, and less granularity for bigger periods and so forth.

Note that this gives the user virtually infinite repositories... if they don't upload movies to their instance, as there is a lot of space even in a small linode instance for git repositories.

This way this is a "pure product" as there is no need to repair or install manually virtual machines, and the company can focus into evolving the offering, developing a better version of gitolite, of the web interface, and so forth.

p.s. I mentioned linode but can well be ec2 or whatever is better suited for this stuff.

icefox3• 15 years ago

What do people really think of this idea? Would you buy it? I have written a web interface for managing git repos called GitHaven (my personal home install is here: http://git.meyerhome.net:8080/). It has the usual goodies, private/public repos etc. I have some friend's companies that run it in the cloud already. Do you think it would actually make money i.e. Would you be willing to join me in a startup on it (in boston)? Would it pass the y-combinator test?

tptacek3• 15 years ago

Probably not. Github is on a path to become the market leader, has a critical mass of customers, and has a business model that harnesses those customers to benefit from network effects. This idea is akin to building a business offering private white-label Stack Overflow clones. Key point: the actual software is not what makes Stack Overflow and Github valuable.

Simpler reaction: Github:FI is already priced at a rounding error for the kinds of companies from which most of the revenue in this space will originate.

GitHub major issues - repos have "lost" commits and site is erroring

Related Stories