Download all of your GitHub data

https://blog.github.com/2018-12-19-download-your-data/

by TimWolla • 8 years ago

151 38 8 years ago

blog.github.com

micky_t• 8 years ago

but I still cannot delete it...?

judge2020• 8 years ago

How would deleting work? They can't really anonymize any commit data without breaking git clones, so I guess deleting data would just delete anything non-git like PR text, issues, etc?

mehrdadn• 8 years ago

Yeah, I think deleting issues etc. should be possible.

avar• 8 years ago

There was a big discussion about this on the Git ML recently: https://public-inbox.org/git/20180417191549.GA1199@helen.PLA...

I.e. whether the argument that you're exempt from deleting data under the GDPR just because it's inconvenient for your storage format is going to fly in the EU. If not there's scary times ahead for Git.

Here's a good example of a 4 year old using (presumably) her full name committing to the kernel: https://github.com/torvalds/linux/commit/690b0543a813b0ecfc5...

If that kid grows up, moves to the EU, and wants Microsoft (GitHub) to stop publishing that content, will they be able to just say "no"?

How about if the kid has had their personal data embedded in the Bitcoin blockchain?

Interesting times indeed.

fingerprinter• 8 years ago

You've been able to permanently delete your GitHub profile and all data associated with it for quite some time.

https://help.github.com/articles/deleting-your-user-account/

I've seen this 'ghost user' around on threads for years....

AdmiralAsshat• 8 years ago

Now what I would really like to see is some tool for keeping my GitHub repos in sync with my GitLab ones.

I did a one-time migration from GH to GL, but my GH account is still open, primarily because some projects are GH-only, meaning I might continue to make forks, contributions, etc.

Some way of making sure that I can periodically get all of those over to my GL would be nice.

kbenson• 8 years ago

Not that I'm all that good at the more esoteric features of git, but you can overwrite the history with the correct flags, right? It seems like all you would need to do to sync them is have the most recent checkout of each, a way to determine which was newer, and push the newer branch over the older one (to be simplistic and destructive).

If you're always overwriting history of one to make it match the other, diffing the git logs of each against each other would likely tell you which had items the other didn't, and if it went just one direction or both (i.e. can you just overwrite one with the other or is some manual intervention required).

MereInterest• 8 years ago

The git repo itself is the easy part. You just make a local clone with two remote repos, pull from one, and push to the other. The hard part is all the parts that aren't managed by git, the issue tracker, PRs, releases, etc.

cfstras• 8 years ago

I‘m using the setup linked below to sync a few repos to gitlab. It uses GL pipelines to do the sync, and a webhook to do it instantly (however you can trigger pipelines using curl, or on a timer in GL)

https://gist.github.com/cfstras/5e40ea264517552e29caf61bbd28...

anon4242• 8 years ago

Git is after all designed for this distributed scenario, so you'd think it would be possible, what about something like:

Setup:

    git add remote gitlab ...
    git add remote github ...

Then whenever you need to sync:

    git fetch --all
    git checkout -b gh --track github/master
    git checkout --track gitlab/master
    git merge/rebase gh
    git push gitlab

Could probably be done shorter but those steps should do it..

johannes1234321• 8 years ago

This has the issue that you have to add this to your Workflow and nice wrappers have to be put on all workstations.

A mirroring setup like provided in GitLab (see sibling post) takes that burden away.

frutiger• 8 years ago

Check out the `--mirror` flag.

scrollaway• 8 years ago

Gitlab itself supports being a mirror of an upstream GitHub remote (or any remote). Last I checked it doesn't deal well with force pushes but I haven't checked in a long time.

sytse• 8 years ago

Thanks for using GitLab! Please also see https://docs.gitlab.com/ee/workflow/repository_mirroring.htm... Pushing free to use, pulling is paid feature but public projects can use it for free too.

mohsen1• 8 years ago

Is this for GDPR compliance?

ceejayoz• 8 years ago

Almost certainly, but it's nice they're making it generally available. I've seen some sites' GDPR exports limited to EU customers, which seems like a dick move.

notimetorelax2• 8 years ago

The thing is, sites cannot use IP geolocation to decide if their users are EU residents or not. I think in the end it’s more expensive to try to manage this than to let everyone do the export and deletion.

Someone1234• 8 years ago

Reminds me of California's labels in that way. Everyone mocks them, but some of them are legitimately useful e.g.:

https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-c...

Same thing here, because border specific stuff aren't worth the hassle, the lowest common denominator (in this case GDPR) becomes the new international standard.

Additionally some US politicians have proposed copying parts of the GDPR into US law which would pretty much make this a permanent staple.

eikenberry• 8 years ago

Anyone know if it includes information on which repositories were given stars?

TimWolla• 8 years ago

It does not. This is what it looks like for me (files with number > 1 omitted for brevity):

    -rw-r--r-- 1 timwolla timwolla  499 Dec 19 19:31 attachments_000001.json
    -rw-r--r-- 1 timwolla timwolla  52K Dec 19 19:31 commit_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla  72K Dec 19 19:31 issue_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla  35K Dec 19 19:31 issue_events_000001.json
    -rw-r--r-- 1 timwolla timwolla  34K Dec 19 19:31 issues_000001.json
    -rw-r--r-- 1 timwolla timwolla  356 Dec 19 19:31 milestones_000001.json
    -rw-r--r-- 1 timwolla timwolla  22K Dec 19 19:31 projects_000001.json
    -rw-r--r-- 1 timwolla timwolla 4.4K Dec 19 19:31 protected_branches_000001.json
    -rw-r--r-- 1 timwolla timwolla  25K Dec 19 19:31 pull_request_review_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla 4.3K Dec 19 19:31 pull_request_reviews_000001.json
    -rw-r--r-- 1 timwolla timwolla  51K Dec 19 19:31 pull_requests_000001.json
    -rw-r--r-- 1 timwolla timwolla 9.0K Dec 19 19:31 releases_000001.json
    -rw-r--r-- 1 timwolla timwolla  80K Dec 19 19:30 repositories_000001.json
    -rw-r--r-- 1 timwolla timwolla   75 Dec 19 19:31 schema.json
    -rw-r--r-- 1 timwolla timwolla  27K Dec 19 19:30 users_000001.json

    attachments:
    total 12K
    drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:31 ./
    drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 c544f416-10c9-11e7-8337-82f88cac9fcd/

    releases:
    total 36K
    drwxr-xr-x 9 timwolla timwolla 4.0K Dec 19 19:31 ./
    drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 5aa3a138-9c86-11e5-99bc-79a6c3e509eb/
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 …
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 dba26980-d9f1-11e8-8357-3f2750c60e51/

    repositories:
    total 12K
    drwxr-xr-x  3 timwolla timwolla 4.0K Dec 19 19:30 ./
    drwxr-xr-x  5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 41 timwolla timwolla 4.0K Dec 19 19:31 TimWolla/

3pt141592• 8 years ago

I want to be able to freeze this. Give me 30 days to OK a bulk data download. I don't want a single lost access token to have such an easy way for someone to get everything.

driverdan• 8 years ago

A single lost access token + the API is an easy way to get everything too. It doesn't take much code to get everything that way.

3pt141592• 8 years ago

Well then I want that to be locked down too! I'm sick of the all-or-nothingness of many of these security measures.

akerl_• 8 years ago

A single lost access token that had global permissions already had the ability to get everything: GitHub’s API is already very full-featured.

The fix here is to use their scoping as intended: it should be amazingly rare to need an access token with such broad access.

benbalter• 8 years ago

When you request an archive of your data, we send the download link to your primary email address (the required token is not available via the web UI). Once you click that link, you'll be asked to re-enter your password. So for this particular feature, an attacker would need both your GitHub password (and your 2FA seed or an active session if 2FA is enabled) and access to your email.

jwilk• 8 years ago

There's also API (https://developer.github.com/v3/migrations/users/), which doesn't involve web UI or email.

The docs says it's "only available to authenticated account owners"; I hope it means you can't use a token for that, but I'm not sure.

ocdtrekkie• 8 years ago

Glad to see more sites jumping on this party, I just wish more tools were out there that interpreted data from these dumps directly and made them more readable.

I've started the practice of keeping backups of somewhat recent "all my data" dumps from each site that offers them.

afezelv2vk• 8 years ago

It seems odd for GitHub to be jumping through GDPR hoops - while at the same time RETROACTIVELY revealing all force-pushes [aka previously hidden and irretrievable by clone] in pull requests.

Of course, that data was compromised the second it was pushed, but, now they're parading the fact that people make mistakes...

What side are you on GitHub? Privacy? Or not?

saagarjha• 8 years ago

Force pushing code doesn’t make your old code private.

Sytten2• 8 years ago

That's an interesting point though, if you made the mistake of publishing a private key at some point and force-pushed your repo to remove it you expect it to be gone from the system, but it's not. I wonder if you could actually retrieve it from the cloned repo. Is it hidden somewhere in git?

EDIT: The cloned repos seems good, but following this post: https://stackoverflow.com/questions/3973994/how-can-i-recove... you can easily find some "lost" commit and retrieve secrets. I checked with one of my repo and I realized I had compromised information...

Sytten2• 8 years ago

Legit question: what about the data stored about you, but that you have no longer access to? I have no clue if that is also covered by GDPR?

stilist• 8 years ago

It wasn't clear to me from the blog post, but the data for events, issues, attachments, and so on is only available for repositories you own. Activity in repos belonging to organizations or other users isn't included.

Download all of your GitHub data

Related Stories