FYI: LLVM-project repo has exceeded GitHub upload size limit (2022)

https://discourse.llvm.org/t/fyi-llvm-project-repo-has-exceeded-github-upload-size-limit/63293

by hippich • 3 years ago

147 78 3 years ago

discourse.llvm.org

mooreds• 3 years ago

Would git prune ( https://git-scm.com/docs/git-prune ) help here? Or is the entire repo of reachable objects more then 2GB?

Arnt4• 3 years ago

The latter. It's >20 years of history, and if you want a straw test as to whether the history is relevant: I personally needed to look at >20-year-old history this winter.

zffr• 3 years ago

Why did you need to look at >20 year old history?

trynewideas• 3 years ago

Considering this from the thread, as a response from GitHub to their repack request:

As it turns out, we only run repacks on a repository network level, which means that repacks need to consider objects from all forks of a given repository.
Repacking entire repository networks will always lead to less optimal pack sizes compared to repacking just objects from a single fork. For GitHub, disk space is not the only thing we optimize for, but also performance across forks and client performance.

Repacking the repo locally more than halves the size:

I tried locally right now to run git repack -a -d -f --depth=250 --window=250 and the size of the .git folder went from 2417MB to 943MB…

But GitHub repacking the repo network reduces the size by only 20%. So presumably GitHub can't aggressively remove things from a repo without affecting how forks work.

This blog post is very old (2015) but there's a section that describes their use of Git alternates to facilitate forks: https://github.blog/2015-09-22-counting-objects/#your-very-o...

secondcoming• 3 years ago

June 2022

t344344• 3 years ago

It only happens if you push entire repo at once. Some script may split upload into multiple smaller parts and solve this.

silverwind• 3 years ago

Sounds like Git CLI could do this automatically, if it can detect these errors (I think it can't).

pionar• 3 years ago

This is a GitHub-specific issue, so I don't think Git should do anything about it.

oauea• 3 years ago

As it turns out, we only run repacks on a repository network level, which means that repacks need to consider objects from all forks of a given repository.
Repacking entire repository networks will always lead to less optimal pack sizes compared to repacking just objects from a single fork. For GitHub, disk space is not the only thing we optimize for, but also performance across forks and client performance.

So the lesson here is you can DoS existing open source projects somehow by forking them and increasing the forked repo size >2GB?

tagraves• 3 years ago

How would the DoS work? As I understand it the issue here only occurs if you try to push the entire project to a _new_ repo. It doesn't affect existing repos.

pointlessone2• 3 years ago

I'm almost certain it affects all repos. It's just more unusual to push huge packs into exiting repos. If for whatever reason you happen to have a huge branch clicking in over 2GB you'd get the same error.

mayli• 3 years ago

True, for github.

pointlessone2• 3 years ago

I don't think that's the case.

The issue is that GH doesn't accept too big packs. git by default pack everything into a single pack. Maximum pack size can be specified either in config or as an argument to repack. The way I read the error message a user can push a huge repo by making sure it's packed into a few packs under 2GB limit.

It doesn't seem like there's an easy way to turn this into a DoS. GH would repack the fork network on its own schedule. A user probably can not trigger repacks. The repacks on GH side would probably be smaller then their limit, too. packs are probably scoped to a fork and the server is an active client so it most likely wouldn't return objects from other forks. I don't think it would be easy to DoS GH just by pushing big packs (either under or over the limit).

CoastalCoder• 3 years ago

I'm not sure what the current status of this issue is, based on the linked Discourse thread.

Is it still a fundamental limitation, but with a known client-side workaround?

rebolek• 3 years ago

It's interesting that the post doesn't deal deal with the basic problem that if your repo is +2GB, the problem may be on your side.

sammyteee• 3 years ago

How so? You realize there can be more than just code in a repo? And even then, what if there is a lot of code?

psychphysic• 3 years ago

He said 'may be'.

Are you really suggesting it's impossible for bad practices to bloat a got repo?

mrguyorama2• 3 years ago

What large data assets does a compiler have? Is there an HD video for you to watch while you compile?

hobofan• 3 years ago

You realize there can be more than just code in a repo?

Yes, and for large assets there are extra solutions like Git LFS (which GitHub has support for).

lwhsiao• 3 years ago

Only tangentially related, but I've found that git-sizer [1] is handy for getting a sense of repository metrics.

[1] https://github.com/github/git-sizer

low_tech_punk• 3 years ago

On a philosophical note, git, being a distributed source control system has no limits to repo sizing, but github, being is a centralized "hub" suffers scaling problems. It's fascinating to watch these tradeoffs play out. Maybe, IPFS would come to the rescue one day.

cabirum3• 3 years ago

The problem here is unbounded growth of a git repo. In this specific case, a size limit was triggered. In other circumstances, it would have required too much time to transfer or no storage space left.

Anyway, the problem is that git stores all changes, forever. A better approach would be to clean old commits, or somehow merge them into snapshots of fixed timespans (like, anything older than a year get compressed into monthly changesets)

mrguyorama2• 3 years ago

Git allows you to rewrite history, so you can "crush" old commits to reduce the size of history as needed.

howinteresting4• 3 years ago

This is not a "problem", this is why source control exists. You should never rewrite published history.

Arnt4• 3 years ago

I believe the problem here is related to LLVM using a CI model, where all changes eventually go to a single tree and tests are run on that single tree. Hosting that CI happen on another site than github wouldn't really change anything. Maybe change the limit, but all such CI hosts have a limit, for the same reason that all the search engines have a size limit for web pages they scrape.

tsimionescu• 3 years ago

I don't understand where git's distributed nature comes into play here. Perforce is fully centralized, and it also doesn't have any limits to repo size - as long as you have enough disk, P4 will handle it.

In fact, Git's distributed nature actually makes it infamously bad at scaling with repo size - since it requires every "participant" in a repo to have a copy of the entire repo before they can start any work on it.

dblitt• 3 years ago

This is where something like VFSForGit [0] helps out. Instead of cloning the entire repo, it creates a virtual file system and fetches objects on demand. MSFT uses it internally for the Windows source tree (which now exceeds 300GB).

[0] https://github.com/microsoft/VFSForGit

rvbissell• 3 years ago

Shallow cloning of Git repos is a thing. Basically, you get a fake head commit (that includes all the files) and none of the real history. Useful if you only intend to build, or make changes locally. If you want to push, you have to unshallow first.

cycomanic• 3 years ago

Funny I just encountered something similar last week. My org runs a gitlab instance and I creates a repo with lecture slides etc.. The slides are html files and some of them have generated videos (for various reasons I wanted to upload the slides not the source scripts). Turns out that my org put a 150MB upload limit and one of the commits exceeded that limit, so my push was failing. The fix was non trivial, but git-sizer was definitely helpful.

MithrilTuxedo• 3 years ago

I ran into this at work trying to push a P4 clone that sat at 20GB in my local .git folders. Needless to say, I didn't have room on my local to actually check out a workspace.

A shell loop can push N commits at a time, using git's handy syntax for that:

`git push origin p4/master~${x}:master`

weinzierl• 3 years ago

Built LLVM about a year ago and not only is the source large but the build artifacts filled my disk to the brim.

mshockwave• 3 years ago

There are some workarounds to shrink the size. I think the most noticeable ones being only build certain backends (-DLLVM_TARGETS_TO_BUILD="X86;..."), built as shared libraries (-DBUILD_SHARED_LIBS=ON) and split (DWARF) debug info for debug builds (-DLLVM_USE_SPLIT_DWARF=ON). Plus, usually I just build the tools I need instead of firing `ninja` or `ninja all`

sixstringtheory• 3 years ago

Maybe at this point it shouldn’t be hosted or even mirrored on GitHub.

CocoaPods and Homebrew both hit similar issues in the past with huge work trees by hosting all their specs on GitHub, resulting in breaking workflows for their users.

Those projects made changes to their workflow to mitigate this. Since the source here is around 6 months old, have LLVM done something similar?

trogdc• 3 years ago

No they still use GitHub, and there are plans to switch code reviews to use GitHub PRs as well.

newaccount74• 3 years ago

Wasn't the problem with brew that they were (maybe still are) using the git repo as a content delivery network? As I recall the problem wasn't the repo size itself, the problem was millions of users have to do a "git clone" to update homebrew and it overwhelmed Githubs infrastructure.

londons_explore• 3 years ago

I think github has some whitelist for large popular projects. Chromium is hundreds of GB for example, but you can still fork it on github and make any changes you like and push them.

madeofpalk• 3 years ago

If I understand correctly, the limitation is on the size of a push.

If you happened to push the entire llvm-project to another new (personal) GitHub repo recently, you might encounter the following error message before the whole process bails out:
The crux here is that we tried to push the entire repo, which has exceeded GitHub’s upload limit (2GB) at once. This is not a problem for majority of the developers who already had a copy of llvm-project in their separated GitHub repos

If you fork the repo on the Github website, they manage the fork server side, and you never have to push up the entire repo.

billti• 3 years ago

Tangentially related, but after installing LLVM on Windows I notice that the clang, clang++, clang-cl, and clang-cpp binaries are all identical. That's four copies of the same 116MB file. The linkers (ld.lld, ld64.lld, lld, lld-link, wasm-ld) are likewise the same 84MB binary repeated 5 times, and similarly llvm-ar, llvm-lib and more are identical.

Overall looks like ~750MB of unnecessary file duplication for one install of LLVM.

I get that Windows doesn't have the same niceties when it comes to symbolic links as macOS and Linux, but that seems really suboptimal and wasteful.

nequo• 3 years ago

If not symlinks, might those be three or four hard links to the same file? I’m not sure how to check on Windows.

tadfisher• 3 years ago

NTFS has had symbolic links since Windows Vista. You enable them in Git with the `core.symlinks` setting. There may be other reasons to avoid them, though; for example, if some tooling like VC++ can't deal with calling symlinked executables. I don't use Windows so this is all conjecture.

WirelessGigabit2• 3 years ago

You still need to enable developer mode to make sure that applications can use it without elevation.

https://blogs.windows.com/windowsdeveloper/2016/12/02/symlin...

nativecoinc2• 3 years ago

I cloned

    git clone https://github.com/llvm/llvm-project

When entering the directory my fancy prompt timed out

    [WARN] - (starship::utils): Executing command "git" timed out.

But it seemed OKish after `git status` (disk cache?)

    time git status
    ...
     real 0m0.287s
     user 0m0.189s
     sys 0m0.392s

The size of `.git/objects` after the clone was 2.5GB.

Output of `git-sizer`: https://pastebin.com/T5HRMfg9

DethNinja• 3 years ago

What is taking so much space though, why not just use LFS? It can't possibly just be the text files.

ok_computer2• 3 years ago

Hi, I’m curious here would git-lfs (large file storage) work along with rewrite the history to remove the larger files and artifacts?

    git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch /path/to/file" HEAD

https://stackoverflow.com/questions/43762338/how-to-remove-f...

Or is git-lfs generally avoided for good reason? Or so all the changes and not necessarily large files cause this repo to exceed the limit?

WirelessGigabit2• 3 years ago

Then you're rewriting history. And this should be avoided at all cost, as git hashes are part of the cryptographic history of a repo.

Who are you going to trust to do this? Someone will need to execute this and then do a force push. Are you going to compare it? Are you going through all the history of files removed to see if there is no change to the source code? And what about the files you're actually going to move to git-lfs? How can you prove that they haven't changed in the migration?

Provenance is a thing. https://slsa.dev/provenance/v0.2

ok_computer2• 3 years ago

I'll admit I've done this early on in history of a pet project repo scrubbing a credential :/, not implementing git-lfs though.

I knew that rewrites compromise the history. It is low stakes and I didn't want to allocate a new repo to start anew, just learned my lesson there.

I'm mostly curious about git-lfs as large file viability or it should be avoided and plan for large build artifacts to host elsewhere.

saurik• 3 years ago

As LLVM is a monorepo of a number of different (giant) projects that kind of feel separate--and in fact used to be separate--I often wish I could have a submodule direct to a subfolder so I don't have to do a checkout of all of LLVM.

GoblinSlayer2• 3 years ago

They seriously wrote 2gb of C++?

nativecoinc2• 3 years ago

The total size of the files in the repo is 1.35 GiB. The total size of all the blobs is 84.9 GiB.

https://pastebin.com/T5HRMfg9

FYI: LLVM-project repo has exceeded GitHub upload size limit (2022)

Related Stories