HuggingFace to Replace Git LFS with Xet

https://huggingface.co/blog/xethub-joins-hf

by skadamat • 2 years ago

18 7 2 years ago

huggingface.co

Kelteseth• 2 years ago

Git Repo: https://github.com/xetdata/xet-core

Sounds promising. Git LFS works ok ish, had often issues with it.

hhthrowaway1230• 2 years ago

The docs say:

it can also be configured against a local data store for a fully free and open source route to managing large repositories.

I can't find the docs for hosting oneself is that maybe available?

wdkrnls2• 2 years ago

Getting git to handle large data sets in regular repositories seems like a great way to go for many reasons. What is preventing the ideas of Xet from landing into git directly?

tredre3• 2 years ago

What is preventing the ideas of Xet from landing into git directly?

Xet needs three things to function (on top of git-xet itself): an auth backend (currently xethub), an S3-compatible backend (currently AWS), and an NFS client for accessing the files (currently your OS, if supported).

Presumably the git authors wouldn't be too keen on forcing their users to depend on those things just to work with a repo.

Git LFS isn't great but at least it doesn't need any extra services (local or otherwise).

wdkrnls2• 2 years ago

Your comment about Git LFS is intriguing. I have been put off from using git lfs because of the perception I needed a separate server to backup the binary files. I had heard of tools like lfs-folderstore (1). That no longer seems maintained and seems to be an extra service. Did git lfs integrate the lfs-folderstore ideas so it doesn't need them?

[1] https://github.com/sinbad/lfs-folderstore

thelastparadise• 2 years ago

does this break huggingface-cli download??

HuggingFace to Replace Git LFS with Xet

Related Stories