The Hugging Face Datasets Server is now open-source

https://github.com/huggingface/datasets-server

by taubek • 4 years ago

219 29 4 years ago

minimaxir• 4 years ago

It's worth noting that the reason this works well is that Datasets is backed by Arrow, which allows streaming remote data. More info in the base library docs: https://huggingface.co/docs/datasets/stream

AnhTho_FR• 4 years ago

So much progress since the tamagotchi!!!

joshxyz• 4 years ago

LMAO welcome to the future!!

jbverschoor3• 4 years ago

Note that you probably need DocArray for this as the core building block

arthurcolle2• 4 years ago

I saw the DocArray post earlier, is it really used in this?

jbverschoor3• 4 years ago

Nah, I'm just kidding. I don't know, probably not :)

I never heard about the hugging face before that post.

f38zf5vdt• 4 years ago

Surprised to see it's all Python.

anaganisk5• 4 years ago

Proves over optimisation(rust, kube etc) vs coded in a language easy to pick, understand and perfect.

stoplying12• 4 years ago

The only thing proved is that the authors chose python. It likely had more to do with necessary library/functionality availability than anything.

j-sizz• 4 years ago

Why just the first few rows

surajs• 4 years ago

what at time to be alive!

The Hugging Face Datasets Server is now open-source

Related Stories