Pre-training a 15B parameter language model over the internet

https://distro.nousresearch.com/

by modeless • 2 years ago

26 7 2 years ago

distro.nousresearch.com

arilotter4• 2 years ago

Yo! I'm on the team at Nous that built this tech, happy to answer any questions :)

modeless• 2 years ago

Awesome work! Are you planning to eventually do a SETI@home style thing where anyone can join? Also, what was Durk Kingma's involvement in DisTrO?

arilotter4• 2 years ago

Are you planning to eventually do a SETI@home style thing where anyone can join?

That's one of the goals of the stuff we're working on right now, I personally hope we can make it work!

What was Durk Kingma's involvement in DisTrO?

As a co-author on the paper, we brought him in to bounce some ideas off of & have him validate DeMo's design and implementation to ensure we weren't hallucinating these results.

dcreater• 2 years ago

Will this enable federated learning? Petals.ml and distributed inference didn't actually work - the technical aspects of what you're doing to enable distributed training is well over my head unfortunately

arilotter4• 2 years ago

you could absolutely use DisTrO for federated learning. The DeMo optimizer on its own doesn't solve the adverserial aspects of training on local-only data, nor does it solve tensor parallelism across devices, so you're still limited to only what fits on your local GPUs, but it does enable distributed data parallelism over the internet at a bandwidth orders of magnitude lower than before.

timtom123• 2 years ago

I'm excited to test the final model. This could be a major breakthrough for open-source LLMs.

arilotter4• 2 years ago

This specific model is only trained on 100 billion tokens, so it's not SOTA by any means, but we've got designs on larger training runs later :)

Pre-training a 15B parameter language model over the internet

Related Stories