Pure Rust implementation of a minimal GPT language model

by keyvank • 3 years ago

20 4 3 years ago

keyvank• 3 years ago

femtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer.

Everything is implemented from scratch, including the tensor processing logic along with training/inference code of a minimal GPT architecture.

allisdust• 3 years ago

Thanks for making this. Was searching for a pure rust GPT implementation just the other day!

mrkeen• 3 years ago

After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output

I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?

I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).

So what are the expectations of this GPT model? Should it be bound by the same limitations?

gandalfff• 3 years ago

Does this have potential for running on an SBC? Mine only has 2GB of RAM and I'm interested in running a toy language model on it.

Pure Rust implementation of a minimal GPT language model

Related Stories