SpaCy PyTorch Transformers

https://explosion.ai/blog/spacy-pytorch-transformers

by syllogism • 7 years ago

106 11 7 years ago

clmnt• 7 years ago

Very cool!

3wolf• 7 years ago

I'm surprised transfer learning via fine-tuning large transformer models hasn't taken off more in the public consciousness a la image recognition models. In my experience, the results can be staggering with very small amounts of training data.

ipsum2• 7 years ago

Google (and I assume most ML providers) offers transfer learning on image models: https://cloud.google.com/vision/automl/docs/

ma2rten• 7 years ago

It's a very new thing. Pretrained ImageNet models were first released around 2011. Pretrained transformer models have only been released recently.

Also, transformers are only useful for short text, not full documents (AFAIK).

bigmit37• 7 years ago

are transformer models are the text version of imagenet model? The first time I am hearing this term.

p1esk• 7 years ago

It has taken off. Pretty much everyone and their grandma has fine tuned gpt-2 to generate all kinds of stuff, even poetry: https://www.gwern.net/GPT-2

shiredude95• 7 years ago

Awesome! Cant wait to try this out. Are there any plans in the future to incorporate sentence/word generation?

syllogism• 7 years ago

That would be cool, yeah. We haven't had anything like that in spaCy so far, but the results from these models can be very good. I think with a good API it could be very useful.

arathore• 7 years ago

Spacy also has one of the best documentations that I have come across. I really like the use of Tufte-style notes for code examples. https://spacy.io/api

jonbaer• 7 years ago

Also the recent SpaCy IRL talks are well worth watching @ https://www.youtube.com/watch?v=hNPwRPg9BrQ&list=PLBmcuObd5A...

goldemerald• 7 years ago

This stuff is great! Will example training files be uploaded for GLUE/SQAUD/etc like on the HuggingFace's implementation?

SpaCy PyTorch Transformers

Related Stories