Understanding word vectors

https://gist.github.com/aparrish/2f562e3737544cf29aaf1af30362f469

by jxub • 8 years ago

354 69 8 years ago

gist.github.com

ZeroCool2u3• 8 years ago

This is a highly localized problem, but I've wanted to read the past couple articles by Allison and cannot, because my org has to block gists.

Allison we know this is a lot of work for a very small group but, if you see this, a couple of us here would be super stoked if you could mirror your articles somewhere else as well!

wyldfire• 8 years ago

Boy, that's a shame for them to do that. But I've been in similar places before. Do they also block reg'lar github.com?

ZeroCool2u3• 8 years ago

Yeah, but we're part of a very small subset of places that genuinely need this level of security. I used to do counter-threat infosec here before I moved to data science, so I know how much work goes into our security.

Regular Github is free and clear, so we're good there.

titanomachy• 8 years ago

They should block stack overflow as well to make absolutely certain nobody can do any work.

What's it like being a software dev in North Korea?

ZeroCool2u3• 8 years ago

The cabbage is subpar.

wenc7• 8 years ago

The cognoscenti already know this, but word vectors are a game changer in the multi-class text classification game. Finding the correct representation of text makes the classification task so much easier.

For my problem (1000+ total classes, 1 class per input), I experimented with Naive Bayes + TFIDF (~50% accuracy, < 1 sec training), then Word2vec + CNN model on GPU (~70% accuracy, 6 hrs training), and finally FastText (99% accuracy, 10 minutes training).

FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams, and there is a default implementation in C++ with a built-in classifier that runs on the command-line (no need to setup Tensorflow or anything like that).

Despite it only running on plain CPUs and only supporting a linear classifier, it seems to beat GPU-trained Word2Vec CNN models in both accuracy and speed in my use cases. I later discovered this paper from the authors comparing CNNs (and other algorithms) to FastText, and their results track my experiences [1].

This goes to show that while GPU-accelerated models are cool, sometimes using a simpler, more suitable model can have a significantly better pay-off.

[0] https://fasttext.cc/docs/en/supervised-tutorial.html

[1] https://arxiv.org/abs/1607.01759

ppod3• 8 years ago

If the word embedding is, say, 400 dimensions, are you averaging the vectors of all the words in the document to get a document vector?

isoprophlex3• 8 years ago

Just piping in to say that I'm very curious as well! How do you go from word vectors to a description of the document?

wenc7• 8 years ago

For the CNN model I was using Magpie [0] which takes a sum of the vectors (so yes, equivalent to unweighted average) to represent the document.

For the FastText model, it would be whatever FastText is doing under the hood. I haven't peeked at the code to see what it is doing, but Section 2 of the paper I cited above seems to imply that an average is taken.

[0] https://github.com/inspirehep/magpie

bomb199• 8 years ago

Thanks for this, I'll have to take a look at FastText. I've been using word2vec before turning it into a matrix and running it through a CNN. I based it off of Yoon Kim's[0] work. I haven't had much luck though on my 92-class problem. Maybe Fast Text will work better, although I think there are a lot of improvements my model can have still.

[0] https://arxiv.org/abs/1408.5882

akhilcacharya• 8 years ago

I'm working on a problem kinda similar to that with a binary classification on a class imbalanced dataset, but fastText and Bidirectional LSTMs appear to work pretty terribly even with oversampling. Is there a better alternative?

wenc7• 8 years ago

Class imbalance is a tricky problem, but it is unrelated to FastText.

There is no silver bullet. The best solution is to collect more data to bring the classes to a balance. The second best approach is to try algorithms like SMOTE.

aisofteng2• 8 years ago

FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams,

To clarify: fasttext does not support n-grams of words, but instead considers n-grams of characters within words.

wenc7• 8 years ago

It supports word n-grams as well as character n-grams https://fasttext.cc/docs/en/options.html

See options -minn, -maxn and -wordNgrams.

melzarei• 8 years ago

For anyone interested in more explanation, Stanford CS224N course has 2 great lectures on them.

ovi256• 8 years ago

A few remarks:

In the "Doing bad digital humanities with color vectors", if you consider colors as 3D vectors, which they do, you'll see that summing enough uniformly sampled vectors always gives medium browns, because that's the color in the middle of the colorspace. Instead, you should model colors in a polar space and sum vectors in that space. This will prevent going inside the sphere and losing color saturation.

It's explained quite well here in the Interpolation section: http://www.inference.vc/high-dimensional-gaussian-distributi...

If you want to understand contemporary use of words embeddings in ML, a nice simple model is explained, with full code, here: https://blog.keras.io/using-pre-trained-word-embeddings-in-a...

The original model comes from Kim 2014, https://arxiv.org/abs/1408.5882 It's a very neat use of CNNs for language processing, instead of the more popular RNNs/LSTMs. CNNs have the advantage of training much faster.

Ethcad• 8 years ago

Sorry if this is a dumb question, but wouldn’t it be a medium gray in RGB? Or are you talking about HSV?

Maybestring• 8 years ago

HSV is a polar model. (Holding constant Value)

wodenokoto2• 8 years ago

Are rnn that common for wordvectors?

I mostly see word2vec and fasttext, neither of which are CNN nor rnn

aglionby6• 8 years ago

Yes, I've not looked at fasttext but word2vec is a simple 1 hidden-layer network to learn word embeddings, which can then be used as pre-trained word embeddings in other tasks.

CNNs and RNNs would be used in the next stage of the pipeline for whatever your task is (machine translation etc), probably as some way of combining the word vectors. RNNs are especially nice as they can deal with variable length sentences. Note also that it's possible for systems to learn their own systems as part of training, rather than using pre-trained ones from word2vec etc.

hoerzu• 8 years ago

Where is T-SNE? ;)

jxub• 8 years ago

For anyone who had to google it like me, T-SNE is T-Distributed Stochastic Neighbor Embedding, and the way I understand it is that it's essentially a statistical way to reduce data dimensionality while still preserving its structure.

b_tterc_p• 8 years ago

Dimension reduction technique. For word vectors, the usual use case is to take the 400 dimension vector and turn it into a 2 dimension vector that you can use to plot like a scatterplot. Similar things will be close together. Dissimilar things will be far apart. It's kind of like principal components analysis but quirkier.

For those who like Tsne. Check out the relatively new UMAP, which seems to be faster and better.

radarsat1• 8 years ago

Gem in here is that I learned about Annoy. https://pypi.python.org/pypi/annoy

Radim3• 8 years ago

You might be interested to know that Annoy is also integrated into Gensim, which allows you to train, use and query word embeddings on your own data. Gensim implements fast Doc2Vec and FastText too, which are a bit newer embedding techniques [0] :-)

[0] https://twitter.com/gensim_py/status/969222857246101504

patelajay2854• 8 years ago

We've recently created an open source library that can help you get started with pre-trained word vectors and Annoy quickly:

https://github.com/plasticityai/magnitude

Radim3• 8 years ago

Interesting effort :-) Unfortunately, your comparison table there is somewhere between misleading and insulting.

Almost all of the "unique" features listed there are in fact a standard part of Gensim. Fast approximative queries (using Annoy), memory mapping with lazy loading, ngrams features, format convertors, Python interface, parallelization, pre-trained models for download…

There is a way to promote cool new libraries, but this ain't it.

stared• 8 years ago

For an technical intro for word2vec (what's that exactly, how to train it): https://lilianweng.github.io/lil-log/2017/10/15/learning-wor... and its readable implementation in PyTorch: https://adoni.github.io/2017/11/08/word2vec-pytorch/.

And just for getting idea why does it work, and play with examples in your browser: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html

visarga2• 8 years ago

Word vectors are at the same time amazing because they contain a huge amount of latent information, and not good enough because they collapse a space of very high dimensionality in ~300 dimensions, so they have a limit to how much they can discriminate between close topics. I have done a lot of experiments on classifying text in thousands of topics, and sometimes they work amazingly well, other times they are really hard to use, depending on how close together are the topics I want to discriminate between.

Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

A third problem of word vectors is out of vocabulary words and words with low frequency. For OOV, the usual solution is to create character or character-ngram embeddings that can be used to compute embeddings for new words. For low frequency words we usually ignore them (apply a cutoff).

Then there is the problem of phrases and collocations - some words go together, such as "New York" and "give up". The meaning of the phrase is different from the sum of the meanings of the component words. In these cases we need to have lists of phrases and replace them in the original text before training the vectors, so we have proper vectors for phrases.

By the way, one amazing tool that goes with word vectors is the library 'annoy' which can do similarity search in log time. So you can do approx 1000 lookups per second per CPU even if the database contains millions of vectors, pretty good. Annoy can be used to find similar articles, or music recommendations. Another remark - my preferred word vectors are computed with Doc2VecC (a variant of doc2vec with corruption). Doc2VecC seems more apt to discriminate between topics, but the secret is to feed it gigabytes of text.

Playing with word vectors has taught me intuitively how it is to navigate a space of high dimensionality. It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective.

doc2vecC: https://github.com/mchen24/iclr2017

annoy: https://github.com/spotify/annoy

hiker512• 8 years ago

Can you recommend an implementation or paper for handling OOV words via character level embeddings?

patelajay2854• 8 years ago

We just open sourced a easy to use library called Magnitude that handles out-of-vocabulary words and uses Annoy indexing for fast most_similar queries for word2vec, GloVE, and fastText:

https://github.com/plasticityai/magnitude

wenc7• 8 years ago

Just an FYI, FastText's default implementation handles OOV words via word n-grams and character n-grams. (see switches -minn, -maxn, and -wordNgram)

https://fasttext.cc/docs/en/options.html

RoboTeddy• 8 years ago

we needed first to find the right sense for each word in a phrase and only then assign the vector

Interesting! I wonder if you could you e.g. arbitrarily split a word into some number of symbols, e.g. two, and each time you're going to apply a training update, only apply the update to one of the symbols -- perhaps initially choosing the symbol facing the greatest loss (forcing the symbols apart in vector space), and then eventually switching over to picking the symbol with the smallest loss (letting each settle onto its own precise meaning)?

aglionby6• 8 years ago

Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

This is interesting, and there seems to be a bit of debate about it (at least with compositional distributional semantic models). [0][1] seem to show that sense disambiguation helps in some contexts, and [2] show that they don't in others. It doesn't seem immediately clear who is right here. I agree with you though that it seems pretty likely that disambiguating would be helpful.

[0] https://www.aclweb.org/anthology/W13-3513

[1] https://www.aclweb.org/anthology/P16-1018

[2] https://www.aclweb.org/anthology/D10-1115

kevinwang2• 8 years ago

It seems obvious to me that disambiguating _correctly_ would necessarily improve performance. As an extreme example, homonyms like bark (the verb) and bark (on a tree) have nothing to do with each other, and ideally would be considered two different words.

nl• 8 years ago

Actually...

This seems like it should obviously make sense, but in practice it doesn’t necessarily. Usually the sense in defined by the context, and so the vector can carry the meaning of multiple contexts at once.

There are some specific tasks where this breaks for some words. I can’t remember one I’ve seen right now, but thinking about the “bark” example, it can cause a problem when both the “woodyness” and “noisyness” of a word cause completely different results. In practice that’s pretty rare.

Generally though, the non-relevant meanings don’t have any negative effects as they can be ignored.

aisofteng2• 8 years ago

Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

There has been work on representing words not as vectors but as multimodal Gaussian distributions in order to try to deal with polysemy, such as [0], of which an implementation (which I have not tried to use) is available on GitHub at [1].

It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective.

I appreciate that the author of the parent comment has found some sort of intuition, but I would caution others from trying to use the above quote in order to develop their own intuition as it is meaningless in any rigorous sense.

[0] https://arxiv.org/abs/1704.08424

[1] https://github.com/benathi/word2gm

debt• 8 years ago

"Poetry is, at its core, the art of identifying and manipulating linguistic similarity."

ah shit this makes me want to stand on a desk.

juanmirocks• 8 years ago

Love the practicality of this simple practical tutorial. The only pity is that it's in Python 2

cup-of-tea• 8 years ago

Yeah, I really don't get why they would use Python 2 for this kind of thing.

sumitgt• 8 years ago

My mind was blown when I found out how easy it was to get started using pre-trained Glove embedding in Keras. Took my Kaggle game up a few notches over-night.

https://blog.keras.io/using-pre-trained-word-embeddings-in-a...

Understanding word vectors

Related Stories