Abstractions my Deep Learning word2vec model made

http://byterot.blogspot.com/2015/06/five-crazy-abstractions-my-deep-learning-word2doc-model-just-did-NLP-gensim.html

by aliostad • 11 years ago

125 55 11 years ago

byterot.blogspot.com

platz• 11 years ago

reminds me of how Chinese words are made up of individual characters that have semantic meaning themselves.

sho_hn2• 11 years ago

The Korean Hangeul alphabet is an interesting compromise. It's an alphabet, but multiple letters are grouped together into syllabic characters when written. Those syllables in many cases map back to Chinese Han characters by way of sound values (a lot of the Korean vocab is Chinese in origin, even though the language have very distinct grammar), which means the boundaries between morphemes often match the character boundaries. This is reflected in the orthography, where in case of multiple options for how to distribute letters over characters, the option that keeps the same morpheme spelled consistently through use in different words is preferred. So while you can write phonetically as in Latin, the written language retains a high level of morphological information and things feel very Lego-like.

robinsloan• 11 years ago

"Lego-like" == what a great description.

aswanson2• 11 years ago

There seems to be a pattern in Asian languages that try to equate symbols that represent things to the physical respresentation. I recall a friend stating the Korean word/character for balance looked like a person holding jugs of water on each shoulder.

lqdc13• 11 years ago

I thought Word 2 Vec isn't "Deep Learning" as both CBOW and skip-gram are "shallow" neural models.

dave_sullivan• 11 years ago

How about a single layer neural net trained with dropout? Not deep learning because there's only 1 layer, but the technique is fairly new and used in deep learning, popularized by some of the guys that popularized deep learning, usually mentioned in conversations about deep learning. It's really a shallow neural model though. Word2vec is similarly related, but you're right in that it is not a perceptron-based neural network with multiple layers (ie "deep neural network")

Still, interesting blog post, worth reading and googling for more information. FWIW I just use the deep learning definition of "neural net or representation learning research since 2006" and find it fits better.

fchollet5• 11 years ago

the technique is fairly new and used in deep learning, popularized by some of the guys that popularized deep learning, usually mentioned in conversations about deep learning

Logistic regression with regularization is fairly new? 'Pioneered' by the same people as deep convolutional neural networks? Are you certain about this?

SilasX5• 11 years ago

So that's the result? That you can find sorta clever vector equations in the results like "Obama + Russia - USA = Putin"?

fiatmoney• 11 years ago

Yes.

There is some nifty work on trying to structure an actual grammar based on the geometry of the embedding space, but "look at this cool thing we found" is a totally worthwhile thing to publish.

(Embeddings themselves are useful for a whole lot more than that, though.)

SilasX5• 11 years ago

If these are the most interesting results, then it's not. Most of the work seems to be in the (human provided) search for happy combinations like these. Are they typical? Is the whole space like this? Is there a consistent metric for deciding that these are in fact clever and it's not just the human doing the work in making it make sense in a funny way?

Frompo• 11 years ago

More significant I guess is how excited the author manges to be over these coincidences. Maybe a word of caution is needed: Overly generous interpretations is how things like Nostradamus or the hidden code of the bible retain their credibility with parts of the population.

And frankly, if you think that stock market \approx thermometer is insightful, you should probably be kept away from positions of responsibility.

Hydraulix989• 11 years ago

I bet you could run a bruteforce search over the possibility space for equations that can be made from simple vector addition/subtraction operations within some margin of error (since nothing perfectly overlaps) and look for interesting candidates manually post-hoc.

Really, there's a lot of \approx going on here -- you have thresholds for distance inside your vector space. Not only that, this shows that he cherrypicked "great" examples.

MrLeap2• 11 years ago

Try some yourself. http://radimrehurek.com/2014/02/word2vec-tutorial/

Scroll down and you have some text boxes you can try some yourself. Here's some I tried (filtering out repeats and plural forms of input words. Those artifacts seem to happen a lot and would be easy to ignore)

cat:dog::bread:butter eh? I guess.

sword:shield::attack:protect okay that works.

up:down::left:leaving Eh, not great. I guess if you think they're analogous in terms of tense it kind of works. Disappointing, word2vec. (to be fair, "right" was third highest)

drive:car::dick:? Whatever it was, it made me giggle immaturely.

bdamos• 11 years ago

How did you select words to compare? Did you have to try many poor combinations before selecting a "good" set?

MrLeap2• 11 years ago

I've played with word2vec, and it took nothing to get really interesting combinations.

The first thing I tried was computer : server :: phone : ? I didn't really have a great answer for that in my head before I ran it. Word2vec decided the closest match was "voicemail". It breaks down when you feed it total nonsense, but what would you expect it to do? :P

I'm constantly super impressed by properties of the vectors.

microtonal2• 11 years ago

Mikolov, et al. 2013 [1] do a proper evaluation of this. E.g. they found that the skip-ngram model has a 50.0% accuracy for semantic analogy queries and 55.9% accuracy for syntactic queries.

word2vec comes with a data set that you can use to evaluate language models.

[1] http://arxiv.org/pdf/1301.3781.pdf

rspeer2• 11 years ago

I would insist on a better dataset before really calling these "semantic analogies" (and don't just take my word for it: Chris Manning complained about exactly this in his recent NAACL talk).

The only semantics that it tests are "can you flip a gendered word to the other gender", which is so embedded in language that it's nearly syntax; and "can you remember factoids from Wikipedia infoboxes", a problem that you could solve exactly using DBPedia. Every single semantic analogy in the dataset is one of those two types.

The syntactic analogies are quite solid, though.

lelf• 11 years ago

word2vec is a Deep Learning technique first described by Tomas Mikolov only 2 years ago but due to its simplicity of algorithm and yet surprising robustness of the results, it has been widely implemented and adopted.

… And patented http://www.freepatentsonline.com/9037464.html

andrewtbham• 11 years ago

If you use word2vec in an agent, like siri or watson, how would google know?

msoad• 11 years ago

That's not a question you can ask in a meeting at Apple!

agibsonccc5• 11 years ago

As someone who has a card at the table with this one, we asked that question on the list fwiw[1].

I don't think there will be any problems here.

1: https://groups.google.com/forum/#!searchin/word2vec-toolkit/...

fchollet5• 11 years ago

The "deep" in deep learning refers to hierarchical layers of representations (to note: you can do "deep learning" without neural networks).

Word embeddings using skipgram or CBOW are a shallow method (single-layer representation). Remarkably, in order to stay interpretable, word embeddings have to be shallow. If you distributed the predictive task (eg. skip-gram) over several layers, the resulting geometric spaces would be much less interpretable.

So: this is not deep learning, and this not being deep learning is in fact the core feature.

ma2rten• 11 years ago

I think you could see it as a sort of recurrent neural network.

fchollet5• 11 years ago

How? There is no recurrent data flow in word2vec. Word2vec maps words and their context with 2 embeddings, a dot product and a sigmoid. That's it.

rndn• 11 years ago

Couldn't you treat the network as a matrix and perform addition and subtraction on it in the same manner?

Smerity• 11 years ago

I'm uncertain whether you mean the algorithm or the output. The question is interesting to both however.

The most common method for producing word vectors, skip-grams with negative sampling, has been shown to be equivalent to implicitly factorizing a word-context matrix[1]. A related algorithm, GloVe, only uses a word-word co-occurrence matrix to achieve a similar result[2].

You can also view the output as an embedding in a high dimensional space (hence the name word vectors) but more surprisingly you can learn a linear mapping between vector spaces of two languages, which lends it immediately useful in translation. From [3]: "Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish".

[1]: "Neural Word Embedding as Implicit Matrix Factorization" http://papers.nips.cc/paper/5477-neural-word-embedding-as-im...

[2] http://nlp.stanford.edu/projects/glove/

[3]: "Exploiting Similarities among Languages for Machine Translation" - page 2 has an intuitive 2D graphical representation http://arxiv.org/pdf/1309.4168.pdf

agibsonccc5• 11 years ago

I think people see it as "deep" (even though it's not!) due to the representation learning component. Word2vec is often used as PART of a deep neural network for feature engineering.

datacog3• 11 years ago

to note: you can do "deep learning" without neural networks

Curious. Can you share a few examples and applications please?

fchollet5• 11 years ago

Here's a relevant link: http://www.researchgate.net/post/Is_deep_learning_with_decis...

Two examples:

- using layers of random forests (trained successively rather than end-to-end). Random forests are commonly used for feature engineering in a stack of learners.

- unsupervised deep learning with modular-hierarchical matrix factorization, over matrices of mutual information of the variables in the previous layers (something I've personally worked on; I'd be happy to share more details if you're interested).

tshadwell• 11 years ago

When I see things like this, it makes me wonder how much data forms each of these vectors; if a single article were to say things about Obama, or humans and animals, would it produce these results?

thisjepisje• 11 years ago

Anyone tried this with the corpus of HN commentary?

fchollet5• 11 years ago

I have, actually. Here's the code for the experiment, with a link to download the data: https://github.com/fchollet/keras/blob/master/examples/skipg...

I also recommend using Gensim for word embeddings.

datacog3• 11 years ago

OP:

Do you have some more results to share coming from your model?

fauigerzigerk• 11 years ago

I wonder if Obama + 2017 == Obama - President

kmicklas• 11 years ago

That would be equivalent to 2017 == -President, which is unlikely.

jmalicki2• 11 years ago

If you want an equivalent you'd have to think in similar units.

Like Obama + 2017 = 1981 + Carter or Obama + 2017 = Bush + 2009

eatonphil2• 11 years ago

I am not getting the "Obama + Russia - USA = Putin" piece nor the "King + Woman - Man" bit either. Nothing particularly meaningful came up on a search for the latter. Could someone explain?

mwsherman• 11 years ago

If I understand your question, the idea is that one can do “arithmetic” on concepts. Essentially the first equation asks “Obama : USA :: ? : Russia”. Similarly, “King : Man :: ? : Woman”.

The way the corpus “talks about” Obama in relation to the USA is similar to how the corpus talks about Putin in relation to Russia. That the system can reveal this is amazing to me.

eatonphil2• 11 years ago

Oh cool, I see! Thanks for the explanation.

Abstractions my Deep Learning word2vec model made

Related Stories