Word2vec: Fish + Music = Bass

https://graceavery.com/word2vec-fish-music-bass/

by tobr • 7 years ago

94 18 7 years ago

xg15• 7 years ago

So. Could anyone explain why it is doing that?

My (very rough) intuition to what w2v does is that it associates each word with a point in a space, then moves the points around so that their distances roughly correspond to how often they are used together with similar other words. (I.e., two words that both are often preceded by "It happened in" would be put close to each other, while words occurring into very different sentences would be placed further apart.)

I can see how this could give some semantic meaning to clusters of words - however, I'm still baffled as to why it gives meaning to directions within the vector space.

I also wonder how consistent it is. Seems to me it would be interesting to investigate the directions further. E.g., are all of them meaningful? What happens if I slightly rotate or scale a direction? How much transformation can I apply before it changes its meaning?

Are there more complex shapes in the space that also have meaning?

GistNoesis2• 7 years ago

The "trick" is that word vectors are just projections of sparse tensor.

For example the word "queen" can be represented by its attributes. Representing in triplet form from wikidata Q16511993 (queen, subclass ,monarch),(queen, subclass, female),... A king would be (king, subclass ,monarch),(king, subclass, male),

Transitioning from queen to king is just changing an attribute in this triplet space, i.e removing a triplet and adding another. All those operations are linear. And then you take the projections.

This is kind of similar to the kernel trick of SVMs. You do the calculations in a lower dimensional space (word vectors) but implicitly you are doing them in the higher dimensional space (graphs of relations, aka triplets lists).

joycian• 7 years ago

The sparse tensor perspective intrigues me, could you perhaps provide me with some references?

zwaps2• 7 years ago

One contention I have with this description (if I understand it at all), is that the projection into the linear space does not preserve the properties of changes in a relation graph.

I mean I "think" I said pretty much the same in my reply, but one big issue with word2vec is that the relations, which are graphical, can be projected, but not all changes in the graph can be. Therefore, I think it is not correct that these operations are linear in the sense that they can be equivalent to linear operations in the embedding space.

But maybe I just don't understand your description.

zwaps2• 7 years ago

As you know the basic idea, which precedes word2vec, is to classify each word in certain meaningful vector dimensions, for example "furryness" for pets.

If you code this manually, it makes sense to manipulate directions. For example, removing furryness from the archetypical pet vector, leaves you with non-furry pets in proximity. It essentially works the same way for word2vec.

Word2Vec encodes words without manual coding of dimensions.

The distributional hypothesis is that if words occur in the same context of other words, they share similar positions on some dimensions, and those dimensions are defined by the contextual words.

Now take two words that appear in more than one "cluster" of words together. Since these clusters differ, their similarity is high in these two groups of dimensions, so we may remove one cluster of dimensions - say related to gender - and we are left with the similarity on the other dimension.

The analogy operation used in the examples works by finding the relevant dimensions from the second word. We pick a word, and then another word for which the semantics are clear to us - like "male". Removing "male" from our starting word, we expect to move the semantic dimensions of "maleness", setting them to zero. Any word now left in proximity, retains the other contextual bundles distinct from gender.

Here you can see that this is not cut and dry. It is not clear that male is situated in a distinct semantic category. It is probably also related to dimensions that we would not want to set to zero. Nevertheless, the approach works well.

For your last question: It should be clear that a linear space can not encode true relations between words - it being linear. Moving toward another cluster usually means moving away from another, which is not true to meaning as a concept. It has been shown, that more sophisticated embeddings encode a non-linear space where such transformations are possible.

namibj• 7 years ago

It has been shown, that more sophisticated embeddings encode a non-linear space where such transformations are possible.

Main paper doi/arxiv of this please? It seems potentially powerful to combine with reference graphs for automatic inference of proper causality from text-based communication streams. Like twitters firehose, GDELT, HN, etc.

I'm trying to track such analysis techniques on the side for applications to auto-crawling/archiving of things likely being short-lived data, yet part of the causality chain leading up to big, global events.

It'd provide an archive for future historians to analyze how humanity fucked up.

I haven't seen an actually feasible way yet, though.

mruts• 7 years ago

word2vec simply is a neural network with a hidden layer of n perceptrons. The number they use in the paper is 300 I believe. Thus, this is the dimensionality of each word.

To train the model, we generate feature-label pairs by taking a word and randomly choosing a word next to it. We feed this into our model. After training, we are left with a 300 long vector for each word (input words x hidden layer since it’s fully connected).

enjoyyourlife• 7 years ago

I think that it is joining the bass as the musical definition of the word and sea bass the fish

boraturan• 7 years ago

I am really interested in how these analogies would appear in different word embedding methods (Bert..), or different implementations of word2vec.

As to different implementations of word2vec, I am still not satisfied why the original code uses two separate embeddings for target and context words. It could use shared embedding layer.

Also for the layer above the sigmoid layer (Dot product), why is cosine similarity not considered for calculating the vector similarity.

I have searched these issues and not found detailed comments. I have created word2vec embeddings with shared embedding layer and cosine similarity. The vectors seems similar to the original code but deciding on which one is better, needs more work.

mkl2• 7 years ago

The typo in "pig : oink :: Fred Flinstone : wassup" bothered me, but even "Flintstone" doesn't produce the "Yabba dabba doo" that Woody Woodpecker (!) produces: pig : oink :: Fred Flintstone : mumble mumble.

If you're wanting to try it, you need about 6GB of RAM free.

userbinator• 7 years ago

The bass/bass confusion seems to have permeated Chinese machine translation, as a search for 低音魚 (literally "low frequency fish") will show.

Another amusing one there that shows a similar ambiguity is: pig : oink :: mouse : keypress

personjerry• 7 years ago

I'm trying to replicate this on my Macbook and it's causing a lot of lag, and taking a long time. Anyone else have this issue, or might be able to help me debug this? I'm assuming I just don't have enough RAM (I only have 8GB here)?

mkl2• 7 years ago

It took several minutes to load off my old SSD, and took 6GB just to load the data. Trying out a bunch of word combinations pushed it up to ~9GB I think. So yeah, it seems like you just don't have enough RAM.

markchen90• 7 years ago

These analogies need to be taken with a grain of salt. For instance, king - man + woman actually returns king (since woman - man ~= 0). It's only queen when you ban the words king, man, or woman (since the embeddings of queen and king are close).

See Ian's tweet https://twitter.com/goodfellow_ian/status/113352818965167718... or the linked paper https://arxiv.org/pdf/1905.09866.pdf for more information.

Word2vec: Fish + Music = Bass

Related Stories