Making Sense of Everything with words2map

http://blog.yhat.com/posts/words2map.html

by lmcinnes • 10 years ago

60 29 10 years ago

blog.yhat.com

minimaxir2• 10 years ago

I'm not fond of the "magic AI does everything" narrative, especially since the code is available on GitHub (https://github.com/overlap-ai/words2map) and it's not magic. That being said, the code is optimized for efficient memory usage (important with the pre-built word2vec models), and since it MIT-licensed, I might be able to develop a few pretty visualizations. :)

legel8• 10 years ago

Thanks for pointing the memory optimizations out. The Google model trained on 100 billion words unzips as a 3.4 GB binary, which often is a non-starter on smaller servers, while these vectors indeed come in at less than 100 MB. Mainly this is because only the most common 100k of the 3m vectors distributed are in here, but also because 32 bit floats for each of 300 elements per vector are converted to 16 bit floats, with virtually no loss in precision. (Research suggests we may be able to compress even further, perhaps to 5 bits of entropy per vector element.) In practice, across a wide variety of words and phrases, this pipeline can quickly derive new vectors by web searching, and thanks to the OP's HDBSCAN, can cluster them quite well. Typically it's the quality of keywords found online that ends up most limiting the quality of maps derived here, while there's probably room for breakthroughs in the keyword extraction algorithm put together here (which relies on word2vec indexes as a proxy for the idf in tf-idf). In any case if greater precision is needed - like when I wanted to map ~50 really great scientists who deserve all the credit here - one trick that can help is to increase the number of websites to scan per unknown word (e.g. from 10 to 50) in the research_keywords function.

mdda4• 10 years ago

Embeddings can be compressed even further than that : https://arxiv.org/abs/1511.06397 (disclaimer: I'm the author)

ilyaeck• 10 years ago

Question to Y-hat folks: why cluster in 2D? Granted, clustering in 300D is hard :) Still, the 2D projection must add a significant metric distortion. Why not a middle ground, say, 5-10D ?

minimaxir2• 10 years ago

Unless you are Mister Mxyzptlk, you cannot plot visualizations greater than 2-3D.

sixhobbits2• 10 years ago

parent is asking about this I think: "This is indeed nice for data visualization, while it’s also very helpful in our pipeline because it removes noise in the derived vectors, by forcing a new mapping based purely on relative similarity. For this reason we will be using the low-dimensional coordinates of each word in our recommender system."

i.e. it's nice for visualisation, and it removes noise. It would be interesting to see discussion from y-hat about where the sweet spot between lack of noise and still keeping relevant information is. I think because the subject matter is pretty simply to cluster, 2D works well enough and keeps everything simple.

BenoitP• 10 years ago

Do you mean replacing

word2vec -> t-SNE(2D) -> HDBSCAN

word2vec -> t-SNE(10D) -> HDBSCAN -> t-SNE(2D) (or just 2D projection that separates well the centroids)

----

About clustering in 300D, specifically for word2vec vectors, to which cosine similarity is important: Random-Projections LSH is supposed to "be compatible" with the cosine similarty. I have been toying with it and getting sort of good recalls (70%) with 100D.

It might help a lot with regards to finding the nearest neighbours. Or were you meaning that the clustering itself will be meaningless, regardless of our ability to find neighbours?

lmcinnes2• 10 years ago

Personally I think that would make some sense. I suspect the catch is in presenting the results to the user: you have to present in 2D, and if you cluster in higher dimensions you may get results that, while perfectly valid, look strange when collapsed to 2D (overlapping/mixed clusters etc.)

To my mind this isn't so bad -- the notion of overlapping clusters is something I think most people actually accept (think of clusters as tags rather than a partitioning and you can get the idea). However I can see why you may prefer to have your clusters present more clearly in 2D.

dandermotj• 10 years ago

Multidimensional scaling is designed to preserve the most variance (or some variation of that idea) among points in a lower dimensional space. So for the most part, the cluster memberships assigned by clustering in an n-dimensional space are the same in the lower dimensional projection.

There's often no need to cluster in a higher space because the outcome is the same, and clustering is more expensive when there's more dimensions.

sixhobbits2• 10 years ago

human + robot ≈ cyborg

electricity + silicon ≈ solar cells

virtual reality + reality ≈ augmented reality

These always seem impressive in word vector models, but in reality, I imagine that "robot" and "cyborg" were already pretty close. The fact that adding "human" nudged the vector closer is likely not as meaningful as it would be nice to believe. The same for "electricity/solar cells" and "virtual reality/augmented reality"

Still a really nice application for word2vec, and I'm looking forward to seeing other similarly practical implementations in future.

crypto5• 10 years ago

They demonstrated few somehow meaningful rules, generated by model, but it is interesting how much garbage it contains?

vinchuco• 10 years ago

Nitpicking: NOT (human + robot) ≈ cyborg BUT average(human + robot) ≈ cyborg

Some things that come to mind:

I'd be interested to see other vector operations such as projection of one word into another in the examples. Also, only nouns yet.

How is ≈ defined, if the distance to the closest word vector is not necessarily unique?

Finally, what is the proportion of words that maintain human meaning when averaged to those that are nonsense? What are the most "meaningful" words, in that sense?

visarga• 10 years ago

I think you can also get pretty good suggestions with plain old bag-of-words, tf-idf and k-means.

ganeshkrishnan4• 10 years ago

bag of words don't give us suggestions. while tf-idf is for searching corpus not clustering.

K-means is clustering and similar to this,correct.

vonnik• 10 years ago

how is this different than TSNE?

https://lvdmaaten.github.io/tsne/

anyone looking for an explanation of word2vec may find this helpful:

http://deeplearning4j.org/word2vec

lmcinnes2• 10 years ago

It uses t-SNE but there other other working parts here, including word2vec (and some nice compression of a pre-trained model), keyword searching to provide context for terms, and clustering to find natural dense groups. It's a nice pipeline that fits together a bunch of independently interesting parts into a single system that can produce quite remarkable results.

ganeshkrishnan4• 10 years ago

Hi, I was in the middle of creating "user personalities" using K-means clustering.

Is it ok to reference your document for our papers? MIT licence is awesome and let us reuse your tech. Our site is at www.shoten.xyz if you are interested to know what we are doing

legel8• 10 years ago

Thanks, shoten looks wonderful and exciting. You're welcome to reference/reuse words2map as you wish, while the authors of word2vec, t-SNE, HDBSCAN, and others deserve all the credit. :)

ganeshkrishnan4• 10 years ago

Thanks for the kind words. We have bit of known unknowns and I am sure plenty of unknown unknowns. What we are trying to implement is a rank based on "personality cluster" of the person querying. So in your example Pablo Picasso searching would get slightly different results compared to Kanye West (because they are different personality clusters)

mdda4• 10 years ago

"We are now at a point in history when algorithms can learn, like people, about pretty much anything. " seems pretty disingenuously worded.

One infers from a quick read ~"Algorithms are now like people, and can learn about anything." But careful parsing of the commas shows that the sentence is true, but in the precise sense that "People can learn about anything. Now, algorithms can also learn about anything." - and the extent of learning/understanding is not being compared.

Perhaps I'm nit-picking, but this statement appears to have been constructed to support an AI pitch, and is literally true, but no 'actual AI' is involved (and no-one is actually claiming it is... unless you /want to believe/).

legel8• 10 years ago

The message is genuine and simple: computers can now learn like humans.

mdda4• 10 years ago

It is a genuine message. It is also simple. But it isn't actually true. Computers are not able to learn like humans : the learning process is entirely different. And what they learn is different - even though it can be visualized in a way that makes it look 'humanlike'.

These techniques are impressive, and yhat is demonstrating that they are very capable. It's just that I feel a little sad that the 'AI pitch' is being turned on, when the 'really good tech' is a much more valid way to understand what they're doing.

Making Sense of Everything with words2map

Related Stories