Show HN: 3d Word Vectors with Neural Networks, T-SNE, and WebGL

http://wordcloud.ersatz1.com/

by dave_sullivan • 13 years ago

30 19 13 years ago

wordcloud.ersatz1.com

Green words on black screen look very intriquing and matrix-like, but I cannot wrap my head around what this clustering actually represents.

dave_sullivan6• 13 years ago

Yeah, some explanation might be in order...

These are a bunch of "word vectors" (generated w/ word2vec https://code.google.com/p/word2vec/) put through a dimensionality reduction/visualization technique called T-SNE and then plotted in 3d. As far as what the clustering represents, check out T-SNE (http://homepage.tudelft.nl/19j49/t-SNE.html), but the short answer is--it's hard to say...

Here's a longer explanation from a webinar I did yesterday where I demoed this: http://www.youtube.com/watch?v=wmlj5uTUTFY (skip to 12:11 to get to the applicable section)

paulgb• 13 years ago

How are the word vectors generated? word2vec?

dhammack4• 13 years ago

Hey Dave - Your visualization is cool, no doubt, but I'm not seeing as much structure in the 3D cloud as I see in 2D visualizations. In 2D, the words form clusters of similarity which are pretty obvious, but I don't really see that here. What training data did you use for word2vec, and for how many iterations did you run t-sne?

I ran some visualizations of my own a while back, and they're a lot less aesthetically pleasing, but you can see more structure and clustering.

https://raw.github.com/dhammack/Word2VecExample/master/visua... https://raw.github.com/dhammack/Word2VecExample/master/visua...

(It could also just be a WebGL thing, because the third dimension seems to have a lot less variance than the other two)

dave_sullivan6• 13 years ago

It's true, I've noticed this too.

I trained word2vec on wikipedia and trained a bunch of different models w/ TSNE for between 800 and 1500 iterations. In 2d, the results actually appear fine (similar to your graphics + others that have been demonstrated). Unfortunately, at least for me, this doesn't seem to be translating to 3d.

It's either a mix of the quality of the vectors (maybe I need to try some other word vectors) and/or T-SNE needs to be optimized a bit better (and/or there's a bug). It's not quite where I want it to be yet, so consider this a "cool tech demo we're still working on" kind of thing.

dhammack4• 13 years ago

Did you roll your own version of t-sne to use in Erstaz? Now that I think about it, I've never seen a 3D visualization with t-sne before (and searching just now didn't find any), it could be that t-sne doesn't work as well in 3D (or has bugs which only appear in >2 dims).

a1k0n4• 13 years ago

Have you tried messing with the target perplexity? That seems to make a big difference.

sp3322• 13 years ago

How did you extract text data from Wikipedia? I've been trying to clean up an XML dump but I'm having trouble getting rid of the wikitext markup.

alexbw• 13 years ago

Poked around the word cloud, but the output of the clustering doesn't seem to many any sense to me.

"boat" is by "scarecrow", "adopted", and "feelings"

"window" is by "insulted", "prize" and "arson"

Rotating around the word-in-question didn't pop up any more similar words.

t-SNE is known for producing immediately interpretable clusterings, but this seems a bit obscure.

a1k0n4• 13 years ago

Ha, I made something very similar with the goal of mesmerizing passersby at a conference where we had a booth: http://www.a1k0n.net/spotify/artist-viz/

Used t-SNE on a vector clustering which came from LDA.

dhammack4• 13 years ago

Nice! I assume LDA was run on the lyrics?

a1k0n4• 13 years ago

Not at all... "documents" were the user's play history and "words" were artists. So if you played one artist twice and another artist once it'd be like a document that says "artist1 artist1 artist2". The assumption is that document topics are analogous to music genres, and each artist creates music within a small set of genres each user prefers music within a small set of genres.

Show HN: 3d Word Vectors with Neural Networks, T-SNE, and WebGL

Related Stories