msalahi

SCIENCE!

"doing all of the web" will cause pretty much any approach to AI/machine learning/NLP to break down. i'm a big believer in it being the responsibility of the engineer employing these techniques to take stock of the problem at hand and find out what constraints you can take advantage of to achieve better performance/accuracy/prettiness of code. there's not really a silver bullet that you can just release on the internet with the task of bringing back incredibly useful information without "knowing what you're trying to analyze before you analyze it."

i've actually found the performance of gensim (the topic modeling python module i use here) to be pretty great. we're not at a scale where CPU performance is make or break just yet, so i haven't done any comprehensive testing of performance. but i've definitely not run into any performance issues worth complaining about. however, gensim is 100% based on lazy evaluation where it can be, so it's relatively light on the CPU. i love NLTK as well, but it did lack in the dimensionality reduction/topic modeling department which gensim did so beautifully. LDA + SVM seemed like an interesting approach to go with, and it didn't disappoint.

In fact, if your training corpus is sufficiently large, you'd be shocked how many words you can eliminate right away for a term frequency of one or two. I went from millions of words in the vocabulary to something like 60k just by ignoring words that happen once or twice in the corpus. Plus, you probably won't learn much about the relationships between words if they only occur a few times in the corpus.

As with most successful applications of machine learning, it's about finessing your approach based on the problem at hand. In our case, we have classes divided on the level of "Medicine," "Real Estate," etc. So, we could throw away lots of words that only occurred once or twice in the massive corpus we crawled to build the language model and still have a pretty robust representation of the subject you're trying to represent.