HN user

ynn4k

15 karma

Semantic Search, NLP, Machine Learning

http://iapps.in

Posts1
Comments18
View on HN

Machine learning consists of parametric optimization to reduce some error function on training data and then use the learnt parametric model on unseen/held-out data and evaluate. The first step is to construct the right parametric model by studying the data domain and then iterate till the performance is achieved within acceptable level. Machine learning research is highly mathematical, but you can start by using some open source ML tools and tweaking the models to get a feel for the capacities of different models.

Some topics you should familiarize are: Probability Theory, EVD/SVD, ANN, ML/MAP estimation, Minimum classification error training, SVM, LMS fitting, PCA/ICA, FSM and HMM.

If you have taken the NLP course, you can work in many interesting areas that have practical applications e.g. improving the performance of a Named Entity recognizer for web data, automatic classification of web pages into predefined categories, natural language parsing for grammar transfer in translation etc.

You should also learn some tools of the trade: regular expressions, machine learning, statistical methods, neural networks, minimum classification error training etc.

New Delhi/Remote possible. iApps.in is looking for

- Business development/marketing manager

- UI/UX designer

- NLP researcher

iApps.in is a semantic search and discovery engine for the App Store that combines the social, semantic and mobile internet technologies to connect users with the apps they want. For more details see http://iapps.in/jobs

It is indeed an extension of the "Website optimization for the search engine" or SEO as they call it. To a search engine developer, SEO means to improve the search engine for better output.

The fight against spam is a constantly evolving one and though some of the techniques from "web" search engine could be applied to the App store, there are unique differences between the two - e.g. the rating, download rankings are not directly analogous to page rank. So Google does seem to have some advantage, but the App store problem will need a fresh research approach.

Everyday Apple approves around 900 NEW iOS and 50 new Mac Apps in the App Store[1]. In such a huge market, it is quite likely that some would try to game it.

The human curation approach falls flat on such a scale. Machine learning and natural language processing can help us in mining the App Store to detect anomalous behavior and improve the search and discovery of apps.

The statistical models of temporal distributions of ratings and rankings are still emerging and such hightlighting provide a useful resource to train the models. So if you see something, say something.

[1] http://twitter.com/iapps_in

A general problem with n-gram is the conundrum of data-sparseness vs reliability of estimation. To have reliable estimation, you need larger order n in n-gram, but it also increases the size of the model which requires larger amount of data and storage. Thanks to the Web as a corpus and cloud computing, we now have upto 5-gram models computable on Terabytes of data provided you are resourceful. One problem with this approach is the selection of the web data to be used for training. The better adaption to the target scenario, the better accuracy.

  i see no services that make use of this.
Most services have proprietary implementations of spell correction that is an amalgamation of several techniques including n-grams, and they might not like to make it public.