Machine learning for the impatient: algorithms tuning algorithms

http://www.aelag.com/147952673

by aelaguiz • 14 years ago

155 31 14 years ago

bencpeters• 14 years ago

I had a few questions about the actual implementation of this stuff. I took the coursera ML course, so much of the terminology and techniques are familiar after that, but Professor Ng structured the exercises in the course around Matlab/Octave and suggested using one of these tools for a first-pass solution when implementing machine learning problems.

Have you used Matlab much in your work? How does the performance (and libraries available) compare with Python?

Also, does anyone know a good resource for finding good, high performance ML libraries for other languages (Ruby, C++, etc.)?

aelaguiz4• 14 years ago

I used to prototype with Octave but I actually find I"m much faster in python. I push my tests to EC2 and just use cProfile to make sure I'm not doing anything silly before hand. I parallelize with celery, but I've also used pp with some limited success. Performance is sufficient for my needs currently but not for production or real-time.

ogrisel• 14 years ago

matlab and python + numpy + scipy + matplotlib give the same kind of performance for the same level of features and ability to do quick exploratory prototyping with high level, vector based datastructures.

However one is free (beer and freedom) and multipurpose (you can preprocess string data and expose your model with a HTTP API) while the other is much less so.

In python + numpy you should always profile and if the bottleneck is in python interpreted code (rather than a low level numpy call) you can always rewrite the offending python loop in cython.

punee• 14 years ago

Scikit-learn for Python, great tutorials/user guide covering various ML techniques, makes prototyping very easy: http://scikit-learn.org/stable/

kilburn• 14 years ago

One of the most famous collections/frameworks for development and usage of (traditional) machine learning algorithms is weka [1]. It is written in java, and actively developed by the machine learning group of the University of Waikato. They also seem to have a forum to discuss, which should be a good resource given that a lot of their users are actually researchers.

The very same group is also developing another framework (also in java), MOA [2], that is oriented towards "big data": the algorithms used in there are specifically tailored for huge datasets and/or realtime (streaming) machine learning.

[1] http://www.cs.waikato.ac.nz/ml/weka/

[2] http://moa.cs.waikato.ac.nz/

scottfr• 14 years ago

Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set.

The more times you go back to your test data set to evaluate the effectiveness of a model, the more optimistic your error predictions will be and the greater your chance of overfitting. Several iterations of his loop will probably improve the model, but if you keep repeating it eventually the true model performance will start to degrade.

aelaguiz4• 14 years ago

Yep, I've definitely seen that. I bet if I were to reserve a portion of the dataset for validation after all tuning, I'd get a much better measure of it's actual ability at prediction. Using it in the way that I did, I wouldn't be surprised if there was significant overfitting. In fact I'd be surprised if there wasn't.

gersh• 14 years ago

You can split it into training, validation, and test sets if you have enough data. Then, the validation set would be used for the training. Presumably by using a genetic algorithm, he isn't going for a perfect fit, which may help avoid overfitting.

brador3• 14 years ago

Question: Why is the data for machine learning split into a training dataset and a test dataset? Wouldn't using the entire dataset to build the model result in greater accuracy of the predictions?

aelaguiz4• 14 years ago

For this I actually had sufficient data available that adding more didn't do much.

You need a testing dataset in order to validate the performance of the model. If you validate against the training set really what you're doing is measuring the model's ability to fit the training data - which it will be able to do with high accuracy. That will, however, result in a much diminished ability to predict any new games - as it isn't "learning" the features of college basketball as much as it is memorizing the contents of the training set.

As others pointed out, it'd probably even be better to add a third grouping which is tested against only after the algorithm has finished - as an objective validation against as of yet unseen data.

numlocked• 14 years ago

To prevent overfitting. A common technique is to use cross-validation, where you train some percentage of the data (90%) and test on the remaining (10%), and cycle through which 10% you use as test (each train/test split is called a "fold"). Once you've identified a model that resists overfitting, you can train on all of the data.

edit: As noted in the other reply, having a truly blind validation set is still ideal.

reginaldo• 14 years ago

When you develop a model, for instance, when implementing a classifier, you supposedly want to apply the developed model to other data, i.e., data you don't have available during development.

In many situations, it doesn't make sense to test your model only when it's put to make or influence decisions in the real world (although you have to test in the real world too). You'll want to test the predictions of your model on data you already have the actual results for. To test your model you'll split your data into data you know and will let the model know about (training dataset), and data you know but the model can't know about (test dataset). That way you can use the data the model doesn't know about to make controlled experiments and compare models (and, if your data is really representative of the real world, your mofrl comparison and the performance of your chosen model will hold).

The moral of the story is: if you don't split your data, you won't have any idea of how it performs in the real world, you'll only know how it performs with data it already knew about.

btilly• 14 years ago

Suppose that I have 2 possible models that I might try to build. I can train them both on my data set, but how do I get a sense as to which is better?

When you have the split, you can then tell which idea seems to work better.

tel• 14 years ago

Yes. Training on the test set overestimates performance, cross validation underestimates it.

danger4• 14 years ago

As another commenter pointed out, the accuracy really needs to be evaluated using a validation set, not the test set--the approach described in the post is training with the testing data. In the field, we call this "cheating".

The basic idea of automatically tuning hyperparameters (the things this post discusses tuning with genetic algorithms) is cool, though, and is becoming a popular subject in machine learning research. A couple recent research papers on the topic are pretty readable:

Algorithms for Hyper-Parameter Optimization:

http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf

Practical Bayesian Optimization of Machine Learning Algorithms:

http://arxiv.org/abs/1206.2944

danger4• 14 years ago

Also, for people who are interested in the application of predicting college basketball with machine learning, there's a Google group that is worth joining:

https://groups.google.com/group/machine-march-madness

aelaguiz4• 14 years ago

Thanks for the information! I've updated the article to reflect this.

Here's a question: where does "the field" hang out? Is there a cohesive community of any sort?

danger4• 14 years ago

I'd say the closest thing to a cohesive community would be the MetaOptimize Q&A forum, but maybe others have other suggestions:

http://metaoptimize.com/qa

gammarator• 14 years ago

The Kaggle forums can be a good resource, and competing there is a good way to polish your skills.

Also, you might check out Random Forest algorithms--they're high performance but still very beginner friendly, as there aren't many parameters to tune. There's a nice implementation in the excellent scikits-learn python library.

pjin• 14 years ago

Hmm, so gaussian distributions are easy to use and ubiquitous and all (they're the basis functions used in SVM), except that I don't see any reason for them to be priors here. But since 2012 > 2008 I feel like I'm obligated (and I'm semi-trolling) to point out the obvious about lazy assumptions based on "flexibility and tractability", which is that they can implode hilariously in your face. C.f. the financial crisis.

[1] http://econometricsense.blogspot.com/2011/03/copula-function...

bayesregressor• 14 years ago

I think you're confusing the Gaussian "process" used in Bayesian optimization with a standard Gaussian distribution. They are very different things - as are Gaussian copulas and what is referred to as the 'Gaussian kernel' (which is not actually a distribution at all) in the SVM. The Gaussian process is a distribution over functions, the properties of which are governed by the covariance function - so the prior over the function, or the assumption about its complexity and form, is determined by the choice of covariance function. Of course it is very important to choose a prior that corresponds to the functional form you are interested in, which is actually discussed and empirically validated in the literature referred to in that post. It's a bit ironic that you are claiming to point out the dangers of making lazy assumptions by doing exactly that.

misiti3780• 14 years ago

This article is very interesting but I am really confused about the edit:

"A few people have pointed out that using the testing set for tuning demands that final measure of effectiveness be doing using a validation test set which is not part of either the training or testing datasets. This is due for the very real potential of over fitting. Also – apparently this technique is called “Hyper-Parameter Optimization.” A helpful commenter over at Hacker News supplied the following resources"

Does that mean there is definitely overfitting going on but it is acceptable for the purposes of the article. That 82.5% accuracy rate has overfitting written all over it.

Good stuff regardless!

Machine learning for the impatient: algorithms tuning algorithms

Related Stories