Unsupervised Learning with Even Less Supervision Using Bayesian Optimization

http://blog.sigopt.com/post/140871698423/sigopt-for-ml-unsupervised-learning-with-even

by Zephyr314 • 10 years ago

85 21 10 years ago

blog.sigopt.com

pizza3• 10 years ago

Now just throw some compressive sensing at the problem ;)

Zephyr3148• 10 years ago

One of the co-founders of SigOpt (YC W15) here. I'm happy to answer and questions about this post or the methods used. More info on the Bayesian methods behind this can be found at sigopt.com/research as well!

pizza3• 10 years ago

Actually, out of curiosity, would there be some way to use the inverse coloring transform + a lil noise to generate some kind of equivalence class of free training examples, sort of a la skip-gram?

idewanck• 10 years ago

Author here, you can definitely talk about augmenting your training data with some slight transforms of the labelled set you have. Common strategies are adding noise, rotation etc for images

bearzoo4• 10 years ago

Well - just my two cents. The title feels inaccurate. You all are tuning hyper parameters with respect to the performance of the classification task. The bayesian optimization is really to optimize the unsupervised -> supervised pipeline. I was expecting some bayesian optimization of strictly unsupervised representation learning (ex. we have an autoencoder and use some bayesian optimization to tune hyper parameters in order to minimize a reconstruction error). This is really just supervised learning with even less supervision (which is quite typical).

Zephyr3148• 10 years ago

Thanks for the note!

We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant).

The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.

tavert3• 10 years ago

No constrained optimization?

Zephyr3148• 10 years ago

SigOpt allows specifying constraint ranges for every parameter. If the parameter space isn't a tensor product you can report constraint violating suggestions as failures via our API [1] and SigOpt will take that into account as it optimizes.

SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking.

[1] https://sigopt.com/docs/endpoints/observations/create

lqdc13• 10 years ago

Is this the first OHAAS (Optimize Hyperparameters As A Service)?

IanCal• 10 years ago

There is/was whetlab which got bought out by Twitter if I remember rightly. It's a shame as I was using them and wanted to do it more.

bearzoo4• 10 years ago

https://github.com/JasperSnoek/spearmint

Zephyr3148• 10 years ago

We provide a very similar interface to Whetlab and I would be happy to get you set up on SigOpt. We offer a free trial to get started [1] and a free academic plan [2].

[1] https://sigopt.com/get_started

[2] https://sigopt.com/edu

Zephyr3148• 10 years ago

We were the first company to launch (Whetlab was bought and shut down before they got out of private beta).

We're currently the only active company offering it as a service.

While hyperparameter optimization is one of the most common use cases of SigOpt right now, the general Bayesian Optimization As A Service we provide has also been used to tune simulations and even manufacturing and process engineering [1].

[1] https://sigopt.com/cases/process_engineering

Unsupervised Learning with Even Less Supervision Using Bayesian Optimization

Related Stories