Bayesian Optimization for Collaborative Filtering with MLlib

http://blog.sigopt.com/post/148703071378/sigopt-for-ml-bayesian-optimization-for

by Zephyr314 • 10 years ago

42 30 10 years ago

blog.sigopt.com

Zephyr3147• 10 years ago

I'm one of the co-founders of SigOpt (YC W15) and am happy to answer any questions about this post (or anything about SigOpt).

More info on the methods behind SigOpt can be found at https://sigopt.com/research.

idewanck2• 10 years ago

Post author here, happy to answer any questions as well.

minimaxir3• 10 years ago

Wait, Spark has built-in Model Hyperparameter selection (http://spark.apache.org/docs/latest/ml-tuning.html), that was not mentioned in the article. What advantages does your service do?

Relatedly, why are you advocating using MLLib/RDDs when they have been deprecated in favor of ML/DataFrames (http://spark.apache.org/docs/latest/ml-guide.html)?

Zephyr3147• 10 years ago

Great questions!

The Spark hyperparameter selection uses an exhaustive grid search approach, which can take a long time (complexity grows exponentially with number of parameters to tune) and produce poor results when compared to other methods [1]. Bayesian optimization is a great way to tune time consuming and expensive functions like ML pipelines, where finding a good configuration in a small number of total attempts is the only tractable way to tune the system.

ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet.

Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib ALS implementation.

[1] http://arxiv.org/abs/1603.09441

minimaxir3• 10 years ago

ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet.

ALS has been available in ML/DataFrames since 1.3.0, per the documentation. (http://spark.apache.org/docs/latest/api/scala/index.html#org...)

Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib cross validation methods.

I edited the comment to correct my error that ALS was not mentioned as being native to Spark. However, for Hyperparameter cross validation, I looked at the code in the repository (https://github.com/sigopt/sigopt-examples/blob/master/spark/...), and while the Spark CrossValidation function is imported at the beginning of the Scala file (from ML, not MLlib), it is not used in the code, in favor of SigOpt.

I'm confused.

lern_too_spel• 10 years ago

Under the hood, all the existing (as of 2.0) ML pipelines models immediately convert the Dataframe to an RDD and cache it, so there is a one iteration slowdown for using the Dataframe models versus an equivalently implemented RDD model. None of them try to get a speedup by directly operating on the Tungsten datastructure, but it is an obvious next step.

nl2• 10 years ago

Relatedly, why are you advocating using MLLib/RDDs when they have been deprecated in favor of ML/DataFrames (http://spark.apache.org/docs/latest/ml-guide.html)?

Note that the DataFrame version of MLLib (which is still called MLLib btw, even if the package is different) isn't feature complete compared to the RDD version.

For example, the DataFrame version of LogisticRegression is binary only, while the RDD one allows multiple classes. I guess the suggested work around is to use the OneVsRest classifier (in Spark 2.0), but that blows the memory out on fairly small tasks.

I like Spark a lot, but MLLib isn't as strong as it appears at first glance.

a1k0n• 10 years ago

So SigOpt was tuning rank (number of latent factors), number of iterations to run the algorithm (in my experience alternating least squares generally converges within 10-20 iterations, but there'd be no downside to running it longer unless it's overfitting), and the regularization strength.

What optimal parameters did it find for these?

Zephyr3147• 10 years ago

Great question. As you point out, all of these parameters (rank, num iterations and the actual reg. term) can in some way contribute to the regularization of the reconstruction of the ratings matrix. We thought it would be interesting to include all of them and here are the optimal parameters SigOpt found for this experiment:

rank = 36

numIter = 30

log_lambda = -2.90405347693

tachim2• 10 years ago

How does SigOpt compare to GPs?

Zephyr3147• 10 years ago

Great question. SigOpt is a hosted, scalable ensemble of different Bayesian optimization methods. Many of these methods use Gaussian Processes (GPs) as part of the modeling aspect of their Sequential Model Based Optimization (SMBO). We've written up a primer on Bayesian optimization [1] that goes over some of the different methods, with lots of citations for diving deeper.

We've found that SigOpt compares very favorably [2] to other Bayesian optimization approaches. In addition to this, our hosted platform allows people to harness the full power of GP backed Bayesian optimization with just a few lines of code [3] instead of the sometimes heavy administration required by other methods.

[1] https://sigopt.ninja/1470077644/pdf/SigOpt_Bayesian_Optimiza...

[2] http://arxiv.org/abs/1603.09441

[3] https://sigopt.com/docs

tachim2• 10 years ago

Thanks. Do you guys do any meta-learning based on information from optimization runs on the many instances that SigOpt runs on? E.g. if Alice uses Sigopt for a neural net architecture and Bob does too, is the information from Alice's run used to improve Bob's?

visarga• 10 years ago

https://sigopt.com/pricing

- Individual: $1,000/month

- Enterprise: Custom pricing

I am not a multi-million $ company, so I guess it's useless for me.

Zephyr3147• 10 years ago

We have a free academic tier as well: https://sigopt.com/edu

apathy5• 10 years ago

I could give a shit about the hyperparameter tuning (CV... it Works For Me) but your writeup of Gaussian processes and why they are called kriging in spatial stats is awesome.

http://blog.sigopt.com/post/130275376068/sigopt-fundamentals...

mccourt2• 10 years ago

Dear apathy (love the name), I wrote that post and am really glad that you liked it. I had another more recent one that focused on a different topic but had a solid paragraph right at the beginning that also stepped through some of the history. It might be useful because it had a bunch of links trying to join content between GP stuff, kriging and RKHS. It is linked below. Have a great day.

http://blog.sigopt.com/post/147952139093/sigopt-fundamentals...

apathy5• 10 years ago

Great post. Why is the kernel trick so important for Gaussian processes? My understanding is that operating in an RKHS is primarily needed to enable that cute little maneuver.

Wait, never mind. If GPR is like an infinite-dimensional linear regression then doing it within an RKHS means you don't actually have to bother with the functional generalization and can get solutions to a potentially ill-behaved loss function along a grid/cube/hypercube/whatever. Is this part of why SigOpt works more efficiently than classical parameter space sampling designs?

Not-so-ninja edit: Time for me to re-read Rasmussen, I think.

apathy5• 10 years ago

Oh, also, for students: https://sigopt.com/edu

I'm worried this is going to be like good Scotch for me.

blahi• 10 years ago

There is package, mlrMBO, created by the great guys who created mlr (absolutely awesome for building pipelines, you will ditch caret in a second!). Not on Spark obviously, but thought some might find it useful.

https://github.com/mlr-org/mlrMBO

Bayesian Optimization for Collaborative Filtering with MLlib

Related Stories