The multi-armed bandit problem (2012)

http://stevehanov.ca/blog/?id=132

by navigaid • 7 years ago

303 78 7 years ago

stevehanov.ca

founderling• 7 years ago

Multi armed bandit algorithms have been used forever to optimize ads, websites, newsletters etc.

Here is an article from 2013 that describes Googles Multi Armed Bandits algo integration into Adsense:

https://www.cmswire.com/cms/customer-experience/google-integ...

The first time I stumbled across the term "Multi Armed Bandit" was when I read Koza's "On the programming of computers by natural selection" in 1992. When I later got involved in e-commerce projects, it was immediately clear to me that this was the way to tackle the involved optimization tasks.

OJFord• 7 years ago

OP itself says 'posted six years ago', so should probably have a (2013) - or thereabouts - in the submission title.

kurthr• 7 years ago

Ahhh, the epsilon greedy strategy... I forgot I had read this until I was a few paragraphs in. If you have rare positive results, it gives you better statistics on your highest signal tests, while still evaluating the others (i.e. more than one alternative).

mhoad• 7 years ago

Previous discussions worth checking out here https://news.ycombinator.com/item?id=11437114

and here https://news.ycombinator.com/item?id=4040022

ermik• 7 years ago

Besides the tree of these and those comments, do you have any other links to relevant subjects/writings?

a13n• 7 years ago

Also one reason a lot of teams can't do more than two options (A, B, C, D, E, F, G, etc. testing) is because you need a TON of traffic for it to be statistically significant.

comicjk2• 7 years ago

Statistical significance isn't necessary for deriving value from information! The point of multi-armed bandit is that you use the best information you currently have, while also not taking that information too seriously. In a context where experiments have a cost and you need results, this makes more sense than gathering more and more data until you meet statistical significance thresholds.

opportune2• 7 years ago

If random noise is more likely to explain the validity of one hypothesis over another, then your information has very little value.

NewsAware2• 7 years ago

At least for us, most A/B/.. tests require a stable assignment of test to user as key metrics like retention would obviously skewed by randomly assigning on each visit/page view.

jfries• 7 years ago

The user id can be used to seed the random number algorithm to achieve that.

NewsAware2• 7 years ago

Hmmm but if you change the distribution, this won't be deterministic anymore unless the randomness only applies to new users entering the test setup.

daenz• 7 years ago

It seems close to simulated annealing in the travelling salesman problem. The basics of it is that you start with a random tour and adjust path segments incrementally. Most of the time, you choose a new path segment that decreases the global route cost, but randomly, you choose a new path segment that increases the global route cost. This random factor is decreased over time, so the route anneals and settles on a close-to-optimal result.

It shares the same principle of choosing reasonable options most of the time, but allowing variation to keep from getting stuck in a local optimum.

opportune2• 7 years ago

Basically you are comparing one strategy to prevent local optima in one optimization algorithm to another strategy in a different optimization algorithm. There are such strategies for basically every optimization problem.

Other examples: momentum in deep learning, tabu search, random search, etc.

The benefit of doing a pure binary A/B test is that the experiment is so simple that as long as you don't break the cardinal rules (you need a fixed experiment size! Most people don't even do this. Also you need to not bias your control/experiment sets) it is easy to get statistically valid results. When doing multivariate optimization using something that is measured (such as user engagement) rather than evaluated, you need a good amount of data for each configuration to evaluate it, or you run the risk of optimizing for random noise. This is true of even the multi-armed bandit problem: if you vary the exploitation strategies over time, then confounding temporal variables (for example, purchases are higher on weekdays because that's when people make business decisions at work) can invalidate the experiment if not controlled for.

marcosdumay• 7 years ago

The GP is comparing the algorithms. MAB is exactly simulated annealing with an exponential temperature.

err4nt2• 7 years ago

The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things.

Also, I work in this field and I will just say that people _do_ behave differently based on traffic source: i.e. users coming from Facebook behave alike, but different than traffic from Reddit who act similarly to each other. If you were running a self-optimizing thing like this it _would_ make sense to split it up by the different traffic sources and handle them separately.

elehack• 7 years ago

Yes. Bandits will often converge more quickly to the optimal strategy, but it is much more difficult to understand why that strategy is optimal and generalize from the bandit outcomes to predict future performance and performance of other strategies.

It isn't impossible - bandits are seeing adoption in medical trials to avoid precisely the problem discussed - but the standard experiment design and analysis techniques you learn in a decent college statistics class or introductory statistics text no longer apply. That's one of the beauties of A/B testing: while it does require substantial thought to do well, the basic statistics of the setup are very well-understood at this point.

orasis9• 7 years ago

I disagree. I’ve spent a lot of time staring at bandit outcomes and usually they match some sort of intuition of why a variant might be exceptional.

edmundsauto3• 7 years ago

But for results to generalize or to understand why, the confounders must be accounted for in the randomization. This is really hard to do well -- there are often subtle influences that aren't sufficiently understood how they impact these non-linear systems. What makes someone convert? A million different factors; changing the color of a button in one context doesn't necessarily tell me much about how people would respond to that experience in another context.

It's easy to underestimate how complex things are, because we only see some superficial aspects of e.g. a user/software interaction model. This flaw is down to how our brains work -- ref "What you see is all there is".

citrablue• 7 years ago

What you said is correct, but I'd like to point out that controlled scientific experiments may not be the right approach for e.g. optimizing conversions on a website.

The reason is that websites are a dynamic environment. All things equal, better controlled experiments are great. However, visitor behavior, especially when from dynamic sources (google serps change weekly), changes all the time.

And that's why I prefer MAB over A/B tests -- A/B tests don't adapt to a dynamic system, so we often wish away the changes in the system to use it. Does anyone go back and re-test their biggest wins?

err4nt2• 7 years ago

Does anyone go back and re-test their biggest wins?

Yes, absolutely! We do research first, then come up with simple, well-controlled tests. Once we have a winner we can either lock it in, but often we continue to research and experiment on the new knowledge we gained. A hefty minority of the tests I implement build on past wins to further flesh out what works and what doesn't with knowledge and the data to back it up.

arendtio• 7 years ago

The purpose of an A/B test isn't to always show the best performing result

The key part is 'not always'.

Using A/B tests just to find the better performing version is a valid use case too. And if you have the traffic, multi-armed bandits are a nice way of automating the whole procedure. Even scientifically speaking there is nothing wrong with them. Their biggest issue is that they require a lot of traffic for significant results.

joshuamorton• 7 years ago

In practice, MAB should require less traffic than an A/B test for the same level of significance. Although it's much more difficult to rigorously describe that significance level with a MAB.

antidesitter• 7 years ago

That’s just a contextual multi-armed bandit.

throwawaymath• 7 years ago

Also, I work in this field and I will just say that people _do_ behave differently based on traffic source: i.e. users coming from Facebook behave alike, but different than traffic from Reddit who act similarly to each other.

Out of curiosity, what is your hypothesis for explaining this difference in behavior? Would you say it's primarily due to differing contexts in which a link is posted, or differing populations on each platform? Or maybe these both contribute about equally?

Stated another way: would you expect the same individual to behave differently coming from Facebook versus coming from reddit, if they happen to be a user of both?

orasis9• 7 years ago

Different demographics, different intent, and different mental context all play a factor.

paulddraper2• 7 years ago

The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things.

Occasionally there is pure scientific interest.

But far more frequently, the purpose of the A/B is to optimize the outcome.

This is why Google Analytics has exclusively chosen multi-armed bandit for its its A/B test framework.

mobjack• 7 years ago

It doesn't have to be a pure science interest.

If you want to use the results from one test to inform what to test next, then A/B tests are better optimizing for truth.

Most website changes dont make a significant difference in conversion. If you use MAB, then you dont know if the winner is really better or the result of random variation.

bduerst• 7 years ago

Wouldn't a simple solution be to create a handler that runs a MAB for each traffic source?

If you're worried about the site changing for people between visits from different traffic sources, you can cookie their MAB/test-assignment on the first visit.

Causality1• 7 years ago

I encounter a/b testing primarily in article headlines, where the choice is always "accurate, informative title" or "inflammatory sensational bullshit clickbait" and the clickbait wins every time.

astazangasta• 7 years ago

In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer.

As a cancer researcher I read this as a just criticism of my work.

messe• 7 years ago

Despite being just text, images and tables, the post doesn't appear at all with js disabled.

gchamonlive• 7 years ago

For me the website is down (its 404'ing, accessing from Brazil) , but it can still be accessed in webarchive: https://web.archive.org/web/20190527144648/https://stevehano...

jedberg• 7 years ago

Before I even clicked I knew this was going to be about multi-armed bandits. I agree with the author -- why don't more people know about this? It's so effective and easy, and you get results much quicker.

sbov• 7 years ago

One thing that is vastly more expensive than picking a winner in your A/B test is maintaining N variations across M features.

XCSme• 7 years ago

Just a small nitpick: this doesn't take into account implementation cost. If you want something dynamic like this it means your app has read access to all the analytics recorded (or at least the ones needed for optimization). Most of the times apps only send data to the analytics services, developers read/analyze them and act based on the data. I personally didn't work on any apps that were using analytics read access within the app. So, in most cases it's a lot more than 20 lines to implement an approach like this (hopefully your analytics platform exposes an API with a read endpoint that you have to then integrate into your app) compared to A/B testing, where you just show several versions and then analyze the data and iterate.

ineedasername• 7 years ago

Good point, the article says "do it in 20 lines of code", but implementation at scale may require a little more engineering than that.

j2kun2• 7 years ago

UCB1 is really not that much more complicated than epsilon-greedy. Some slightly sloppy code I wrote a few years ago, maybe 20 lines of code: https://github.com/j2kun/ucb1/blob/master/ucb1.py#L7-L35

Sure you have to read a bit more to know why it works, but if you write your code well you could plug this in without any extra trouble. It's not like you need a special optimization solver as a dependency.

nestorD• 7 years ago

And, for not much more effort (computing the variance of your samples), you can use UCB1-tuned [0] which gets rid of the 'c' parameter and tends to be even better.

I personnaly think that it should replace UCB1 as a baseline when trying bandit algorithms.

[0] https://homes.di.unimi.it/~cesabian/Pubblicazioni/ml-02.pdf

606542• 7 years ago

That's a nice paper, thanks for posting this!

j2kun2• 7 years ago

It's funny, I had read that paper a few times while learning about bandit learning, and I never noticed their version, which funnily enough outperforms vanilla UCB1 in all of their tests!

606542• 7 years ago

Multi-armed bandits are also well known in game AI. They got popular with the introduction of Monte Carlo Tree Search, where MAB are used there to select which subtrees to search for largest expected payoff, e.g.:

https://www.aaai.org/ocs/index.php/AIIDE/AIIDE13/paper/view/... https://courses.cs.washington.edu/courses/cse599i/18wi/resou... etc.

For what it's worth the MAB algo in the original post looks like Epsilon Greedy. It's probably better to look into Upper Confidence Bound variants like UCB1 that dynamically adjust how much to explore vs exploit, e.g.:

https://towardsdatascience.com/comparing-multi-armed-bandit-...

praptak• 7 years ago

UCB on game trees (MCTS) was the first breakthrough that created decently playing Go programs, if I remember correctly.

cgearhart• 7 years ago

You are correct. MCTS + UCB and other variants were state of the art leading up to AlphaGo. And even then, MCTS was also used in AlphaGo.

The main change in AlphaGo was using a deep learning network to encode a value network for fast rollouts and a policy network for move selection (rather than using the UCB rule). They later removed the value network and rollouts entirely, but even AlphaZero uses MCTS.

lowdose• 7 years ago

Microsoft had good talk about contextual bandits and machine learning. The business case discussed was webpage conversion optimization which increased by 80%.

https://www.microsoft.com/en-us/research/blog/new-perspectiv...

https://podcasts.apple.com/nl/podcast/microsoft-research-pod...

cfarm2• 7 years ago

MAB seems to find a local maxima subject to input biases whereas an AB test is aimed to figure out a scientific truth and isolates out all potential biases in the system. I'd be curious to hear where a MAB approach and an AB test did not yield the same results and why that happened.

orasis9• 7 years ago

AB testing doesn’t eliminate time based biases such as novelty of new features, special events, pay day, tax day, etc.

cfarm2• 7 years ago

Sure - my point is that you could design the experiment such that it could. It's much more scientific.

ampersandy• 7 years ago

We (Facebook) just released our Adaptive Experimentation platform as open source: https://github.com/facebook/Ax. It supports bandit optimization for discrete sets of candidates and bayesian optimization for continuous/very large search spaces.

https://ax.dev/docs/why-ax.html

_eigenfoo• 7 years ago

Bandit algorithms can also be approached from a Bayesian point of view! This lets you quantify the uncertainty in your estimates (e.g. how uncertain you are than ad A has a higher CTR than ad B), which a lot of other bandit methods don't offer.

To toot my own horn a bit, I wrote a blog post about Bayesian bandits: https://eigenfoo.xyz/bayesian-bandits/

seoulsister• 7 years ago

Great to see Multi-Armed Bandits getting coverage here. I wrote an article last year to help with gaining a deeper understanding on the topic:

https://towardsdatascience.com/bandits-for-recommender-syste...

dr_dshiv• 7 years ago

I published this work in CHI on the use of multiarmed bandits in educational games. My biggest takeaway was the importance of choosing the right metric -- because our optimization worked too well.

https://www.researchgate.net/publication/301935710_Interface...

barbarr• 7 years ago

This was an interesting read. I played the game, too! Interesting to see how the optimizer converged on such a large ship size, even though you only allowed that case to verify whether the optimizer would successfully reject it.

moultano2• 7 years ago

Does anyone have a reference for solving multi-armed bandit problems with a finite time horizon? I would like something that derives rules or heuristics for how your explore/exploit tradeoff changes as the horizon approaches.

This seems like an obvious extension, and something that someone should have worked on given how long this problem has been around, but I've been unable to find anything on it. Any pointers?

bhl• 7 years ago

What do you mean? Most analyses of multi-armed bandit algorithms assume a finite time horizon. And if not, they use the doubling trick for infinite time horizons.

moultano2• 7 years ago

Thank you, now I realized that I had misunderstood the notation.

Aeolun• 7 years ago

How is a bandit related to this?

Based on just the article it could be a multi armed chef.

masonicb00m• 7 years ago

A slot machine is a “one armed bandit”.

everyone• 7 years ago

"hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. Instead, they have been refining techniques for getting you and me to click on banner ads."

Just out of curiosity... Have you ever purposefully clicked on an ad on the internet? I honestly dont think I ever have.

ps. I mean an outright overt straight up ad, not, for example, some article linked on HN that is a thinly veiled promo piece for something.

orasis9• 7 years ago

I clicked on an ad for women’s yoga pants once. The retargeting makes the web a constant stream of delights.

WrtCdEvrydy• 7 years ago

Interesting...

The multi-armed bandit problem (2012)

Related Stories