The Importance of Sequential Testing

http://www.austinrochford.com/posts/2014-01-01-intro-to-sequential-testing.html

19 6 13 years ago

noelwelsh2• 13 years ago

[I'm assuming the author posted this. MidsizeBlowfish has no profile information.]

Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives?

My field is bandit algorithms, which are a generalisation of hypothesis tests. Here there are algorithms like Empirical Bernstein Stopping (http://icml2008.cs.helsinki.fi/papers/523.pdf) which achieve the same early stopping from a different approach. If this is of interest sign up to the course I'm running (http://noelwelsh.com/data/2013/08/16/free-bandit-course-draf...) -- I plan to cover this algorithm soon.

BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)

MidsizeBlowfish3• 13 years ago

Haha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school.

The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.

noelwelsh2• 13 years ago

It would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.

nitrogen• 13 years ago

A heads up to the author: with narrower browser window sizes (e.g. less than ~1024px), the left side menu overlaps the text.

MidsizeBlowfish3• 13 years ago

Just noticed that on my iPad, will look into it.

The Importance of Sequential Testing

Related Stories