Ask HN: What is your one tip about A/B testing you would share?

https://news.ycombinator.com/item?id=8929813

by raycloyd • 12 years ago

4 9 12 years ago

news.ycombinator.com

Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!

onion2k• 12 years ago

Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_wi...

Someone1234• 12 years ago

That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.

However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.

I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.

mtmail• 12 years ago

If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.

domrdy• 12 years ago

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.

siddharthdeswal• 12 years ago

Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.

dairgram• 12 years ago

Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right?

I have seen statistically significant differences in outcomes in A/A testing.

A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.

seekingcharlie• 12 years ago

Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.

catman01• 12 years ago

Kittens will always improve conversion rates.

hkiely• 12 years ago

Choose and test one independent variable at a time. Then, check your results for statistical significance.

Ask HN: What is your one tip about A/B testing you would share?

Related Stories