How to A/B Test Using Javascript in Your Posterous Blog

http://themetricsystem.rjmetrics.com/how-to-ab-test-using-javascript-in-your-poste

by rj_metrics • 15 years ago

19 3 15 years ago

themetricsystem.rjmetrics.com

3pt141592• 15 years ago

I say this all the time, but be very very careful split testing in JavaScript. It is best left for very small elements on your site. Otherwise your results will be misleading. Once you get to a nontrivial size implement it server side.

sshrin• 15 years ago

Can you elaborate on why you think the results will be misleading? Does it have to do with data integrity issues of some sort?

3pt141592• 15 years ago

Because for large elements there is a delay, especially when your user has a slow browser and is talking to a third party service that is far away.

For example, say you ran a company that had about 100 paying users, almost all of which were in the SV tech scene. A fine opportunity to use JS because they all have fast browsers and the odds that the delay between KISS Metrics or Mixpanel and them are fairly low.

If, on the other hand, you sell Bingo Card Creators to thousands of teachers around the globe the latency is going to kill you. A teacher in England will see the button (s)he wants to click on, then all of a sudden it is gone and replaced with a intro video, or, even worse, the prices have all gone up and there are different options on the pricing page.

Not only is your split test reporting the wrong winner (because of a negative bias on the B side of the test) but it can also erode the confidence visitors had in your service.

How to A/B Test Using Javascript in Your Posterous Blog

Related Stories