‘Try demo’ or ‘Buy now’: A/B testing finds which button increased clicks by 47%

http://visualwebsiteoptimizer.com/split-testing-blog/testing-call-to-action-buttons/

by paraschopra • 15 years ago

38 13 15 years ago

Egregore• 15 years ago

Have the resulting sales been measured? Because I think some people might click on demo button thinking that it is a free product.

paraschopra3• 15 years ago

Yes, the customer hinted that their sales did increase and were very happy with results but they don't fully reveal this data (as it can be sensitive).

Jabbles• 15 years ago

Where's the raw data?

danmaz74• 15 years ago

I just went to the website that did the test, and found that they currently have two buttons: "Try Demo for Free" and "Plans and Pricing", not just the "Try Demo for Free" button as in the winning test.

This makes sense to me - as a matter of fact, while reading the post I was thinking that before trying a demo I usually check the price I would have to pay in the end - but I wonder why this variation wasn't talked about in the article. "Plans and Pricing" isn't a call to action, but having that second button is for sure different than having just one and knowing which clickthrough it gives would be very relevant.

saurik• 15 years ago

In case anyone finds this interesting (as it is so glaringly apropos to this "Buy Now" blog post): I ran an A/B test recently of "Buy Now" vs "Purchase" (which is a drastically different and more subtle question than comparing whether people want to install a demo vs. make a purchase), and found almost no difference.

Unfortunately, due to the usage of a multi-armed bandit algorithm (which attempts to "exploit" already learned knowledge to not lose sales during the test), my data is somewhat "skewed" (in that I have an order of magnitude more tests for one of the hypotheses), but here are the raw results:

"Purchase": 35,715 sales from 3,255,882 impressions (1.097%)

"Buy Now": 4,042 sales from 376,227 impressions (1.074%)

If you compare the confidence interfaces with a Beta distribution it is difficult to feel comfortable claiming Purchase is a winner, but that small benefit is why the algorithm kept trying to use it over the other case. Put differently: despite the large sample, I believe that tiny difference is not statistically significant.

(Additionally, for completeness, and as this is important for anyone who might care about this experiment: my app had previously said "Purchase", so there are likely guides online that tell the user how to buy things, or people may have had memories, which may have caused "Buy Now" to be ever so slightly more confusing.)

0x12• 15 years ago

You may want to disable your skewing algo for the first 10K or so sales and/or if the difference between the two tests is within certain bounds.

dlss• 15 years ago

This performs poorly when testing variations that convert at 0%

makira• 15 years ago

Where's the original page ? Only the 3 variations are presented...

pjo• 15 years ago

For one of our customers, "Start Checkout" was a win versus "View Cart". You can find more case studies on our company blog.

http://monetate.com/2011/10/website-testing-wins-button-mess...

revorad• 15 years ago

Did you control for the fact that the 'Try demo' button is red and the 'Buy now' button is blue? I'm sure I've read about A/B test studies (probably on the VWO blog itself), which showed red buttons had higher clickthrough rates than other colours, which could be a big confounding factor.

paraschopra3• 15 years ago

Excellent point! No, I don't think they controlled for this.

brador• 15 years ago

"selling" them on the demo version is not the same as selling the product out (which brings in direct cash in the bank and is the effective "end game").

You need to multiply the demo download conversion with the number who go ahead and make a purchase after trying said demo.

The fact that this was not done brings questions upon the quality of methodology of visualwebsitreoptimiser. Especially when they put out blog posts pimping the results. Up to this point I've been relatively happy with their work and contributions to the A/B field. Which makes this dodgy conclusion a bit of a shame really.

paraschopra3• 15 years ago

I wrote the case study. The results we publish actually depends on what our customers care for in those particular tests. We don't actively recommend them which tests to run (unless they ask for) so if they don't measure impact on sales (which we would have recommended had they asked for), we can't ask them for the data.

Most customers would know if their sales were negatively impacted during a test, so if you happen to increase demos while still not negatively impacting sales there's nothing wrong with that.

‘Try demo’ or ‘Buy now’: A/B testing finds which button increased clicks by 47%

Related Stories