Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

http://www.fourhourworkweek.com/blog/2009/08/12/google-website-optimizer-case-study/

by divia • 17 years ago

51 22 17 years ago

videophile7• 17 years ago

There is some "New Math" being invoked here. If the conversion rate goes from 24.4% to 29.6%, it's disingenuous to claim a 21.2% improvement.

I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment).

Edit: Downmodded for asking for real math over marketing math?

jbm• 17 years ago

It's been a while since I've taken a math class, but I'm not sure what the problem is with calling it a 21.2% improvement.

The old conversion rate was 24.4%. The new one was 29.6%.

(29.6 - 24.4) / 24.4 = .213 = 21.3%.

Is it misleading because their number of conversions is so low? (Not trying to be snarky, I have no idea)

videophile7• 17 years ago

The improvement in the conversion rate is (29.6 - 24.4) = 5.2%.

Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.

onreact-com2• 17 years ago

"Conclusion: Simplified design improved conversion by an average of 20.45%." is the most important sentence in this post.

To be honest you don't have to use Google Website Optimizer to notice that.

patio112• 17 years ago

It is easy to believe the things you want to believe. A/B testing is useful for convincing you of the things you don't want to believe.

Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right?

Without ruining the surprise: that is a testable hypothesis. ;)

onreact-com2• 17 years ago

A/B testing is great, no doubt. The lesson of solely simplifying is just not a great outcome of such a test. It's like driving a car for the first time and then telling everybody that it's faster than walking.

ramit• 17 years ago

Yeah you do. I ran a test on a friend's site, convinced that a short version would outperform the long-copy version. Conversions dropped by about 30%, which would have cost him tens of thousands of dollars per year.

Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

Related Stories