Does A/B testing lead to crappy products?

http://andrewchenblog.com/2009/03/02/does-ab-testing-lead-to-crappy-products/

by wmorein • 17 years ago

32 4 17 years ago

From TFA: "Thus, my answer to the title question is that no, A/B testing does NOT lead to crappy products, but poor decision-making around data can absolutely lead to it"

A more accurate title would have been "Metrics don't replace strategy."

madh• 17 years ago

I like the point. With the gobs of data available in web product development, I think there is sometimes too much focus on metrics. Metrics are absolutely critical but can be wrong. They are really just an input into a sound strategy.

suhail• 17 years ago

Metrics aren't wrong, it's how you perceive the results of what you're testing. If you make too many assumptions, it's your fault.

There's too many variables involved for metrics to be the final solution towards success, part of building a startup is taking risks and your intelligence to come out on top.

joe_the_user• 17 years ago

Are crappy products unprofitable?

He says Hollywood keeps making sequels even though they turn out to be bad. But does he show that Hollywood loses money making bad sequels?

IF bad inevitably translated to unprofitable, Hollywood would have a strong incentive to make good movies. If movie goers flock to movies that are just a mediocre mishmash of the themes they've seen before, then Hollywood has every reason to keep putting them out.

Does A/B testing lead to crappy products?

Related Stories