How Much Optimization Is The Right Amount?

https://training.kalzumeus.com/newsletters/archive/premature_optimization_postmature_optimization

by dennisgorelik • 13 years ago

93 17 13 years ago

training.kalzumeus.com

dennisgorelik• 13 years ago

Finally Patrick seriously addressed the topic of when A/B testing is NOT needed.

entendre• 13 years ago

relevant xkcd: http://xkcd.com/1205/

mebreuer• 13 years ago

Title reads: How to optimize your optimization.

Now what I'd really like to know is, how much optimization of my optimization is the right amount?

Jokes asides, good read!

rcthompson• 13 years ago

Premature optimization of your optimization is the root of the root of all evil.

gk1• 13 years ago

You're doing it all wrong! You're only optimizing for the local maxima of your optimization's optimization!

pstuart• 13 years ago

As someone whose goal it is to have viable startup next year, this is pure gold. Thank you Patrick!

patio112• 13 years ago

Thanks, glad to have helped.

jmduke• 13 years ago

As a point of context, I'm guessing this blog post was based a little bit on the "20 days as a growth hacker" submission (and Patrick's subsequent comment): https://news.ycombinator.com/item?id=6883357

patio112• 13 years ago

That did make me tie a string around my finger, but there's no particular line in the article that you should read as me referring to him.

The editorial calendar basically dictates that I have to talk about A/B testing for the time being (product launch and all), and I generally prefer to say new things rather than saying old things. I've written an awful lot about A/B testing from different angles, but have rarely said "Actually, hold up, don't do it."

I previously had, if anything, a little too much "All I have is a HammerFactory so all problems must implement INail or they will be recast until they do" going on due to my own business history. I've talked to many people who have that to varying degrees over the years.

So "Hmm, as I come to be a little more mature in my understanding of this, there's actually some times where I wouldn't reach for scalable, metrics-driven, quantitative conversion optimization as the first solution" seemed like a pretty good skeleton of a piece.

cperciva• 13 years ago

Patrick: Sorry to hear you've been sick!

Everybody else: There's another reason to "not optimize yet" in many cases: If your traffic doesn't support A/B testing. Taking the example of 100 visitors/day of whom 2.5/day sign up for a free trial -- you're going to have anywhere from 0 to 5 signups/day on a regular basis. If could increase your yield from 2.5% to 3.5%, you should be thrilled -- but an A/B test would take over a month to distinguish that from a random fluctuation.

timr2• 13 years ago

That's true, but there's still a valid reason to test things on small traffic: you don't necessarily need to have statistical certainty on every decision. A lot of the time, it's sufficient to know that when comparing two dramatically different options, you can't quickly and easily distinguish between them. That's a strong hint that you should put your future efforts elsewhere. It makes no sense to dedicate lots of effort to a project if the impact is statistically indistinguishable from noise.

Moreover, the changes that matter early on are the 10x changes, and they'll often be subjectively visible to you, even if the math shrugs and tells you that it isn't confident. So while it probably isn't worth testing button copy or images on 100 visitors a day, it might be worth testing complete redesigns, or entirely different value propositions.

teleclimber• 13 years ago

10x changes are hard to come by, even at an early stage. That's like going from 3% to 30% conversion. If your site is mostly broken and one change takes you to 30% conversion, then you probably didn't need an a-b test to figure it out.

EGreg• 13 years ago

Simple: optimize the amount of optimization, and you're good to go!

abyx• 13 years ago

Lots of good points there, one of my favorites is the Collison Close: “Open up your laptop. I'll get you an account and code the integration right now.”

ufmace• 13 years ago

Good article. I especially liked the section on how to get your first user for a more serious business app and how you should treat them. That fleshed out a lot of what I was already thinking about how to go about launching the app that I'm working on.

exo_duz• 13 years ago

As always your stuff is great to read and very insightful! Thanks again Patrick!

How Much Optimization Is The Right Amount?

Related Stories