We didn't use A/B tests to make the iPhone at Apple

https://twitter.com/kocienda/status/1113450945705177088

by branko_d • 6 years ago

36 12 6 years ago

robbyking• 6 years ago

I understand why they didn't -- when it comes to new technologies, most end users can only imagine extensions of what they already have, not entirely new paradigms. (Just look at the technology predictions from the 1960s!)

A/B testing is best suited for determining user preference between existing designs, not new ones.

FindMySocks• 6 years ago

Exactly this. It's to measure the nuances between designs to see if they improve user interaction.

If you're at the AB stage you should have a fairly mature prduct where you're optimizing to move forward from the low-hanging fruit customers.

jldugger2• 6 years ago

Didn't Apple argue in Apple v Samsung that Apple invested millions in focus groups?

vgel• 6 years ago

A focus group isn't an A/B test. An A/B test gives you a number per option. A focus group is a conversation with customers where they can actually talk to you and explain their opinions. An A/B test could tell you customer click on the darkest blue shade of links the most, but a focus group could tell you that they were clicking on them by mistake.

jldugger2• 6 years ago

Sure, but the dichotomy being offered here is: "instead of listening to our users, we just had to listen to our own good taste." The original "41 shades of blue at Google" A/B rant is a tale of designer disenfranchisement. And of course a former Apple designer would latch onto that -- its carefully curated brand identity is 'good design.'

Ken goes on to point out that instead Apple relies on exec demos (true); yet zero mention of consumer focus groups. Unless your idea of a good product focus group is one slightly egocentric billionaire.

And really, the entire argument comes down to: "we don't have enough hours in the day to consider every detail, so we just focus on the stuff execs see." When Ken brags about not consulting data in building his software, I view it as equal parts damnation and praise. If he didn't consult any user metrics in designing the software keyboard autocorrect[1], we now know why its reputation with users sucked for years.

[1] https://qz.com/1380188/ken-kocienda-qa/

xt00• 6 years ago

Use of A/B tests does feel like an organization where the leadership is not there to either help make hard decisions or the culture doesn't allow people lower down the ranks to make these decisions. The excerpt mentions "good taste".. well plenty of people don't have good taste, so while saying we don't use A/B tests because we have good taste, I think its likely a combination of good taste and leadership in the company that allows the good taste to drive the decisions and some level of agreement among a group of leaders of what "good taste" is..

pardonmesir• 6 years ago

this is maybe missing the point but how about the blue/green of iMessage? surely they went to great lengths to find the most displeasing shade of green for iPhone owners to talk to non-iPhone owners.

ksec• 6 years ago

surely they went to great lengths to find the most displeasing shade of green for iPhone owners to talk to non-iPhone owners.

This is genius! It might very well be true.

drzoltar• 6 years ago

There’s a lot of false equivalence in this post. The message seems to be “if you want to build amazing new things, don’t A/B test.” This is vacuously true. You certainly won’t discover fusion, or make the next iCar through rote testing.

What’s hidden though is that out of the thousands of companies that will try, only a few will succeed. The rest will burn through millions and have nothing to show for it. A/B tests are done because they work in the short term, providing runway for the long term. It’s true that many companies abuse these tests and heavily lean on incremental improvements. Management can become complacent, spineless and rudderless, thereby refusing to make hard choices that go against the data. But, it keeps volatility down and appeases the overlords holding shares and vested interest in the company.

It does make wonder though how, despite being so secretive, Apple got a lot of things right with the iPhone and iPod. So whatever their process was for internal focus groups is definitely worth exploring. We’ve all heard how Jobs was insanely attentive to details, but I still wonder where the overall cohesion of design came from. No one person could have possibly gotten all of that right.

systemvoltage2• 6 years ago

Design arrogance of Apple is misplaced and frankly, annoying. Sottsass, Dieter Rams, Bob Noorda, Paul Rand, etc would not invite Apple designers to their parties. I don't think Apple designers have good taste. They're just fashionable half baked designers with a sense of minimalism with no idea about pragmatism and functionalism. Using premium materials doesn't necessarily make good Design. In the Design circles, we call those things - luxury goods.

jacques-noris• 6 years ago

You are wrong about Dieter Rams, he likes Apples Design: "You can count the companies that really take design seriously on ten fingers. Apple is one of them."

https://www.faz.net/aktuell/technik-motor/digital/designer-d...

systemvoltage2• 6 years ago

Thanks for the correction.

We didn't use A/B tests to make the iPhone at Apple

Related Stories