Is Apple making implausible iPhone satisfaction claims?

https://www.perfectrec.com/posts/is-apple-making-implausible-iphone-satisfaction-claims

by nowooski • 3 years ago

121 102 3 years ago

aryan14• 3 years ago

Any situation where a company can be put in a better position with statistics will 99% of the time lead to the company hiring a statistics firm that favours them.

post_break• 3 years ago

Just like how they compare the M2 processors to an Intel iMac in bench marks which in itself is kind of ridiculous, this is unsurprising.

threeseed2• 3 years ago

They do that to encourage existing Apple users to upgrade.

Apple is growing market share but not massively so. Hence new revenue will need to come from it's existing user base.

arvindh-manian• 3 years ago

A number this high is absurd to get in a survey of the general public -- always remember Lizardman's Constant [1].

[1] https://slatestarcodex.com/2013/04/12/noisy-poll-results-and...

dpkonofa2• 3 years ago

It’s not a survey of the general public, though. That’s why the OP article is nonsense. This is most-likely a NetPromoter survey. It is a survey of verified iPhone purchasers/owners just after the 14 day return window who decided to keep it. It’s self-selective but it’s not unattainable like the article claims.

howinteresting2• 3 years ago

It is not a NetPromoter survey as clearly outlined in the article. It is a non-representative 451 Research survey.

nisse72• 3 years ago

TIL: Obama is a hippopotamus

pm22223• 3 years ago

If only there’s no carrier lock and photos can be backed up easily to personal storage.

matricaria• 3 years ago

Is carrier lock still a thing in the US?

RandallBrown• 3 years ago

I think it is, but it's not like you can't get unlocked iPhones in the US. I think carriers also are required to unlock your phone if you ask them, but I may be imagining that.

lotsofpulp2• 3 years ago

If you buy from a carrier, yes.

RhodesianHunter• 3 years ago

Generally it's only a thing any more if you get your phone discounted as part of signing up with or renewing at a certain carrier.

If you just go out and buy a phone outright you can generally use it anywhere.

pm22223• 3 years ago

It is. I paid discounted full price through my employer and because it’s on a business plan it’s locked.

WA• 3 years ago

Photo Backup easily: PhotoSync. Been using it for years.

jjtheblunt• 3 years ago

that all has worked in the US for 15 years or more, i'm one who has done that at least as far back as 18 years ago, though it might not be what everyone does.

possibly one can get a phone subsidized and locked until an extortion fee is paid, though, or get a phone from work that an employer didn't pay to have unlocked.

worrycue• 3 years ago

Carrier locks are not an Apple-only thing - https://en.wikipedia.org/wiki/SIM_lock

Seem to be a common feature on most phones. Don’t like it? Bug your government to outlaw the use of the feature like “Canada,[5] Chile,[6] China, Israel,[7] and Singapore”.

iTunes backups my photos. No idea how to access them though - apart from restoring to a working phone.

jrockway2• 3 years ago

iTunes backups my photos. No idea how to access them though - apart from restoring to a working phone.

You should be able to go to iCloud.com and log in. If you have the E2E encryption enabled, it will ask you to give iCloud the decryption key, so the webserver can serve you the decrypted files. (This ability can be turned off if you're really paranoid.)

beebmam• 3 years ago

I've yet to use a single iphone that I was satisfied with.

My last experience with an iphone was when my GPS began to malfunction and Apple's proposed solution was to have me buy a brand new iphone at no discount.

nashashmi• 3 years ago

You must be in the top 1%

ricardobeat• 3 years ago

GPS malfunction? There’s something I haven’t heard in 15 years of smartphones.

asutekku2• 3 years ago

They are using Google reviews as proxy for customer satisfaction. I really can't take rest of the article seriously after that, knowing anyone can leave a review at Google and there's no guarantee they own the phone.

Sure, 99% satisfaction rate is pretty much impossible, but if all Apple asked was "are you satisfied with iPhone 14" with "yes" & "not at all" options and only surveyed iPhone 14 owners, I can easily see the number being 99%.

ars3• 3 years ago

I really can't take rest of the article seriously after that, knowing anyone can leave a review at Google and there's no guarantee they own the phone.

Do you mean you can't take Apple seriously? The article is calling out this issue, not engaging in it.

I can easily see the number being 99%.

No, it can't. It's utterly impossible for it to be 99% unless it's fake data.

sezycei2• 3 years ago

utterly impossible

No, it's not. Unbelievably unlikely? For sure.

dclowd99012• 3 years ago

They mention in TFA that even surveying anomalies (people accidentally answering negatively) can’t be represented by a 99% figure.

That said, if I had a survey that asked a couple times in different ways if someone was satisfied with the iPhone and they gave conflicting answers, I’d probably simply throw their answer out for being nonsensical. Maybe that’s what’s going on here: they’re teasing out survey anomalies.

nowooski7• 3 years ago

I'm the post author. I suspect aside from conducting extremely unrepresentative surveys, they are also aggressively filtering out respondents. I didn't include that since it is purely speculative on my part, but I'd guess that is part of what is happening.

I'd note that other seemingly innocuous things could have an impact too. For instance, if they surveyed only people who've owned their phone for 90 days or whatnot, you would eliminate the vast majority of people who didn't like it and returned it from the sample.

99% is an extraordinary claim, which is why we think Apple should be more transparent about their methodology.

nowooski7• 3 years ago

Hey - I'm the author of the post. Thanks for reading.

I included the Google reviews as a secondary example since it is public information. Google reviews are actually more positive on the iPhone than the third party surveys, which found satisfaction rates in the low 80s for 5G iPhones.

dpkonofa2• 3 years ago

Yeah, especially when you consider that Apple likely uses NetPromoter as its primary metric for customer satisfaction. NetPromoter only sends out surveys to verified purchasers and the question is a 1-10 scale for satisfaction. Anything 8-10 is “satisfactory”. This is not only plausible but likely and is why the rating is so high. There wouldn’t be very many anomalous responses of people not understanding the question because the question is literally “Would you recommend the iPhone (whatever model) to friends and family?”. That’s not a hard question to understand.

This is not a seriously written article.

FireBeyond2• 3 years ago

Apple likely uses NetPromoter as its primary metric for customer satisfaction

The article is entirely about a completely different firm, 451 Research, who actually is not and does not use NetPromoter. They state that they use a pool of "high wallet spend early adopters and technical people".

(For added irony, 451 Research says it's own customers are 98%+ satisfied with them...)

nowooski7• 3 years ago

Hey - I'm the post author.

Apple may use NetPromoter internally, but on the earnings calls they are citing satisfaction surveys from 451 Research.

In 2018, they shared some of the response categories from the survey. They are customer satisfaction questions "Very Satisfied, Somewhat satisfied." Not recommendation questions.

FireBeyond2• 3 years ago

I really can't take rest of the article seriously after that, knowing anyone can leave a review at Google and there's no guarantee they own the phone.
but if all Apple asked was "are you satisfied with iPhone 14" with "yes" & "not at all" options and only surveyed iPhone 14 owners, I can easily see the number being 99%.

"and only surveyed iPhone 14 owners"

451 Research's own statement on their survey mentions _nothing_ about owning the phone:

The findings are based on a December 14th – January 22nd survey of 3,568 primarily North American consumers from 451 Research’s user insight service, Voice of the Connected User Landscape and its Leading Indicator panel of 25,000 accredited business and technology professionals whose application for inclusion identifies them as having a high wallet share being used for personal technologies, and a high readiness to try new products and services. The service captures consumer and business spending through weekly tracking studies, delivering a continuous view of user perceptions and purchase activity as new products and services enter the market.

And also, how valid would you consider a "satisfied" rating from an iPhone 14 owner who'd never owned anything but an iPhone? There's value in their ongoing satisfaction, but relatively speaking what does it mean? It could just as easily be interpreted as "not annoyed enough to change ecosystems".

SllX• 3 years ago

And also, how valid would you consider a "satisfied" rating from an iPhone 14 owner who'd never owned anything but an iPhone?

Why wouldn’t it be valid? If somebody says they’re satisfied with something, take them at their word. There’s no shortage of people who complain about anything and everything, especially if they paid money for the thing in question, and especially if they paid a lot of money.

Apple is a single vendor that makes a very small number of models and earned $205.49 billion on iPhones alone in 2022. This excludes a lot of their “services” revenue of which I’m convinced the Lion’s share and then some is also basically just iPhone revenue that wouldn’t exist without specifically the iPhones tied to them. You don’t get anywhere near that for one company on one hardware product division in one year without some seriously high customer satisfaction and repeat buyers.

photonerd• 3 years ago

how valid would you consider a "satisfied" rating from an iPhone 14 owner who'd never owned anything but an iPhone?

Very? It’s a subjective position no matter what but they know their own satisfaction level better than you do, certainly.

goldenjm2• 3 years ago

I'm quoted in the post. We're using Google reviews as one of several imperfect comparison points to Apple's 99% claim. You're absolutely right about a key downside to Google reviews. The limited information 451 provides on their survey methodology shows that they aren't asking about satisfaction as a yes-or-no style question.

If you want to go really deep on online review manipulation, I'm co-author of this research paper on ratings inflation in online marketplaces: https://pubsonline.informs.org/doi/abs/10.1287/mksc.2022.135.... If you get in touch, I can send you more papers on fake reviews that I like.

camillomiller2• 3 years ago

America’s favorite holiday, Thanksgiving, enjoys 82% popularity according to a 2023 survey by Statista. Memorial Day comes in second at 77%.
The three most popular movies of all time on IMDB are The Shawshank Redemption, The Godfather and The Dark Knight. They have ratings of 9.2, 9.2 and 9.0 respectively.

These examples are so orange and apples it’s baffling the authors even came up with them.

You compare the most advanced personal pocket computer of all time with a patriotic holiday (assuming patriotism as an absolute value) or a movie about inmates escaping a prison. Like, really?!

throw47474777j4• 3 years ago

I'm quoted in the post.

Quoted by your employee who wrote the post? At your request? To generate traffic for your site?

howinteresting2• 3 years ago

Do you think that (even if that was the survey) it is an honest survey? Do you not care about the richest company in the world being so blatantly dishonest?

mgh2• 3 years ago

A sample limited to accredited business and technology professionals who consider themselves early adopters and spend a lot on gadgets is, of course, not representative of iPhone purchasers generally.

1. Granurality: as OC mentioned, w/ an overall satisfaction "yes or no" method can reach there.

2. Bias: Pretty sure they did the research before releasing the product to get feedback from a loyal subset base/target customers.

Why would a study include non-Apple users if they have never used or refuse to use Apple products? Wouldn't this be biased if they are included to be "representative"?

I know people who don't use Apple products yet are vocal critics...humans can be highly ignorant creatures

razemio3• 3 years ago

"They opened our email 73 times, but didn’t respond."

This counter does not work right? I know it is a minor thing, but I immediately stopped reading further. I have a problem...

minsc_and_boo2• 3 years ago

Email pixel tracking is an industry standard these days. Every time the email loads the pixel, it counts +1.

Some platforms like Gmail partially obfuscate it.

razemio3• 3 years ago

I am totally out of the loop when it comes to these things. My client always opens in plain text or html render with JS and external content disabled. I believe this is true for most clients. On the other hand security scanners, browsers engines and other Middleware stuff can trigger such a counter without any user interaction if I am not mistaken.

ridiculous_fish2• 3 years ago

Right, especially with Apple.

Apple's Protect Mail Activity feature, which is on by default, "opens" all emails received on a server, to hide the user's IP address. This causes emails with tracker pixels to be falsely reported as read.

For a while, I was confused why all of my GitHub notifications were being marked as read!

https://twitter.com/ridiculous_fish/status/14604379978667376...

jonas-w3• 3 years ago

Later on they said they use Mailtrack [0]. I looked at their FAQ and they just use a tracking pixel. I don't really think that mail clients in a coorporate environment (i.e. Apple) download external images automatically. Mailtrack seems to be confident, that only 5-10% of recipients aren't "trackable" [1], but I haven't seen a mail client that by default downloads external media in a while.

[0] https://mailtrack.io [1] https://mailtrack.io/hc/en-us/articles/360005941037-Why-is-M...

ridiculous_fish2• 3 years ago

See my peer comment: Apple Mail really does download external images automatically, on a server to hide your IP address. It then rewrites those emails to reference the cached images from the server. Every tracker pixel sent to a mail account with this feature will be counted as read, whether or not the user opens it.

withinrafael• 3 years ago

The Windows Mail app still does this by default. And I know folks that use it over the thick Outlook in corporate environments. (Yikes, I know.)

mytailorisrich• 3 years ago

What is "satisfaction"? Minor gripes are still "satisfaction", IMHO. At the price point I would expect someone not satisfied to return the phone.

hedgehog• 3 years ago

For what alternative? My 14 Pro is way too big, and the camera software is meh, but the lower models have really nerfed cameras and I'm too entangled in the Apple ecosystem to switch to Android even if there were better options. 80% satisfaction seems reasonable, most people seem to like the big phones, and 20% of people with some major complaint or other sounds plausible.

ars3• 3 years ago

I can't remember what it's called (Edit: Lizardman's Constant) but around 4 to 20% of people will give nonsense replies to any survey.

Like if you asked Are you Human? 4% (or more) of people will reply "No" just for fun, or because they don't read.

So you can try to adjust for this by asking a silly question, and if they reply that way ignore their entire survey, but that could lead to issues where they reply correctly on most of it, but false on the "fun" question.

fsociety999• 3 years ago

I have received Apple satisfaction surveys before and the options are basically this:

How satisfied are you with your new iPhone?

* Insanely satisfied

* Extremely satisfied

* Very very satisfied

* Quite satisfied

* Satisfied

* Mostly satisfied

* Somewhat satisfied

* Neither satisfied or dissatisfied

* Neutral

* No opinion

* Dissatisfied

I’m exaggerating a little to make my point, but I suspect that’s more or less how they get these insanely high scores. Also I’m not sure they send the satisfaction surveys to everyone. Probably you have to have a history of purchasing Apple products.

I don’t think they are manipulating the data itself as this article seems to suggest though. I think the survey methodology itself is skewed.

nowooski7• 3 years ago

Hey - I'm the post author.

Thank you for reading! I don't think they are manipulating the data, rather that they are surveying an unrepresentative population of tech enthusiasts, and presenting it as if representative of the entire customer base.

throw47474777j4• 3 years ago

They don’t present it as if it’s representative of the entire base, nor do they ever compare it to anyone else’s metrics, and your claim is based on information openly disclosed about how they create their metric.

The piece is just clickbait.

Your final paragraph is this decontextualized “Quote”:

“We’re not calling on the SEC to investigate these customer satisfaction claims, since they have more important cases to go after,” Golden said. “But extraordinary claims require extraordinary evidence, and we believe Apple should review the survey results they trumpet on every earnings call. If they truly think they are valid, they should publish their methodology, and if not, they should issue retractions.”

Which is inherently disingenuous in the way it is written. (We’re not threatening you..) but worse, it’s a quote from your own CEO.

The piece is in far worse faith than anything Apple is doing. Please take it down.

barbariangrunge2• 3 years ago

Give me an iPhone 5 that’s fast enough to run an os that has all the software bloat we have today. Then, stop adding more bloat to force us to buy newer hardware

asutekku2• 3 years ago

Ironically, in one of their other articles (also based on google reviews) they claim iPhone 5 is the least liked iPhone.

rottencupcakes• 3 years ago

Give me an Intel Celeron that's fast enough to run Windows XP and then stop changing anything.

Give me a 386 that's fast enough to run DOS and stop changing anything.

I'm surprised you've made it from mailing lists all the way to Hacker News.

wilg2• 3 years ago

Bloat is often just things you don't care about.

barbariangrunge2• 3 years ago

What are you talking about? Computers are thousands of times faster than they used to be, but performance is like half. Look up videos of people running old operating systems on old hardware, how everything happens instantly, even without ssds. The only improvement we've had is in games and web apps, and web apps are a mess

NineStarPoint• 3 years ago

So obviously 99% is impossibly high, but iPhones (and well made flagship phones in general) being some of the most popular things ever made wouldn’t be very surprising to me. People’s lives revolve around their phones. They’re probably people’s most used item after their bed. And for the most part they work really well. That 20% of people don’t like Thanksgiving and the Shawshank redemption isn’t surprising to me, and I don’t think that means the iPhone couldn’t have higher than 90% approval rating.

(Especially since a lot of people who don’t like iPhones would have gone to Androids at this point anyway, so current iPhone customers is a somewhat self selecting demographic)

jrockway2• 3 years ago

Yeah, I agree with that. The article asks:

If Apple’s claims of 99% satisfaction were true, it would mean the iPhone 14 was not only the most popular mass market consumer product of all time, but it would also probably make it the most popular anything of all time.

Is it not the most popular mass market consumer product of all time? People spend their lives on their phone these days, and I've never heard anyone say anything about their iPhone. I have one and it's fine. I'm satisfied. I have end-to-end encrypted backups, the battery lasts long enough, the screen is nice to look at, etc. If I like some sort of tech product, it's probably hard to find people that don't.

bredren• 3 years ago

I don’t think that means the iPhone couldn’t have higher than 90% approval rating.

I would be surprised if iPhone didn't have 90%+ customer satisfaction narrowly scoped to usage of the phone.

The product as hardware combined with software and supporting services are the best money can buy. Apple regularly pushes the smartphone category forward on fronts neglected by other companies.

That said, consumers seem to think more broadly about their experience than just the phone. For example an unexpected failure of an official or 3rd party case might influence overall satisfaction. The article mentions a 3% difference based on 5G availability, as another.

So without an explanation of the carve out, 99% does seem a bit too good to be true.

Apple should publish how these numbers are arrived at. If its a decent measurement it could help set an evolving standard for the category.

goldenjm2• 3 years ago

I'm quoted in the post. I agree, it is plausible that iPhones have a 90%+ satisfaction rate. Apple's claim isn't 90%, or 92% or even 95%, but 99%. I agree that 99% is impossibly high.

hiccuphippo• 3 years ago

My thinking is maybe iPhone 14 customers are satisfied by it, but I'm satisfied by my phone, not because it's a great phone but because any modern phone has the bare minimum I need to get satisfied by it.

Those people aren't using other phones so they wouldn't know if they'd be satisfied by them.

dktnj• 3 years ago

Trick in getting 99% satisfaction is asking you about it 30 nanoseconds after you bought it or before you even received it.

rickstanley• 3 years ago

Prime Day I bought my first iPhone, iPhone 14, with a good discount and cashback, I was between it and Zenfone 9. Not sure what to expect in general to be honest, but I expect a good battery life and updates, these were two factors that contributed to my final choice.

I'll sure miss sideloading certain apps, but that's the price for a walled garden and security (I guess?).

Let's see if I'm going to be one of the "99%"...

havelhovel• 3 years ago

Trader Joe’s fans will stand in line for hours for groceries but still give the store just an 84% satisfaction rating, according to a survey by the American Customer Satisfaction Institute.

It’s tough competition, but I think this is the most absurd sentence in the article. Nevermind the difference between a premium consumer good and a grocery store, my takeaway is that reported satisfaction could be much higher among TJ shoppers without the hours long waits.

tedunangst• 3 years ago

Satisfaction might even be higher among people waiting an hour in line; if they weren't happy they wouldn't wait.

_Nat_• 3 years ago

[This PDF](http://media.wix.com/ugd/bbfb8a_2ea66bffc09b40d099237bb83813... ) appears to provide information about their methodology -- noting it was linked from [this article](https://www.macrumors.com/2015/07/20/apple-watch-tops-iphone... ).

In short, it looks like potential-respondents first had to apply (e.g., in response to [this Facebook post](https://www.facebook.com/Wristly.co/posts/apple-watch-owner-... )), and then they pre-screened potential-respondents.. and it sounds like respondents may have had some sort of a weekly commitment?, as the document includes:

To finish, a big thank you to the 1,100+ strong members of the Wristly Inner Circle- we wouldn’t be able to learn so much so fast about the Apple Watch without your weekly contribution.

So it doesn't sound like a random sampling of Apple-product users.

Anyway, then apparently they ask if a respondent's satisfied with a product, giving 4 options: "Very Satisfied/Delighted", "Somewhat Satisfied","Neither Satisfied or Dissatisfied", and "Somewhat Dissatisfied".

Then, they add up the first 2 categories as their "key metric of customer satisfaction".

---

To note it, [this blog-post](https://daringfireball.net/linked/2018/04/20/iphone-x-custom... ) claims to quote "Ben Bajarin, on the result of a survey of iPhone X owners conducted last month" saying:

When it came to overall customer satisfaction, iPhone X owners in our study gave the product an overall 97% customer satisfaction. [...] Just to contrast that with the original Apple Watch research with Wristly I was involved in, 66% of Apple Watch owners indicated they were very satisfied with Apple Watch, a product which also ranked a 97% customer satisfaction number in the first Apple Watch study we did.

Point being that, while the above PDF appears to talk about the Apple Watch, it sounds like they're strongly implying that the customer-satisfaction figures for the iPhone were largely done in a similar manner -- and, at least in the case of the above-quoted speaker, by some of the same people.

user_named• 3 years ago

Apple is NPD manifested as a corporation. Everything they do is self-gratifying, so of course also this.

"All-new"... Pffft.

Insanity• 3 years ago

NPD?

spywaregorilla• 3 years ago

somewhat satisfied is an amusing answer. it implies you're not actually satisfied with the product.

nowooski7• 3 years ago

Hey - I'm the post author. Thank you sharing these details on the Watch survey.

meepmorp• 3 years ago

Hey - I'm the post author. Thank you sharing these details on the Watch survey. I wonder if 451 Research is doing something similar with their iPhone survey.

Isn't that something you should've researched before writing the post? Methodology seems like an important factor here.

autoexec• 3 years ago

There are people who practically worship apple. A new iphone could ship broken and a certain percentage of buyers would praise it as being brilliant. They'll never be unsatisfied with an apple product. There's another group whose satisfaction has nothing to do with the phone itself and everything to do with knowing that it was new and highly expensive and out of reach for many. As long as they can pull it out of their pocket and show it off they'll be happy.

Adding to that, people already have a tendency to convince themselves after the fact that an expensive purchase was a good deal, and that companies pay for fake reviews and I'm not surprised iphone reviews seem inflated.

nowooski7• 3 years ago

I do think there is something to the idea that people want to satisfied by something they spend a lot of money on. Though, 99% is a bit much.

YetAnotherNick• 3 years ago

Apple has 14 days return policy in many countries. Wouldn't non return count as satisfaction? Does Apple announce that number?

loeg• 3 years ago

The original title, "Apple claims 99% iPhone 14 satisfaction. Is that possible? (No). We investigate" (which this was submitted under originally) is less click-baity than the current version: "Is Apple making implausible iPhone satisfaction claims?"

addisonl• 3 years ago

These both seem the same to me as far as click bait goes…

lolinder• 3 years ago

I pinged dang to ask him to change the title to match the title of the actual article, because it was obvious from the rate of upvotes and the quality of comments that the title as submitted was inflamatory.

I can't measure clicks, but the pace of upvotes on this submission slowed dramatically after dang changed it: it had reached ~50 upvotes in about half an hour (~100 upvotes an hour), and now it's been here for another hour with only ~30 more upvotes (~15 upvotes an hour).

tgv• 3 years ago

The advantage is that the answer to the title question is yes for once.

wereHamster• 3 years ago

Is there a legal limit how much you can fake the numbers before you are held accountable? Seems like Apple is pulling a lot of numbers out of their rear end. See also LTT's recent video about Apple quite clearly faking Mac Studio performance numbers: https://www.youtube.com/watch?v=buLyy7x2dcQ

yuters• 3 years ago

I'm curious about the stat of the email being opened 73 times. I'm wondering if someone at apple who has access to that contact email address has the option "Protect Mail Activity" turned off on their Apple Mail client or if this is a bug.

BSEdlMMldESB• 3 years ago

maybe they're also considering the satisfaction of the slate of hardware?

/joking

bell-cot• 3 years ago

At what age is it reasonable to assume that a modern western consumer understands that Marketing routinely uses some torturously contorted standards for "truth"?

Family lore: My kid brother was ~5 years old when he bit into a spoonful of a breakfast cereal that he had asked for after having seen it advertised - and unhappily blurted out "it tasted good on TV!".

Is Apple making implausible iPhone satisfaction claims?

Related Stories