How Samsung inflated its performance scores

http://www.theguardian.com/technology/2013/oct/13/samsung-benchmarking-apps

by antimora • 13 years ago

52 30 13 years ago

bsullivan012• 13 years ago

Samsung is one shady company. They get away with murder in South Korea, given that they make up a huge chunk of SK's GDP.

http://en.wikipedia.org/wiki/Lee_Kun-hee#Samsung_scandal

They do make decent electronics, but I wouldn't trust anything they say.

NatW3• 13 years ago

Concerning that Samsung operates like this even though they are the dominant market leader - with a plurality of the market share for mobile devices, 30.4% of the global market Q2, 2013 [1]. 1 http://bgr.com/2013/07/26/smartphone-market-share-q2-2013-sa...

snogglethorpe• 13 years ago

I think Samsung recognizes that their position as a "dominant market leader" is due more to luck than anything else, and that their products really aren't very different from those of their main competitors.

They have good reason to be nervous, maybe even more so than their competitors: Samsung gains a lot from the public perception that they're "in the lead", and thus have more to lose should that perception change...

belorn2• 13 years ago

So the chairman was found guilty of financial wrongdoing and tax evasion.

Tax evasion is indeed serious business, but murder? I guess I would like to see several large business CEO's charged with facilitating murder by tax evasion. Would make for some interesting times, and likely tax cuts for the general public.

ceejayoz• 13 years ago

To "get away with murder" is a very common way of saying someone is successfully avoiding the consequences of their actions.

epo• 13 years ago

Well, their washing machines are OK. Anything computer or phone related is cheap rubbish for those with no money or with low standards.

rsanders• 13 years ago

I've been an iPhone user since the first one shipped, and just changed to a Galaxy Note 3. It's neither cheap nor rubbish. But I do feel a bit uneasy about supporting them in light of their history of "extreme homage" in design. I'm not at all surprised to hear about them acting shady in this way.

steve19• 13 years ago

The funny thing is that despite being caught out doing this a while ago with the S4, they have no choice but to continue doing it regardless how embarrassing it becomes. If they don't, their next generation devices may score only slightly more (or less than) than the current gen devices.

They painted themselves into a corner from which there is no escape. And they only have themselves to blame.

bsullivan012• 13 years ago

But it's not like the market punished them at all. They are still selling lots of them.

Apple is the only that held them accountable in court, at least a little it. These b*stards are shameless http://cdn.mactrast.com/wp-content/uploads/2011/09/Samsung-A...

nodata2• 13 years ago

Your proof that "Samsung are bastards" is because they have a connector that is either a PDMI connector, or something that looks suspiciously similar?

masklinn3• 13 years ago

The phone giant has proved that benchmarking apps are vulnerable

The only thing they have proved is their willingness to game benchmarks. ATI and NVidia were gaming 3DMark 10 years ago.

Also interesting to note, though Samsung is definitely more than willing to game benchmark despite their protests (and looks like the worst offender by a pretty long shot), Anandtech found out they're far from the only company doing so: http://anandtech.com/show/7384/state-of-cheating-in-android-...

Edit: turns out this was mentioned in the article, but the way it's presented, what the fuck Guardian?

Anandtech, a site where you could get the (wrong) impression that benchmarks are the only reason to own a gadget

belorn2• 13 years ago

Perfectly correct. The only way to produce real benchmarks is to run useful software that are identical on two different hardware setups. Sadly because of to DRM, this is impossible to do on lock down devices.

You could run two similar applications (say angry avialaes), but the only thing you would test is that device implementation of said game.

NatW3• 13 years ago

Anandtech's Article:

Cheaters: Samsung, LG, Asus, HTC

non-cheaters: Motorola, Google, Nvidia, Apple

See any patterns?

nyrina• 13 years ago

Not really, care to enlighten me?

czr80• 13 years ago

Let me guess: Suppliers of commodity hardware have an incentive to make that hardware look as attractive as possible, suppliers competing on other dimensions lie about other things ("We're open!, "It's magical")

Or did you mean something else?

slacka• 13 years ago

The CPU/GPUs keep getting faster but this idiotic PPI race is forcing manufactures to cheat so they can appear faster than the previous generation. On a 5" display anything over 720p is overkill.

devx• 13 years ago

If Apple moves to a 5" screen or something close next year, I'm sure they'll double up the resolution again to 2272x1280. That could force Android OEM's to think they need to go to 2560x1440 now, to be higher than Apple, but I agree that would be silly.

It could help devices like Oculus Rift, though, and it could also lead to 4k displays on 10-12" tablets and notebooks faster than otherwise ("only" 2x the pixels from the current resolutions for 10" tablets).

masklinn3• 13 years ago

A number of benchmarks have an "off-screen" mode with fixed-resolution rendering specifically to avoid native resolution artefacts. That provides both the absolute hardware power in comparison to other devices and the device's "feeling", how its hardware meshes together (aka if it has enough power to drive its own screen). The second one is arguably the more important one wrt user experience and the ability to actually use the device.

GLBenchmark has both onscreen and offscreen modes, and Anandtech posts both every time.

sharpneli• 13 years ago

"We've already started work on making sure that all future versions of benchmarks we get will come with unique package names." - Anandtech

After that Samsung will just figure out some other identifying feature to recognize the benchmarks. Naturally we might go overboard and do random obfuscation on everything, that might solve the issue for the journalists who use them one off.

However for benchmarks that are published in the Google play store that is impossible, so there is no real way for benchmark developers to 'immunize' their products for this sort of approach.

Disclaimer: I am employed in developing Mobile benchmarks, and our product is something they cheat on. Avoiding devices that cheat is very much in our interest.

masklinn3• 13 years ago

After that Samsung will just figure out some other identifying feature to recognize the benchmarks.

Yep, reminds me of 3DMark2003 when NVidia and ATi "optimized" their drivers by disabling, altering or swapping the benchmark's shaders for their own or less complex versions

hclee• 13 years ago

People still buys. In business sense, it is funny. They cheated and impressed as much as they can and made really large volume of sale. Media reporting like this does not seem to make much difference. Other than few hackers/engineers/programmers, majority of consumers don't give a thing on benchmarks. Too bad.

badman_ting• 13 years ago

All I can think is, imagine if Apple did this.

How Samsung inflated its performance scores

Related Stories