Note 3’s benchmarking “adjustments” inflate scores by up to 20%

http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20

by AndrewDucker • 13 years ago

103 63 13 years ago

MBCook• 13 years ago

For clarification, Note 3 means Samsung Galaxy Note 3, not the name of a company that produces a benchmark (which was my first thought).

Samsung is at it again. Doesn't matter if it's already a fantastic device, they still pull this kind of stuff.

fsniper2• 13 years ago

I can't understand the issue here. If you are using a benchmarking app or a game that's heavy on the cpu, these same optimizations would kick in. So you will cry for the speed of the device while crying for the battery juice draining.

What is the issue here?

JonFish85• 13 years ago

Would they actually, though? If these optimizations would kick in, why does Samsung feel the need to force the optimizations to kick in for specific apps, rather than via CPU usage like any other game/app?

epo• 13 years ago

Because they are incapable of producing competitive products (their phone market is people who can't afford anything better or who are motivated by fanboyish prejudice) and because their corporate culture appears to emphasise cheating, deception and dishonesty above all other values.

I mean really, even if you thought Android was a better solution why would you buy Samsung?

tekacs• 13 years ago

It appears, from the extracted code, to be the case that a 'special' (read: hardcoded) set of optimisations is being enabled solely for benchmarking apps.

Whilst 'smart' behaviour might enable the phone to switch on some such functionality when the SoC/CPU is put under heavy load (by something such as a game), the effect of such automated switching is unlikely to have anything like the effect of a hardcoded force-to-high-power-mode as shown here.

wuliwong3• 13 years ago

And you know this based on what? "hardcoded" != "faster"

It just guarantees that it will happen in that case. Unless you have some reason to believe that this mode is unattainable under any real world circumstance, I think the original point still has some credence.

sp3322• 13 years ago

It's not about CPU-heavy apps. The cores change their clock speeds as soon as you open the benchmarking app, before you even run the benchmark.

fsniper2• 13 years ago

If that's the case, Samsung is doing direct manipulation on benchmarks and this is fraud.

macspoofing7• 13 years ago

What is the issue here?

The issue is that these 'optimizations' don't actually kick-in for non-benchmarking apps.

darkarmani• 13 years ago

It's only triggered if it is a benchmarking app. Did you read the article at all? They look at the name of the app running and if it matches the name of a benchmarking app, they shift into a mode that only exists for benchmarking apps.

devx• 13 years ago

I think it's a very misleading tactic, because then many customers buy based on those benchmarks.

Ironically, I believe these smartphone OEM's are a lot less bad than PC OEM's/chip makers. I hate how Intel, Nvidia and AMD all "win benchmarks" or show numbers much higher than the practical limit of those chips.

All of them seem to have a "base clock speed", and then a "turbo speed" that goes even up to twice as high. What do you think gets shown in the benchmarks? The base clock speed, that you'll be using most of the time? Of course not. It's the "turbo" burst speed that will be calculated by the benchmarks.

This ends up being very misleading because for example I bought a laptop with an Nvidia GPU that was supposedly 3x faster than the integrated Intel one. But after about 30 minutes of playing a game, it will invariably get throttled to about half of that speed, and then it will only be 50 percent faster than the integrated one.

Intel is even worse than this, because they're becoming more desperate about the ARM competition, they start to become increasingly more misleading to make the appearance that they're "closing the gap" with ARM much faster than they really are.

Before I start, I want to say that I use a Core i7 in my laptop, but I just hate shameless misleading of consumers, whether it's from Intel, Nvidia, AMD or Samsung.

There are 3 "misleading strategies" that Intel has used lately:

1) Turbo-boost, of course. Their new Atom chips are actually 1.3-1.5 Ghz, but they market them as 2-2.4 Ghz chips, a speed that also helps them do very well in CPU benchmarks (they're still about 2x behind any other ARM GPU, but I think they'll start using turbo speed for the Atom GPU's in the next generation, too, again to make it "seem" like they caught up - except not really).

2) Launching the most powerful version they have first, even if it's not going to be used in most of the devices the consumers want, but this way the press is all over it, and then the consumers remember that "brand" that was very powerful, even if it's not what they're getting in their device.

Example: Haswell's Iris Pro. It was the first to be sent to sites and benchmark, and it showed much bigger GPU performance than before in laptops. The only problem? It's probably never going to be in a laptop. It was meant for PC's mostly, and again as a "halo product", to make people think that "Haswell is very powerful". In fact most laptops have only seen about 20 percent increase in GPU performance with Haswell, compared to IVB.

Second example: They launched Bay Trail first, the "tablet version", to make people think "how powerful Atom is" and that it's "good for smartphones", and then they will release Merrified for smartphones, half a year later (which would already put them behind competition), and probably even weaker than Bay Trail performance wise. But most consumers will just remember that Bay Trail was great in benchmarks (see 1, also), even that's not what they're getting.

3) They're starting to push the "SDP" measuring, so tech websites can start writing about and promoting their chips as "5W chips" or whatever, and hope that even if they're now using the SDP name after the power consumption number, they will eventually forget to mention it ("Intel's 5W chips", etc).

SDP is nothing but an absolute shameless scam from Intel. They want to have their cake and eat it, too: a) declare super high performance with the burst speed that it's not practical at all for too long, and b) declare super low-power with SDP, even if the chips themselves can get a lot hotter than Intel is implying with SDP, especially if they start using the Turbo-Boost speed.

These guys explain it a lot better than I could:

http://semiaccurate.com/2013/07/25/intel-keeps-up-the-unethi...

http://www.theverge.com/2013/1/9/3856050/intel-candid-explai...

If I can't use my laptop at the advertised clock speeds for more than 10-30 minutes continuously, then it's nothing but a scam, meant to fool consumers into thinking their chips are either a lot more powerful or a lot more efficient than they really are.

Lagged2Death3• 13 years ago

If your laptop doesn't have the cooling to support the maximum continuous performance of a CPU or GPU, how is that the chip-maker's fault? The chips offer you as much speed as possible under the power/cooling constraints - isn't that exactly what you want, as a user?

I think the benchmark manipulation by Samsung is in a different class; it configures the system in an impractical mode that actually isn't available to the end user at all.

In contrast, high-consumption turbo modes in laptop CPUs and GPUs really do benefit the end user, even if the benefit isn't everything you were hoping for.

macspoofing7• 13 years ago

Yeah. I don't really care what the manufacturers show in their marketing material. As far as I'm concerned they are all biased and their numbers should be taken with a grain of salt. There is an ethical problem, however, if you try to game benchmarks used by independent agents. At the very least you should be transparent about what you're doing.

makomk• 13 years ago

I think you missed the part where in order to appear competitive with ARM, Intel are advertising their chips as having lower total power consumption and lower cooling requirements than are actually required to run them at full performance. They're then configuring them to throttle back in order to meet the advertised thermal figures.

hmottestad• 13 years ago

Samsung also has a screen brightness booster that only works when the brightness is being measured. [1]

It also has a battery life extender for apps that measure battery life [2]

Don't forget the wifi range extender that boosts up the wifi decibel [3]

They also say their phones can use 16TB memory cards [4]

And don't forget that their phone is the prettiest [5]

1-4: I'm just making things up. 5: When compared to a naked mole rat.

So next time Samsung says anything about anything. Just remember that they are willing to stretch the truth more than the average company.

simias• 13 years ago

I'm sorry but I don't understand your point at all. Are you saying TFA makes up its claims?

jfb• 13 years ago

If people buy on easily gameable microbenchmarks, we should expect companies to game said microbenchmarks.

wuliwong3• 13 years ago

Well said @jfb.

mercurial• 13 years ago

This is pathetic. And not even necessary:

The ironic thing is, even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn't appear to have been necessary in the first place.

The Samsung phone division upper management and marketing team don't seem burdened by an overabundance of neither ethics nor common sense. Or maybe I'm just being naive.

radley• 13 years ago

There's another possible explanation: some Samsung head demanded an extra 20% in unobtainable improvements (maybe offered bonuses?), so the engineers found a quick and dirty way to make it happen.

bentcorner• 13 years ago

Reminds me of this: http://dilbert.com/strips/comic/1995-11-13/

Kronopath• 13 years ago

Reminds me of how some old ATI graphics drivers detect whether you were running Quake 3, and turn down the graphics quality for the sake of better benchmarks.

Renaming quake3.exe to quack3.exe would fool the detectors and skip these "optimizations".[1]

[1] http://www.hardocp.com/article/2001/10/23/optimizing_or_chea...

dman• 13 years ago

Nvidia has been caught doing exactly this multiple times. Same with intel compilers. Same with car makers who optimize for EPA testing conditions. Given a benchmark people will "optimize".

mikro2nd• 13 years ago

Or... "Tell me how you measure me, and I'll tell you how I behave"

diminish• 13 years ago

Moreover. Weren't Oracle, Sybase, Ms SQL Server doing same type of optimizations just for benchmarking purposes since their beginnings?

Pxtl2• 13 years ago

To be fair, Quake 3 fans are known for prioritizing frames-per-second beyond all other considerations. One could argue that tweaking the graphical quality for Q3 to encourage high framerates is just giving customers what they want.

moogleii2• 13 years ago

One could, but that'd be a stretch. It should be a discrete option left up to the user. If they're tweaking that much, they'll know or seek the knowledge on how to activate it themselves.

roc• 13 years ago

Except this was done against the explicit graphics settings the customers had set and only for a handful of games that were regularly used as benchmarks.

Customers who actually want framerate-beyond-all-else don't enable/would disable those settings in the first place.

So if it were "giving the customer what they want" they'd have respected the customer's settings.

What it actually was, was a simple recognition that people used "framerate at max quality" as a proxy for net card performance and an attempt to "game" such measures by not actually running at max quality, for those games that were common benchmarks.

macspoofing7• 13 years ago

To be fair ...

Do whatever the fuck you want, but be upfront and transparent about it. If you aren't, you're shady and sleazy.

yapcguy2• 13 years ago

Devil's advocate here.

"the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off."

Good! I want to know what the maximum performance is from a benchmarking app.

I don't know why regulars apps are unable to get the Note 3 to use all 4 cores. Perhaps they're not multi-threaded or they haven't enabled the right settings? Surely 3D games will access all cores?

At the end of the day, Samsung wants the benchmarking to show the theoretical maximum. Other manufacturers may not be too bothered and are happy to show benchmarking of normal usage. It's like reading the specs on a car, you're never going to drive at 160 mph, but at least you know you could.

Tyrannosaurs2• 13 years ago

Because the OS is trying to balance performance and battery life.

Saying I want to see the maximum performance is fine but they should also be forced to show how that impacts other elements of system performance.

Personally I think it would be good to see two sets of benchmark typical and maximum, but the most important thing is that whatever scores are being shown are clear and honest and I think it's hard to say that these are.

mjs• 13 years ago

I don't expect a benchmarking app to be throttled by the OS under any circumstances--ideally the benchmarking app would run on the raw hardware, with no second-guessing by the OS about whether it should be throttled because the battery is down to 10%, or because some other process would like some network or CPU, etc.

The way this probably should work is that the benchmarking apps themselves have a flag telling the OS they want all cores at full power all of the time, instead of having to hardcode this list into the OS. (Perhaps this could be one of the permissions you have to accept when installing the app.)

gibwell6• 13 years ago

In this case it's not the theoretical maximum. It isn't available to regular apps because the OS and CPU don't allow it, for good reason - to conserve power. The reason benchmarking apps can do it is that Samsung added code to specifically detect them.

Your car spec analogy is a good one. If the car maker removed the seats and carpet as well as inessential engine components before testing, that would be the equivalent of what Samsung have done here.

It would also be fraud.

macspoofing7• 13 years ago

I think the key idea here is transparency. There's nothing wrong with showing theoretical maximums, but be transparent about it!! Otherwise it looks like you're trying to get away with something .

lostlogin• 13 years ago

There is very little that is transparent about Samsung and their relatively recent history is pretty seedy. High level corruption, bribery etc. That the culture of deception flows down the company into actual products doesn't surprise me.

rplacd• 13 years ago

At the end of the day, Samsung wants the benchmarking to show the theoretical maximum. Other manufacturers may not be too bothered and are happy to show benchmarking of normal usage. It's like reading the specs on a car, you're never going to drive at 160 mph, but at least you know you could.

And they should be vilified for re-interpreting preconditions that would usually guarantee valid relative comparisons and are used like so. ("Theoretical maximum" is a problematic end to work towards, though - why shouldn't clock-speed specifications suffice, then? I presume you're talking about stock speeds sans the software stack - but cranking up frequency scaling isn't much of a meaningful hop and skip away from just running the benchmark.)

umsm• 13 years ago

Your comparison with a sports car is very appropriate.

Many cars have "race" modes which allow the computer to use the maximum power an engine has while not trying to force it into an "efficient" mode.

99% of the time you want your car to be as efficient as possible while others you want to see what it's capable of.

The Nissan GT-R does this automatically when you're located on a track:

http://www.engadget.com/2007/12/22/nissan-gt-r-disables-spee...

gibwell6• 13 years ago

This is nothing like that. The note doesn't have two modes that the user or the software can choose from. The adjustments are made secretly, and only when specifically named benchmarks are run.

kalleboo• 13 years ago

It's like reading the specs on a car, you're never going to drive at 160 mph, but at least you know you could.

It's like testdriving your car on the manufacturer's track, seeing it go to 160 mph, then taking the car to race day and you hit a 90 mph limiter they built into the car for fuel efficiency.

Ironically, there's a Nissan car that does enable and disable the limiter based on the GPS detecting it being on a race track or not. But they're transparent about it.

bonzoesc• 13 years ago

Ironically, there's a Nissan car that does enable and disable the limiter based on the GPS detecting it being on a race track or not. But they're transparent about it.

It's the GT-R, and it's only on the Japanese version: [trigger warning: gadget blog] http://www.engadget.com/2007/12/22/nissan-gt-r-disables-spee...

venomsnake• 13 years ago

Which is great. And I totally agree with them - if they had disclosed it at least to the tech community.

After all when Nvidia 6xx and 7xx series self overclock when there is room in the power envelope available this was disclosed and taken into account by the reviewers.

moogleii2• 13 years ago

I don't know why regulars apps are unable to get the Note 3 to use all 4 cores. Perhaps they're not multi-threaded or they haven't enabled the right settings? Surely 3D games will access all cores?

That doesn't explain why the exact same app performed worse (or normal, rather) when renamed .

The customer is buying a complete product, not a cpu. Maximum cpu performance without context is not meaningful. A 500 horsepower engine is pretty awesome, until you drop it into a cargo freighter. Then it's worthless.

Pxtl2• 13 years ago

... could somebody please explain to Samsung's marketing department that they're a world-leader and not a sketchy fly-by-night knock-off company?

macspoofing7• 13 years ago

You are what you are. If you read anything about the way Samsung is run (especially in South Korea), this is par for course.

pwthornton• 13 years ago

Samsung's entire business model is being a sketchy fly-by-night knock-off company. First it was Sony and other Japanese electronics makers and now it is Apple. And I don't think Samsung would really deny it. They're pretty good at what they do.

Somewhere along the way they became a huge global company that makes a lot of money, but they forgot that they need to start acting like one.

They don't need to fake benchmarks or anything like this. They are the main event when it comes to Android.

Samsung is who we thought they were: http://www.youtube.com/watch?v=kw5UWqpcsDs

gibwell6• 13 years ago

Why do they need to act any different? Their current tactics are working very well.

Nursie• 13 years ago

Don't care. Mobile processors in all of the high-end handsets long ago surpassed my needs. As far as I'm concerned they are all now awesome and we're living in the future.

gibwell6• 13 years ago

A good reason to find deliberate deception ok.

beauzero• 13 years ago

...and they thought they could get away with it.

Geee• 13 years ago

Mobile devices are energy/power-limited (battery life / chip temperature) and you always have to optimize between those and performance. Theoretically you can add performance as much as you'd like but with a huge power draw. Basically, all these benchmarks are meaningless without knowing the power draw.

YOSPOS• 13 years ago

Shocker.

NatW• 13 years ago

So, Samsung, the dominant (#1 manufacturer of mobile phones (Q2 2013 sales: 107 million, market share: 24.7%)[1]) has been caught unambiguously and intentionally deceiving the public on their product's performance. Approval of cheating marks a disgusting corporate culture.

sources: [1] http://www.rediff.com/business/slide-show/slide-show-1-tech-...

anonymous• 13 years ago

Question: why don't benchmark sites then install a third-party android distro and benchmark with that?

Note 3’s benchmarking “adjustments” inflate scores by up to 20%

Related Stories