Mozilla finds CPU bug (bad store forwarding) in Samsung Galaxy S20

https://bugzilla.mozilla.org/show_bug.cgi?id=1833315

by kens • 3 years ago

246 68 3 years ago

bugzilla.mozilla.org

LispSporks22• 3 years ago

The only thing better than being able to blame the compiler for something, is being able to blame the CPU

print_goto_ten• 3 years ago

Great tshirt

peheje• 3 years ago

Saving that quote.

amelius2• 3 years ago

But how are you going to fix it then?

wongarsu• 3 years ago

Change your code to not trigger the bug, and leave that workaround there for the next decade or so until the buggy CPU generation isn't relevant anymore.

Or hope for microcode updates to fix the bug at the CPU level.

JonChesterfield• 3 years ago

You change the compiler to carefully step around the stuff that doesn't work in the hardware.

pajko• 3 years ago

Compilers can be patched to automatically do some masking in the code generation step after every LDRB instruction in this case, when compiling for that specific CPU. For example, the Blackfin DSPs have many similar bugs which are fixed this way. Some of which the fixes done automatically, while others could be disabled for performance critical code which has been implemented while keeping the various issues in mind, like -mcsync-anomaly and -mspecld-anomaly.

Certain MIPS CPUs also have incorrectly implemented instructions and the -mfix-* compiler parameters could be used to enable workarounds.

JIT compilers can do the same.

jeffbee• 3 years ago

This is where the vocal minority of HN posters who are in a snit about telemetry are wrong. This is the kind of issue you can discover at global scale, but only if you collect the data.

n4r9• 3 years ago

It looks like this could have been discovered by simply having (anonymised) crash report telemetry? I don't know about others, but that sort of telemetry doesn't sound so bad.

Nextgrid3• 3 years ago

It's very hard to have useful anonymized telemetry. In this case, you actually need at least a partial memory or register dump to understand what's going on, but those can contain sensitive data that the failing code was processing.

Telemetry doesn't have to be 100% anonymized - it can (and in some cases needs) to contain PII, just needs to be collected respectfully and transparently. Ask the user before sending something, and let the user review it. The user can then make their own decision.

mixedbit• 3 years ago

Such crash data can be collected with an old fashion 'Firefox has crashed, send a problem report to Mozilla [OK] [Cancel]' dialog, which is much less privacy-intrusive than telemetry.

ksenzee• 3 years ago

Right, and Firefox for Android does in fact collect its crash data like this.

vardump• 3 years ago

Indeed. Telemetry is often used for good purposes, like improving features that cause trouble to end users and to fix crashes.

I'm not saying bad actors don't exist, just that telemetry is often used for good, with no nefarious goals.

burkaman2• 3 years ago

Personally I don't have a problem with automatically sending the type of telemetry that makes detecting this possible, but Mozilla could also discover this issue by in-house testing on a wide variety of devices, and/or prompting users to submit telemetry after a crash.

It's not feasible for a smaller developer, but it wouldn't be unreasonable for Mozilla to have a big lab bench with like 100 different devices on it that are just constantly running a script to start Firefox and load a suite of test sites.

sjsdaiuasgdia• 3 years ago

I don't think you could really approach the kind of coverage automatic crash telemetry can give with a bunch of test devices. You will absolutely find a non-zero number of issues with the test devices, but it's not really the same.

Without substantially increasing the cost and work to maintain the test pool, you'll be testing exactly one combination of OS version/patch level/settings, one internet connection medium/carrier, one set of browser config options, etc for each device in the pool.

The investigation into any fault isolated to a single device in the test pool has to start with suspicion of the health of the specific device. The test S20 crashing doesn't tell you that every S20 will crash in the same way, it could be an isolated local fault. Yes you can throw more devices but that's further increasing the cost and effort of maintaining the test pool.

jfk13• 3 years ago

It's not remotely feasible to test more than a tiny fraction of the variety of devices, configurations, and sites that a product like Firefox is going to encounter in the real world.

toast02• 3 years ago

User volunteered crash telemetry is better than nothing, but automatic crash telemetry is going to have a more representative sample. A compromise would be to grab just a stack trace and phone model / firmware version etc automatically, and a full dump with user affirmative consent; but if that's not compatible with your privacy stance, asking everybody for everything is reasonable but limiting.

Android has an immense variety of devices. Without a representative sample, you wouldn't know which devices to buy. Regional variants sometimes have a huge difference and sometimes not; this bug seems to be tied to non-US Samsung flagships, which you might not buy if you're US based and don't know what people are using.

rollcat• 3 years ago

We've had crash reports for about as long as we've had computer programs. The problem is not in the mechanism itself, but in its abuse for (overwhelmingly, non-consensual) behavioral profiling or spying.

Even when not abused to violate our privacy, the end result is often highly questionable. In place of good design, common sense, and user studies, companies try to extrapolate trends/patterns from data where there are none, resulting in mediocre decisions at best, and self-feeding cycles of devolution. ("Users are clicking this button a lot, let's make it larger" - users are actually clicking the button by accident.)

SanderNL2• 3 years ago

This minor issue that also is not Firefox’s fault is not enough to sacrifice anything let alone privacy.

dralley• 3 years ago

Unless it doesn't actually sacrifice any privacy.

Nextgrid3• 3 years ago

Even if telemetry is useful (this is a rare example - the vast majority of telemetry is either useless or actively works against the users), it's still a breach of privacy and confidentiality (crash dumps - which are necessary to investigate/reproduce such bugs - can contain highly confidential data including passwords or session cookies).

but only if you collect the data.

If the problem is impacting the user and they want it fixed they'll happily tell you. This bug (which causes crashes) would've been investigated just as well by asking users "Firefox has crashed - send bug report?", while giving the user the choice to decline if they had sensitive data in the browser's state.

Obviously nobody in their right mind should opt-in to generic telemetry because all the "product improvements" excuses never panned out - in the last decade software has only become worse, more annoying, and less feature-full, so people are making the rational decision and are not opting into something that's not actually benefiting them. But opt-in telemetry in specific cases where the user actually has a problem like this occurrence can work just fine.

ptx• 3 years ago

You're wrong about them being wrong. Nobody is arguing that telemetry could not possibly have any upsides (primarily for the developer), only that the downsides (primarily for users) make the tradeoff unacceptable.

dspillett• 3 years ago

Your point would be correct if that was the only telemetry being collected. The pile of stuff most things are sending back home contains much that would not be useful here but is potentially privacy breaking (or just purely for marketing purposes with no intention of being useful to the user). Similarly, in either case telemetry is fine if it is opt-in, and if extra is collected the opt-in is repeated (i.e. no stalking via the back door by changing what is collected and sent home after consent is given).

temac• 3 years ago

Bullshit, CPU bugs have been found before wide usage of telemetry and will continue to be found in various context, number of them not doing it even if they could.

Telemetry is probably just one more excuse to ship crap in the first place.

nottorp• 3 years ago

but only if you collect the data

You spelled 'send crash dumps with the user's permission' wrong.

msla• 3 years ago

Normal software testing (remember that?) can catch bugs without invasions of privacy.

howinteresting• 3 years ago

Have you ever worked on a large-scale software product used by millions of people across thousands of kinds of devices? Telemetry is always going to be useful.

botanical• 3 years ago

I allow telemetry to Mozilla but no one else

garganzol• 3 years ago

When I was a kid, I always suspected a compiler or CPU bug when my program did not behave as expected. I heard similar stories on more than several occasions from other people, so being naïve must be a common thing among us software engineers when we make our first steps. Kudos to Mozilla, their finding is a real CPU bug.

pengaru2• 3 years ago

For ages in *nixland the phrase "select is not broken" has been an aphorism for such situations. I'm not sure of its origin, but it's referenced in The Pragmatic Programmer:

  >  Tip #33, pg. 95:
  >
  > “select” Isn't Broken
  >
  > It is rare to find a bug in the OS or the compiler, or even a third-party 
  > product or library. The bug is most likely in the application.

AshamedCaptain2• 3 years ago

I really don't know when this became "common knowledge". Most OSes and compilers were utter crap not more than 10-20 years ago. We would spend afternoons ranting about MSVC++ bugs. gcc was a joke, and even if quickly improved, does nobody really remember that by the 2010's gcc was so buggy -O3 was barely usable? No Gentoo users around?

toast02• 3 years ago

Yeah, I don't know where that comes from. Everything is broken.

If you think select is broken, read the docs and make sure you're using it correctly. If you still think select isn't doing what the docs say, try to reduce your actual use into a minimal test case and see. Go look at the source for select on your OS. Search the internet and see if anyone else ran into the same thing (but be careful, because there's lots of bad leads out there).

syncsynchalt• 3 years ago

Ironically we later found that `select` is broken by design. `poll` was created as an improvement, and syscalls like `epoll` later refined it further.

mastax• 3 years ago

If you step into microcontroller land you run into hardware bugs all the time, at least if you have to write any low-level code. My very first major project I ran into at least 3 silicon bugs which wasted weeks of my life. Rookie mistake: check the errata. Though one of the bugs wasn't in the errata.

0xffff2• 3 years ago

It's a good guideline when you're early on in your programming career, but I've been surprised by how many compiler bugs I have independently re-discovered in my career. One of the curses of my job is that I work with ancient compilers. About half a dozen times I have stumbled on a bug, spent the better part of a day debugging my code, only to find that I'm being bitten by a documented GCC bug that was fixed a decade ago.

IshKebab• 3 years ago

One of the big reasons to keep software up to date IMO.

dkarras• 3 years ago

yeah same, from almost 30 years ago. Interesting shift. I distinctly remember having the instinct that it must be the computer that must be wrong, and what I wrote was obviously correct. Thanks for reminding me of that. With some experience, you figure out that you are wrong %99.99 of the time and after that loop, the thought never crosses your mind. I've never had to deal with hardware bugs myself, but had my share of compiler and platform bugs over the years.

londons_explore• 3 years ago

It's not a real CPU bug unless you can write a 20 line program in C/assembly and run it to demonstrate the problem.

Then, when you've done that, send it to samsung so they patch it in their CPU design. They may even be able to patch it in their devices in the field (it's common for CPU designs to have special hardware whose purpose is runtime detecting certain trigger conditions and running replacement code to avoid hardware bugs discovered after the CPU was produced).

They can also issue an errata which might let compiler authors modify their code generators to avoid any pattern that triggers the bug, therefore avoiding other projects being impacted.

bpye2• 3 years ago

There are a surprising (or unsurprising depending on your perspective) number of hardware bugs out there. There are published errata that I had hit before they were documented. It’s not always very much fun - and leads to a lot of head scratching.

jakub_g• 3 years ago

A few years ago I was doing Android native development. Back then Samsung as well as Huawei (and other cheap Chinese vendors) have been known to be doing weird things to the operating system and in Google Play console there'd often be wild device-specific crashes coming from those vendored Androids.

I never loved Samsung devices but since then I avoid them even more.

satiric• 3 years ago

This site has a list of phones with the same CPU as the Samsung Galaxy S20 they mentioned. https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865

I wonder if it happens to them too?

SketchySeaBeast• 3 years ago

Given that it's only effecting S20's I'd be surprised if it was the snapdragon 865 as that's in a ton of phones as you point out. I'd expect it to be the Exynos 990, which is Samsung's custom chip for every region but North America and was only used in the S20 series.

scrlk2• 3 years ago

Wouldn't be surprised if it was an Exynos bug either. Samsung LSI haven't produced a decent flagship Exynos SOC for multiple generations now. Always behind Snapdragon on overall balance of performance (CPU/GPU/radios), heat and battery life.

I refuse to buy any phone with an Exynos SOC ever since I got screwed with a poorly tuned Exynos 9810 in my S9. Andrei Frumusanu did a great two part article on turning the 9810 when he used to write for Anandtech:

https://www.anandtech.com/show/12615/improving-exynos-9810-g...

https://www.anandtech.com/show/12620/improving-the-exynos-98...

adrian_b• 3 years ago

Depending on the region where they were sold, Samsung Galaxy S20 could use either Snapdragon 865 or Exynos 990 CPUs.

The Mozilla messages fail to mention this information that is essential for any CPU bug, which CPU model was used in the phones with crashes.

It is more likely that the bug is present in the Samsung Exynos 990 CPUs and not in Snapdragon, because Cortex-A77 was used much more widely, so it is improbable for such a bug to not also be noticed elsewhere.

happycube• 3 years ago

The bug report shows a Mali GPU, which rules out Snapdragon (Adreno GPU). Also, the phone model when googled shows the S20 FE, which has Exynos.

johntb86• 3 years ago

One of the crashes listed is https://crash-stats.mozilla.org/report/index/f585f1a9-0ed8-4... . That lists a "Mali-G77" GPU, which isn't used by Qualcomm. So the affected chip is probably the Exynos 990, which uses

* Octa-core (2x2.73 GHz Mongoose M5 & 2x2.50 GHz Cortex-A76 & 4x2.0 GHz Cortex-A55) - Global

* Octa-core (1x2.84 GHz Cortex-A77 & 3x2.42 GHz Cortex-A77 & 4x1.80 GHz Cortex-A55) - USA

(from https://www.gsmarena.com/samsung_galaxy_s20-10081.php )

dotdi2• 3 years ago

A few years ago I also ran into a hardware (or at least very low-level software) bug that was only affecting Samsung devices.

I had taken over maintenance of a library that allowed predicting satellite orbits from TLE data, which was used by the European Space Agency (and others) for the occasional mobile app.

Sporadically, we were getting reports of strange situations where the altitude of a satellite was way off, by hundreds or thousands of kilometers. The bug was really difficult to track down and reproduce, and by chance one of our Samsung tablets finally started showing the behaviour.

It turned out that Dalvik (the now obsolete Android VM) was figuring out that the TLE calculation was in a hot path and would JIT it to run more efficiently, and the optimized code used some native arithmetic (or was it trigonometic? can't recall) call that had a bug. We saw that the incorrect values started showing up right around the time when the JIT started kicking in. Fortunately, we could turn off JIT for specific devices and we rolled out an update that fixed the issue.

Fun times!

spuz• 3 years ago

How do you trace what the JIT is doing in an Android app?

dotdi2• 3 years ago

If I recall correctly, there were some TRACE or DEBUG logs one could turn on that would - amid a ton of unhelpful output - also emit logs when the JIT was doing its thing. Then, while running a small reproducer app and combing through the output, we noticed values were always correct before the JIT ran, then when the JIT started to kick in, we had a transition period where sometimes wrong values started to crop up (I assume until all threads had switched to the optimized code), and then always produce the wrong result.

This gave me a hunch that it could be related to JIT, and then I tried completely turning it off, which caused the issue to disappear.

FartyMcFarter• 3 years ago

It's never a compiler or CPU bug, until it is.

omoikane• 3 years ago

Summary: Samsung's LDRB doesn't zero-extend when reading a byte, so it's not following ARM specs[1]. Fix was to workaround in the JIT[2].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13

[2] https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.1...

bch• 3 years ago

Here’s the first story I read (~11 years ago) with an occurrence like this that cracked my concept of hardware/software divide…

https://www.theregister.com/2012/03/07/amd_opteron_bug_drago...

febed• 3 years ago

Kudos to these guys for figuring this out from the dump, not a trivial task! These kind of issues can take months to triage if one is unlucky. I remember our team going crazy trying to figure out the reason for random crashes until finally discovering a firmware bug in proprietary hardware.

Mozilla finds CPU bug (bad store forwarding) in Samsung Galaxy S20

Related Stories