A few CPU hardware bugs

https://www.taricorp.net/2026/a-few-cpu-bugs/

by signa11 • 6 months ago

130 42 6 months ago

taricorp.net

6K76981-O• 6 months ago

Writing software in embedded processor pipelines for bugs in the IT81202 CPU.

Microcode errata re-writes to GPR, compiling low level "mul," and "output," CPU RISC V to system archictecture.

Retr0id• 6 months ago

The GenuineIotel thing fascinates me because I can't fully grasp how it could happen. I can imagine a physical defect causing a permanent wrong-bit in a specific piece of silicon, but it seems more widespread than that. Perhaps some kind of bug in the logic synthesis process?

nippoo• 6 months ago

My favourite one of this kind is the Rockchip RK808 RTC, where the engineers thought that November had 31 days, needing a Linux kernel patch to this day that translates between Gregorian and Rockchip calendars (which are gradually diverging over time).

Also one of my favourite kernel patch messages: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

nasretdinov• 6 months ago

It's always November, isn't it? I've once made a log collection system that had a map of month names to months (had to create it because Go date package didn't support that specific abbreviation for month names).

As you might've guessed, it lacked November, but no one noticed for 4+ months, and I've left the company since. It created a local meme #nolognovember and even got to the public (it was in Russia: https://pikabu.ru/story/no_log_november_10441606)

lsofzz• 6 months ago

Rockchip calendars
.< haha i remember this

Gibbon12• 6 months ago

That's gold.

That hardware real time clocks keep time in date and time drives me batty. And no one does the right thing which is just a 64 bit counter counting 32khz ticks. Then use canned tested code to convert that to butt scratching monkey time.

Story my old boss designed an STD Bus RTC card in 1978 or something. Kept time in YY:MM:DD HH:MM:SS 1/60 sec. And was battery backed. With shadow registers that latched the time. Couple of years later redesigned it as a 32 bit seconds counter. With a 32khz sub seconds counter. Plus a 48 bit offset register. What was a whole card was now a couple of 4000 series IC's on the processor card. He wrote 400 bytes of Z80 assembly to convert that to date and time. He said was tricky to get right but once done was done.

Someone• 6 months ago

I would guess that’s because those chips were designed for use in systems that didn’t have a CPU, but reading the data sheet, it doesn’t look as if you can easily hook up this thing to 7-segment LEDs, so maybe this is a matter of “this is how we always did it, and if it ain’t broke, don’t fix it”, and then ‘fix’ it, anyways?

kevin_thibedeau• 6 months ago

Some of them do have an epoch counter in addition to broken down time.

alsetmusic• 6 months ago

I had some interactions with the guy responsible for the code that made our system do the right thing around Daylight Saving. Listening to him talk out loud as he thought about bugs was fascinating. He was clearly one of the smartest people I've met and I would quickly fall behind as he rationalized problems to himself. What a marvelous mind.

quotemstr• 6 months ago

That one is up there with the all time greats.

b1temy• 6 months ago

the characters ’n’ and ‘o’ differ by only one bit; an unpredictable error that sets that bit could change GenuineIntel to GenuineIotel.

On a QWERTY keyboard, the O key is also next to the I key. It's also possible someone accidentally fat-fingered "GenuineIontel" , noticed something was off, and moved their cursor between the "o" and "n", and accidentally hit Delete instead of Backspace.

Maybe an unlikely set of circumstances, but I imagine a random bit flip caused at the hardware-level is rare since it might cause other problems, if something more important was bit-flipped.

matja• 6 months ago

I like this theory - I can totally imagine some big spreadsheet of processor model names where someone copy/pastes the model name to some janky firmware-programming utility running on an off-the-shelf mini PC on the manufacturing floor, implemented as a "temporary fix" 5 years ago, every time the production line changes CPU model.

userbinator• 6 months ago

I am reminded of the old AMD CPUs with "unlockable" extra cores, which would when unlocked change the model name to something unusual.

"GenuineIotel" is definitely odd, but difficult to research more about; I suspect these CPUs might actually end up being collector's items sometime in the future.

because inserting no-op instructions after them prevents the issue.

The early 386s were extremely buggy and needed the same workaround: https://devblogs.microsoft.com/oldnewthing/20110112-00/?p=11...

pm2153• 6 months ago

Some of the 386 bugs described there sound to me like the classic kind of "multiple different subsystems interact in the wrong way" issue that can slip through the testing process and get into hardware, like this one:

For example, there was one bug that manifested itself in incorrect instruction decoding if a conditional branch instruction had just the right sequence of taken/not-taken history, and the branch instruction was followed immediately by a selector load, and one of the first two instructions at the destination of the branch was itself a jump, call, or return.

Even if you write up a comprehensive test plan for the branch predictor, and for selector loads, and so on, it might easily not include that particular corner case. And pre silicon testing is expensive and slow, which also limits how much of it you can do.

Taniwha2• 6 months ago

This sort of bug, especially in and around pipelines are always hard to find. In chips I've built we've had one guy who built a system that would build random instruction streams to try and trigger as many as we possibly could

adrian_b2• 6 months ago

80386 (1985) did not have a branch predictor, which was used first only in Intel Pentium (1993).

Nevertheless, the states of the internal pipelines, which were supposed to be stopped, flushed and restarted cleanly by taken branches, depended on whether the previous branches had been taken or not taken.

pjc50• 6 months ago

The revenge of the MIPS delay slot (the architecture simply didn't handle certain aspects of pipelining, so NOPs were required and documented as such).

ralferoo2• 6 months ago

It's not quite true to same NOPs were required. It was fairly common to just reorder instructions so that the branch instruction was just moved one forward so that the following instruction would execute before the branch target.

This also wasn't that uncommon. Sparc also had a delay slot that operated similarly to MIPS.

charcircuit2• 6 months ago

The workaround for this is to cripple the system

That is not the workaround in the documentation that was just linked.

  Workarounds:
  The solution to this problem is to put two instructions that do not require write back data after the mul instruction.

This seems reasonable for your compiler vendor to implement without getting rid of multiplication altogether.

direwolf204• 6 months ago

even if you don't know which instructions they are, just place two nops after every mul, problem solved

NobodyNada• 6 months ago

There's a difference in effort of several orders of magnitude between "change a setting so the compiler doesn't emit multiplies" and "convince GCC/LLVM to add a special-case flag for one very rare chip, or maintain your own fork". The vendor's workaround is the "ideal" solution, but disabling multiplies is a lot more practical if you don't need the performance.

They also mention in the next sentence that they adopted the "correct" workaround (by providing a multiplication library function for the compiler to call).

charcircuit2• 6 months ago

The company selling the chip can create a fork. They are typically the ones providing all of the sdks for you to use in order to use it, flash it, debug it, etc.

IshKebab• 6 months ago

To me, this issue doesn’t seem as embarrassing as Intel’s wrong CPUIDs. Pipelined CPUs are hard to build

I disagree. Misspelling a name in the CPUID is kind of easy to do, somewhat awkward to test (in a non-tautological way), and pretty easy to work around.

Having `mul ...; lw ...;` fail show that they've done very little testing of the chip. Any basic randomised pipeline testing would hit that trivial case.

Essentially all CPUs are pipelined today. In-order pipelined CPU execution semantics are not particularly hard to test. Even some open source testing systems could detect this bug, e.g. TestRig or RISCV-DV.

direwolf204• 6 months ago

When you have a known hardware bug like needing a nop after every mul, compilers can do this. You don't need to turn off mul entirely.

0xTJ• 6 months ago

The issue is that it's no longer actually RISC-V M at the point, you're changing the instruction set. If you're compiling RISC-V M code, that doesn't need the extra NOP.

That being said, the disabling of MUL is being done at a software project level here, not by the CPU vendor. It's in the same linked commit that added in the NOP instructions to the arithmetic routines.

direwolf204• 6 months ago

If your software runs on any chip and your chip runs any software, you have a problem, but in embedded cases, you know which chip runs which software, because you designed them together.

direwolf204• 6 months ago

Will someone register the Iotel trademark and sue Intel? That was the purpose of the Intel string in reverse!

qiqitori2• 6 months ago

I came across a CPU bug that prevented Linux from booting on 3rd gen i3/i5/i7 CPUs. Did a bunch of printf debugging until I was right before the freeze. Then found something relevant in the CPU errata. It could be "fixed" by passing in noapic. I had a decent writeup on the old CentOS forums, but they're gone, and I don't have a copy of my writeup anymore.

jraph• 6 months ago

Maybe the Wayback machine archived it by any chance?

jamesfinlayson• 6 months ago

Depends when it was - Web Archive didn't seem to archive forums for a long time (maybe it does now?).

pixl97• 6 months ago

noapic seemed to be a really common 'fix' for CPU and BIOS issues back then.

qiqitori2• 6 months ago

I just checked an old note I found, maybe it was noclflush actually. Affected one or more versions starting with 2.6.32-754.

mzs• 6 months ago

links to this entertainment: https://crackhead.technology/anjoy-ssc338q-libtools/

nabbed• 6 months ago

Wasn't there also a Pentium division bug of some sort. I didn't pay much attention to microcomputers back then (being a mainframe programmer at the time), but I remember hearing about it from the mainstream news.

chme• 6 months ago

I had to deal with Intel Quark SoC X1000 on a Galileo board years ago, where the LOCK prefix instruction caused segfaults. Since the SoC is single threaded, the lock prefix could just be patched out from resulting binaries, before the compiler/build system was patched.

https://en.wikipedia.org/wiki/Intel_Quark#Segfault_bug

A few CPU hardware bugs

Related Stories