My Snapdragon Dev Kit was healthy and working fine until a Windows update failed

https://jasoneckert.github.io/myblog/how-microsoft-killed-my-snapdragon-devkit/

by jasoneckert • 7 months ago

207 118 7 months ago

jasoneckert.github.io

GaryBluto• 7 months ago

Not that you are at fault here, but I'd be very hesitant to install any system updates so shortly after they brick my computer, especially when Microsoft is involved.

g947o4• 7 months ago

Or for an experimental device that has reached its EOL with no support for either software or hardware.

I would just completely disable Windows Update, act as if the computer is already compromised, and only do work where security is not an issue. That's the most "reliable" way to keep it working.

Of course, hindsight something something...

emeril• 7 months ago

having windows update running is already a compromise in some respects

I haven't run windows update in like 20 years

_ache_• 7 months ago

Snapdragon is already canceled so I guess, they just don't care about this device. It's Microsoft on ARM. Sad to say, but don't expect full support or quality update on this.

Ref:

- https://www.youtube.com/watch?v=XrA2Xe9f7e8 - https://www.jeffgeerling.com/blog/2024/qualcomm-snapdragon-d...

OptionOfT• 7 months ago

No, the dev kit was cancelled.

There are ARM laptops out there from multiple manufacturers, and there is a SnapDragon 2 on the horizon.

9999000009994• 7 months ago

I wanted to order one of these and then Qualcomm cancelled it.

Then I knew Windows ARM probably wasn't going to make it. Why any technical person would want a PC( not including Macs)that explicitly can't run Linux I'll never know.

kllrnohj3• 7 months ago

Why any technical person would want a PC that explicitly can't run Linux I'll never know.

Huh? https://www.phoronix.com/review/snapdragon-x1e-september

pjmlp4• 7 months ago

Technical person that knows UNIX since being introduced to it via Xenix in 1993, and has used plenty of UNIX flavours since then.

Some of us like the experience of Visual Studio, being able to do graphics development with modern graphics APIs that don't require a bazillion of code lines, with debuggers, not having to spend weekends trying to understand why yet again YouTube videos are not being hardware accelerated, scout for hardware that is supposed to work and then fails because the new firmaware update is no longer compatible,....

kllrnohj3• 7 months ago

The Snapdragon Dev Kit is canceled. Snapdragon as a whole sure as hell isn't canceled, and Windows on Snapdragon isn't, either. There's loads of Windows laptops using Snapdragon with more continuing to release.

motorpixel• 7 months ago

It's just the "dev kit". Snapdragon for the laptop form factor is alive and well. You don't need a devkit for a laptop running Windows and QCOM easily figured that out.

cobalt• 7 months ago

you mean like the new generation they just announced? https://www.qualcomm.com/laptops/products/snapdragon-x2-elit...

GranPC2• 7 months ago

Considering that you were seeing unpredictable behavior in the boot selector, with it randomly freezing, I would assume a hardware component (RAM?) kicked the bucket. If it were firmware corruption, it would consistently fail to present the menu, or wouldn't boot at all.

Microsoft's code quality might not be at its peak right now, but blaming them for what's most likely a hardware fault isn't very productive IMO.

ankurdhama• 7 months ago

So the "hardware failure" happening exactly at the same time the Windows update installation failed are not related? That sounds like a one in a billion kind of coincident.

eli• 7 months ago

An upgrade process involves heavy CPU use, disk read/writes, and at least a few power cycles in short time period. Depending what OP was doing on it otherwise, it could've been the highest temperature the device had ever seen. It's not so crazy.

My guess would've been SSD failure, which would make sense to seem to appear after lots of writes. In the olden days I used to cross my fingers when rebooting spinning disk servers with very long uptimes because it was known there was a chance they wouldn't come back up even though they were running fine.

olyjohn• 7 months ago

I've fixed thousands of PCs and Macs over my career. Coincidences like this happen all the time. I mean, have you seen the frequency of updates these days? There are always some kind of updates happening. So the chances of your system breaking during an update is not actually that slim.

santoshalper• 7 months ago

Two bugs occurring at the same time is definitely not one in a billion, and with billions of computers in the world, weird shit is going to happen.

nightfly• 7 months ago

windows update just doing a normal write causing the active chunk of flash memory being used to hold something in the boot loader to a different failed/failing section

GranPC2• 7 months ago

For all we know, this thing was on its last legs (these machines do run very hot!) and the update process might have been the final nail in the coffin. That doesn't mean Microsoft set out to kill OP's machine... Same thing could have happened if OP ran make -j8 -- we wouldn't blame GNU make.

justsomehnguy• 7 months ago

"Hardware failure" => "WinUpdate failure" => "Corrupted system" conforms the Occam's razor.

Graziano_M2• 7 months ago

I had a friend's dad's computer's HDD fail while I was installing Linux on it to show him it. That was terrifying. I still remember the error, and I just left with it (and Windows) unable to boot. Later my friend told me that the drive was toast.

Come to think of it, maybe it was me. I might have trashed the MBR? I remember the error, though, "Non system disk or disk error".

croes• 7 months ago

Like winning the lottery?

Happens quite often

tobyjsullivan• 7 months ago

Over my 35 years of computer use, most hardware failures (very, very rare) happen during a reboot or power-on. And most of my reboots happen when installing updates. It actually seems like a very high probability in my limited experience.

Of course, it’s possible that the windows update was a factor, when combined with other conditions.

pdpi• 7 months ago

I think it's fair to say they're related, yes. But causality can well be the other way around — that Windows upgrade failed because of flaky hardware.

taneq• 7 months ago

A software update can absolutely trigger or unmask a hardware bug. It’s not an either/or thing, it’s usually (if a hardware issue is actually present) both in tandem:

wvenable• 7 months ago

If had happened any other time, there wouldn't be a blog post about it and we wouldn't be reading about it.

ezfe• 7 months ago

This happens all the time, people always doubt it - but the patterns are always consistent: large updates kill hardware that's in progress of failing

Aurornis2• 7 months ago

That sounds like a one in a billion kind of coincident

Hardware is more likely to fail under load than at idle.

Blaming the last thing that was happening before hardware failed isn't a good conclusion, especially when the failure mode manifests as random startup failures instead of a predictable stop at some software stage.

wnevets• 7 months ago

This reminds me of the 3090 hardware problems being exposed by Amazons New World [1]. Everyone really wanted to blame the software.

https://www.pcgamer.com/amazon-new-world-killing-rtx-3090-gp...

Aurornis2• 7 months ago

Yeah, I agree. This just feels like an appeal to anti-Microsoft clicks

From the article:

It won’t get past the Snapdragon boot logo before rebooting or powering off… again, seemingly at random.

Random freezing at different points of the boot process suggests a hardware failure, not something broken in the software boot chain.

colechristensen• 7 months ago

It could very well be something poorly configured in the boot chain leading to random failures. There are plenty of hardware things configured in software which can lead to plenty of different kinds of random failures.

numpad02• 7 months ago

Or something hit max program-erase cycle counts and are returning corrupt/old data. Flash ROMs tend to become "sticky" with previous states as you write more to them. I think it's possible that ROMs used for early SoC boot firmware or peripherals firmware still don't have wear leveling that they could become unusable after just a hundred or so of writes.

fortran772• 7 months ago

This just feels like an appeal to anti-Microsoft clicks

Exactly. Did you notice the one comment on his blog? It's a Linux zealot saying "Linux".

ErroneousBosh2• 7 months ago

Random freezing at different points of the boot process suggests a hardware failure, not something broken in the software boot chain.

Power issues all day long. It'll be fine until the SoC enables enough peripherals for one of the rails to sag down.

That being said, it's a hell of a coincidence that it failed exactly when a software update failed.

deckar01• 7 months ago

I would test the CPU cooler since the fans ran so hard. Temps ramp up around the login screen, then stay hot and reboots get unpredictable.

I recently had a water cooler pump die during a Windows update. The pump was going out, but the unthrottled update getting stuck on a monster CPU finished it off.

inferiorhuman• 7 months ago

With the original Arduino Due there was some fun undocumented behavior with the MCU (an Atmel Cortex-M3) where it would do random things at boot unless you installed a 10k resistor. From booting off of flash or ROM at random to clocks not coming up correctly.

I swear I was doing just fine with it booting reliably until I decided to try flashing it over the SWD interface. But wouldn't you know it, soldering a resistor fixed it. Mostly.

shakna• 7 months ago

The update listed in the article contains two UEFI patches, intended for "handheld devices".

It would be entirely unsurprising to me if this trashed UEFI for this particular ARM device, from firmware corruption.

tallanvor• 7 months ago

That's plausible, but I'd expect the UEFI patches to come from a vendor, not Microsoft. So if one came from Qualcomm, and they didn't properly specify the devices it should be installed on, that wouldn't make it Microsoft's fault.

p0w3n3d2• 7 months ago

when something works flawlessly and starts to fail after an update (so no user actions there) this could mean that update made the hardware fail. For example overuse of flash in ssd (it's been already reported https://community.spiceworks.com/t/anyone-else-have-their-ss...) or reflashing a component too many times (simple error in scripts)

MrGilbert• 7 months ago

Or it might not be related at all. Correlation and causality might be in charge here.

everforward• 7 months ago

I'm not so sure, I've had a similar-ish issue on a W10 PC. I vaguely suspect a race condition on one of the drivers; I've specifically got my eye on the esp32 flashing drivers.

Sometimes it boots fine, sometimes the spinning dial disappears and it gets hung on the black screen, sometimes it hangs during the spinning dial and freezes, and very occasionally blue screens with a DPC watchdog violation. Oddly, it can happen during Safe Mode boots as well.

I would think hardware, but RAM has been replaced and all is well once it boots up. I can redline the CPU and GPU at the same time with no issues.

jasoneckert• 7 months ago

Sage point. But it actually turned out to be non-hardware-related (I added an "Update: It's alive!" section to the blog post).

schmuckonwheels2• 7 months ago

Yeah they (probably) did not.

Almost certainly a soft hardware failure, likely the SSD.

I've run into a similar situation - except the culprit was Linux not Windows. Tossed the machine in a closet for a few months, when it miraculously started working again. Until it broke again a day and a half later. It's disk or RAM corruption.

Give it up dude, it's the hardware, but let not an opportunity to smash Microsoft go unfulfilled.

llmslave2• 7 months ago

Considering that when the problems started, many people online were having a similar issue I think it's unlikely it was a hardware failure.

bigyabai2• 7 months ago

Or Microsoft deliberately bought shoddy RAM/SSDs, intending to force the bathtub curve down onto devkits they wanted dead ASAP.

geerlingguy2• 7 months ago

From the article:

I opened the system and reseated everything, including the SSD. No change. I even tested the SSD in another machine to rule it out, and it’s fine too.

But that doesn't mean it's not bad RAM, a bad SSD controller, who knows what... there are only a few of these boxes in the wild regardless, so it's unlikely it can be debugged :(

schmuckonwheels2• 7 months ago

Hi, Jeff! Didn't realize the SSD was removable. Murphy's Law dictates the problem will always be in a soldered-down component whenever this happens.

andwur• 7 months ago

Considering the number of x86 machines I've come across in fleet deployments that were put into various states of brickdom from Windows Update, I would not be at all surprised if it was a bad update-rollback sequence.

Laptops seem particularly susceptible to whatever (anti) magic Microsoft utilise for their update rollback process, but it happens to every device class seemingly at random. Besides the run of the mill "corrupt files at random in System32", which is common and simple enough to fix with a clean install, I've had a few cases where it appears an attempt at rolling back a BIOS update has been interrupted by the rollback manager and left those machines hard bricked. They could only be recovered by flashing a clean BIOS image with an external programmer and clip (or hand soldering leads), after which they ran without issue.

As much as it's valid to question the unconditional anti-Microsoft mentality, they are still far from infallible and from my experience they are getting notably more unreliable in recent years.

jamesnorden• 7 months ago

Almost certainly a soft hardware failure, likely the SSD.

If you actually read the article, you'd know it wasn't. Besides, Windows updates can and do deliver firmware/bios updates.

byte_0• 7 months ago

Not sure how much it could help, but is there a possibility you connect the SSD to another machine with the same architecture, run Windows install in it, then once Windows is installed and running, shut down, move the SSD back to the Snapdragon kit and attempt to boot? Just an idea...

petcat• 7 months ago

I doubt this is a Windows issue.

I would replace your ram sticks. I had a similar mysterious issue on an old Intel nuc. Got some new sticks off Amazon and never had the problem again

geerlingguy2• 7 months ago

Sadly it's one of the Arm Qualcomm Snapdragon boxes; none that I've seen offer replaceable RAM sticks.

smileybarry• 7 months ago

I recall one of the issues leading up to their abrupt cancellation was fulfillment, so I can't help but suspect there's some potential (long-term) issue they couldn't work out for this dev kit's chipset. Maybe some part of the chain was held together with glue and "this shouldn't fail but continue anyway" and whatever hardware issue eventually hit something critical. (And they intended to fix this some time after shipping, and gave up halfway through fulfillment)

ggm• 7 months ago

Nobody has the time or energy to chase companies up for this stuff, and you know somewhere in the T&C they inserted a legal clause which is expensive to contest or un-contestable to liability.

But, that said, it saddens me we've normalised "oh well" when it comes to kit. even dev kit. If MS can't manage release engineering to keep dev/test things alive, then it's not helpful to the belief they can do it for production things either.

I inherited an IBM PC/RT back in the 90s. It was well outside what most people would consider its support lifetime. IBM could not have been more helpful working out how to keep it alive. I suspect this influences why when I later had some financial authority I was happier to buy thinkpad, than any other hardware we had available: I knew from experience they stood behind their maintenance guarantees. The device was configured to run BSD, not the IBM supported OS of the day, made no difference. It was end of life product line, made no difference.

This was before Lenovo of course. But the point stands: people with positive support stories, keep that vendor in their top-set

arjie• 7 months ago

These devices are nightmares. I'm sure things will pay off at some point but this feels like all those years where everyone was cursing Nvidia on Linux and praising AMD's dedication to open source but my computer would constantly lock up regardless until I switched to Nvidia. There was this massive disconnect between my experience and what everyone told me was best supported.

Similarly, I'm constantly hearing about Qualcomm's renewed interest in Linux and this and that and how the X2 Elite will be fully supported but I have never known them to be like this. A decade or so ago we were trying to work for a school project on one of their dev kits and the documentation was so sparse.

Then I see that the Snapdragon X Elite comes in this Ideacentre stuff but looking online no one has gotten Linux anywhere close to as good as Linux is on a Mac M2. That, for me, is the marker. If a Mac can run Linux better than whatever chipset you've released, it's just not hardware worth buying. If you're not Apple, you have to support Linux. Otherwise, to borrow Internet lingo, you're "deeply unserious".

blowsand• 7 months ago

Completely off topic: the font on that site is quite readable and easy on the eyes and brain, the way the dyslexia font on the Kindle is.

cratermoon• 7 months ago

It's Overlock https://fonts.google.com/specimen/Overlock

elzbardico• 7 months ago

Man, it looks like a memory issue. Is the memory user replaceable on this machine?

10000truths• 7 months ago

Sounds like this could potentially be some defective RAM. Memtest86 can boot from UEFI directly, so it should hopefully show up in BDS. A run should tell you what regions of RAM are bad, if any.

tgtweak• 7 months ago

change the SSD and retry (the same ssd in another machine may not trigger the same error btw, this is not unilateral process of elimination) - those windows updates do a lot of disk writes and a small miss there can screw up an entire install since it shuffles things around in preboot environment (moving them on disk) and that can corrupt things and prevent a new install in the same way.

You can also try to live boot into Ubuntu 25.04 arm64 since that iso has experimental snapdragon elite support and has some built-in drivers for storage and network - you can extract firmware from the windows drivers with qcom-firmware-extract - they recommend doing this from a windows partition which you should have (albeit possibly corrupted).

If that still fails - you have a ram issue as others have pointed out. I've had the exact same symptoms (hardware instability after windows update) and it was nvme ssd (an early samsung one) and ram, in both instances.

Not saying the windows update didn't also come with some junk firmware that got loaded into some of your devices, but that would be a distant diagnosis from ssd/ram (and many others would have seen the exact same thing during their update if it was that).

hamonrye• 7 months ago

Cool. Running development for fork Ruby on Rails /src/ crates to match kit for enterprise-level software firmware recovery paths.

pdyc• 7 months ago

are you able to stop at uefi stage and system is stable in bootloader stage? if yes than it may not be software issue. Others have covered checking ram and ssd. I suspect it could also be thermal or voltage issue.

wewewedxfgdf• 7 months ago

We just bought a Snapdragon Windows device.

I trust Microsoft 0% to keep developing Windows for it.

numpad02• 7 months ago

Hopefully this serves as a reminder to decision makers with Web backgrounds to NOT push random non-critical _firmware_ updates without clear merit, or random updates in general.

Security is not fluids. It doesn't naturally evaporate. So don't try to add like they're washer fluids.

Those low-level software and associated hardware don't take software overwrites very well, even today. They might have total cumulative max overwrites, or manufacturer supplied update codes can still be dubious. It's (not)okay if you are meaning it to be a tool for your planned obsolescence strategy, otherwise, just don't touch it for the sake of doing it.

fortran772• 7 months ago

THere's a chance that there was a hardware failure and _that's_ why the update was failing. And not a failed update that caused the hardware failure.

Either way, may the memroy of your Snapdragon Dev Kit be a blessing.

ehnto• 7 months ago

Only adding to this because it's likely a hardware failure, and I had a challenging time debugging a similar issue in an aftermarket engine ECU years ago. Thought it might make a fun anecdote.

The car would run fine once started, but the car just wouldn't start sometimes (quite modified so I knew the systems well). The started would turn as that was a simple relay, but all ECU controlled devices wouldn't trigger. Plugging into the ECU, no error codes and all looked normal.

Eventually we tracked the issue to some corruption in the ROM that was only getting read in certain circumstances, since the ECU stores maps for engine parameters based on things like pressure and temperature you might only hit the corrupted bits of a table in very specific circumstances.

Reflashed the ROM and all was good afterwards. The suspected cause of corruption was intermittent power supply that had been fixed a while earlier.

danans2• 7 months ago

I was pleased to discover recently that Ubuntu is supporting my NVidia Jetson going forward after NVidias official support period ends:

https://canonical.com/blog/ubuntu-now-officially-supports-nv...

So there is at least one ARM devkit with long term Linux support.

jogu• 7 months ago

Jetson is such a confusing product and it's difficult to tell exactly what they're supporting. Looking at the image download page it seems to be only Orin and newer?

https://ubuntu.com/download/nvidia-jetson

danans2• 7 months ago

Yeah. I have the Orin devkit

bbhjbhjbmnn• 7 months ago

Soo.. Qualcomm can use a Windows drive to receive calibration data and other configurations. If you have a virus or something, you might brick a board, if its connected. We used 3-4 days in the factory to figure out why our boards were bricked. The PCs on the production line were all infected.

sedatk• 7 months ago

As a data point, my Windows Dev Kit 2023 (also a Snapdragon) has been working great so far.

trvz• 7 months ago

Seems like the typical Microsoft experience nowadays.

My ROG Ally ran fine on Windows 11 at the beginning, but a year later always randomly crashed, even when idle, on a fresh OS install. After switching to SteamOS it runs stable again.

ChocolateGod• 7 months ago

If it updated the UEFI it could be that somehow the UEFI now starts the hardware wrong (RAM setup maybe?) that causes it to be unstable.

nubskr• 7 months ago

The holy trinity of modern hardware: update servers, firmware locks, and abandoned devkits

megous• 7 months ago

If it's a proper devkit it should have accessible and documented test points for all voltage rails and should have come with a complete schematic. It should be possible to go through them with a voltmeter or oscilloscope and see if everything looks ok.

Given the symptoms (random crashes not right away at boot), and given that qcom is anal about secure boot, my guess is that it's unlikely that it's a firmware (in SPI-NOR or wherever) corruption that initially caused this. Firmware is checked each boot.

Might be as simple as degraded capacitor, or something similar.

And I can imagine that it's not hard to destroy this kind of HW physically with a SW update. PMICs can often produce voltages way higher than Vmax of connected components. But it's unlikely that if bug like that happened, that it would only affect one devkit out there, and not a whole range of devices.

anonymousiam• 7 months ago

After decades of experience, my normal practice is to make regular multi-generational backups of the entire disk (usually compressed). When something like this goes wrong, I can revert to the last known-good image and go from there. It saves a lot of time and trouble.

0xFFFC• 7 months ago

That Snapdragon Kit you have was immediately recalled due to known issues. I read somewhere only a couple hundred ever shipped. I am one of the lucky ones to get one as well. If I were you I would get one of the Lenovo arm64 desktops and save what you have as a relic.

My Snapdragon Dev Kit was healthy and working fine until a Windows update failed

Related Stories