SSD drives’ 40k-hour “death bug” continues to catch enterprises unawares

https://thestack.technology/ssd-death-bug-40000-hours-sandisk/

by ghostpepper • 4 years ago

109 54 4 years ago

pclmulqdq2• 4 years ago

I wonder how the institutional knowledge on correlated disk failure got lost. This used to be common with HDDs, and one large enterprise I am aware of makes sure that boot drives in a datacenter (the only SSDs of this type) are always from a mix batches at a mix of vendors.

reaperducer• 4 years ago

Possibly because so much of what happens in the tech industry has been abstracted away.

The vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable.

There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.

hinkley3• 4 years ago

That, and the curse of working the same 18 months 4 times on the way to 'senior'.

All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008.

But for drives in particular, people are pretty superstitious about giving them up.

notacoward3• 4 years ago

"Point and click" is unnecessarily derogatory. During my career as a developer I spent my share of time racking my own systems, diagnosing (both network and disk) cable faults, etc. I also spent time working with the kinds of software tools needed to manage dozens of clusters up to 10K machines each at 36 disks per machine. At that scale you really need to minimize the number of people who get to muck with the physical bits, and the amount of time they spend doing so. The complexity has merely shifted, not disappeared, and most developers or sysadmins from the previous generation would themselves be hard pressed to master the management of modern systems. It's no more "point and click" now than it was "lift heavy things" before.

if all you know is cloud, you don't need to know things like this.

That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?

pragmatic• 4 years ago

ClickOps is a term I heard second hand for this.

majewsky• 4 years ago

Once you cross a certain size, how else is it supposed to work, realistically speaking? My team administers servers in Brazil, Australia, Japan, Netherlands, the US, and several other countries. We can't do racking, cabling and hardware maintenance by ourselves. If we did, we'd spend 50% of our work hours just commuting on international flights. The only realistic option is to rely on local DC Ops teams to perform the manual steps involved in hardware buildup, maintenance and decom.

toast04• 4 years ago

It's just so much more convenient to single source things and have a consistent experience... No worrying about why these drives failed and these other ones didn't; not so great when they all fail though.

Fargoan2• 4 years ago

I'm just some random guy who doesn't work in tech and even I know about this. I was hesitant to buy any SSDs when they were new because I was worried about drive failure.

cyanydeez• 4 years ago

This isn't "SSD failure due to random overuse"

This is a specific bug, like an overflow that is consistently triggered after about 4 years of use.

zitterbewegung• 4 years ago

This is quite meta (literally and also not related to the company).

sprayk• 4 years ago

Meta (the company) too, maybe. They have a pretty big datacenter footprint.

notacoward3• 4 years ago

They also know about mixing brands/batches/etc. to avoid exactly this kind of correlated failure. During one of my oncalls, we had an uncomfortably large number of failures fitting this kind of pattern, but they were well enough distributed and we had enough data redundancy that there was no data loss. Sometimes there's safety in numbers, if you know how to make those numbers work for you rather than against you.

loeg• 4 years ago

Mostly HDDs, though.

goatcode6• 4 years ago

It's been a while since I've been in the hardware-level IT side of things, but is it not unwise to run a production server with SanDisk drives?

batmansmom1• 4 years ago

What is the industry standard for this?

goatcode6• 4 years ago

No idea now. I usually look to BackBlaze's periodic drive performance reports when I'm shopping, but even that's been a while for me.

walrus014• 4 years ago

samsung or intel would be strongly preferable in my opinion

cjensen• 4 years ago

For an SSD, you need to buy "Enterprise" or "Data Center" class drives for this kind of application. That class of drive have higher write endurance which is important since enterprise applications can have orders of magnitude more writes to their drives than a typical desktop.

That said, there's no real standard blanket vendor recommendation so long as the "Enterprise" criteria is met.

NavinF2• 4 years ago

been a while since I've been in the hardware-level

Well yeah that’s probably why you see “SanDisk” and think of consumer SD cards.

goatcode6• 4 years ago

I'd imagine. Do they make stuff now that's intended to be spec'd for enterprise servers?

tjoff2• 4 years ago

The update corrects a logging check: “Assert had a bad check to validate the value of circular buffer’s index value. Instead of checking the max value as N, it checked for N-1,” Dell’s advisory said.

So the code was never tested even once?

nix0n• 4 years ago

It could have been checked once, it just wasn't checked N times in a row.

swatcoder• 4 years ago

Possibly, but if you don’t think that’s common you may be grossly overestimating how thorough even the best testing process can be.

Engineering processes are systems with their own bugs, and the people/systems that implement them can’t deliver perfectly either.

There are always going to be things that slip through, and some of them are bound to look embarrassingly sloppy.

tjoff2• 4 years ago

Testing that a circular buffer can wrap is a very very low bar though.

I'm not talking automated testes or anything. Just once, on your own machine.

I agree that it is a bit unfair to pick an example after the fact. But it does not inspire confidence.

hinkley3• 4 years ago

A lot of preventable accidents come from the human brain discounting the threat after a large number of positive interactions. For all we know there once was a test three refactorings ago and someone fenced it away.

notacoward3• 4 years ago

in a specific line of older, end-of-life SanDisk SAS SSDs

They weren't end-of-life when they were sold this way. That's just the PR department trying to make their failure seem less severe than it was.

toast04• 4 years ago

Well, we now know the lifetime for those SSDs is 40k hours. At least we don't have to wonder.

detaro• 4 years ago

Or you can choose to install the firmware update fixing the issue and get back that sweet sweet uncertainty that your drive might live longer.

treeman79• 4 years ago

Can’t find it. But there was an old article where some young guy at a conference was explaining to Netflix why SSDs weren’t going to work due to drive failure cost.

Response was epic.

Everything you say is true.

But we don’t care, we rent the servers.

supernova87a• 4 years ago

Response was epic.

As in, no one believed him at the time?

And what does Netflix have a need for SSDs for? Their load seems entirely predictable / serveable / stable with spinning disks and reasonably (but not lightning) fast access patterns? I would have guessed the network latency is always dominant.

jasonwatkinspdx• 4 years ago

There's still a huge long tail, so iops matter. Netflix definitely wants to minimize any time you ever see a waiting on network spinner.

I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.

SteveNuts• 4 years ago

Netflix has a lot more to it than just serving video data

stefan_• 4 years ago

Because "SSD fails at a fixed amount of hours" is a recall-level bug and well.. where is the recall then? This should be a class-action lawsuit.

kabdib• 4 years ago

Is there anything on the SanDisk site about this? I checked a few weeks ago and saw nothing.

jonas21• 4 years ago

There's no need for a recall because it can be fixed with a firmware update. And as the article mentions, vendors did issue urgent/critical firmware updates for it back in 2020.

https://www.dell.com/support/home/en-us/drivers/driversdetai...

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...

koheripbal• 4 years ago

Firmware updates on hardware have a very very low update rate outside of professional datacenters.

DannyBee• 4 years ago

Uh, what? In most industries, it would be a recall and fix. The recall is used to notify people, and to pull unfixed products off shelves.

ksec• 4 years ago

Well, I thought it should be well known by now since it took down HN a while back.

SSD drives’ 40k-hour “death bug” continues to catch enterprises unawares

Related Stories