Unpowered SSD endurance investigation finds data loss and performance issues

https://www.tomshardware.com/pc-components/storage/unpowered-ssd-endurance-investigation-finds-severe-data-loss-and-performance-issues-reminds-us-of-the-importance-of-refreshing-backups

by progval • 1 year ago

123 87 1 year ago

tomshardware.com

csdvrx2• 1 year ago

For long term storage, prefer hard drives (careful about CMR vs SMR)

If you have specific random IO high performance needs, you can either

- get a SLC drive like https://news.solidigm.com/en-WW/230095-introducing-the-solid...

- make one yourself by hacking the firmware: https://news.ycombinator.com/item?id=40405578

Be careful when you use something "exotic", and do not trust drives that are too recent to be fully tested: I learned my lesson for M2 2230 drives https://www.reddit.com/r/zfs/comments/17pztue/warning_you_ma... which seems validated by the large numbers of similar experiences like https://github.com/openzfs/zfs/discussions/14793

dragontamer2• 1 year ago

If you care about long term storage, make a NAS and run ZFS scrub (or equivalent) every 6 months. That will check for errors and fix them as they come up.

All error correction has a limit. If too many errors build up, it becomes unrecoverable errors. But as long as you reread and fix them within the error correction region, it's fine.

csdvrx2• 1 year ago

run ZFS scrub (or equivalent) every 6 months

zfs in mirror mode offers redundancy at the block level but scrub requires plugging the device

All error correction has a limit. If too many errors build up, it becomes unrecoverable errors

There are software solutions. You can specify the redundancy you want.

For long term storage, if using a single media that you can't plug and scrub, I recommend par2 (https://en.wikipedia.org/wiki/Parchive?useskin=vector) over NTFS: there are many NTFS file recovery tools, and it shouldn't be too hard to roll your own solution to use the redundancy when a given sector can't be read

WalterGR3• 1 year ago

What hardware, though? I want to build a NAS / attached storage array but after accidentally purchasing an SMR drive[0] I’m a little hesitant to even confront the project.

A few tens of TBs. Local, not cloud.

[0] Maybe 7 years ago. I don’t know if anything has changed since, e.g. honest, up-front labeling.

[0*] For those unfamiliar, SMR is Shingled Magnetic Recording. https://en.m.wikipedia.org/wiki/Shingled_magnetic_recording

ErneX• 1 year ago

I use TrueNAS and it does a weekly scrub IIRC.

AshamedCaptain• 1 year ago

(careful about CMR vs SMR)

Given the context of long term storage... why?

0cf8612b2e1e• 1 year ago

After I was bamboozled with a SMR drive, always great to just make the callout to those who might be unaware. What a piece of garbage to let vendors upsell higher numbers.

(Yes, I know some applications can be agnostic to SMR, but it should never be used in a general purpose drive).

whoopdedo• 1 year ago

Untested hypothesis, but I would expect the wider spacing between tracks in CMR makes it more resilient against random bit flips. I'm not aware of any experiments to prove this and it may be worth doing. If the HD manufacture can convince us that SMR is just as reliable for archival storage it would help them sell those drives since right now lots of people are avoiding SMR due to poor performance and the infamy of the bait-and-switch that happened a few years back.

sitkack4• 1 year ago

Tape is extremely cheap now. I booted up a couple laptops that have been sitting unpowered for over 7 years and the sata SSD in one of them has missing sectors. It had zero issues when shutdown.

seszett• 1 year ago

Is tape actually cheap? Tape drives seem quite expensive to me, unless I don't have the right references.

matheusmoreira4• 1 year ago

Tape sucks unless you've got massive amounts of money to burn. Not only are tape drives expensive, they only read the last two tape generations. It's entirely possible to end up in a future where your tapes are unreadable.

CTDOCodebases• 1 year ago

The issue with tape is that you have to store it in a temperature controlled environment.

fpoling• 1 year ago

While the tape is relatively cheap, the tape drives are not. The new ones typically starts at 4K USD, although sometimes for older models the prices can drop below 2K.

vlovich1234• 1 year ago

- make one yourself by hacking the firmware: https://news.ycombinator.com/item?id=40405578 Be careful when you use something "exotic", and do not trust drives that are too recent to be fully tested

Do you realize the irony of cautioning about buying off the shelf hardware but recommending hacking firmware yourself?

userbinator4• 1 year ago

That "firmware hack" is just enabling an option that manufacturers have always had (effectively 100% "SLC cache") but almost always never use for reasons likely to do with planned obsolescence.

Brian_K_White• 1 year ago

They did not recommend. They listed.

rkagerer• 1 year ago

I would never buy a no-name SSD. Did it once long ago and got bit, wrote a program to sequentially write a pseudorandom sequence across the whole volume then read back and verify, and proved all 8 Pacer SSD's I had suffered corruption.

WalterGR3• 1 year ago

That’s also fairly common for cheap ‘thumb drives’, as I understand it. I’ve been bitten by that before.

(Edit: Allegedly if you use low-numbered storage blocks you’ll be okay, but the advertised capacity (both packaging and what it reports to OS) is a straight-up lie.)

WalterGR3• 1 year ago

(Edit: Retail packaging, I mean. Too late to edit my comment.)

gnabgib• 1 year ago

Discussion on the original source: (20 points, 3 days ago, 5 comments) https://news.ycombinator.com/item?id=43702193

Related: SSD as Long Term Storage Testing (132 points, 2023, 101 comments) https://news.ycombinator.com/item?id=35382252

bityard• 1 year ago

I didn't think it was controversial that SSDs are terrible at long term storage?

wmf3• 1 year ago

I wouldn't say it's controversial but I suspect most people don't know about it. There's been a lot of discussion about SSD write endurance but almost none about retention.

creatonez• 1 year ago

Right, but the news is that someone finally actually tested the unplugged-is-worse theory in a long term real world test. So much of the existing data about SSD endurance has been exclusively SSDs in datacenters plugged in 24/7, so it's nice to see some data showing what the difference actually looks like.

jeffbee• 1 year ago

Endurance is proportional to programming temperature. In the video, when all four SSDs are installed at once, the composite device temperature ranges over 12º. This should be expected to influence the outcomes.

ein0p2• 1 year ago

This is a known issue. You have to power up your SSDs (and flash cards, which are based on even more flimsy/cost optimized version of the same tech) every now and then for them to keep data. SSDs are not suitable for long term cold storage or archiving. Corollary: don't lose that recovery passphrase you've printed out for your hardware crypto key, the flash memory in it is also not eternal.

RicoElectrico• 1 year ago

Please explain to me how is that supposed to work. For all I know the floating gate is, well, isolated and only writes (which SSDs don't like if they're repeated on the same spot) touch it through mechanisms not unlike MOSFET aging i.e. carrier injection. Reading on the other hand depends on the charge in floating gate altering Vt of the transistor below, this action not being able to drain any charge from the floating gate.

ein0p2• 1 year ago

If you at least read the data from the drive from time to time, the controller will "refresh" the charge by effectively re-writing data that can't be read without errors. Controllers will also tolerate and correct _some_ bit flips on the fly, topping up cells, or re-mapping bad pages. Think of it as ZFS scrub, basically, except you never see most of the errors.

wmf3• 1 year ago

According to a local expert (ahem), leakage can occur through mechanisms like Fowler-Nordheim tunneling or Poole-Frenkel emission, often facilitated by defects in the oxide layers.

jsheard2• 1 year ago

A not-so-fun fact is that this even applies to modern read-only media, most notably Nintendo game carts. Back in the day they used mask ROMs which ought to last more or less forever, but with the DS they started using cheaper NOR or NAND flash for larger games, and then for all games with the 3DS onwards. Those carts will bit-rot eventually if left unpowered for a long time.

vel0city• 1 year ago

I've noticed a number of GBA carts I've picked up used (and probably not played in a long while) fail to load on the first read. Sometimes no logo, sometimes corrupted logo. Turning it off and on a couple of times solved the issue, and once it boots OK it'll boot OK pretty much every time after. Probably until it sits on the shelf for a long while.

Dylan168074• 1 year ago

Do we have confirmation that the carts are able to refresh the data?

computator• 1 year ago

You have to power up your SSDs every now and then for them to keep data.

What is the protocol you should use with SSDs that you’re storing? Should you:

- power up the SSD for an instant (or for some minutes?) without needing to read anything?

- or power up the cells where your data resides by reading the files you had created on the SSD?

- or rewrite the cells by reading your files, deleting them, and writing them back to the SSD?

mikequinlan• 1 year ago

What is the protocol you should use with SSDs that you’re storing?

The correct protocol is to copy the data to a more durable medium and store that.

gblargg• 1 year ago

I'd at least just read all the used blocks on the drive. partclone is the most efficient that comes to mind, because it just copies used sectors. Just redirect to /dev/null.

    partclone.filesystem --source /dev/sdx --clone --output /dev/null

rapjr9• 1 year ago

Maybe someone should design and sell a "drivekeeper" that you can plug all your backup SSD's into and it will power them up on a time table and do whatever is necessary to cause them to maintain integrity. Could be something like a Raspberry Pi with a many-port USB hub, or with a PCB with a bunch of connectors the raw drives can plug into. Could maybe even give a warning if a drive is degrading. Possibly it could be a small device with a simple MCU and a battery that you snap directly onto the SSD's connector?

matheusmoreira4• 1 year ago

don't lose that recovery passphrase you've printed out for your hardware crypto key, the flash memory in it is also not eternal

Yeah. Paper is the best long term storage medium, known to last for centuries.

https://wiki.archlinux.org/title/Paperkey

It's a good idea to have a backup copy of the encryption keys. Losing signing keys is not a big deal but losing encryption keys can lead to severe data loss.

smittywerben• 1 year ago

Paper is the best long term storage medium, known to last for centuries.

Carve it into stone if you it need to last longer than the Romans.

bobmcnamara• 1 year ago

Good paper too: low acid content in a low cellulose eating insect environment.

zamadatix• 1 year ago

The article states as much but to sum it all up as just that is leaving most of the good stuff out.

Perhaps the most interesting part of the experiment series has been just how much longer these cheap drives with tons of writes have been lasting compared to the testing requirements (especially with so much past write endurance on the one just now starting to exhibit trouble). Part of the impetus for the series seemed to be lots of claims on how quickly to expect massive problems without any actual experimental tests of consumer drives to actually back it up. Of course n=4 with 1 model of 1 brand drives but it's taken ~20x longer than some common assumptions to start seeing problems on a drive at 5x its endurance rating.

jauntywundrkind2• 1 year ago

Note: the worn drives were very worn. 128GB drive with 280TB written is ~2400 cycles. >5x it's 480x rating!

Even though it's a cheap drive, it's rated endurance wasn't really that low. 600 cycles (1200TBW/2TB) is pretty common for consumer SSD: that's higher than 480x but not vastly higher.

Glad folks are chiming in with the temperature sensitivity notes. I have parts coming in for a much bigger home-cloud system, and was planning on putting it in the attic. But it's often >110°F in the summer up there! I don't know how much of a difference that would make, given that the system will be on 24/7; some folks seem to say having it powered on should be enough, others note that usually it's during read that cells are refreshed.

Doing an annual dd if=/nvme0n1 of=/dev/zero bs=$((1024*1024)) hadn't been the plan, but maybe it needs to be!

ilikepi• 1 year ago

Ok I gotta ask...why of=/dev/zero and not of=/dev/null ?

jauntywundrkind2• 1 year ago

That's probably much better, oops!

ahofmann• 1 year ago

Just my 2 cents: I've installed quite a few servers and racks of hardware in very unsuitable spaces. LTO drives in dusty rooms, important disks in hot and cold (summer and winter), dusty and vibrating rooms and so on. I was very worried about them dying way sooner than people would expect. The only thing that died after two years was the LTO drives. Everything else is still running. So I wouldn't worry much about your hot attic. 43° Celsius outside means constantly over 50° inside the server, which sounds horrible but in the last 15 years of my experience nothing died because the room was too hot, or cold, or humid, or shaky. I've installed dust Filters in front of all air intakes, though. My reasoning is, that dust kills way faster than temperature.

userbinator4• 1 year ago

15 years ago the average MLC flash was rated for closer to 5k cycles, not 600.

The fact that even the "fresh" drive, with just 1 cycle, is already showing some degradation is also concerning.

stego-tech2• 1 year ago

I mean, this reads as I expected it to: SSDs as cold storage are a risky gamble, and the longer the drive sits unpowered (and the more use it sees over its life), the less reliable it becomes.

Man, I’d (figuratively) kill for a startup to build a prosumer-friendly LTO drive. I don’t have anywhere near the hardware expertise myself, but I’d gladly plunk down the dosh for a drive if they weren’t thousands of dollars. Prosumers and enthusiasts deserve shelf-stable backup solutions at affordable rates.

0manrho6• 1 year ago

I agree with the general sentiment, but just a headsup for those who might be unaware...

thousands of dollars

I know this varies a lot based on location, but if you're stateside, you can get perfectly functional used LTO6 drives for $200-$500 that support 6TB RW cartridges that you can still buy new for quite cheap (20pk of 6TB RW LTO6 carts[0] for $60), much cheaper than any other storage medium I'm aware of and still reasonably dense and capacious still by modern standards. Sure it's 3-4 generations behind the latest LTO's, and you might need to get yourself a SAS HBA/AIC to plug it in to a consumer system (those are quite cheap as well, don't need to get a full RAID controller), but all in that's still quite an affordable and approachable cold storage solution for prosumers. Is there a reason this wouldn't work for you? Granted, the autoloader/enclosed library units are still expensive as all hell, even the hold ones, and I'd recommend sticking to quantum over IBM/HPE for the drives given how restrictive the later can be over firmware/software, but just wanted to put it out there that you don't necessarily need to spend 4 figures to get a perfectly functional tape solution with many many TB's worth of storage.

0: https://buy.hpe.com/us/en/storage/storage-media/tape-media/l...

throwaway815234• 1 year ago

LTO6 capacity is 2.5TB. 6TB is "compressed" which these days is near worthless because data collections that size are likely to already compressed (example: video).

If the idea is to back up a hard drive, it's not nice to require big piles of tape to do it. Right now even the current LTO generation (LTO9, 18TB native capacity) can't back up the current highest capacity hard drives (28TB or more, not sure). In fact HDD is currently cheaper per TB (20GB drive at $229 at Newegg) than LTO6 tape ($30 for 2.5TB) even if the tape drive is free.

LTO would be interesting for regular users if the current generation drive cost $1000 or less new with warranty rather than as crap from ebay. It's too much of an enterprise thing now.

Also I wonder what is up with LTO tape density. IBM 3592 drives currently have capacity up to 50TB per tape in a cartridge the same size as an LTO cartridge, so they are a couple generations ahead of LTO. Of course that stuff is even more ridiculously expensive.

0manrho6• 1 year ago

Not surprsing. Flash is for hot/warm storage, not for cold storage, but using literal bottom of the bargin-bin barrel no-name drives that are already highly worn really doesn't tell us anything. Even if these were new or in powered on systems for their whole life, I wouldn't have high confidence in their data retention, reliability or performance quite frankly. Granted, there is something to be said about using budget/used drives en masse and brute forcing your way to reliability/performance/capacity on a budget through shear scale, but that requires some finesse and understanding of storage array concepts, best-practices, systems and software. By no means beyond the skills of an average homelaber/HN reader if you're willing to spend a few hours of research, but importantly you would want to evaluate them as an array/bulk, not individually in that instance, else you lose context. That also typically requires a total monetary investment beyond what most homelabbers/consumers/prosumers are willing to invest even if the end-of-the-day TB/$ ratio ends up quite competitive.

There's also many many types of flash drives (well beyond just MLC/TLC/QLC), and there's a huge difference between no names, white labels, and budget ADATA's and the like, and an actual proper high end/enterprise SSD (and a whole spectrum in between). And No, 990/9100 Pro's from Samsung and other similar prosumer drives are not high end flash. Good enough for home gamers and most prosumers, absolutely! They would also likely yield significant improvements vs these levens in OP. I'm not trying to say those prosumer drives are bad drives. They aren't (The levens though, even new in box, absolutely are).

I'm merely saying that a small sample of some of the worst drives you can buy that are already beyond their stated wear and tear is frankly a poor sample to derive any real informed opinion on flash's potential or abilities. TL;DR: This really doesn't tell us much other than "bad flash is bad flash".

ksec• 1 year ago

That is basically saying dont store your data on USB Memory.

Are there any way we could fix those corrupted bit flip files with USB Flash?

asdefghyk• 1 year ago

I wonder how ssd endurance is effected - if at all , by storage temperature. I’m thinking if it was kept at 3 degrees or even colder like frozen. Since chemical reactions proceed slower with a lower temperature. Photographic negatives can have deterioration greatly slowed if stored at some low temperatures. Would need to take steps to avoid condensation. … Would be interesting to know for hard drives to ….

Unpowered SSD endurance investigation finds data loss and performance issues

Related Stories