Intel has screwed up their DC S3500 SSDs

http://utcc.utoronto.ca/~cks/space/blog/tech/IntelDCSSDSectorSizeMistake

by luu • 12 years ago

178 38 12 years ago

utcc.utoronto.ca

fivedogit• 12 years ago

"and of course it completely ruins the day of people who are trying to have and maintain a spares pool"

I used to maintain a distributed system of TV recording servers with hundreds of analog TV tuner cards inside and understand this pain all too well. After years of frustration trying to get these cards and all the different revisions to work together on whatever version of Linux I'd adopted for the system (kernel upgrades were a huge risk), I swore off hardware altogether for future projects. Even though all the devices had the same chipset, I couldn't keep it all working at the same time and it sucked all the time and energy I should have been spending on my actual product.

God bless the rise of cloud computing. Seriously.

I can't even imagine what it must be like to maintain the amount of hardware they have at AWS or Google. Speaking of which, how the fk does a startup like Digital Ocean do it?

toomuchtodo2• 12 years ago

Speaking of which, how the fk does a startup like Digital Ocean do it?

Carefully with DevOps/Networking/Infrastructure folks.

I used to manage several thousand physical Linux servers. It can be done fairly painlessly when you control the environment.

I noticed you mentioned recording servers with analog tuner cards; today it'd be much easier with SDR hardware streaming RTMP streams to cloud servers writing the data out to network attached storage or even S3.

akira2501• 12 years ago

I'm not trying to take the piss here, but why would you use a real-time protocol to stream to an archive? And why SDR instead of a dedicated tuner/decoder?

madengr• 12 years ago

DTV decoding is really processor intensive, and is really only now possible on the fastest processor. You can do it in an FPGA, but those are expensive. ASIC are still the cheapest way to go. It's like hardware vs software H.264 decoding; night and day speed and efficiency difference.

wmf• 12 years ago

Amazon and Google design their own hardware so it is much less likely to change out from under them.

mark-r2• 12 years ago

Do they also design their own components? Because that's what we're talking about here.

Igglyboo• 12 years ago

Honestly it's probably a lot easier at that scale. When you're a single dude maintaining 10 boxes you're fucked if one dies on you. When you're 50 people maintaining tens of thousands of boxes it's not such a big deal.

yingerj• 12 years ago

Also, when you're buying that quantity you qualify the hell out of your hardware. The big boys have groups dedicated to making sure that they get what they want from their suppliers.

tw043• 12 years ago

DO orders from Dell who has contracts in place with their hardware suppliers to supply the same part for the life of the product. Someone like Amazon or Google order enough parts that they have the same setup, just directly with the ODM.

digi_owl• 12 years ago

God bless the rise of cloud computing. Seriously.

In other words, god bless that you can pay someone else to deal with it so you can make the shiny service?

chris_wot• 12 years ago

Well, yes. That's the whole idea.

schtinky• 12 years ago

Pew just released a study that showed people are starting to feel that they've lost control of their private data (which they have):

http://www.pewinternet.org/2014/11/12/public-privacy-percept...

Your vision of the future assumes that folks will remain complacent about how much they trust their doctors, health insurance companies, medical device companies and hospitals with their minute-to-minute activity.

rodgerd3• 12 years ago

I read this at the time and, frankly, my takeaway was more that:

There are applications where 512b drives and 4K drives are not compatible; for example, in some ZFS pools you can't replace a 512b SSD with a 4K SSD

...the ZFS design was fundamentally fucked up. Intel have merely exposed a core design problem, because sooner or later you aren't going to be able to find 512 byte drives at all.

mark-r2• 12 years ago

If you know you can't get suitable replacements, it's an easy problem - copy the data to a new pool and retire the old one. In this case, since the part number didn't change, there was no way to know.

Sector size is a pretty fundamental property of a disk drive.

w0rm• 12 years ago

He mentions it's only in some pool configurations. I imagine there might be configurations where 512b or 4k makes a difference and isn't compatible.

barrkel• 12 years ago

Zfs sector size (ashift parameter) is set at vdev creation time. That is, when you get a bunch of drives together, and you want to have some redundancy or striping, you create a vdev that is composed of several drives, for your desired mode of redundancy (and / or striping). A pool is composed of multiple vdevs; zfs file systems all allocate from a common pool. So it's generally only a problem if you are replacing an existing drive in a vdev after it failed.

Zfs doesn't support a bunch of things. It has no defragmentation. Filling a zfs pool much north of 90% tends to kill its performance even after you delete stuff to bring it back down again. The usual answer to these things is "wipe the pool and restore from a backup", or "zfs send <snapshot> | zfs receive <filesystem>". The answer to changing the sector size of a vdev is similar, just like it is for removing a disk from a vdev, or reconfiguring your redundancy in most cases.

This is just how zfs is currently implemented. It was designed for Sun's customers, for whom having backup for the whole pool, or having a whole second pool to stream to, is not a big deal. Using it in a home or small business context consequently requires more care and forethought.

rodgerd3• 12 years ago

Zfs doesn't support a bunch of things.

I am well aware of this, having been running production systems with it since 2008, shortly after it stopped silently and irretrievably corrupting data.

It was designed for Sun's customers, for whom having backup for the whole pool, or having a whole second pool to stream to, is not a big deal.

The idea that I have to destroy and re-create pools for so many no especially uncommon events is one that runs pretty counter to the way ZFS generally does a good job of being an enterprise filesystem. "Throw it away and restore from backup" is not a good answer.

Rapzid• 12 years ago

I believe 80% is where it switches allocation strategies by default. Ideally you don't want to cross that threshold.

EwanToo• 12 years ago

For those who wonder why, this seems to be a decent explanation of the issue:

http://lists.freebsd.org/pipermail/freebsd-stable/2014-Septe...

So you can have ZFS pools with 4K blocks, it's just if you've chosen 512-bytes at the start, you're going to struggle

protomyth• 12 years ago

Doesn't the current install of FreeBSD default to 4k blocks even on 512-byte drives?

tw043• 12 years ago

Yes. As do all of the illumos derived builds.

Freaky• 12 years ago

No. There is a vfs.zfs.min_auto_ashift sysctl you can poke now instead of messing about with gnop, but it still defaults to 9 (512b).

DiabloD3• 12 years ago

I think ZoL does 4k by default now as well.

ericd• 12 years ago

Hm, does anyone know if this would mess up hardware RAID (specifically LSI MegaRAID running RAID10)?

We have some of these drives in RAID, along with some spares sitting around. All bought around the same time, but who knows if they were manufactured during the transition.

kevin_thibedeau• 12 years ago

All that changed are the default settings. It's unfortunate that Intel didn't document the change but the fix is to add a new step to the drive replacement procedure to reconfigure the firmware. This is a scriptable action. Not that big a deal.

tw043• 12 years ago

It's a minor inconvenience at best. It's not like ZFS let's you add the drive rendering the pool unusable. You get an error telling you it's the wrong sector size, at which point you fix the error and move on with life.

rincebrain• 12 years ago

You...can't, that's kind of the point.

The statement is that you can't rollback the FW rev and it's not an end-user configurable.

nisa• 12 years ago

Totally off-topic but I really enjoy this blog. I've discovered it while struggling with ZFS and btrfs and it's a concise opinionated no bullshit honest sysadmin blog from someone far more knowledgeable as myself. You may not agree but the presentation and style is really great.

rodgerd3• 12 years ago

Chris is one of my must-read sysadmin blogs. Smart guy, good writer, lots of interesting things to say.

IvyMike• 12 years ago

I think anyone who was on the supplier side of Intel's "Copy Exact" requirements would find this particularly ironic.

http://www.intel.com/content/www/us/en/quality/exact-copy.ht...

Intel has screwed up their DC S3500 SSDs

Related Stories