Since when did SSDs need water cooling?

https://www.theregister.com/2023/05/27/liquid_cooled_ssd/

by LinuxBender • 3 years ago

75 72 3 years ago

One of the issues with M.2 in desktop PCs is how buried from the airflow they are, and often they're literally on the exhaust side of the GPU (many GPUs exhaust on both long edges, and many motherboards just so happen to have an M.2 slot under the PEG) or in the air-flow dead-zone between PEG and CPU cooler.

Overall the AT(X) form factor, with extension cards slotting in at a 90° angle, just doesn't work all that well for efficient heat removal. DHE takes away I/O slot space and requires high static pressures (so high fan RPMs), it works for headless servers, but that's about it. The old-fashioned way of a backplane and orthogonal airflow does work much better for stuff like this; but it also requires a card cage and is not very flexible in terms of card dimensions. The one saving grace of ATX is that cards and their cooling solutions can grow in length and height, GPUs are much taller than a normal full-height card, and many are much longer than a full-length card is supposed to be as well.

jackmott422• 3 years ago

This isn't really an issue because everyone's M.2 is working fine. You have to construct absurd scenarios to cause problems. Use a case with bad airflow, a hot GPU, and a workload that is pushing the gpu and m.2 and cpu to their limits indefinitely, which isn't a real life thing.

and if it IS a real life thing because you have some special use case, you use a case with good airflow.

zamadatix• 3 years ago

I think this depends on what the definition of fine and problems is. IMO most don't even notice when their drive throttles due to thermals so it's probably fine that the drives get hot. At the same time, as newer drives keep drawing more and more power, this is going to start to push the limits of "well why did I buy the fast drive in the first place" if they didn't come with these ever increasing cooling solutions as well.

alberth2• 3 years ago

”Overall the AT(X) form factor, with extension cards slotting in at a 90° angle, just doesn't work all that well for efficient heat removal.”

To give an example of this, here’s a server from a huge cloud provider for a brand new AMD 7700 on an ATX board.

Those 90° angles make for horrible airflow.

https://twitter.com/PetrCZE01/status/1637122488025923585

mordae• 3 years ago

Yeah, but X470D4U and similar boards are so overpriced one can somewhat relate to people using gaming boards for servers. Especially since a lot of them route ECC pins nowadays.

I sure wasn't happy paying extra just to have a different board layout with mostly the same components.

Well, there's IPMI at least. Still not worth the price tag.

dx034• 3 years ago

It should be clear that this was only shot for marketing purposes. I don't think they actually run cables like that, but it probably looked better to have cables visible in the picture.

ilyt4• 3 years ago

That look more like "you put powersupply on wrong side".

But yeah, most servers have risers that flip the cards to be parallel to the board.

hgsgm• 3 years ago

That looks like a ribbon cable blocking the fan?

CrimsonRain• 3 years ago

All of it doesn't matter because SSDs like being hot.

ilyt4• 3 years ago

Sure if you dont like your data

coldtea2• 3 years ago

In the same that an ice cream is too cold and could use some heat

Asooka• 3 years ago

I'm water cooling both the CPU and GPU in my PC and have found out that leads to virtually no airflow over the m.2 slot. For now I've simply placed a fan aimed directly at the slot on top of the GPU and that keeps the SSD at a 50 to 60C. I am considering installing a water block on the SSD when I do maintenance next.

numpad02• 3 years ago

People are buying gaming branded PEG propping sticks and sustainer wires because high end PEGs are sagging, and neither the case nor the card support that front slot for full length cards. It’s well past the time for a card cage spec as far as I can see from the user perspective.

jmclnx• 3 years ago

While NAND flash tends to prefer higher temperatures there is nothing wrong with running it closer to ambient temperatures

New one on me :) I did not know NAND liked to be hot, if true does not bode well for laptops for over-clockers.

To me, the end result seems to be, yes and no, up to you. But I still prefer HDD anyway, I am very old school.

formerly_proven3• 3 years ago

NAND is bad for cold storage, because writes cause more wear when it's not warm. Meanwhile data retention benefits from lower temperatures.

adgjlsfhk1• 3 years ago

cold storage by it's nature doesn't have a lot of writes...

jackmott422• 3 years ago

You do not actually prefer HDD, and nothing bodes badly for overclockers or laptops. You are just looking for ways to be contrarian.

EscapeFromNY• 3 years ago

What's wrong with HDD? It's actually quite convenient having time for your morning jog and a shower while you wait for your computer to boot up.

detrites• 3 years ago

Preferring a storage medium for its reliability regardless of the amount of writes it endures is utility - I could see that might be preferable to SSD in some specific case. Maybe there are other upsides, eg, it's often much cheaper.

Regardless, some people drive an old, dangerous, slow, gas-guzzling car - and maintain it at great expense - just because they prefer it. Aesthetic and sentimental appeal is highly personal and knows no bounds.

coldtea2• 3 years ago

Please tell us more about the psychology of the parent commenter.

You seem to have studied it quite well, or perhaps find that ad-hominems make for the best arguments!

Am4TIfIsER0ppos• 3 years ago

I love my HDDs. I just invested in 90TB. That would have cost me a little more than two and a half times as much and be six times as many drives. I do not have the the sata ports or the power connectors for that.

ksec• 3 years ago

There were early PCI-E 5.0 SSD samples pushing closer to theoretical max of 16GB/s but were consuming up to 25W. The current PCI-E 5.0 are only about ~11GB/s. But stays within a 12W power envelop.

I do wonder if we have hit law of diminishing return. With Games optimised for System on PS5 and Xbox's DirectStorage, developer are already showing 80-95% of load time are spent on CPU already.

swarnie• 3 years ago

What's the issue with a jump from 12w to 25w?

wtallis5• 3 years ago

We're talking about an SSD form factor that's 22x80mm and is fed by a couple of card edge pins carrying 3.3V. 12W was already pushing it.

ilyt4• 3 years ago

The M.2 sockets on motherboards don't exactly have great cooling, and SSDs don't usually even come with radiator.

The positioning can also be pretty iffy, mine have one next to CPU, another just under GPU (no chance getting fan there) and those are the "fast" (directly connected to CPU) ones!

The another 2 slots are again under GPU (one filled with wifi/bt card), and only last 2 are far away from other hot components and get its own heatsink. but those are not directly connected to CPU

londons_explore2• 3 years ago

As soon as SSD's are faster, developers will find ways to waste more space and do more IO operations...

mastax• 3 years ago

Maybe we should revive U.2 on consumer desktops. The vast majority of desktops will be fine with only a single M.2 drive, but back in the day the vast majority of desktops were fine with a single SATA drive and yet motherboards commonly came with 8 SATA ports.

jeffbee3• 3 years ago

This author is confused, or the article is just badly written. The thing that draws all the power in these newer SSDs is the controller, not the memory. Shoveling 2 million IOPS to the host CPU is a difficult task … your high-power host CPU can barely keep up. But the article goes on and on about the flash and hardly mentions the controller.

justin66• 3 years ago

This author is confused, or the article is just badly written.

I see you have discovered The Register. Welcome!

zokier2• 3 years ago

I'd question could we not use more the powerful, well-cooled, cpu that we already have in our computers instead of pushing the ssd controller complexity and power ever further. What if we used something like UBIFS or F2FS and removed/simplified FTL?

Sakos2• 3 years ago

SSD controllers are just ASICs. I'm not sure why you want to go from a chip tailored for a specific task to a general purpose one that already has way too much on its plate. Then there's things like latency and how the controller abstracts away the details of how an SSD works internally. All that complexity doesn't go away by putting it on your CPU. You're just moving it from one place to another for no benefit, and adding other complexities.

Could ask the same thing about all the extra silicon in GPUs that adds hardware acceleration for video encoding/decoding.

phire• 3 years ago

Once you get to this level of performance, the bulk of a SSD controller ASIC is essentially just high-performance switching fabric; directing the flow of data between the 8-12 channels of NAND flash, the PCIE bus, it's internal buffers and the DRAM.

If you have any experience with high performance networking equipment, you know that pure switching fabric ASICs generates a lot of heat on its own. Hell, even a dumb 5-port gigabit ethernet switch generates a surprising amount of heat, they are always warm to the touch.

I really doubt that handling the FTL layer on the controller adds that much extra power draw. A dumb PCIE <-> NAND switching ASIC will also have cooling problems.

numpad02• 3 years ago

Speaking from memory and coarse grained understanding,

1) MLC/TLC/QLC work more like 4/8/16-tone grayscale e-paper than flash: e.g. 0x10 = (1,0,1,0), that’s “4 level/bits per cell”. And it’s not a single pulse of 0x10 voltage into a memory cell, more like repetitive pulses from 0b1111 to enough millivolts below 0b1011. Readout is probably more complicated, let alone lifecycle management. Those businesses might be more involved than it’s worth filesystem researchers time.

2) It was often said, at least years ago, that the considerable fraction of heat in NVMe SSDs comes from PCIe serialization/deserialization(SerDes), rather than payload data processing or NAND programming.

If both of above are true, maybe it’s PCIe that should be replaced, with something more like the original PCI?

Out_of_Characte• 3 years ago

Although, with flash memory cells nearing their physical limits in lithography, pretty soon you'll need active cooling for bigger stacks.

marcosdumay• 3 years ago

I doubt one is far from the other.

Shoveling IOPS into a bus is an easily parallelizable problem, while NAND-flash memory has a very high theoretical floor on its capacitance. Any good engineer would optimize the CPU part up to the point where it's only a bit worse than the flash, and stop there because there isn't much gain on going further.

If that's the case, you will see the CPU being the bottleneck on your device, but it's actually the memory that constrains the design.

That is, unless the CPU comes from some off the shelve design that can't be changed due to volume constraints. But I don't think SSDs have that kind of low volume.

zinekeller• 3 years ago

That is, unless the CPU comes from some off the shelve design that can't be changed due to volume constraints.

Most SSDs (with exceptions like Samsung's) simply use SiliconMotion's IP (https://www.siliconmotion.com/products/client/detail) for their controllers.

But I don't think SSDs have that kind of low volume.

If a custom design adds a cent or two to the BOM then it doesn't matter, but when you need to verify that the changes works as intented and that the data isn't corrupted (beyond specifications) that's a lot of cents to be saved. Plus, SiliconMotion can request to TSMC to fabricate it at a lower cost per unit (because there is only one pattern to manufacture) than to customise the controllers for each drive.

KennyBlanken2• 3 years ago

Because the controller and flash are on the same physical device in a very small amount of space, at least somewhat thermally coupled, not just by the PCB, but by "heat spreaders" often sold with the drive or part of the motherboard. Google around and you'll see lots of thermal camera images of M2 drives.

As the article points out, these drives consume up to ~10W under load. That's actually a lot of power for something with very, very little thermal mass - around 10 grams, and a heat capacity of around 400J/KgC is common for PCBs and chips. 0.4J/gC means that for just one second under full load, if the heat is generated evenly across the entire device, it will heat up 2.5 degrees C. Assuming no cooling, that's 24 seconds until it hits is thermal throttling point.

From the article:

The amount of activity taking place on the gumstick-sized M.2 form factor means higher temps not only for the storage controller, but for the NAND flash itself.
NAND, Tanguy explains, is happiest within a relatively narrow temperature band. "NAND flash actually likes to be 'hot' in that 60° to 70° [celcius] range in order to program a cell because when it's that hot, those electrons can move a little bit easier," he explained.
Go a little too hot — say 80°C — and things become problematic, however. At these temps, you risk the SSD's built-in safety mechanisms forcibly powering down the system to prevent damage. However, before this happens users are likely to see the performance of their drives plummet, as the SSD's controller throttles itself to prevent data loss.

FYI, Tanguy according to his linkedin is the principle product engineer for Micron.

frou_dh• 3 years ago

Design priorities probably get warped by doing well at artificial benchmarks/torture-tests in reviews coming to the fore.

wtallis5• 3 years ago

Absolutely. CrystalDiskMark is bad for the consumer SSD market.

tinus_hn• 3 years ago

Unfortunately the alternative is trusting the manufacturers data which leads to cheating.

j16sdiz• 3 years ago

For those don't read the article:

"NAND flash actually likes to be 'hot' in that 60° to 70° [celcius] range in order to program a cell because when it's that hot, those electrons can move a little bit easier," he explained. ... Go a little too hot — say 80°C — and things become problematic

londons_explore2• 3 years ago

Sure... erasure energy is lower when it's hot... But there are lots of other downsides, like a much reduced endurance, and more noise in sense amplifiers meaning there is a higher chance of needing to repeat read operations.

valine• 3 years ago

Does anyone know of a PCIE 5 SSD designed for sustained read/write? Most of these new drives are meant for short bursts of data transfer, I can’t imagine water cooling would be necessary for most drives.

matja• 3 years ago

Kioxia CD8 : https://www.storagereview.com/news/kioxia-cd8-series-pcie-5-...

rowanG077• 3 years ago

I don't understand why pushing 16gb/s requires so much power. A fully custom ic where the data path is in silicon should be able to handle that speed no sweat.

wtallis5• 3 years ago

SSD controllers aren't just moving a lot of data. Between the PCIe PHY and the ONFI PHY there's a lot of other functionality. In particular, doing LDPC decoding at 16GB/s (128Gb/s) is not trivial.

andromeduck• 3 years ago

ECC/Crypto is pretty energy intensive - other bookkeeping like wear leveling and r/w disturb is also quite complicated.

imtringued• 3 years ago

16GB/s over a serial interface with 4 lanes is 32Ghz per lane.

Since when did SSDs need water cooling?

Related Stories