Memory has grown to nearly two-thirds of AI chip component costs

https://epoch.ai/data-insights/ai-chip-component-cost-shares

by intelkishan • 2 months ago

445 499 2 months ago

slicktux• 2 months ago

I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!

ksec2• 2 months ago

It is one of the thing with consumer when they remember they brought it at the absolutely lowest price point when DRAM maker were bleeding money.

Those are not normal pricing. Before the pricing collapse in early 2020, 96GB DDR5 would have cost about $450 to $500. And I will need to restate again the cost of DRAM hasn't really changed much in the past 20 years. Its price just goes up and down in cycles.

So in reality it is more like going from $500 to $1300. But consumer felt it was more like going from $200 to $1300.

Crucial are already selling DRAM made by CXMT. And China are already throwing money at it. I doubt the memory bubble will burst in next 12-24 months. As in going back to money losing DRAM pricing. As they will all pivot to HBM or other money making products. But the bulk of lower end consumer DDR5 or LPDDR5 will goes to Chinese Foundry. Assuming they have figure out how to do them well. Which they have improved but are still so far away from industry leaders.

Normally memory maker will push the next DDR standard to market just to push out Chinese competitors, I am not sure it will work the same this time around. DDR5 have plenty of other usage / demands.

DoctorOetker9• 2 months ago

Crucial are already selling DRAM made by CXMT.

Crucial was disestablished this year.

cogman10• 2 months ago

Its price just goes up and down in cycles.

Historically the price has always trended downward. When I first got into computing $200 could buy you 128 MB (yes M) of ram. Really nice systems had 512 MB.

That's obviously changed over the decades as process shrinks have lead to higher memory density. We should generally expect that ram will cheaper up and until the point where process shrinks stop happening. They've definitely slowed, but they haven't stopped.

adroitboss• 2 months ago

I paid $279 for crucial 96gb DDR5 5600 MHz SO-DIMM ram October 22 of last year. Amazon has the same kit going for $1,048.90 right now.

Joel_Mckay3• 2 months ago

Nice, you were lucky. =3

burnt-resistor• 2 months ago

CORSAIR Vengeance 96GB (2 x 48GB) SO-DIMM DDR5 5600 CMSX48GX5M1A5600C48

Bought an extra one by accident, paid $218.99 March 2025

Goes for $1400 now. I haven't gotten around to selling it.

bushbaba• 2 months ago

Makes prior assumptions that getting tens of gigs of ram is cheap thrown out the window. Would likely lead to super fast SSDs such as optain being way more valuable

moregrist• 2 months ago

The price of SSDs is similarly depressing.

dawnerd3• 2 months ago

I’m so mad I didn’t max out my main server when I had the chance. Used enterprise sticks were dirt cheap on eBay.

Forgeties795• 2 months ago

Used enterprise HDD’s also jacked up now. It’s absurd lol

glouwbug• 2 months ago

People spend that much a month on restaurants

IshKebab• 2 months ago

I bought a couple of used computers with 256 GB of DDR 4 (total) a year ago. The ram is worth more than I paid for the whole machines now.

DarkUranium• 2 months ago

Someone was selling an Epyc machine with 512GB RAM @ 500 EUR last year. I regret not buying it now ...

giancarlostoro2• 2 months ago

Ramflation

trollbridge4• 2 months ago

I bought 192GB of DDR3 a year ago for literally $60 ($5 a stick). It's about $22 a stick now, so more like $350 today. What on earth is _anybody_ doing with DDR3?

chinathrow• 2 months ago

Being desperate?

manquer• 2 months ago

All memory products use many shared resources in the supply chain, so if there is high demand in one product line, others have to raise prices to compete for the resources or stop making those lines altogether.

That is to say at least you were able to buy them at $350 today, with the current trajectory there will be no supply at all in few months.

jlokier• 2 months ago

Demand for DDR3 is up because people who want DDR5 or DDR4 but can't afford either any more are choosing DDR3 and old DDR3-compatible systems to put it in, instead of what they really want.

kristopolous4• 2 months ago

there's an economic term for this: substitute good. https://en.wikipedia.org/wiki/Substitute_good

zozbot2347• 2 months ago

You could set up swap space on Intel Optane media, it'll be about the same performance as DDR3 and sells for ~$1/GB on the secondary market. Though it will be a lot more power hungry than Flash, let alone DRAM - so not suitable for all uses.

Forgeties795• 2 months ago

2x16gb for $105 total April of 2025. $600 for that now. Makes no sense.

shevy-java3• 2 months ago

My main computer has 64GB. I bought that one in late 2022 or so.

Looking at the current prices, even of the same RAM, is just insane. Those companies really need to pay us compensation damage here. The whole "free market" notion does not work when you have de-facto monopolies and mega-corporations abuse average Joe and average Jane.

jmspring2• 2 months ago

I just found two 4tb Samsung EVO drives - unused - while organizing my garage.

jmspring2• 2 months ago

I forgot to add, I paid ~500 each, Samsung for the same drive is quoting $2k on their site (maybe a new sku). These were bought 2ish years ago. Strange things are a foot at the Circle-K.

journal• 2 months ago

yea, but people now have more money.

rldjbpin2• 2 months ago

paid a bit more than that just for a half-decent 16 gig stick recently :)

i compensate by never paying for AI

positron26• 2 months ago

The algorithm advances are going to crash this so hard.

Coffeewine• 2 months ago

I mean, god willing, but it'll be just as likely that we'll blissfully consume 100 million token contexts in that case.

iamtheworstdev• 2 months ago

isn't there a law for that? as things become cheaper you consume more?

Legend24402• 2 months ago

Or will more efficient algorithms just mean we run even more AI models, increasing the demand for AI chips even more?

Npovview3• 2 months ago

I heard Greg Brockman on a podcast saying they are limited by computer and memory. They have line of sight in solving many different kinds of problems. But they also have to survive in the meantime. Hence the focus on enterprise recently. They could just ask Government to fund them doing other research areas

helterskelter2• 2 months ago

Better algos = more demand

Memory squeeze will get worse before it gets better.

fHr• 2 months ago

classic uneducated algo copium talk

oceansky2• 2 months ago

Awful time for gamers and PC hobbyists not fully into AI.

lacunary• 2 months ago

also for ones fully into AI

aunty_helen3• 2 months ago

This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram.

People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.

johnvanommen6• 2 months ago

Yes, this will definitely renew interest in Stadia type products.

Joel_Mckay3• 2 months ago

Indeed, Gamers Nexus is doing interviews with PC component manufacturers, and some are hurting bad right now. The PC market is no longer in competition, but rather survival mode. =3

https://www.youtube.com/@GamersNexus/videos

themafia• 2 months ago

It's more likely to kill the AI market. They're overbuilding capacity and most of it is going unused. The upcoming haircut is going to kill a lot of the major players.

They've intentionally crafted an unsustainable business model in an effort to get users in the front door and raise their MAUs. We've seen this story before. We should know precisely where it's headed.

throwatdem123112• 2 months ago

Don’t you worry - Microsoft and Amazon will have you covered with cloud streaming.

Can’t afford a computer because they bought up all the supply? They’ll conveniently sell it back to you with a subscription!

You’ll own nothing and be happy.

luqtas• 2 months ago

there's much more than triple A video-games running at 240 Hz on Ultra settings... a 200 USD laptop/computer has enough power to run hundreds of interesting indie games and AAA from the past

Ray20• 2 months ago

I don't understand the threat to the PC market.

Prices haven't risen THAT much and are quite affordable. And if you look at the improved quality of upscalers (DLLS 4.5 for example), gaming is now more affordable than ever, despite the increased cost of components.

Of course, the 5090 prices are insane, as are for SOME memory models, but that's nothing new and represents a fairly small market share.

When I started building gaming pcs, the top top card was 750$ (NZD)

When I started building gaming PC, the top $700 cards didn't even provide comfortable performance or graphics. Back then, you were supposed to have several of this connected SLI or somethin. And even then, it wasn't always reliable, and it resulted in stuttering, lags, and graphical artifacts (in cases when it worked). Today, even $700 graphics cards are a much better product from a user perspective than the high-end cards of that time (and that's not even taking into account that $700 cards back then were much more expensive).

tpurves2• 2 months ago

This has already happened. Home PC market is practically dead already due to memory, ssd and graphics card price inflation. Makers of components like PC cases and power supplies etc. are seeing demand down 30-40% year over year and this is going to put many suppliers out of business. NVDIA has stopped even listing gaming revenue on their earnings reports. Both NVDIA and AMD are not seriously interested in supplying the consumer GPU market anymore either.

The only hope left is really Apple, but even apple has conspicuously delayed the launch of M5-gen mac minis and mac studio. Mostly because even Apple can't source enough DRAM to fully supply all their product lines.

doom2• 2 months ago

It might kill the console gaming market, too. Typically consoles get cheaper over time post-release. Instead, all the latest gen consoles are getting price hikes and at least one company is potentially pushing back the next gen release (PS6). A PlayStation 5 for $900? I'll just wait and be happy with my perfectly usable Switch 1 (since the 2 is also more expensive than it should be).

paulmist2• 2 months ago

I think it's the opposite. Sure in short term hobbyists are getting squeezed, but the amount of capital that they can put into pushing the edge is small compared to Fortune 500. Sooner or later hobbyists will benefit, especially if the market crashes.

baq3• 2 months ago

If it crashes after it kills the PC we’ll be left with… nothing? Path matters as much as destination

oceansky2• 2 months ago

I fully agree, the billion dollar question is when it will come.

deadbabe6• 2 months ago

Here’s the thing, what if memory manufacturers take this opportunity to collude and basically never reduce the price of memory below the current levels since it’s too hard for a new competitor to just rise up and undercut them? Everything I hear about is how hard and risky it is to spin up a new fab.

And by doing this, they ensure local LLMs never become feasible for the vast majority of people and AI companies solidify subscriptions forever.

aurareturn11• 2 months ago

Keeping prices at this level is precisely how one or more competitor will rise up. Making memory isn’t super hard. That’s why it is a commodity. The problem with the memory market is that up and down cycles have bankrupted the vast majority of players in the past. Now we only have 3 players left except for a few smaller ones in China.

The reason memory prices can stay high for years in this mega cycle is because the 3 players will be very cautious on overbuilding. They’d rather under build, make great profit (not maximum) and reduce the risk of going bust if this suddenly ends.

Same for TSMC in chips.

Great opportunity for Chinese companies though. This shortage is exactly what Chinese companies need to scale.

jazzyjackson• 2 months ago

up and down cycles have bankrupted the vast majority of players in the past

Exactly, so what’s the incentive for anyone to sink half a billy into building out more capacity.

The existing players get to rest on their laurels and succeed whether or not the AI bubble busts.

dymk2• 2 months ago

Making memory isn’t super hard.

Then why do only 3 companies make it?

petra• 2 months ago

//Making memory isn’t super hard. That’s why it is a commodity.

These two aren't related.

Dram is a commodity because the you can replace a chip from hynix with a chip from micron, the have the same behaviour.

And a price competitive Dram isn't easy manufacture, or China would have made it already.

shaky-carrousel• 2 months ago

Then China will come and eat their lunch. I for one will only buy Chinese RAM from now on, no matter the prices.

granzymes• 2 months ago

I for one will only buy Chinese RAM from now on, no matter the prices.

Memory is a commodity, so I think you will be very lonely in your quest.

stavros• 2 months ago

Then that's a cartel and hopefully regulators will act.

deadbabe6• 2 months ago

They won’t.

YetAnotherNick• 2 months ago

If the collude to say make the price $1000 for a component that costs them $100(including opportunity costs), then either a new company or a greedy company in the collusion can make their price secretly $900 and get massively more profit.

Right now their opportunity cost is too high.

risky it is to spin up a new fab

You don't need a new fab. You can build memory in 20 years old fab.

elorant4• 2 months ago

Bought a second hand Dell server a week ago. The entire rig with a 12-core CPU and 32GB DDR4 ecc RAM cost as much as I'd pay to buy 64 GB of DDR RAM alone. I hope there's an end to this absurdity soon enough otherwise the pain will affect other markets too. I read the other day that PC case sales have collapsed by more than 40%.

nik2820002• 2 months ago

I feel like by the time the AI bubble bursts the PC market will be irreparably damaged. Manufactures who have been making "enterprise" parts aren't going to go back to making consumer parts because there will be no market for it. And with a glut of datacenters not making any money on slop, they are going to be repurposed for saas, stuff like OnShape but for every application.

Most users don't seem to care about storing everything they generate in cloud services and this could easily be sold as an alternative to owning "expensive" desktop or laptop hardware.

dawnerd3• 2 months ago

They’re going to pivot to you renting desktop cloud compute instead of owning anything.

MattDamonSpace• 2 months ago

“Bubble”

finebalance• 2 months ago

Poor people are already being priced out of cheap phones due to rise in RAM-related unit costs. https://www.cnet.com/tech/mobile/smartphone-sales-to-plummet...

lostlogin2• 2 months ago

It makes me sad for the Neo 2.0. More ram is the only thing stopping me switching to it from a Pro.

Npovview3• 2 months ago

I have an alternative take.

If hyperscalers are using more RAM, and that RAM is not available for consumers, it means all the heavy stuff will happen in the cloud. Why would we want both the hyperscalers and consumers to have RAM simultaneously? Consumers would want more RAM to run local models but then hyperscalers capacity will be unused.

elorant4• 2 months ago

Because RAM isn’t in PCs only. It’s in tablets, phones, laptops, DIY computers like the Raspberry, mini PCs, watches, smart TVs, game consoles, cars, routers, cameras, all smart appliances from refrigerators to washing machines, fitness trackers, printers etc. Cloud services are irrelevant to most of these categories.

MrGilbert2• 2 months ago

I assume that memory manufacturers don’t really care where the money is coming from, as long as the "numbers go up" game is working.

NVIDIA in their recent quarterly report stopped categorizing "Geforce" as a single category, and merged it into "Edge-Computing".

If you are a PC Gamer or PC Enthusiast as I am, then we have some dark times ahead.

reactordev3• 2 months ago

Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all…

Or, we could be fucked.

kg• 2 months ago

If DLSS 5 becomes the norm it's possible that just makes things worse. The DLSS 5 demos required an entire separate card to run the model, though IIRC NVIDIA did claim it would eventually work on a single card. Given what the model is doing (yassifying the whole scene instead of just upscaling/reconstructing) it makes sense to me that it would increase compute demand instead of reduce it like previous versions of DLSS.

MrGilbert2• 2 months ago

From my point of view, I suppose we will enter a "Let AI generate entertainment" era. In which you just might rent everything, including games. No need for a beefy computer at home, you just need a slim endpoint:

"Order yours now, for just $99.99 per month, hardware included! Order today, and you will get three months of 'Office Suite' for free, with a small additional cost of $49.99 after month 4. On a tight budget? Switch to the yearly subscription, and pay comfortably in 18 installments."

amazingamazing• 2 months ago

A commodity rapidly increasing in price. What could go wrong?

KronisLV2• 2 months ago

I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).

johnvanommen6• 2 months ago

What if this is the lowest that prices will ever be?

mrandish• 2 months ago

As Yogi Berra famously said, "It's tough to make predictions, especially about the future." But based on historical tech industry trends, a price increase in one component that's this rapid and extreme, is likely to eventually regress somewhat toward the long-term trend line - even if that trend line experiences a longer-term shift upward.

As always, some interpret certain recent events as reason to conclude "but this time it's different." Occasionally they are correct. But that doesn't change the fact that it's reasonable to assume some of the recent extreme, rapid price inflation is due to shorter term market distortion. It's also pretty clear that some of the recent increase in demand represents a stable increase in the long-term trendline. The question is how much is long-term stable and how much is short-term distortion.

KronisLV2• 2 months ago

Then I will make my build last as long as it can, in protest of that. I do expect at least a performative price drop in the coming years, though.

willis9363• 2 months ago

Then I better divert all of my investment into memory maker stocks.

stringfood• 2 months ago

Memory manufactures don't want your money anymore, Micron just left consumer market 6 months ago and says we want to be B2B from now on, and who can blame him? https://investors.micron.com/news-releases/news-release-deta...

brcmthrowaway• 2 months ago

Anyone invested in Micron stock?

lostlogin2• 2 months ago

Up 700% in a year.

WallstreeetBets has been disturbingly accurate in its predictions - basically anything related to AI.

DoctorOetker9• 2 months ago

It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability?

As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost.

So which is it?

AnotherGoodName• 2 months ago

It’s fab capacity. Fwiw dram is different enough that fabs are not transferable between dram memory and other usages. It’s nice to think ‘wow if they made the current 10nm dram on the latest 2nm processes it’d be much faster’ but it doesn’t work that way. The specific size is needed for the capacitance. Sram can be made on fabs that make other circuitry since it’s transistor not capacitor based but is less dense.

Dram is just extremely specialised.

DoctorOetker9• 2 months ago

I know the differences between SRAM, DRAM, ...

I asked for evidence different people keep feeding me opposite stories: one insists its not fab capacity but wafer competition, with a recent article claiming HBM3E takes 3 times as much wafer area per bit than LPDDR5X. Others tell me the complete opposite: its fab capacity, not wafer shortage.

Do we have citable references to ground either set of claims?

jacekm2• 2 months ago

There is a good article (featured on HN a couple of days ago) that explains the issue: https://davidoks.blog/p/ai-is-killing-the-cheap-smartphone

DoctorOetker9• 2 months ago

And that article is contradicting other voices. If that article were correctly identifying the bottleneck as wafer shortage due to switching to HBM, why is everybody discussing the memory makers instead of the boule growers. Memory makers can expand operations all they can, which makes no sense if wafer supply doesn't follow, and the article is suspicously light on semiconductor boule / wafer mfr's.

So which is the bottleneck: fabs or boule growing?

also consider how most solar panels are monocrystalline silicon, how credible is silicon wafer shortage ... really? there is so much disinformation in this market...

stevenwoo• 2 months ago

This covers it pretty well https://news.ycombinator.com/item?id=48229319, TLDR -memory for AI uses more wafers from same production line as other memory and is more profitable, building new fab very risky historically for companies. The companies have cut production of other memory to favor memory for AI and the market for memory for AI is still unfulfilled so prices still go up for customers of every type.

regularfry4• 2 months ago

Regardless of the specific mechanics of the bottleneck, we know what the proximate source of the problem is: openai locking up 40% of Samsung and SK Hynix wafer capacity for the next few years. That's what triggered the madness.

plipt• 2 months ago

Is there an understanding of what OpenAI intends to do with that memory?

Surely they need GPU capacity and would need memory for those GPUs but OpenAI doesn't build GPUs or any hardware, right? So did they pay to keep the supply locked up, or do they have the ability to put that ram into use?

Legend24402• 2 months ago

I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.

epistasis2• 2 months ago

I'm not sure if they should vertically integrate, it would probably be a better idea to directly fund the expansion of capacity, much like Apple does when they scale up a new technology for iPhones.

However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have.

AI companies claim they will need a ton of massive expansion, but are unwilling to take on the risk of the capital needed for that expansion.

I'm hearing a lot of sad whining from AI folks about how these chip makers are holding them back, but who actually has the money to finance the expansion easily? Chip makers have been through this game far longer, when Sam Altman went around claiming it was time for $7T of fabs the AI companies made it clear that they were willing to make ridiculous claims, eliminating credibility.

What's needed now is for them to funnel a tiny amount of their massive piles of cash into financing fabs directly.

energy123• 2 months ago

Oracle is getting sold because of how much capex they're spending on new data centers in the middle of a high rates environment. It's not like they're stockpiling cash due to doubting AI.

alecco• 2 months ago

[...] better idea to directly fund the expansion of capacity [...] > However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have.

With what money? They have to spend the money they get on hardware ASAP else they are left behind.

jacekm2• 2 months ago

A fab takes years to build even when you have the necessary know-how. If you don't it'll take some additional experimenting before you can compete with the established manufacturers. By the time you can produce a usable chip the shortage might be over.

nicoburns• 2 months ago

Because fabs are about the most complex cutting edge technology out there: the "rocket science" of our day (or one of them). And merely having the money is not sufficient. It would be very easy to blow several billion dollars and end up with nothing to show for it.

Just look at how Intel has struggled to compete in recent years, and they have been in the business for decades.

tjwebbnorfolk4• 2 months ago

Intel struggled because they bet the company that Moore's law was over back in ~2014, and instead of upgrading their fabs to EUV they sent the money back to shareholders.

They forgot Moore's main lesson: only the paranoid survive. They thought they could coast, and it nearly killed them.

treis• 2 months ago

A fab costs a billion dollars (really a lot more) and 5 years. It doesn't do anything for anyone today.

elorant4• 2 months ago

A fab costs $15-20bn and it takes at least five years to build. Plus it requires expertise that none of these companies have.

rcxdude2• 2 months ago

Fab margins are on average super thin compared to the margins of big tech companies, and come with a lot of risk because of that. It's not something they are likely to be keen to integrate.

redanddead2• 2 months ago

Another guy answered it ITT. Intel did that, it’s not great because fabs are expensive and risky and it’s less risky to amortize the cost across multiple customers instead of just yourself

skiing_crawling• 2 months ago

I recently built a system at insane ddr4 prices ($2000 for 256gb). But that’s only after seeing how ddr5 prices were 3-4x that!

preisschild• 2 months ago

Yeah I upgraded all of my systems to DDR5 last year, so now I have to buy for ddr5 memory upgrades.

Joel_Mckay3• 2 months ago

Had to fork over almost $1k for a 64G DDR5 kit a few weeks back. At least AMD chips large L3 cache allows folks to get away with lower grade udimms.

Also had to do an Intel build, and there was no way we were going cudimm at current prices. =3

TheGrassyKnoll• 2 months ago

I wish I had figured that out a year ago. MU up ~10x, SNDK up ~37x. My crystal ball is woefully under performing.

chvid• 2 months ago

Time to let ASML sell to the Chinese memory producers … or not.

mchusma• 2 months ago

Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic.

I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to bankrupt themselves, but it feels like there's got to be some way to push that risk off onto model providers and other people in the ecosystem to allow us to grow ram capacity more like 50% per year.

minraws5• 2 months ago

I mean the biggest risk is Chinese CXML benefits and capturing markets that others are leaving hanging and then being able to compete and push out the others when costs start to normalize.

As for 20-25% growth not being enough, I think it's not that far off, if we assume data center build out plans hit a wall and slow down significantly, and the AI heat starts to cool off.

I don't think 20-25% may be enough in the short term but if the AI build out stops within this year, we have a massive oversupply instead of a under supply.

zx8080• 2 months ago

What is the risk? Competition is good for consumers.

galangalalgol• 2 months ago

Is there any indication research is being focused on reducing menory footprint of inference for frontier class models? Is the low hanging fruit already gone there?

blululu2• 2 months ago

Looking at the history of the memory industry the biggest risk is that a firm would over produce and go bankrupt. Maybe this time is different but so far no memory chip maker has gone under because their competition increased capacity.

DoctorOetker9• 2 months ago

According to the recent article HBM memory is 3x less efficient wafer area wise than LPDDR; but the bandwidth is more than triple.

What if its in everyone's interest to buy computers at say 1/3rd the rate and switch everything over to HBM?

the discrepancy between compute and memory has been growing for ages, perhaps a painful switch to HBM is exactly what we need?

Would you rather have 3 intermediate computers with low memory bandwidth, or wait a little longer statistically so that we can all enjoy a new computer at 1/3rd the rate but much higher bandwidth than the area ratio?

FuckButtons• 2 months ago

These are fundamentally different points in design space though, hbm doesn’t have a 10mw idle draw like lpddr does.

aurareturn11• 2 months ago

Can’t put HBM in smartphones and laptops. The power drain is too great.

thfuran2• 2 months ago

Not many workloads are RAM bandwidth limited. Power and latency are much more common bottlenecks, and HBM loses on both of those.

edg5000• 2 months ago

I hear people are doing AI workloads on apple hardware, which is LPDDR but with a wider memory bus (1024bit). This requires the SoC to support this; from what I understand not many of any beyond Apple offer this. A wider memory bus may be all we need.

foota• 2 months ago

In theory the new futures markets for chip components would help here, since it would allow DRAM suppliers to insulate themselves from that risk.

regularfry4• 2 months ago

The openai deal would be absorbed by two years of that. And it would be inefficient for the RAM makers in a competitive market to leave buyers unsold-to.

I don't actually know what the rate of growth before October was, I'm sure someone round here will though.

Traubenfuchs• 2 months ago

Why did this happen so suddenly?

Why were tech savy investors unable to figure this out when the datacenter craze had already started?

How to explain this lag between quickly rising demand for all datacenter components besides memory?

johnvanommen6• 2 months ago

Nine years after Google's seminal paper lit the fuse on AI, a total lack of manufacturing foresight has trapped over a trillion dollars of incoming capital in a hardware bottleneck.

The entire sector is now facing a critical RAM starvation crisis where memory manufacturers are actively slow-rolling supply just to keep prices high and avoid running out entirely.

This has created an unprecedented supply-and-demand distortion where desperate companies are getting rejected even at a 5x markup, and mission-critical SKUs are skyrocketing to 10x and 20x their baseline value.

It is a macroeconomic squeeze at a staggering scale, and the massive venture scale opportunity lies in capturing the value created by this memory gatekeeper.

From the perspective of an armchair economist, the winners will be the investors who invest in RAM wisely. The losers will likely be cash strapped SAAS companies. They’re almost completely dependent on a fleet of servers in the hyperscalers, and they’re leasing those servers and services. That leaves small SAAS companies exposed to incoming inflation in the cost of hosting.

irthomasthomas• 2 months ago

A lot of words to say that Sam Altman bought up the worlds total supply of ram chips for the next few years.

vb-8448• 2 months ago

Capex expenditure start exploding after covid with the chart going hockey stick at the end of 23/start of 24, almost 2.5 years ago.

A lot of capex is supposed to go into the datacentres, didn't they know that datacentres need to be filled among other stuff with RAM? I wonder if at some point we will discover that there is a shortage of fibre optic cables of SFPs ...

PS: Obviously armchair economist here too ... but for it doesn't seem too difficult to foresee the increase of the demand.

chairmansteve• 2 months ago

"That leaves small SAAS companies exposed to incoming inflation in the cost of hosting".

Which they will pass on to their customers. If their product provides enough value the customers will pay.....

skybrian• 2 months ago

RAM is a boom-and-bust industry, so memory manufacturers were reluctant to invest. Here's a good blog post on the economics:

https://davidoks.blog/p/ai-is-killing-the-cheap-smartphone

Maybe long-term purchase agreements from big buyers might have helped convince them it's okay to build, but apparently it didn't happen.

LPisGood2• 2 months ago

The same reason they didn’t all sell everything to buy NVIDIA the day chatGPT came out

johnvanommen6• 2 months ago

I really don’t want to give anyone ideas, but doesn’t this make the Nvidia 5090 an unbelievably good deal right now?

The VRAM in the 5090 is only made by one country in the world.

The 50xx series is special, because its ram is so dependent on a single commodity. It’s not like a 4090 or a 3090; their VRAM chips have been around for years.

If there’s a shortage or interruption in DDR7 VRAM, it seems like every GPU that requires it would explode in value.

I hope I don’t regret posting this because I’d really like to buy one myself…

mattmanser• 2 months ago

It's gone up like 300% in cost in the last year.

JacobAsmuth2• 2 months ago

Which surely is the highest it'll ever be! You're suggesting that the price will go down in the future? Would love to hear more about your thought process!

johnvanommen6• 2 months ago

I believe msrp is $2000 right?

EnPissant7• 2 months ago

There was only a very brief time it was selling for MSRP (last fall for $2000). Even if you use that as the previous data point, it's only 200% increased.

forrestthewoods2• 2 months ago

if you can buy one!

The RTX 5090 is faster than an H200. It just has less ram (32 vs 141), doesn't have NVLink, and technically isn't allowed to be used in a datacenter.

The datacenter GPUs sell at an 80% margin. They're incredibly overpriced. But the laws of supply and demand are undefeated and so here we all are.

alphabeta3r56• 2 months ago

The RTX 5090 is faster than an H200. It just has less ram

H200 has HBM and much more 64-bit compute

layer8• 2 months ago

An unbelievably good deal at $4000 plus?

johnvanommen6• 2 months ago

Possibly the best deal there is

I really need to shut up, or bite the bullet and by one.

If you graph the tokens per second on the 5090, your jaw will hit the floor at how cheap it is

I_am_tiberius• 2 months ago

It seems to me the max memory you can buy in a laptop stagnated for the past 3 years or so.

giancarlostoro2• 2 months ago

I have always felt insulted that most laptops even offer a low 4 GB of RAM I rather take 16 GB in previous gen memory

ffaccount2• 2 months ago

My several years old laptop has 128GB of RAM, is that not enough? I admit that it's a pretty heavy one.

superkuh• 2 months ago

And the max storage in pre-built computers has stagnated at 2010 levels (~1TB). This was first due to the switch to the much more expensive and much faster charge trap flash. In the 2020s it finally started to approach 2010 sizes in pre-builts but then the corporate finance wars re: fab capacity happened.

rldjbpin2• 2 months ago

for the most part, unless soldered down, it has been hard to find higher than dual channel (maybe quad for a massive odm gaming laptop). each stick and platform having set maximum memory capacity has put a glass ceiling for those machines.

doesn't matter anyway when things are not reasonably priced. i am stuck at the same memory capacity in my personal system for the better part of two decades, partially due to the above and the current pricing today.

gpm4• 2 months ago

An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).

Waterluvian• 2 months ago

What’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?

jmalicki2• 2 months ago

For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.

eldenring3• 2 months ago

2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively

BearOso2• 2 months ago

Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.

gpm4• 2 months ago

Probably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out.

Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily.

The 2x from returning to previous costs is interesting because it's practically guaranteed, and it's on top of everything else. We're just currently "overpaying" (relative to the stable market price) for the manufacture of dram because of a sudden increase in demand.

sandworm101• 2 months ago

Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.

fitblipper• 2 months ago

I have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?

aDyslecticCrow• 2 months ago

Patents is not the issue here. Not even close.

The up-front investment of a memory fab is measured in billions, and takes years to construct and get running. The margin on the chips themselves is terrible, so without scale its not worth even trying. DDR5 is a industry standard that takes some effort to conform to, but the licence fees is a drop in the bucket to the cost of creating a fab.

The fabricators were cautious about increasing production, and slow to start planning. It takes further time to build up capacity, and if the demand drops down, they may end up producing dram at a loss when the market flips over to oversupply. The demand whiplash could kill any company that dared betting on increasing production. See the "bullwhip effect" https://en.wikipedia.org/wiki/Bullwhip_effect which has killed semiconductor fabricators before.

There is a discussion to be had about how to maintain national semiconductor production in Europe and US as a strategic industry, but historic attempts have all failed.

cubefox5• 2 months ago

For some reason I still haven't heard any predictions on when new fabs will come online to meet the current demand. This shouldn't be too hard to find out, since the building time of fabs is very predictable process.

The difficult question is more whether foreseeable memory demand will remain at the current level, grow even further, or shrink again.

wmf2• 2 months ago

No new DRAM fabs are being built. That's why you don't see any predictions.

xadhominemx3• 2 months ago

It's very easy to find out when the new fabs come online. Try asking Claude or ChatGPT.

andrepd2• 2 months ago

I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...

jfim• 2 months ago

That's already the case with say bf16

refulgentis2• 2 months ago

This has been around for quite some time, to the point I had to read this a couple times to understand what you meant. Mighta predated LLMs even.

Dylan168074• 2 months ago

Notoriously terrible?

Posits do a little better if your numbers are biased enough toward 1, but not much better. A 16 bit posit in a near-ideal situation matches an 18 bit IEEE float, and in a pretty wide range of situations loses to either fp16 or bf16.

Training anything at 8 bits is going to be tough, and it's hard to say if the flexible exponent is worth the precision tradeoffs.

shevy-java3• 2 months ago

a path to a ~3x hardware cost reduction

Really?

How long do we have to wait until that ... cost reduction hits us?

gpm4• 2 months ago

For supply to meet demand. Depends very much on how aggressively producers scale and on how demand grows or shrinks.

Safe to say at least a year or two. It'd be shocking if it took a decade.

da_chicken• 2 months ago

All the projections I've seen have said that the earliest we might see the curve flatten is 2030.

It just takes that long to get a fab up and running.

radialstub• 2 months ago

The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.

jayd163• 2 months ago

Apple could always decide to build their own fab or some such thing.

ec109685• 2 months ago

Supply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.

brandensilva• 2 months ago

China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply.

But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer.

I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

mlinsey• 2 months ago

If the existing memory makers retains control of the market and don't defect from the optimal-long-term equilibrium for themselves, that's true. It just takes one player to defect for short term gains as we've seen with some past boom-and-bust cycles. Alternatively, it takes a sufficiently-resourced player with enough incentive to enter the market themselves (NVidia, Google, Amazon, the PRC government through one of many companies...)

itopaloglu83• 2 months ago

Reminds me of how Samsung is giving out $340,000 per person bonuses. Shows you how much of a stronghold they have in market.

djeastm• 2 months ago

Relevant article posted on HN about this a few days ago: https://davidoks.blog/p/ai-is-killing-the-cheap-smartphone

tooltalk• 2 months ago

This is wrong. It is NOT in their nature to keep the market under-supplied -- eg, Samsung, the industry's largest company, was notorious for expanding their capacity during the industry downturn to gain market share while everyone else was cutting back to minimize loss.

I'm guessing you are also probably unfamiliar with the terms like "chicken game" which refers to the cutthroat, high-stakes price wars where dominant semiconductor manufacturers intentionally overproduce and slash prices. This is literally how the industry went from dozens to just three majors today since the 80's.

dev1ycan• 2 months ago

CXMT is scaling up incredibly fast, they are on a clock (south koreans) their monopoly will end relatively soon, although I'm guessing that the AI companies will crash before that anyways.

gimmeThaBeet2• 2 months ago

I struggle to think of a line of business as cyclical as DRAM, maybe like certain kinds of mining would be my only thought.

The DRAM fabs have been on a roundabout for 40 years going from getting accused of price fixing and cartel behavior, to struggling to keep the lights on.

And imo it's not really their fault, it's all the lead time of advanced semiconductors, combined with the commodity dynamics of oil. And the goal is to match that supply to the demand of everything from consumer electronics to more datacenters than you can shake a stick at.

It's maddening to try and solve that, so at this point I really don't fault them for prioritizing survival.

cromka• 2 months ago

What you described only works if the manufacturers agree to price fix. Otherwise, in a free market, they'll race to increase their earnings by meeting the demand.

Ey7NFZ3P0nzAe• 2 months ago

It is in the nature of their business to keep the market under-supplied

What?! If they did an anti competitive agreement sure. Otherwise no as each supplier is incentivized to produce more than its competitor and less than the demand, while divesting just enough to survive the oversupply risk.

overfeed3• 2 months ago

It sure looks like Sam Altman's masterful gambit to corner the memory market has had unforeseen consequences.

roxolotl• 2 months ago

Is any of this actually unforeseen? Buying the vast majority of the world’s supply of something does have mostly predictable consequences.

dragonwriter2• 2 months ago

“Unforeseen consequences” in the same way death of the target is when someone aims a loaded gun at their head and pulls the trigger.

weitendorf5• 2 months ago

If you factor in Nvidia’s profit margin due to the scarcity of the current bleeding-edge chips there is a path to a much larger cost reduction still.

There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point.

In a decade training LLMs to the current level of sophistication, which is in my opinion rather advanced and probably has lots of additional upside just from constructing better RL training regime independently of hardware advancement, will become just as table stakes as running a database is now. I highly recommend everyone look into the Allen Institute’s projects in GitHub and HF because they have open source training materials (including an LLM from scratch off common crawl, and some quite interesting tunes of qwen) to get a taste for what will be in the near future afternoon projects or educational material. The future is going to be wild

oblio2• 2 months ago

These crazy hardware price increases will probably delay everything by at least 2-5 years. Then add at least 5-10 years for all these refinements and optimizations to permeate universally.

Until everything matures, most likely the current iteration of OpenAI and Anthropic will be long gone, along with their current business models.

refulgentis2• 2 months ago

Well, no: manufacturers charge more than input price generally, here specifically, Nvidia wouldn’t lower prices because RAM went down.

fittingopposite5• 2 months ago

Really wondering what this might mean for local LLMs when RAM costs plummet...

fittingopposite5• 2 months ago

either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike

Or the more likely scenario that the AI bubble bursts and the hyperscalars realize they have built too many data centers.

liccil• 2 months ago

What demand? Can't shake the notion that it's fictive considering the amount od data centers being built and GPUs sitting in containers, where they will spend quite some time before being even integrated, even more until used...

willis9363• 2 months ago

This line of thinking makes sense if we're talking about opex like power usage. This is capex though and we'll be financing this overpaying for a long time after the hardware has "aged out". Not really sure there is an upside to it.

Also, inference cost predictions were made before this price jump, so we really haven't started paying for it yet. Inference will not be getting cheaper.

ck2• 2 months ago

if we survive the bubble bursting and there isn't a "too big to fail" bailout with public money manipulation by bought politicians

we are going to have amazing cheap used hardware for a decade

ecommerceguy• 2 months ago

As models gain efficiency, will the need for ram cool?

throwatdem123112• 2 months ago

They’ll just fill up the ram with bigger models. Demand will INCREASE, not decrease.

helterskelter2• 2 months ago

Every time we add capacity with almost anything, we find ways to saturate it.

kingstnap• 2 months ago

Jevons paradox is at play. Right now frontier AI is very expensive which heavily suppresses demand.

If you made it 10x cheaper right now you would see a truly unimaginable wave of bot slop.

shevy-java3• 2 months ago

I think the companies that drive up the prices here, need to pay an extra-tax to all of us. I fail to see why I now have to pay more due to the AI monster companies ruining the economy.

proee• 2 months ago

Memory manufactures sit on a war chest of IP. So even if someone has excess fab capacity and wants to get into memory manufacturing, they will have to fight an uphill battle of about a zillion patents.

Most memory companies have backroom deals to exchange tit-for-tat patent violations against each other.

Not sure how a new memory manufacture comes into being without getting sunk from licensing costs?

byzantinegene2• 2 months ago

china?

notnullorvoid• 2 months ago

Good time to focus on more memory efficient means of training and inference.

SeedLM from Apple is an interesting approach for inference memory efficiency. I'd like to see someone try and build that into training so that it's not a post training compression step.

blindriver• 2 months ago

Since January, I've been lucky and picking up various used DDR4 memory sticks for cheap-ish. I got a total of 64 GB for $180. I feel like I hit the jackpot!

inciampati• 2 months ago

Memory makes computation universal.

cineticdaffodil• 2 months ago

I find it deeply ironic, that iran has blocked helium supply- while it relies on AI created slopaganda to subvert its advesary. Its one of those afterwits of history.

Ylpertnodi• 2 months ago

iran...slopaganda

A US soldier i know commented that the iranian ai slop is "scary and powerful".

flykespice• 2 months ago

Since memory is becoming an expensive commodity, I guess the old ways of being precious on the efficient memory usage of your program (like it running on the constrained 1mb memory back then) are making a comeback.

I only feel sorrow for the electron devs, they will have a hard time.

cloudengineer94• 2 months ago

With how things are going, I'm really wondering how we are gonna tackle the consumer market for things like gaming and machine learning.

No doubt Cloud Gaming is in the cards for the future, only purists like myself with an RTX 5090 will pay premium for offline gaming

weitendorf5• 2 months ago

In the long run cloud gaming is inevitable, it’s just more economically efficient for the cost of the hardware required to render graphics to be amortized across consumers and not sit idle when being unused by collocating them with game assets in POPs.

Once enough gaming compute runs at the edge it also allows for more technically advanced games than would currently be economically feasible (but aren’t made mostly for lack of a market/adoption of cloud gaming and the resulting lack of technical know-how). So I think it will stick and probably end up winning over the holdouts, once the cost of rendering the games they want to play with consumer hardware becomes too large to stomach.

Marsymars2• 2 months ago

You could make the same economic argument for any SaaS, but the margins SaaS providers look for make it so that the only time it isn't cheaper to run your own software/hardware stack in place of SaaS is when the hardware requirements are very low, not high. SaaS makes sense economically when you take into account the admin, compliance, etc. costs... and the admin costs of a Nintendo Switch are pretty low.

willis9363• 2 months ago

Economic efficiency does not win the day because the free market is a myth. Cloud gaming is a technically worse solution because the latency floor is higher. It's a microeconomic disaster (rent vs buy, buy wins). The only reason it would become a thing is if the multinationals succeed in concentrating more wealth and power, which consumers aren't interested in supporting. It's a bad deal and consumers know it. They would have to be forced into it by having the consumer hardware market taken off the table (which is happening and the only possible avenue for a technical regression like cloud gaming to have a market).

luxuryballs• 2 months ago

it’s fun and ironic that “having a memory” is what AI appears to lack the most in practice while at the same time it demands more computer memory than anything to run

abhaynayar• 2 months ago

How can I use this information to MY advantage? Do I started going into something to do with AI chip memory-stuff? If so, how? But just on a software level cause hardware is hard.

Escapade5160• 2 months ago

And four-fiths the cost of a consumer PC build.

maxnevermind• 2 months ago

I wonder if it is reasonable to assume the propagation of shortages further. At first it was GPUs, then RAM, then what?

aceazzameen• 2 months ago

Fresh water?

Jasonwang123• 2 months ago

The cost of memory should continue go up as we tend to have the AI to have context and remember lots more.

ElenaDaibunny• 2 months ago

unified memory architectures are getting more interesting for inference workloads.

IAmGraydon• 2 months ago

Built a new machine with 64GB DDR5 and 5TB SSD in January 2025. It's sheer luck that I dodged that bullet.

emsign• 2 months ago

AI is choking the computing economy. Many companies will die. It's already a mass extinction event and will leave behind deserts.

zeristor• 2 months ago

Or to put it another way, the prices will only come down the other side of an intense catastrophe.

AI growth is locked in now, only if it were to stop will demand be abated.

Memory has grown to nearly two-thirds of AI chip component costs

Related Stories