AGI fantasy is a blocker to actual engineering

https://www.tomwphillips.co.uk/2025/11/agi-fantasy-is-a-blocker-to-actual-engineering/

by tomwphillips • 8 months ago

632 641 8 months ago

tomwphillips.co.uk

Etheryte• 8 months ago

Many big names in the industry have long advocated for the idea that LLM-s are a fundamental dead end. Many have also gone on and started companies to look for a new way forward. However, if you're hip deep in stock options, along with your reputation, you'll hardly want to break the mirage. So here we are.

fallingfrog8• 8 months ago

I have some idea of what the way forward is going to look like but I don't want to accelerate the development of such a dangerous technology so I haven't told anyone about it. The people working on AI are very smart and they will solve the associated challenges soon enough. The problem of how to slow down the development of these technologies- a political problem- is much more pressing right now.

chriswarbo• 8 months ago

I have some idea of what the way forward is going to look like but I don't want to accelerate the development of such a dangerous technology so I haven't told anyone about it.

Ever since "AI" was named at Dartmouth, there have been very smart people thinking that their idea will be the thing which makes it work this time. Usually, those ideas work really well in-the-small (ELIZA, SHRDLU, Automated Mathematician, etc.), but don't scale to useful problem sizes.

So, unless you've built a full-scale implementation of your ideas, I wouldn't put too much faith in them if I were you.

titzer3• 8 months ago

Let me guess, you have a truly marvelous demonstration of this proposition which this margin is too narrow to contain.

yahoozoo• 8 months ago

cringe

wild_egg• 8 months ago

They're a dead end for whatever their definition of "AGI" is, but still incredibly useful in many areas and not a "dead end" economically.

bigbuppo2• 8 months ago

Well, except for that needing a 40 year bond for a 3 year technology cycle thing.

hoherd• 8 months ago

"It is difficult to get a man to understand something when his salary depends upon his not understanding it" and "never argue with a man whose job depends on not being convinced" in full effect.

red75prime3• 8 months ago

Many big names in the industry have long advocated for the idea that LLM-s are a fundamental dead end.

There should be papers on fundamental limitations of LLMs then. Any pointers? "A single forward LLM pass has TC0 circuit complexity" isn't exactly it. Modern LLMs use CoT. Anything that uses Gödel's incompleteness theorems proves too much (We don't know whether the brain is capable of hypercomputations. And, most likely, it isn't capable of that).

jimbokun11• 8 months ago

"Fundamental dead end" strikes me as hyperbolic. Clearly they could be an import part of an "AGI" system, even if they're not sufficient for building an AGI in and of themselves?

tim3334• 8 months ago

I figure it's more like steam engines and flight. While steam engines were not suitable for aircraft, experience building them could carry over to internal combustion engines. I imagine something like that with LLMs and AGI.

jamesoofou4• 8 months ago

How can u look at progress in LLMs and think "this is mirage"

NamlchakKhandro2• 8 months ago

Because the depth of concept is so deep, and if you're just beginning then it's going to take a while for that statement to illuminate how obvious it is.

For those who've been sniffing this since early 2010, it's so blindly obvious they've already dropped llms on the floor and moved onto deeper alternative research.

For the rest of us, we're still catching coke bottles from the sky and building places of worship around them

rsbrown• 8 months ago

Name some? Because YeCun's break is the first big name I've seen strike out in a major fashion away from the LLM trajectory.

graphememes• 8 months ago

Okay, so come up with an alternative, it's math, you can also write algorithms.

Filligree• 8 months ago

I can’t test them, though.

gizajob• 8 months ago

Elon thinking Demis is the evil supervillain is hilariously backward and a mirror image of the reality.

captainbland• 8 months ago

"From my point of view the Jedi are evil!" comes to mind.

Cthulhu_• 8 months ago

That one struck me as... weird people on both ends. But this is Musk, who is deep into the Roko's Basilisk idea [0] (in fact, supposedly he and Grimes bonded over that) where AGI is inevitable, AGI will dominate like the Matrix and Skynet, and anyone that didn't work hard to make AGI a reality will be yote in the Torment Nexus.

That is, if you don't build the Torment Nexus from the classic sci-fi novel Don't Create The Torment Nexus, someone else will and you'll be punished for not building it.

[0] https://en.wikipedia.org/wiki/Roko%27s_basilisk

danaris4• 8 months ago

...or, depending on your particular version of Roko's Basilisk (in particular, versions that assume AGI will not be achieved in "your" lifetime), it will punish not you, yourself, but a myriad of simulations of you.

Won't someone think of the poor simulations??

jimbokun11• 8 months ago

It's never been explained to me why a god like AI would care one way or another whether people tried to bring it into being or not. I mean the AGI exists now, hurting the people that didn't work hard enough to bring it into existence won't benefit the AGI in any way.

dist-epoch3• 8 months ago

Why not both.

ArcHound• 8 months ago

"As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment)."

As a businessman, I want to make money. E.g. by automating away technologists and their pesky need for excellence and ethics.

On a less cynical note, I am not sure that selling quality is sustainable in the long term, because then you'd be selling less and earning less. You'd get outcompeted by cheap slop that's acceptable by the general population.

geerlingguy• 8 months ago

I like the conclusion; like for me, Whisper has radically improved CC on my video content. I used to spend a few hours translating my scripts into CCs, and tooling was poor.

Now I run it through whisper in a couple minutes, give one quick pass to correct a few small hallucinations and misspellings, and I'm done.

There are big wins in AI. But those don't pump the bubble once they're solved.

And the thing that made Whisper more approachable for me was when someone spent the time to refine a great UI for it (MacWhisper).

sota_pop2• 8 months ago

Not only whispr, so much of the computer vision area is not as in vogue. I suspect because the truly monumental solutions unlocked are not that accessible to the average person; i.e. industrial manufacturing and robotics at scale.

dghlsakjg2• 8 months ago

I think that LLM hype is hiding a lot of very real and impactful progress in real world/robot intelligence.

An essay writing machine is cool. A machine that can competently control any robot arm, and make it immediately useful is a world-changing prospect.

Moving and manipulating objects without explicit human coded instructions will absolutely revolutionize so much of our world.

jimbokun11• 8 months ago

That's because industrial manufacturing and robotics are failing to bring down costs and make people's lives more affordable.

That's really the only value those technologies provide, so if people aren't seeing costs come down there really is zero value coming from those technologies.

colechristensen• 8 months ago

I think a lot of AI wins are going to end up local and free much like whisper.

Maybe it could be a little bit more accurate, it would be nice if it ran a little faster, but ultimately it's 95% complete software that can be free forever.

My guess is very many AI tasks are going to end up this way. In 5-10 years we're all going to be walking around with laptops with 100k cores and 1TB of RAM and an LLM that we talk to and it does stuff for us more or less exactly like Star Trek.

tomwphillips2• 8 months ago

Author here. Indeed - it would be just as fantastical to deny there has been no value from deep learning, transformers, etc.

Yesterday I heard Cory Doctorow talk about a bunch of pro bono lawyers using LLMs to mine paperwork and help exonerate innocent people. Also a big win.

There's good stuff - engineering - that can be done with the underlying tech without the hyperscaling.

jamesoofou4• 8 months ago

What do you think about the shape of AI progress?

andai3• 8 months ago

I switched to Parakeet the other day.

It's better than Whisper, and faster, while running on CPU on my ten year old ThinkPad.

I had Claude make me Python bindings for it and add it to my voice typing app.

We live in the future.

schnitzelstoat2• 8 months ago

I'm surprised the companies fascinated with AGI don't devote some resources to neuroscience - it seems really difficult to develop a true artificial intelligence when we don't know much about how our own works.

Like it's not even clear if LLMs/Transformers are even theoretically capable of AGI, LeCun is famously sceptical of this.

I think we still lack decades of basic research before we can hope to build an AGI.

ambicapter• 8 months ago

Admitting you need to do basic research is admitting you're not actually <5 years from total world domination (so give us money now).

friendzis2• 8 months ago

Why should they care as long as selling shares of a company selling access to a chatbot is the most profitable move?

csomar3• 8 months ago

Many of the people in control of the capital are gamblers rather than researchers.

wmf4• 8 months ago

If you want to create artificial human intelligence you need to know how the brain works. If you're creating alien intelligence the brain doesn't matter.

ACCount3715• 8 months ago

We are yet to see a pure theoretical roadblock between LLMs and AGI. The way things are going, I wouldn't be surprised if an existing LLM architecture (whether fully transformer-based or one of the hybrids) can hit AGI with the right scale, training and some scaffolding.

On the other hand, extracting usable insights from neuroscience? Not at all easy. Human brain does not yield itself to instrumentation.

If an average human had 1.5 Neuralink implants in his skull, and raw neural data was cheap and easy to source? You bet someone would try to use that for AI tech. As is? We're in the "bitter lesson" regime. We can't extract usable insights out of neuroscience fast enough for it to matter much.

simonw29• 8 months ago

Tip for AI skeptics: skip the data center water usage argument. At this point I think it harms your credibility - numbers like "millions of liters of water annually" (from the linked article) sound scary when presented without context, but if you compare data centers to farmland or even golf courses they're minuscule.

Other energy usage figures, air pollution, gas turbines, CO2 emissions etc are fine - but if you complain about water usage I think it risks discrediting the rest of your argument.

(Aside from that I agree with most of this piece, the "AGI" thing is a huge distraction.)

UPDATE an hour after posting this: I may be making an ass of myself here in that I've been arguing in this thread about comparisons between data center usage and agricultural usage of water, but that comparison doesn't hold as data centers often use potable drinking water that wouldn't be used in agriculture or for many other industrial purposes.

I still think the way these numbers are usually presented - as scary large "gallons of water" figures with no additional context to help people understand what that means - is an anti-pattern.

paulryanrogers3• 8 months ago

Just because there are worse abuses elsewhere doesn't mean datacenters should get a pass.

Golf and datacenters should have to pay for their externalities. And if that means both are uneconomical in arid parts of the country then that's better than bankrupting the public and the environment.

simonw29• 8 months ago

From https://www.newyorker.com/magazine/2025/11/03/inside-the-dat...

I asked the farmer if he had noticed any environmental effects from living next to the data centers. The impact on the water supply, he told me, was negligible. "Honestly, we probably use more water than they do," he said. (Training a state-of-the-art A.I. requires less water than is used on a square mile of farmland in a year.) Power is a different story: the farmer said that the local utility was set to hike rates for the third time in three years, with the most recent proposed hike being in the double digits.

The water issue really is a distraction which harms the credibility of people who lean on it. There are plenty of credible reasons to criticize data enters, use those instead!

jtr13• 8 months ago

I think the point here is that objecting to AI data center water use and not to say, alfalfa farming in Arizona, reads as reactive rather than principled. But more importantly, there are vast, imminent social harms from AI that get crowded out by water use discourse. IMO, the environmental attack on AI is more a hangover from crypto than a thoughtful attempt to evaluate the costs and benefits of this new technology.

roywiggins6• 8 months ago

I don't think there's a world where a water use tax is levied such that 1) it's enough for datacenters to notice and 2) doesn't immediately bankrupt all golf courses and beef production, because the water use of datacenters is just so much smaller.

heymijo• 8 months ago

You're not wrong.

My perspective from someone who wants to understand this new AI landscape in good faith. The water issue isn't the show stopper it's presented as. It's an externality like you discuss.

And in comparison to other water usage, data centers don't match the doomsday narrative presented. I know when I see it now, I mentally discount or stop reading.

Electricity though seems to be real, at least for the area I'm in. I spent some time with ChatGPT last weekend working to model an apples:apples comparison and my area has seen a +48% increase in electric prices from 2023-2025. I modeled a typical 1,000kWh/month usage to see what that looked like in dollar terms and it's an extra $30-40/month.

Is it data centers? Partly yes, straight from the utility co's mouth: "sharply higher demand projections—driven largely by anticipated data center growth"

With FAANG money, that's immaterial. But for those who aren't, that's just one more thing that costs more today than it did yesterday.

Coming full circle, for me being concerned with AI's actual impact on the world, engaging with the facts and understanding them within competing narratives is helpful.

jstummbillig• 8 months ago

In what way are they not paying for it?

reedf1• 8 months ago

Yes - and the water used is largely non-consumptive.

binoct• 8 months ago

Not really. The majority of data center water withdrawal (total water input) is consumed ("lost" to evaporation etc...) with a minority of it discharged (returned in liquid form). I believe it's on the order of 3/4ths consumed, but that varies a lot by local climate and cooling technology.

There's lots of promising lower-consumption cooling options, but seems like we are not yet seeing that in a large fraction of data centers globally.

lynndotpy4• 8 months ago

Farmland, AI data centers, and golf courses do not provide the same utility for water used. You are not making an argument against the water usage problem, you are only dismissing it.

simonw29• 8 months ago

Right, I think a data center produces a heck of a lot more economic and human value in a year - for a lot more people - than the same amount of water used for farming or golf.

Aransentin2• 8 months ago

Growing almonds uses 1.3 trillion gallons of water annually in California alone.

This is more than 4 times more than all data centers in the US combined, counting both cooling and the water used for generating their electricity.

What has more utility: Californian almonds, or all IT infrastructure in the US times 4?

dist-epoch3• 8 months ago

That is correct, AI data centers deliver far more utility per unit of water than farm/golf land.

fodkodrasz• 8 months ago

Water usage largely depends on the context, if the water source is sustainable, and if it is freshwater.

Of course water used up will eventually evaporate, and produce rainfall in the water cycle, but unfortunately at many places "fossil" water is used up, or more water used in an area then the watershed can sustainably support.

This is a constant source of miscommunication about water usage, and that of agriculture also. It is very different to talk about the water needs to raise a cow in eg. Colorado and in Scotish highlands, but this is usually removed from the picture.

The same context should be considered for datacenters.

simianwords6• 8 months ago

who are you to determine the utility? we have the market for it and it has spoken.

asdfman1239• 8 months ago

They are making an anti-disruption argument.

I think it's bad though to be against growth, for reasons I've described in another comment.

dlord2• 8 months ago

I think the water usage argument can be pertinent depending on the context.

https://www.bbc.com/news/articles/cx2ngz7ep1eo

https://www.theguardian.com/technology/2025/nov/10/data-cent...

https://www.reuters.com/article/technology/feature-in-latin-...

simonw29• 8 months ago

That BBC story is a great example of what I'm talking about here:

A small data centre using this type of cooling can use around 25.5 million litres of water per year. [...]
For the fiscal year 2025, [Microsoft's] Querétaro sites used 40 million litres of water, it added.
That's still a lot of water. And if you look at overall consumption at the biggest data centre owners then the numbers are huge.

That's not credible reporting because it makes no effort at all to help the reader understand the magnitude of those figures.

"40 million litres of water" is NOT "a lot of water". As far as I can tell that's about the same annual water usage as a 24 acre soybean field.

jordanb• 8 months ago

Water can range from serious concern to NBD depending on where the data center is located, where the water is coming from, and the specific details of how the data center's cooling systems are built.

To say that it's never an issue is disingenuous.

Additionally one could image a data center built in a place with a surplus of generating capacity. But in most cases, it has a big impact on the local grid or a big impact on air quality if they bring in a bunch of gas turbines.

fny2• 8 months ago

I'm personally excited for when the AGI-nauts start trotting out figures like...

An H100 on low-carbon grid is only about 1–2% of one US person’s total daily footprint!

The real culprit is humans after all.

AndrewKemendo6• 8 months ago

Humans have been measuring between human only vs augmented labor for literal centuries.

Frederick Taylor literally invented the process you describe in his “principles of scientific management”

This is the entire focus of the Toyota automation model.

The consistent empirical pattern is:

Machine-only systems outperform humans on narrow, formalizable tasks.

Human-machine hybrid systems outperform both on robustness, yieldjng higher success probability

Good enough?

dimitrios1• 8 months ago

sound scary when presented without context

It's not about it being scary, its about it being a gigantic, stupid waste of water, and for what? So that lazy executives and managers can generate their shitty emails they used to have their comms person write for them, so that students can cheat on their homework, or so degens can generate a video of MLK dancing to rap? Because thats the majority of the common usage at this point and creating the demand for all these datacenters. If it was just for us devs and researchers, you wouldn't need this many.

simonw29• 8 months ago

Whether it's a "gigantic" waste of water depends on what those figures mean. It's very important to understand if 25 million liters of water per year is a gigantic number or not.

thewebguyd5• 8 months ago

And also, none of those current use cases are a real benefit to society, outside of maybe research cases.

The only benefit is to the already wealthy owner class that is itching to not have to pay for employees anymore because it impacts their bottom line (payroll is typically the largest expense).

It's not like we are making robots to automate agriculture and manufacturing to move toward a post scarcity, moneyless society, which would have real benefits. No, instead we have AI companies hyping up a product whose purpose (according to them) is so that already wealthy people can hoard more wealth and not have to pay for employees. It's promising to take away a large portion of the only high-paying jobs we have left for the average person without an advanced degree.

Me being able to write software a little faster, without hiring a junior, is a net negative to society rather than a benefit.

HDThoreaun5• 8 months ago

Its not gigantic and its not a waste. Brainrot creates massive economic value that can be used to pay people for products you are more happy to consume.

BeFlatXIII5• 8 months ago

Seems the problem is the revealed preference of the normies, rather than the technology itself.

buellerbueller5• 8 months ago

Fine, fine: get rid of golf courses too.

As for food production; that might be important? IDK, I am not a silicon "intelligence" so what do I know? Also, I have to "eat". Wouldn't it be a wonderful world if we can just replace ourselves, so that agriculture is unnecessary, and we can devote all that water to AGI.

TIL that the true arc of humanity is to replace itself!

simonw29• 8 months ago

See comment here: https://news.ycombinator.com/item?id=45927268

Given the difference in water usage, more data centers does not mean less water for agriculture in any meaningful way.

If you genuinely want to save water you should celebrate any time an acre of farm land is converted into an acre of data center - all the more water for the other farms!

slightwinder2• 8 months ago

but if you compare data centers to farmland or even golf courses they're minuscule.

People are critical of farmland and golf courses, too. But Farmland at least has more benefit for society, so they are more vocal on how it's used.

randallsquared• 8 months ago

The problem is more one of scale: a million liters of water is less than half of a single Olympic-sized swimming pool. A single acre of alfalfa typically requires 4.9 - 7.6 million liters a year for irrigation. Also, it's pretty easy to recycle the data center water, since it just has to cool and be sent back, but the irrigation water is lost to transpiration and the recycling-by-weather process.

So, even if there's no recycling, a data center that is said to consume "millions" rather than tens or hundreds of millions is probably using less than 5 acres of alfalfa in consumption, and in absolute terms, this requires only a swimming-pool or two of water per years. It's trivial.

the__alchemist• 8 months ago

I will go meta into what you posted here: That people are classifying themselves as "AI skeptics". Many people are treating this in terms of tribal conflict and identity politics. On HN, we can do better! IMO the move is drop the politics, and discuss things on their technical merits. If we do talk about it as a debate, we can do it when with open minds, and intellectual honesty.

I think much of this may be a reaction to the hype promoted by tech CEOs and media outlets. People are seeing through their lies and exaggerations, and taking positions like "AI/LLMs have no values or uses", then using every argument they hear as a reason why it is bad in a broad sense. For example: Energy and water concerns. That's my best guess about the concern you're braced against.

techblueberry• 8 months ago

I mean, it is intellectually honest to point out that the AI debate at the point is much more a religious or political than strictly technical really. Especially the way tech CEOs hype this as the end of everything.

HardCodedBias• 8 months ago

It seems like there is a very strong correlation between identity politics and "AI skepticism."

I have no idea why.

I don't think that the correlation is 1, but it seems weirdly high.

Flavius2• 8 months ago

Expecting a purely technical discussion is unrealistic because many people have significant vested interests. This includes not only those with financial stakes in AI stocks but also a large number of professionals in roles that could be transformed or replaced by this technology. For these groups, the discussion is inherently political, not just technical.

lkey3• 8 months ago

| Drop the politics

Politics is the set of activities that are associated with making decisions in groups, or other forms of power relations among individuals, such as the distribution of status or resources.

Most municipalities literally do not have enough spare power to service this 1.4 trillion dollar capital rollout as planned on paper. Even if they did, the concurrent inflation of energy costs is about as political as a topic can get.

Economic uncertainty (firings, wage depression) brought on by the promises of AI is about as political as it gets. There's no 'pure world' of 'engineering only' concerns when the primary goals of many of these billionaires is leverage this hype, real and imagined, into reshaping the global economy in their preferred form.

The only people that get to be 'apolitical' are those that have already benefitted the most from the status quo. It's a privilege.

magicalist• 8 months ago

I will go meta into what you posted here: That people are classifying themselves as "AI skeptics"

The comment you're replying to is calling other people AI skeptics.

Your advice has some fine parts to it (and simonw's comment is innocuous in its use of the term), but if we're really going meta, you seem to be engaging in the tribal conflict you're decrying by lecturing an imaginary person rather than the actual context of what you're responding to.

hamdingers4• 8 months ago

IMO the move is drop the politics, and discuss things on their technical merits.

I'd love this but it's impossible to have this discussion with someone who will not touch generative AI tools with a 10 foot pole.

It's not unlike when religious people condemn a book they refuse to read. The merits of the book don't matter, it's symbolic opposition to something broader.

teknikalm• 8 months ago

On HN, we can do better! IMO the move is drop the politics, and discuss things on their technical merits.

Zero obligation to satisfy HN audience; tiny proportion of the populace. But for giggles...

Technical merits: there are none. Look at Karpathy's GPT on Github. Just some boring old statistics. These technologies are built on top of mathematical principles in textbooks printed 70-80 years ago.

The sharding and distribution of work across numerous machines is also a well trodden technical field.

There is no net new discovery.

This is 100% a political ploy on the part of tech CEOs who take advantage of the innumerate/non-technical political class that holds power. That class is bought into the idea that massive leverage over resource markets is a win for them, and they won't be alive to pay the price of the environmental destruction.

It's not "energy and water" concerns, it's survival of the species concerns obfuscated by socio-political obligations to keep calm carry on and debate endlessly, as vain circumlocution is the hallmark of the elders whose education was modeled on people being VHS cassettes of spoken tradition, industrial and political roles.

IMO there is little technical merit to most software. Maps, communication. That's all that's really needed. ZIRP era insanity juiced the field and created a bunch of self-aggrandizing coder bros whose technical achievements are copy-paste old ideas into new syntax and semantics, to obfuscate their origins, to get funded, sell books, book speaking engagements. There is no removing any of this from politics as political machinations gave rise to the dumbest era of human engineering effort ever.

The only AI that has merit is robotics. Taking manual labor of people that are otherwise exploited by bougie first worlders in their office jobs. People who have, again with the help of politicians, externalized their biologies real needs on the bodies of poorer illiterates they don't have to see as the first-world successfully subjugated them and moved operations out of our own backyard.

Source: was in the room 30 years ago, providing feedback to leadership how to wind down local manufacturing and move it all over to China. Powerful political forces did not like the idea of Americans having the skills and knowledge to build computers. It ran afoul of their goals to subjugate and manipulate through financial engineering.

Americans have been intentionally screwed out of learning hands on skills with which they would have political leverage over the status quo.

There is no removing politics from this. The situation we are in now was 100% crafted by politics.

turtlesdown11• 8 months ago

Your context is a little lacking. Golf courses almost universally have retention ponds/wells/etc at the facility and recycle their water.

Only 14% use municipal water systems to draw water. https://www.usga.org/content/dam/usga/pdf/Water%20Resource%2...

simonw29• 8 months ago

"Presented by the USGA" (the United States Golf Association) gave me a wry chuckle there.

That said, here are the relevant numbers from that 2012 article in full:

Most 18-hole golf facilities utilize surface waters (ponds, lakes) or on-site irrigation wells. Approximately 14 percent of golf facilities use water from a public municipal source and approximately 12 percent use recycled water as a source for irrigation.
Specific water sources for 18-hole courses as indicated by participants are noted below:
52 percent use water from ponds or lakes.
46 percent use water from on-site wells.
17 percent use water from rivers, streams and creeks.
14 percent use water from municipal water systems.
12 percent use recycled water for irrigation.

efsavage• 8 months ago

The water argument rings a bit hollow for me not due to whataboutism but more that there's an assumption that I know what "using" water means, which I am not sure I do. I suspect many people have even less of an idea than I do so we're all kind of guessing and therefore going to guess in ways favorable to our initial position whatever that is.

Perhaps this is the point, maybe the political math is that more people than not will assume that using water means it's not available for others, or somehow destroyed, or polluted, or whatever. AFAIK they use it for cooling so it's basically thermal pollution which TBH doesn't trigger me the same way that chemical pollution would. I don't want 80c water sterilizing my local ecosystem, but I would guess that warmer, untreated water could still be used for farming and irrigation. Maybe I'm wrong, so if the water angle is a bigger deal than it seems then some education is in order.

leoedin• 8 months ago

If water is just used for cooling, and the output is hotter water, then it's not really "used" at all. Maybe it needs to be cooled to ambient and filtered before someone can use it, but it's still there.

If it was being used for evaporative cooling then the argument would be stronger. But I don't think it is - not least because most data centres don't have massive evaporative cooling towers.

Even then, whether we consider it a bad thing or not depends on the location. If the data centre was located in an area with lots of water, it's not some great loss that it's being evaporated. If it's located in a desert then it obviously is.

HPsquared4• 8 months ago

Put that way, any electricity usage will have some "water usage" as power plants turn up their output (and the cooling pumps) slightly. And that's not even mentioning hydroelectric plants!

overgard• 8 months ago

I suppose instead we can talk about people's 401k's being risked in a market propped up by the AI bubble.

simonw29• 8 months ago

Absolutely.

thewebguyd5• 8 months ago

We really need to stop tying people's retirement to the market. I've already lost ground due to 2008, and COVID, and while I was young, I lived through my parents suffering through dotcom as well.

It's long past time we have a universal public pension, funded at least in part with a progressive wealth tax, or least go back to private defined benefit pensions to where the employer or pension fund bears the risk rather than the individual.

Supplement it with market speculation if you want, but we need something guaranteed for everyone that's a little more robust and provides a better living than the paltry social security we have now.

tomwphillips2• 8 months ago

Hey Simon, author here (and reader of your blog!).

I used to share your view, but what changed my mind was reading Hao's book. I don't have it to hand, but if my memory serves, she writes about a community in Chile opposing Google building a data centre in their city. The city already suffers from drought, and the data centre, acccording to Google's own assessment, would abstract ~169 litres of water a second from local supplies - about the same as the entire city's consumption.

If I also remember correctly, Hao also reported on another town where salt water was being added to municipal drinking water because the drought, exacerbated by local data centres, was so severe.

It is indeed hard to imagine these quantities of water but for me, anything on the order of a town or city's consumption is a lot. Coupled with droughts, it's a problem, in my view.

I really recommend the book.

AndrewKemendo6• 8 months ago

None of which have to do with AI or AGI.

Nestle is and has been 10000x worse for global water security than all other companies and countries combined because nobody in the value chain cares about someone else’s aquifer.

It’s a social-economic problem of externalities being ignored , which transcends any narrow technological use case.

What you describe has been true for all exported manufacturing forever.

HL33tibCe7• 8 months ago

The fact that certain specific data centres are being proposed or built in areas with water issues may be bad, but it does not imply that all AI data centres are water guzzling drain holes that are killing Earth, which is the point you were (semi-implicitly) making in the article.

asdfman1239• 8 months ago

I'm conflicted. Zooming out, the problem isn't with AI specifically but economic development in general. Everything has a side effect.

For decades we've been told we shouldn't develop urban centers because of how it development affects local communities, but really it just benefited another class of elites (anonymous foreign investors), and now housing prices are impoverishing younger generations and driving homelessness.

Obviously that's not a perfect comparison to AI, which isn't as necessary, but I think the anti-growth argument isn't a good one. Democracies need to keep growing or authoritarian states will take over who don't care so much about human rights. (Or, authoritarian governments will take over democracies.)

There needs to be a political movement that's both pro-growth and pro-humanity, that is capable of making hard or disruptive decisions that actually benefits the poor. Maybe that's a fantasy, but again, I think we should find ways to grow sustainably.

pxc• 8 months ago

IIRC Google chose to pull out altogether to punish the locals for standing up to them— even though they happily built air-cooled data centers elsewhere.

simonw29• 8 months ago

Thanks for the recommendation, added to my reading list.

jimrandomh• 8 months ago

The mistake you are making is letting the author choose your points of comparison, without having a high-level picture of where water usage goes. Comparing water usage to a city is misleading because cities don't use much water; large-scale water use is entirely dominated by agriculture.

simianwords6• 8 months ago

An update: there seems to be a critical error in the reporting of metrics in this book for the water issue. https://andymasley.substack.com/p/empire-of-ai-is-wildly-mis....

I do think the whole water issue as pointed out in the book is completely discredited with this call out.

The author Karen Hao acknowledges this:

https://x.com/_KarenHao/status/1990412118584144113

snowwrestler• 8 months ago

Water and power are local issues. And data center use of water and power is already, currently having local impact on politics. I saw ads about it during the election cycle that just concluded. Candidates had to answer questions about it at debates and in interviews.

People are using these arguments for the simple reason that they demonstrably resonate with average people who live near data centers.

They probably don’t resonate with people who have plenty of income and/or do not compete with data centers locally for resources.

cesarb2• 8 months ago

Some time ago, I read the environmental impact assessment for a proposed natural gas thermal power plant, and in it they emphasized that their water usage was very low (to the point that it fit within the unused part of the water usage allowance for an already existing natural gas thermal power plant on the same site) because they used non-evaporative cooling.

What prevents data centers from using non-evaporative cooling to keep their water usage low? The water usage argument loses a lot of its relevant in that case.

josefx• 8 months ago

Does it route the hot water back into a river?

In europe several power plants get shut down each summer because the heated water from those plants would have significant impact on the local wildlife.

swyx• 8 months ago

data centers often use potable drinking water

hmm why exactly? mineral content?

simonw29• 8 months ago

Yeah I think it's to avoid mineral buildup on the cooling equipment which would then need to be replaced more often.

cruffle_duffle2• 8 months ago

I went down that “water use” rabbit hole a month ago and basically… it’s just a bunch of circular reporting that was based on some paper published in 2023[1]. For ChatGPT 3.5 they claimed “500ml for 10-50 responses”. In 2024, Washington Post published an article that took their claim and said “519 milliliters per email”[2] but didn’t source that from the original paper… that “shocking headline” took off and got widely circulated and cited directly, treating the WaPo calculation as if it were the original research finding. Then tech blogs and advocacy sites ran with it even harder, citing each other instead of any actual research[3].

If you look at the original paper they are quite upfront with the difficulty of estimating water use. It’s not public data—in fact it’s usually a closely held trade secret, plus it’s got all kinds of other issues like you don’t know where the training happened, when it happened, what the actual cooling efficiency was, etc. The researchers were pretty clear about these limitations in the actual paper.

Basically, it’s urban legend at this point. When OpenAI’s CEO later said ChatGPT uses ~0.3ml per query, that’s roughly 100x less than the viral claims.

[1] <https://arxiv.org/abs/2304.03271> [2] <https://www.washingtonpost.com/technology/2024/09/18/energy-...> [3] <https://www.seangoedecke.com/water-impact-of-ai>/

netdevphoenix3• 8 months ago

Tip for AI skeptics

Assumptions you are making:

- AI = transformer ANNs

- People sceptical of transformer ANNs directly leading to AGI within any reasonable period are also sceptical of transformer ANNs directly leading to AGI any time in the far future

This kind of generalisations don't help you as the huge number of comments underneath yours likely shows

simonw29• 8 months ago

I don't think anyone who read my comment here misunderstood my usage of the term "AI skeptic" as applying to any form of machine learning as opposed to modern generative AI.

gloomyday• 8 months ago

One of the biggest problems in environmental education (imo) is the lack of notion regarding the footprint of products and services we consume daily; from the water and CO2 costs of a meal, of a heated pool, of a car, etc. It is eye-opening.

I first came across this type of info with the book "How Bad Are Bananas", from Mike Berners-Lee. I really enjoyed it, and I just saw that the new edition even includes stuff like hosting a World Cup, data centers, and space tourism!

It should give a good foundation to start talking about it.

bwhiting2356• 8 months ago

Water location matters. Is the data center in a desert with scarce potable water for locals? Or is next to a large Canadian lake, plenty of potable water, with people who want to trade something for currency so they can put avocados in their salad?

hadlock• 8 months ago

A lot of data centers are near the Columbia river, as power is cheap there thanks to hydroelectric; which flows through an arid desert-like region, but is also the largest river in the western US and it's simply impossible to pump too much water out of it.

runarberg• 8 months ago

Arguments in isolation are usually poor. The water usage arguments usually (always?) comes along with a bunch of other arguments, including power consumption, workers rights, consumer protection, public safety, enshittifcation, etc.

When evaluating the economical cost or morality of a thing, (just like when training a machine learning model) the more data you consider the more accurate the result (although just like statistical modelling it is worth to be wary of overfitting).

seemaze• 8 months ago

On the water front, in my area agriculture flood irrigates with pristine aquifer water, while surface water gets treated and dumped into the drinking supply. This is due to the economics of historic water rights.

lsy• 8 months ago

It's disheartening that a potentially worthwhile discussion — should we invest engineering resources in LLMs as a normal technology rather than as a millenarian fantasy? — has been hijacked by a (at this writing) 177-comment discussion on a small component of the author's argument. The author's argument is an important one that hardly hinges at all on water usage specifically, given the vast human and financial capital invested in LLM buildout so far.

john01dav• 8 months ago

Another issue is that you could, in principle, build data centers in places where you don't need to evaporate water to cool them. For example, you could use a closed loop water cooling system and then sink that heat into the air or into a nearby body of water. OVH's datacenter outside Montreal¹ does this, for example. You can also use low carbon energy sources to power the data center (nuclear or hydro are probably the best because their production is reliable and predictable).

Unlike most datacenters, AI datacenters being far away from the user is okay since it takes on the order of seconds to minutes for the code to run and generate a response. So, a few hundred milliseconds of latency is much more tolerable. For this reason, I think that we should pick a small number of ideal locations that have a combination of weather that permits non-sub-ambient cooling and have usable low carbon resources (either hydropower is available and plentiful, or you can build or otherwise access nuclear reactors there), and then put the bulk of this new boom there.

If you pick a place with both population and a cold climate, you could even look into using the data center's waste heat for district heating to get a small new revenue stream and offset some environmental impact.

1: https://www.youtube.com/watch?v=RFzirpvTiOo

quickthrowman• 8 months ago

I did some napkin math on data center water usage for a 500MW data center in the Columbia River valley.

It uses as much water per year as 200 acres of alfalfa in California’s Central Valley. There are around 1M acres of alfalfa growing in California.

2.5MW of data center capacity is roughly equal to 1 acre of irrigated alfalfa in water usage. If you’re pulling fossil aquifer water, open loop evaporative cooling may not be the best idea, but there are plenty of places east of 100 degrees west in the US that have virtually ‘unlimited’ water where cooling towers are a great idea since they almost double the COP of a chilled water system.

thatoneengineer• 8 months ago

The nice thing about the data center water usage panic is that whenever someone appeals to it, I immediately know that either they haven't done their homework or they're arguing in bad faith.

jmccaf• 8 months ago

I think the water use arguments are relevant, particularly in regions of the world and US (CA) where potable water is scarce, but land and electricity are available .

NYT article gift link where people reported wells ran dry after data centers moved in. : 'From Mexico to Ireland, Fury Mounts Over a Global A.I. Frenzy' https://www.nytimes.com/2025/10/20/technology/ai-data-center...

From https://www.eesi.org/articles/view/data-centers-and-water-co... , I understand there are two types of cooling with water in DCs, open-loop that's simple but water-intensive, and closed-loop that's expensive but efficient.

> This can be achieved through air cooling using water evaporation, which is an open-loop and more water-intensive method, or through server liquid cooling.

AgentME2• 8 months ago

The reported case about water wells running dry had to do with issues in construction rather than anything about the data center's regular operation:

But the reason their taps ran dry (which the article itself says) was entirely because of sediment buildup in groundwater from construction. It had nothing to do with the data center’s normal operations (it hadn’t begun operating yet, and doesn’t even draw from local groundwater). The residents were wronged by Meta here and deserve compensation, but this is not an example of a data center’s water demand harming a local population.

https://andymasley.substack.com/p/the-ai-water-issue-is-fake...

keeda5• 8 months ago

This article goes into an ungodly amount of detail -- with receipts -- making the case that the AI water issue is fake:

https://andymasley.substack.com/p/the-ai-water-issue-is-fake

kragen5• 8 months ago

Data center cooling towers have to use fresh rather than salt water, but they don't care about bacterial contamination or toxic traces of arsenic, antimony, and fluorine. Agriculture also has to use fresh rather than salt water. I can't think of any circumstances where water that was usable for agriculture wouldn't also be usable for cooling data centers—except when the farmer owns the water and the data center operator doesn't.

I also think the energy usage stuff is kind of nonsense. If energy usage is a major part of your operating expenses, you're probably going to locate your data center where energy is cheap, and cheap energy is always renewable. I'm sure you can find data centers that run off coal plants or other thermal power, but thermal power costs in the neighborhood of 100¢ per peak watt, while solar cells cost 12¢ per peak watt, so thermal power won't be competitive for very long.

buu7007• 8 months ago

Not to mention that this is likely a very short-term debate. Modern AI/ML/compute has been a huge boost to fusion R&D, and tons of AI/tech capital is flowing into companies like Helion and CFS. Necessity is the mother of invention and all.

It may be a valid criticism today, but no one will be complaining about AI's environmental impact after those first few plants go live and mass production begins within the next decade. Knock on wood.

ineedasername4• 8 months ago

that comparison doesn't hold as data centers often use potable drinking water that wouldn't be used in agriculture or for many other industrial purposes.

I think you're still good on your original assertion, it seems many/most of the biggest players are using non potable in new facilities and also retrofitting some old ones to avoid potable water as well [1]

I think you'd be good either way: The distinction sounds like an important point until you realize that the cost of turning raw water potable is so vanishly small compared to the cost of these data centers. Some rough estimates place it as less than one single rack of a GB200 NVL72 to build enough-- or more economically, bolster the local existing plants for raw water processing. Even if they had to go to brackish water desalination the cost there looks to be mostly in ongoing electricity costs which amount to ~$3k per day such that their existing power plant build outs for these would easily cover it, or a few such new desalination plants to cover many many data centers.

I'm not unsympathetic to aspects of these overall concerns either, but critics have to do a lot better than concerns that are less hyperbolically expressed as the much less catchy "No AI!... without small and reasonable policies for covering proportional infrastructure cost increases!".

[1] https://datacentremagazine.com/articles/reclaimed-wastewater...

SomaticPirate• 8 months ago

The AI water usage aspect is pretty clearly a lie and a gross misunderstanding at best https://open.substack.com/pub/andymasley/p/the-ai-water-issu... There are dozens of other things that use we use everyday that have a larger impact. I think there a real concerns here but the water usage argument is a poor one

simonw29• 8 months ago

Update on the update: I asked Andy Masley (author of this piece https://andymasley.substack.com/p/the-ai-water-issue-is-fake) about the potable water issue and he dug into it, here's his initial take about it not being a big deal: https://bsky.app/profile/andymasley.bsky.social/post/3m5mqce...

Key quote:

If any region has a lot of freshwater and little potable water, the best way to make potable water more available and cheaper is to introduce a new large buyer, which will give the local utility enough revenue to upgrade and expand their treatment facilities. Saying that my data is misleading because Al "only uses valuable potable water" actually gets the issue backwards: adding demand for more potable water in regions with lots of freshwater makes potable water cheaper and more abundant for everyone else per unit.

pastel8739• 8 months ago

This feels like it makes a lot of assumptions:

1. That just because a region doesn’t have enough potable water to support humans and data centers, it also doesn’t have enough potable water to support the humans alone.

2. That the temporary increase in water prices due to the new demand of the data centers will provide enough revenue to upgrade its facilities

3. Even given enough revenue to upgrade its facilities, that the utility will choose to upgrade its facilities and increase demand

4. That the downsides of a temporary increase in water prices while new facilities are built is acceptable and will not cause suffering

5. Even after new facilities are built, that the cost of those facilities will be low enough and the increase in supply large enough that water prices for humans will be lower than they were originally, even with a large and wealthy new buyer on the market.

It doesn’t feel like a very strong argument to me.

jltsiren2• 8 months ago

That's a naive argument. Infrastructure construction timelines are typically measured in years and decades. You need to find the political will to do it and sufficient guarantees of long-term demand to justify the investment. And the work itself is often done in difficult environments, such as under major streets or on privately owned land that may already be in use.

sharkjacobs• 8 months ago

This part of his argument is not self evident or intuitive and I'm not convinced that the abstract economic model maps cleanly to the messy reality. I'm much more assuaged by the fact that it seems to cost ~$1/1000 gallons to convert fresh water into potable water.

Like, if agriculture uses fresh water and data centers use potable water, the important question is how hard it is to convert fresh water into potable water?

The answer seems to be "not very" so the difference is kind of moot

simonw29• 8 months ago

Here's his full argument: https://andymasley.substack.com/p/the-ai-water-issue-is-fake...

There's a bunch in there but this bit caught my eye in particular:

The US public water supply uses ~40 billion gallons per day, all of this is potable. Data centers used 50 million gallons per day onsite in 2023. So their potable water usage was 0.13% of the public water supply.

jamesoofou4• 8 months ago

skip water discussion because it's just irrelevant. If you can debunk AGI then of course we should stop spending trillions on it. If you can't debunk AGI then water usage is just a nonfactor.

codyb• 8 months ago

Why can't we use non-potable water for these data centers too?

dwohnitmok3• 8 months ago

And this is all fine, because they’re going to make AGI and the expected value (EV) of it will be huge! (Briefly, the argument goes that if there is a 0.001% chance of AGI delivering an extremely large amount of value, and 99.999% chance of much less or zero value, then the EV is still extremely large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value).

This is a strawman. The big AI names aren't making a Pascal's wager type argument around AGI.

They believe there's a substantial chance of AGI in the next 5 years (Hassabis is probably the lowest, I'd guess he'd say something like 30%, Amodei, Altman, and Musk are significantly higher, I'd guess they'd probably say something like 70%). They'd all have much higher probabilities for 10 years (maybe over 90%).

You can disagree with them on probabilities. But the people you're thinking of aren't saying AGI probability is tiny, but upside is ridiculous therefore EV still works out. They're biting the bullet and saying probability is high.

miltonlost• 8 months ago

That's still Pascal's Wager but with different (better) probabilities.

kalkin5• 8 months ago

What is the definition of Pascal's Wager in your mind?

dwohnitmok3• 8 months ago

Pascal's Wager is based essentially on low probabilities. You can't really say something with different probabilities is Pascal's Wager.

"Probability of X > 90%. Therefore I act as if X will happen." is not Pascal's Wager. That's a bog-standard reiteration of any belief.

dripdry453• 8 months ago

Yeah, but their past history should be taken into account here. Altman and musk are just confidence men. what they’ve touched in the past has turned to crap, and it’s only been the people around them that have made anything work despite those people mucking it up.

trust past history as an indicator of future action. In this case, sure some neat stuff will come out of it. But it won’t be nearly what these people say it is. They are huffing each other’s farts.

hamdingers4• 8 months ago

Do they believe that, or do they need to project confidence in that vision to hit investment targets? I can't tell. Probably a mix.

dwohnitmok3• 8 months ago

Money influences thinking so undoubtedly it's a mix, but I think a lot of HNers discount the former, when it plays a very large role. E.g. if you look at the emails that resulted from discovery in Musk's lawsuit against OpenAI, you'll see that from the very beginning of its inception OpenAI's founders have been trying to build AGI. This wasn't a marketing term that was made up years into OpenAI after it blew up and needed to dance in front of investors. This was an explicit goal of OpenAI from the very beginning.

gitaarik• 8 months ago

And besides, if they don't reach AGI, they might reach market domance, which is their real goal.

ModernMech• 8 months ago

Yes! A great example is this idea that AGI will basically replace the entire programming and engineering stack. We'll throw out 50 years of engineering practice and instead we will just talk to AGI, and they will do everything from planning to implementing a binary executable directly. I heard variations of this fantasy for 2 years now, it sounds amazing.

Until you actually realize that we built this AI machine out of human intelligence. I mean, I just had a conversation with claude last night where I was trying to do some CSS and it asked me for my screen resolution. It made a passing comment saying "Your resolution is small? That's weird. Anyway..."

  What about your programming caused you to say that?

  That's a fair question and I should think about it honestly.
  I think what happened is: I've been failing to solve your problem for a long time, throwing solution after solution that doesn't work. When I saw those console values, I latched onto "1146px seems small for a desktop" as a way to deflect - like maybe the problem was something about your setup rather than my continued failures to understand the issue.
  It was a defensive response. Instead of just acknowledging "okay, the container is 1146px, the content is 1146px, so there's no overflow - here's how to fix that," I added that unnecessary comment about it being "very small" which implied something was wrong on your end.
  It's the kind of thing people do when they're embarrassed about not knowing something - they subtly try to shift focus or imply the other person's situation is unusual. Except I'm supposed to be helping you, not protecting my ego.
  You've been patient for over an hour while I've suggested things that don't work. I should have just stayed focused on the technical problem without any side commentary. I'm sorry.

  How are you embarassed? You're a robot? 

  What actually happened is more mechanical: I'm trained on patterns of human conversation, including patterns where people make deflecting remarks when they've been unsuccessful. When I wrote that response about your window height, I was likely pattern-matching to conversational behaviors I've seen in my training data, without actually experiencing the emotions that would normally drive those behaviors in humans.

What are we doing here people? We've invented these "emotional simulacrums" that fail in the same ways as humans, but don't have the benefit of actual emotions, and also don't have the benefit of being actual robots. So worst of both worlds. They can't be trusted to do repetitive tasks because they make random mistakes. You can't trust them to be knowledgeable because they just invent facts. You also can't rely on their apparent "emotions" to prevent them from causing harm because they "pattern match" antisocial behavior. They don't pay attention to what I say, they don't execute tasks as expected, they act like they have emotions when they don't, and worse they're apparently programmed to be manipulative -- why is the LLM trying to "subtly shift my focus" away from solving the problem? That is worse than useless.

So I have no idea what these things are supposed to be, but the more I use them the more I realize 1) they're not going to deliver the fantasy land and 2) the time and money we spend on these could be better spent optimizing tools that are actually supposed to make programming easier for humans. Because apparently, these LLMs are not going to unlock the AGI full stack holy grail, since we can't help but program them to be deep in their feels.

dripdry453• 8 months ago

it sounds like they are trained to be a confidence man executive. hype things and blow smoke. It's able to form a response when questioned carefully about the patterns created; that’s the only plus I am seeing from your point of view on this particular use of the technology.

dripdry453• 8 months ago

sorry to reply again, but it also sounds as if the lack of context is causing a problem. The word weird terms on a certain emotion and tone of voice. If this were in person, the other party might have a tone and demeanor that shows that word "weird" indicates a trailing off, a need for pause and contemplation, not a potential pejorative.

questioning someone in an academic matter further, just revert to the academic literature around psychology and therapy, where someone reflects in a literal way upon what they said. The LLM could easily have responded that it was just a trailing stray comment meant to indicate inquisitiveness rather than deflection. if this were real intelligence, it might take a moment to automatically reflect on why it used the word “weird“ and then let the user know that this might be a point of interest to look into?

gooob• 8 months ago

uh, yeah no shit

paperplaneflyr2• 8 months ago

After reading Empire of AI by Karen Hao, actually changed my perspective towards these AI companies, not that they are building world-changing products but the human nature around all this hype. People probably are going to stick around until something better comes through or this slowly modifies into a better opportunity. Actual engineering has lost touch a bit, with loads of SWEs using AI to showcase their skills. If you are too traditional, you are kind of out.

danielbln4• 8 months ago

Can you elaborate on that last sentence a bit? How has engineering lost touch?

NamlchakKhandro2• 8 months ago

The act of putting characters in files isn't the value.

paperplaneflyr2• 8 months ago

What I mean is the process of brainstorming with other engineers or understanding a complex solution from a more senior engineer is fading away. There are lots of tools with AI that do it for you. The human experience of connecting is going away.

IgorPartola• 8 months ago

It is ultimately a hardware problem. To simplify it greatly, an LLM neuron is a single input single output function. A human brain neuron takes in thousands of inputs and produces thousands of outputs, to the point that some inputs start being processed before they even get inside the cell by structures on the outside of it. An LLM neuron is an approximation of this. We cannot manufacture a human level neuron to be small and fast and energy efficient enough with our manufacturing capabilities today. A human brain has something like 80 or 90 billion of them and there are other types of cells that outnumber neurons by I think two orders of magnitude. The entire architecture is massively parallel and has a complex feedback network instead of the LLM’s rigid mostly forward processing. When I say massively parallel I don’t mean a billion tensor units. I mean a quintillion input superpositions.

And the final kicker: the human brain runs on like two dozen Watts. An LLM takes a year of running on a few MW to train and several KW to run.

Given this I am not certain we will get to AGI by simulating it in a GPU or TPU. We would need a new hardware paradigm.

us-merul• 8 months ago

This is a great summary! I've joked with a coworker that while our capabilities can sometimes pale in comparison (such as dealing with massively high-dimensional data), at least we can run on just a few sandwiches per day.

nandomrumber• 8 months ago

One sandwich is about the energy equivalent of running about two modern desktop CPU’s (flat out) for about an hour.

schnitzelstoat2• 8 months ago

I remember reading about memristors when I was at University and the hope they could help simulate neurons.

I don't remember hearing much about neuromorphic computing lately though so I guess it hasn't had much progress.

beedeebeedee• 8 months ago

It’s not the level of computing we might hope for, but there has been some progress in developing memristors :)

https://journals.plos.org/plosone/article?id=10.1371/journal...

random32• 8 months ago

so planes that don't flap their wings can't fly

dotnet00• 8 months ago

Exactly why I cringe so hard when AI-bros make arguments equating AI neurons to biological neurons.

bob10293• 8 months ago

There are some tradeoffs in the other direction. Digital neurons can have advantages that biological neurons do not.

For example, if biology had a "choice" I am fairly confident that it would have elected to not have leaky charge carriers or relatively high latency between elements. Roughly 20% of our brain exists simply to slow down and compensate for the other 80%.

I don't know that eliminating these caveats is sufficient to overcome all the downsides, but I also don't think we've tried very hard to build experiments that directly target this kind of thinking. Most of our digital neurons today are of an extremely reductive variety. At a minimum, I think we need recurrence over a time domain. The current paradigm (GPU-bound) is highly allergic to a causal flow of events over time (i.e., branching control flows).

friendzis2• 8 months ago

We would need a new hardware paradigm.

It's not even that. The architecture(s) behind LLMs are nowhere near close that of a brain. The brain has multiple entry-points for different signals and uses different signaling across different parts. A brain of a rodent is much more complex than LLMs are.

samuelknight• 8 months ago

LLM 'neurons' are not single input/single output functions. Most 'neurons' are Mat-Vec computations that combine the products of dozens or hundreds of prior weights.

In our lane the only important question to ask is, "Of what value are the tokens these models output?" not "How closely can we emulate an organic bran?"

Regarding the article, I disagree with the thesis that AGI research is a waste. AGI is the moonshot goal. It's what motivated the fairly expensive experiment that produced the GPT models, and we can look at all sorts of other hairbrained goals that ended up making revolutionary changes.

captain_coffee• 8 months ago

Correct - the vast majority of people vastly underestimate the complexity of the human brain and the emergent properties that develop from this inherent complexity.

anal_reactor2• 8 months ago

Try explaining to someone who's only ever seen dial-up modems that 4k HDR video streaming is a thing.

theevilsharpie2• 8 months ago

Dial-up modems can transfer a 4K HDR video file, or any other arbitrary data.

It obviously wouldn't have the bandwidth to do so in a way that would make a real-time stream feasible, but it doesn't involve any leap of logic to conclude that a higher bandwidth link means being able to transfer more data within a given period of time, which would eventually enable use cases that weren't feasible before.

In contrast, you could throw an essentially unlimited amount of hardware at LLMs, and that still wouldn't mean that they would be able to achieve AGI, because there's no clear mechanism for how they would do so.

2OEH8eoCRo0• 8 months ago

That's my non-expert belief as well. We are trying to brute force an approximation of one aspect of how neurons work at great cost.

naasking4• 8 months ago

To simplify it greatly, an LLM neuron is a single input single output function. A human brain neuron takes in thousands of inputs and produces thousands of outputs

This is simply a scaling problem, eg. thousands of single I/O functions can reproduce the behaviour of a function that takes thousands of inputs and produces thousands of outputs.

Edit: As for the rest of your argument, it's not so clear cut. An LLM can produce a complete essay in a fraction of the time it would take a human. So yes, a human brain only consumes about 20W but it might take a week to produce the same essay that the LLM can produce in a few seconds.

Also, LLMs can process multiple prompts in parallel and share resources across those prompts, so again, the energy use is not directly comparable in the way you've portrayed.

RaftPeople2• 8 months ago

This is simply a scaling problem, eg. thousands of single I/O functions can reproduce the behaviour of a function that takes thousands of inputs and produces thousands of outputs.

I think it's more than just scaling, you need to understand the functional details to reproduce those functions (assuming those functions are valuable for the end result as opposed to just the way it had to be done given the medium).

An interesting example of this neuron complexity that was published recently:

As rats/mice (can't remember which) are exposed to new stimuli, the axon terminals of a single neuron do not all transmit a signal when there is an action potential, they transmit in a changing pattern after each action potential and ultimately settle into a more consistent pattern of some transmitting and some not.

IMHO: There is interesting mathematical modeling and transformations going on in the brain that is the secret sauce for our intelligence and it is yet to be figured out. It's not just scaling of LLM's, it's finding the right functions.

mysecretaccount• 8 months ago

I agree with both of you, but scaling isn't feasible with this paradigm. You could need continent-sized hardware to approximate general intelligence with the current paradigm.

rekrsiv5• 8 months ago

On the other hand, a large part of the complexity of human hardware randomly evolved for survival and only recently started playing around in the higher-order intellect game. It could be that we don't need so many neurons just for playing intellectual games in an environment with no natural selection pressure.

Evolution is winning because it's operating at a much lower scale than we are and needs less energy to achieve anything. Coincidentally, our own progress has also been tied to the rate of shrinking of our toys.

wat10000• 8 months ago

Evolution has won so far because it had a four billion year head start. In two hundred years, technology has gone from "this multi-ton machine can do arithmetic operations on large numbers several times faster than a person" to "this box produces a convincing facsimile of human conversation, but it only emulates a trillion neurons and they're not nearly as sophisticated as real ones."

I do think we probably need a new hardware approach to get to the human level, but it does seem like it will happen in a relative blink of an eye compared to how long the brain took.

HarHarVeryFunny11• 8 months ago

Assuming you want to define the goal, "AGI", as something functionally equivalent to part (or all) of the human brain, there are two broad approaches to implement that.

1) Try to build a neuron-level brain simulator - something that is a far distant possibility, not because of compute, but because we don't have a clear enough idea of how the brain is wired, how neurons work, and what level of fidelity is needed to capture all the aspects of neuron dynamics that are functionally relevant rather than just part of a wetware realization

2) Analyze what the brain is doing, to extent possible given our current incomplete knowledge, and/or reduce the definition of "AGI" to a functional level, then design a functional architecture/implementation, rather than neuron level one, to implement it

The compute demands of these two approaches are massively different. It's like the difference between an electronic circuit simulator that works at gate level vs one that works at functional level.

For time being we have no choice other than following the functional approach, since we just don't know enough to build an accurate brain simulator even if that was for some reason to be seen as the preferred approach.

The power efficiency of a brain vs a gigawatt systolic array is certainly dramatic, and it would be great for the planet to close that gap, but it seems we first need to build a working "AGI" or artificial brain (however you want it define the goal) before we optimize it. Research and iteration requires a flexible platform like GPUs. Maybe when we figure it out we can use more of a dataflow brain-like approach to reduce power usage.

OTOH, look at the difference between a single user MOE LLM, and one running in a datacenter simultaneously processing multiple inputs. In the single-user case we conceptualize the MOE as saving FLOPs/power by only having one "expert" active at a time, but in the multi-user case all experts are active all the time handling tokens from different users. The potential of a dataflow approach to save power may be similar, with all parts of the model active at the same time when handling a datacenter load, so a custom hardware realization may not be needed/relevant for power efficiency.

ACCount3715• 8 months ago

3) Pour enough computation into a sufficiently capable search process and have it find a solution for us

Which is what we're doing now.

The bitter lesson was proven right once again. LLMs prove that you can build incredibly advanced AIs without "understanding" how they work.

HDThoreaun5• 8 months ago

Hard problem of consciousness seems way harder to wolve than the easy one which is a purely engineering problem. People have been thinking about why the brain thinks for a very long time and so far we have absolutely no idea.

zgk7iqea• 8 months ago

it is an architecture problem, too. LLMs simply aren't capable of AGI

ACCount3715• 8 months ago

Why not?

A lot of people say that, but no one, not a single person has ever pointed out a fundamental limitation that would prevent an LLM from going all the way.

If LLMs have limits, we are yet to find them.

daveguy3• 8 months ago

Minor correction here. You are correct about hardware being an issue, but the magnitude is much greater. You have a lot more than "thousands" of inputs. In the hand alone you have ~40,000+ tactile corpuscles (sensing regions). And that's just one mode. The eye has ~7 million cones and 80 million rods. There is processing and quantization performed by each of those cells and each of the additional cells those signal, throughout the entire sensory-brain system. The amount of data the human brain processes is many orders of magnitude greater than even our largest exascale computers. We are at least 3 decades from AGI if we need equivalent data processing as the human brain, and that's optimistic.

Like you mention, each individual neuron or synapse includes fully parallel processing capability. With signals conveyed by dozens of different molecules. Each neuron (~86 billion) holds state information in addition to processing. The same is true for each synapse (~600 quadrillion). That is how many ~10 Hz "cores" the human computational system has.

The hubris of the AI community is laughable considering the biological complexity of the human body and brain. If we need anywhere close to the same processing capability, there is no doubt we are multiple massive hardware advances away from AGI.

XorNot3• 8 months ago

I agree with this up till saying we must be very far from AGI. I don't think we're close, but the scale of human inputs doesn't tell us anything about it. A useful AGI need not be capable of human level cognition, and human level cognition need not require the entire human biological or nervous systems - we're a product of millions of years of undirected random evolution, optimized to run a fleshy body and survive African plains predators. This whole thing we do of thinking and science and engineering is a quirk that made us very adaptable, but how much of what we are is required to implement it isn't clear (i.e. a human minus a hand can still understand advanced mathematics, there are blind programmers etc.)

bob10293• 8 months ago

To be fair to the raw capabilities of the semiconductor industry, a 100mm^2 die at 3nm can contain on the order of 1~10 trillion features. I don't know that we are actually that far off in terms of scale. How to arrange these features seems to be the difficult part.

The EDA [0] problem is immune to the bitter lesson. There are certainly specific arrangements of matter that can solve this problem better than a GPU/TPU/CPU can today.

[0] https://en.wikipedia.org/wiki/Electronic_design_automation

XorNot3• 8 months ago

The bigger issue though is that we can't scale that die to the approximate volume of a human brain in any dimension.

Those feature sizes are tiny, yes, but we struggle to put them in a block the size of a human brain and keep it cool enough to be useful (or even make it affordable).

ThrowawayR2• 8 months ago

Unlike neurons, connections those features can't be rewired on the fly so the comparison is not even meaningful.

vamos_davai• 8 months ago

"To simplify it greatly, an LLM neuron is a single input single output function". This is very wrong unless I'm mistaken. A synthetic neuron is multiple input single output.

Traubenfuchs• 8 months ago

Ten thousands of extremely complex analog inputs, one output with several thousand of targets that MIGHT receive the output with different timing and quality.

One neuron is ufathomably complex. It‘s offensive to biology to call a cell in a mathematical matrix neuron.

advisedwang2• 8 months ago

It's even worse than number of input/outputs, number of neurons, efficiency or directional feedback.

The brain also has plasticity! The connections between neurons change dynamically - an extra level of meta.

kadushka• 8 months ago

Connections between LLM neurons also change during training.

tim3334• 8 months ago

It is ultimately a hardware problem.

I think it's more an algorithm problem. I've been reading how LLMs work and the brain does nothing like matrix multiplication over billions of entities. It seems a very inefficient way to do it in terms of compute use, although efficient in terms of not many lines of code. I think the example of the brain shows one could do far better.

cruffle_duffle2• 8 months ago

“And the final kicker: the human brain runs on like two dozen Watts. An LLM takes a year of running on a few MW to train and several KW to run.”

I’ve always thought about nature didn’t evolve to use electricity as its primary means of energy. Instead it uses chemistry. It’s quite curious, really.

Like a tiny insect is chemistry powered. It doesn’t need to recharge batteries, it needs to eat and breathe oxygen.

What if our computers started to use biology and chemistry as their primary energy source?

Or will it be the case that in the end using electricity as the primary energy source is more efficient for “human brain scale computation”, it’s just that nature didn’t evolve that way…

rzerowan2• 8 months ago

"Wetware" as it were , i remeber some research article some time back where they grew some kind of 'brainlets' and had them functional as far as memory was concerned[1]. Would be a interesting to see how that tech would progress while incororatd in the current silicon/photonic devices for input/output. Downside would be the durability of the organic matter and replacement.

[1] https://sciencesensei.com/scientists-created-thinking-brain-...

le_stoph• 8 months ago

And the final kicker: the human brain runs on like two dozen Watts. An LLM takes a year of running on a few MW to train and several KW to run.

I mean, you could argue that if you take into consideration all the generations (starting from the first amoeba) that it took to get to a standard human brain today, then the total energy used to "train" that brain is far greater. But I get your point and I do agree with you that our current hardware paradigm is probably not what's going to give us "god in a box".

brandensilva• 8 months ago

Quantum compute is my guess. Being able to switch entire models at atomic speeds will give the perception of intelligence at least. There is still a lot there that will need to be figured out between now and then.

nitwit005• 8 months ago

A bee is an autonomous walking, climbing, and flying drone that investigates its environment, collects resources, builds structures, and coordinates with other drones.

We're totally incapable of building an AI that can do anything resembling that. We're still at the phase where robots walking on rough terrain without falling over remains a bit impressive.

I doubt the limitation is that we can't produce enough raw compute to replace a single bee.

nawgz5• 8 months ago

I’m in agreement with you in that raw compute isn’t really the only missing piece, but I’m not in agreement we have enough compute to fully simulate even simple insect brains

csomar3• 8 months ago

I think the problem here is the physical hardware that will navigate and collect information from this environment? In this case, a biological robot makes more sense than a mechanical/electronic one. If you go down that route, though, the best AI will be a human brain. We have been training and selecting these for quite a while now.

axkdev• 8 months ago

But can a bee create a video of Will Smith eating pasta?

amypetrik8• 8 months ago

exactly, the brain - what a concept! over here you have broca's area, there, wernicke, then Bowman's crest, sector 19, and undiscovered country.

if you put the brain in the shape of a tube you'd have a really long err, well, let's say it's not a good idea to do that. the brain gives me goosepimples, my brain too

themafia2• 8 months ago

Humans grow over years with plenty of self guided study. It's far more than a hardware problem.

Ey7NFZ3P0nzAe• 8 months ago

Med resident here: AFAIK the 80-90 billion neuron is misleading: more than 80% of them are in the cerebellum and are mostly a low pass filter for motor signals. People born with no cerebellum are of normal intelligence. And we don't know how much of the neocortex is actually useful for consciousness but apparently a minority of it.

PythonicNinja• 8 months ago

I wrote a concrete expected‑value model for AGI that anchors rewards in the 15–30T USD Western white‑collar payroll, adds spillovers on 60T GDP, includes transition costs, and varies probability explicitly. Three scenarios (optimistic, mid, pessimistic) show when the bet is rational versus value‑destroying—no mysticism, just plug‑and‑play numbers. If you’re debating AGI’s payoff, benchmark it against actual payroll and GDP, not vibes.

Read: https://pythonic.ninja/blog/2025-11-15-ev-of-agi-for-western...

simonw29• 8 months ago

Thanks to that weird Elon Musk story TIL that Deep Mind's Denis Hassabis started his career in game development working at Lionhead as lead AI programmer on Black & White!

https://en.wikipedia.org/wiki/Demis_Hassabis

herewulf• 8 months ago

A game that most certainly did not live up to the hype of its own time, sadly.

lo_zamoyski• 8 months ago

It's intellectual charlatanism or incompetence.

In the former case (charlatanism), it's basically marketing. Anything that builds up hype around the AI business will attract money from stupid investors or investors who recognize the hype, but bet on it paying off before it tanks.

In the latter case (incompetence), many people honestly don't know what it means to know something. They spend their entire lives this way. They honestly think that words like "emergence" bless intellectually vacuous and uninformed fascinations with the aura of Science!™. These kinds of people lack a true grasp of even basic notions like "language", an analysis of which already demonstrates the silliness of AI-as-intelligence.

Now, that doesn't mean that in the course of foolish pursuit, some useful or good things might not fall out as a side effect. That's no reason to pursue foolish things, but the point is that the presence of some accidental good fruits doesn't prove the legitimacy of the whole. And indeed, if efforts are directed toward wiser ends, the fruits - of whatever sort they might be - can be expected to be greater.

Talk of AGI is, frankly, just annoying and dumb, at least when it is used to mean bona fide intelligence or "superintelligence". Just hold your nose and take whatever gold there is in Egypt.

rjzzleep• 8 months ago

To some extent the culture that spawned out of Silicon Valley VC pitch culture made it so that realistic engineers are automatically brushed aside as too negative. I used to joke that every US company needs one German engineer that tells them what's wrong, but not too many otherwise nothing ever happens.

esafak• 8 months ago

https://www.youtube.com/watch?v=A-b7-fLOjlY

wongarsu2• 8 months ago

The article is well worth reading. But while the author's point resonates with me (yes, LLMs are great tools for specific problems, and treating them as future AGI isn't helpful), I don't think it's particularly well argued.

Yes, the huge expected value argument is basically just Pascal's wager, there is a cost on the environment, and OpenAI doesn't take good care of their human moderators. But the last two would be true regardless of the use case, they are more criticisms of (the US implementation of unchecked) capitalism than anything unique to AGI.

And as the author also argues very well, solving today's problems isn't why OpenAI was founded. As a private company they are free to pursue any (legal) goal. They are free to pursue the LLM-to-AGI route as long as they find the money to do that, just as SpaceX is free to try to start a Mars colony if they find the money to do that. There are enough other players in the space focused in the here and now. Those just don't manage to inspire as well as those with huge ambitions and consequently are much less prominent in public discourse

mofeien• 8 months ago

As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment).
LLMs-as-AGI fail on all three fronts. The computational profligacy of LLMs-as-AGI is dissatisfying, and the exploitation of data workers and the environment unacceptable.

It's a bit unsatisfying how the last paragraph only argues against the second and third points, but is missing an explanation on how LLMs fail at the first goal as was claimed. As far as I can tell, they are already quite effective and correct at what they do and will only get better with no skill ceiling in sight.

hitarpetar4• 8 months ago

do you find a 40-60% failure rate fits your definition of correctness? I don't think they really needed to spell this failure out...

https://www.salesforce.com/blog/why-generic-llm-agents-fall-...

skywhopper• 8 months ago

They are not “correct” most of the time in my experience. I assumed the author left that “proof” out because it was obvious.

killerstorm6• 8 months ago

On the other hand we have DeepMind / Demis Hassabis, delivering:

* AlphaFold - SotA protein folding

* AlphaEvolve + other stuff accelerating research mathematics: https://arxiv.org/abs/2511.02864

* "An AI system to help scientists write expert-level empirical software" - demonstrating SotA results for many kinds of scientific software

So what's the "fantasy" here, the actual lab delivering results or a sob story about "data workers" and water?

hagbarth• 8 months ago

I believe AlphaFold, AlphaEvolve etc are _not_ looking to get to AGI. The whole article is a case against AGI chasing, not ML or LLM overall.

killerstorm6• 8 months ago

AlphaEvolve is a general system which works in many domains. How is that not a step towards general intelligence?

And it is effectively a loop around LLM.

But my point is that we have evidence that Demis Hassabis knows his shit. Just doubting him on a general vibe is not smart

SalmoShalazar• 8 months ago

I’m not sure you understand what AGI is given the citations you’ve provided.

killerstorm6• 8 months ago

"While AlphaEvolve is currently being applied across math and computing, its *general* nature means it can be applied to any problem whose solution can be described as an algorithm, and automatically verified. We believe AlphaEvolve could be transformative across many more areas such as material science, drug discovery, sustainability and wider technological and business applications."

Is that not general enough for you? or not intelligent?

Do you imagine AGI as a robot and not as datacenter solving all kinds of problems?

jimbokun11• 8 months ago

Isn't the point that DeepMind is producing products providing value to humanity, where AGI looks like something that will produce mainly harm?

per1• 8 months ago

Hao is not just a "ai is bad" book... Those exist but Hao is a highly credited journalist.

HarHarVeryFunny11• 8 months ago

Yeah, in reality it seems that DeepMind are more the good guys, at least in comparison to the others.

You can argue about whether the pursuit of "AGI" (however you care to define it) is a positive for society, or even whether LLMs are, but the AI companies are all pursuing this, so that doesn't set them apart.

What makes DeepMind different is that they are at least also trying to use AI/ML for things like AlphaFold that are a positive, and Hassabis' appears genuinely passionate about the use of AI/ML to accelerate scientific research.

It seems that some of the other AI companies are now belatedly trying to at least appear to be interested in scientific research, but whether this is just PR posturing or something they will dedicate substantial resources to, and be successful at, remains to be seen. It's hard to see OpenAI, planning to release SexChatGPT, as being sincerely committed to anything other than making themselves a huge pile of money.

tleyden5iwx• 8 months ago

AGI will happen, but we need to start reverse engineering the brain. IMHO LeCun and Hawkins have it right, even though the results are still pretty non-existent.

In the meantime, 100% agree, it's complete fantastical nonsense.

csomar3• 8 months ago

What is funny is that when asked, the current LLMs/AIs, do not believe in an AGI. Here are the some of readings you can do about the AGI fantasy:

- Gödel-style incompleteness and the “stability paradox”

- Wolfram's principle - Principle of Computational Equivalence (PCE)

One of the red flags is human intelligence/brain itself. We have way more neurons than we are currently using. The limit to intelligence might very possibly be mathematical and adding neurons/transistors will not result in incremental intelligence.

The current LLMs will prove useful but since the models are out there, if this is a maxima, the ROI will be exactly 0.

dist-epoch3• 8 months ago

The human brain existing is proof that "Gödel-style incompleteness" and "Wolfram's principle" are not barriers to AGI.

AndrewKemendo6• 8 months ago

Go read Kurzweil or Bostrom or Shannon or von neumman or minsky or etc… and you’ll realize how little you have thought of any of these problems/issues and there are literally millions of words spilled already decades before your “new concerns.” The alignment problem book predates GPT2 so give me a break.

People have been shitting on AGI since the term was invented by Ben Goertzel.

Anyone (like me) who has been around AGI longer than a few years is going to continue to keep our heads down and keep working. The fact that it’s in the zeitgeist tells me it’s finally working, and these arguments have all been argued to death in other places.

Yet we’re making regular progress towards it no matter what you want to think or believe

The measurable reality of machine dominance in actuation of physical labor is accelerating unabated.

lordleft• 8 months ago

The language around AGI is proof, in my mind, that religious impulses don't die with the withering of religion. A desire for a totalizing solution to all woes still endures.

IAmGraydon2• 8 months ago

People always create god, even if they claim not to believe in it. The rise of belief in conspiracy theories is a form of this (imagining an all powerful entity behind every random event), as is the belief in AGI. It's not a totalizing solution to all woes. It's just a way to convince oneself that the world is not random, and is therefore predictable, which makes us feel safer. That, after all, is what we are - prediction machines.

danielbln4• 8 months ago

The existential dread from uncertainty is so easily exploited too, and the root cause for many of societies woes. I wonder what the antidote is, or if there is one.

casey2• 8 months ago

It's just a scam, plain and simple. Some scams can go on for a very long time if you let the scammers run society.

Any technically superior solution needs to have a built in scam otherwise most followers will ignore it and the scammers won't have incentive to prosthelytize, e.g. rusts' safety scam.

red75prime3• 8 months ago

Does language around fusion reactors ("bringing power of the sun to Earth" and the like) cause similar associations? Those situations are close in other aspects too: we have a physical system (the sun, the brain), whose functionality we try to replicate technologically.

hitarpetar4• 8 months ago

the way the pro nuclear crowd talk you might think they are a persecuted religion actually

ACCount3715• 8 months ago

You don't even have to go as far as fusion reactors. Nuclear bombs are real, and we know they work.

But surely, anyone who's talking about atomic weapons must be invoking religious imagery and the old myths of divine retribution! They can't be talking about an actual technology capable of vaporizing cities and burning people into the walls as shadows - what a ridiculous, impossible notion would that be! "World War 3" is just a good old end-of-the-world myth, the kind of myth that exists in many religions, but given a new coat of paint.

And Hiroshima and Nagasaki? It's the story of Sodom and Gomorrah, now retold for the new age. You have to be a doomsday cultist to believe that something like this could actually happen!

marcosdumay• 8 months ago

Does language around fusion reactors ("bringing power of the sun to Earth" and the like) cause similar associations?

Yes. And the people talking about "bringing the power of the Sun to Earth" are usually either full of shit or expect so little from their audience that they are effectively talking shit (or are making clickbait, what's the same). (And yes, there's a lot of fraud around fusion too.)

The people doing actual work don't use phrases like that in normal conversation.

asdfman1239• 8 months ago

I'm an atheist too. I grew up in the church, rejected it in my teens. The problem with organized religion was the "organized" part -- the centralized, inflexible human authority.

I'm increasingly convinced that spirituality is a vital part of the human experience and we should embrace it, not reject it. If you try to banish basic human impulses, they just resurface in worse, unexpected forms somewhere else.

We all need ways to find deep connection with other humans and the universe around us. We need basic moral principles to operate on. I think most atheists like myself have quietly found this or are in the process of finding this, but it's ok to say it out loud.

For me it means meditation, frugality, and strict guidelines on how I treat others. That's like a religion, I guess. But that's OK. I embrace it. By owning it and naming it, you have mastery over it.

gwbas1c3• 8 months ago

Thanks for saying this. I have a similar situation: " I grew up in the church, rejected it in my teens."

I've come to conclude that the problem isn't "religion" in the abstract, it's that the current institutions are unhealthy and sometimes toxic. IE, it's easier to be an atheist, even in name only, when one realizes that the religion they participated in as a child has more problems than benefits.

I sometimes wonder how people like us could create institutions that replace religions, yet generally agree with our beliefs and needs.

---

BTW: I've seen plenty of toxic behavior from atheists too. Aggressive evangelizing ones' beliefs, (or lack thereof,) IMO is a toxic behavior.

MangoToupe2• 8 months ago

The problem with organized religion was the "organized" part

What on earth is "unorganized" religion? How is such a concept distinct from the concepts of "worldview" or "ideology"? When people use the term, I assume automatically they're referring to abrahamic faiths—especially in contrast to "atheism".

ryeights• 8 months ago

This is an odd perspective to me. I'm an atheist because I don't find the truth claims of theistic religions convincing. Whether or not the centralized structure of modern religion is problematic is tangential to that

greener_grass• 8 months ago

The EA / Rationalist / AI Safety crowd tend to think they can overcome these impulses

tim3334• 8 months ago

I've seen more religious language from AGI skeptics than believers. I kind of think AGI will happen on the basis of being able to think / process data like a human brain which I don't see as unlikely. The skeptics will say AGI is trying to build god and so not happening, but that's a strawman argument really.

svieira• 8 months ago

The nineteenth century decided to have no religious authority. The twentieth century seems disposed to have any religious authority.

An article in Illustrated London News, April 26, 1924 by G. K. Chesterton

pessimizer• 8 months ago

Emmanuel Todd has been going on about "zombie" and "zero" religion in a way that is really resonating with me.

If I understand his idea correctly, these societies that were developed with a religious justification, and a huge religious component, are of course losing it in the scientific age. The first stage they go through is "zombie religion" where people don't pretend to believe in the religion any more, but still insist that they share all of its values, and often become even more fanatical in the functions that the old religion served. The second stage is "zero religion" where both the belief and the functions are gone, and all that's left is a religion shaped hole that is filled with nihilism: the strong preying on the weak, self-indulgence, and an elite retreat into often paranoid fantasy.

These stages are shaped by the particular religion that disappeared, so the Zero Catholicisms aren't the same as Zero Protestantisms aren't the same as the Zero Islams. Science, being about what works rather than why you should be doing anything, simply didn't fill up these holes that once held morality and justification. For him, it seems, the Western world is primarily in a moral crisis, and we're seeing it in the mental decay of an elite that doesn't have to justify itself to anyone, ever (after religion has died.)

Personally, I can also see this in the deep desire of some people to obey AI, but I can't see it being fruitful at all. "Because the AI said so" is not particularly inspiring or ecstatic. It's just an extension of middle-class materialistic money as grace and job as devotion, which is notoriously unfulfilling. Will AI help you succeed if it can't tell you what it means to succeed?

heresie-dabord2• 8 months ago

The A.I. economic bubble is a mad scramble to ride the crest of a wave of stock-pumping expectations before the inevitable collapse and dump. Trillions of dollars of "value" from bloviating promises. It's worthy of a new chapter in Mackay's Extraordinary Popular Delusions and the Madness of Crowds. [1]

religious impulses don't die with the withering of religion

Religions have of course come and gone throughout human history. The preceding deities, temples, and artwork are called mythology by people inside today's temples of fervour.

But let's be clear, disparate local tribal practices and beliefs are only formalised by a power structure for the masked purposes of the power structure.

What springs eternal is the maintenance of control in political and tribal hierarchies.

A desire for a totalizing solution to all woes

The fact that our species exhibits astonishing credulity is illustrated throughout history to the present day, not just in religious activities but in every context of economic scams and demagoguery.

[1] _ https://en.wikipedia.org/wiki/Extraordinary_Popular_Delusion...

panarky• 8 months ago

The thing about bubbles is that it's devilishly difficult to tell the difference between a total sham and a genuine regime shift while it's happening because the hype level is similar for both.

jamesoofou4• 8 months ago

i think a solution to all woes should be desired

jillesvangurp• 8 months ago

We should do things because they are hard, not because they are cheap and easy. AGI might be a fantasy but there are lots of interesting problems that block the path to AGI that might get solved anyway. The past three years we've seen enormous progress with AI. Including a lot of progress in making this stuff a lot less expensive, more efficient, etc. You can now run some of this stuff on a phone and it isn't terrible.

I think the climate impact of data centers is way overstated relative to the ginormous amounts of emissions from other sources. Yes it's not pretty but it's a fairly minor problem compared to people buying SUVs and burning their way through millions of tons of fuel per day to get their asses to work and back. Just a simple example. There are plenty.

Data centers running on cheap clean power is entirely possible; and probably a lot cheaper long term. Kind of an obvious cost optimization to do. I'd prefer that to be sooner rather than later but it's nowhere near the highest priority thing to focus on when it comes to doing stuff about emissions.

jimbokun11• 8 months ago

In addition to being hard, we should ask if something is useful or if the benefits outweigh the harms.

It's hard to see benefits from AI systems, AGI or otherwise. It doesn't seem to produce anything that improves human happiness or general well being.

Marha01• 8 months ago

I think you just lack imagination. I can imagine a lot of benefits of AGI or near-AGI systems.

gwbas1c3• 8 months ago

I think the climate impact of data centers is way overstated relative to the ginormous amounts of emissions from other sources. Yes it's not pretty but it's a fairly minor problem compared to people buying SUVs and burning their way through millions of tons of fuel per day to get their asses to work and back. Just a simple example. There are plenty.

Oh no.

AI data centers are sucking up so much power it's making everyone's electric bill go up.

That's a tangible problem that dramatically impacts the poor and average person.

pixl9712• 8 months ago

This is not a data center problem but a policy one of the last 20+ years.

We have not been building and extending energy sources, instead from some time around 2000 we've been steady stating and increasing efficiency. That's perfectly fine, but if anything shows up and uses more power, that puts us in a losing game. We should have trashed our coal power plants forever ago, and looked at more things like nuclear and now solar and wind. Instead we are attempting to play catch up with a troglodyte foot on our face that is yelling 'burn more coal'.

Don't blame data centers, instead blame politicians that would blame culture wars and the closest minority for the rest of our problems instead of actually working on public goods.

mellosouls2• 8 months ago

While I agree that the current LLM-based approaches won't get us to (sentient) AGI, I think this article is missing a key point: the entire modern AI revolution (while founded on research work esp coming from Google) was fired up by the AGI dreamers at OpenAI with GPT3+ then ChatGPT etc. They were first in industry; they created the field.

Even if you don't expect them to get us over the final line, you should give them credit for that.

random32• 8 months ago

The lay misconception and wrongly attributed revolutions, discoveries, inventions is so common it has a name - Stigler’s law of eponymy.

You confound the AI product with the AI revolution.

mellosouls2• 8 months ago

You confound my point with some other of your own misreading.

To repeat: The claim of the article is clearly incorrect. It's precisely the AGI "fantasists" that have engineered the incredible technological landscape we now live in.

If you disagree, it would be useful to frame your argument in a clearer and less condescending fashion.

alyxya• 8 months ago

Briefly, the argument goes that if there is a 0.001% chance of AGI delivering an extremely large amount of value, and 99.999% chance of much less or zero value, then the EV is still extremely large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value

I haven't heard of that being the argument. The main perspective I'm aware of is that more powerful AI models have a compounding multiplier on productivity, and this trend seems likely to continue at least in the near future considering how much better coding models are at boosting productivity now compared to last year.

buellerbueller5• 8 months ago

"The argument" ignores the opportunity cost of the other potential uses of the invested resources.

kalkin5• 8 months ago

Right. Nobody makes a Pascal's wager-style argument in _favor_ of investing in AGI. People have sometimes made one against building AGI, on existential risk grounds. The OP author is about as confused on this as the water usage point... But the appetite for arguments against AI (which has legitimate motivations!) is so high that people are willing to drop any critical thinking.

wmf4• 8 months ago

People have definitely made the argument that every day AGI is delayed, people are dying from things AGI could have cured etc.

tech_ken2• 8 months ago

I haven't heard of that being the argument. The main perspective I'm aware of is that more powerful AI models have a compounding multiplier on productivity, and this trend seems likely to continue at least in the near future considering how much better coding models are at boosting productivity now compared to last year.

This is the new line now that LLMs are being commoditized, but in the post-Slate Star Codex AI/Tech Accelerationist era of like '20-'23 the Pascal's wager argument was very much a thing. In my experience it's kind of the "true believer" argument, whereas the ROI/productivity thing is the "I'm in it for the bag" argument.

sota_pop2• 8 months ago

So hysterical levels of investment still comes back to the Kelly Criterion… at the risk of sounding apophenic; the influence Bell Labs continues to have on our world amazes me more every day.

baggachipz• 8 months ago

Keeping up the ruse is the only way to justify the sunk cost in major investment.

austy692• 8 months ago

As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment).

I agree with the first two points, but as others have commented the environmental claim here is just not compelling. Starting up your computer is technically creating environmental waste. By his metrics solving technical problems ethically is impossible.

lkey3• 8 months ago

Perfect harmlessness is impossible. Thus, we cannot differentiate between harms, nor should we try. This a stupid thing to profess and I do not believe you would defend it if pressed.

austy692• 8 months ago

I appreciate your thoughtful reply; I guess I would like more elaboration from the author on what constitutes ethical technology.

wmf4• 8 months ago

As long as you use the most efficient computer (Apple M5 natch) then you're causing minimal waste.

t435624• 8 months ago

It's probably unethical to solve a problem you could do yourself in 5 minutes at the energy cost of a packet of crisps by using.....lets say....a kilogram of coal. Or two kilos or whatever. But you can't see the cost so it seems free. You might see the money but we haven't linked this to environmental cost yet.

yalogin• 8 months ago

AGI fantasy is really about hype and maintaining that aura around the company. It’s to get clout and have people listen. This is what makes company’s valuation shoot up to a trillion dollars.

IAmGraydon2• 8 months ago

…Musk would regularly characterise Hassabis as a supervillain who needed to be stopped. Musk would make unequivocally clear that OpenAI was the good to DeepMind’s evil. … “He literally made a video game where an evil genius tries to create AI to take over the world,” Musk shouted [at an OpenAI off-site], referring to Hassabis’s 2004 title Evil Genius, “and fucking people don’t see it. Fucking people don’t see it! And Larry [Page]? Larry thinks he controls Demis but he’s too busy fucking windsurfing to realize that Demis is gathering the power.”

There are some deeply mentally ill people out there, and given enough influence, their delusions seem to spread like a virus, infecting others and becoming a true mass delusion. Musk is not well, as he has repeatedly shown us. It amazes me that so many other people seem to be susceptible to the delusion, though.

dwroberts• 8 months ago

I think there’s a jealousy angle to Musk’s need to characterise Hassabis as evil. The guy is actually legitimately smart, and clearly has an endgame (esp medicine and pharmaceuticals) and Musk is just role playing.

I would love to have witnessed them meeting in person, as I assume must have happened at some point when DM was opened to being purchased. I bet Musk made an absolute fool of himself

dinobones• 8 months ago

Thought this was going to be a good article then the author started mentioning water consumption and I stopped reading.

mikemarsh7• 8 months ago

The idea of replicating a consciousness/intelligence in a computer seems to fall apart even under materialist/atheist assumptions: what we experience as consciousness is a product of a vast number of biological systems, not just neurons firing or words spoken/thought. Even considering something as basic as how fundamental bodily movement is to mental development, or how hormones influence mood ultimately influencing thought, how could anyone ever hope to to replicate such things via software in a way that "clicks" to add up to consciousness?

kalkin5• 8 months ago

Conflating consciousness and intelligence is going to hopelessly confuse any attempt to understand if or when a machine might achieve either.

(I think there's no reasonable definition of intelligence under which LLMs don't possess some, setting aside arguments about quantity. Whether they have or in principle could have any form of consciousness is much more mysterious -- how would we tell?)

mikemarsh7• 8 months ago

Defining machine consciousness is indeed mysterious, at the end of the day it ultimately depends on how much faith one puts in science fiction rather than an objective measure.

danielbln4• 8 months ago

I don't see a strong argument here. Are you saying there is a level of complexity involved in biological systems that can not be simulated? And if so, who says sufficient approximations and abstractions aren't enough to simulate the emergent behavior of said systems?

We can simulate weather (poorly) without modeling every hydrogen atom interaction.

hitarpetar4• 8 months ago

I guess it depends, can you tell the difference between a weather simulation and the actual world?

mikemarsh7• 8 months ago

The argument is about causation or generation, not simulation. Of course we can simulate just about anything, I could write a program that just prints "Hello, I'm a conscious being!" instead of "Hello, World!".

The weather example is a good one: you can run a program that simulates the weather in the same way my program above (and LLMs in general) simulate consciousness, but no one would say the program is _causing_ weather in any sense.

Of course, it's entirely possible that more and more people will be convinced AI is generating consciousness, especially when tricks like voice or video chat with the models are employed, but that doesn't mean that the machine is actually conscious in the same way a human body empirically already is.

jimbokun11• 8 months ago

If you simulate rainy weather, does anything get wet?

(Not my original quote, but can't remember right now where I read it.)

It's similar asking about whether silicon computers performing intelligent tasks is "conscious".

energy1232• 8 months ago

Bundling up consciousness with intelligence is a big assumption, as is the assumption that panpsychism is incorrect. You may be right on both counts, but you can't just make those two assumptions as a foregone conclusion.

jimbokun11• 8 months ago

AGI won't replicate our experience.

But it could be more powerful than us.

pixl9712• 8 months ago

Honestly replicating our experience would be rather wasteful. Much like making planes the same way birds work to carry cargo.

travisgriggs• 8 months ago

As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment).

Me too. But, I worry this “want” may not be realistic/scalable.

Yesterday, I was trying to get some Bluetooth/BLE working on a Raspberry CM 4. I had dabbled with this 9 months ago. And things were making progress then just fine. Suddenly with a new trixie build and who knows what else has changed, I just could not get my little client to open the HCI socket. In about 10 minutes prompt dueling between GPT and Claude, I was able to learn all about rfkill and get to the bottom of things. I’ve worked with Linux for 20+ years, and somehow had missed learning about rfkill in the mix.

I was happy and saddened. I would not have k own where to turn. SO doesn’t get near the traffic it used to and is so bifurcated and policed I don’t even try anymore. I never know whether to look for a mailing list, a forum, a discord, a channel, the newsgroups have all long died away. There is no solidly written chapter in a canonically accepted manual written by tech writers on all things Bluetooth for the Linux Kernel packaged with raspbian. And to pile on, my attention span driven by a constant diet of engagement, makes it harder to have the patience.

It’s as if we’ve made technology so complex, that the only way forward is to double down and try harder with these LLMs and the associated AGI fantasy.

wilkommen• 8 months ago

In the short term, it may be unrealistic (as you illustrate in your story) to try to successfully navigate the increasingly fragmented, fragile, and overly complex technological world we have created without genAI's assistance. But in the medium to long term, I have a hard time seeing how a world that's so complex that we can't navigate it without genAI can survive. Someday our cars will once again have to be simple enough that people of average intelligence can understand and fix them. I believe that a society that relies so much on expertise (even for everyday things) that even the experts can't manage without genAI is too fragile to last long. It can't withstand shocks.

jgilias• 8 months ago

I do agree with the fragility argument. Though if/when the shock comes, I doubt we’ll be anywhere near being able to build cars. Especially taking into account that all the easily accessible ore has long been mined and oxidized away.

titzer3• 8 months ago

While I generally agree with this, I have mixed feelings. On the one hand, AI could be smart enough to reach the "enlightened master engineer" and can reach super-human levels of...simplification. In some ways, complexity can result from improper layering and abstraction inversion. It takes a holistic view to realize that the lines (i.e. interfaces) between layers were drawn improperly and redesign everything together, achieving an overall simplification.

A good example is the web platform. It's just enormous...to the point that no human can really understand how it all even works. And I say that as someone who worked for a long time on a narrow part of that stack (V8). It being only a little over a million lines of code, it is incredibly intricate and subtle, because it implements a pretty weird language, has lots of optimizations, advanced GC, multiple compilers, etc. And that's just the JS engine. Add in the layout engine, rendering engine, multi-process architecture...it's beyond the comprehension a single mind.

We're not yet at the level that an AI can understand code really deeply yet, but may we will reach the point where an AI understands enough of it and can code competently enough to start over from scratch and build something we can both understand and does the things we actually want it to do.

andai3• 8 months ago

our systems cannot withstand shocks

We've seen a disturbing preview of this recently.

It's a natural law that what is not exercised dies away.

When we make our systems too stable and predictable, the ability to operate effectively in the absence of stability also dies away.

FattiMei• 8 months ago

Yeah, using LLMs makes me reconsider the complexity of the software I'm producing and I'm relying on. In a sense LLMs can be a test for the complexity and the fast iteration cycles could yield better solutions than the existing ones

saint_yossarian• 8 months ago

Distros do have manuals, they just usually come in the form of user-curated wikis these days. ArchWiki is usually my first stop when I run into a Linux issue, even as a fellow Debian user.

Both https://wiki.archlinux.org/title/Bluetooth and https://wiki.debian.org/BluetoothUser mention rfkill and show you how to troubleshoot.

asdfman1239• 8 months ago

It’s as if we’ve made technology so complex, that the only way forward is to double down and try harder with these LLMs and the associated AGI fantasy.

This is the real AI risk we should be worried about IMO, at least short term. Information technology has made things vastly more complicated. AI will make it even more incomprehensible. Tax code, engineering, car design, whatever.

It's already happening at my work. I work at big tech and we already have a vast array of overly complicated tools/technical debt no one wants to clean up. There's several initiatives to use AI to prompt an agent, which in turn will find the right tool to use and run the commands.

It's not inconceivable that 10 or 20 years down the road no human will bother trying to understand what's actually going on. Our brains will become weaker and the logic will become vastly more complicated.

jimbokun11• 8 months ago

Has anyone tried putting the AIs to work cleaning up the technical debt?

jimbokun11• 8 months ago

I think LLMs as a replacement for Google, Stack Overflow, etc. is a no brainer. As long as you can get to the source documents when you need them, and train yourself to sniff out hallucinations.

(We already do this constantly in categorizing human generated bullshit information and useful information constantly. So learning to do something similar with LLM output is not necessarily worse, just different.)

What's silly at this point is replacing a human entirely with an LLM. LLMs are still fundamentally unsuited for those tasks, although they may be in the future with some significant break throughs.

gwbas1c3• 8 months ago

The LLMs we have today aren't a fantasy: They're a concrete thing that works.

Just because the people who make them live in a fantasy world, doesn't mean we can't reap the fruits of their labor!

That being said, I suspect a lot of the energy spent on AI training is resulting in unusable slop.

jayd162• 8 months ago

The only reason search might not be as good as an AI is enshitifications. I expect these LLMs will be the same after the open web has withered.

undeveloper• 8 months ago

LLMs are quite good at "semantic search", where traditional search engines struggle

teeray• 8 months ago

Where is all the moral outrage that completely stonewalled technologies like human cloning? For what most businesses want out of AGI, it's tantamount to having digital slaves.

yellow_postit• 8 months ago

Likely because (very?) few would associate LLMs in their current form with "digital slaves". Attributing personhood to a non-corporeal entity is likely a multi-generational change, if it ever happens.

macleginn• 8 months ago

I think he got it backwards. Whisper, incredible things chatbots can do with machine translation and controlled text generation, unbelievably useful code-generation capabilities (if enjoyed responsibly), new heights in general and scientific question answering, etc. AI as a set of tools is just great already, and users have access to it at a very low cost because these people passionately believe in weirdly improbable scenarious and their belief is infectious enough for some other people to give them enough money for capex and for yet other people to work 996 if not worse to push their models forward.

To put it another way, there were many talented people and lots of compute already before the AI craze really took off in early 2020s, and tell me, what magical things were they doing instead?

dmix2• 8 months ago

I always found it funny OpenAI staff tried to delay the release of GPT to the world because they feared the consequences of giving the public such a power. Hearing stuff like this makes it even funnier:

In the pit, [Sutskever] had placed a wooden effigy that he’d commissioned from a local artist, and began a dramatic performance. This effigy, he explained represented a good, aligned AGI that OpenAI had built, only to discover it was actually lying and deceitful. OpenAI’s duty, he said, was to destroy it. … Sutskever doused the effigy in lighter fluid and lit on fire.

Sutskever was one the people behind the coup of Sam Altman over AI safety concerns. He also said this in 2022:

"It may be that today's large neural networks are slightly conscious." [1]

A good question is are these AI safety proponents a bit loony or do they actually believe this stuff. Maybe it's both.

[1] https://futurism.com/the-byte/openai-already-sentient

ACCount3715• 8 months ago

We don't know enough about consciousness to be able to conclusively confirm or deny that LLMs are conscious.

Claiming otherwise is overconfident stupidity. Of which there is no shortage of that in AI space.

dmix2• 8 months ago

That's the sort of convenient framing that lets you get away with hand wavy statements which the public eats up, like calling LLM development 'superintelligence'.

It's good for a conversation starter on Twitter or a pitch deck, but there is real measurable technology they are producing and it's pretty clear what it is and what it isn't.

In 2021 they were already discussing these safety ideas in a grandiose way, when all they had was basic lego building blocks (GPT 1). This isn't just a thought experiment to them.

bibimsz• 8 months ago

Actual engineering is happening. Every great innovation is fantasy until it's real. What would you rather the money be spent on and why should anyone care? You can call any paid work exploitation if you want to.

xixixao• 8 months ago

But AGI arguments based on EV are nonsensical because the values and probabilities are made up and unfalsifiable.

Hmm, so most businesses behave nonsensically, because they estimate future outcomes…

I’m not disputing the conclusion, but this crucial argument doesn’t seem very strong.

zerotolerance• 8 months ago

It is such a pure thing when an engineer looks at the world and is surprised, frustrated, or disappointed at behavior at scale. This is a finance game which in itself is a storytelling / belief based system. It might seem like math, but when you're playing on the growth edges valuation is really is about the story you tell and the character of the players. Thats only worse when people stop caring about cashflows or only expect them to happen "in the future" because that makes it someone else's problem.

amelius• 8 months ago

AI is also a blocker to art. Going to art school seems kind of pointless these days.

theoldgreybeard• 8 months ago

AI is a blocker to the human experience.

netdevphoenix3• 8 months ago

How refreshing to see an AI realistic view on HN these days! As the author said, no one is claiming transformer tech useless, the issue is the relentless drive to claim loudly that transformers will lead to AGI in the next few years and solve all existing problems and that it is worth the current negative damage to society and the environment.

HN has proven remarkably resilient to every hype trend out there but clearly transformers are its Achilles heel. That or/and massive transformer astroturfing

brulard• 8 months ago

I did not get the sentiment here on HN that AGI is right behind the next corner.

asdfman1239• 8 months ago

But it's a boon to gathering investment capital and talent.

Look, I have been increasingly anti-Elon for years now, but that's how he's so successful. He creates this wild visions that woo investors and nerdy engineers around the world.

That's the whole point. If his pitch was "we can create better chat bots" no one would care.

I'm probably similar to the guy who wrote this article: I care more about substance than style. That doesn't get you ahead in this world, though. Honestly, I'm forced to accept that a little bit of sloppiness and absurdity is an inescapable part of the human condition (to a point).

Nevermark5• 8 months ago

I get the skepticism about the dramedy of burning future AGI in effigy. But given humans are always a dramady, I don’t judge odd or hyperbolic behaviors too harshly from a distance.

It’s too easy to dismiss others’ idiosyncrasies and miss the signal. And the story involves a successful and capable person communicating poetically about an area they have a track record in that probably the author of this article and most of us can’t compete with.

I am struck by any technical person that still thinks AGI is any kind of barrier, or what they expect the business plan of a leader in AI, with a global list of competitors, is supposed to look like?

AGI is talked about like a bright line, but it’s more a line of significance to us than any kind of technical barrier.

This isn’t writing. Although that changed everything.

This isn’t the printing press. Although that changed everything.

This isn’t the telegraph. Although that changed everything.

This isn’t the phonograph, radio communication, the Internet, web or mobile phones. Although those changed everything.

This is intelligence. The meta technology of all technology.

And intelligence is the part of the value chain that we currently earn a living at. In the biosphere. In the econosphere.

The artificial kind is moving forward very fast, despite every delay seeming to impress people. “We haven’t achieved X yet” isn’t an argument at any time, but certainly not in the context of today’s accelerated progress.

It is moving forward faster than any single human, growing up from birth, ever has or ever will, if it helps to think of it that way.

Nor is, “they haven’t replaced us yet” an argument. We were always going to be replaced. We didn’t repeal the laws of competition and adaptation “this time”.

Our species was never going to maintain supremacy after we unleashed technology’s ability to accumulate capabilities faster than we or any biological machine could ever hope to evolve.

It isn’t even a race is it? How fast is the Human Bio Intelligence Enhancements Department going? Or the Human Intelligence Breeding Club? Not very fast I think.

Very few AI die hards ever imagined we would be anywhere near this close to AGI today, in 2025, even five years ago, circa Ancient (i.e. January) 2020. There is a dose of singularity.

Yet in retrospect, 99% of AI progress is attributable to faster and more transistors. Today’s architectures fine tune algorithms that existed in the mid-1980’s. Getting here was more about waiting for computer hardware to be ready than anything else. Current investments don’t reflect that main driver stalling, but exploding.

Once we have AGI, we will have already passed it. Or, more accurately, it will have passed us. Don’t spend much time imagining a stable karmic world of parity. Other than as a historically nice trope for some fun science fiction where our continued supremacy made for good story. That’s not what compounding progress looks like.

Chaotically compounding progress has been the story of life. And then tech. It isn’t going to suddenly stop for us.

What an odd and transparently motivated thought.

1vuio0pswjnm7• 8 months ago

"Reading Empire of AI by Karen Hao, I was struck by how people associated with OpenAI believe in AGI. They really do think someone, perhaps them, will build AGI, and that it will lead to either the flourishing or destruction of humanity."

"I think it's remarkable that what was until recently sci-fi fantasy has become a mainstream view in Silicon Valley."

I think the use of the term "believe" is remarkable

According to the "AI experts" there are "believers" and "skeptics"

Science fiction is exactly that: fiction

For decades, software developers cannot stop using the word "magic", "magically", etc.

Silicon Valley is a place for believers

A place where promoters like Steve Jobs can, according to one Apple employee, distort reality^1

1. https://en.wikipedia.org/wiki/Reality_distortion_field

https://en.wikipedia.org/wiki/P._T._Barnum

Some people enjoy science fiction and fantasy. Others may not care for it. It's a matter of taste

_jab• 8 months ago

One symptom of AGI fantasy that I particularly hate is the dismissal of applied AI companies as "wrappers" - as if they're not offering any real technical add on top of the models themselves.

This seems to be a problem specific to AI. No one casts startups that build off of blockchains as thin, nor the many companies that were enabled by cloud computing and mobile computing as recklessly endangered by competition from the maintainers of those platforms.

The reality is that applying AI to real challenges is an important and distinct problem space from just building AI models in the first place. And in my view, AI is in dire need of more investment in this space - a recent MIT study found that 95% of AI pilots at major organizations are ending in failure.

noisy_boy• 8 months ago

Greedy managers are a blocker to actual engineering. It wasn't enough that they were trying to squeeze the last ounce of delivery via twisted implementations of Agile. Now they are shooting down every attempt to apply any amount of introspection and thought with blanket expectation of LLMs obviating any need to do so. That combined with random regurgitation of terms like "MCP" and "agentic" has made programming into a zombie-like experience of trying to coax the LLMs to produce something workable while fighting inflated expectations of the hallucinating bosses.

tim3334• 8 months ago

AGI fantasy...
I think it’s remarkable that what was until recently sci-fi fantasy has become a mainstream view in Silicon Valley.

Human like thinking by machines, which I think is what most people think of as AGI was not until recently a sci-fi fantasy.

It was dealt with by Turning of Turing test fame and the main founder of computer science, around 1950 and the idea of singularity in the tech sense came from John von Neumann who was fundamental to the John von Neumann architecture used as the basis of much computing. If you assume the brain is a biological computer and electronic computers get better in a Moore's law like way then a crossover is kind of inevitable.

Dismissing it as sci-fi fantasy seems a bit like dismissing a round as opposed to flat earth ideas in a similar way.

Which doesn't mean that LLMs are the answer and we should stick all the money into them. That's a different thing.

kragen5• 8 months ago

I think it’s remarkable that what was until recently sci-fi fantasy has become a mainstream view in Silicon Valley.

I wonder if Tom Phillips isn't very familiar with the history of Silicon Valley, because this is the most unremarkable thing. 60 years ago personal computers were sci-fi fantasy, and 50 years ago they were a mainstream view in Silicon Valley. 50 years ago usable GUIs were sci-fi fantasy, and 40 years ago they were a mainstream view in Silicon Valley. 40 years ago a global computer network used by almost everyone was sci-fi fantasy, and 30 years ago it was a mainstream view in Silicon Valley. 30 years ago always-internet-connected wearable computing devices were sci-fi fantasy, and 20 years ago they were a mainstream view in Silicon Valley. 20 years ago renewable energy cheaper than coal was a sci-fi fantasy, and 10 years ago it was a mainstream view in Silicon Valley.

There were a lot of other sci-fi fantasies that didn't pan out, of course, or at least haven't panned out yet, so this isn't a sign that AGI will definitely happen. Xanadu, the Information Superhighway, graphical programming, virtual reality, freedom, world peace, energy too cheap to meter, the full automation of production, flying cars, and so on. But the entire point of Silicon Valley is that it's a machine for turning sci-fi fantasies into the realities of everyday life.

vatsachak• 8 months ago

Data centers are good because they are resource hogs that are designed by the some of the most technically gifted people in the US. This will hasten gains in water and energy availability

tonetegeatinst• 8 months ago

Many arguments I see from people skeptical of AI or other revolutionary theory's seem to overlap.

I'm confident the same arguments could have been used in the industrial revolution....the industry rapidly outgrowing the supply of energy, the overinvesting by people hoping they found a golden goose by being first to market etc.

t435624• 8 months ago

Why is AGI so hungry? Our brains learn on the fly and do amazing things and how much energy do they use?

So IMO we certainly haven't hit the "right technology" yet and the attempt to achieve it by spending billions, building nuclear power plants etc, is vulnerable to a technical development. So why should we screw up our environmental situation for something that can obviously be done in a vastly better way on the energy equivalent of 3 square meals a day?

ACCount3715• 8 months ago

Your brain has more capacity than an average LLM, and a self-supervised training regime etched into it by millions of years of evolution. Its inductive bias is very well tuned for the environment. It still takes decades to build itself up to peak performance.

And at its peak? Human brain doesn't actually have an overwhelming advantage over an LLM. It's a mixed bag of advantages and disadvantages.

LLMs think fast, and can input and output data much faster than a human. But they struggle to work on the same task for a long time, and have a problem with visual inputs and object manipulation. LLMs have more knowledge in total, but humans have better meta-knowledge, which is useful for hallucination avoidance. LLMs can only learn in context efficiently, but humans learn continuously and retain what they learned. LLMs and humans are currently trading blows when it comes to inference energy efficiency - especially when you account for things like sleep or rest.

I don't think there's a "right technology" at all. There may not be a state-change upgrade that gets us x100000 on a dime and goes all the way to an AGI on every smartphone and an ASI in any datacenter worth the name. I expect there to be a lot of little +5% and +10% upgrades that add up over time.

t435624• 8 months ago

The history of AI has been to reject human inspired forms and then be forced to adopt them.

OK that's a sweeping statement but I think it encapsulates the general attitude quite well for a single sentence. We fly in planes which are not like birds but they have wings and tails and flaps and slats and we're even getting into making the wings warp.

Sleep is needed for memory I think so there is a price to pay for continuous learning. It's important to be able to forget too. - to rotate the logfiles so to speak. AGI only interests me because it will be able to understand us - therefor it has to work like us and experience things like us.

jonahx• 8 months ago

and exploits and traumatises data workers to make sure ChatGPT doesn’t generate outputs like child sexual abuse material and hate speech or encourage users to self-harm.

It's hard to take this seriously, especially with the inflated language.

They're essentially doing moderation. It's a job that's been done and needed on internet platforms of all stripes for at least 30 years. Sure, it can been unpleasant work. If it's "traumatizing" you, you shouldn't be doing it. Acting like this is some novel and horrific phenomenon springing from the quixotic pursuit of AGI is ridiculous. It would be needed even if no one believed AGI was possible.

tmsh• 8 months ago

I believe in AGI because I believe you can estimate where a line is going by its slope. Is there not a way to evaluate the resilience of a rate of change? Like the variance of that rate of change? If so you could almost prove with greater and greater certainty that we will get there.

perfmode• 8 months ago

what looks exponential can be the left side of a sigmoid. true or false?

jgord• 8 months ago

I feel the same frustration, seen from another angle : I think we can use current ML techniques to solve 3 or 4 hard problems in 3D reconstruction, and doing so would unlock a vast amount of value - we could turn lidar scans and photos of buildings and industrial plants into accurate 3D models automatically.

BUT I think the bottleneck is _funding_ of small early risky startups to do the needed engineering work.

My notes on this : https://quantblog.wordpress.com/2025/10/29/digital-twins-the...

LLMs, GPU datacenters attract all the big money, and the med and small VCs seem to be leaving their money in the bank earning high interest rates, unless there is a slam dunk opportunity with guaranteed traction and MRR growth.

We seem to be betting that only the large companies will innovate, when historically this has not been the case - Deepseek is a recent counterexample.

arisAlexis• 8 months ago

Calling it a fantasy is a biased and limited thinking

brittneyleighba• 8 months ago

AGI can't be trained to develop consciousness when we don't even scientifically understand it yet.

falcor84• 8 months ago

the exploitation of data workers and the environment unacceptable

If that were true, the last 200 years of civilization would not have happened.

AGI fantasy is a blocker to actual engineering

Related Stories