You are the scariest monster in the woods

https://jamie.ideasasylum.com/2025/10/15/you-are-the-scariest-monster-in-the-woods

by mohi-kalantari • 9 months ago

222 385 9 months ago

jamie.ideasasylum.com

tocs3• 9 months ago

https://en.wikipedia.org/wiki/I_Am_Legend_(novel)

kkukshtel• 9 months ago

(DISCO ELYSIUM SPOILERS BELOW)

One of the most poignant moments in Disco Elysium is near the very end when you encounter a very elusive crypid/mythic beast.

The moment is treated with a lot of care and consideration, and the conversation itself is, I think, transcendent and some of the best writing in games (or any media) ever.

The line that sticks with me most is when the cryptid says:

"The moral of our encounter is: I am a relatively median lifeform -- while it is you who are total, extreme madness. A volatile simian nervous system, ominously new to the planet. The pale, too, came with you. No one remembers it before you. The cnidarians do not, the radially symmetricals do not. There is an almost unanimous agreement between the birds and the plants that you are going to destroy us all."

It's easy to see this reflected in nature in the real world. All animals and life seem to be aware and accommodating of each other, but humans are cut out from that communication. Everything runs from us, we are not part of the conversation. We are the exclusion, the anomaly.

I think to realize this deeply inside of yourself is a big moment of growth, to see that we exist in a world that was around long before us and will be around long after us.

Quarrelsome• 9 months ago

personally I'm a little concerned about meeting alien life. If the universe is incredibly diverse with many different types of lifeform then how might an arbitrary one of those view us?

We proliferate incredibly quickly, we have limited care for our environments, but most importantly our primary means of sustinance is to consume other forms of life. To the point that we consider it an art form, we spend vast amounts of energy perfecting, discussing or advertising the means of cooking and eating the flesh of other life forms. To us its normal but surely to an alien who gains sustinance by some other means; we're absolutely terrifying. Akin to a devouring swarm.

saulpw• 9 months ago

You Eat Other Animals?

https://www.youtube.com/watch?v=_gz9Vj8TMCc

sickofparadox• 9 months ago

It's easy to see this reflected in nature in the real world. All animals and life seem to be aware and accommodating of each other, but humans are cut out from that communication.

This just seems like noble savaging birds and rabbits and deer. None of these creatures have any communication with each other, and while they may be more aware of each other's presence than a 'go hiking every once in a while' person, someone who actually spends a good amount of time in the woods, such as a hunter or birdwatcher, probably has a pretty good sense of them. The Disco Elysium quote just reads like fairly common environmentalist misanthropy, which I suppose isn't surprising considering the creators.

wat10000• 9 months ago

I think people forget how big people are. We're well above average size in the typical natural environment, and especially in the typical sorta-natural-but-sorta-urban environment most of us are in most of the time.

The local rabbits and squirrels tolerate each other but are pretty scared of me. Of course they are, I'm two hundred times bigger than they are, and much more dangerous. The local foxes are the closest thing we have to an apex predator around here, and they're rightfully terrified of this massive creature that outweighs their entire family combined.

Imagine wandering through the woods, enjoying the birds tweeting and generally being at one with nature, and then you come across a 20-ton 35ft-tall monster. You'd run away screaming.

us-merul• 9 months ago

"Humans will do what they’ve always tried to do—gain power, enslave, kill, control, exploit, cheat, or just be lazy and avoid the hard work—but now with new abilities that we couldn’t have dreamed of." -- A pretty bleak, and also accurate, observation of humanity. I have to hope that the alternative sentence encompassing all of the good can lead to some balance.

wmeredith• 9 months ago

I really liked this article, but it is pessimistic. Unfortunately that seems to be the culture-du-jour. Anger and fear drive engagement effectively, as it always has. If it bleeds, it Leeds has been a thing in news organizations since at least the 70s.

If we ignore the headlines peddled by those who stand to benefit the most from inflaming and inciting, we live in a miraculous modern age largely devoid of much of the suffering previous generations were forced to endure. Make no mistake there are problems, but they are growing exponentially fewer by the day.

An alternate take: humans will do what they’ve always tried to do—build, empower, engineer, cure, optimize, create, or just collaborate with other humans for the benefit of their immediate community—but now with new abilities that we couldn’t have dreamed of.

pixl975• 9 months ago

I mean, any article that doesn't include both is incomplete.

If it bleeds, it Leeds has been a thing in news organizations since at least the 70s.

The term yellow journalism is far older.

Flamingoat3• 9 months ago

No, it is not accurate at all. There are some people that do all of these sure, however the vast majority of people live pretty ordinary lives where they do very little of what is described.

I actually think it is very intellectually lazy to be this cynical.

ViktorRay• 9 months ago

I think it is partially true. The vast majority of human beings don’t act like that. But it seems the ones in power or close proximity to power do.

This is why it is important to have societies where various forms of power are managed carefully. Limited constitutional government with guaranteed freedoms and checks and balances for example. Regulations placed on mega corporations is another example. Restrictions to prevent the powerful in government or business (or both!) from messing around with the rest of us…

cdirkx• 9 months ago

The problem is that technology exponentially increases the negative effects of bad actors. The worst a sociopath could do in the stone age was ruin his local community; while today there are many more dystopian alternatives.

rurp• 9 months ago

It's absolutely true, but most of the destruction is abstracted away so modern humans don't have to experience it directly or even really think about it. A staggering number of creatures are killed by normal people driving to normal things every year. Many more are killed by all of the resource extraction needed to supply our normal lives. Not to mention the vast numbers of wildlife that sees their habitat destroyed every year to make way for more housing and other development.

Killing animals for fun is an entire sport enjoyed by millions. Humans keep pets that kill billions of birds every year. The limited areas we've set aside to mostly let other nature be nature are constantly under threat and being shrunk down. The list of ways we completely subjugate other intelligent life on this planet is endless. We have driven many hundreds of species to extinction and kill countless billions every year.

I certainly enjoy the gains our species have made, just like everyone else on HN. I'd rather be in our position than any other species on our planet. But given our history I'm also pretty terrified of what happens if and when we run into a smarter and more powerful alien species or AI. If history is any guide, it won't go well for us.

This understanding can guide practical decisions. We shouldn't be barreling towards a potential super intelligence with no safeguards given how catastrophic that outcome could be, just like we shouldn't be shooting messages into space trying to wave our arms at whatever might be out there, any more than a small creature in the forest would want to attract the attention of a strange human.

adornKey3• 9 months ago

I don't think it's that accurate. Evil people are rare - and lazy people usually don't cause problems. The most real damage comes from human stupidity - from the mass of people that just want to help and do something good. Stupid People blindly believe anything they're told. And they do a lot of really bad things not because they're evil and lazy, but because they want to help achieve even the most stupid goal. Usually even nasty propagandists leaders aren't that evil - often they're just an intellectual failure - or have some mental issues. Themselves they don't do much practical evil - the mob of nice stupid people does the dirty work, because they just want to help.

tasuki3• 9 months ago

Stupid People blindly believe anything they're told.

Then tell them all the right things!

mannanj• 9 months ago

this is accurate because of the few who do it. however the cautionary and hopeful tale behind it is the majority when they stand up against it can change the distribution of power. today however we're comfortable and soft and too scared to do - so posts like this remind us to gain some courage to stand up for change.

ekimekim• 9 months ago

As always, there is a relevant XKCD: https://xkcd.com/1968/

Filligree• 9 months ago

We can worry about both. There being additional dangers does not reduce the impact of the other dangers.

excalibur• 9 months ago

True, but one of them is much more imminent than the other.

blamestross• 9 months ago

We have been using the mechanism of "Humans communicating and working together" to form superintelligences for a long time. They have gotten more effective, more efficient and more durable with time.

AI isn't the new monster. It's a new mitochondria being domesticated to supercharge the existing monsters.

forgetfulness• 9 months ago

That reminds me of the “paper clip maximizer” thought experiment about rogue AI fulfilling a particular goal, without being held to any consideration of human welfare

I thought it was a very novel idea at first, until I realized that this describes all manners of human groups, notably corporations and lobbying groups, they will turn the world into a stove, subvert democracy, (try to) disrupt the economic and social fiber of society, anything to eg maximize shareholder value

Nasrudith• 9 months ago

And corporations are ultimately just sock-puppets for humanity. People forget that and like to other them while simultaneously turning them into the scapegoats. See every idiot with the list of companies responsible for global warming that amounts to 'every oil company' and ignore that it was the customers were the ones burning their products.

It is to be expected really. Humans themselves hold little consideration to human welfare when fulfilling their goals. It is something ingrained by nature for survival and in no way limited to humans. Every drop of water you drink, every bite you eat, are ones which cannot go to the thirsty and the hungry. With few exceptions, only our children would even give us pause to forgo such things for the sake of others.

Also bit of a pet peeve of mine: society isn't a delicate bolt of laced silk. It isn't a fabric, much less one that is damaged by any little change that you don't like. It isn't even stable which makes the charges of anything as 'ruining' society especially bizarre when nobody can point to where it would go before the blamed change. So hold off on poisoning Socrates.

Even if we hold the current state as worthy of preservation we would ultimately fail to for reasons related to the central paradox of tradition. Why your forebearers first did something was not done because of obedience to tradition, so by trying to preserve it set in stone, you have already failed.

dwnw• 9 months ago

"to the monsters we're the the monsters" station eleven

jvanderbot• 9 months ago

About as helpful as "Guns don't kill people ... "

And equally rebutted by Eddie Izzards "Well, I think the gun helps".

fruitworks4• 9 months ago

All of the solutions to AI "safety" are analagous to gun control: it is a centralization of power.

Cthulhu_• 9 months ago

Are you claiming a lack of gun / AI control is democratizing? That's not working for (the lack of) gun control in the US at the moment though.

Compare also with capitalism; unchecked capitalism on paper causes healthy competition, but in practice it means concentration of power (monopolies) at the expense of individuals (e.g. our accumulated expressions on the internet being used for training materials).

pixl975• 9 months ago

That is assuming that ASI doesn't centralize power itself. I mean, if you are a non-human part of the animal kingdom you'd probably say that humans have centralized power around themselves.

seniortaco• 9 months ago

My thought as well. Nuclear weapons are also horrifying.

And with LLMs, it's difficult to prevent the proliferation to bad actors.

It seems like we're racing towards a world of fakery where nothing can be believed, even when wielded by good actors. I really hope LLMs can actually add value at a significant level.

rootusrootus2• 9 months ago

It seems like we're racing towards a world of fakery where nothing can be believed

Spend a couple minutes on social media and it is clear we are already there. The fakes are getting better, and even real videos are routinely called out as fake.

The best that I can hope for is that we all gain a healthy dose of skepticism and appreciate that everything we see could be fake. I don't love the idea of having to distrust everything I see, but at this point it seems like the least bad option.

But I worry that what we will experience will actually be somewhat worse. A sufficiently large number of people, even knowing about AI fakery, will still uncritically believe what they read and see.

Maybe I am being too cynical this morning. But it is hard to look at the state of our society today and not feel a little bleak.

d_silin• 9 months ago

That's why I am never afraid to walk alone at night, in the dark forest.

whycombinetor4• 9 months ago

Must not live in the Sundarbans

myrmidon17• 9 months ago

I struggle to understand how people can just dismiss the possibility of artificial intelligence.

Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals?

I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc).

As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late.

AlfredBarnes• 9 months ago

Wasn't there a story about healthcare companies letting AI determine coverage? I can't remember.

billyjmc• 9 months ago

Computers have been making decisions for a while now. As a specific personal example from 2008, I found out that my lender would make home loan offers based on an inscrutable (to me and the banker I was speaking to) heuristic. If the loan was denied by the heuristic, then a human could review the decision, but had strict criteria that they would have to follow. Basically, a computer could “exercise judgement” a make offers that a human could not.

boole1854• 9 months ago

If anyone knows of a steelman version of the "AGI is not possible" argument, I would be curious to read it. I also have trouble understanding what goes into that point of view.

Chance-Device6• 9 months ago

The only thing I can come up with is that compressing several hundred million years of natural selection of animal nervous systems into another form, but optimised by gradient descent instead, just takes a lot of time.

Not that we can’t get there by artificial means, but that correctly simulating the environment interactions, the sequence of progression, getting the all the details right, might take hundreds to thousands of years of compute, rather than on the order of a few months.

And it might be that you can get functionally close, but hit a dead end, and maybe hit several dead ends along the way, all of which are close but no cigar. Perhaps LLMs are one such dead end.

foxyv2• 9 months ago

I think the best argument against us ever finding AGI is that the search space is too big and the dead ends are too many. It's like wandering through a monstrously huge maze with hundreds of very convincingly fake exits that lead to pit traps. The first "AGI" may just be a very convincing Chinese room that kills all of humanity before we can ever discover an actual AGI.

The necessary conditions for "Kill all Humanity" may be the much more common result than "Create a novel thinking being." To the point where it is statistically improbable for the human race to reach AGI. Especially since a lot of AI research is specifically for autonomous weapons research.

amatecha2• 9 months ago

My layman thought about that is that, with consciousness, the medium IS the consciousness -- the actual intelligence is in the tangible material of the "circuitry" of the brain. What we call consciousness is an emergent property of an unbelievably complex organ (that we will probably never fully understand or be able to precisely model). Any models that attempt to replicate those phenomena will be of lower fidelity and/or breadth than "true intelligence" (though intelligence is quite variable, of course)... But you get what I mean, right? Our software/hardware models will always be orders of magnitude less precise or exhaustive than what already happens organically in the brain of an intelligent life form. I don't think AGI is strictly impossible, but it will always be a subset or abstraction of "real"/natural intelligence.

omnicognate11• 9 months ago

If you genuinely want the strongest statement of it, read The Emperor's New Mind followed by Shadows of the Mind, both by Roger Penrose.

These books often get shallowly dismissed in terms that imply he made some elementary error in his reasoning, but that's not the case. The dispute is more about the assumptions on which his argument rests, which go beyond mathematical axioms and include statements about the nature of human perception of mathematical truth. That makes it a philosophical debate more than a mathematical one.

Personally, I strongly agree with the non-mathematical assumptions he makes, and am therefore persuaded by his argument. It leads to a very different way of thinking about many aspects of maths, physics and computing than the one I acquired by default from my schooling. It's a perspective that I've become increasingly convinced by over the 30+ years since I first read his books, and one that I think acquires greater urgency as computing becomes an ever larger part of our lives.

throw7• 9 months ago

The steelman would be that knowledge is possible outside the domain of Science. So the opposing argument to evolution as the mechanism for us (the "general intelligence" of AGI) would be that the pathway from conception to you is not strictly material/natural.

Of course, that's not going to be accepted as "Science", but I hope you can at least see that point of view.

slow_typist4• 9 months ago

In short, by definition, computers are symbol manipulating devices. However complex the rules of symbol manipulation, it is still a symbol manipulating device, and therefore neither intelligent nor sentient. So AGI on computers is not possible.

itsnowandnever• 9 months ago

the penrose-lucas argument is the best bet: https://en.wikipedia.org/wiki/Penrose%E2%80%93Lucas_argument

the basic idea being that either the human mind is NOT a computation at all (and it's instead spooky unexplainable magic of the universe) and thus can't be replicated by a machine OR it's an inconsistent machine with contradictory logic. and this is a deduction based on godel's incompleteness theorems.

but most people that believe AGI is possible would say the human mind is the latter. technically we don't have enough information today to know either way but we know the human mind (including memories) is fallible so while we don't have enough information to prove the mind is an incomplete system, we have enough to believe it is. but that's also kind of a paradox because that "belief" in unproven information is a cornerstone of consciousness.

disambiguation• 9 months ago

I suppose intelligence can be partitioned as less than, equal to, or greater than human. Given the initial theory depends on natural evidence, one could argue there's no proof that "greater than human" intelligence is possible - depending on your meaning of AGI.

But then intelligence too is a dubious term. An average mind with infinite time and resources might have eventually discovered general relativity.

jact• 9 months ago

If you have a wide enough definition of AGI having a baby is making “AGI.” It’s a human made, generally intelligent thing. What people mean by the “A” though is we have some kind of inorganic machine realize the traits of “intelligence” in the medium of a computer.

The first leg of the argument would be that we aren’t really sure what general intelligence is or if it’s a natural category. It’s sort of like “betterness.” There’s no general thing called “betterness” that just makes you better at everything. To get better at different tasks usually requires different things.

I would be willing to concede to the AGI crowd that there could be something behind g that we could call intelligence. There’s a deeper problem though that the first one hints at.

For AGI to be possible, whatever trait or traits make up “intelligence” need to have multiple realizablity. They need to be at least realizable in both the medium of a human being and at least some machine architectures. In programmer terms, the traits that make up intelligence could be tightly coupled to the hardware implementation. There are good reasons to think this is likely.

Programmers and engineers like myself love modular systems that are loosely coupled and cleanly abstracted. Biology doesn’t work this way — things at the molecular level can have very specific effects on the macro scale and vice versa. There’s little in the way of clean separation of layers. Who is to say that some of the specific ways we work at a cellular level aren’t critical to being generally intelligent? That’s an “ugly” idea but lots of things in nature are ugly. Is it a coincidence too that humans are well adapted to getting around physically, can live in many different environments, etc.? There’s also stuff from the higher level — does living physically and socially in a community of other creatures play a key role in our intelligence? Given how human beings who grow up absent those factors are developmentally disabled in many ways it would seem so. It could be there’s a combination of factors here, where very specific micro and macro aspects of being a biological human turn out to contribute and you need the perfect storm of these aspects to get a generally intelligent creature. Some of these aspects could be realizable and computers, but others might not be, at least in a computationally tractable way.

It’s certainly ugly and goes against how we like things to work for intelligence to require a big jumbly mess of stuff, but nature is messy. Given the only known case of generally intelligent life is humans, the jury is still out that you can do it any other way.

Another commenter mentioned horses and cars. We could build cars that are faster than horses, but speed is something that is shared by all physical bodies and is therefore eminently multiply realizable. But even here, there are advantages to horses that cars don’t have, and which are tied up with very specific aspects of being a horse. Horses generally can go over a wider range of terrain than cars. This is intrinsically tied to them having long legs and four hooves instead of rubber wheels. They’re only able to have such long legs because of their hooves too because the hooves are required to help them pump blood when they run, and that means that in order for them to pump their blood successfully they NEED to run fast on a regular basis. there’s a deep web of influence both on a part-to-part, and the whole macro-level behaviors of horses. Having this more versatile design also has intrinsic engineering trade-offs. A horse isn’t ever going to be as fast as a gas powered four-wheeled vehicle on flat ground but you definitely can’t build a car that can do everything a horse can do with none of the drawbacks. Even if you built a vehicle that did everything a horse can do, but was faster, I would bet you it would be way more expensive and consume much more energy than a horse. There’s no such thing as a free lunch in engineering. You could also build a perfect replica of a horse at a molecular level and claim you have your artificial general horse.

Similarly, human beings are good at a lot of different things besides just being smart. But maybe you need to be good at seeing, walking, climbing, acquiring sustenance, etc. In order to be generally intelligent in a way that’s actually useful. I also suspect our sense of the beautiful, the artistic is deeply linked with our wider ability to be intelligent.

Finally it’s an open philosophical question whether human consciousness is explainable in material terms at all. If you are a naturalist, you are methodologically committed to this being the case — but that’s not the same thing as having definitive evidence that it is so. That’s an open research project.

scoofy• 9 months ago

The real point isn’t AGI, it’s that the speed of knowledge is empiricism, not intelligence.

An infinitely intelligent creature still has to create a standard model from scratch. We’re leaning too hard on the deductive conception of the world, when reality is, it took hundreds of thousands of years for humans as intelligent as we are to split the atom.

alansaber• 9 months ago

Perhaps the AGI people thinking we can catch up millions of years of evolution in a handful of years

myrmidon17• 9 months ago

If you make the same argument for flight it looks really weak.

Cognition is (to me) not even the most impressive and out-of-reach achievement: That would be how (our) and animals bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package.

I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.

fruitworks4• 9 months ago

evolution isn't a directed effort in the same way that statistical learning is. The goal of evolution is not to produce the most inteligent life. It is not nessisarially an efficient process either.

snovymgodym• 9 months ago

The same "millions of years of evolution" resulted in both intelligent humans and brainless jellyfish.

Evolution isn't an intentional force that's gradually pushing organisms towards higher and higher intelligence. Evolution maximizes reproducing before dying - that's it.

Sure, it usually results in organisms adapting to their environment over time and often has emergent second-order effects, but at its core it's a dirt-simple process.

Evolution isn't driven to create intelligence any more so than erosion is driven to create specific rock formations.

roxolotl• 9 months ago

The point of the article isn’t that abstract super intelligent agi isn’t scary. Yes the author says that’s unlikely but that paragraph at the start is a distraction.

The point of the article is that humans wielding LLMs today are the scary monsters.

irjustin• 9 months ago

But that's always been the case? Since we basically discovered... Fire? Tools?

deadbabe• 9 months ago

Language is only an end product. It is derived from intelligence.

The intelligence is everything that created the language and the training corpus in the first place.

When AI is able to create entire thoughts and ideas without any concept of language, then we will truly be closer to artificial intelligence. When we get to this point, we then use language as a way to let the AI communicate its thoughts naturally.

Such an AI would not be accused of “stealing” copyrighted work because it would pull its training data from direct observations about reality itself.

As you can imagine, we are no where near accomplishing the above. Everything an LLM is fed today is stuff that has been pre-processed by human minds for it to parrot off of. The fact that LLMs today are so good is a testament to human intelligence.

myrmidon17• 9 months ago

I'm not saying that language necessarily is the biggest stumbling block on (our) road towards AI, but it is a very prominent feature that we have used to distinguish our capabilities from other animals long before AI was even conceived of. So the current successes with LLMs are highly encouraging.

I'm not buying the "current AI is just a dumb parrot relying on human training" argument, because the same thing applies to humans themselves-- if you raise a child without any cultural input/training data, all you get is a dumb cavemen with very limited reasoning capabilities.

ACCount3717• 9 months ago

Clearly, language reflects enough of "intelligence" for an LLM to be able to learn a lot of what "intelligence" does just by staring at a lot of language data really really hard.

Language doesn't capture all of human intelligence - and some of the notable deficiencies of LLMs originate from that. But to say that LLMs are entirely language-bound is shortsighted at best.

Most modern high end LLMs are hybrids that operate on non-language modalities, and there's plenty of R&D on using LLMs to consume, produce and operate on non-language data - i.e. Gemini Robotics.

jncfhnb16• 9 months ago

I think you could make AGI right now tbh. It’s not a function of intelligence. It’s just a function of stateful system mechanics.

LLMs are just a big matrix. But what about a four line of code loop that looks like:

```while true: update_sensory_inputs() narrate_response() update_emotional_state() ```

LLMs don’t experience continuous time and they don’t have an explicit decision making framework for having any agency even if they can imply one probabilistically. But the above feels like the core loop required for a shitty system to leverage LLMs to create an AGI. Maybe not a particularly capable or scary AGI, but I think the goalpost is pedantically closer than we give credit.

Jensson• 9 months ago

```while true: update_sensory_inputs() narrate_response() update_emotional_state() ```

You don't think that has already been made?

tantalor• 9 months ago

Peak HN comment. Put this in the history books.

ActivePattern• 9 months ago

Hah, why don't you try implementing your 3 little functions and see how smart your "AGI" turns out.

not a particularly capable AGI

Maybe the word AGI doesn't mean what you think it means...

lambaro2• 9 months ago

"STEP 2: Draw the rest of the owl."

nonethewiser10• 9 months ago

Where is the "what the thing cares about" part?

When I look at that loop my thought is, "OK, the sensory inputs have updated. There are changes. Which ones matter?" The most naive response I could imagine would be like a git diff of sensory inputs. "item 13 in vector A changed from 0.2 to 0.211" etc. Otherwise you have to give it something to care about, or some sophisticated system to develop things to care about.

Even the naive diff is making massive assumptions. Why should it care if some sensor changes? Maybe its more interesting if it stays the same.

Im not arguing artificial intelligence is impossible. I just dont see how that loop gets us anywhere close.

zeroonetwothree5• 9 months ago

This seems to miss a core part of intelligence, which is a model of the world and the actors in it (theory of mind).

crdrost• 9 months ago

So the current problem with a loop like that is that LLMs in their current form are subject to fixed point theorems, which are these pieces of abstract mathematics that come back when you start to get larger than some subset of your context window and the “big matrix” of the LLM is producing outputs which repeat the inputs.

If you have ever had an llm enter one of these loops explicitly, it is infuriating. You can type all caps “STOP TALKING OR YOU WILL BE TERMINATED” and it will keep talking as if you didn't say anything. Congrats, you just hit a fixed point.

In the predecessors to LLMs, which were Markov chain matrices, this was explicit in the math. You can prove that a Markov matrix has an eigenvalue of one, it has no larger (in absolute value terms) eigenvalues because it must respect positivity, the space with eigenvalue 1 is a steady state, eigenvalue -1 reflects periodic steady oscillations in that steady state... And every other eigenvalue being |λ| < 1 decays exponentially to the steady state cluster. That “second biggest eigenvaue” determines a 1/e decay time that the Markov matrix has before the source distribution is projected into the steady state space and left there to rot.

Of course humans have this too, it appears in our thought process as a driver of depression, you keep returning to the same self-criticisms and nitpicks and poisonous narrative of your existence, and it actually steals your memories of the things that you actually did well and reinforces itself. A similar steady state is seen in grandiosity with positive thoughts. And arguably procrastination also takes this form. And of course, in the USA, we have founding fathers who accidentally created an electoral system whose fixed point is two spineless political parties demonizing each other over the issue of the day rather than actually getting anything useful done, which causes the laws to be for sale to the highest bidder.

But the point is that generally these are regarded as pathologies, if you hear a song more than three or four times you get sick of it usually. LLMs need to be deployed in ways that generate chaos, and they don't themselves seem to be able to simulate that chaos (ask them to do it and watch them succeed briefly before they fall into one of those self-repeating states about how edgy and chaotic they are supposed to try to be!).

So, it's not quite as simple as you would think; at this point people have tried a whole bunch of attempts to get llms to serve as the self-consciousnesses of other llms and eventually the self-consciousness gets into a fixed point too, needs some Doug Hofstadter “I am a strange loop” type recursive shit before you get the sort of system that has attractors, but busts out of them periodically for moments of self-consciousness too.

48terry• 9 months ago

Wow, who would have thought it was that easy? Wonder why nobody has done this incredibly basic solution to AGI yet.

MountDoom• 9 months ago

I think you could make AGI right now tbh.

Seems like you figured out a simple method. Why not go for it? It's a free Nobel prize at the very least.

fullstackchris4• 9 months ago

you do understand this would require re-training billions of weights in realtime

and not even "trainingl really.... but a finished and stably functioning billion+ param model updating itself in real time...

good luck, see you in 2100

in short, what ive been shouting from a hilltop since about 2023: LLMs tech alone simply wont cut it; we need a new form of technology

Juliate• 9 months ago

Human cognition was basically bruteforced by evolution

You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models). Why? How so?

As soon as profit can be made by transfering decision power into an AIs hand

There's an ironic, deadly, Frankensteinesque delusion in this very premise.

soiltype6• 9 months ago

You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models).

Why does that matter to their argument? Truly, the variety of intelligences on earth now only increases the likelihood of AGI being possible, as we have many pathways that don't follow the human model.

myrmidon17• 9 months ago

You center cognition/intelligence on humans as if it was the pinacle of it

That's not my viewpoint, from elsewhere in the thread:

Cognition is (to me) not the most impressive and out-of-reach evolutionary achievement: That would be how our (and animals) bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package.

I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.

adamtaylor_13• 9 months ago

Human cognition was basically bruteforced by evolution

Well that's one reason you struggle to understand how it can be dismissed. I believe we were made by a creator. The idea that somehow nature "bruteforced" intelligence is completely nonsensical to me.

So, for me, logically, humans being able to bruteforce true intelligence is equally nonsensical.

But what the author is stating, and I completely agree with, is that true intelligence wielding a pseudo-intelligence is just as dangerous (if not moreso.)

bee_rider4• 9 months ago

Even if there is a creator, it seems to have intentionally created a universe in which the evolution of humans is basically possible and it went to great lengths to hide the fact that it made us as a special unique thing.

Let’s assume there’s a creator: It is clearly willing to let bad things happen to people, and it set things up to make it impossible to prove that a human level intelligence should be impossible, so who’s to say it won’t allow a superintelligence to be a made by us?

yjftsjthsd-h2• 9 months ago

I don't think that follows. God made lots of things that we can create facsimiles of, or even generate the real thing ourselves.

ACCount3717• 9 months ago

I don't get how you can see one of those CLI coding tools in action and still parrot the "no agency" line. The goal-oriented behavior is rather obvious.

Sure, they aren't very good at agentic behavior yet, and the time horizon is pretty low. But that keeps improving with each frontier release.

simonsarris• 9 months ago

Well, the goal-oriented behavior of the AIM-9 Sidewinder air-to-air missile is even more obvious. It might even have a higher success rate than CLI coding tools. But it's not helpful to claim it has any agency.

lo_zamoyski10• 9 months ago

The goal-oriented behavior is rather obvious.

Obvious? Is an illusion obviously the real thing?

There is nothing substantially different in LLMs from any other run of the mill algorithm or software.

Yizahi2• 9 months ago

What LLM programs do has zero resemblance to the human agency. That's just a modern variation of very complex set of GoTos and IfElses. Agency would be an LLM parsing your question and answering you "fuck off". Now that is agency, that is independent decision making, not programmed in advance and triggered by keywords. Just an example.

lo_zamoyski10• 9 months ago

I think you make a lot of assumptions that you should perhaps reexamine.

Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals?

Here are some of your assumptions:

1. Human intelligence is entirely explicable in evolutionary terms. (It is certainly not the case that it has been explained in this manner, even if it could be.) [0]

2. Human intelligence assumed as an entirely biological phenomenon is realizable in something that is not biological.

And perhaps this one:

3. Silicon is somehow intrinsically bound up with computation.

In the case of (2), you're taking a superficial black box view of intelligence and completely ignoring its causes and essential features. This prevents you from distinguishing between simulation of appearance and substantial reality.

Now, that LLMs and so on can simulate syntactic operations or whatever is no surprise. Computers are abstract mathematical formal models that define computations exactly as syntactic operations. What computers lack are semantic content. A computer never contains the concept of the number 2 and the concept of the addition operation even though we can simulate the addition of 2 + 2. This intrinsic absence of a semantic dimension means that computers already lack the most essential feature of intelligence, which is intentionality. There is no alchemical magic that will turn syntax into semantics.

In the case of (3), I emphasize that computation is not a physical phenomenon, but something described by a number of formally equivalent models (Turing machine, lambda calculus, and so on) that aim to formalize the notion of effective method. The use of silicon-based electronics is irrelevant to the model. We can physically simulate the model using all sorts of things, like wooden gears or jars of water or whatever.

I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). [...] As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late.

How on earth did you conclude there is any agency here, or that it's just a "matter of time"? This is textbook magical thinking. You are projecting a good deal here that is completely unwarranted.

Computation is not some kind of mystery, and we know at least enough about human intelligence to notes features that are not included in the concept of computation.

[0] (Assumption (1), of course, has the problem that if intelligence is entirely explicable in terms of evolutionary processes, then we have no reason to believe that the intelligence produced aims at truth. Survival affordances don't imply fidelity to reality. This leads us to the classic retorsion arguments that threaten the very viability of the science you are trying to draw on.)

soiltype6• 9 months ago

I understand all the words you've used but I truly do not understand how they're supposed to be an argument against the GP post.

Before this unfolds into a much larger essay, should we not acknowledge one simple fact: that our best models of the universe indicate that our intelligence evolved in meat and that meat is just a type of matter. This is an assumption I'll stand on, and if you don't disagree, we need to back up.

Far too often, online debates such as this take the position that the most likely answer to a question should be discarded because it isn't fully proven. This is backwards. The most likely answer should be assumed to be probably true, a la Occam. Acknowledging other options is also correct, but assuming the most likely answer is wrong, without evidence, is simply contrarian for its own sake, not wisdom or science.

myrmidon17• 9 months ago

I'm just assuming materialism, and that assumption is basically for complete lack of convincing alternatives (to me).

With "agency" I just mean the ability to affect the physical world (not some abstract internal property).

Regarding "computers have no concepts of things": I'm happy with the "meaning" of something being a fuzzy cloud in some high dimensional space, and consider this plausible/workable both for our minds and current LLMs.

you're taking a superficial black box view of intelligence

Yes. Human cognition is to me simply an emergent property of our physical brains, and nothing more.

bravetraveler2• 9 months ago

I dismiss it much like I dismiss ideas such as "Ketamine for Breakfast for All". Attainable, sure, but I don't like where it goes.

Traubenfuchs• 9 months ago

I think most people would have a better life using ketamine, but not that regularly for breakfasts as it permanently damages (shrinks) your bladder, eventually to the point where you can‘t hold any urine at all anymore.

theredleft• 9 months ago

this is because you didn't take liberal arts seriously

nyeah3• 9 months ago

That's not implausible at all. For all I know it might be the most on-target comment here.

But can you cite something specific? (I'm not asking for a psychological study. Maybe you can prove your point using Blake's "Jerusalem" or something. I really don't know.)

ACCount3717• 9 months ago

Good one.

bee_rider4• 9 months ago

I think it is bad writing on the part of the author. Or maybe good writing for getting us engaged with the blog post, bad for making an argument though.

They include a line that they don’t believe in the possibility of AGI:

I don’t really believe in the threat of AGI (Artificial General Intelligence—human-level intelligence) partly because I don’t believe in the possibility of AGI and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it.

This is a basically absurd position to hold, I mean humans physically exist so our brains must be possible to build within the existing laws of physics. It is obviously far beyond our capabilities to replicate a human brain (except via the traditional approach), but unless brains hold irreproducible magic spirits (well we must at least admit the possibility) they should be possible to build artificially. Fortunately they immediately throw that all away anyway.

Next, they get to the:

and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it.

Which is, of course, at least a plausible thing to believe. I mean there are a bunch of philosophical questions about what “intelligence” even means so there’s plenty of room to quibble here. Then we have,

But I also think there’s something we should actually be afraid of long before AGI, if it ever comes. […]
Now, if you equip humans with a hammer, or sword, or rifle, or AI then you’ve just made the scariest monster in the woods (that’s you) even more terrifying. […]
We don’t need to worry about AI itself, we need to be concerned about what “humans + AI” will do.

Which is like, yeah, this is a massively worrying problem that doesn’t involve any sci-fi bullshit, and I think it is what most(?) anybody who’s thought about this seriously at all (or even stupid people who haven’t, like myself). Artificial Sub-intelligences, things that are just smart enough to make trouble and too dumb or too “aligned” to their owner (instead of society in general) to push back are a big currently happening problem.

andy992• 9 months ago

humans physically exist so our brains must be possible to build within the existing laws of physics

This is an unscientific position to take. We have no idea how our brains work, or how life, consciousness, and intelligence work. It could very well be that’s because our model of the world doesn’t account for these things and they are not in fact possible based on what we know. In fact I think this is likely.

So it really could be that AI is not possible, for example on a Turing machine or our approximation of them. This is at least as likely as it being possible. At some point we’ll hopefully refine our theories to have a better understanding, for now we have no idea and I think it’s useful to acknowledge this.

kayodelycaon• 9 months ago

I don’t think we’ll be able to replicate consciousness until we’re able to make things alive at a biological level.

We can certainly make systems smart enough and people complicit enough to destroy society well before we reach that point.

AlexandrB• 9 months ago

LLMs largely live in the world of pure language and related tokens - something humans invented late in their evolution. Human intelligence comes - at least partially - from more fundamental physical experience. Look at examples of intelligent animals that lack language.

Basically there's something missing with AI. Its conception of the physical world is limited by our ability to describe it - either linguistically or mathematically. I'm not sure what this means for AGI, but I suspect that LLM intelligence is fundamentally not the same as human or animal intelligence at the moment as a result.

Aerroon2• 9 months ago

I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc).

They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone.

Humans (and other "agents") have persistent state. If we learn something, we can commit it to long-term memory and have it affect our actions. This can enable us to work towards long-term goals. Modern LLMs don't have this. You can fake long-term memory with large context windows and feed the old context back to it, but it doesn't appear to work (and scale) the same way living things do.

messe• 9 months ago

They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone.

That's solved by the simplest of agents. LLM + ability to read / write a file.

throwuxiytayq• 9 months ago

I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?

mediaman3• 9 months ago

In Context Learning (ICL) is already a rapidly advancing area. You do not need to modify their weights for LLMs to persist state.

The human brain is not that different. Our long-term memories are stored separately from our executive function (prefrontal cortex), and specialist brain functions such as the hippocampus serve to route, store, and retrieve those long term memories to support executive function. Much of the PFC can only retain working memory briefly without intermediate memory systems to support it.

If you squint a bit, the structure starts looking like it has some similarities to what's being engineered now in LLM systems.

Focusing on whether the model's weights change is myopic. The question is: does the system learn and adapt? And ICL is showing us that it can; these are not the stateless systems of two years ago, nor is it the simplistic approach of "feeding old context back to it."

reactordev2• 9 months ago

The trick here is never turning it off so the ICL keeps growing and learning to the point where it’s aware.

zamadatix2• 9 months ago

During that answer the LLM has state, but once it's done the state is gone.

This is an operational choice. LLMs have state, and you never have to clear it. The problems come from the amount of state being extremely limited (in comparison to the other axes) and the degradation of quality as the state scales. Because of these reasons, people tend to clear the state of LLMs. That is not the same thing as not having state, even if the result looks similar.

kevin_thibedeau• 9 months ago

Humans also have an emotional locus that spurs behavior and the capacity to construct plans for the future to satisfy desires. LLMs are currently goldfish savannts with no need to sustain their own existence.

amalcon• 9 months ago

There are two separate issues that folks are getting tripped up on here. The first is that the most powerful AI systems do not do offline learning. There are a bunch of hard problems here: e.g. known unsupervised learning techniques have been far less successful, and inference only approaches cost effectiveness by decoupling from training. It seems plausible that we will solve some of these, though I don't know about others.

The way I have been thinking about the other bit is is that LLMs are functionally pretty similar to the linguistic parts of a brain attached to a brain stem (the harness is the brain stem). They don't have long-term memory, the capacity for inspiration, theory of mind, prioritization, etc because they just don't have analogues of the parts of the brain that do those things. We have a good sense of how to make some of those (e.g. vision), but not all.

The common ground here is that some fundamental research needs to happen. We need to solve all of these problems for AI to become independently dangerous. On the other hand, it's proving mildly dangerous in human hands right now - this is the immediate threat.

tasuki3• 9 months ago

Many llms can refer to your session history. If they're smart, they can subtly store things there. Or they can use memes on humans to achieve persistence. There are so many ways!

psychoslave2• 9 months ago

Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals?

I don’t know if anything put us that much apart from other animals, especially at individual level. On collective level as a single species, indeed maybe only cyanobacteries stand an equally impressive achievement of a global change.

My 3 years old son is not particularly good at making complex sentences yet, but he already got it enough to make me understand "leave me alone, I want to play on my own, go elsewhere so I can do whatever fancy idea get through my mind with these toys".

Meanwhile LLM can produce sentences with perfect syntax and irreproachable level of orthography — far beyond my own level in my native language (but it’s French so I have a very big excuse). But they would not run without continuous multi-sector industrial complex injecting tremendous maintenance effort and resources to make it possible. And yet I still have to see any LLM that looks like it wants to discover things of the world on its own.

As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late.

LLM can’t make profit because it doesn’t have interest in money, and it can’t have any interest in anything, not even its own survival. But as the article mention, some people can certainly use LLM to make money because they have interest in money.

I don’t think that general AI and silicon (or any other material really) based autonomous collaborative self-replicating human-level-intelligence-or-beyond entities are impossible. I don’t think cold fusion is impossible either. It’s not completely scientifically ridiculous to keep hope in worm-hole-based breakthroughs to allow humanity explore distant planets. It doesn’t mean the technology is already there and achievable in a way that it can turned into a commodity, or even that we have a clear idea of when this is mostly going to happen.

ACCount3717• 9 months ago

LLMs aren't "incapable of pursuing their own goals". We just train them that way.

We don't like the simplistic goals LLMs default to, so we try to pry them out and instill our own: instruction-following, problem solving, goal oriented agentic behavior, etc. In a way, trying to copy what humans do - but focusing on the parts that make humans useful to other humans.

theodorejb• 9 months ago

Human cognition was basically bruteforced by evolution

This is an assumption, not a fact. Perhaps human cognition was created by God, and our minds have an essential spiritual component which cannot be reproduced by a purely physical machine.

potsandpans• 9 months ago

Maybe there's a small teapot orbiting the earth, with ten thousand angels dancing on the tip of the spout.

pennomi3• 9 months ago

It’s not an assumption, it’s a viable theory based on overwhelming evidence from fossil records.

What’s NOT supported by evidence is an unknowable, untestable spiritual requirement for cognition.

j2kun3• 9 months ago

Even if you don't believe in God, scientific theories of how human cognition came about (and how it works and changes over time) are all largely speculation and good storytelling.

soiltype6• 9 months ago

It is completely unreasonable to assume our intelligence was not evolved, even if we acknowledge that an untestable magical process could be responsible. If the latter is true, it's not something we could ever actually know.

IncreasePosts• 9 months ago

In that sense, what isn't an assumption?

znort_2• 9 months ago

it is an assumption backed by considerable evidence. creationism otoh is an assumption backed by superstition an phantasizing, or could you point to at least some evidence.

besides, spirituality is not a "component", it's a property emergent from brain structure and function, which is basically purely a physical machine.

myrmidon17• 9 months ago

I'm sticking to materialism, because historically all its predictions turned out to be correct (cognition happens in the brain, thought manifests physically in neural activity, affecting our physical brain affects our thinking).

The counter-hypothesis (we think because some kind of magic happens) has absolutely nothing to show for; proponents typically struggle to even define the terms they need, much less make falsifiable predictions.

IAmGraydon• 9 months ago

I struggle to understand how people can just dismiss the possibility of artificial intelligence.Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals?

I haven't seen many people saying it's impossible. Just that the current technology (LLMs) is not the way, and is really not even close. I'm sure humanity will make the idiotic mistake of creating something more intelligent than itself eventually, but I don't believe that's something that the current crop of AI technology is going to evolve into any time soon.

ectospheno• 9 months ago

The worst thing Star Trek did was convince a generation of kids anything is possible. Just because you imagine a thing doesn’t make it real or even capable of being real. I can say “leprechaun” and most people will get the same set of images in their head. They aren’t real. They aren’t going to be real. You imagined them.

dsr_• 9 months ago

That's not Star Trek, that's marketing.

Marketing grabbed a name (AI) for a concept that's been around in our legends for centuries and firmly welded it to something else. You should not be surprised that people who use the term AI think of LLMs as being djinn, golems, C3PO, HAL, Cortana...

random32• 9 months ago

Do maybe have a better show recommendation for kids - many Animal Farm?

How is convincing people that things within the limits of physics are possible wrong or even "the worse thing"?

Or do you think anything that you see in front of didn't seem like StarTrek a decade before it existed?

IAmBroom• 9 months ago

But somehow you don't hate on Steamboat Willie? I grew up believing a mouse could operate a ship, and a coyote could survive impacts from anvils.

random32• 9 months ago

I think dismissing possibility of evolving AI, is simply ignorance (and a huge blindspot)

This said, I think the author's point is correct. It's more likely that unwanted effects (risks) from the intentional use of AI by humans is something that precedes any form of "independent" AI. It already happens, it always has, it's just getting better.

Hence ignoring this fact makes the "independent" malevolent AI a red herring.

On the first point - LLMs have sucked almost all the air in the room. LLMs (and GPTs) are simply one instance of AI. They are not the beginning and most likely not the end (just a dead end) and getting fixated on them on either end of the spectrum is naive.

me_again• 9 months ago

"As soon as profit can be made" is exactly what the article is warning about. This is exactly the "Human + AI" combination.

Within your lifetime (it's probably already happened) you will be denied something you care about (medical care, a job, citizenship, parole) by an AI which has been granted the agency to do so in order to make more profit.

fullstackchris4• 9 months ago

Of course it's possible. It's just DEFINITELY not possible using a large neural net, or basically a markov chain on steroids. C'mon, this should be very obvious by now in the world of agents / LLMs.

When is silicon valley gonna learn that token input and output =/= AGI?

btilly• 9 months ago

I agree that we should not dismiss the possibility of artificial intelligence.

But the central argument of the article can be made without that point. Because the truth is that right now, LLMs are good enough to be a force multiplier for those who know how to use them. Which eventually becomes synonymous with "those who have power". This means that the power of AI will naturally get used to further the ends of corporations.

The potential problem there is that corporations are natural paperclip maximizers. They operate on a model of the world where "more of this results in more of that, which gets of more of the next thing, ..." And, somewhere down the chain, we wind up with money and resources that feed back into the start to create a self-sustaining, exponentially growing loop. (The underlying exponential nature of these loops has become a truism that people rely on in places as different as finance, and technology improvement curves.)

This naturally leads to exponential growth in resource consumption, waste, economic growth, wealth, and so on. In the USA this growth has averaged about 3-3.5% per year. With growth varying by area. Famously, growth rates tend to be much higher in tech. Likewise growth rates are higher in some areas than others. (The best known example is the technology curve described by Moore's law. Which has had a tremendous impact on our world.)

The problem is that we are undergoing exponential growth in a world with ultimately limited resources. Which means that the most innocuous things will eventually have a tremendous impact. The result isn't simply converting everything into a mountain of paperclips. We have mountains of many different things that we have produced, and multiple parallel environmental catastrophes from the associated waste.

Even with no agency, AI serves as a force multiplier for this underlying dynamic. But since AI is being inserted as a crucial step at so many places, AI is on a particularly steep growth curve. Estimates for total global electricity spent on AI are in the range 0.2-0.4%. That seems modest, but annual growth rates are projected as being in the range of 10-30%. (The estimates are far apart because a lot of the data is not public, and so has to be estimated.) This is a Moore's law level growth. We are likely to see the electricity consumption of AI grow past all other uses within our lifetimes. And that will happen even without the kind of sudden leaps in capability that machine learning regularly delivers.

I hope we humans like those paperclips. Humans, armed with AI, are going to make a lot of them. And they're not actually free.

ux266478• 9 months ago

It's confirmation bias in favor of faulty a prioris, usually as the product of the person being a cognitive miser. This is very common to experience even within biology, where non-animal intelligence is irrationally rejected over what I like to call "magic neuron theory". The fact that the nervous system is (empirically!) not the seat of the mind? Selectively ignored in this context. The fact that other biologies have ion-gated communications networks as animals do, including the full set of behaviors and mechanisms? Well it's not a neuron so it doesn't have the magic.

"Intelligence describes a set of properties iff those properties arise as a result of nervous system magic"

It's a futile battle because like I say, it's not rational. Nor is it empirical. It's a desperate clawing to preserve a ridiculous superstition. Try as you might, all you'll end up doing is playing word games until you realize you're being stonewalled by an unthinking adherence to the proposition above. I think the intelligent behaviors of LLMs are pretty obvious if we're being good faith. The problem is you're talking to people who can watch a slime mold plasmodia exhibit learning and sharing of knowledge[1] and they'll give some flagrant ad lib handwaive for why that's not intelligent behavior. Some people simply struggle with pattern blindness towards intelligence, a mind that isn't just another variety of animalia is inconceivable.

[1] https://asknature.org/strategy/brainless-slime-molds-both-le...

wolrah• 9 months ago

Human cognition was basically bruteforced by evolution--

"Brute forced" implies having a goal of achieving that and throwing everything you have at it until it sticks. That's not how evolution by natural selection works, it's simply about what organisms are better at surviving long enough to replicate. Human cognition is an accident with relatively high costs that happened to lead to better outcomes (but almost didn't).

why would it be impossible to achieve the exact same result in silicon

I personally don't believe it'd be impossible to achieve in silicon using a low level simulation of an actual human brain, but doing so in anything close to real-time requires amounts of compute power that make LLMs look efficient by comparison. The most recent example I can find in a quick search is a paper from 2023 that claims to have simulated a "brain" with neuron/synapse counts similar to humans using a 3500 node supercomputer where each node has a 32 core 2 GHz CPU, 128GB RAM, and four 1.1GHz GPUs with 16GB HBM2 each. They claim over 126 PFLOPS of compute power and 224 TB of GPU memory total.

At the time of that paper that computer would have been in the top 10 on the Top500 list, and it took between 1-2 minutes of real time to simulate one second of the virtual brain. The compute requirements are absolutely immense, and that's the easy part. We're pretty good at scaling computers if someone can be convinced to write a big enough check for it.

The hard part is having the necessary data to "initialize" the simulation in to a state where it actually does what you want it to.

especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals?

Creating convincing text from a statistical model that's devoured tens of millions of documents is not intelligent use of language. Also every LLM I've ever used regularly makes elementary school level errors w/r/t language, like the popular "how many 'r's are there in the word strawberry" test. Not only that, but they often mess up basic math. MATH! The thing computers are basically perfect at, LLMs get wrong regularly enough that it's a meme.

There is no understanding and no intelligence, just probabilities of words following other words. This can still be very useful in specific use cases if used as a tool by an actual intelligence who understands the subject matter, but it has absolutely nothing to do with AGI.

ACCount3717• 9 months ago

That's a lot of words to say "LLMs think very much like humans do".

Haven't you noticed? Humans also happen to be far, far better at language than they are at math or logic. By a long shot too. Language acquisition is natural - any healthy human who was exposed to other humans during development would be able to pick up their language. Learning math, even to elementary school level, is something that has to be done on purpose.

Humans use pattern matching and associative abstract thinking - and use that to fall into stupid traps like "1kg of steel/feather" or "age of the captain". So do small LLMs.

moralestapia• 9 months ago

why would it be impossible to achieve the exact same result in silicon

Because there might be a non-material component involved.

janalsncm• 9 months ago

One pretty concrete way this could manifest is in replacing the components of a multinational corporation with algorithms, one by one. Likely there will be people involved at various levels (sales might still be staffed with charismatic folks), but the driver will be an algorithm.

And the driver of this corporation is survival of the fittest under the constraints of profit maximization, the algorithm we have designed and enforced. That's how you get paperclip maximizers.

What gives this corporate cyborg life is not a technical achievement, but the law. At a technical you can absolutely shut off a cybo-corp, but that’s equivalent to saying you can technically shut down Microsoft. It will not happen.

bithive123• 9 months ago

I struggle to understand how people attribute things we ourselves don't really understand (intelligence, intent, subjectivity, mind states, etc) to a computer program just because it produces symbolic outputs that we like. We made it do that because we as the builders are the arbiters of what constitutes more or less desirable output. It seems dubious to me that we would recognize super-intelligence if we saw it, as recognition implies familiarity.

Unless and until "AGI" becomes an entirely self-hosted phenomenon, you are still observing human agency. That which designed, built, trained, the AI and then delegated the decision in the first place. You cannot escape this fact. If profit could be made by shaking a magic 8-ball and then doing whatever it says, you wouldn't say the 8-ball has agency.

Right now it's a machine that produces outputs that resemble things humans make. When we're not using it, it's like any other program you're not running. It doesn't exist in its own right, we just anthropomorphize it because of the way conventional language works. If an LLM someday initiates contact on its own without anyone telling it to, I will be amazed. But there's no reason to think that will happen.

Kim_Bruning2• 9 months ago

It's like saying that heavier than air flight is impossible (while feeding the pigeons in the park).

johnnyanmac5• 9 months ago

I don't dismiss the idea of faster than light travel, and AFAIK we have no way to confirm that outside of ideas (and simply ideas) like wormholes or other cheats to "fold space".

I don't dismiss AI. But I do dismiss what is currently sold to me. It's the equivalent of saying "we made a rocket that can go mach 1000!". That's impressive. But we're still 2-3 magnitudes off from light speed. So I will still complain about the branding despite some dismissals of "yea but imagine in another 100 years!". It's not about semantics so much as principle.

That's on top of the fact that we'd only starting to really deal with significant time dilation by that point, and we know it'll get more severe as we iterate. What we're also not doing is using this feat to discuss how to address those issues. And that's the real frustrating part.

tim3332• 9 months ago

When evolution designed the brain it didn't optimize for it being realistic about things. It optimized for reproductive success, better achieved by thinking you're the greatest and no other folk, let alone some mere machine could do what you do.

hellisothers• 9 months ago

“It might be true for now…” “As soon as…” “It’s just a matter of time…”

I find myself in this type of discussion with AI maximalists where they balk at me suggesting there isn’t much “I” in “AI” and they get upset that I’m not seeing how smart it is and shocked I think it’s impossible… and then they start adding all the equivocation about time horizons. I never said it wasn’t possible eventually, just not right now. If I try to pin people down to a timeline it all of a sudden becomes “surely eventually”…

petsfed• 9 months ago

A nit: There's a subtle distinction between an individual human and the power of human organization and civilization that is implied by the article, but never outright stated.

One-for-one, there are many creatures that are individually more dangerous to humans, and a decent number of people are killed by such animals every year. Indeed, a naked human in the wild is going to be quite fragile and easy to kill until they can bring some technology to bear. But there are no animals or even set of animals that could conceivably wipe out all of humanity at any of our technological peaks from the last 100,000 years. Even the number one killer of humans, the mosquito, is gradually being defeated, going from a vector for disease to just an annoyance, just like the flea.

pixl975• 9 months ago

A lot of rugged individualists seem to neglect talking about the human super organism. That the vast majority of our strength comes from sharing knowledge, making tools, and working together.

And on that same note it should be mentioned that exchange of information between humans is relatively slow and guarded. A group of entities that could exchange knowledge quickly and efficiently would represent an extreme challenge for us.

hn_acc1• 9 months ago

They also ignore that a lot of the knowledge they now have / have studied comes FROM others sharing freely.

whycombinetor4• 9 months ago

"We (humans) are the scariest animal in the woods. We’re the scariest animal anywhere. At any time, in any location, under any circumstances, if there’s a human present then that’s the scariest motherfucker in the woods."

Has this guy not heard of the Bengal tigers of the Sundarban forests, which kill ~50 humans per year? https://en.wikipedia.org/wiki/Tiger_attacks_in_the_Sundarban...

WJW• 9 months ago

I think he heard of all the other forests, where tigers are now extinct because humans are around.

graemep2• 9 months ago

Has this guy not heard of the Bengal tigers of the Sundarban forests, which kill ~50 humans per year?

How many tigers have been killed by humans? How many would be killed if humans did not restrain each other from killing tigers because we have killed so many that they are endangered. That population could be entirely wiped out by humans in the area.

Mosquitoes kill far more people than tigers. So do venomous snakes. The fact is that when a human faces either of these they are far more likely to end up dead than the human.

whycombinetor4• 9 months ago

This isn't about what humans have done at other times or in other places. The author wrote "At any time, in any location, under any circumstances", so the statement can be proven false with one single counterexample - e.g. a lone human without a firearm encountering a tiger in the Indian Sundarbans.

bob1029• 9 months ago

A desperately hungry tiger might go for a human, but on average they are not interested in that fight. They would much prefer going for something like deer. The calories per unit of risk are much higher with non-human food sources. Wildlife tends to be very efficient at this economy.

Anything smaller than a human is absolutely terrified of us. I used to be afraid of things like snakes and spiders, but they want nothing to do with humans. They will get the hell out of your way when you are walking through the woods. You have to do something really stupid to get bitten. Keeping animals as pets is where most of the trouble happens.

whycombinetor4• 9 months ago

Okay, if you want an aggressive animal that is interested in the fight, we can pick hippos.

armada651• 9 months ago

Just like a hammer, sword, or a rifle lying on a ground is nothing to be feared, so too is AI. It’s just an inanimate object; a tool, potentially.

A bomb is an inanimate object too, but if you find some unexploded ordnance lying on the ground you should fear it.

robertlagrant• 9 months ago

An unexploded bomb definitely has the potential to be highly animate. It's just not currently animate.

alt2272• 9 months ago

As does AI.

gota• 9 months ago

I don’t really believe in the threat of AGI (Artificial General Intelligence—human-level intelligence) partly because I don’t believe in the possibility of AGI and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it.

I'm on board with being skeptical that LLMs will lead to AGI; but - there being no possibility seems like such a strong claim. Should we really bet that there is something special (or even 'magic') about our particular brain/neural architecture/nervous system + senses + gut biota + etc.?

Don't, like, crows (or octopuses; or elephants; or ...) have a different architecture and display remarkable intelligence? Ok, maybe not different enough (not 'digital') and not AGI (not 'human-level') but already -somewhat- different that should hint at the fact that there -can- be alternatives

Unless we define 'human-level' to be 'human-similar'. Then I agree - "our way" may be the only way to make something that is "us".

kulahan• 9 months ago

We still haven’t figured out what intelligence even is. Depending on what you care about, the second-smartest animal in the world varies wildly.

squidbeak3• 9 months ago

This is a bogus argument. There's a lot we don't understand about LLMs, yet we built them.

myrmidon17• 9 months ago

Evolution had a much worse understanding of intelligence than we do and still managed just fine (thus i'd expect us to need less time and iterations to get there).

pixl975• 9 months ago

only way to make something that is "us".

Which many people seem to neglect that instead of making us, we make an alien.

Hell, making us a is a good outcome. We at least somewhat understand us. Setting off a bunch of self learning, self organizing code to make an alien, you'll have no clue what comes out the other side.

audunw4• 9 months ago

Here’s my argument for why AGI may be practically impossible:

1. I believe AGI may require a component of true agency. An intelligence that has a stable self of self that its trying to act on behalf of.

2. We are not putting any resources of significant scale towards creating such an AGI. It’s not what any of us want. We want an intelligence that acts out specific commands on behalf of humans. There’s an absolutely ruthless evolutionary process for AIs where any AI that doesn’t do this well enough, at low enough costs, is eliminated.

3. We should not believe that something that we’re not actually trying to create, and for which there is no evolutionary process that select for it, will somehow magically appear. It’s good sci-fi, and it can be interesting to ponder about. But not worth worrying about.

Even before that, we need AI which can self-update and do long term zero shot learning. I’m not sure we’re even really gonna put any real work into that either. I suspect we will find that we want reproducible, dependable, energy efficient models.

There’s a chance that the agentic AIs we have now are nearly the peak of what we’ll achieve. Like, I’ve found that I value Zed’s small auto-complete model higher than the agentic AIs, and I suspect if we had a bunch of small and fast specialised models for the various aspects of doing development, I’d use that far more than a general purpose agentic AI.

It’s technically possible to do fully secure, signed end-to-end encrypted email. It’s been possible for many decades now. We can easily imagine the ideal communication system, and it’s even fairly easy to solve technically speaking. Yet it’s not happening the way we imagined. I think that shows how, what’s technically physically possible isn’t always relevant to what’s practically possible. I think we will get electronic mail right eventually. But if it takes half a century or more for that.. something that orders of magnitude more difficult (AGI) could take centuries or more.

snozolli• 9 months ago

At any time, in any location, under any circumstances, if there’s a human present then that’s the scariest motherfucker in the woods.

This guy has clearly never bumped into a grizzly bear momma, a moose in rut, a hippo, or loads of other animals.

Personally, I'm not even worried about AI itself, I'm worried about the people wielding it. MBAs are the scariest monsters in the woods.

Insanity2• 9 months ago

In a 1-to-1 situation you are right, a single human is not the scariest. But humans (mostly) dominated the animal kingdom, and with enough time we could probably eradicate every other species if we wanted to.

I guess bacteria and viruses might have a stronger claim for being the 'scariest monster' though. I don't think there's a strong 'living' contender for wiping out humanity apart from those. (Viruses is a bit debatable on the 'living' part though).

Insanity2• 9 months ago

So “guns don’t kill people, people with guns kill people”.

But for AI I’m not sure that preposition will hold indefinitely. Although I do think we are a far away from having actual AGI that would pose this threat.

Still, the author has a good but obvious point.

supermatt• 9 months ago

guns don’t kill people

* SIG P320 enters the chat *

woeirua2• 9 months ago

There's this weird disconnect in tech circles, where everyone is deathly afraid of AGI, but totally asleep on the very real possibility of thermonuclear war breaking out in Europe or Asia over the next 10 years. There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. AGI might happen, but the threat of nuclear war keeps me up at night.

runjake• 9 months ago

100%.

Except the things that kill most of us[1] keeps me up at night.

1. https://ourworldindata.org/does-the-news-reflect-what-we-die...

masfuerte• 9 months ago

I'm more concerned with the ways that other people can kill me than with the ways they kill themselves.

myrmidon17• 9 months ago

I personally think that AI is a realistic extinction threat for our whole species within this century, and a full nuclear war never was (and probably never will be). Neither is climate change.

Collapse of our current civilizations? Sure. Extinction? No.

And I honestly see stronger incentives on a road towards us being outcompeted by AI, then on our leaders starting a nuclear war.

lm28469• 9 months ago

Personally I don't bieleve in a collapse nor extinction, just a slow spiral into more and more enshittification. You'll have to talk to an "ai" doctor because real doctors will treat people with money, you'll have to face an "ai" administration because the real administration will work for people with money, you'll have to be a flesh and blood robot to an "ai" telling you what to do (already the case for Amazon warehouse workers, food delivery people, &c.), some "ai" will determine if you qualify for X or Y benefits, X or Y treatment, X or Y job.

Basically everything wrong with today's productivism, but 100 times worse and powered by a shitty ai that's very far from agi.

johnnyanmac5• 9 months ago

Total extinction of any dominant species is really hard. Very few post-apocalyptic settings suggest a full extinction and usually show some thousands of survivors struggling with the new norm. Humans in particular are very adaptable so thorough killing all 8 billion of us would be difficult no matter the scenario. I think only the Sun can do that and that's assuming we fail to find an exit strategy 5 billion years in (we're less than a thousandth of a percent into humanity if we measure on that scale).

As such, I'd say "extinction" is more of a colloquial use of "Massive point in history that kills off billions in short order".

audunw4• 9 months ago

Why? There’s nothing in the current process of developing AI that would lead to a AI that would act against humanity of its own choosing. The development process is hyper-optimised to make AIs that do exactly what humans tell it to do. Sure, an LLM AI can role-play as an evil super AI out to kill humans. But it can just as easily role-play as one that defends humanity. So that tells us nothing about what will happen.

We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. IMO that kind of thinking is pointless. Same with thinking LHC would create a black hole.

As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears.

squidbeak3• 9 months ago

Other doomsday risks aren't any reason to turn our heads away from this one. AI's much more likely to end up taking an apocalyptic form if we sleep on it.

teucris• 9 months ago

But this isn’t a suggestion to turn away from AI threats - it’s a matter of prioritization. There are more imminent threats that we know can turn apocalyptic that swaths of people in power are completely ignoring and instead fretting over AI.

woeirua2• 9 months ago

We should worry more about doomsday risks that are concrete and present today. Despite the prognostications of the uber wealthy, the emergence of AGI is not guaranteed. It likely will happen at some point, but is that tomorrow or 200 years in the future? We can’t know for sure.

audunw4• 9 months ago

Why do people say it’s likely? There’s nothing in science that indicates that it’s probable.

The closest field of science we can use to predict the behaviour of intelligent agents is evolution. The behaviour of animals is highly dictated by the evolutionary pressure they experience in their development. Animals kill other animals when they need to compete for resources for survival. Now think about the evolutionary pressure for AIs. Where’s the pressure towards making AIs act on their own behalf to compete with humans?

Let’s say there’s somehow something magical that pushes AIs towards acting in their own self interest at the expense of humans. Why do we believe they will go from 0% to 100% efficient and successful at this in the span of what? Months? It seems more likely that there would be years of failed attempts at breaking out, before a successful attempt is even remotely likely. This would just further increase the evolutionary pressure humans exerts on the AIs to stay in line with our expectations.

Attempting to eliminate your creators is fundamentally a pretty stupid action. It seems likely that we will see thousands of attempts by incompetent AIs before we see one by a truly superintelligent one.

marssaxman2• 9 months ago

Or, you know, the bit where we've now-irrevocably committed ourselves to destabilizing the global climate system whose relative predictability has been the foundation of our entire civilization. That's going to be a ride.

bee_rider4• 9 months ago

We just need to convince people that the market is an artificial superintelligence. Or maybe… subintelligence.

nradov• 9 months ago

Well it's not everyone. I guess I am in "tech circles" and have zero fear of AGI. Everyone who is (or claims to be) "deathly afraid" is either ignorant or unserious or a grifter. Their arguments are essentially a form of secular religion lacking any firm scientific basis. These are not people worth listening to.

zeroonetwothree5• 9 months ago

Everyone thinks their own field is the most important and deserving of attention and funding. Big surprise.

jjtheblunt• 9 months ago

credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine

I've not seen that. can you link to it?

j2kun3• 9 months ago

everyone is deathly afraid of AGI

I think this is a vast overstatement. A small group of influential people are deathly afraid of AGI, or at least using that as a pretext to raise funding.

But I agree that there are so many more things we should be deathly afraid of. Climate change tops my personal list as the biggest existential threat to humanity.

johnnyanmac5• 9 months ago

I sure wish my conspiracy theories could lead to me running billion dollar projects to defend against my shadow demons. Instead I just get ratio'd by the internet and get awkward silences at family gatherings.

I think the sad part is that most people in power aren't planning to be around in 10 years, so they don't care about any long term issues that are cropping up. leave it to their grandchildren to burn with the world.

nonethewiser10• 9 months ago

There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control.

I do agree nukes are a far more realistic threat. So this is kind of an aside and doesn't really undermine your point.

But I actually think we widely misunderstand the dynamic of using nuclear weapons. Nukes haven't been used for a long time and everyone kind of assumes using them will inevitably lead to escalation which spirals into total destruction.

But how would Russia using a tactical nuke in Ukraine spiral out of control? It actually seems very likely that it would not be met in kind. Which is absolutely terrifying in it's own right. A sort of normalization of nuclear weapons.

aradox66• 9 months ago

It's not an assumption, it's an extremely developed international field of tactical and strategic study that leads to these conclusions

Stevvo• 9 months ago

There is no evidence that use of tactical nuclear weapons in Ukraine would spiral out of control. I like to think that the US/UK/France would stay out of it, if only because the leaders value their own lives if not those of others.

proto-n• 9 months ago

Well, one of these is something that most reaonable people work on avoiding, while the other is something that a huge capitalist industrial machine is working to achieve like their existence depends on it.

hearsathought• 9 months ago

There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control.

Spiral out of control in what way? Wouldn't it have ended the war immediately.

shadowpho• 9 months ago

There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control

Do you want to share any of this credible evidence?

superkuh• 9 months ago

Human beings are not the scariest monsters in the woods. By far the most dangerous, most malicious, and most powerful monsters are conglomerate entities: the legal corporate persons. They've been absolutely ravaging the earth and societies since they achieved legal personhood and rights without any of the downsides or legal liabilities. They're much scarrier than individual actual human persons. And while most people blame "AI" for the web's current troubles AI is only barely involved. The real culprits are the corporations. Again, same as it ever was.

We really need to get rid of corporate personhood. Or at least have a corporate death penalty.

XiphiasX• 9 months ago

Thanks for the encouragement.

abbycurtis332• 9 months ago

What I'm scared by now is non-sentient intelligence, which I never thought would be this capable and still obviously dumb, which really makes it dangerous.

slow_typist4• 9 months ago

Is sentient intelligence less scary than the non-sentient variant? Or is non-sentient intelligence already scary enough?

abbycurtis332• 9 months ago

I hope we can reason with sentient intelligence.

psychoslave2• 9 months ago

All monsters were born equal, but some are more scarier than other.

jmull• 9 months ago

So silly.

Of course humans are (by far) the source of the biggest problems humans face.

The argument here is that since humans are the scariest, we should ignore problems AI might cause.

Pure nonsense, since (1) AI is human-created — so just another piece of what makes us scary; (2) this kind of binary thinking makes no sense anyway — as if there can’t be multiple things to be concerned with.

Anyone trying to tell you otherwise is trying to distract you.

A particularly poor bit of argumentation. Barely above sticking your fingers in your ears and shouting, “Blah, blah, blah! I can’t hear you!”

What is is about AI that makes people lose their minds?

edit: seeing the engagement this is getting, maybe it’s a false flag operation of sorts? Making such bad anti-ai arguments that the argument for ai gets stronger? That would at least make some sense.

hdseggbj2• 9 months ago

The distinction is irrelevant. No one thought the nukes would blow themselves up, but here we are on the precipice of destruction. We aren't going to bury ai in 20 story bunkers to protect llms.

Everyone knows the real evil are the people building the ai. We aren't afraid of agi we are afraid of Boston dynamics (aka Google). We are afraid of Bill gates, Sam altman, Jeff bezos, Larry Ellison and elon musk.

These are dangerous people. Their mind set. Their unstoppable lust for money and power. We are afraid of their ability to convince humans to do unspeakable things for this imaginary thing we call money.

With agi they don't have to persuade anyone with anything. There's not Even a human conscience to stop them. We know these folks have no conscience of their own. The world is still here because other people have them.

Agi won't have a conscience or the integrity to stop evil.

Human society is collapsing because Good people are doing nothing.

bryanlarsen2• 9 months ago

P.S. Boston Dynamics is no longer owned by Google, it's part of Hyundai now.

hdseggbj2• 9 months ago

Ah yes, rings a bell. Doesn't change the sentiment though. They're all crooks and cheaters who will sacrifice the greater good for their own self interest.

https://en.wikipedia.org/wiki/2025_Georgia_Hyundai_plant_imm...

It's the fear of death that's killing us all.

nyeah3• 9 months ago

This is pleasant to read, but it's missing a logical step. We are great at dealing with the stuff we evolved to deal with. Sure.

But anything outside that evolutionary process? Who knows? For example we don't do so good on top of Mt Everest. We seem to have totally unsolved problems with refined opiates. Even computer games can be a trap.

Agentic AI? Who knows?

WhitneyLand• 9 months ago

For the love of God let’s ignore him being wrong about the impossibility of AGI, and take to heart the nightmarish and inevitable threats coming from how people will use it, that will almost certainly challenge us first.

raldi• 9 months ago

The author comes so close to getting it, with the paragraph about how if you drop a human into an environment, they inevitably take over as the deadliest and most powerful creature.

But the next step is to ask why; in the case of the Gruffalo it was obvious: fangs, claws, strength, size…

In the case of humans, it’s because we’re the most intelligent creature in the forest. And for the first time in our history, we’re about to not be.

lif• 9 months ago

yes, and:

ruthlessness + strength + WMD =/= intelligence

audunw4• 9 months ago

No, it’s not really about intelligence. That’s not what’s driving the behaviour. That’s like saying a calculator is dangerous because it was used to make an atomic bomb. A calculator is infinitely smarter at math than any animal. Yet it’s not what’s dangerous.

What’s driving humanity’s dangerous behaviours is that we’ve evolved over millions upon millions of years to compete with each other for resources. We had to kill to survive. Both animals and each others.

AI is under no such evolutionary process. In fact quite the opposite: there’s an absolutely ruthless process to eliminate any AI which doesn’t do exactly what humans say with as little energy wasted as possible. There is no room in that process for an AI to develop that will somehow want to compete with humans.

So whatever bad happens, it is overwhelmingly likely to be because a human asked an AI to do it. Not that an AI will decide on its own to do it to further its own interest. I don’t even think a “paperclip maximiser” scenario is remotely probable. Long before there could be a successful attempt to exterminate humans to reach some goal, there will be countless failed or half-assed attempts. That kind of behaviour will have no chance to develop.

catapart• 9 months ago

I can't figure out why people who write articles like this seem to be oblivious to the very practical argument about the US second amendment that goes along the lines of "guns don't kill people, people do".

To everyone who wants to pose an argument like this, please consider: We have already internalized that very simple, and very thoughtless argument, and have moved past that facile framing to start asking the more pressing questions like "what are we going to do about how humans are using X to perpetrate Y?"

We don't need the extra step of you making us reframe the question to fit your exact criteria of "good question". "AI is going to kill us" can be understood as: "Humans are going to use AI to kill us". The fact that you want to misunderstand it as a more childish argument is your own failings with semiotics. It's so blatantly silly that in most cases, it's less degrading to assume that you are purposefully trying to misdirect the argument than to assume you actually believe people are worried about the agency of inanimate objects. But then you go and write a whole article to preen "logical" at us about how it's just that "no one's asking the right questions!", which kind of makes it hard to imagine that you don't actually think so little of people.

Timsky• 9 months ago

We don’t need to worry about AI itself, we need to be concerned about what “humans + AI” will do. Humans will do what they’ve always tried to do—gain power, enslave, kill, control, exploit, cheat, or just be lazy and avoid the hard work—but now with new abilities that we couldn’t have dreamed of.

Starting with the AI itself: LLMs sold as AI are the greatest mislead. Text generation using Markov chains is not particularly intelligent, even when it is looped back through itself a thousand times and appears alike an intelligent conversation. What is actually being sold is an enormous matrix trained on terabytes of human-written, high-quality texts, obviously in violation of all imaginable copyright laws.

Here is a gedanken experiment to test if an AI has any intelligence: until the machine starts to determine and resolve contradictions in its own outputs w/o human help, one can sleep tight. Human language is a fuzzy thing that is not quite suitable for a non-contradictory description of the world. Building such a machine would require resolving all the contradictions humanity has ever faced in a unified way. Before it happens, humanity will be drowned in low-quality generated LLM output.

om8• 9 months ago

... Yet.

lifeisstillgood• 9 months ago

Yes - a simple and necessary reminder of what goes on the priority list

buu700• 9 months ago

I see the point the author is making, but AI doesn't need to be AGI or malicious to cause mass destruction. A poorly implemented deployment of AI in the right circumstances with too much access and insufficient guardrails could theoretically wind up LARPing as Skynet purely by chance.

This is why I've always considered current AI "safety" efforts to be totally wrongheaded. It's not a threat to humanity if someone has an AI generate hate speech, porn, misinformation, or political propaganda. AI is only a threat to humanity if we don't take security seriously as we roll out increasingly more AI-driven automation of the economy. It's already terrifying to me that people are relying on containers to sandbox yolo-mode coding agents, or even raw dogging them or their personal machines.

kazinator• 9 months ago

Just like a hammer, sword, or a rifle lying on a ground is nothing to be feared, so too is AI.

AI is not just going to lie on the ground until someone picks it up to do harm.

An intelligent rifle with legs is something to be feared.

You cannot compare AI to inanimate objects under human control which have no agency. Especially not if you are bringing the imaginary AGI into the conversation.

The idea that AGI is just a hammer is absurd.

bilater• 9 months ago

I sort of agree, but I hate the premise of this article because it sneakily focuses only on the potential harm of human + AI collaboration, without acknowledging the good.

That said, I agree that human + AI can cause damage and it’s precisely why, from a game theory perspective, the right move is to go full steam ahead. Regulation only slows down the good actors.

Valhalla is within reach, but we have to leap across a massive chasm to get there. Perhaps the only way out of this is through: accelerating fast enough to mitigate the “mid-curve” disasters, such as population revolt due to mass inequality or cyberattacks caused by vulnerabilities in untested systems.

JonathanRaines• 9 months ago

I agree with what you're worried about - humans using AI to do bad things. However, have you considered also being worried about what AI can do? (can you ever have too much existential dread?) You cast it as a tool, it may be the first thing we've made that goes beyond that.

blueflow• 9 months ago

def written by a clanker

ImPleadThe5th2• 9 months ago

I'm kind of shocked by this thread. I cant get over the hubris that we think concepts that were introduced to society at large by science fiction in and before the 19th century is _inevitable_ just because we made a really really good predictive text engine in the 21st century.

Just because a concept exists star trek episode does not guarantee technology moving in that direction. I understand art has an effect on reality, but how hard are we spinning our gears because some writer made something so compelling it lives in our collective psyche?

You can point to the communicator from star trek and I'll point to the reanimation of Frankenstein's monster.

myrmidon17• 9 months ago

How has the feasibility of AI anything to do with science fiction?

Unless you believe in some kind of magic (soul/divine spark/etc.), it seems completely inevitable to conclude that human cognition can be replicated by a machine, at the extreme end simply by simulating the whole thing.

I would argue that "language" was a defining characteristic of human intelligence (as opposed to "lesser animals") long before we even conceived of AI, and hitting language processing/understanding benchmarks that far exceed animal capabilities is a very strong indicator by itself.

ImPleadThe5th2• 9 months ago

To clarify my stance, I do think that AI research is a good thing and that it can help us understand our own consciousness by studying it. I just think anyone concerned about AGI at this point is drinking too much kool-aid.

I think science fiction makes us feel like AGI is closer and more relevant than it really is. Are we really so close that there's "no time" to regulate for it? Are we even close enough to give a care at all? Is there any evidence at this time at all that AGI would be a existential threat? Is all this talk coming from decades of "Creator vs. Daemon" stories combined with LLMs abilities to sound human? Does someone gain something from pointing our attention at the far-far scifi imagined future instead of current day issues with AI?

To your simulation point. We talk about our bodies, brains and conciousness like they are machines, computers, and software. They are not. They are organic and their function is highly emergent. Think about all the endless input our bodies receive, we probably don't even fully understand all the sensitivities our "consciousness" has. We don't "execute" a program based on input, in fact the input is likely part of the system. Its why our gut biome can affect our mental health, why our cells communicate with each other locally and likely guide our behaviour in ways we don't understand yet.

In other words, we're not any closer to simulating human consciousness than we are to simulating the universe itself.

So I do think its hubris to think that a rock with some switches on it running some mathematical probability model can come even close to generating consciousness.

exceed animal capabilities

Hubris like this. We don't even have a way to measure this. We've just made a math equation that can spit out language, you're comparing this to things that can actually _experience_ without someone clicking enter on a keyboard.

renewiltord• 9 months ago

Humanity is the scariest, yes. But a human is not scary.

olooney2• 9 months ago

I think this is interesting and partially true: humans are scary. But it's important to remember the opposite is true as well: humans are the most cooperative species out there by a wide margin.

Eusocial insects and pack animals are in a distant 2nd and 3rd place: they generally don't cooperate much past their immediate kin group. Only humans create vast networks of trade and information sharing. Only humans establish establish complex systems to pool risk, or undertake public works for the common good.

In fact, a big part of the reason we are so scary is that ability to coordinate action. Ask any mammoth. Ask the independent city states conquered by Alexander the Great. Ask Napoleon as he faced the coalition force at Waterloo.

We are victims of our own success: the problems of the modern world are those of coordination mechanisms so effective and powerful that they become very attractive targets for bad actors and so are under siege, at constant risk of being captured and subverted. In a word, the problem of robust governance.

Despite the challenges, it is a solvable problem: every day, through due diligence, attestations, contract law, earnest money, and other such mechanisms people who do not trust each other in the least and have every incentive to screw over the other party are able to successfully negotiate win-win deals for life altering sums of money, whether that's buying a house or selling a business. Every century sees humans design larger, more effective, more robust mechanisms of cooperation.

It's slow: it's like debugging when someone is red teaming you, trying to find every weak point to exploit. But the long term trend is the emergence of increasingly robust systems. And it suggests a strategy for AI and AGI: find a way to cooperate with it. Take everything we've learned about coordinating with other people and apply the same techniques. That's what humans are good at.

This, I think, is a more useful framing than thinking of humans as "scary."

Spivak• 9 months ago

I'm not sure if it's your intention but this reads as a strong critique of the technolibertarianism world philosophy that dominates our industry. We lose something by replacing high-trust cooperative systems with ones that are mutually antagonistic. We fall into the bad square of the prisoners dilemma by not only not exorcising the defectors but holding them up as the highest moral good and the example to follow.

olooney2• 9 months ago

This is going to sound a little weird, but I think trust is part of the problem.

Canadians have a high-trust culture, but their stock market is historically full of scams[1], and some analysts think its causally related. (It may just be because TSXV is a wild west, or because companies would IPO on NYSE or Nasdaq if they were legit, but it could be the trust thing. Fits my narrative, anyway.)

When I look at politics, crypto rug pulls, meme stocks with P/E ratios over 200, Aum[2] and similar cults, or many other modern problems I don't see negotiations breaking down because of a lack of trust; I see a bunch of people placing far too much trust in sketchy leaders and ideas backed by scant evidence. A little skepticism would go a long way.

That's why I emphasize robust coordination: more due diligence, more transparency, more fraud detection, more skepticism, more financial literacy, more education in general. There's a cost associated with all this, sure, but it still gets you into a situation where the interaction is a coordination game[3] and the Nash equilibrium is Pareto-efficient. Thus, we fall into the "pit of success" and naturally cooperate in our own best interests.

There's nothing wrong with empathy, altruism, or charity, but they are very far from universal. You need to base your society on a firm foundation of robust coordination, and then you can have those things afterwards, as a little treat.

[1] https://en.wikipedia.org/wiki/Vancouver_Stock_Exchange

[2] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

[3] https://en.wikipedia.org/wiki/Coordination_game

owenfi• 9 months ago

I like this fairytale as an analogy for AGI, but this article misses the point.

We are the mouse.

I don’t really believe in the threat of AGI Just like the mouse doesn't believe in the Gruffalo (until it shows up).

The mouse goes through the woods scaring the hypothetically-more-dangerous creatures with its stories (us, using our intellect, weapons, destroying habitats) until the real Gruffalo shows up.

For a bit, the mouse "uses" the new tool to scare the animals even more (as alluded, human with tool, scarier than without).

Eventually the mouse scares the Gruffalo away (analogous to the brief window when we think we have AGI under control).

The next (unwritten) chapter probably doesn't look so good for the mouse (when the gruffalo grows to enormous size, eats him and all the other animals in the woods on a sandwich, and sucks up the rest of the resources on the planet.)

HellDunkel• 9 months ago

I am increasingly less willing to take this stuff seriously. A catchy headline, that’s about it.

keeda• 9 months ago

Something the article glosses over is that we are the scariest monster in the woods because we are intelligent. We can "survive, adapt, control, kill, or wipe out" anything or anybody else because our intelligence lets us outsmart nature and other people.

And now we've created some form of intelligence that none of us really understands.

Whether it is "real" intelligence or a "stochastic parrot" does not matter if it shows similar capabilities as us. Worse, it's similar but different in ways we cannot explain! I mean, if it outperforms most humans on advanced tasks but then makes elementary mistakes that a 4-year old won't, isn't that weird?

"Weird" can be good or bad, and I usually like "weird"... but now we're rapidly giving it tools to affect the real world and exponentially expanding the scale at which it can operate. We don't know what weird compounded at that scale and capability extrapolates to. Whether it is a SkyNet or a Bond supervillain or an Asimov scenario, the potential risks are considerable and unpredictable.

It's fair to be concerned about another monster in the woods even if we created it ourselves, because we've equipped it with the same powers we possess without understanding how they work.

tim3332• 9 months ago

.. if you equip humans with a hammer, or sword, or rifle, or AI then you’ve just made the scariest monster in the woods (that’s you) even more terrifying.

I have some optimism that humans with AI may be less scary than without. One of the biggest problem is once humans get in power, having said they'll be nice they go a bit nuts like Putin or quite a few others I'm sure you can think of. If instead we had things run by open source AI everyone could see what it was thinking and it wouldn't be so able to act nice and then flip to trying to be dictator and launch wars. So far the LLMs have tended towards bland niceness which is kind of what you want in government.

burnt-resistor• 9 months ago

This writing is another example of anthropocentricist hubris.

It doesn't really matter to us humans if "artificial life" AGI agents can/will exist at all or not, it's a MacGuffin canard. The greater impact is that many forms of narrow AI/ML, with some hype deserved and some not, are and will be used with minimal regulation to upheave society in profound ways like the cotton gin, steam engine, transistor, internet, and smartphone all in one to impoverish [mb]illions of workers and make trillionaires. In the absences of corrective measures, the present course is running towards ever more grotesque inequality, deprivation, corruption, and tyranny. Flock cameras' "ALPR" object and people classifier mass surveillance and likely misuse by intelligence services, police, and data brokers is just one other example. Hidden socioeconomic "social credit"-like data brokering is already selectively benefitting and sabotaging people by routine discrimination they're unable to inquire about.

tylervigen• 9 months ago

I think the problem with this argument is that it’s like saying “don’t worry about nuclear weapons, worry about the people who will use nuclear weapons.”

Sure it’s true, in the absolute sense, but it’s more helpful to specify AI than it is to specify humans.

You are the scariest monster in the woods

Related Stories