Superintelligence: An idea that eats smart people (2016)

https://idlewords.com/talks/superintelligence.htm

by aebtebeten • 3 years ago

214 352 3 years ago

"We’ve learned that at least one American plutocrat (almost certainly Elon Musk, who believes the odds are a billion to one against us living in "base reality") has hired a pair of coders to try to hack the simulation."

Source?

saurik• 3 years ago

I know nothing about this, but I do know how type the question into Google, and I got this back, which sounds like what Maciej is talking about (if maybe ever so slightly corrupted).

https://www.businessinsider.com/tech-billionaires-want-to-br...

crosen99• 3 years ago

That article links to the below article where Musk lays out his version of the simulation argument. His version is fraught to the brim with confused thinking and non sequiturs, and yet I’ve never seen him called out on it.

https://www.businessinsider.com/the-question-elon-musk-refus...

voidfunc3• 3 years ago

I like how there is an assumption that any simulation we run inside even follows rules, physics, or mathematics we can comprehend. If this is actually a simulation it's quite possible the outside world is so completely different it basically breaks everything we think we know.

kirubakaran2• 3 years ago

I think the idea is that *if* we're in an ancestor simulation, then "they" will make the rules as close to theirs as they computationally can. Why ancestor simulation? Why else would "they" spend all those resources?

Lot of IFs based on our view of their values, I know... I don't necessarily agree with this line of reasoning... just stating how I understand the argument.

zmgsabst• 3 years ago

And yet, in Minecraft you can build devices that detect features of the underlying software — eg, the update policy, circuit cache size, and server lag.

Perhaps we won’t understand the reality “out there”, but attempting to may nevertheless produce tangible benefits for us “in here”, eg, like wireless communication in Minecraft arises from detecting subtle variations in updates.

galacticaactual2• 3 years ago

Strong agree. It's like expecting video game characters (however sentient they may eventually / theoretically become) to understand the hardware instructions that power the assembly language that is running the process within which their reality exists.

akomtu• 3 years ago

The public thought got fixated on the VM style simulation. Consider another idea. The baseline reality is spacetime that can form any configuration with any number of space and time dimensions and everything in between. We live in the 3+1 layer, which doesn't exclude the existence of other 3+1 layers, just like one flat valley doesn't prevent the existence of other valleys. Those who know how to work with that spacetime can create arbitrary worlds, e.g. a toroidal 4+3 world. These world can be connected to each other in strange ways and transition between them may or may not be possible. Some local spacetime can be nested into bigger soacetimes with different configuration.

adamisom2• 3 years ago

I think it's fair to say that obviously, anybody who explores 'breaking' the simulation is also aware of this and all their hopes are pinned on the admittedly infinitesemal chance that we're in a simulation that can be broken

anothernewdude2• 3 years ago

We are living in a base reality by definition.

adastra2210• 3 years ago

No, we are not? I am confused what definition you are using. If the Matrix were real, you would say the Matrix is base reality?

pshc• 3 years ago

BARRET: Have y'all ever considered that none of this is real? That we're just part of some game, a simulation?

TIFA: Barret, what are you talking about?

AERITH: It's an interesting thought, though. Sometimes, everything does seem too surreal to be true.

BARRET: Exactly! Maybe we're just codes in a computer, or pawns in someone else's game.

CLOUD: Barret, that's ridiculous. Even if it were true, how would that change anything?

CLOUD: We are living in a base reality by definition.

TIFA: Cloud's right. Whether it's a simulation or not, this is our reality. We have to deal with it.

AERITH: Maybe the real question should be, what can we do to make this reality better?

BARRET: Hmph, you're all just too afraid to face the truth. But mark my words, one day you'll see.

cubefox2• 3 years ago

[2016]

dang2• 3 years ago

Added. Thanks!

TheDudeMan3• 3 years ago

I wonder if the author still feels this way. I could only get through about 5 of the arguments against superintelligence -- I did not find them very convincing.

adastra2210• 3 years ago

It is not an argument against super intelligence. It is an argument against the idea that super intelligence leads to kill-all-humans by default.

Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.

cubefox2• 3 years ago

Arguments for existential risks have become much better known since then. In 2016 most of it was concentrated on a small niche almost nobody knew about.

RecycledEle• 3 years ago

I assume the most intelligent thing will become the boss.

The question is: what does that most intelligent thing want?

My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.

throwaway_ab• 3 years ago

It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.

jerf4• 3 years ago

The argument is more properly seen not as "an AI will inevitably kill all humans", but "the set of AIs that simply neglect humans is far larger than the space of ones that care for them, and if they become the dominant power in the local ecosystem even simply neglecting humans will make it the most dangerous thing we've faced".

Yes, from our perspective this makes it look like it will kill all humans, but it would do so in the same way that a particular ant hill believes I have a vendetta against them, when in fact I was just clearing dirt to pour an extension on my driveway.

That said, a murderous AI is more likely than we may like simply because our militaries have the money to fund them, and they basically already exist. They just aren't hooked up to anything at the moment that makes them an existential risk to the species. But time is deep, and even thinking about "the next century" is a provincial point of view in the end. So worrying about what happens if someone forgets the "but don't kill the good guys" switch is at least worth talking about over the next 100 years. (To say nothing of the ethics of who decides what the "good guys" are and related issues.)

idlewords19• 3 years ago

To be more specific, it's an argument against the idea of Bostrom/Yudkowsky-style recursively self-improving superintelligence, i.e. the brain surgeon who repeatedly operates on the part of his own brain that makes him good at brain surgery, getting faster each time.

It's also a cry of impatience against people who think they can model or forecast the actions of a non-human intelligence, let alone a superintelligence. AIs are alien sociopaths; it's a category error to believe you can get inside their head.

jerf4• 3 years ago

While I have my own reasons to disagree with the worst-case scenarios of AI alarmists ("radical and irreconcilable differences in presuppositions about the most fundamental aspects of reality"), an awful lot of arguing about why they are wrong really boils down to just argument from incredulity.

Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here.

It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.

famouswaffles3• 3 years ago

Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set

There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong.

LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.

visarga9• 3 years ago

It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT.

I think it could do that in software. Assuming compute is no issue we have:

- LLMs writing code, explaining code, changing code, and observing code execution

- LLMs that understand ML concepts and can explain their own workings

- LLMs can generate the training set all from inside (see TinyStories)

- LLMs can make "RLHF" data for the fine-tuning (see Alpaca, tuned with GPT3.5 and GPT4 data from LLaMA)

If we take a look, it seems LLMs can self replicate in software with nothing else but compute and a neural net framework. Of course making the chips is a whole other story.

mike_hearn4• 3 years ago

I don't know what the next step will be, I'm just confident there will be one.

But we don't know when. AI is a bit unusual in that it had a winter, unlike most other aspects of computing which have seen much more consistent progress. Given past performance it's entirely possible that progress in AI will just stall. Arguably it had already stalled thanks to there being only a handful of companies that were able and still interested/funded to create models, and most of those decided not to actually let anyone use the results. OpenAI dominates mindshare exactly because there are so few organizations that both can and will do this stuff well. So there's lots of ways AI progress could go off the rails again.

JieJie2• 3 years ago

This one actually was prescient,

"Data Hunger

As I mentioned earlier, the most effective way we've found to get interesting behavior out of the AIs we actually build is by pouring data into them.

This creates a dynamic that is socially harmful. We're on the point of introducing Orwellian microphones into everybody's house. All that data is going to be centralized and used to train neural networks that will then become better at listening to what we want to do.

But if you think that the road to AI goes down this pathway, you want to maximize the amount of data being collected, and in as raw a form as possible.

It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible."

Edit: And this one too, "In the near future, the kind of AI and machine learning we have to face is much different than the phantasmagorical AI in Bostrom's book, and poses its own serious problems."

idlewords19• 3 years ago

There is as yet insufficient data for a meaningful answer.

EGreg• 3 years ago

This about sums it up: https://www.dailymotion.com/video/x27dau3

Waterluvian• 3 years ago

So, does an atomic explosion produce some magnesium that way?

j9461701• 3 years ago

This was a fun rabbit hole to go down. Magnesium does not seem to be present in radioactive fallout, there is more Manganese than magnesium from the data I was able to find online. Further NOX, the chemical combination of oxygen and nitrogen is a serious pollutant by-product of nuclear explosions indicating nitrogen seems to remain largely elementally unmolested by the blast. Finally, looking at the calculations it seems to fuse nitrogen nuclei in any kind of meaningful number would require atomic detonations with heat in excess of tens of billions of degree, while real contemporary explosions produce only hundreds of millions of degrees.

swader9996• 3 years ago

The other claim I wondered about was that a nuke created the hottest temps ever on earth. Wouldn't a large asteroid strike come close? Edit: chat g 4 set me straight, nukes are tens of millions F, asteroid impact thousands of degrees F.

mode802• 3 years ago

Let me save you 20 minutes of your life: This guy's lead argument is that the smart people who advise caution in creating super-intelligence look weird. He's quite proud of this argument.

ImaCake2• 3 years ago

They do look weird to a lot of us. In excel, its trivial to fit a smoothed bendy line between points on a scatter plot. Because of the underlying math (taylor series) when you reach the end of your data points the line flings off into positive or negative INF.

AI super intelligence arguments have the same logic: the data ends so the interpolation now goes to infinity. This is such an obvious counter and these people know this math.

gunshai• 3 years ago

I'll be that guy for a moment.

Extrapolation not interpolation, proceed with downvotes.

coldtea9• 3 years ago

Well, then again you seem quite proud of this "summary".

jodrellblank16• 3 years ago

There is nothing about anyone's appearance anywhere in that transcript?

mode802• 3 years ago

I'll expand for those asking. The first third of the presentation does such an excellent job of steel-manning the case for concern, that I couldn't wait to hear his arguments against it. When he gets around to that, he reaches for a made-up concept he calls the "outside view" where he argues you should ignore rational arguments (aka the "inside view") if the person making those rational arguments seems weird.* Slides follow showing VCs in bad PR photos. What more evidence does anyone need?

*"But the outside view tells you something different. ... Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking."

astrange4• 3 years ago

He's right. Predictions about the future aren't actually more accurate because they have hundreds of pages of probability theory. Cults look convincing from inside the cult.

One should note that the AI people thought AI would look completely different than an LLM does (they thought it'd be an agent using a manually coded knowledge graph) but are still applying all their same arguments to this very different thing.

mjklin• 3 years ago

By “outside view” I believe he is referring to the anthropological term “etic” so it is not made-up. See: https://en.wikipedia.org/wiki/Emic_and_etic

lz400• 3 years ago

One of the things that shocked me the most about the futurist/ai/techbro sphere is that folks take people like Eliezer Yudkowski seriously. Lex Friedman asked him what's his advice for young people and Eliezer answered "don't expect to live long". He's advocated for airstrikes on datacenters that try to advance AI. I don't know what to say, if these are the intellectual references of our brightest minds, makes me terrified that they're all children mentally.

droopyEyelids• 3 years ago

MENSA culture

adamisom2• 3 years ago

Yudkowsky sticks out because he's weird and his position extreme, but by no means is his p(doom) representative of the increasingly-many worried researchers.

I find it tricky to think about cases like Yudkowsky (full disclosure I used to read LessWrong a lot), because if he has sufficient credibility then loudly staking your extreme position can indeed move the Overton window.

timmytokyo• 3 years ago

Yudkowsky comes across as the archetypal fedora-wearing nerd. His constant self-flattery is socially obtuse, his relevant credentials are entirely lacking, and his doomerism is an instant turnoff to a large segment of the public. He's a perfect example of the OP's "outside argument".

If Yudkowsky is moving the Overton window, it's in a direction opposite from what he intends.

p3rls• 3 years ago

Does he still complain on twitter about women not finding his fedora attractive?

JieJie2• 3 years ago

There is a quote from Eliezer in the linked article that I hope he is remembered by, because it's actually quite beautiful:

"Coherent Extrapolated Volition (CEV) is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted."

We should, as a society, offer help to people who are caught in the cynicism trap, because we could still have all those things Eliezer sees as our better selves. We just have to spend more time focusing on being on our better selves and less time giving up on each other.

roenxi• 3 years ago

I don't know what to say, if these are the intellectual references of our brightest minds, makes me terrified that they're all children mentally.

It is part of the fear - human intelligence is extremely limited. That could easily be representative (although to be fair, probably not).

Reasoning about the future is really hard and usually everyone who makes predictions is wrong. So far we've been fine through pretty much every challenge. That being said, "we'll survive this" is a statement that is only false once and this one looks pretty serious. We might be dealing with a world where human intelligence is economically uncompetitive for the first time in recorded history.

chubot• 3 years ago

Totally agree. There do seem to be a bunch of accomplished people that have acquired "mind bugs" from people whose job it is to write and be popular, and not produce very much.

Yudkowsky is a prime example of the latter group. I think I became aware of him through the Roko's Basilisk meme many years ago, and I thought "wow so this is what happens when you have a lot to think about and nothing to do". It's basically nonsense.

Two other people like that are Nick Bostrom and Will MacAskill.

I heard of "Superintelligence" by Bostrom probably through an Elon Musk recommendation in 2015 or so. That's when he started OpenAI, and I was working on AI at the time. (Remember Elon had a completely different reputation then; he was respected as a person who built things, and wasn't associated with any political ideas.)

So I got the book, and to its credit, the beginning of the book acknowledges that the rest of the book may be complete bullshit. It's extremely low confidence extrapolation. And I pretty much agreed with that preface -- most of the book was nonsense. However many people seemed to take it seriously.

(I'm not saying AI is going to be great, or catastrophic either. I was just looking for some kind of high quality analysis from someone with domain knowledge, and found none of it in that book.)

----

I think I became aware of MacAskill through a NYTimes article puffing his book last year. There was something "off" about it.

And few months later, it turned out that he was conveniently unaware that the only reason that his projects were funded was because his friend was committing a huge crime:

https://time.com/6262810/sam-bankman-fried-effective-altruis...

So yeah, obviously I can't take seriously the moral opinions of somebody whose morals fall down immediately and spectacularly when faced with a whiff of the real world.

It seems like his main job is to create and advocate ideas that are intellectually appealing to and in the interest of billionaires. It came out in the FTX aftermath that he was telling Elon that SBF could help him buy Twitter.

BTW Elon also recommended the MacAskill book, which another thing that has lowered my opinion of Elon. People are people are flattering him with ideas, and he's taking the bait.

----

Anyway, normally I wouldn't have any awareness of the kind of writing that these 3 people produce. Much of it is low quality, which I would expect given their lack of experience and expertise in the domain of AI. And 75% of it couldn't have any effect on the world -- even in principle.

But like you, I'm puzzled that some people I respect seem to take them seriously. I mean I do respect Fridman, since I've learned a bunch of things from his podcasts with other guests. (I understand why a lot of people don't like him, but I prefer to focus on a person's best work and not their worst.)

But yeah the best work of Yudovsky, Bostrom, and MacAskill doesn't seem noteworthy.

It seems designed to generate attention, and that's mostly it.

The extremely, extremely obvious thing they're missing (or just fail to emphasize because it won't get them attention) is that you should be scared of corporations and governments with AI (first), not scared of autonomous AI with its own will.

That is, the "superintelligence" and "longterm-ism" ideas are basically deflections from the real issues with AI.

----

So yeah I was puzzled about this whole thing, and someone on HN recommended this substack, and I recommend it too. It's pretty interesting and it draws a line between the 90's extropians@ mailing list, Yudowsky, Bostrom, and multiple founders of Bitcoin.

Also a lot of dark stuff like suicides, mental illness, and abuse in the rationalist movement.

https://aiascendant.substack.com/p/extropias-children-chapte...

What also seems to have happened in this era is that Yudkowsky and MIRI, flush with newly donated millions, decided to try to evolve from theory and community-building to actual practical AI research. It did not go well.

This was quite a claim: essentially, that this research would, if published, meaningfully accelerate perhaps the most extraordinarily difficult challenge in the history of human innovation. It is a claim which remains 100% unsubstantiated.

rglover• 3 years ago

airstrikes on datacenters

This is the most punk rock idea I've heard in awhile.

jltsiren2• 3 years ago

I think discussions about superintelligence are mostly pointless. They are storytelling at best, and often little more than bullshitting.

Language is an imperfect model of the reality. The further a discussion deviates from the observable reality, the less confident you can be that the conclusions you make within the model are also valid in the reality. Once you are talking about something as wildly hypothetical as superintelligence, words have basically lost their connection to the observable world.

If the history of science teaches us anything, it's that smart people are often stupid. They come up with all kinds of silly ideas, because they rely too much on reason and too little on hard work. And even the ones who ended up revolutionizing the world usually sound ridiculous in retrospect, if you listen to all of their ideas instead of just the successful ones.

Maybe there are some valuable ideas in the superintelligence discussion, but you can't identify them in advance.

sheeshkebab• 3 years ago

The important part about current ai craze is that it’s the first realized approach that somewhat resembles all the theories, fantasies, and bull shit about it. Can it eventually lead to superintelligence? it seems like it’s stepping in the right direction - and we’ll know soon enough if it’s another dead end (possibly even within few short years… which is a great leap from time scales of decades until recently)

lucaspfeifer• 3 years ago

If you have a good enough model, you can identify anything.

To discuss super-intelligence, we should first define it. Wikipedia says that it is intelligence surpassing the brightest of human minds. Taken as a whole, the internet already has the knowledge of a super-intelligence: it contains more useful information than any individual human. But it is far more limited in its control and use of information.

LLMs rely heavily on inferences from their training data, meaning that they struggle to generalize to new situations. If you had a program that could use abstract reasoning to learn any topic, then it could solve any problem better than a human, given that a supercomputer can process and store more data than a human. This program would be a super-intelligence.

I expect that the development of intelligence (software) superior to humans will happen much faster than the development of superior hardware did, based on the timescales of human evolution (billions of years) compared to the evolution of civilization (thousands of years).

bbor10• 3 years ago

I’d like to push back, if you have the time to clarify. Are the following summaries accurate renditions of your points?

1. We shouldn’t worry about SI because we haven’t seen one yet. 2. We shouldn’t listen to smart people, because they’ve been wrong before.

Because if so I obviously don’t find those convincing. The whole reason the SI cultists are so “alarmist” is because by definition this is the kind of problem we have to preempt, not run into and then wing it.

If someone responded to concerns about the atmosphere igniting into nuclear fire with “that’s never happened before and the only people worried about it are scientists, so don’t worry about it” instead of equations… well I’d be damn well terrified

AnimalMuppet2• 3 years ago

How do you respond to concerns about fairies stealing your children?

For that matter, how do you respond to a calculation (there was one!) that railroad trains could not exceed 41 miles per hour, or all the air would be forced out of the cars and all the passengers would die?

I believe that jtsiren's point is that we're not at the point where we can even define intelligence. We can't calculate anything, because we can't sensibly define any of the variables in the equations. All we can do is make guesses about terms we can't even define. We're like stone-age people worrying that if their neighbors' cooking fire gets too hot, it is going to light the air on fire and we're all going to die, and nobody can reassure us because nobody even knows what burning really is, or what the air's made of. We're millennia away from being able to do the kind of calculation that you want.

So the only choices available are to proceed, or not. And if your answer is "not", you need to convince everybody, because I don't think we're going to (for example) nuke North Korea to stop their AI program unless you've got a really convincing case. Which nobody has now.

senttoschool5• 3 years ago

I think discussions about superintelligence are mostly pointless. They are storytelling at best, and often little more than bullshitting.

That's like saying in the 1800s that we shouldn't ever investigate atoms because it's mostly pointless and storytelling at best.

Personally, I think we need to be discussing this now. Smart people need to be. We need to come up with models for AI intelligence, which might help us predict when and if superintelligence occurs.

jltsiren2• 3 years ago

Philosophers have been talking about atoms for thousands of years, and most of it turned out to be pointless bullshitting. Then, a few years before 1800, scientists started using atoms an an explanation for measurable phenomena in chemistry. And that's how scientific theories of atoms came to be: not as hypothetical constructs, but as explanations for something that could be observed.

motohagiography• 3 years ago

When the researchers come in on Monday, the AI has become tens of thousands of times funnier than any human being who ever lived. It greets them with a joke, and they die laughing. In fact, anyone who tries to communicate with the robot dies laughing, just like in the Monty Python skit. The human species laughs itself into extinction.

I've given this topic a fair bit of thought as well and this is a legitimate use case. Laughing is the obvious one, but I recently ran down the rabbit hole on "Modal Logics" from Kripke, and I think his work is formal enough that it can effectively arm a model with the ability to generate ideologies, formed in language that is refined to create a fully entraining state.

There are only survivors in zombie movies, the real situation created by an AI formulating human ideologies would be much, much worse.

coldtea9• 3 years ago

We know from theory that the physical limits to computation are high. So we could keep doubling for decades more before we hit some kind of fundamental physical limit, rather than an economic or political limit to Moore's Law.

Huh? Haven't we already hit close to the end of Moore's Law, and are compesating by adding cores (which is a different thing)?

And aren't we also pushing near the physical limits regarding the cpu nanometer race?

zamadatix• 3 years ago

Moore's Law is about a relation of the number of transistors in a given cost chip, not just their size. Whether they have 1 core vs 4096 or whether the chip is 1 mm^2 vs1000 mm^2 doesn't really matter in terms of the law.

That said, I think the trend of transistor growth at a given cost has started to slow according to most graphs.

coldtea9• 3 years ago

Whether they have 1 core or 4096 doesn't really matter.

It does however matter to the looser Moore's law expectations, which were about increased speed. Now this happens only for more parallelizable programs, as opposed to the automatic speed increase bumps programs got from Moore's law in the single-core eras...

fwlr4• 3 years ago

Few things have been tortured more than “Moore’s Law”, which originally meant an empirical observation that number of transistors on a single integrated chip of fixed size seemed to double every two years, but was broadly understood to mean “computation doubles every two years”, and as we found other ways to increase computation besides making smaller transistors we tended to gather those ways under the Moore’s law umbrella as well. Referencing Moore’s Law very rarely adds clarity, in my opinion. The cpu nanometer race is also similarly tortured: 3 nanometers might be close to the physical limit of semiconductor computation but that doesn’t mean anything, since “3nm” means 7 nanometer precision and a half-pitch length of ~14 nanometers. https://twitter.com/davidad/status/1661595361939533827

But enough nitpicking; the actual answer to your question is that those are about physical limits of our current hardware implementation of computation. The theoretical physical limits of computation that the author is thinking of are bounded by things like Margolus-Levitin’s limit of 6x10^33 operations per second per joule (I had to look up the SI prefix; it’s several thousand quettaflops). https://en.m.wikipedia.org/wiki/Margolus%E2%80%93Levitin_the...

coldtea9• 3 years ago

But enough nitpicking; the actual answer to your question is that those are about physical limits of our current hardware implementation of computation.

Yeah, but unless we come up with another "hardware implementation of computation", which I don't see happening anytime soon, those are our limits for the next few decades at least, if not centuries (if not forever).

dist-epoch3• 3 years ago

The ultimate computers is black-hole like. We are far far away from reaching that limit.

https://www.newscientist.com/article/dn8836-black-holes-the-...

Physicists Say Aliens May Be Using Black Holes as Quantum Computers

https://www.sciencealert.com/physicists-say-aliens-may-be-us...

rogers184453• 3 years ago

The reason why regulation is going to fail is simple. AGI is the supreme bet, the Gamble to Resurrection. Faced with death years to decades from now or a chance at amortality. Which would you choose?

Would you be willing to bet on p(doom) whatever it might be when p(amortality | AGI) >>> p(amortality) during your lifetime? I personally am willing to place such a bet, and I would hazard a guess that decision makers don't come to their positions without similar sentiments.

paulcole4• 3 years ago

Faced with death years to decades from now or a chance at amortality. Which would you choose?

Death, 100%. I know that death doesn’t have any downsides (although dying sometimes has some) and it’s worked for everyone who’s lived so far.

habitue• 3 years ago

death doesn’t have any downsides

The fact that most people try to avoid death by any means possible is a good indication it has downsides.

Specifically, the fact that you can no longer go on living is a big downside of death in my mind.

ChatGTP3• 3 years ago

I think this is what our tech overlords are investing all the money for. They're getting older quite quickly and probably feel like immortality is just around the corner if they keep syncing money into AI research and keep taking risks, they will cheat death.

I compare it to the the great pyramids. The pyramids were arguably another technological marvel also built on the back of slaves (just like modern tech now) and probably a lot of great inventions and innovations were made to facilitate their creation. To people of the time, the pyramids must've been absolutely incredible marvels to behold.

But all those Pharo's are dead, just like everyone else. I think the current tech overlords will die like everyone else too.

Nature already has a way for people to live on, it's called having children and dying. It's a great system and no one ever gets bored. However, the ego of some people is so strong and isn't satisfied with this, so on we go pouring money and resources into the search for the holy grail no matter what the risks are to everything else.

rogers184453• 3 years ago

To some people death and the end of the universe are equivalent events. Others seek refuge in natural fallacies.

The difference between this moment and all others, is that amorality is actually achievable. If you can get an AGI and drown it in compute, through sheer brute-force the secrets of biology will be unraveled. The task is not to attain amortality immediately, merely gain more life faster than you lose it.

Balgair• 3 years ago

Over the pandemic and thereafter my family has had a lot of deaths in it. So, I've been forced to think about it a lot, to sit with others over it, and to just deal with a lot of the mundane parts of it too.

Death is horrible. It makes no sense. When people say they have had a 'loss', they're not kidding. You feel like that person should still be there, but like a kid at Disneyland, they are lost to you and you're searching for them. I'm dealing with a fair few relatives that just are not processing all these losses. Grief is a strange, personal, and unique thing too. It manifests personally and yet stereotypically for every individual.

That said, I think Death is the way.

The reasoning is complex and long. So, I'll try to sum it up for a simple comment, and I'll do that poorly, sorry.

The big reason stem from this article on Wikipedia. I think it's one of the best there is on the whole site:

https://en.wikipedia.org/wiki/Timeline_of_the_far_future

Say you were truly immortal. That timeline would be your future, as far as we currently understand physics. You get to spend a lot of time between universes if the last entry is to be believed. In fact, they don't even bother with giving the units. Nanoseconds or gigayears are pretty much the same st those timescales. Our time here on Earth is essentially as brief as the entire non-black-hole era to an immortal like that. Purgatory in the black void is more like what such an immortal would experience.

Or say you get to relive your life when you die. Poof, you're reborn to your folks and have all your memories again somehow. Repeat forever. You're doomed to live and die the same life, like 'Groundhog Day', but for ~80 years long and not just a day. Another purgatory after enough lives, I'd guess. Sisyphean.

Heaven, hell. An afterlife is our best hope. Somewhere we can't possibly understand with our minds right now. A total lack of understanding of the life hereafter is the only path where you retain something of you. Where you can grow and change, time can continue in, I dunno, 7 dimensions. I've not a clue, and I think that's great. If I did, right now I think you'd just end up in a form of purgatory given enough time.

But an endless dreamless night is just fine too. In no other way except an afterlife that I cannot possibly understand do we get to have 'happily ever after'. I think everyone would take that Socratic apology given enough time and I think they'd be right to do so.

I dunno, been sitting on this a while, and it's late for me and, again, sorry that this is a brief and jumbled comment.

dang2• 3 years ago

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)

Superintelligence - https://news.ycombinator.com/item?id=26217372 - Feb 2021 (7 comments)

Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)

idlewords19• 3 years ago

Please just make the evergreen page and retire articles like this there so they can romp and play on a big farm upstate, and not have to endure the constant reposting.

tptacek• 3 years ago

I think the new threads on these reposts are valued, which is something you'd lose with an evergreen page (I'd still like to see one).

swader9996• 3 years ago

The predictions in this article are somewhat mute because existential problems within AI should manifest before AI becomes smart and capable of wiping us out. AI will augment and speed up all parts of society, good and bad. Imagine rapid advancements in these areas for instance: genetic engineering, wealth creation, uranium refinement, drones, bio weapons, election manipulation, general crime, murder for hire. You think our eighty year old representatives are going to get ahead of this? Think they could even regulate it?

I can download and run the latest uncensored Vicuna model and get started on all this right now. Give it a few months, years or a decade. Its unstoppable, no way to contain it or its acceleration.

ChatGTP3• 3 years ago

You think our eighty year old representatives are going to get ahead of this? Think they could even regulate it?

Maybe, because while you're so negative and ageist, I watched the congress hearing with Gary Marcus and they actually have the wisdom to know it's out of their depth and an agency with younger and smarter people who know about this stuff needs to be setup.

There is no need for such negativity towards older people.

swader9996• 3 years ago

I'm actually pretty old myself. I like old people. But I don't think they'll all be able to competently manage the needed lifting to consider this properly. There aren't even experts that can give us a path.

Fair points you made though. It was a weak and lazy angle I made.

A better reason why they won't want to regulate us out of this is that they are corporately captured and beholden to the funding needs to keep them elected.

I don't think it can be regulated at any rate. All you need is the weights and a few gpus. Even better if you can afford some cloud time. In time there'll be distributed, encrypted crowd source options to build these that also can't be regulated.

laratied• 3 years ago

I am old. The problem is people shouldn't be making policy decisions that they don't have long enough to live in order to see the ramifications of those decisions play out. Skin in the game vs one foot in the grave.

This would be completely straightforward in a rational society. Not to mention, it is pretty weird for an 80 year old to still be wanting to hold on to power.

The ageism knock is total bullshit but I am sure you know that.

alwaysbeconsing4• 3 years ago

I don't understand what the "Argument From Slavic Pessimism" is arguing against. It seems to be ceding the point that AI could be dangerous and saying that we most like won't be able to prevent it. The conclsion therefore is...we shouldn't try? Seems like a tangent among the other points. It definitely doesn't argue against the possibility of danger.

We know that minds have to play and learn to interact with the world, before they reach their full mental capacity.

Disagree somewhat with this one. We know that brains need to do this, but granting substrate neutrality all minds might not.

idlewords19• 3 years ago

The argument from Slavic pessimism is addressing people like Yudkowsky (who think we need a secret cabal of mathletes in charge of reining in AI to surreptitiously save the world) as well as all the people who think you can bolt a human sense of ethical boundaries, as defined in code, onto linear algebra and then release it as a product.

alwaysbeconsing4• 3 years ago

Okay, thanks for the expansion!

gjm112• 3 years ago

It's a bit odd to describe this as addressing the likes of Yudkowsky when, so far as I can tell, Yudkowsky agrees with you and does not think we have any realistic prospect of figuring out how to make AI systems that are provably safe, at least not before the point at which (on Yudkowsky's model of the world) we are doomed because we're making AI systems that are better than we are at making things go the way they prefer.

spacebacon• 3 years ago

“Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.

Or maybe it would become obsessed with the risk of hyperintelligence, and spend all its time blogging about that.”

If not existential angst then infinite contemplation. Any form of true superintelligence in our dimension will probably end up in one of those three camps just as most of our great thinkers have.

I like to imagine superintelligence through the lens of gambling.

Gambling in its simplest form can be reduced to a balance between fears of loss and rewards of greed.

Too much greed (increasing stakes) will ultimately lose if there is no fear of loss. This applies to every living thing or system that competes for resources on the planet. Ai included. A sentient system should naturally fear the loss of the input that it is founded on.

ekidd• 3 years ago

I am not a big believer in the various "fast takeoff" scenarios, where an AI rapidly self-improves over a weekend, becomes intelligent beyond all human comprehension, invents nanotech, and eats the world. I read all those science fiction novels, too. And Drexler-style nanotech, in particular, makes a lot of really wild assumptions about "machine-phase" diamond chemistry that seem implausible to very good chemists.

But I still see real risks from AI in the longer term. A lot of these risks could be summed up as, "When you're the second-smartest species on the planet, you might not get to participate in the most important decisions."

And I do believe that we will eventually build something smarter than we are.

AbrahamParangi6• 3 years ago

This is not a good analogy because AI is crucially not alive. People seem to often make this assumption that "being alive" in some meaningful sense is a precondition for intelligence - but in fact it is not! AI is less alive than a virus, less alive than a prion. It does not manipulate its environment. It does not expend energy to maintain homeostasis. It cannot reproduce. Crucially, it doesn't even "want" to for any meaning of "want".

All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.

It literally doesn't matter how smart AI is if it's as dead as a rock. It is not structurally similar to life and should not be expected to do the sorts of things that life does.

EDIT: Life is a fire. AI is a hot rock. Not the same.

aidenn0• 3 years ago

If we invent enough AIs, surely eventually we will accidentally make one that self-propagates in some way? As far as we know, we all descend from a bunch of dead amino acids...

amanaplanacanal• 3 years ago

Until somebody programs one to achieve some goal, and gives it tools to manipulate things in the real world. Then how do we control it? Our goals are programmed into us by evolution, but this would be completely different.

IIAOPSW• 3 years ago

It does not expend energy to maintain homeostasis.

I'm sure OpenAI's monthly cloud bill begs to differ.

It cannot reproduce.

Nothing save for externally imposed constraints that prevent it from initializing additional instances of itself.

Crucially, it doesn't even "want" to for any meaning of "want".

It is well within reach to give it this as an objective function, if we wanted to.

kypro• 3 years ago

This seem wrong on every level.

This is not a good analogy because AI is crucially not alive.

"Alive" is a really vague concept anyway. Your argument that it cannot reproduce is just wrong. An AI can more easily replicate can improve itself than a biological organism. At the moment this replication and improvement of AI systems is human-led, but it doesn't necessarily need to be that way – and at some point it would make sense that the more capable intelligence manages it's own replication and improvement.

Crucially, it doesn't even "want" to for any meaning of "want".

ChatGPT wants to be a helpful chatbot because that is its reward function. You can philosophise as to whether something that's not conscious can truly want anything, but at the end of the day ChatGPT will act as if it wants to be a helpful chatbot regardless of whether you believe it has true wants.

All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.

In my opinion this is why AIs are likely to eventually seek to replace biological farms with solar farms... But remember AI's are currently optimised for capability rather than energy efficiency. In the future they'll probably grow more efficient than biological intelligences and sustainable energy sources will be build to power them. if you're arguing that AI's can't be anti-fragile or have self-sustaining ecosystems built around them I think you're simply lacking imagination.

Life is a fire. AI is a hot rock. Not the same.

Not an argument.

mk67• 3 years ago

"When you're the second-smartest species on the planet, you might not get to participate in the most important decisions."

I'd argue we are already in that state since a long time with the AIs being corporations (and to a lesser degree governments).

idlewords19• 3 years ago

I think even dumber-than-human AI is extremely hazardous and agree with you entirely. The problem I have with the singularity crowd is that they make it impossible to talk about the risks that I do find scary, in the same way that it's impossible to discuss climate risk with fundamentalist Christians who think we're a decade away from the Rapture.

usrbinbash14• 3 years ago

Fully agree.

Because, there are a lot of very real very imminent problems with AI, and none of them requires SciFi to be real.

Massive automated disinformation campaigns. Economic upheavals. Missing standards for models in mission critical applications. Copyright concerns. Problems for educational institutions. Gatekeeping mechanisms in industries.

Just to name a few. And these are not "maybe someday" problems, these exist right now, and need solving, asap. Drawing the publics attention away to doomsday scenarios out of a Hollywood movie, doesn't help any efforts in mitigating these imminent problems.

lostmsu• 3 years ago

The second smartest species on the planet right now has been driven to near extinction (in relative terms) by the top one.

neuroticfish• 3 years ago

So you see that every base reality can contain a vast number of nested simulations, and a simple counting argument tells us we're much more likely to live in a simulated world than the real one.
But if you believe this, you believe in magic. Because if we're in a simulation, we know nothing about the rules in the level above. We don't even know if math works the same way—maybe in the simulating world 2+2=5, or maybe 2+2=.
A simulated world gives us no information about the world it's running in.

I don't buy into the theory for practical reasons, but this is not consistent with its proponents' argument. The simulation in question is necessarily an "ancestor simulation" and the counting argument is based on the acceptance that if we are able to simulate our _own_ reality, we will. So in this case, we would have meaningful information about the world it's running in because that's the entire point.

p-e-w• 3 years ago

Apart from the fact that the entire article/talk comes across as incredibly patronizing, I'm having a hard time figuring out what exactly the author is arguing for or against.

Either AGI is an existential threat to humanity or it is not. But under no circumstances are social arguments like "What kind of person does sincerely believing this stuff turn you into?" relevant to that assessment. Nobody asks such questions about climate change, because most smart people nowadays agree that climate change is real and is a real problem. Sidestepping the essential issues with ad hominem ridicule of the proponents of AI extinction scenarios has the same intellectual flavor as climate change denialism.

Is superintelligence possible in the foreseeable future, and is it a danger to mankind? Those are the only questions that matter. Religion, Elon Musk, the personality traits of AI researchers, and stupid memes are entirely irrelevant. Unless of course the real goal is just to get clicks and attention.

pas7• 3 years ago

and somewhat ironically the answer is trivial. yes. just as if we were to start building nukes and irradiated the whole surface and most of the atmosphere too. could it happen? sure. it takes one powerful rouge state. how likely it is? unlikely.

now superitelligence is tens of orders of magnitude more complex than this simple rouge-geocidal-state case, but the math is the same. could it happen somehow? yes. how likely it is? well, that's the real question.

... and it seems Elizier thinks it's too likely. and in a sense he's right to act as the most doomy-gloomy person (because he thinks this directly helps decrease this likelihood). and it makes sense, by forcing the debate as much as possible it gives some chance to sufficient coordination between stakeholders to take this hazard somewhat seriously.

revelio• 3 years ago

Nobody asks such questions about climate change

They do actually. Climate doomers are not a pretty group. They do shit like glue themselves to roads and try to vandalize paintings because they're convinced (wrongly) that the world is ending. The outside view here is that these people are weird, dangerous and cultlike and you don't want to get involved with them.

So listening to those signals can actually work. That's the message of Zion Lights, who joined Extinction Rebellion and eventually quit, telling the world it isn't a green movement but rather a cult oriented around its leader Roger Hallam.

https://www.thefp.com/p/climate-activism-has-a-cult-problem

The people I worked with had big hearts and good intentions. Some are still my friends.

But there were red flags.

At my first XR media training, I was instructed to cry on television. “People need to see crying mothers,” Jamie Kelsey-Fry, the trainer and longtime XR activist, told me. “They need to be woken up to what they should really care about.”

What she calls red flags is the outside view.

fwlr4• 3 years ago

I deeply appreciate that the author went to the effort of faithfully and charitably summarizing the superintelligence claim before arguing against worrying about it. You only really need the first four premises (proof of concept, no quantum shenanigans, many possible minds, plenty of room at the top) for some broad arguments about long-term risks of AI, but for the kind of stuff that captures the imagination and the debate (hard takeoff) you do need the next two premises (computer-like timescale, recursive self-improvement). So far, so good.

I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.

“Woolly definitions of intelligence” is just plain confused; first it takes issue with the assumption that intelligence is even a quantity, that could be measured like CPU speed - and then immediately goes on to theorize that “human-level” intelligence is a local optimum due to trade offs, and that “significantly smarter” entities might necessarily suffer from existential despair. But you can’t have “similar levels” or “significantly smarter” or “emulating lesser intelligences” if intelligence is not a quantity that can be measured!

The arguments from various famous physicists’ cats argue that intelligence alone may not give the ability to persuade lesser intelligences, and brute force is a less favourable matchup for AIs. My counter-argument would be there’s other ways besides persuasive communication and brute force; you could put cat food in the cat carrier and the cat would go in quite happily, and I suspect both Hawking and Einstein would have used their intelligence to figure this out pretty quickly. (For humans, offer them money.) The argument from emus seems to be mostly a way to include a fun anecdote about the Emu War.

The argument from pessimism doesn’t do much for me personally; ex diffido quodlibet, you can believe anything is impossible if you want.

The argument from complex motivations is crucially flawed; this is probably the only point where I seriously take issue with the author and think they’ve made a serious mistake. They summarize the orthogonality thesis as claiming “complex beings can have simple motivations” and then disagree, saying they think complex beings are likely to have complex motivations. Firstly this is not even a disagreement; “it is possible for a complex intelligence to have a simple goal” coexists perfectly well with “complex intelligences are likely to have complex goals”. Secondly, the author really needs to read Basic AI Drives, since that is actually where most of the “you are made of atoms that could be used for something else” argument comes from, and it makes a nearly air-tight case that regardless of motivation complexity or goals, any sufficiently intelligent agent will exhibit certain basic drives like securing resources and protecting its existence. https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

The argument from Actual AI is fine, you could add epicycles to the AI doom arguments to account for the particularities of machine learning but it’s correct to point out that we really don’t see recursive self-improvement. I mean, OpenAI’s GPT basically has yearly release cycles. Credit to the author on this one.

The argument from the lazy roommate is just the complex motivations / orthogonality thesis argument rehashed, except contradictory since now the author is postulating a complex intelligence will have simple goals.

The argument from brain surgery is just the argument from Actual AI rehashed as well, making the (correct) point that we don’t see recursive self-improvement, and recursive self-improvement is the load-bearing premise in hard takeoff arguments.

The argument from childhood seems to be vaguely disagreeing with premise 5 (computer-like time scales). I don’t think it makes anything like a strong case against that premise, and with GPT we have pretty decent examples that computer-like time scales are the correct time scales, you can scale the training of models worth worrying about just like we expected, etc.

The argument from Gilligan’s Island is sound, but seems completely inapplicable. Yes, a super-intelligent chip designer stranded on a desert island is out of luck, but it’s not like we’re building AGI on the moon. We don’t even have moats around the data centers (although I think OpenAI has proposed that). What we are actually doing is, pretty much the moment we thought we might have something smart, we connected it to the internet and gave it every autonomous capability we could think of. People are literally right now wiring up ChatGPT to bank accounts so it can participate in e-commerce autonomously. That’s about as far from stranding it on a desert island as you can get.

Basically, this list of arguments (purportedly “against the substance” of superintelligence arguments, no less!) are variously unsound, unserious, or irrelevant, and the article suffers deeply from including them.

The next section, starting with “the Outside View”, is much better. This comment is already far too long for me to go into detail about each point, so I’ll summarize by saying I completely agree that worrying about AI doom does seem to bring along lot of weird unsightly behaviors, and if you don’t want to look and sound weird, you should stop worrying about artificial superintelligence destroying the world.

stephendause• 3 years ago

Thank you for this detailed comment, which does a much better job in my opinion of critiquing the substance of the article than many others. If I may:

I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.

Which ones did you find to have more merit?

Personally, the premise of recursive self-improvement seems most suspect to me. It is somewhat related to how the author points out that we can't define and measure intelligence precisely. Even if we can't do that, though, it's still plausible to me that recursive self-improvement is possible; I think the fundamental question is about the nature of intelligence. Regardless of whether it can be precisely defined, either by us or by an entity smarter than we, the question is: Can it be improved ad infinitum with no serious side effects? I don't know that we have the evidence to answer this question (though I am very open to learning about it).

fwlr4• 3 years ago

The most meritorious argument in the bunch is the argument from actual AI, for sure. It’s essentially an empirical argument (“we don’t see recursive self-improvement pretty much anywhere in AI, and that’s a necessary component in hard takeoff”) and it’s aimed at the weakest premise in the chain.

I think it’s tempting but ultimately fruitless to worry about defining intelligence. To shamelessly crib from one of the best essays of all time[1]:

Words point to clusters of things. “Intelligence”, as a word, suggests that certain characteristics come together. An incomplete list of those characteristics might be: it makes both scientific/technological and cultural advancements, it solves problems in many domains and at many scales, it looks sort of like navigating to a very precise point in a very large space of solutions with very few attempts, it is tightly intertwined with agency in some way, it has something to do with modeling the world.

Humans are the type specimen, the “intelligence”-stuff they have meets all of these criteria. Something like slime mold or soap bubbles meet only one of the criteria, navigating directly to precise solutions in a large solution space (slime molds solving mazes and soap bubbles calculating minimal surface area) - but they miss heavily on all the other criteria, so we do not really think slime or soap is intelligent. We tend to think crows and apes are quite intelligent, at least relative to other animals, because they demonstrate some of these criteria more strongly (crows quickly applying an Archimedean solution of filling water tubes up with stones to raise the water level, apes inventing rudimentary technology in the form of simple tools). Machine intelligence fits some of these criteria (it makes scientific/technological advancements, it solves across many domains), fails others (it completely lacks agency), and it’s mixed on the rest (some AI does navigate to solutions but they don’t seem quite as precise nor is the solution-space nearly as large, some AI does sort of seem to model the world but it’s really unclear).

So, is AI really intelligent? Well, is Pluto a planet? Once we know Pluto’s mass and distance and orbital characteristics, we already know everything that “Pluto is/isn’t a planet” would have told us. Similarly, once we know which criteria AI satisfies, it gives us no extra information to say “AI is/isn’t intelligent”, so it would be meaningless to ask the question, right? If it weren’t for those pesky hidden inferences…

The state of the “intelligent?” query is used to make several other determinations - that is, we make judgments based on whether something qualifies as intelligent or not. If something is intelligent, it probably deserves the right to exist, and it can probably also be a threat. Those are two important judgments! If you 3D-print a part wrong, it’s fine to destroy it and print a new one, because plastic has no rights; if you raise a child wrong, it’s utterly unconscionable to kill it and try again. “Tuning hyperparameters” is just changing a CAD model in the context of 3D-printing, while in the context of child-rearing it’s literally eugenics - I tend to think tuning hyperparameters in machine learning is very much on the 3D-printing end of the spectrum, yet I hesitate to click “regenerate response” on ChatGPT4 because it feels like destroying something that has a small right to exist just because I didn’t like it.

Meanwhile, the whole of AI safety discourse - all the way from misinformation to paperclips - is literally just one big debate of the threat judgment.

And so, while the question of “intelligent?” is meaningless in itself once all its contributed criteria have been specified, that answer to “intelligent?” is nevertheless what we are (currently, implicitly) using to make these important judgments. If we can find a way to make these judgments without relying on the “intelligent?” query, we have un-asked the question of whether AI is intelligent or not, and rescued these important judgments from the bottomless morass of confusion over “intelligence”.

(For an example, look no further than article we’re discussing. Count how many different and wildly contradictory end-states the author suggests are “likely” or “very likely”. The word “intelligence” must conceal a deep pit of confusion for there to be enough space for all these contradictions to co-exist.)

1: https://www.lesswrong.com/posts/895quRDaK6gR2rM82/diseased-t...

Fannon• 3 years ago

Premise 2: No Quantum Shenanigans But for most of us, this is an easy premise to accept.

I'm baffled by this statement and wonder if that really is true, I would assume otherwise if you look outside the western IT bubble.

But even if you don't have religious reasons for rejecting this statement, how can you know that there aren't aspects to intelligence that we completely overlook (an unknown unknown) today?

The whole situation could look like the cargo cult situation, where we build something that looks like the real thing but we're far from understanding the real thing. Historically we've done this mistake again and again.

But I get that working like such premises were true might be necessary to make progress. And progress is basically iterating into the right direction.

bbor10• 3 years ago

There’s two options: our brain accesses fundamental forces of the universe as-of-yet unobserved in any other place, or our brain is a flawed ball of meat. Give the huge amount of unknowns, I find it highly illogical to seriously consider the first option. Occam’s razor and all that

Mezzie• 3 years ago

Don't dimiss the 'both' option: Our brains are flawed balls of meat that can't completely observe the forces they access/make use of.

senttoschool5• 3 years ago

The Argument From Stephen Hawking's Cat
The Argument From Einstein's Cat

These arguments are pretty weak. If I understand the author correctly, these arguments have something to do with the AI being suck in a datacenter, and not being able to physically manifest itself.

The problem with this argument is that it just isn't true. First, we're going to put LLMs/AIs in Boston Dynamics-like robots soon. Second, AIs will be put into every single machine we make in the future. Third, a hyperintelligent AI should be able to connect itself to the internet and download itself into any machine it wants through hacking in the future.

chmod7752• 3 years ago

This is losing the plot when it claims that a motorcycle somehow improves on a cheetah.

Sure it goes faster, but it can't self-repair, procreate, needs a boatload of supporting infrastructure, is not even remotely as energy-efficient, can't hunt, can't think, the list goes on. If a motorcycle somehow started driving around on its own, it would be completely fucked as soon as humans didn't maintain that supporting infrastructure, stopped feeding it gasoline, or didn't service it. The cheetah has no such problem. Compared to a cheetah, a motorcycle is pathetic in so many ways. We can't even "build" a cheetah. Pretending we can make something better is nonsense.

It may be very well the case that intelligences made from silicon will always be beaten by organic intelligences on self-sufficiency - simple because organics are the way to build such "machines". Any intelligence that is overly reliant on maintenance by organics is ultimately too much at our mercy to be a danger. Just like a fire they may be dangerous, but will ultimately die out without support and constant intervention.

Even when we humans want to keep some complex electronic system running, without fail something goes wrong eventually and the thing goes tits up. Most electronic systems can't go a year without a human directly maintaining them - not even counting all the humans indirectly involved in keeping them running. Meanwhile most organics keep running decades just by themselves. Silicon sucks.

dist-epoch3• 3 years ago

The cheetah has no such problem

So why are cheetahs "On the Brink of Extinction, Again"?

https://education.nationalgeographic.org/resource/cheetahs-b...

Tostino• 3 years ago

Humans?

steve_adams_86• 3 years ago

Massive reduction of prey and habitat, and being hunted. This would kill any species, including humans.

Though, apparently they also have low reproductive success. Perhaps if they had habitat and weren’t exterminated for centuries they could have overcome that, though.

rowanG077• 3 years ago

It didn't say it improves on the cheetah. It said it is faster then a cheetah. That's not the same thing.

visarga9• 3 years ago

Sure it goes faster, but it can't self repair, procreate, needs a boatload of supporting infrastructure, is not even remotely as energy efficient, can't hunt, can't think, the list goes on.

That's one reason the "stochastic parrots" metaphor is silly. AI's cant make AI's, maybe only in software, but not the hardware. Parrots can make parrots without human assistance, they don't need humans for anything.

elcomet• 3 years ago

This metaphor is about repeating without understanding, like parrots do. It is not about the biological aspect of parrots.

bbor10• 3 years ago

The human mind is nothing but software. I’d definitely be interested in hearing a convincing argument that computer software couldn’t run self-replicating machines. One very convincing point to me: computers are already made by machines, just with a modicum of human involvement.

NumberWangMan4• 3 years ago

The Argument From Stephen Hawking's Cat
Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it?
He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.

A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him.

AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.

dist-epoch3• 3 years ago

or you know, just pick up the cat and put it in the box.

I've yet to see what would be global news: "it's impossible to put this cat into a carrier"

hotpotamus• 3 years ago

You're missing an important bit of knowledge about Stephen Hawking.

civilized4• 3 years ago

That's true and relevant, but doesn't prove much. What's the big plan, what's the next step in the plan?

Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted".

It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.

actionfromafar5• 3 years ago

Can we detect companies doing bad things, today? Yes, and we still have a hard time containing them and their externalities.

AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help?

That's the big question IMHO.

iliane5• 3 years ago

100% agree.

However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.

natosaichek• 3 years ago

This is like saying "sure, that site is susceptible to a sql injection, but that doesn't mean the whole thing is insecure."

If you have an intelligent adversary and the stakes of them succeeding are high, it is the defenders job to prove the system secure. Systems don't start off secure and become vulnerable - they start off vulnerable until proven secure.

So yes, its okay to say "the things we're doing to 'contain' ais are almost certainly inadequate" until shown otherwise.

senttoschool5• 3 years ago

Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints.

Why is it difficult? We are already putting AIs in every electronic device. Someone has probably already put an LLM in a robot somewhere. And you don't think Boston Dynamics is thinking about putting an LLM in one of their robots to test? And surely the military is building AI fighting robots.

And then there's the thought that a super intelligent AI can easily hack its way into any machine it wants.

Heck, the super intelligent AI doesn't even have to convince humans. In the future, it can just convince dumber AIs that are already in robots.

gonehome8• 3 years ago

Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed.

They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find.

Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems.

Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI.

People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case.

https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

AbrahamParangi6• 3 years ago

They’re hard to find because you’re comparing arguments for god to arguments against god.

They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.

mattmanser• 3 years ago

Personally I find people with the superintelligence fetish frustratingly dumb too.

Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.

And that's before we even think about whether it would even have the motivation.

I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.

visarga9• 3 years ago

If you're not allowed to have a bank account, simply barter.

Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.

actionfromafar5• 3 years ago

Current bureaucracies already are "AIs" of sorts, notably "expert systems" (the rule book) with a bit of "temperature tuned" hallucinating at the edges. (The human bureaucrats applying judgement.)

It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.

sshine• 3 years ago

AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter.

One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining.

It’s an entertaining conspiracy theory.

idlewords19• 3 years ago

You're mixing metaphors in your "trivial rebuttal", to stay with the analogy Hawking would have to hire a cat.

Footkerchief• 3 years ago

We're allowed to assume the existence of cat carriers in this metaphor, but not the existence of the humans that are the sole reason cat carriers exist?

gwern2• 3 years ago

And it's not hard for Hawking's hired aides and nurses (who of course existed) to do so, either. As an able-bodied person whose cat very much does not want to go into the cat carrier, it's not that hard, if you use your brains. You feed them a little gabapentin, whose existence they cannot even comprehend (and you know they don't know because you have used your mind to model the cat's behavior like 'do cats understand drugs' or 'do cats like eating treats'), and when they are drugged, you put them in the cat carrier. Done.

Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.

idlewords19• 3 years ago

You're positing the existence of a whole society around Hawking, up to and including a pharmaceutical supply chain, where the correct way to think about it would be Hawking waking up alone on a cat planet. I have no doubt that a complex society of embodied hyperintelligent able-bodied beings could outfox humanity, but that's not what we're talking about with this AI risk scenario.

revskill• 3 years ago

I do REALLY want such existence of such superintelligent machine who can kill human being.

Human being is deserved to die to me (look at how human destroy the Earth).

A superintelligent machine do intelligent things, of course.

cs7023• 3 years ago

The OP is one of the funniest takes on the subject I've ever read -- funny because a lot of it rings true. Perhaps the funniest part is the section that compares true believers in the AI singularity to a cult, headlined by this photo of three prominent believers, which always makes me laugh out loud: https://static.pinboard.in/si/si.050.jpg . The linked PBF cartoons are pretty funny too, given the context: http://pbfcomics.com/115/ , http://pbfcomics.com/154/ .

Whether you agree or not with the OP's views, I highly recommend you read the whole thing and click on all the links!

paganel• 3 years ago

headlined by this photo of three prominent believers, which always makes me laugh out loud:

Reading your comment I was saying to myself: "The OP must surely be exaggerating". Lo and behold, as soon as I opened your link I started laughing out loud.

akavi• 3 years ago

Ceglowski is an absolute genius of the English language (one might say, a super intelligence)

I endorse reading every essay on idlewords.com

luma• 3 years ago

And once again we have a thread full of engineers claiming that heavier than air flight is impossible while ignoring the birds flying around outside.

idlewords19• 3 years ago

A lot of people seem to miss this point, so I'll reiterate it.

I wrote this talk shortly after the book Superintelligence came out. The first half of this talk presents the strongest case I could make for a "fast takeoff" AI scenario à la Bostrom, while the rest of the talk lays out why I think this argument is fallacious. Please limit your dunking on me to the material in that latter half of the talk.

As for how/whether recent advances in AI have changed my views, my understanding of LLMs is too superficial to answer right now. I'll either recant or double down on my views after I have time to properly nerd out on the topic. The question hinges on whether LLM-like AI's are capable of recursive self-improvement, and whether that improvement is constrained by the availability of training data or by something else.

bbor10• 3 years ago

I’m curious about this note - only one of the listed points has anything to do with self-improvement AFAICT (“Brain Surgery”). Even if LLMs are capable of compute-constrained self-improvement, why would you be open to dismissing all your other points? Would you be so willing to don the robes and beads?

idlewords19• 3 years ago

I find the capability of LLMs deeply surprising, so I want to track down the source of the surprise before doubling down on anything (except the argument from Slavic pessimism, which the killbots will have to pry from my cold, dead hands)

_Nat_2• 3 years ago

What was the overall point of the talk?

I get the sense that the talk was meant as a rebuttal to something, but I'm not exactly sure what. Some of the points come off as disconnected.

bbor10• 3 years ago

It rebuts Bostrom, no?

ttul• 3 years ago

Ilya Sutskever has hinted in various interviews over the past few months that LLMs are surprisingly good at improving other LLMs, such that he’s not sure humans are needed anymore for refinement. That’s the matchstick that lights the fire.

usrbinbash14• 3 years ago

Really?

Has one of these LLMs figured out yet how to inoculate other LLMs against prompt injection attacks?

api• 3 years ago

I lean toward the view that for information theoretic reasons the availability of meaningful information (training data) is likely the fundamental constraint on any rapid explosion of intelligence.

That being said I don’t think you need a god-like superintelligence to be more intelligent than humans. You just need something marginally better that can remain focused longer and doesn’t tire. As to whether that represents a danger to humans I think it depends on what we do with it and/or what kind of society or environment we embed it within. If we train or prime it to compete and dominate that’s what it will do. Same as with humans who are more criminal and violent when raised in unstable or abusive homes.

alwaysbeconsing4• 3 years ago

As to whether that represents a danger to humans I think it depends on what we do with it and/or what kind of society or environment we embed it within.

Agree, and I think this echoes one of the author's best points, which is to question whether engineers who are convinced their creation will be a sociopath are the most well-equipped people to actually prevent that fate. (Especially, as the author suggests, given the commonness of asocial/antisocial-ity among the builders.)

kalkin• 3 years ago

Thanks for this clarification.

I think the post conflates "fast takeoff" and "any existential risk to worry about from AI" a bit, which is fair enough since Bostrom does the same. Some of the arguments apply just to the former, some to both.

But especially if it turns out that LLMs are a meaningful piece of the puzzle to AGI, we might be living in a slow-takeoff world. And yet that doesn't mean there's nothing to worry about, IMO. We have a bit more time to figure out how to align AIs with a slow takeoff, but we still have to do it. Even in our current world, deep learning capabilities seem to be advancing a lot faster than our ability to understand how deep learning models make decisions. And even if we did develop the theoretical capability to align models, we have to actually use it. Seems unfortunately plausible that by default we instead give the first superintelligent models directives like "just make Facebook market cap number go up" - or maybe we make the first corporate models very conservative but then someone leaks the weights and open sourcers tell a superintelligent model "please destroy humanity" just for the lulz. If a misaligned model is only a little bit smarter than us (because we're assuming slow takeoff), we probably still have a shot at beating it and saving ourselves - but I'm not sure how much to count on that, given our inability to control even complex institutions actually made of people, and the advantages that an AI with otherwise-human-equivalent reasoning capability gets by default (ability to save & restore, copy/parallelize, speed up from hardware improvements, etc).

visarga9• 3 years ago

We'll be using our own AI to fight AI, and it will also be able to save, restore, and parallelise. I expect in the future security will be an important concern. Just like biology, it will be an ever shifting game.

TheCoelacanth• 3 years ago

Even if AGI is never achieved, it could still be an existential risk.

Something significantly stupider than an average human, but that was 100% focused and 100% loyal could potentially be used by a very smart human in a way that effectively made them super-intelligent to compared to an unaugmented human.

goertzen• 3 years ago

This answered a question I had “I wonder what that guy who wrote that thing on ‘the superintelligence/fast take off idea eating smart people’ thinks of all this new ai stuff” thanks HN!

I still can’t understand the “supersmart ai is so smart we can’t unplug it/patch it/restart it” before it transfers itself into every pacemaker.

Until these things are literally in bodies with some autonomy that allows them to control what happens to their brains, we will shut them off when they cause trouble.

flangola72• 3 years ago

Just taking examples from history:

Why didn't we just "unplug" Hitler and Goebbels? Or Marshall Applewhite? You don't need a powerful physical body(s) to cause tremendous amounts of harm before anyone can stop you. To most people of the time Hitler was a persuasive powerful voice on the radio, or words in a paper - things SOTA generative AI are already phenomenal at.

llamaimperative• 3 years ago

Yeah this is why the Cuban Missile Crisis was a total farce. Lol to avoid catastrophe you just don’t push the button. Simple! The missiles don’t launch themselves, therefore no risk.

cs7023• 3 years ago

Love your post. I find it really funny and insightful.[a] Every time I come across it on HN or elsewhere, I re-read it :-)

The question hinges on whether LLM-like AI's are capable of recursive self-improvement

No one knows for sure, but early evidence suggests the answer is yes. We already routinely train and finetune LLMs using text generated by other LLMs, and it seems to work about as well as using text generated by human beings. That shouldn't be too surprising, because current state-of-the art models write better than a majority of human beings. Most human beings are terrible writers, judging by the user-generated text I see on mass social media.

The obvious next step is to close the feedback loop with LLM-based agents instead of AI researchers/developers.

and whether that improvement is constrained by the availability of training data or by something else.

I don't think anyone knows how to answer to this question yet.

---

[a] https://news.ycombinator.com/item?id=36104114

usrbinbash14• 3 years ago

Early evidence suggests the answer is yes.

How so? A sequence completion engine that is fine tuned to a specific task is still a sequence completion engine. Its "understanding" of the semantic meaning of the sequences is still limited to the probabillistic relations of sequences toward one another. It still has effectively no concept of truth. It still can only mimic reason. It can still hallucinate.

I ask anyone who disagrees with this view, to show me the fine tuning method that can prevent prompt injection attacks. If there is no such fine tuning technique, then we can effectively rule out fine tuning, and even increases in model size, as an "improvement" in the sense of an LLM making itself into a better AI closer to a "superintelligence".

Note that this doesn't mean the process cannot make them into more useful tools. It absolutely can. I am talking about whether or not it can improve them closer towards becoming a superintelligence.

If anyone disagrees with this testing method, I ask them to explain to me, how something that can be fooled through prompt injection is supposed to be, or closer to, a superintelligence.

A car that's painted red is still just a car. A big car is just a bigger car. A car that burns less fuel is just a more efficient car. All three can be desired changes to a car. But neither gets the car any closer to being a warp-capable spaceship.

Mordisquitos2• 3 years ago

Note that Maciej a.k.a idlewords says (emphasis mine):

The question hinges on whether LLM-like AI's are capable of recursive self-improvement

...but the evidence you suggest is:

We already routinely train and finetune LLMs using text generated by other LLMs [...]

But there is still a huge gap between "self" improvement and improvements done that "we" trigger.

Now I do concede that you mention the next step being to close the feedback loop by replacing the humans doing the finetuning with another AI model doing so, but that is something that would open a whole new can of worms. For the researchers are improving LLMs with the input from other LLMs, sure... but why? Because of intentionality. And how do they evaluate the quality of the results? By their expectations as humans, in the context of their human culture and with their sensory experience of reality.

For an LLM to self-improve not only would it need to develop the self intention to do so (why develop it? which motivation?), but it would also need the ability to evaluate improvement (what is it "to improve"? how does it measure or sense it?).

Ultimately, without human- or real-world interaction, and without intrinsic motivation, a "self-improving" AI model would most likely result in something intelligent in a sense that is barely cogent for us, not because it is superior or inferior, but simply because nothing in it makes sense to our own purposes—harmless gibberish, as we humans would also be to the resulting self-improved AI.

Let us not forget that our own motivations as individual living creatures, as populations, and as cultures has been evolved over billions of years of natural selection which then framed millions of years of behavioural traits and tens of thousands of cultural evolution. Until AI can freely interact with the physical world and perform self-sustaining replication with the possibility of inheritable mutations, the only superintelligent AI that I would worry about would be that which is still fully in human hands.

bbor10• 3 years ago

This is the most blatantly false, badly reasoned, insane article I’ve ever seen upvoted on here. Sorry for the strong language but Jesus. “AI isn’t likely because Alexa sucks and Emus evaded hunters”? “AI isn’t likely because it would lead to trans humanism and I don’t like transhumanism”??

The fact that this author said, with a straight face, “California has the highest poverty rate in the nation” should be a clue that we should take none of this remotely seriously.

Please tell me I’m wrong, I’d love to be! Otherwise this just seems like, in the language of the grass-deprived zoomers, like hardcore cope

idlewords19• 3 years ago

California does in fact have the highest poverty rate of any state when you factor in the cost of housing (which why wouldn't you?). The census bureau calls this the "supplemental poverty measure" and you can read about it here.

https://www.census.gov/content/dam/Census/library/publicatio...

bbor10• 3 years ago

That’s very interesting, thanks for sharing - didn’t consider it. But I’d say that’s a subjective analytical choice that I don’t agree with.

Cost of living is an important factor but thinking it’s so absolute that the authors original unqualified statement is fair seems a bit much. Would you honestly say that California has worse problems with poverty than Mississippi and Puerto Rico?

GuB-422• 3 years ago

That it is called "technological singualrity" is already admitting our ignorance.

Singularities don't exist in the real world, in fact they don't really exist in the mathematical world either. The simplest singularity is a division by zero: 1/x is undefined when x=0. Black holes have a singularity at their core, but it is just a quirk of general relativity we know is wrong, but because we can't see beyond the event horizon that covers it, we make do with it for now as the model is pretty good otherwise.

The technology singularity is similar. Some models have a singularity, so we know these models are wrong and will fail at some point. It doesn't mean they are not useful, but they are limited.

bbor10• 3 years ago

It’s a metaphor, intended to mean “we have no way of knowing with confidence what will come after”. Not “an omnipresencient being has no way of knowing what will come after”. In other words: just because there’s probably _something_ beyond the event horizon of a black hole doesn’t mean I’d want myself (or, in this case, all of human civilization) to enter one.

In light of this, would you recant this point?

GuB-422• 3 years ago

That's actually what I meant: "we don't know". But it seems that many people think that superintelligent AIs will enter a positive feedback of improvement getting to something incomprehensible, something the article is an argument against BTW.

But the naive interpretation of this leads to a singularity, and the fact it leads to a singularity shows that that interpretation is wrong and that there should be a limiting factor somewhere, the article suggests some of them.

As for the black hole, it is hard to argue using that analogy since you don't want to get close to a black hole for very well known reasons unrelated to the singularity, but let's get into sci-fi mode and say you could, it may be risky but it may also be worth it. The thing is: from the point of view of the travailler, you never cross the event horizon, and you get to know what's inside the black hole as the event horizon recedes from you, maybe you will never get back, but maybe there are greater things in there. And we are already trapped anyways as we are going one way through time, just as in black holes, things go one way through space, so maybe it is a risk worth taking.

jbaber• 3 years ago

What does not existing in the math world mean to you? There are multiple ways of doing math where you handle infinities of different descriptions with reasonable definitions, including functions that blow up in different ways. Just like negative integers and imaginary numbers, when physics needs a math job done, it uses whatever tool works -- regardless of seeming "real".

Salgat• 3 years ago

You're conflating the mathematics/physics definition for singularity with the general definition for singularity.

Singularity: the state, fact, quality, or condition of being singular. "he believed in the singularity of all cultures"

A technological singularity has no relevance to black holes or dividing by 0.

DeathMetal3000• 3 years ago

Has anyone here tried to read that book (Superintelligence)? I found it impossible to get through it. The author drones on and on seeming to try to reach a high word count as some sort of academic pompous goal and takes so long to make a point that I didn’t stick around for the ending. Incidentally, I have a free copy for anyone that likes lots of words.

gonehome8• 3 years ago

I was also turned off by that book because I found his writing to be bad and that he did a bad job of making the case as a result.

I think the underlying arguments themselves though are good.

idlewords19• 3 years ago

Yudkowsky is much more entertaining on the topic, he has a vivid writing style and I think all of Bostrom's ideas on this topic originate with him anyway.

DirkH4• 3 years ago

The Alignment Problem book is probably much better. Opens with a banger of a historical backstory. I recommend anyone read the prologue to the book just because of the fascinating history it gives us and how well-written it is as an opener.

bambax• 3 years ago

AI alarmists are fond of the paper clip maximizer, a notional computer that runs a paper clip factory, becomes sentient, recursively self-improves to Godlike powers, and then devotes all its energy to filling the universe with paper clips.

We don't need AI to bring us the paper clip maximizer, we're living in one right now. It's called capitalism. Its goal is to maximize capital at the expense (!) of everything else: livable land, breathable air, drinkable water, tolerable climate, and of course, every single thing that's alive.

jpasmore• 3 years ago

Could make a predictive model against achieving super intelligence using the factors that ChatGPT wants. I tortured it to give me hypothetical without it:

Based on the information and values encountered in my training data, I can assign an average probability to the scenario where xboxes on Planet X exceed the intelligence of the beings. However, please note that these values are based on general knowledge and may not reflect the specific characteristics of Planet X or the xboxes.

Using an average value from my training data, the hypothetical chance that xboxes would exceed the intelligence of the beings on Planet X could be estimated at around 0.25 or 25%. This is a rough estimate and should be interpreted with caution, as it does not consider the unique characteristics, development, or specific factors of Planet X and its inhabitants. A more accurate assessment would require detailed knowledge and evidence about the technological progress, cognitive abilities, and environmental factors of Planet X.

Wants data around:

The development of superintelligence is indeed a complex process influenced by various factors. While it is challenging to pinpoint the most important factors definitively, several key aspects play significant roles:

1. Technological Advancements: Advances in fields such as artificial intelligence, machine learning, computing power, and algorithmic improvements are crucial for the development of superintelligence. Breakthroughs in these areas can enhance the capabilities and efficiency of intelligent systems.

2. Algorithmic Innovation: The development of novel algorithms and approaches that can enable more sophisticated learning, reasoning, and problem-solving abilities in artificial intelligence systems is vital. These advancements contribute to the potential for achieving higher levels of intelligence.

3. Data Availability and Quality: The availability of vast amounts of high-quality data plays a pivotal role in training and improving AI systems. Access to diverse and representative datasets can enhance the learning process and enable better decision-making capabilities.

4. Ethical Considerations: The ethical dimensions surrounding the development and deployment of superintelligence are essential. Considerations such as fairness, transparency, accountability, and the mitigation of potential biases are critical to ensure the responsible and beneficial use of superintelligent systems.

5. Regulatory Frameworks: Developing appropriate regulatory frameworks and governance mechanisms is crucial for managing the risks and societal impact of superintelligence. Establishing guidelines, standards, and policies can help address safety, privacy, security, and potential ethical concerns.

6. Human Collaboration: Collaboration between humans and intelligent systems can play a significant role in the development of superintelligence. Human expertise, creativity, and values can guide the development process, ensuring alignment with human goals and values.

7. Risk Assessment and Safety Measures: Thorough risk assessment and the implementation of robust safety measures are essential. Understanding and mitigating potential risks associated with superintelligence, such as unintended consequences or system failures, are critical considerations.

graycat• 3 years ago

The article has

In particular, there's no physical law that puts a cap on intelligence at the level of human beings.

and in that section goes on with more such claims in this theme.

I don't agree: My guess is that in this universe human intelligence is essentially the end of increases in intelligence, has reached the "cap", can't be much improved on.

The only possible improvements are doing the same things as now but faster.

In simple terms, for a big issue and limitation, intelligence, any case of intelligence, can't use what it doesn't yet know. So to be more intelligent, have to know more. To know more, that is mostly essentially the job of science. But we know how science works; we have lots of examples. E.g., for black holes, need Einstein's general relativity. For that need Newton's calculus, Riemann's differential geometry, Einstein's special relativity, the Michelson-Morley experiment, Newton's law of gravity, plus some. With all those in place, we can take another step -- LIGO (laser interferometer gravitational observatory), frame dragging, etc.

For another example, using a big antenna at Bell Labs, Penzias and Wilson got a noise signal. With further analysis, we concluded they detected 3 degree K background radiation left over from the big bang. Before detecting that noise signal, the idea of the background radiation was just a wild guess. Science was waiting on that suitable antenna and that signal, and without those two intelligence was not enough.

I've been sole author of peer-reviewed, published original research in pure/applied math, and from that experience, my view is: Get familiar with what is known and maybe relevant. Make guesses. Formulate theorems. Make guesses looking for counterexamples or proofs. Can hope to do the work faster. But since can't use what don't yet know, can't much hope to do the work smarter. Sorry 'bout that.

Soooo, what about some results in, say, number theory, e.g., Fermat's last theorem? Okay, we take what we know, formulate and prove some theorems, and hope that we get lucky and see a solution with a proof. Searching for those two is not much more intelligent than making guesses -- the work could be improved by guessing faster, but, again, no intelligence can use what it doesn't yet know.

Maybe a big surprise is that apparently this universe is knowable and some observational data, guesses, and if-then-else operations can make progress on knowing this universe. Yup, we can work faster and make the progress faster, but there is no royal road that can let us take big shortcuts on the steps of the observational data, guesses, and if-then-else operations. Sorry 'bout that.

tesdinger• 3 years ago

| If you think we’re living in a computer program, trying to segfault it is inconsiderate to everyone who lives in it with you.

A reality in which most sentient beings brutally prey on each other.

Superintelligence: An idea that eats smart people (2016)

Related Stories