Is Intelligence Self-Limiting?

http://ieet.org/index.php/IEET/more/eubanks20120310

by lavin • 14 years ago

148 76 14 years ago

"Mobile AI robots do all the work autonomously. If they are damaged, they are smart enough to get themselves to a repair shop, where they have complete access to a parts fabricator and all of their internal designs. They can also upgrade their hardware and programming to incorporate new designs, which they can create themselves."

Okay it's too soon to talk about self-programmed robots. But I think it could be interesting to think about easier self-programmed things. They may not be too far away from now.

carsongross2• 14 years ago

Reality is self-limiting.

There are no exponentials, only sigmoid curves.

Unfortunately there is a lot of money to be made convincing people that an early stage sigmoid is actually an exponential.

robertskmiles9• 14 years ago

Obviously intelligence tops out at the high end of a sigmoid curve - there's only so much matter and energy in the universe to convert to computronium - but that's no reason to suspect any value smaller than that as the limit. It must eventually level off, but the point at which it levels off may be hundreds of billions of times more intelligent than a human.

So the fact that intelligence is limited, doesn't in any way mean that hyperintelligence isn't possible. There is a limit, but we have no reason to believe that limit is on anything like the same order of magnitude as current intelligence.

iamgopal3• 14 years ago

I like it. Intelligence more defer at Hope, Anger and Desire than at Innovativeness.

carsongross2• 14 years ago

We also have no reason to suspect that it is possible.

The wall that silicon has smashed into should give us (well, not me, really the Singularitists) pause.

cousin_it• 14 years ago

Not all possible AI designs are based on reinforcement learning or reward channels. You can code the AI with a formally specified goal instead. For example, "is there a proof of the Riemann hypothesis in ZFC shorter than a million symbols?" If such a goal requires converting the solar system into computronium, then the AI will do that. It won't settle for wireheading because wireheading doesn't give an answer to the formal problem.

To everyone who thinks intelligence might be limited in principle: there's no reason to think humans are anywhere close to the upper limit. In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.

ihnorton• 14 years ago

I think there is a necessary discontinuity between a self-modifying solver algorithm that acts within an internally-consistent formal framework; and an intelligence that is able to modify the formal system itself. On some level this might be considered analogous to self-awareness. Past that discontinuity though, my guess is that all bets are off. There are many historical examples of the depths of madness that may be plumbed in unfulfilled pursuit of a formal goal.

To your second point, absolutely agreed: http://infoproc.blogspot.com/2012/03/only-he-was-fully-awake...

randallsquared2• 14 years ago

In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.

While I don't disagree that humans are essentially at the lowest level of intelligence that make civilization possible (else why would it have taken hundreds of thousands of years to get started?), this claim has no bearing on the claim that the upper limit of intelligence is immediately above human genius level. You seem to be assuming that there is necessarily a wide gap between the lowest civilization-producing level and the highest practical level, and that's not at all clear. Some (weak) evidence that we're already near the top can be found in the higher incidence of mental health issues among very intelligent humans: perhaps this is a result of a limit on complexity rather than merely a feature of human brains.

pjscott• 14 years ago

Imagine a human genius. Now imagine that same genius with a brain that thinks the same thoughts, but a hundred times faster. Now imagine a million instances of that person.

At no point in this thought experiment have we changed the thoughts this person can think. If any of these sped-up million geniuses took an IQ test or something, the score would be the same. And yet, they would seem superhumanly intelligent by any reasonable definition of the phrase.

I doubt that evolution -- with a crazy biological substrate, no less -- somehow managed to find an upper limit on intelligence.

Dn_Ab• 14 years ago

I agree with you but only while limiting the scope to the best that can be done for a biological agent which must pass through a birth canal and evolved under strong resource limits.

Can you apply intelligence to more than one person? Can a group of 10 people be considered intelligent as a whole? What about 100? Can you distinguish between the intelligence of different equally sized groups? I say yes to all of those.

Let us say we put groups of 1, 10, 100, 10^n people in a box and gave each box intelligence tests. Which would be most intelligent? Eventually communication costs will outweigh size advantages but only at some n >>0. So a trivial example of an entity whose intelligence surpasses a single human intelligence. Now why do you expect that this level of computation and knowledge management is not replicable on a smaller more efficient device not distracted by the need to calculate proprioception?

Next consider effective intelligence. Do computers when used in the right way amplify intelligence? Is it possible that a human/computer interface is possible? Forget self awareness, is it possible that a computer could do what all of what we call intelligent work better than humans? The answer to all of these is more subjective, especially the last, but I say yes. The human brain is one architecture for intelligence. It is riddled with weaknesses that only make sense when considering the resource limited nature of its development. It was created under a specific set of constraints - a vehicle to manage the more effective transmissions of genes of an energy limited biological entity that must pass a birth canal. I believe that the human mind set on designing an intelligence can do better.

Will a superhuman mind do better at designing a mind in a proportionally similar way to how humans will surpass evolution? Or will they hit too many NP complete problems? The answer to this I cannot give but if forced to bet would be yes, yes.

bermanoid3• 14 years ago

But there's no particular reason to believe that the upper limit of intelligence is pretty much right where we are, either.

In fact, I'd argue that based on what we know about other evolved systems, our priors for "the biologically evolved system that does X has plenty of room for improvement" should be much, much, much higher than "the biologically evolved system for X is almost optimal for achieving X".

Like, a thousand to one or more - there are very few tasks that evolution has found optimal solutions to (which is not surprising, its "goal" is gene survival, everything else is an accident), and most of those are extremely low level physical things, chemical processes and the like. I'd need to hear some really strong argument that suggested that there's something special about human intelligence that should make us think it's near the limit, otherwise the odds ratio is just too hard to overcome...

robertskmiles9• 14 years ago

What's the purpose of this 'pleasure' construct in the AI's mind? If it's able to calculate the value of a utility function in order to set the 'pleasure' variable, and it bases its actions on the value of this 'pleasure' variable, why not just cut out the middle man and have it base its actions directly on the result of the utility function? The variable functionally buys you nothing, but introduces this problem that adjusting the variable directly can cause the AI to take actions inconsistent with its utility function.

Without the variable, the problem doesn't happen. The AI values collecting ore. If it has enough self-awareness to reliably modify itself, it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want. The action of modifying the utility function naturally rates very low on the utility function itself.

You don't want to murder people, so not only do you choose not to murder people, but if you are presented with a pill which will make you think it's good to murder people and take great joy in it, you will choose not to take that pill. No matter how enjoyable and good murder may be for you if you take the pill, your own self-knowledge and current utility function prohibit taking it.

The model of intelligence described can be thought of as self-limiting. Luckily it is not by any means the only viable model of intelligence.

Androsynth6• 14 years ago

From a systems perspective, a pleasure center in the 'brain' is much more scalable than hardwiring something to 'do more x'. These are semi-sentient beings that he is describing, not dumb robots.

Androsynth6• 14 years ago

edit: I also want to point out that a murder-happy pill is a straw man argument. It will never happen for two reasons: 1-we have evolved to find murder repugnant, so the rewards wouldn't seem appealing to us 2-there are real life repercussions to murdering someone, so the joys would be limited in their duration.

We already have happy medicines that are legal. Valium, alcohol, WoW... I would bet that most people who are reading this have self-medicated on at least one of those before. And those are rather weak medications. They pale in comparison to what the robot was able to do by short circuiting the pleasure center in his brain.

harshreality• 14 years ago

it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want.

Why would it want or not want anything, if it doesn't have a pleasure construct (which might also be called a motivation construct, since an AI might not be capable of the same subjective experience of pleasure that we are)?

I think it's a question of program design whether there's a utility function which decides whether to trigger the pleasure construct, or whether certain sensory input modules directly trigger the pleasure construct. To limit hacking potential, routing everything through a tamper-proof utility function might be better, except that it would also limit the AI's adaptability (short of recreating its own hardware to remove the tamper-proof module... which it might never do depending on the details of its motivation construct).

robertskmiles9• 14 years ago

> Why would it want or not want anything, if it doesn't have a pleasure construct

Why does a calculator want to add, without a pleasure construct? If it doesn't enjoy giving the right answer to 2 + 3, what motivates it to choose 5? The answer is it doesn't necessarily 'want' to do it, it just does it. That was how it was programmed.

You program the AI to choose the plan that maximises the expected utility function. Someone looking at that AI in action might suppose it 'wants' to maximise the utility function. Whether they'd be right to suppose that is a question for the philosophers, but the point is, the thing doesn't just sit there apathetically just because it has no 'motivation construct'. You could say that the AI follows its programming, but it would be more accurate to say the AI is its programming.

eaten_by_a_grue2• 14 years ago

I think it would work better if when self-modifying, it ran simulations on the result with its current utility function, but there would be a problem if it did end up modifying the utility function or modified away the simulation requirement.

It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.

robertskmiles9• 14 years ago

> It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.

I think that for such an AI, the concept of a "more useful utility function" would be a nonsense. The AI's definition of 'useful' is the utility function. No other utility function can ever rate higher against the current utility function than the current utility function does.

rbanffy2• 14 years ago

> why not just cut out the middle man and have it base its actions directly on the result of the utility function?

If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.

One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.

This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.

robertskmiles9• 14 years ago

> it can also modify the utility function to return MAXINT every time

Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.

You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.

Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]

Swizec• 14 years ago

Valid argument, but we have a clear example that it is wrong.

Humans.

A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business?

Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.

ams6110• 14 years ago

why aren't we all simply taking drugs and avoiding all this messy "doing things" business

Some of us are, but that is also self-limiting. If too many of us did it we'd start dying out.

iamgopal3• 14 years ago

We are driven by multiple different value system. e.g. I am extremely curious to know if god's existence can or can not be proved scientifically. I can go to great length to know it, I do not give a damn about money. On the other hand, there are people who do not give damn about god and can go to great length for money. So Humanity can not go towards single goal. Some of them could go for shortcut and are going.

rbarooah2• 14 years ago

I'm not sure humans are a counterexample. Drugs don't generally relieve the need to work to buy food, and most serious drug addicts don't look as though they enjoying themselves.

hackinthebochs2• 14 years ago

I've been thinking about this recently: drug addicts "seem" unhappy because they don't match the usual outward appearance of happiness. But who's to say in their lifetime they haven't experienced many orders of magnitude more happiness than even the happiest non-addicts? Drug addiction breaks the usual profile of happiness, it doesn't mean that they're not actually the happiest people who ever lived.

zerostar072• 14 years ago

It might not have to do with "figuring it out", in the case of humans it's natural selection: individuals who choose to stimulate their reward centers only had much lower probability of reproducing. (like mice who stimulate their nucleus accumbens till they starve to death)

Androsynth6• 14 years ago

Drugs, in their current incarnations, are simply not that powerful a motivator. If you were to point at a heroin junky or basehead and say 'that person is in ecstasy right now, do you want their life?', the majority of people would say no, because all they see is a dirty, homeless junky.

hackinthebochs2• 14 years ago

Most of us haven't actually experienced these powerful drugs. I wonder how many people could willingly pass it up if everyone had a significant experience with them.

Karunamon• 14 years ago

Was going to say something like this. There's a significant difference between:

(researcher) points at a junkie on the street: This person is in ecstasy. How would you like to feel like that forever?

(researcher) gets you comfortable and injects you professionally and painlessly with a controlled dose of heroin, leaves and returns in a few hours: How would you like to feel like that forever?

Outward appearance counts for a lot.

goodside2• 14 years ago

The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly. Somehow, humans are able to "want to be happy" while still refusing to take heroin or pain killers, and we have no reason yet to believe that AI is fundamentally unable to grasp this distinction between "stuff I care about" and "the different stuff I would care about if I changed my own design".

iamgopal3• 14 years ago

The overlooked point here is, We do not have any control over how we build next generation. We have a little control by teaching them. The example author gave assumes each generation having lots of control over the next generation, and hence figured out what bugs them and solved it. I do not think we know what bugs us. For some people other people having different religions bugs them, so they kill them and be happy. For some people unknown unified theory bugs them, they will not be happy till it found. So It is not possible to find universal solution to different problem. But It could happen that people with similar problem join a group and find a shortcut way of being happy, without the problem being solved.

Helianthus• 14 years ago

>We do not have any control over how we build next generation.

Says who?

The author assumes uncertainty and exploits uncertainty to make a certain conclusion, when the uncertainty should just prevent him from forming a conclusion at all.

In fact it's fairly regular, I think, for singulitarians to let their imaginations run amok this way.

tel• 14 years ago

Then again isn't there that study about mice given a functional pleasure button dying of thirst?

goodside2• 14 years ago

The point is that there are humans who will refuse to have that machine hooked up, even though they understand completely that it'll make them feel so good they won't care if they use it.

stcredzero2• 14 years ago

A transhuman AI worthy of the name would be very concerned with how well it was able to model the real world -- if for no other reason than the sake of its own survival. Think of that as a 2nd order "survival imperative" of sentient beings.

bermanoid3• 14 years ago

I was actually surprised a bit to see that the author was somewhat familiar with Eliezer Yudkowsky's writings on the topic (he cited http://lesswrong.com/lw/wp/what_i_think_if_not_why/), because the line of thought doesn't seem to incorporate a real understanding of what he's said on the topic (which, to be fair, is a huge body of work...).

Most of EY's "Friendly AI" worries are rooted in this idea that when considering the entire universe of algorithms that could be described as intelligent or self improving, we need to be exceptionally careful not to assume that more than a negligible percentage of them share anything in common with human intelligence, because for the most part, they won't, unless they're carefully and explicitly designed to do so.

Here, the author assumes that the AI is simply trying to optimize some internal measure of happiness, with complete disregard for the meaning of that measure. This is an incredibly naive view of how deeply important and carefully constructed any optimization target would have to be in any self-improving intelligent machine; it's literally the core of the entire problem of friendly AI, and to trivialize it by assuming that such an AI would ever even consider rewriting its "happiness button" to be always-on is to miss the entire difficulty of the problem.

Hell, it's even the core of the problem of non-friendly AI, because it doesn't even require human level intelligence to realize that if you rewrote your own code so that you were always thrilled with the result, that's the easiest way to increase "utility". Any self-rewriting algorithm that's capable of real self improvement has to, by design, be able to consider the likelihood that changes to its objective function will end up with negative expected value.

None of which is to say this isn't a valid concern, by any stretch. But it's not a proof of universality; in fact, getting around this type of problem is exactly what any real AI designer must contend with. It's an issue that's very well known, and it's certainly not well-accepted as an insurmountable hurdle.

zerostar072• 14 years ago

Human civilization is currently limited to planet Earth with a few minor exceptions, so it makes sense to consider it one big system.

Actually it makes sense to consider all life as one big system. It was created by the planet itself, so, who knows we might even have to ascribe motivations to the planet. It's as if the planet (kind of like Lem's Solaris) has been brewing organisms for millions of years in order to do something with them . We might not be able to conceive these purposes with our antrhropomorphic thinking.

So, the human race is now coming to the point where it can modify and advance itself by tinkering with its own circuits. What we don't know is 1) what the planet plans to do with us and b) what are its movitivation and reward signals. It's not explained in the article how the AI knows what are the reward signals of its creator or why it would ever want to change them.

jimrandomh• 14 years ago

This is called "wireheading", and while it's an interesting possibility, unfortunately what we know indicates that AIs won't do it. See Stephen Omohundro, "The Basic AI Drives" (http://selfawaresystems.files.wordpress.com/2008/01/ai_drive...). The basic argument is that a sufficiently intelligent mind will make a distinction between the world being a certain way, and it perceiving that the world is a certain way, and that its goals will be defined in terms of the former; and since misperceiving the world impairs its ability to influence it, it will protect its perceptions from corruption.

watmough2• 14 years ago

Louis Wu's tasp in Ringworld also comes to mind.

bluekeybox• 14 years ago

Three immediate points:

1) Acquiring a mate is an essential external motivator which is acted upon by Darwinian laws, and there is no escape from it... If we get to the stars, it will probably be because of women. Not really touched upon by the article.

2) The author poo-poos Facebook "friends" as being on par with a virtual world, but Facebook friends are anything but "virtual". In fact, some real-world friends are actually all too often nothing more than the MOOF agents described, while some (admittedly not all) Facebook friends may offer valuable advice about where to shop, for example, or which car to buy, or even engage with you into a discussion on politics or whatnot. Very real-world and relevant.

3) The definition of "intelligence" can be easily extended to exclude self-limiting types of intelligence.

There are probably many more things that could be picked away... I'll leave it at that.

Androsynth6• 14 years ago

Iirc, there was an article a while back (or maybe a link from a comment) about how video games were the real reason behind the fermi paradox. The argument was that a civilization's entertainment would constantly be improving to the point where we could jack into a self-controlled matrix-like world where we had complete control and, in essence, are deities in our self-fabricated worlds. If we built robots to ensure our real-life conditions were safe and sanitary, it would be very difficult for the majority of mankind to resist the urge to directly modify the signals and ... MOOF!

Considering how many ways there are to modify the signals now: playing WoW, using drugs, even using alcohol to give a quick and easy boost. I'm not sure how the vast majority would be able to turn down such a machine.

dgallagher2• 14 years ago

It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them.

--------------------

Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Superiorly-intelligent entities seeded by our inventions will, but not humans.

Humans are this weird, version 0.01 of intelligence. We're part sentient, part beast. To assume we're the final perfect end product of ~4,000,000,000 years of evolution, having only been around for ~200,000 years is laughable. Pop culture and religion say otherwise because it feels good to think "we're special!", but we're only special relative to what's around us, and we're "extremely" tiny (http://www.phrenopolis.com/perspective/solarsystem/). We just happen to be the lucky first who got to v0.01, floating around on a grain of sand.

Humans are messy. Our brains are significantly limited. We die quickly. We sleep 1/3rd of our life. We do stupid things. We kill each other. We're tied to the Earth. If we leave Earth, we have to create and bring a mini-Earth along for the ride. That's an extremely large amount of overhead to carry. Efficient use of energy is likely one of the most important aspects of space travel. Anything which can do it even 1% better has a competitive advantage over us. This is why we sends robots to Mars and not people.

Imagine a form of intelligence which can travel through space, back its brain up, and never dies. If it blows up, restore from backup. Imagine a computer the size of the sun. INSANE! A human brain to a sun brain is like a grain of sand to Einstein's. It self-upgrades. It makes copies of itself and scatters throughout the universe. Trillions of eyes observing everywhere, networked together, in a giant universal wireless-mesh-network of intelligence, communicating with neutrinos (they go through planets, radio waves do not).

Major advances in hardware, software, and A.I. are key ingredients in this happening. What exists in 1,000 years will be derived from all this, much like humans are derived from a common ancestor. I don't expect a sun-sized computer in 1,000 years, but likely "intelligence" existing on every planet and moon in our solar system, with many headed to explore Alpha-Centauri.

Since humans likely won't want to be left out in all this, we'll probably transition our own intelligence/consciousness into this technology. We'll depreciate our bad and carry along our good. We're already doing this by augmenting our existences with smart phones and other gadgets. One day these will be built inside of us, and eventually will replace us. An upgraded, better version of us. Still intelligent, but vastly more-so.

Androsynth6• 14 years ago

I completely agree with everything you say. I think that in order to avoid a cambrianesque extinction event within the human race (whether by 99% of the population entering the matrix or simply by the destructive capability of weapons becoming too much), we will need to upgrade ourselves. Most likely by meshing with machine technology.

htf• 14 years ago

Great comment. On an evolutionary scale, humans just recently became smart enough to get the ball rolling and accumulate technology from generation to generation. The potential for improvements in intelligence is enormous.

Karellen2• 14 years ago

Individuals (10 billion/1 million) do not evolve - populations evolve. Evolution takes a long time, and even the spectacularly obvious gross phenotypic changes you see between various breeds of dog, occuring over the last 10,000 years or so, has not produced any actual speciation yet. And really, unless the amount of interbreeding between two "separate" populations is infinitesimally small (or actually zero, because they are truly separate, e.g. on different sides of an impassable mountain range or sea), speciation simply doesn't happen.

I can't forsee any situation in at least the next 100,000 years which could produce two viable populations of humans which would remain separate enough for long enough for them to be considered two distinct species, and for one of them to become extinct. Evolution doesn't happen that way, or quickly enough.

eaten_by_a_grue2• 14 years ago

I find it funny that you combine awareness that nutrinos go through planets with a suggestion that they'd be good for communication.

watmough2• 14 years ago

"It only takes a tiny percentage to turn down said machine. If 10 billion people plug-in, but 1 million abstain and continue to evolve past the 10 billion, the majority stagnates. Those 1 million will eventually gain a competitive advantage over the 10 billion, evolve past them, and likely extinct them."

If you look at the distribution of wealth, there's a pretty good argument this already happened.

randome3889• 14 years ago

It is not. Because power is the ultimate drug in human society -- and only a finite amount of people can have it.

frankyh• 14 years ago

"1. AI isn't close to human level (HL) yet. I don't think we can really know what HL will be like till we get a lot closer.

2. You can't get people to seriously discuss policy until HL is closer. The present discussants, e.g. Bill Joy, are just chattering.

3. People are not distinguishing HL AI from programs with human-like motivational structures. It would take a special effort, apart from the effort to reach HL intellignece to make AI systems wanting to rule the world or get angry with people or see themselves as oppressed. We shouldn't do that. "

--john mccarthy

dochtman• 14 years ago

I found it quite annoying that the FOOM/MOOF acronyms didn't seem to be explained anywhere on the page...

biot• 14 years ago

http://wiki.lesswrong.com/wiki/FOOM

It appears to be a word made up to explain a rapid growth in AI due to the fact that such an intelligence can rewrite its source code and modify its hardware, whereas humans are relatively stuck with the limitations of our wetware. A take on the onomatapoetic word BOOM I think as it represents an explosion of intelligence/capability. By contrast, MOOF would be subverting the reward mechanism that underlies the FOOM growth, thereby resulting in a whimper (MOOF) rather than an explosion (FOOM).

cheatercheater• 14 years ago

TLDR: author starts out by comparing the human being to mythical cyber-creatures that never feel pain or hunger, then continues to exploit the comparison by supposing if you do enough coke you can live forever. Finally, humans are described as anarchistic, self-destructive babies in grown up bodies that repeatedly crap themselves without noticing, while inadvertently performing genocide on the whole humanity at once.

seiji• 14 years ago

You should read The Metamorphosis of Prime Intellect before discussing pleasure-to-death scenarios: http://localroger.com/prime-intellect/mopiidx.html

kyberias• 14 years ago

It's utterly frustrating when a writer refers to terminology or acronyms (here FOOM) that are not explained at all.

robertskmiles9• 14 years ago

It's onomatopoeic, and capitalised not to indicate an acronym, but for emphasis. I don't know who originated the term but it was popularised by Eliezer Yudkowsky and Robin Hanson. The idea is that once an AI is able to self-modify to become smarter and consequently better at self-improvement, that feedback loop creates an 'explosion' of intelligence, and the AI "goes FOOM!".

"FOOM!" here is usually accompanied by some form of hand gesture evocative of an explosion.

http://wiki.lesswrong.com/wiki/FOOM

whateverer• 14 years ago

Dead comment, perhaps not a masterpiece but certainly not worthy of being shadow banned:

by donnawarellp

  I have often thought that life is self limiting. For example, yeast is a
  living thing, it consumes sugars and it's waste is ehtanol and eventually
  they all kill themselves off by drowning in their own waste (which, I
  guess later over a meal I will be enjoying a glass of yeast pee, but I
  digress). From a molecular perspective, human DNA is not that different
  from yeast, and at a macro view our behaviour is not that different
  either it seems. Perhaps all sentient life is destine to the same fate,
  perhaps it is a law of nature. I recall reading about an argument that
  Stephen Hawking and Kip Thorne have regarding time travel. Stephen
  Hawking argues against time travel because we have never met any time
  travellers from our future. Maybe time travel is theoretically possible,
  but like yeast, we do not as a species survive long enough to develop
  the necessary technology. ugh, sorry to bring everyone down, might as
  well have a glass of yeast pee.

Karellen2• 14 years ago

I thought yeast only produces ethanol if it's respiring anaerobically, which is not a normal state for it to be in and is a short-term emergency alternative to dying immediately.

Kind of like lactic acid production in humans. We normally don't make much of it, certainly less than the rate that we can flush it through our system. We can produce more than we can handle for short periods of time if needed, but it's not sustainable. Put us in a situation where we have to keep producing lactic acid beyond sustainable levels for more than a few minutes, and we won't last long either. That doesn't make lactic acid proof that human bodies are self-limiting. I mean, we don't keep going forever, but lactic acid is not the reason for that.

ilaksh• 14 years ago

This relates to subjects I am slightly familiar with and very interested in so I am glad this is on the front page and I think he did a good job.

I have a couple of criticisms though.

The idea that we could build this powerful general human-like intelligence but it would turn out to necessarily be an addict seems unlikely, because I don't think you can build a reward system without including something to prevent shortcuts. Although creating an effective shortcut-prevention system may be challenging. See a bunch of stuff written by Eliezer S. Yudkowsky and his friends.

Also, when he writes

"It seems like the most important motivations of human civilization are related to near-term goals. This is probably a consequence of the fact that motivation writ large is still embodied in individual humans, who are driven by their evolutionary psychology. Individually, we are unprepared to think like a civilization. Our faint mutual motivation to survive in the long term as a civilization is no match for the ability we have to self-modify and physically reshape the planet."

To me that is implying a fairly obvious and typical line of leftish reasoning that for most of my life I assumed was self-evident and didn't require elaboration. After deliberately 'exposing' myself to some 'right-wing' thought and 'extreme' 'left-wing' thought, I now realize that aspiring for collectivism is not an adequate solution.

What happens with self-interested individuals in our capitalist society is that they aggregate power and resources for their own individual use. Unfortunately, something very similar happens in collectivism: power and resources are still aggregated in the hands of a few.

In other words, both systems tend towards centralization. Theoretically, the collectivists will aim for redistributing power and resources for the good of the many, but there are a couple of practical problems: they are still living in hierarchical societies and so have strong motivation for enriching themselves and the controllers cannot see enough detail to know what is really best for the rest of society.

Command economies don't work because of general problems with centralization and lack of distribution and the limitations of the types of technologies employed. A peer-to-peer network is more robust than an internet with only a few backbones.

Unfortunately, capitalists don't realize that their system also leads to monopoly and centralization.

I believe that we do need wholistic measurement and analysis of things like global resources and human equality, and a common principle of supporting the collective good. But at the same time we need to decentralize and distribute production locally. We need local decision making that operates on the basis of a global information schema with wholistic data.

A system can't be efficient or evolve without being localized, but it also can't be integrated without an accurate shared knowledgebase and an egalitarian perspective.

ryanackley• 14 years ago

Our social structure prevents self-limiting behavior. For example, eventually the robot has to be re-charged. If it isn't contributing to society then why would someone give it free energy to recharge itself.

indrax• 14 years ago

This person needs to finish reading the sequences.

showmustgn• 14 years ago

A robot has to decide is the show must go on or not.

dkrich• 14 years ago

The problem is that the instincts that humans possess today, (ie, maximizing short-term comfort, possibly at the expense of long-term survival) is what got us this far. That cannot suddenly be decoupled and discarded as useless simply because we now live in a remarkably stable environment that enables us to focus on things other than immediate survival.

I suspect that if there were some horrible catastrophe and the human race were suddenly thrown back into an archaic society without any of the technological advancements that we have at our disposal today, it would be those MOST focused on short-term gain who would be most likely to perpetuate their own, and consequently, human existence.

rsanchez1• 14 years ago

So, drugs are just as effective in AIs as they are in human intelligences.

The author calls drugs "primitive pre-FOO hacks", but since they already have such a powerful effect on human intelligence, what's the point of differentiating between pre- and post-FOO hacks? One will just lead to self-destruction much quicker than the other.

Is Intelligence Self-Limiting?

Related Stories