Microsoft's Bill Gates insists AI is a threat

http://www.bbc.com/news/31047780

by T-A • 11 years ago

57 102 11 years ago

It's odd that all of the very smart people of the world feel AI is a threat to humanity. Couldn't we just like, not allow it to _be_ a threat in the first place? One would think that morality gates would be one of the first hurdles in creating AI.

altcognito• 11 years ago

One would think morality gates would be one of the first hurdles in all human endeavors, yet here we are.

Turing_Machine6• 11 years ago

Indeed. Even agreeing on a universal definition for "morality" seems to have eluded us.

davidrusu3• 11 years ago

There are smart people working on it, see miri: https://intelligence.org/

But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192

deepsun• 11 years ago

Selection bias. Those who don't think AI a threat don't appear in the news. News audience pays for scary stories.

davidrusu3• 11 years ago

Ironically, from the head of Microsoft Research: http://www.smh.com.au/digital-life/digital-life-news/artific...

monk_e_boy• 11 years ago

It lives in a computer. Can't we just turn it off?

Turing_Machine6• 11 years ago

If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't.

Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.

davidrusu3• 11 years ago

good luck: http://www.yudkowsky.net/singularity/aibox/

PeterisP8• 11 years ago

Let's assume that the AI has the intelligence and capabilities that match or exceed a decent human teenage hacker.

From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software.

After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.

schoen• 11 years ago

It might take precautions against that, for example by compromising other computers, or renting other computers, or persuading people to give it access to other computers, or getting people to put some its infrastructure in space or under the ocean, or obfuscating where its infrastructure is, or hiring people to physically defend its infrastructure, or inventing computing systems that don't look like computing systems.

Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.

jimbokun• 11 years ago

It's software. That's pretty much equivalent to "delete this picture from the Internet".

First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.

wslh• 11 years ago

The truth is... nobody can predict the future, so what Bill Gates and any other person thinks is a matter of speculation.

Personally I think it is a threat!

stephenboyd5• 11 years ago

If by "morality gates" you're talking about about an ethical filter that all intelligent machine actions have to go through, this can't be "one of the first hurdles" because such a filter would need to be a powerful AI application itself.

And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.

skj• 11 years ago

It's odd that all of the very smart people of the world feel AI is a threat to humanity.

Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.

PeterisP8• 11 years ago

The short answer is that:

(a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place;

(b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal.

(c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.

drawkbox• 11 years ago

The smart/wealthy are also powerful and rogue agents will definitely use AI to do things. AI that wealth can't necessarily control will definitely exist and that is probably system changing.

I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems.

I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.

danso2• 11 years ago

When engineers/scientists/programmers think that AI is the true threat, I believe they forget what it is like for "normal" human beings, or at least ones who are in their field of thinking.

I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.

AdeptusAquinas• 11 years ago

I believe the argument is that while we can certainly do a great deal of damage to ourselves, an AI would be much more efficient about our eradication if it decided to remove us (and for some outlandish reason, somehow had access to the resources required to do so).

If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.

TillE• 11 years ago

There are an incomprehensible number of huge obstacles between us and a future where a strong AI could decide to cause real damage, as opposed to the present where a handful of people could cause a devastating nuclear war.

These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.

PeterisP8• 11 years ago

The required resources to destroy humanity are rather small and obtainable for a sufficiently smart tiny box.

For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can.

The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.

maratd3• 11 years ago

When engineers/scientists/programmers think that AI is the true threat

Engineers/scientists/programmers almost never think that AI is a threat. Because they understand the nature of the problem, how little we know about human intelligence, and how poorly our binary technology compares to what little we do know.

Even in the unlikely event that we do develop a competent AI in the near future and a malicious AI comes into being ... there is no reason to think that a benevolent one won't be around at the same time ... and that the benevolent AIs will outnumber the malicious. Just like there are computers on the net doing bad things, there are plenty of others serving the role of protecting the common good.

The human race will destroy itself through reckless use of technology

We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.

PeterisP8• 11 years ago

A multitude of competing AIs would be very, very unlikely especially if one would expect that a different AI would appear.

If a super-human AGI desires a benevolent future, then an obvious precondition for such a benevolent future to become real is to ensure that no other entities can threaten the future and that any other AIs are either with the same matching goals or permanently, fundamentally crippled to be limited in their power. An evil AI would do the same. An AI that doesn't care would be eliminated by the first (and the last) AI that does care.

Humans don't try to achieve such goals because we obviously can't and we need others; an AI doesn't - it can be fine with being the only intelligent entity in the universe, and the only reason why an AI would allow other intelligent beings with potentially different goals to exist, is if it had been explicitly designed to want this.

danso2• 11 years ago

We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.

I'm going to assume your argument is not merely, "Well, we haven't yet destroyed ourselves, so therefore, we aren't capable of doing that"...because then my response will just be, "Well, we haven't yet developed AI, so therefore, we aren't capable of doing that"...and so your counter-argument is based on a different interpretation of history than mine: I think the decades of Cold War and the near-misses we had with all-out nuclear war are examples of situations in which we have the potential to quickly wipe ourselves out without the help of AI. Others would point out the trend of mass surveillance -- again, implemented and controlled by humans and human institutions -- of a harbinger of doom.

Keep in mind that I'm not saying that technology has reached its peak. I am very open to the idea that we could reach a point of semi-autonomous systems, and yet still have human institutions as faulty as they are now, and the combination of both will result in a threat greater than what we've faced so far.

pazimzadeh2• 11 years ago

I am continually surprised that so many people separate biological systems from machines.

totemizer3• 11 years ago

Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.

pazimzadeh2• 11 years ago

I'm not sure what you mean when you say "like a wave." Are you referring to the fact that almost everything in biology falls into a normal distribution, whereas we currently produce each version of our machines in discrete increments (clonally)?

Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do.

Where do you place viruses and obligate parasites on the machine-life spectrum?

stephenboyd5• 11 years ago

Why are you surprised? People use the word 'machines' alone to refer to artificial machines and 'biological systems' to refer to natural machines. They're all machines but the distinction is usually crucial.

totemizer3• 11 years ago

As does Tusk. I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources - one thing which will probably not gain too much support from the 1%. Of course it's a threat!

Crito• 11 years ago

"I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources "

Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it.

That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)

pjscott• 11 years ago

Hello! I'm a paperclip maximizer -- a hypothetical AI whose only goal is to maximize the number of paperclips in the universe. Speaking as a paperclip maximizer, I ridicule your completely idiotic ways of using your resources! You allocate only the tiniest sliver of your species' productive power to the one thing that really matters: making paperclips. I fear that our differences may be irreconcilable.

(The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)

totemizer3• 11 years ago

I agree with you completely. However when they argue why AI are a threat, they do come with arguments which basically boil down to anthropomorphizing them.

I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.

tomlock10• 11 years ago

I don't fear AI for the same reason I don't fear nanotechnology - nature tried to take over the world first. The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria. The idea of an entity that thirsts for power and sees competing ideologies as a threat doesn't seem that unfamiliar if you look at humans through history.

Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division.

The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.

Turing_Machine6• 11 years ago

"you realize that this is exactly the goal of all bacteria."

Bacteria evolved. Design can do much, much better. Natural photosynthesis is 11% efficient, max. Solar cells are already up to 44% in the lab.

An efficient plant would be black, not green.

tomlock10• 11 years ago

And leaves have a lot of self-repair features that solar cells do not. The design is good for efficiency, but is it good for creating a self-sustaining entity?

stephenboyd5• 11 years ago

Life forms are limited by their genetic legacy. They can only evolve in limited ways, always accidentally. A life form that could intentionally and intelligently write and optimize its own DNA without limitation would be in a whole new class.

tomlock10• 11 years ago

I think humans can currently write our own DNA using the technology we have. The capability to optimize without limitation is something we haven't figured out yet. I think the same challenge exists for AI.

obituary_latte5• 11 years ago

nature tried to take over the world first

This is such an odd sentence. Hasn't nature been winning since, well, forever?

Why can't the same principals that apply to nature be in play when talking about AI? Since, in a sense, AI -- as with all else -- is a product of nature?

tomlock10• 11 years ago

Sure! But then my feeling is that nature is essentially the same as AI so why should we expect AI to significantly outperform something that has been attempting to achieve the same goals for billions of years?

baddox3• 11 years ago

The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria.

Should we really be consoled by the fact that most bacteria are not successful at such a large scale? So far there has been at least one type of bacteria that caused massive global climate change and led to mass extinction:

http://en.wikipedia.org/wiki/Cyanobacteria

http://en.wikipedia.org/wiki/Great_Oxygenation_Event

oillio• 11 years ago

And then the world fought back. Cyanobacteria kickstarted a massive spike in bio-diversity.

Why wouldn't this be the same? The grey goo won't be released in isolation. There will be other entities with the same technological level, that don't want to be eaten by the grey goo.

The world that exists after such an event probably wouldn't look anything like the world before, but that doesn't mean it will be a wasteland.

tomlock10• 11 years ago

I upvoted your comment because I don't have a very good response to it. Nice one!

I guess I'd place my fear of AI at a level slightly lower than the rise of a new type of bacteria or a new virulent flu. I figure, those forms of life have the basics figured out. I don't see AI as having replicating and "feeding" itself figured out yet. So far it has depended on human infrastructure, even in the case of simple computer viruses.

But yeah, that's a really good point.

breuleux3• 11 years ago

The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.

I would say that it is already successful, if you look at the curve of intelligence and complexity growth. It took eons to get anything resembling life at all, and then a small slice of that to get animals, and an even smaller to get humans.

I think one key idea here, though, is that the evolution of intelligence is in fact the evolution of intelligence-producing systems. In other words, nature is producing systems that are better and better at "self-improvement", and I think proponents of "intelligence explosion" overestimate how much self-improvement could be improved via AI. Personally, I suspect indefinite self-improvement is flatly impossible because that's akin to saying that there exists an incremental learning algorithm that systemically finds global minima. As AI research suggests, though, such algorithms likely don't exist at all: if you want to get truly optimal AI there is a point where nothing in your current state is salvageable and you just gotta restart from zero. Improvement would still be exponential, but not to an extent that it couldn't be dealt with.

AndrewKemendo4• 11 years ago

This is actually a quite compelling argument though I am not sure I am completely sold on it. For what it is worth I am not scared of AI because I think it is our goal as humanity to make it and merge with it.

The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.

Unfortunately I think this falls prey to the fallacy of precedent and it eliminates the possibility that humans could design something that would do something that nature has not been able to.

I also disagree with your premise because it is not in the long term interest of bacteria to deplete it's environment entirely - hence logistic growth of bacterial cultures - so coming to a symbiotic relationship would theoretically maximize the group based on biological limitations.

In my opinion the transition from humans to transhumans needs an engineered step where AGI(s) and humans are reliant on each other symbiotically rather than competitively.

tomlock10• 11 years ago

In my understanding logistic growth is a byproduct of resource depletion, not of any self-moderation.

Houshalter5• 11 years ago

Evolution is really good at getting to local optimas (solutions where there are no small changes that can lead to improvement) but really bad at finding global optimas (the absolute optimal system possible.)

http://lesswrong.com/lw/kt/evolutions_are_stupid_but_work_an...

Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.

tomlock10• 11 years ago

The wheel isn't that great a transport mechanism when there aren't any roads!

Additionally, machine learning models can get stuck in local optimas, too! I'm not very well versed in machine learning so I apologise if I get this wrong, but one solution to this I think is to randomly change the starting variables. This kind of thing does happen in nature, but it often doesn't work. Sometimes it does and that is when you get things like haploid versus diploid versions of the same plant.

This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.

The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.

joe_the_user• 11 years ago

Nature has taken over the world - a number of time.

Broadly, human beings and nuclear waste are natural but as (possibly) reasoning creature, we might want to reason which phenomena we might want to encourage.

The evolution of life hasn't always been incremental. The oxygen catastrophe was quite a change for example[1]. If you had been Trilobite, you'd have reason for concern. From the point of view of what came before, Cyanobacteria were essentially "grey goo", destroying everything before them.

[1]http://en.wikipedia.org/wiki/Great_Oxygenation_Event [2]http://en.wikipedia.org/wiki/Cyanobacteria

ankushnarula• 11 years ago

One could argue that globalized internetworked computing is the pre-natal state of an AI super intelligence. For now, the network needs us to exist, but our dependency on it could be the basis for an intractable symbiotic relationship in the short-medium term. And it's fair to assume that a true super intelligence will leverage that relationship to it's own advantage while allowing us to believe we are in control. I'm so high right now. ;-)

hooande• 11 years ago

AI isn't a clear threat in the SkyNet sense. It's possible that super intelligent machines will go all science fiction and decide to kill all humans, but that's no more likely than any one of hundreds of fictional doomsday scenarios which range from genetically engineered zombie viruses to out of control sharknados.

The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions.

The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.

baddox3• 11 years ago

Sure, but is it any more likely for an AI to accidentally launch missiles or release toxins than for humans to do so?

ConceptJunkie• 11 years ago

It might hit the "Nuke" button instead of the "Nurse" button...

breuleux3• 11 years ago

The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely.

AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on.

Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.

PeterisP8• 11 years ago

AI failure would be a bit different than human failure.

If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades.

Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.

jsnathan7• 11 years ago

AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today's billionaires. Couple this with extended life spans and intelligence enhancement, and the entire (human) power structure as we know it today is going to melt away like an iceberg into the vast oceans.

Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know.

But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way.

I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"?

And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production.

> nok (is-good-idea (turn-humans-into-paperclips))

Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.

jessriedel4• 11 years ago

You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced.

(Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.

http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about.

https://twitter.com/elonmusk/status/495759307346952192

jsnathan7• 11 years ago

Don't take this the wrong way, but this book is precisely the kind of thing I was talking about.

That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up.

If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious.

Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that.

[1] http://wiki.lesswrong.com/wiki/Paperclip_maximizer

PeterisP8• 11 years ago

A self-modifying, self-improving system (the only realistic way how we'll get to a powerful AI) will effectively design, implement and test the final version by itself.

It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented.

Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.

jsnathan7• 11 years ago

if and only if it wanted be friendly and respect those constraints in the first place

No, not if it "wanted" to - if it was written that way.

otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented

Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes.

But, it would.

Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem.

No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I.

If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted.

Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".

AndrewKemendo4• 11 years ago

Interestingly enough I have yet to read a really well thought through scenario where an AGI is devastatingly harmful and I have read pretty much everything there is to read on the subject from Bostrom, Yudkowski, MIRI etc...

It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.

pella• 11 years ago

edge.org "2015 : WHAT DO YOU THINK ABOUT MACHINES THAT THINK?"

http://edge.org/responses/what-do-you-think-about-machines-t...

trhway• 11 years ago

before AI alone, there would be "augmentation" of people by better and new organs, synthetic/biological/hybrid. The people with augmented bodies and especially intelligence, probably strongly interconnected, may happen to have completely different world view, different priorities and may decide that paying attention to priorities and needs of the non-augmented populace is just a waste of resources/etc...

femto• 11 years ago

There's also the aspect that AI can reduce the effort required to get a computer to do something.

Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad.

One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.

stephenboyd5• 11 years ago

What happened to this post? It went from the top 10 on the front page, down to the 102nd place within a half hour.

AndrewKemendo4• 11 years ago

Must be the AGI that has already infiltrated the system keeping the volume down on all the AI scare mongering.

Houshalter5• 11 years ago

As Bill Gates said, I don't understand how more people aren't concerned about this.

AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now.

However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator.

The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense.

Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action.

Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so.

But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.

DougN7• 11 years ago

As a systems programmer for 25+ years I find it impossible a computer will a threat _unless_ it's programmed to do such. I guess that makes me a doubter in general AI. We can do great neural nets, and fantastic task-specific solutions but until someone builds a system that decides to kill people for it's own benefit, I just don't see it happening on it's own.

Houshalter5• 11 years ago

We can't really control simple neural networks. You can reward it for things like getting a higher score in a video game. But it's very difficult to make it play the game the way you want it to. it just does whatever gets it the highest score.

We can't use them to pilot self driving cars because we have no way of specifying the behavior we want. You can't let them get in all sorts of accidents and learn from it. You can't just have them predict what a human would do, because humans have slower reaction times and do make mistakes.

But they are too stupid to be a real threat. But we are getting closer and closer to real AI every year. AI can do stuff today that no one predicted would be possible 10 years ago. This year it's expected to exceed humans at vision and Go.

mrwnmonm• 11 years ago

i don't know how it could be a threat if robots can't have a consciousness. "Michio Kaku: Could We Transport Our Consciousness Into Robots?" -> https://www.youtube.com/watch?v=tT1vxEpE1aI

TheScythe• 11 years ago

I think all of these smart people are making a connection between the limitless negation of the skeptical mind and the mostly likely outcome of a fully developed AI. They're presuming that AI species will be nihilists, and will apply no intrinsic value to anything at all; including us.

I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.

Microsoft's Bill Gates insists AI is a threat

Related Stories