Elon Musk donates $10M to keep AI beneficial

by cocoflunchy • 12 years ago

127 145 12 years ago

FatalLogic4• 12 years ago

I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity?

Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though.

Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.

blfr• 12 years ago

It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.

FatalLogic4• 12 years ago

Is there any sure way to stop the super intelligent AI from changing its own inherent values though?

I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.

rl35• 12 years ago

Boström argues that containment isn't really possible.

He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI.

Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system.

Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely.

Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.

pavel_lishin• 12 years ago

I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI.

The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?

ceejayoz• 12 years ago

how do you trap and imprison something smarter than you?

Jails seem to have this figured out just fine.

meepmorp2• 12 years ago

Apparently, he succeeded - his subject elected to release the fake AI.

Hasn't he refused to release the transcripts for this experiment? And I think there've been subsequent runs of the experiment, with some losses as well as wins.

Edit: I see that not releasing the transcripts is part of the design of the experiment. Which seems kind of like it'd seriously hamper any kind of independent analysis of the results or the quality of the experiment.

breuleux6• 12 years ago

It's obvious that many subjects would release the AI, if only because they don't really know why it's in a box in the first place. Furthermore, the experiment grants AI capabilities it would probably not have, such as read/write access to its own source code. It assumes the first AI we box will be thousands of times smarter than we are, which is bizarre, since we should already have access to a whole continuum of AI dumber and smarter than us. It deprives the gatekeeper from a lot of tools they would probably have access to in a real situation, for instance raw computing power that far exceeds the AI's and runs powerful analytics on the AI's brain, or the ability to take snapshots of the AI's state and run many conversation paths.

mentos• 12 years ago

What about limiting AI to a certain clock rate?

kanzure• 12 years ago

What about limiting AI to a certain clock rate?

Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)

FatalLogic4• 12 years ago

Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.

JulianMorrison• 12 years ago

http://www.rifters.com/real/2009/01/iterating-towards-bethle...

It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.

rl35• 12 years ago

I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution.

For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.

gradys2• 12 years ago

If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?

eli_gottlieb6• 12 years ago

Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.

Synthpixel• 12 years ago

The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies.

I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry.

Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.

butwhy3• 12 years ago

" The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.

yoha• 12 years ago

About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment

evanspa2• 12 years ago

"If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual."

They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.

pzxc• 12 years ago

Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended.

God, I'm such a nerd. :P

comrade1• 12 years ago

He doesn't sound that smart...

Retra10• 12 years ago

Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists?

We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions.

This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish.

So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous.

Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.

ThomPete10• 12 years ago

It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.

alexvr• 12 years ago

My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.

steve-benjamins3• 12 years ago

Philosophically I think it's a mistake to believe there is any possibility of creating artificial human intelligence (and the mistake is in our assumptions of what human intelligence is— computers are not human beings and human beings are not computers) ... But I've got to hand it to Elon Musk— when he believes in something he commits to doing something about it.

EDIT - Saying I disagree that AI is a foregone conclusion on Hacker News is probably a bit trollish... Unfortunately I'm not smart enough to condense an explanation to this comment box— but my disagreement stems from philosophers such as Hubert Dreyfuss (What Computers Can't Do) who are working off of Heidegger and phenomenology.

EDIT 2 - I know how douchey it is to namedrop Heidegger, but I really believe in this case it's necessary ...

jsutton2• 12 years ago

Why wouldn't there be any possibility of artificial human intelligence? Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. There's nothing inherently preventing us from recreating that.

psycr• 12 years ago

recreating

And that's the crux of the question isn't it? How do we determine if the intelligence is recreated or merely mimicked? Does such a distinction matter? Searle says yes. More here if you're interested: http://plato.stanford.edu/entries/chinese-room/

steve-benjamins3• 12 years ago

"Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. "

That's where I would disagree. But I'm not fit to give a full explanation here— thinkers such as Hubert Dreyfuss (What Computers Can't Do) do a much better job of explaining the critique (Dreyfuss is working off of Heidegger and other phenomenalist) .

cgearhart2• 12 years ago

I don't mean to be snarky, but it can be illuminating to change your statement just a little bit:

>Why wouldn't there be any possibility of making artificial humans? We are just a manifestation of a physical system of bonds between organic molecules that follow the laws of physics. There's nothing inherently preventing us from recreating that.

Obviously, such a summary vastly oversimplifies the problem. It is easy to look at how far we've come and think that we must be close to the goal, but it's a bit presumptuous to think that we are near to making a strong general AI.

nyrulez• 12 years ago

Well how are you so sure of that ? You must have verified or modeled it somehow I guess. Wait... you haven't.

ajuc4• 12 years ago

You can easily simulate einvironment for the AI in your computer. If you cut some senses it is feasible even now, and blind people are certainly human and intelligent so it's not required.

I don't see impossibility there, just impracticability (why create mechanic horse if we can just create car).

ThomPete10• 12 years ago

Thats not what worries people like Musk. It's that we will create something far superior to human intelligence.

And if you believe that something as complex as the brain came through evolution but life started out as immaterial matter then I find it hard to understand why you don't think this can be recreated?

drzaiusapelord2• 12 years ago

I'm not sure why human cognition is the exception in a world of machines that can emulate anything. My grandfather would never have believed that we have cars that drive themselves. Many baby boomers I work with are still wowed by the basic AI of Siri/GNow/Cortana, or even google search. Common sense and philosophical musings just isn't enough to figure out if something is feasible.

Thinking isn't that hard of a nut to crack, imho. Enough of a ruleset/branch logic is thinking. We don't need to emulate a person perfectly, the same way we don't have self-driving cars losing their tempers or tailgating people out of spite. We just need to distill the intelligence we want and voila, you have AI.

AI isn't just a HAL-like character. Its a lot of things. In fact, it surrounds us. There's expert systems, fuzzy logic, etc controlling pretty much the entirety of our technological lives. I have no idea what drugs Musk is on, but its dangerous for him to piss on AI publically. I imagine, from a practical pov, spreading FUD about AI makes sense for him economically. For someone who wants human spaceflight, talking down about robots and AI is a means to an end. Now the public doesn't want a cheap-ish robot mission on Mars with some guy in Houston wearing an Oculus-like headset. Or autonomous robots on the moon. They want a meatbag on the surface, for their own emotional and philosophical needs, price be dammed. "Price be damned" in spaceflight makes Musk a very happy camper, considering he's going to get a lion's share of that money.

There's something really neo-luddite about human spaceflight. Its like we can't admit to ourselves that machines, AI, etc just are a better fit for it. Anyone else notice that about three months ago there's been endless anti-AI sentiment in the press? Where is this stuff coming from? Who is funding it? I really think Musk and others see it as a major threat to their companies and social capital and are trying to change the narrative to suit them and their goals. I don't think we typically appreciate how one wealthy person or organization can easily astroturf the web and the press.

Personally, I think the more feasible futurist path is actually taking the environment seriously, pushing the path to peace politically, trying to bring in a post-scarcity world via cheap replication, and using AI the same way we use all tools -- for our benefit. Fleeing to some other planet or moon without fixing life on Earth first guarantees we bring our terrible politics and baggage to this new destination. Let's fix things here first and then worry about moving to Mars. "Fixing things" more than likely involves strong AI implementations in various parts of our lives.

happyscrappy• 12 years ago

What happens when AI reaches provably better decision outcomes whose chain of logic is opaque to mere humans? Now you have a trusted black box that cannot be audited by humans.

enoch_r• 12 years ago

using AI the same way we use all tools -- for our benefit

Musk and others are concerned about very different things than "we'll accidentally use AI wrong." And they're not concerned about the AI we already have, and they're certainly not "pessimistic" about whether AI technology will advance.

The concern is that we'll develop a very, very smart general artificial intelligence.

The concern is that it'd be smart enough that it can learn how to manipulate us better than we ourselves can. Smart enough that it can research new technologies better than we can. Smart enough to outclass not only humans, but human civilization as a whole, in every way.

And what would the terminal goals of that AI be? Those are determined by the programmer. Let's say someone created a general AI for the harmless purpose of calculating the decimal expansion of pi.

A general, superintelligent AI with no other utility function than "calculate as many digits of pi as you can" would literally mean the end of humanity, as it harvested the world's resources to add computing power. It's vastly smarter than all of us put together, and it values the digits of pi infinitely more than it values our pleas for mercy, or our existence, or the existence of the planet.

This is quite terrifying to me.

A good intro to the subject is Superintelligence: Paths, Dangers, Strategies[1]. One of the most unsettling books I've read.

[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

breuleux6• 12 years ago

Skimming Dreyfus's criticism, it is mostly applicable to early AI research which tried to reproduce intelligence through high level symbolic manipulation and rule-based systems (and he was right: that approach failed). Modern AI research goes in the direction of connectionism and machine learning, so his criticism doesn't really hold (and neither does Musk's, I believe).

gradys2• 12 years ago

I encourage you to explore some other philosophers' thoughts on AGI. Dan Dennett in particular is an AI-optimistic philosopher of mind who directly addresses Dreyfus' AI-skepticism, quite effectively in my opinion.

steve-benjamins3• 12 years ago

I have read Dennett and he's a fine analytic philosopher. But it's important to note that any discussion between Dennett and Dreyfus is a very broad discussion of continental vs analytic ... which is not to discredit Dennett's argument (afterall it's Dreyfus who picked the fight in the first place)

31reasons• 12 years ago

AI is already dangerous with its recommendation engines shaping our lives and used for decision making in lot of unethical places we don't know about. Every technology has a dark side. It does not need to reach human level intelligence or beyond it.

fixedd• 12 years ago

Can you give some examples of places where it might be currently being unethically used?

olalonde• 12 years ago

It's not a philosophical question and the AI we are talking about is not defined philosophically either.

eli_gottlieb6• 12 years ago

An AI can still perform tasks and act for goals without being conscious or having subjective experiences. The philosophers should either engage the real subject or shut up.

creack• 12 years ago

I love the date of the article.

utopiah• 12 years ago

Scary isn't it. They have an AI AND a time machine...

bhouston2• 12 years ago

Is FutureOfLife.org a competitor to Machine Intelligence Research Institute? So now we have two charitable organizations researching how to make AI's beneficial? Who knew this was going to be a growth industry.

nbouscal2• 12 years ago

They're not competitors so much as allied organizations using different methods, as far as I can tell.

eli_gottlieb6• 12 years ago

They're not so much competitors as FoL and FoI institutes are exploiting the comparative advantage of having vastly more famous and reputable staff who can be taken seriously by the academic community at large. I hang out on LessWrong sometimes and know some of the MIRI people, and while I appreciate the technical work they do on their subject, what they simply don't have is brand image. So they've gladly let someone with more brand-image than them take the "marketing and advocacy" parts of their cause off their hands, freeing the math nerds to concentrate on math.

Kutta• 12 years ago

Tallinn, Tegmark and Bostrom are long-time MIRI supporters, so this probably won't be positioned as a competitor.

It also seems that there was a not-very-advertised AI conference in January, with pretty much everyone in the AI-risk scene attending, most of MIRI staff and Elon Musk included.

olssy• 12 years ago

If only humans could agree on what makes things beneficial to themselves. Are self-driving cars and rockets to Mars beneficial to humanity? Depending on how you look at it the answer is both yes and no.

sgt• 12 years ago

Where is Ray Kurzweil's name on that list?

DanAndersen• 12 years ago

From what I understand, Kurzweil tends to be much more optimistic about AI, predicting that humanity will merge with advancing AI rather than it running away from us.

Chathamization2• 12 years ago

And predicting that it’d happen five years ago…

CodeCube• 12 years ago

I know it's a bit dramatic at this point to take this stance ... but this almost gives me deja-vu of racism. Like a modern version of: http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Ro...

Here's what I mean; I'm not suggesting that AI has, or will have, some innate set of "rights". Nor am I suggesting that anyone is wrong for wanting to fund and research "AI Safety". It's just the first thing that came to mind while reading that post, as it spoke about need to steer, control, and regulate AI so that it's beneficial for "us" ... already setting up an "us" vs "them" dynamic.

Anyways, just thought it was an interesting juxtaposition to mention ... thoughts?

ObviousScience2• 12 years ago

I think it's incredibly selfish and poorly thought out to create another kind of mind only if it serves us.

Much like parenting, I think it should be done because there's another kind of mind to be had, regardless of the exact outcome (which we shouldn't necessarily try to fully control).

eli_gottlieb6• 12 years ago

Parents do not routinely expect to be murdered by their children, nor is moral to expect or require such.

proveanegative• 12 years ago

I think your analogy demonstrates the perils of anthropomorphism when thinking about AI. If we design a resource-consuming autonomous entity it should be assumed that it will compete with us for the resources it uses until proven otherwise. That is at the core of the concern about AI safety. It is about preventing human extinction.

mattcwilson• 12 years ago

I think it depends a great deal on how charitable our intentions are towards the AI.

Few would argue that, in raising a child, acting on the child's behalf and shaping their learning is an inappropriate arrogation of control. On the other hand, never relinquishing when it becomes clear the child is independent would be.

If that child goes on to be superior in intellect, in capability, etc, that child's opinions and values are likely going to be a direct reflection of the parents', or so one hopes. Sure, it's no guarantee, but is there a better approach?

When commenters talk about "trapping" or "imprisoning" a rogue AI, I feel like we've already missed the point. And yeah, such talk does have echoes of sins of the past.

I think it's also relevant to ask what our expectations of the AI would be, and how much of a burden it places on the AI. If you're thinking you need to "capture" an AI that's trying to escape so it can continue "working" for you... that sounds like you're doing something terribly wrong.

On the other hand, continuing the parenting analogy, if what you're asking for amounts to a trivial favor, or some occasional assistance with a burden, that doesn't seem at all unfair.

"Bobby, could you come over this afternoon and unconfuse our legislative body? It's all gummed up and going nowhere."

buro9• 12 years ago

The default assumption seems to be that if we create a thing of superior intellect, then that thing is going to be a threat to humans.

I believe that it is possible to create intelligent machines, and that those machines will challenge our perception of what it means 'to be'.

I don't necessarily agree with the implicit assumption that when that time comes, that a thing of superior intellect is obviously going to be 'evil' towards things of lesser intellect. Unless we are actually making a statement extrapolated from our own experience of how we perceive our relation to other life-forms on this planet.

drdeca• 12 years ago

I don't think the common claim is that it would be "evil" or malicious, but that it could be a threat, being both powerful and unpredictable.

The idea that I've often seen people mention, is the idea of a "paperclip optimizer", in which an AI superintelligence primarily optimizes for there being more paperclips, not taking into account side effects of this, which other people might find harmful. If I understand the idea correctly, which I very well might not, it is meant to suggest that if an AI superintelligence is created with a particular goal in mind, we would do well to be very careful when choosing the AI's goals.

It would I suppose be essentially a similar problem to that of a "literal genie". (Except harsher, because the AI would go through steps to accomplish the task, instead of just magically making the task so, and therefore would have more side effects?)

listic• 12 years ago

A post from the future! http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zp...

k__• 12 years ago

He fears Roko's basilisk ;)

http://rationalwiki.org/wiki/Roko%27s_basilisk

bhouston2• 12 years ago

A theory that those that do not donate to AI-promoting groups will be punished in the future by AIs? Whoa, weird. Great for promoting fundraising activities though. Very church like.

butwhy3• 12 years ago

And Musk will be punished for donating to a cause potentially preventing its inception. ;)

JamesArgo3• 12 years ago

No one has every believed in the basilisk, save for maybe Yud. Everyone on LessWrong thought it was dumb from day one. LessWrong, now mostly dead, was a fun community full of smart people. That article is trying to make the place out as a cult, but no one bought any of that stuff, not even Roko. He was just playing around with local memes. It's pretty gross how this whole basilisk thing has turned out.

Keyframe2• 12 years ago

I thought he was strapped for cash. Divorce and all.

zodiakzz• 12 years ago

Feeding the troll but he had his wife sign a contract where she can't claim ownership of any of his assets.. smart ;)

Keyframe2• 12 years ago

Not a troll :) I remember these headlines: http://venturebeat.com/2010/05/27/elon-musk-personal-finance... I wasn't paying attention much since then, so was wondering.

mikeyouse• 12 years ago

Elon's stakes in his various companies:

- Tesla - 28M shares @ $193 = $5.4B

- Solar City - 20M shares @ $51 - $1.0B

- SpaceX - Still private, but some wild assumptions that Elon owns ~33% at a valuation of 4x total funding - $350M

So his net worth is somewhere near (and probably north of) $7 Billion. If his wife takes half, he'll still be able to donate plenty to AI research.

honeybooboo123• 12 years ago

I doubt he can really believe that dropping 10M on a project will somehow prevent Skynet from happening. AI will just get nasty, or it won't. No one knows.

butwhy3• 12 years ago

Fine, it will reduce the likelihood of it "getting nasty".

bra-ket• 12 years ago

we already have The Three Laws of Robotics

http://en.wikipedia.org/wiki/Three_Laws_of_Robotics

rfrey• 12 years ago

You're probably joking, and my humour unit is impaired today. But in case you're not...

Those laws are the requirements document - the $10mm is for the implementation.

Symmetry• 12 years ago

If you can define "harm" well enough for the first law to be useful then you've solved the problem and laws 2 and 3 are probably superfluous.

breuleux6• 12 years ago

A lot of Asimov's work deals with loopholes and inadequacies in these laws, though: for instance, what is a human being? Does surgery violate the first law? If a human is going to injure another human, can the first law be overridden? If a robot believes that some entity is not human, do they have the responsibility to check before injuring them?

I mean, realistically, AI safety is not going to be achieved with a list of written laws. Heck, the way it's going, we wouldn't know how to enforce such a list at all: first we would need the AI to understand human languages, and then we would need the AI to care, that is to say, robots would need to know and follow the second law before you can tell them about it.

kbody• 12 years ago

How can you control the creation of code?

This news sounds like "donating to keep software bug-free". If someone wants or even accidentally, they can create buggy code or "evil" AI.

cjfont• 12 years ago

Perhaps the incentive here is to educate coders as to what patterns to avoid which will have an inclination toward "evil" AI. This won't stop others for whom this is the goal.

davmre• 12 years ago

You can donate to keep software bug-free by funding research into safer programming languages (e.g., Rust), tools for formal verification and static analysis, testing frameworks, software engineering methodology, etc. Sure, someone could always ignore all the research advances and just write unsafe, untested C code. But for the most part, people want to eliminate bugs, so as the techniques for doing so mature, they will tend to become adopted in practice and certain types of bugs will become much less frequent.

Similarly, there are technical research questions related to how to design an intelligent agent that is "controllable", in that it acts to achieve its given goals but not to such an extent that it would resist being switched off or reprogrammed with new goals. Making progress on answering these questions doesn't guarantee that any given developer will use whatever techniques are discovered. But insofar as pretty much everyone, even evil masterminds, wants to maintain control over their AIs, the availability of techniques for 'safe' AI will at least decrease the likelihood that a powerful AI is built without any safeguards.

reacweb• 12 years ago

AI is a buzzword that is coming more and more in car. Creating a mental association between AI and safety is a very smart PR idea.

pekk• 12 years ago

AI is just software.

So any discussion about AI gone wrong is really a discussion about malware and other kinds of software which are used by people to do bad things. If my software runs a power plant for an evil facility (full of sharks with lasers), is it evil software? The evilness of the software is parasitic on the intent of its operators.

But if the issue is software that goes badly wrong in unexpected ways, we are really just talking about software engineering again, segfaults and Ariane 5 and how to handle user passwords.

Obviously if we plan to outsource big tasks to chaotic software that we have trustingly given power over, let's say, all the cruise missiles, that is less of a scary emergent singularity scenario and more a plain-old-stupid scenario.

ggreer• 12 years ago

Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

-- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies[1]

A lot of people in this thread seem to be falling into the same attractor. They see that Musk is worried about a superintelligent AI destroying humanity. To them, this seems preposterous. So they come up with an objection. "Superhuman AI is impossible." "Any AI smarter than us will be more moral than us." "We can keep it in an air-gapped simulated environment." etc. They are so sure about these barriers that they think $10 million spent on AI safety is a waste.

It turns out that some very smart people have put a lot of thought into these problems, and they are still quite worried about superintelligence as an existential risk. If you want to really dig into the arguments for and against AI disaster (and discussion of how to control a superintelligence), I strongly recommend Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. It puts the comments here to shame.

1. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

ThomPete10• 12 years ago

I always think about it like this.

If immaterial matter could turn into conscious beings like us how can anyone claim that we are the last step in that evolution.

rl35• 12 years ago

"Along with research grants, the program will also include meetings and outreach programs aimed at bringing together academic AI researchers, industry AI developers and other key constituents to continue exploring how to maximize the societal benefits of AI; ..."

Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope.

For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse).

However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise.

For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it).

Humans instruct it to solve the problem of death and disease. It does so.

Humans instruct it to solve the problem of crime. It does so.

Humans instruct it to solve the problem of war. It does so.

Humans instruct it to solve the problem of poverty. It does so.

Humans instruct it to explore the universe and solve the most vexing existential questions. It does so.

In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima.

That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering?

Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence.

For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing.

While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.

lione2• 12 years ago

I'd think that that friendly super intelligence would realize the issues associated with post-scarcity society and would act in the best interests of humanity.

It would weight human happiness as more important and work to avoid the pitfalls that would be undesirable.

There's more utility for it in the human population being happy/not giving up on life then most of the things it would do (assuming it's primary goal is to further and safeguard the human race).

There would be aspects of society it COULD run, but feel that giving us purpose would be more beneficial then the increase in productivity or whatever of it running that aspect of society.

Maybe it can make beautiful gardens and art, but instead leaves it to the humans to give them some sort of purpose.

I don't see how or why a post-scarcity society would do away with art/culture/many of the subjective things that make life worth living and provide purpose.

Most people don't quit doing the things they like just because someone is better at it.

My two cents.

rl35• 12 years ago

I think you may be right. Admittedly, most of what I said was really only applicable to a Genie superintelligence that implements our wishes on a command-by-command basis.

A friendly Sovereign superintelligence (i.e. operating under its own goal/value-directed volition) would probably not rob us of purpose, though the societal changes in either scenario would likely still be quite profound.

Of course, a positive superintelligence outcome would be a miracle in and of itself, so there's that. Coming to terms with a newly created utopian society sure beats extinction.

eli_gottlieb6• 12 years ago

Excuse me, but if you can't come up with anything to do with yourself after suffering and despair are gone from the world, that's your failure of imagination. I'm pretty sure that if really placed into such a situation, you will find a way to cope, just like the rest of us.

rl35• 12 years ago

I'd like to think I would cope just fine. In fact, I'd probably be elated.

My concern was more what would happen on a societal scale, what would become of humanity as it was known.

Elon Musk donates $10M to keep AI beneficial

Related Stories