Learning to Cooperate, Compete, and Communicate

https://blog.openai.com/learning-to-cooperate-compete-and-communicate/?source=hn

by maccaw • 9 years ago

199 36 9 years ago

blog.openai.com

EGreg5• 9 years ago

Am I the only one that finds this scary?

As I have repeatedly said, so far the intelligence we have been producing is NOT the kind that applies abstract logic rules to figure out meaning.

It is the kind that takes full advantage of computers' strengths: perfect copying and speed.

So good ideas are copied and propagate. Neural networks are just this on steroids... basically extracting signals from noise and doing a search in a space to maximize something, and storing the results.

Humans were able to transmit knowledge, then produce books etc. Now bits can be perfectly copied with checksums.

This isn't general intelligence in the human sense. But that's what makes it scary. It can solve these problems with brute force. Resistance soon may really be futile. Not just in running, avoiding capture and death. But also in ANY human system we rely on, including voting, due process of law, trust, reputation, family, sex, humor, etc.

zitterbewegung• 9 years ago

As someone that is teaching themselves Machine learning solving problems with "brute force" doesn't actually happen and Neural Networks are a way around using brute force. It seems that you are caught in the hype around machine learning. Systems that can copy ANY human system are very far off. Also, there are human systems where AI's would not be found to be acceptable.

EGreg5• 9 years ago

Just to be clear: I didn't say they can copy any human system. I am saying they can find new ways to beat humans or groups of humans at the games we have set up. Chess and Go are just examples. Trust and Reputation are easily attacked. Voting is easily manipulated (look at gerrymandering and the increasingly shrill "fake news" claims.) Look, they are already better at making diagnoses than doctors.

I am saying that the machines won't really "understand" the human meaning behind things. They will just have "deep" insight in having explored a huge amount of variations and extracted the combinations that maximize some goal. You think you are persuasive now? You think comedians are funny? Fast forrward 17 years when any software can deliver blistering putdowns to rappers, destroy the best comedian to the point where every human agrees the guy looks dull in comparison. The AI that can easily outcompete guys at wooing a girl online. And so on. Forget the Turing test. We are in for a future where programs simply "solve" the systems we set up faster and better than we can. And share that knowledge in compact form (such as artificial neural net calibration).

Most of our systems rely on the inefficiency of an attacker. That they can't instantly start up 3,000,000 businesses or do sybil attacks in voting or destroy someone's credit overnight or win an election. But that is going to change. And I am saying that's just in the near future. Humor and trust is a little further.

hacker_94• 9 years ago

Still, better this is done in the open. Otherwise Russia and China will just continue in secret, which is definitely a world I don't want to live in.

EGreg5• 9 years ago

I agree it should be done in the open. Doesn't make it much less scary.

justicezyx2• 9 years ago

Not sure what you want to say. US can do it open, China and Russia can still do it in secret.

While, as a matter of fact, you will be naive if you believe that there is no secret research happening inside US government agencies.

obscolescence2• 9 years ago

The question was "is this good", not "could this be done in a worse way".

Russia and China

Those use the US as an excuse. Of course, these entities are only convenient monoliths when it comes to pushing the interests of very small groups within them, not when a poor American or Russian or Chinese asks a wealthy one to share.

Also, why is something done in secret and something done out in the open a zero sum game? I would think the opposite is the case, when you already have the infrastructure for 99.9% and only have to set the evil bit, that's a lot easier than doing it secretly when nobody else is doing it at all.

Danihan• 9 years ago

You're not the only one who finds it scary, as there are massively popular books on the topic..

https://www.amazon.com/Superintelligence-Dangers-Strategies-...

gonehome• 9 years ago

I found it difficult to make it through the first couple of chapters.

Having just read The Gene, his analysis of the artificial selection option for super intelligence came across as very wrong - underestimating the complexity of polygenic traits and making (likely inaccurate) assumptions about their heredity. This type of thing has a dangerous history.

The idea that a super intelligence could emerge from the internet unexplained also seemed pretty weak, but the first issue was so bad I found it hard to take anything else seriously (didn't trust the author's analysis).

There are interesting issues with artificial consciousness, but I think they're in some ways similar to the issues with biological consciousness - the testing data the neural net is exposed to and it's underlying model can lead to minds that wouldn't be considered intelligent (and dangerous outcomes as a result).

tedsanders• 9 years ago

No, I don't find this scary at all. My impression is that these algorithms are brittle, working only on very simple and structured problems. And these algorithms and their hyperparameters have to be specifically set by human researchers to match the structure of the problem. And if computer hardware progresses to reach orders of magnitude more performance, that may come at the cost of its present advantages in reliability, copying, and speed.

jboggan• 9 years ago

This is great. I've always thought intelligence can only be defined as an emergent property of self-replicating systems operating under stress, and this provides a good framework for that "stress".

TeMPOraL3• 9 years ago

intelligence can only be defined as an emergent property of self-replicating systems operating under stress

That's a prescription for making one, but it isn't a good definition of intelligence. It doesn't tell us what to expect from it.

Personally, I prefer intelligence defined as extremely strong, cross-domain optimization power.

mgummelt• 9 years ago

a multiagent environment has no stable equilibrium

It does. https://en.wikipedia.org/wiki/Evolutionarily_stable_strategy

ilya_su• 9 years ago

True in the general case, but the idea is to set things up (possibly by constantly changing the environment) so that there isn't one. This has happened at least once, when the brains of our ancestors have tripled in size relatievly quickly.

tomcam• 9 years ago

I was honestly hoping this would be about communications within the openai organization itself.

deepnet• 9 years ago

Peter Abeel ( OP link author with Igor Mordatch ) explains his groups work [1] as a guest lecturer for Berkeley's cs294-112 Deep Reinforcement Robotics.

[1] https://www.youtube.com/watch?v=f4gKhK8Q6mY&list=PLkFD6_40KJ...

This great talk starts with his work at OpenAI on neural net safety and adverserial images, then the OP research paper Emergence of Grounded Compositional Language in Multi-Agent Populations[2] concluding with his work ( with Andrew Ng ) reinforcement learning helicopter flight and stunt controllers from human pilots.

The OP multi-agents divide labour and apparent collaborative plans appear. That the goal seeking agents split up, appear to dance to and from from their goal, distracting the predators from their kin is to all appearance coordinated and clever.

Dawkin's Selfish Gene espouses alturism as an inevitability of genetic relatedness, the individual sacrifices but the genes persist in siblings.

In this work alturism emerges purely memetically.

The Nash equilibrium of cooperation jumps the local minima of selfishness in this prisoners dillemma.

Multi agent enviroments are difficult to learn with many false minima for the learners.

This work hints that the loose coupling of language, rather than direct sharing of memories or genes, is noisy enough to find more global solutions that appear complex or 'plan-like'.

Maybe these AI's should be considered as planners, yet in a bottom-up immediate-heuristic emergent way.

[2] https://arxiv.org/abs/1703.04908

Learning to Cooperate, Compete, and Communicate

Related Stories