Show HN: Recurrent neural net plays 'neural slime volleyball' in JavaScript

http://otoro.net/slimevolley

by hardmaru • 11 years ago

147 74 11 years ago

otoro.net

chillingeffect• 11 years ago

This is massively frustrating :) although I started to develop some strategies once I got used to the controls.

I have a question though: In useful.js, there's a function like this:

function getHeight() { return $(window).height()-200; }

Why is 200 subtracted from the window's height?

squeaky-clean2• 11 years ago

It looks like it's actually

    return $(window).height()-20*0;

Which is just subtracting zero from the result. Not sure why. The only reason I know for doing that is to cast a string to a number without the function overhead of parseInt(). Furthermore, I don't actually see any code using that function.

juliangregorian2• 11 years ago

Probably leftovers from development phase.

ebbv• 11 years ago

This is cute but I think it's just a really hard game to play as a human. Your little dude can't move fast enough.

uuave• 11 years ago

Yeah, judging the angles and getting the ball to bounce in the right direction is really hard. But I'm gonna see if my kids can play this. I'm guessing they probably can.

I like these old-school games.

Ntrails• 11 years ago

i'm finding a real problem with delay on jumping/direction which seems to be part of the problem? As though I'm expecting movement to behave slightly differently. could just be making excuses for myself though!

hardmaru15• 11 years ago

I should have done a better job for the ball dynamics

Perhaps the easy version is more suitable for the kids

http://otoro.net/slimevolley/easy.html

Thanks for the feedback it is very helpful.

eghri• 11 years ago

I agree, but I'm not sure I would have said that without first getting clobbered by the AI. The list of really hard things for a human seems to be growing these days.

billconan• 11 years ago

very interesting! now you owe me 10min of my life. :)

nine_k• 11 years ago

Is the graph of the top a representation of the neural net? That is, are 11 neurons in 2 layers enough to operate the game?

sweezyjeezy2• 11 years ago

Since RNNs are self connected, in some sense it has a lot more depth than just 2, so they can learn very non-linear features (but the weights of the connections are shared across the layers).

deepnet21• 11 years ago

[accompanying blog post](http://blog.otoro.net/2015/03/28/neural-slime-volleyball/)

I too am perplexed as to what the neurons represent, the blog describes the RNN as 19 inputs connected to 7 outputs.

Examining slimevolley_pro.js (Agent.prototype.drawstate) shows only the agent states are depicted onscreen x,y,vx,vy,bx,by,bvx,bvy in the top row and actions forward, jump and backward below.

The code is well documented and builds on convnet.js by Karpathy.

The network is trained by genetic algorithm and self play - so this is a neural net trained with reinforcement learning.

The author's method seems unique and effective.

The blog post comparison of the resultant 'genomes' of network weights seem to show that the space has been searched exhaustively.

The game is played with trained networks but the code contains a training flag - it will be interesting to watch them train with self play.

albertzeyer• 11 years ago

The method is not unique. See some of the work of Schmidhuber's group. They are doing a lot of reinforcement learning for recurrent nets (LSTMs) and also via evolutionary algorithms. Eg see Evolino.

juliangregorian2• 11 years ago

It's just an abbreviated form of the nets shown in this (http://blog.otoro.net/wp-content/uploads/sites/2/2015/03/sli...) image, the 3 outputs displayed correspond to the "forward", "jump", and "back" on the diagram, and the 8 inputs correspond to the agents' and balls' position vectors (which you can see in the Brain() code in slimevolley_pro.js).

seanalltogether2• 11 years ago

I feel like I'm missing something here, can anyone give a brief synopsis on the significance of this?

sweezyjeezy2• 11 years ago

The AI of the computer player is an RNN, which is a type of neural network. The objective of the game, what the controls do, and strategies were never hard coded into the AI, it learned them through trial and error, the way a human would. It's a lightweight implementation too, done in JS. One neat feature is that you can see the real-time activations of the neurons which are connected to the control nodes in the background.

seanalltogether2• 11 years ago

Is the "learned" behavior stored in the javascript somewhere, or is it coming from a socket in real time? Is it just a giant data array for handling possible decisions?

discardorama3• 11 years ago

The interesting thing, to me, is how was network the trained?

deepnet21• 11 years ago

Other networks learn - this network evolved.

By breeding the winners of many self-play tournaments a perfect player was born.

Training and learning show examples and mark the student against a correct answer - the student network tries to minimise their error.

Genetic Algorithms are a type of Reinforcement Learning, i.e. learning by trial and error.

The agent can direct it's own learning and explore.

Useful for training robots and other embodied agents such as oppenents.

discardorama3• 11 years ago

I am aware of all of those terms. But I'd prefer to hear details about how this particular network was evolved, as you put it.

cowpig• 11 years ago

Whoa.... any chance you can write up something on how this works? I've never seen a working implementation of a recurrent net before. I would be very grateful!!

Either way, this is very impressive :)

deepnet21• 11 years ago

The author drops into this reddit thread to answer questions

http://www.reddit.com/r/MachineLearning/comments/31eepl/recu...

fpgaminer• 11 years ago

His blog post describes how it was created: http://blog.otoro.net/2015/03/28/neural-slime-volleyball/

donatj• 11 years ago

Does it continue to learn when I take over control, eg: learn from me, or is the learning done at the time I can play?

deepnet21• 11 years ago

AFAIK the game is not learning when playing a human.

There is a flag to control learning in the javascript trainingMode = false;

bufordsharkley• 11 years ago

Very interesting-- slime volleyball was quite the thing back in my high school days. (See it here if you can get the Java applets to run in 2015 [1])

I was surprised to see a few months ago that the version we played back in high school had the AI programmed by Chris Coyne (of SparkNotes, OKCupid, and now Keybase). I'm curious what was behind the AI in 2004, and furthermore, impressed by Chris Coyne's seeming ubiquity...

1: http://slimegames.eu

hiharryhere• 11 years ago

Massive blast from the past. Slime games were awesome. I spend a ton of time in the highschool library playing volleyball. I think they guy behind the AI in the one I played was an aussie guy called Daniel Wedge. We used to play on his University provided page. Slime Cricket was great too

data_scientist• 11 years ago

Please modify the bindings, the directional keys are international, without a qwerty keyboard I can't play - and no, I will not modify it systemwide for just a JS game.

Nice one nevertheless!

deepnet21• 11 years ago

use the cursor keys for player 2

hardmaru15• 11 years ago

Oops I haven't thought about that!

The arrow keys should work right?

Is there an easy way to detect non-qwerty keyboards so can implement it elegantly in JS?

thomasfoster962• 11 years ago

I'm a little confused - the convnet.js library doesn't do recurrent neural networks, but the recurrent.js library (by the same author) does. Yet convnet.js seems to be being used for this, not recurrent.js.

Still, this is pretty awesome.

hardmaru15• 11 years ago

Hi Thomas

I manually take the outputs of a feed forward network and out them back as inputs in the next time step, kind of making it a fully connected RNN.

Here are some details of the implementation:

http://blog.otoro.net/2015/03/28/neural-slime-volleyball/

thomasfoster962• 11 years ago

Ah, right. That makes sense (even if it's a little tedious).

hardmaru15• 11 years ago

As requested, I put up a demo of neural slime volleyball without any pre-trained agents so you can see them improve gradually as they evolve:

http://otoro.net/slimevolley/training.html

Basically, I left the computer running for a day or so, and came back realising that they have become quite good. But a day of training on a macbook pro is probably equivalent of a _few real decades_ of simulation at 30 frames per second, so to answer someone's question is that evolutionary approach will probably not work well when training with actual human players in human-time. Other approaches such as deep-q-learning may be more suitable. But you gotta wonder about the power and possibilities of simulating time at super human speeds to artificially evolve cool stuff!

sahilwasan• 11 years ago

Hi hardmaru.

Thats great stuff. I always want to learn AI and ML. But do not know where to start. Can you please guide where to start and how can you done this. Thanks

hardmaru15• 11 years ago

Hi Sahilwasan

If you want to learn I found it is best to get your hands dirty and code rather than just watch videos from MOOCs. I didn't know how to do any of this stuff a few months ago.

I recommend Karpathy's Hacker's Guide to Neural Networks if you like JS

http://karpathy.github.io/neuralnets/

Most of my intro to neural nets is just by going thru programming examples in this stanford tutorial on deep learning:

http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutori...

deepnet21• 11 years ago

Mindblowing that the trained agents are 10s of years old - this puts their skill in perspective.

Very interesting that we humans may be too 'realtime' to train evolving agents.

Do your plank avoiding creatures evolve faster ?

http://otoro.net/ml/planks/index.html

Watching the creatures avoiding planks it is hard to reconcile the deterministic mathematics with the seeming agency and 'beingness' of the simulated life.

hardmaru15• 11 years ago

The creatures evolved much faster! If you run the training demo at http://otoro.net/planks/training.html I noticed that after 50-100 generations they develop insect like characteristics.

But for the slime agents, they needed much evolution. I think when I did it I actually had them evolve on the easy version first, and then increased the area size to evolve them on the larger play area. This sort of step by step evolution strategy was covered in some neuroevolution papers I read. I also applied it to the pendulum example.

hardmaru15• 11 years ago

Posted a small write up here with regards to the evolution setup used:

http://blog.otoro.net/2015/05/13/neural-slime-volleyball-evo...

deepnet21• 11 years ago

While Deep-Q-Learning is powerful, it uses backprop - your use of CNE genetic evolution rather than backprop may provide a more global search than Mnih's Deep-Q.

The methods seem complementary , I have an inkling that the combination may be greater than the sum of the parts.

Another innovation in the Atari paper, Mnih's deep-Q-agent remembers successful episodes and trains more from memory than experience.

Of course they are not training against a human player but perhaps this memory training could be used with human games to exploit expert knowledge.

The recent Neural Net Go players by Edinburgh's Clarke & Storkey group and Deepmind's Maddison & Huang both train on corpii of expert human play.

But the Go players are passive backprop learners.

Tesauro's TD-Gammon learned to expert level through self play.

I am speculating that a Reinforcement Learner could learn to utilise the benefits of human play with memory, expert corpii and self-play.

This is a fascinating parallel area - two equal players rather than Mnih's single player Atari - perhaps there are good 2-player Atari games that could be added to the ALE benchmark - Joust ?

deepnet21• 11 years ago

*references

Teaching Deep Convolutional Neural Networks to Play Go by Clarke & Storkey http://arxiv.org/abs/1412.3409

Move Evaluation in Go Using Deep Convolutional Neural Networks by Maddison , Huang , Silver & Sutskever http://arxiv.org/abs/1412.6564

Tesauro's TD-Gammon http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node108.htm...

Playing Atari with Deep Reinforcement Learning , Mnih et al. https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Otoro's CNE Bibliography http://blog.otoro.net/2015/01/27/neuroevolution-algorithms/

Neuro Evolution - http://en.wikipedia.org/wiki/Neuroevolution

technologia• 11 years ago

A blast from the past and I'm still losing to the computer. Nothing has changed.

Show HN: Recurrent neural net plays 'neural slime volleyball' in JavaScript

Related Stories