Show HN: AI QLearning Robot Simulation in JavaScript and WebGL

http://qlearning.4ck5.com/

by javierluraschi • 11 years ago

42 12 11 years ago

qlearning.4ck5.com

hmate92• 11 years ago

Looks very nice, I have one question:

How does R2D2 "see" the blocks? Is it by pixels or is the game hard-coded in this sense?

belltyler• 11 years ago

It seems like board is kept as a grid and merely represented with three.js; look into the page's source and you'll see a GameBoard object as well as its logic.

javierluraschi3• 11 years ago

Currently, it sees a board of 3x3 states, as in:

_ _ G

G B B

_ _ G

A next step would be to use real computer vision over a 3D camera in the environment aligned to R2D@ and a deep neural network to interpret pixels instead of states. DeepMind has a paper describing this approach: http://arxiv.org/pdf/1312.5602.pdf

dlsym• 11 years ago

Food / Poison break even was at 88. A chart would be interesting.

However - since R2D2 seems like a slow learner it would be nice to speed up the time a little bit.

hmate92• 11 years ago

I am currently at 175 vs 75

balazsdavid987• 11 years ago

After ~30mins I got ratio of 0.77 (375 greens, 111 blacks), it got pretty good at avoiding blacks after reaching 0.7.

doorhammer• 11 years ago

Warning: I only glanced through the code real quickly, so I haven't checked to see if this has unintended side effects to the game logic, or to make sure it's a complete solution.

It looks like there's a variable that controls how quickly the board itself is updated:

  var boardGameUpdateInterval = 10;

And a function deciding how quickly the visual representation gets updated:

  var render = function () {
      setTimeout( function() {          
          requestAnimationFrame(render);
      }, 1000 / 15);

      if (gameTicks % boardGameUpdateInterval == 0) {
          updateBoard();
      }
  
      animateBoard();
  
      gameTicks++;
      renderer.render(scene, camera);
  };

They're global variables so it's pretty easy to just drop into the developer console and overwrite them.

I just changed the denominator of the setTimeout to 30 instead of 15, and the boardGameUpdateInterval to 5 instead of 10. Of course, you don't have to change the render speed to get the logic speedup, but I figured I'd do both.

This appears to have sped the simulation speed up by a factor of two.

ymmv

javierluraschi3• 11 years ago

I've gone as far as ~1000 food ~100 poisson blocks to make sure it's on the right track. Speeding this up is a good idea, I'll make some time to add this and post an update. You can ping me on twitter @javierluraschi.

akyu• 11 years ago

Very Cool.

I did a similar project using a Neural Network instead of a look up table, similar to how deepmind did their Atari game system. The neural network adds some overhead to the training time because of backprop, so for small state spaces which the table can handle easily it is a bit overkill. But the neural network can scale up to much more complex state spaces that would be impractical to use a table for.

javierluraschi3• 11 years ago

Do you have a library for your project? It sounds really interesting. I think this space is interesting enough that would benefit from a GutHub project to collaborate.

nrox2• 11 years ago

It looks based on this repo: https://github.com/nrox/q-learning.js and this example: http://nrox.github.io/q-learning.js/test2.html

nrox2• 11 years ago

that makes me proud :) do you have suggestions to improve this library ?

Show HN: AI QLearning Robot Simulation in JavaScript and WebGL

Related Stories