Deep Reinforcement Learning

by dcre • 10 years ago

146 29 10 years ago

awwaiid• 10 years ago

"Previous attempts to combine RL with neural networks had largely failed due to unstable learning. To address these instabilities, our Deep Q-Networks (DQN) algorithm stores all of the agent's experiences and then randomly samples and replays these experiences to provide diverse and decorrelated training data."

... so, made the machines dream. Fancy!

ehsanu1• 10 years ago

Aha, so that's what (human) dreaming is for.

Houshalter2• 10 years ago

IIRC there is evidence rats replay their experiences, sped up, while they are asleep. Dreaming may be something else entirely though, because my dreams aren't anything like my memories of the day before.

Artificial neural networks can "dream" by predicting what frame it will see next. This is a really cool technique. They've shown slightly blurry videos of atari games being played that comes entirely from the network's dream. With no interaction with the game at all. You can even train the reinforcement learning on the dream sequences and improve it's performance.

But this also doesn't seem quite like what human dreams are. Human dreams are wild and unrealistic, while the NN dreams try to match the training data as closely as possible.

visarga• 10 years ago

I thought it was for regularization.

justifier• 10 years ago

i've dreamt up a few possible explanations for dreams

currently my favourite is for converting short term to long term memory

my most vivid dreams are when i experience something new, or have an unexpected reminder about something i last thought about years ago

i imagine the surreal nature of the dreamscape is some consequence of developing neuronal paths

storing a new idea about a boat forms a path to a neuronal pathway referencing fire and your dream has you sailing across a sea of fire

aivosha• 10 years ago

i have read somewhere the dreams are for training purposes and present the dreamer situations either from the past or present or combination. that would explain why dreams are mostly dangerous or "wild". this is how our mind tries to figure out how to deal with those situations in the future. good examples of this are same recurring dreams over and over. and also i think there is no sense of time in the brain while dreaming so it does not differentiate childhood "situations" from any current ones. It just keeps fucking trying to solve that damn math exam problem....

cosmoharrigan2• 10 years ago

For historical background on this part of the algorithm, called "experience replay", see this paper from Long-Ji Lin in 1992:

Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching

http://link.springer.com/article/10.1023/A:1022628806385

as well as his excellent 1993 PhD thesis:

Reinforcement Learning for Robots Using Neural Networks

http://www.dtic.mil/dtic/tr/fulltext/u2/a261434.pdf

syngrog66• 10 years ago

next up: make them dream of electric sheep

mtgx• 10 years ago

Let's hope Google doesn't feed it video games like Battlefield, where it learns how to most effectively kill humans.

jamessb• 10 years ago

The comparison to dreaming reminds me of a comment in Information Theory, Inference, and Learning Algorithms:

"One way of viewing the two terms in the gradient (43.9) is as 'waking' and 'sleeping' rules. While the network is 'awake', it measures the correlation between x_i and x_j in the real world, and weights are increased in proportion. While the network is 'asleep', it 'dreams' about the world using the generative model (43.4), and measures the correlations between x_i and x_j in the model world; these correlations determine a proportional decrease in the weights. If the second-order correlations in the dream world match the correlations in the real world, then the two terms balance and the weights do not change."

tintor• 10 years ago

Labyrinth? I have a feeling that Doom is next.

shogunmike• 10 years ago

You might find the VizDoom project interesting: http://vizdoom.cs.put.edu.pl/

seanwilson3• 10 years ago

Labyrinth? I have a feeling that Doom is next.

This would be really interesting to see. I'm curious if Google would avoid this game though since that the media would likely have a field day reporting about a murderous Google developed AI if they did this. Technically we already have this in games anyway though (e.g. bots in FPS games).

seanwilson3• 10 years ago

When it's playing a game (e.g. breakout) and it's being fed the pixels on the screen, how is the AI being told what the score/progress is? Does it have access to some numeric metric that is chosen by the researchers for each game?

sanxiyn• 10 years ago

Yes.

For example, Breakout saves score in address 76 and 77. Arcade Learning Environment has code to read the score, one per game. Code for Breakout is here: https://github.com/mgbellemare/Arcade-Learning-Environment/b...

seanwilson3• 10 years ago

Thanks for that! So I was looking at the one for Montezuma's Revenge: https://github.com/mgbellemare/Arcade-Learning-Environment/b...

The reward seems to be only the game score which I believe is meant to be problematic for this game because your score doesn't go up very often (so you have to perform a lot of actions to get any feedback)? The lives are recorded but aren't part of the "get reward" method...are the lives factored into decision making somewhere else? Seems like knowing you just lost a life would really help decision making for such a game.

msohcw• 10 years ago

Yes, it's fed the score as the reward value used. If I'm not wrong, they didn't normalise it across games for the initial paper but normalised it to some range for some of the following research experiments.

cosmoharrigan2• 10 years ago

The original papers [1][2] for the deep Q-network used reward clipping: "As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged".

[1] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

[2] http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...

Dolores12• 10 years ago

They can measure time you played. More time in game mean higher score.

Deep Reinforcement Learning

Related Stories