Prefrontal cortex as a meta-reinforcement learning system

https://deepmind.com/blog/prefrontal-cortex-meta-reinforcement-learning-system/

by godelmachine • 8 years ago

243 29 8 years ago

deepmind.com

spitfire• 8 years ago

Paper is behind a paywall. Sci-Hub doesn't have it. Does anyone have a link to the paper PDF?

dcre2• 8 years ago

https://www.biorxiv.org/content/early/2018/04/13/295964

(I searched it on Google Scholar.)

ColanR• 8 years ago

http://sci-hub.tw/10.1038/s41593-018-0147-8

gumby• 8 years ago

Scihub link: https://sci-hub.tw/10.1038/s41593-018-0147-8

mindcrime2• 8 years ago

So here's something interesting to note. The "buy this paper" price for this article is almost reasonable in and of itself. $22.00. In fact, for somebody who works in the field or even a tangentially connected field, they could probably justify spending the $22.00 for it.

But... hilariously, a one year subscription to the journal, is only $59.00 (for individuals, online only).

https://www.nature.com/neuro/subscribe

Something seems a bit out of whack when the price of one article is about 1/3rd of the price of the journal for an entire yet.

I'm almost tempted to subscribe. The problem with this model though, is that it doesn't scale. Even at "only" $59.00, how many journals can one afford to subscribe to before the aggregate cost breaks the bank? sigh

keithwhor2• 8 years ago

It scales because nobody subscribes individually. If you can't access a journal, you get your department head to forward things up the chain so your institution can just add another line item to the millions of dollars they spend annually on journal subscriptions for every researcher and student.

AFAIK almost all of that money goes directly to Elsevier [1] who, well, they pretty much run a monopoly.

[1] https://en.wikipedia.org/wiki/Elsevier

taneq• 8 years ago

It's priced that way precisely because they want you to go "uh, but chances are I'll want to read another paper or two so I might as well get a subscription", even though you probably won't find two more papers that aren't available elsewhere that you want to read enough to pay $22 for.

kriro• 8 years ago

This is very likely not an accident but by design. For a good popsci read on similar topics I can recommend "Predictably Irrational, Revised: The Hidden Forces That Shape Our Decisions".

dcre2• 8 years ago

Full preprint available here:

https://www.biorxiv.org/content/early/2018/04/13/295964

cs7022• 8 years ago

Very interesting.

Oversimplifying and ignoring a lot of important details, the key idea proposed by the authors is that the brain's phasic dopamine system is a model-free reinforcement-learning system that learns to train the prefrontal cortex as a more efficient model-based reinforcement-learning sytem -- a form of meta-learning which the authors accurately refer to as meta-reinforcement learning.

The "Results" section provides compelling evidence that the authors might be on to something. The authors show and discuss the outcomes of six different kinds of computer experiments in which a (relatively simple) meta-reinforcement learning software system is shown to learn and behave in qualitatively similar ways as, for example, monkeys and rats in equivalent lab experiments.

I'm still digesting the implications.

Highly recommended reading.

epicureanideal• 8 years ago

Although the behavior of the monkeys and rats was simply selecting the correct image to receive a reward, so not an especially complicated task.

red75prime• 8 years ago

Well, it took 50+ years to solve this simple problem somewhat reliably.

dlwdlw• 8 years ago

To simplify even further, would it be right to say that dopamine is not just a carrot being used to adjust weights but that the entire dopamine system actively learns how/when/why to use the carrot to effectively train?

if so, some interesting real world parallels i can see:

1/ this is why good teachers early on have such a profound effect. Teachers act as a dopamine system so good teachers teach meta learning via whatever they teach

2/ teaching the meta directly is difficult as the meta of the meta is too abstract. Which is why grammar/math may tend to feel out of touch. They are already the meta.

3/ drugs can mess up the dopamine system and throw the feedback loop out of whack. Even if you have the best resources, the “teacher” is now inept

epberry• 8 years ago

One implication for products, if this work pans out, is that the systems which "adapt to you" (learn your voice, your schedule, what your face looks like) won't have to change the weights of their network. I believe this could lead to much better systems because adjusting the weights of an already-deployed network is dangerous - you could make the system perform poorly. It's also simpler for engineers because you can scrap all that logic and just deploy one RL network.

alex_duf2• 8 years ago

Sorry my head just exploded, but are you suggesting we have personal "prefrontal cortexes" used to customise general neural networks?

brockf• 8 years ago

No reason to oversimplify and ignore a lot of important details — they're doing that for you!

cs7022• 8 years ago

The original link was to the actual paper[a], not the blog post!

A moderator, dang, changed the link after I posted my comment; see his comment [b] elsewhere in this thread.

Had the original link been to the blog post, I agree, there would have been no need to simplify.

[a] https://www.nature.com/articles/s41593-018-0147-8

[b] https://news.ycombinator.com/item?id=17068449

yonoataraxia2• 8 years ago

Somewhat interesting: Bananas contain the ingredients necessary to produce dopamine.

I can see how this all plays out as a way for some apes to get some bananas. Even social structures, fairness and other things (I love this video - two monkeys getting unequal pay https://www.youtube.com/watch?v=meiU6TxysCg ) can be explained by a dopamine system that just likes to give us bananas and sex. And it's incentivized to conserve energy because this is better for survival (also called lazy) - interesting is that humans have some tendency to do stuff that don't necessarily is conserving energy. Maybe this is what makes us successful: Sometimes we invest excessive energy to try new things which happens to let us survive better through innovation (which leads to genetics that encode this behavior).

I like it, although it isn't necessarily what I call my meaning of life (I'm a dualist, because materialism is too damn dry). I like bananas, though (and yes, the role of bananas may be exaggerated here).

Thought long about it. From a materialistic standpoint there are only quantitative differences between a organism which has a sensor, some type of memory and an actuator and human beings, although they sure look different. Still following the same principles.

When I further simplify, we are all just energy (and matter, which is just a form of it) following some first principles hallucinating our consciousnesses trying to evade the eventual entropy that we'll reach nevertheless because this is how the universe works.

- - -

Cool fact is that animals are capable of rational thinking (crows drop nuts to break them approx. at the minimal height necessary to achieve that - optimal energy usage. I'm pretty sure they don't even realize that their brains calculate this based on their experience) and I'm sure that all people act rational w.r.t their training data (some outliers like traumatic events and other life circumstances that differ from the average just change some weights in a way that it looks irrational for outsiders). This is indeed difficult to defend because it depends on the semantics on the word "irrational" which is man-made after all.

Interesting resources for this topic:

a theory trying to explain human behavior and the emergence of consciousness using knowledge of psychology: https://unifiedtheoryofpsychology.files.wordpress.com/2011/1... (principles how the brain works)

https://unifiedtheoryofpsychology.files.wordpress.com/2011/1... (why culture and religion emerges)

https://www.youtube.com/watch?v=lyu7v7nWzfo (consciousness as a hallucination)

https://www.youtube.com/watch?v=HRVGA9zxXzk (a bird which can identify itself in a mirror, simple self-awareness - I like the fact that the brains of birds are more efficient because they are space-constrained to be able to fly better)

(the brain as a neural net with meta-learning capabilities that tries to guess what happens next and what it should do next. Emotions and some pre-wiring based on genetics enables us that we don't start with a complete random brain structure because it gives us better surviving abilities if we're able to see and feel as soon as we get out of our mothers).

sudomake• 8 years ago

anything that contains the amino acids tyrosine or phenylalanine has the ingredients for dopamine

mr_overalls• 8 years ago

A more accessible discussion by the paper's authors is here at DeepMind:

https://deepmind.com/blog/prefrontal-cortex-meta-reinforceme...

vvkmnn• 8 years ago

Thanks!

dang• 8 years ago

We've changed to that from https://www.nature.com/articles/s41593-018-0147-8. Thanks!

Prefrontal cortex as a meta-reinforcement learning system

Related Stories