Learning a hierarchy

https://blog.openai.com/learning-a-hierarchy/

by gdb • 9 years ago

267 39 9 years ago

blog.openai.com

anon4041234• 9 years ago

super cool that this was done by a high schooler

akhilcacharya4• 9 years ago

More discouraging to me to be completely honest.

fjsolwmv2• 9 years ago

Why have a whole humanity if you only think a single best person has value?

anon4041234• 9 years ago

no reason to be discouraged. plenty of good times to go around. AI is wide open and there's plenty of basic discoveries to be had by those willing to look.

Furthermore, outside of AI there is so much fun to be had in the world that it's probably not worth being discouraged by discoveries made by some preternatural high schooler. https://xkcd.com/1024/

fjsolwmv2• 9 years ago

? The blog post has 5 authors

anon4041234• 9 years ago

first author on the paper is a high school student

tejohnso• 9 years ago

Article about him in Wired magazine:

https://www.wired.com/story/meet-the-high-schooler-shaking-u...

zardo2• 9 years ago

I was mulling over this idea yesterday in the context of RTS games... There's no reason to consider changing your overall strategy every frame. Nice to see it works!

It will be interesting to see how it performs with more tiers in the hierarchy, and with more structured tasks.

Controlling a virtual arm to play a board game for example.

sputknick2• 9 years ago

I don't understand where the 'hierarchy' comes into play? This reads to me as a standard computer program where you execute code, and some of those lines execute other segments of code which might be much more complex than what I see. If I execute the line 'printline('Hello World')' I only excuted one line, but many other things happened that I did not directly execute. I'm sure I'm missing something, and this is somehow different and novel, but I'm just missing it from this blog post.

zardo2• 9 years ago

It is effectively a system of reinforcement learning agents working in a command hierarchy to solve problems that single reinforcement learning agents fail to.

It's (somewhat)obvious that this is an idea worth trying. But that doesn't mean actually getting it to work is easy.

sputknick2• 9 years ago

Got it, okay, so it is different from a traditional computer program, and more like a business or military unit, where the agent at a high level "determines" an action, and delegates the action to a lower level entity that doesn't necessarily have the knowledge as to why it's doing this thing?

gthinkin• 9 years ago

Great work, Kevin!

kevinfrans• 9 years ago

indescions_2017• 9 years ago

Next step: transfer learning and sharing amongst sub-policies in the graph hierarchy. If an Ant Agent learns to "move up" to avoid obstacle or reach goal. Why can't it infer the same for any cardinal or diagonal direction, after observing the world around it. It's just a rotation or translation after all.

Also, for small numbers of sub-policies, would Monte Carlo playouts be faster. Where we are searching over the next step the Any may encounter. Which presumably is a finite set of possible "wall-floor" configurations ;)

In any case, great work! Always love watching OpenAI vids...

derefr• 9 years ago

It's just a rotation or translation after all.

My intuition from working on various computer vision tasks, is that animal brains would do this more by rotating the perspective at the post-optic synapses, rather than having a generalized plan. We still only know how to "move up"; we just change the angle we're understanding the scene from, and so change what "up" means.

jng• 9 years ago

Well, it's really hard to read text upside down.

fiddlerwoaroof• 9 years ago

I’ve heard this many times, but I’ve never had any issues reading text at any angle.

hacker_92• 9 years ago

Does this optimise the hierarchy as the environment changes? For example when cooking, I unpackage food as needed, but when it starts to clutter the workspace I make a decision to fit in a 'clean up cycle' while waiting on some other food to cook.

sharemywin2• 9 years ago

As far as I understood it, it learned sub-tasks then learned to apply those sub tasks.

Kind of reminds me of the Soar system except using Deep learning instead.

https://en.wikipedia.org/wiki/Soar_(cognitive_architecture)

canjobear• 9 years ago

It seems to me there's been an interesting turn in AI recently, toward focusing on adaptability as a goal in itself. Deep learning has shown that there is incredible power in stochastic gradient descent over a space of functions, but so far that has mostly been applied to rigid tasks. Now work like this is about turning that power towards adaptability itself as a goal, and it seems to me that this brings us towards "real" intelligence.

The logical extreme of this thinking would be agents that actually maximize entropy of future actions as the only objective function, like in [1]

[1] http://paulispace.com/intelligence/2017/07/06/maxent.html

hacker_92• 9 years ago

Structures that can query and adapt their own structure. Reminds me of reflection in a managed language.

zan2434• 9 years ago

Related article on similar hierarchical / compositional policies learned by maximum entropy optimization: http://bair.berkeley.edu/blog/2017/10/06/soft-q-learning/

ionforce• 9 years ago

Is this like maximizing for movement options in a Chess AI?

Retra• 9 years ago

This reminds me of a thought I had some years ago. The idea was that we can think of general intelligence not as an optimization for a specific given goal, but as optimization for a special position from which some wider set of goals can be most rapidly converged upon. Thus optimizing for future flexibility rather than current results.

At the time, I remember being excited to hear of a physics paper[1] concerning an inverted pendulum, where they solved the system for some dynamic forces which would keep the system at the position of maximum instability, and claimed that it was, in some sense, a description of dynamic intelligence. The analogy there is that this is the unique position from which the pendulum can be efficiently made to move quickly in any 'required' direction (the 'goal'.)

I still think that idea has some merit, but putting together a coherent formalization of it seems really tricky and requiring some genius far beyond my own meager pondering.

[1] I found the article: https://physics.aps.org/articles/v6/46

sixdimensional• 9 years ago

A little bit philosophical and not scientific, but I found this blog post [1] to be good for musing about the relationship of "intelligence" and "adaptability".

[1] http://www.actinginbalance.com/intelligence-is-adaptability/

sharemywin2• 9 years ago

Found the paper from the wired article below

https://s3-us-west-2.amazonaws.com/openai-assets/MLSH/mlsh_p...

ohitsdom• 9 years ago

There are buttons below the first video to read the paper and view the code.

setr• 9 years ago

Is it just me or is there something revolting about the character model?

Good work nonetheless but for god's sake give it six legs and make it black

Learning a hierarchy

Related Stories