Why I’m Remaking OpenAI Universe

https://blog.aqnichol.com/2017/06/11/why-im-remaking-openai-universe/

by evc123 • 9 years ago

224 58 9 years ago

blog.aqnichol.com

evc1234• 9 years ago

https://github.com/unixpickle/muniverse

https://github.com/unixpickle/demoverse

dswalter• 9 years ago

I'm a little surprised, but this seems like a good idea. HTML5 certainly has a brighter present and future than flash, and skipping the OCR stem should save quite a few CPU cycles.

misiti3780• 9 years ago

Did openAI really unofficially abandoned universe ?

toisanji• 9 years ago

yes

adewinter• 9 years ago

They switched over to OpenAI Gym which is much broader in scope (able to play Steam based video games).

aerovistae3• 9 years ago

Yeah, really interested in hearing their take on this. It's not often you see a Musk-sponsored enterprise cast a major project aside without public comment.

chronic61a• 9 years ago

It's because the people actually working on AI, including OpenAI, finally knocked some sense into Elon Musk. He finally realized how far behind AI is (it is a glorified linear regression) and we won't be seeing general AI for at least another 40 years.

Source: Am an AI research scientist.

backpropaganda3• 9 years ago

The main reason people in the AI community believe Universe has been abandoned is because the engineers who worked on it have been laid off, and also because none of the promised updates actually materialized. This doesn't preclude the possibility of a fresh non-VNC take in universe with a smaller team of course, perhaps also with more focus on benchmarking (like Atari, Labyrinth) than universality.

zach417• 9 years ago

I echo all of your issues with running Universe. I have a decrepit Macbook, and it was actually not possible for me to use it at all.

forgotmyhnacc• 9 years ago

If you have trouble running universe, how are you going to run RL algorithms that use lots of gpu and CPU?

gdb• 9 years ago

(I work at OpenAI.)

Great project. We've found that the VNC Universe environments are hard for today's RL algorithms primarily due to the their async nature. We're currently working on a new set of Universe environments without VNC; I'm very happy to see others inspired by the core ideas of Universe as well.

Aqueous• 9 years ago

It seems like you might be duplicating work? At the end he mentions he's dropping VNC in favor of headless Chrome.

evc1234• 9 years ago

recruit Alex Nichol (unixpickle).

du_bing• 9 years ago

Oh, Greg, nice to see you here, I am eager to see some solid Universe environments without VNC, that will be interesting.

unixpickle4• 9 years ago

(Author here). Hi Greg! I am excited to hear about the new Universe environments. I want as many RL environments as possible for my upcoming project, so I will probably draw from Universe and ALE as well as µniverse.

I took a lot of inspiration from Universe and am grateful for OpenAI's work on RL in general :). I probably wouldn't have started on this project if a company like OpenAI hadn't already decided it was a worthy goal.

pixelHD2• 9 years ago

honest question, how interested is the academia/industry in deep learning libraries & game engines integrations? I've worked on unreal and tensorflow the last semester, and I found out that there aren't any existing integrations. I will probably work on a plugin, but I wanted to know if there is any interest?

The way I see it, having hooks into the engines themselves helps with what the article talks about - not needing to go through VNCs or other _glue_ to get realtime data. It could potentially send the framebuffers themselves directly from the game/simulation and tie in the actions back to the game/simulation. And using framebuffers is just one direction, we could instead stream the co-ords/the current payoff/etc.

Also, having such plugins would help with the adoption in both directions - games now have an always updating/learning AI (might need a network connection + cloud backend), and researchers can have training/testing environments.

evc1234• 9 years ago

Arthur Juliani is about to open source his interface for connecting ML agents to Unity3D game engine: https://twitter.com/awjuliani/status/879142906178785281

hackpert• 9 years ago

This is great. Using HTML5 games in a headless browser makes a lot of sense because the need for VNC is circumvented. However, I think that while OpenAI's implementation is certainly not the best, having access just the information on the screen is not a bad idea in itself as a (maybe optional) constraint. With access to the game's internal state we don't even need RL for solving a large number of games - algorithms like NEAT are sufficient.

Houshalter6• 9 years ago

This project doesn't change that. The agents still only get screenshots of the game as far as I understand.

However I think this approach is bad. Machine vision is a separate problem from reinforcement learning. You shouldn't need to be able to do both well. Machine vision consumes a ton of processing power and researcher time in figuring out the hyperparameters. And all it's doing is figuring out information that's already in memory like the location of various objects and the score. It really limits what can be done. E.g. the famous atari playing AIs by deepmind were limited to no memory and only knowing the last few frames, because backpropagating through thousands of frames was too expensive.

Because of the way NNs work, it's trivial to separate out the machine vision into a separate module. So if you have a good RNN reinforcement learning system, you can easily add a machine vision learning system to it later if you need.

unixpickle4• 9 years ago

In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401.

strin• 9 years ago

Awesome project.

Despite the flaws, the nice thing with VNC is its universality to support any apps on a computer. Using HTML5 in a browser limits the scope of things we could encapsulate as environments, and makes it less "universe".

However, there is a difference between the universality of the tech stack and the exposed interface. In my opinion, the future universe would be rich clusters of RL environments with unified API, each of which implemented using different underlying technology to meet the desired synchronicity and frame performance.

HTML5 could deliver one of such clusters.

unixpickle4• 9 years ago

I'm pretty sure that was the goal of OpenAI Gym. Gym tries to provide a generic interface for RL environments, and imho it does a nice job. I am working on Python bindings for µniverse now, which should allow µniverse to integrate with Gym.

tomjacobs• 9 years ago

Missed opportunity for a Rick and Morty Microverse reference here as the name

Houshalter6• 9 years ago

Why not use game emulators? With popular NES emulators you can advance the game frame by frame. You can read the raw memory addresses that correspond to the score. You can dump the memory at any time and reload the game to a specific game state. You can even manipulate the games in many fun ways by messing around with the game memory. Or give an AI algorithm access to memory addresses as additional information, instead of relying on pure machine vision, if you want to do that..

Here's an example of a guy who made a general game playing algorithm that brute forces it's way through any NES game: https://www.youtube.com/watch?v=xOCurBYI_gY This isn't necessarily interesting from an AI perspective - the playing algorithm is just brute force. But it shows what can be done with the platform, easily reloading to previous states and exploring counterfactual futures (which is exactly the sort of thing RL algorithms do.) He also has a cool algorithm for finding the objective function of an arbitrary game, by watching a human play, and seeing what memory addresses increment. Which is a lot more easy to use than writing OCR code to read the score and game over states from the screen.

zzh8829• 9 years ago

I am also working on related project. Flash and HTML5 games in chrome are great but they are very far away from the initially promised full blown GTA5, Starcraft and other complex envs. I am in process of remaking the Universe framework for host machine, since running those computation intensive games at reasonable frame is nearly impossible inside docker or virtual machines.

namuol• 9 years ago

Funny, I have an old (unfinished) HTML5 space-exploration game by the same name:

https://github.com/namuol/muniverse

If I had more time I'd submit a PR to integrate it...

Cellestro• 9 years ago

Congratulations on the initiative, it looks very cool! Indeed, we found that running asynchronous environments, while possible, proved to be too cumbersome for research. We're now working on a synchronous set of environments for universe that are easier to use.

daveguy• 9 years ago

According to the author, "Universe never really took off in the AI world."

That's a bit premature for a project that was just released less than 7 months ago, isn't it?

https://blog.openai.com/universe/

Edit: that said the project seems to have some interesting and needed improvements (esp time adjustment). Glad to see dialog between muniverse and openai here.

make32• 9 years ago

I wonder what's happening with OpenAI. Most big names are leaving.

fggh• 9 years ago

Please elaborate...

make32• 9 years ago

Well, Ian Goodfellow and Andrej Karpathy for starters

Why I’m Remaking OpenAI Universe

Related Stories