MIRA: Multiplayer Interactive World Models Trained on Rocket League

by ethanlipson • 14 days ago

99 31 14 days ago

vvolhejn2• 15 days ago

Václav here from the team, we're happy to answer questions :) The most surprising part to me is the auto-recovery behavior we mention at the end of the blog post, since any other model I've seen always stays diverged once it goes off the rails once. But MIRA really doesn't like to be out-of-distribution. To be completely honest we're not entirely sure why this happens.

shay_ker• 14 days ago

Great work! My question is about what you mentioned at the end - how well do world models operate when out of distribution? In some sense we hope these models learn something "deeper" about how the world works and can apply that knowledge to different tasks.

I saw lots of awesome ablations in the paper (loved it!), but I'm curious if you analyzed the latents to get an intuition for what the model actually learned. Or, is it just that it learned the training data distribution really, really well?

blackfireWM• 5 days ago

I'm insanely impressed with MIRA! I mean WOW, I've been doing a lot of diffusion world model experiments and I had thought of trying stuff like "visual anchors" to support some form of long form continuity but you guys really blew the field out of the water. My intuition(i've still yet to fully read the research paper i apologize) is that feeding in 4 highly dynamic(yet correlated, due to focusing on the ball) perspectives for each frame is what gives the incredible coherence. As said in the blog and paper, essentially infinite rollout. Thank you, and all of the team, for making your work public! I'm not above begging for an internship just to be clear https://github.com/blackfirebitcoin/hyrule-dreamer-wm

_willmanning• 15 days ago

it would have been easier to just go to FNAC and buy Rocket League like a normal person :)

cataPhil• 15 days ago

you should do that too! the goal is not to replace the game but to foster research on these method, and hopefully apply them to data-constrained settings like robotics

danking00• 15 days ago

Wow! At first, I expected this to be a demonstration of an AI playing rocket league, but I rapidly realized this is actually a model simulating rocket league. Wild! It feels just like the real game.

exortaz• 15 days ago

this is insane - what’s your thinking on how this improves model grounding and efficiency vs single pov outputs?

vvolhejn2• 15 days ago

A lot actually, since the model has all information given to it in the four views, it doesn't have to deal with any "theory of mind" of modeling the other players or being consistent over long times. See [1], there's a video of a single-player model where a car disappears behind a ball and never reappears. Multiplayer has its own risks such as the four views desynchronizing, but overall it gives a big boost to the model.

[1] https://mira-wm.com/blog-post/#hidden-information

LorenDB• 15 days ago

Is this now the easiest way to play Rocket League on Linux?

MasterScrat5• 15 days ago

The checkpoint also weights half less than the game install! ;-)

ggarnhart• 15 days ago

Nice to know my inability to play Rocket League with any level of skill carries over to this world model

MasterScrat5• 15 days ago

We're happy to release MIRA, a collaboration between General Intuition, Kyutai, and Epic Games.

Mira was trained on 10k hours of Rocket League data. The model has 5B parameters and runs 4-player games at 20 fps on a single B200 GPU.

We've released a playable online demo, an in-depth technical report as well as a 1k hour dataset of 4-players gameplay:

Technical report: https://mira-wm.com/paper Repo: https://github.com/mira-wm/mira

lostmsu• 15 days ago

How much compute did it take to train the model?

jorl17• 14 days ago

This was a much better experience than I expected. Rather unbelievable!

Side-effect of the data: clearly the model is better than I normally am at playing, as it spontaneously did several things I had not told it to do and wouldn't really know how to do (at least not with a keyboard).

Really remarkable, congrats!

superkuh2• 14 days ago

It feels like playing on a very slow computer. Except that sometimes it just randomly decides you pressed the flip button. Really impressive.

superkuh2• 13 days ago

I played again but this time I drove my car backwards and the model does much, much worse in this weird situation. It often doesn't do what I input at all and keeps straightening itself out. And on kick-off you always go for it, even if you don't.

MasterScrat5• 12 days ago

Playing backward is actually a great idea for testing generalization

MasterScrat5• 14 days ago

Hey all, happy to see this here! This was a colab between General Intuition (that I’m part of), Kyutai and Epic Games.

You can read plenty of details in the blog post and tech report but the TLDR is that we trained a multiplayer world model on 10k hours of Rocket League data. We optimized it to be playable at 20fps on a single GPU.

So what you see in the demo is fully generated: there’s no graphics or physics engine. Instead it’s a 5b neural network that takes actions in and gives pixels out.

pizzathyme• 14 days ago

Tim Sweeney’s interviews on the uses of GenAI for game development have been some of the best takes I’ve heard. He’s mentioned how GenAI is great at filling in the gaps or treating assets, but no world simulation means no deep persistence or authoring for a whole new unique game world.

What is the conversation like within Epic now? Is this still the view? What is the future for simulations like this?

pvillano• 14 days ago

Could a network be trained to transform physics state directly into the latent state and back?

Having a direct transformation would enable some interesting experiments.

How is the latent state different when everything else stays the same, but you change one physics value, like player one velocity? Is there a cyclical pattern of activation that correlates strongly with the seconds digit of the clock? Can you decode the latent state, give players full boost, and then re-encode it for infinite boost, without losing continuity?

Edit: There sure are a lot of papers on interpretability.

MasterScrat5• 14 days ago

Would be a great idea to see how much we could manipulate the latent space and whether it has some internal structure w.r.t the physical state. I guess the only unknown is how the world model would show robustness to latent states that are transformed through this network

sliding-penguin• 14 days ago

Very cool, and publishing a slice of the dataset and all of the training code is fantastic, but if reproducing the model and the video representation codec is encouraged, why not open source the models or at least some variant of them?

I'd be interested in seeing if fine-tunes that include human gameplay data would be possible.

bschwindHN• 14 days ago

Where is the option to call all of my tm8s trash? That's an essential part of the experience!

MitziMoto• 14 days ago

What a save! What a save! What a save! (Chat disabled for 3s)

coip• 14 days ago

Calculated.

skibz• 14 days ago

Nice shot!

avaer• 14 days ago

If the data and code is all there, why not release the 5B weights?

in-silico• 14 days ago

I feel like the data should have been generated by a much less predictable policy.

It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what would happen next during training without paying attention to the inputs)

amarant• 14 days ago

Pretty insane that you can get this close to the real thing this way.

Rocket league is one of my favourite games, and I'm pretty decent at it (rank champion 1). I kinda felt like my controller was a bit broken when playing this, a lot of commands were just ignored, and forget doing stuff like speed flips. But I did feel like was controlling the car, and everything about the game looked very much like the real thing. Ball movement was on point, I didn't notice any weird bounces or anything.

The lack of opponents pulling triple flip resets and double-tapping musty's (musties?) was the most notable difference from the real thing

RugnirViking• 14 days ago

The demo button, and most of the features mentioned, dont seem to work for me, on edge or chrome. Project sounds really interesting, so I wish I could try!

MIRA: Multiplayer Interactive World Models Trained on Rocket League

Related Stories