Mastering Real-Time Strategy Games with Deep RL: Mere Mortal Edition

https://clemenswinter.com/2021/03/24/mastering-real-time-strategy-games-with-deep-reinforcement-learning-mere-mortal-edition/

by cwinter • 5 years ago

129 66 5 years ago

clemenswinter.com

FartyMcFarter2• 5 years ago

This trend has culminated in the defeat of top human players in the complex real-time strategy (RTS) games of DoTA 2 [1] and StarCraft II [2] in 2019.

Not quite:

- OpenAI's DoTA 2 system wasn't playing the full game. I think the final version could play 17 of the 117 heroes, and the opposing human players were also restricted to playing this subset of the game.

- DeepMind's StarCraft II system reached a level above "above 99.8% of officially ranked human players.", so it isn't trivial to argue that this amounts to defeating top players.

exdsq• 5 years ago

The difference between the top 0.2% and top 500 players is huge too

porphyra2• 5 years ago

At Blizzcon 2019, Alphastar beat Serral, undeniably a top player (although he didn't get to use his own keyboard and settings or get to prepare). Serral was able to beat the Terran agent though.

https://www.youtube.com/watch?v=nbiVbd_CEIA

ZephyrBlu8• 5 years ago

People also cheesed the shit out the bot and won though. None of these AIs have proved to be robust to exploitation yet.

TulliusCicero6• 5 years ago

IIRC each individual agent is essentially built around a single strategy/playstyle, not to mention the agents relied heavily on mechanical advantages to win.

AlphaStar basically got to 'good enough' strategy, then won on control, which computers obviously have a massive advantage with.

rollcat• 5 years ago

In StarCraft 2, if you're used to playing with specific settings (esp. graphics, keybindings, mouse speed), having to revert to standard is a huge handicap. There are also OS settings (keyboard delay and repeat rate) that if left at default basically make a "standard" game unplayable, especially for Zerg.

kevinwang2• 5 years ago

< OpenAI's DoTA 2 system wasn't playing the full game. I think the final version could play 17 of the 117 heroes, and the opposing human players were also restricted to playing this subset of the game.

The bigger issue in my eyes was that while OpenAI 5 defeated the world champion team OG, when they let anyone in the world fight it, some ingenious players figured out a pretty robust method to consistently exploit and defeat the bot. As I haven't heard any buzz about OpenAI 5 since then, I think it was more or less unsuccessful unless they can show that their training method produces unexploitable bots (instead of bots that are really good against certain strategies)

Gunax• 5 years ago

They can train the bots based on on those games though, right? Seems more like a flaw in the training data than the principle.

I am not sure if the training is done live or not--that is does the algorithm learn based on each game against a real, live player? Or do they just train the model offline, then allow players to play against the static model?

taberiand• 5 years ago

I think this is kind of pedantic. They built an AI agent to take pixel data as input, and provide mouse movements and clicks as output, and rather than just flail around like a baby it actually played the games with a sophisticated competency. This to me is such an incredible achievement that I have no doubt that it could be enhanced to defeat top players consistently and easily.

As another commenter remarks, there are holes to plug in terms of exploitable behaviours that are locked into the model, but this too I'm confident they will find a general method of preventing; on the other hand, it's not like humans aren't susceptible to similar exploits by competitors in situations where they decide to cease innovation/learning

ZephyrBlu8• 5 years ago

As another commenter remarks, there are holes to plug in terms of exploitable behaviours that are locked into the model, but this too I'm confident they will find a general method of preventing

The problem is, I don't think there is a "general method of [prevention]" because that's not how neural networks work.

It's not easy to fix things like this because you can't just say "yeah just don't do that dumb thing anymore", the network has to be re-trained to learn the exploit.

The way DeepMind tried to get around this is by having a league of AIs playing against each other which try to exploit each other and expose their weaknesses. It worked pretty damn well, but people still found ways to exploit the AI.

TaupeRanger• 5 years ago

Not at all. It is a computer, of course it beats humans at optimization problems and speed. Not remotely surprising or interesting, any more than a calculator doing arithmetic faster than a human.

FartyMcFarter2• 5 years ago

I don't think either system was trained from pixels, but I agree it's impressive nonetheless.

However, it doesn't follow that it's easy to extend to beating top players consistently. If that was the case, it probably would have been done.

wnevets5• 5 years ago

OpenAI's DoTA 2 system wasn't playing the full game. I think the final version could play 17 of the 117 heroes

Limiting the number of playable heros in DoTA2 really isn't important when it comes to evaluating the skill of the AI. Most real players trying hard to win already play with a limited hero pool dictated by the current patch verison.

alach11• 5 years ago

It is important. They removed a lot of champions with complicated mechanics that could have been much harder for the AI to play against.

TulliusCicero6• 5 years ago

AlphaStar wasn't beating top pros, though. And even the players it was beating, it was with a heavy mechanical advantage -- not just on strategy and tactics.

That AI's can beat human players via superior interface control is obvious, of course, and uninteresting from an AI standpoint. Starcraft has had AI's with perfect roach/stalker/marine/etc control for a while. The problem was that the overall strategies weren't good enough.

AlphaStar did make massive improvements there, to be sure, very impressive ones. But it still relied on out-controlling human players to get the edge against pros.

tialaramex• 5 years ago

A legacy of the AlphaStar work is ongoing amateur SC2 AI play. If you're comfortable writing software in, say, Python, and can play SC2 at least a little, you can see whether the reason you aren't a world famous player is just that you were too slow or whether your strategy actually isn't that great even if executed perfectly :)

https://sc2ai.net/competitions/3/

Unlike Alphastar, these AIs are intended primarily to play each other because as you say inhuman perfection in execution is not an interesting difference. This means it makes sense for them to exploit behaviour in the game itself that would be inaccessible to humans (e.g. "speed mining" by individually controlling every worker) as well as executing ludicrously multi-pronged mid-game attacks since they can just as easily manage six individual small battles as one larger frontal assault.

That site links a Twitch channel which automatically plays random games between bots with auto-camera, but if you prefer human commentators (and as a bonus, speeding up the period when the game is clearly lost but bots rudely never resign since it's not as though politeness scores points) there's https://www.youtube.com/watch?v=oLpEzq_6_go which is the next ESChamps tournament cast later on Thursday.

Yenrabbit• 5 years ago

This is an excellent project with a great write-up. Most articles this long would loose me but this is engaging and clear, a joy to read. And I'm in awe of the amount of work that has gone into every aspect of this.

Seeing as my policies are currently the world’s best CodeCraft players, we’ll just have to take their word for it for the time being.

I really hope this inspires some competition! How long until there is a leaderboard? :)

shmageggy• 5 years ago

Agreed, this is better than the vast majority of machine learning papers that actually get published. The ablation section is particularly nice. It is really a major failing of the field that in most papers, it's entirely unclear what aspect of the model (or which particular hacks) are really carrying the weight.

mindfulplay• 5 years ago

This is a fantastic project and a great blog! As games start to include RL, it will be a lot of fun that could spawn a while new generation of interesting games (especially if games are made with an RL-first mindset as opposed to using RL later on to beat human beings).

Do you have recommendations to learn more about RL? Is CodeCraft a game?

cwinter2• 5 years ago

Thank you for the kind words! I am also quite excited about the new points in game design space that RL will unlock and am planning write another blogpost on that topic.

I quite like https://karpathy.github.io/2016/05/31/rl/ as an introduction to some of the ideas behind modern RL. Beyond that, I just recently found out about https://github.com/andyljones/reinforcement-learning-discord... which lists a lot of other high-quality resources.

CodeCraft is a programming game which you can "play" by writing a Scala/Java program that controls the game units. It's not actively developed anymore but still functional: http://codecraftgame.org/

CyberRage8• 5 years ago

Interesting blog-post.

I found some similarities with what occurred with Deepmind's Alphastar AI.

One of the weaknesses that seem to manifest in this piece too is the handling of unfamiliar scenarios.

The AI is very confused once it experiences something that was rarely seen in its learning data. Destroyer's big drones confused the bot quite a bit.

Deepmind solved it by intentionally creating agents that introduce different\bizzare strategies(which they called exploiters) in order to develop robustness against such strategies.

cwinter2• 5 years ago

The bot has actually never seen Destroyer's big drones during training even once, so I found it somewhat surprising that it even works as well as it does!

Completely agree that adding something like the "League" used by AlphaStar would be one of the top priorities if you wanted to push this project further. I don't think CodeCraft is sufficiently complex to really allow for several very distinct strategies in the same way as StarCraft II, but I would still expect training against a larger pool of more diverse agents to increase robustness quite a bit.

CyberRage8• 5 years ago

What amazes me at the end of the day is that brute-forcing seem to do much better than I initially thought it would do.

Trying random stuff just sounds stupid but with enough compute and data, I guess it could overpower smart creatures like us.

I agree that CodeCraft is vastly simpler than StarCraft but the idea is the same. just try random stuff(sometimes with better logic behind it) until something works and then optimize it to perfection.

andyljones• 5 years ago

In case anyone misses the links, this is twinned with two other superb posts - one about general lessons the author learned over the course of the project

https://clemenswinter.com/2021/03/24/my-reinforcement-learni...

and one history of the project

https://clemenswinter.com/2021/03/24/conjuring-a-codecraft-m...

Mastering Real-Time Strategy Games with Deep RL: Mere Mortal Edition

Related Stories