DeepMind has open-sourced the heart of AlphaGo and AlphaZero

https://twitter.com/DrJimFan/status/1625538305889820673

by mariuz • 3 years ago

328 87 3 years ago

I can't help but wonder when this stuff happens that the devs (or the leads) are worried about being let go, so they open source their stuff to be able to take it elsewhere.

Anyone here from DeepMind?

Jensson2• 3 years ago

The people who left probably said that they would want to work on stuff usable by the public, this seems like an answer to that.

throwaway815232• 3 years ago

These particular programs were bleeding edge when they were announced, but have been recreated and improved on outside of DeepMind since then. For example, LCZero today is stronger at chess than any of the DeepMind chess programs that were shown to the public. Of course we don't know if they got further behind the scenes. Of course LCZero and everything else relied on the published papers of DeepMind. That is, the papers were more important than the code.

largepeepee2• 3 years ago

Agreed, most "newly released" "open sourced" products have typically been in the pipeline for at least half a decade.

That said many have been retooled for a more specific purpose before public release.

mtlmtlmtlmtl7• 3 years ago

There's an interesting bias I've noticed on HN where a lot of people still believe AlphaZero is the state of the art in computer chess, when that hasn't been true except for a short while after the release. AZ still gets posted and upvoted today while newer improvements in other engines are discussed far less.

I guess it's because AZ came out of Google?

Closi• 3 years ago

I can't help but wonder when this stuff happens that the devs (or the leads) are worried about being let go, so they open source their stuff to be able to take it elsewhere.

Open-sourcing isn't a dev-level decision - it is a business leadership decision.

kubb• 3 years ago

Must feel pretty bad being a rank and file dev at DeepMind and realizing that your leads had all the talent and funding in the world that they could have used for building language model products and instead they heavily invested in solving board games.

Then you realize they get paid >x10 of what you are and they're fine, but the next layoff is likely gonna get you.

est• 3 years ago

They have their own LLMs such as PaLM

tyler33• 3 years ago

why they dont make public apps?(at least)

ogoparootbbo• 3 years ago

The human in the loop reinforcement learning paper that powered chatgpt's training arose from deepmind's experiments with boardgames (and games). AGI is still an unsolved problem and deep RL that arose from the success of deepmind's experiments wth boardgames and games so far, will likely play a huge part in it

leereeves• 3 years ago

Like Bell Labs of old, Google has excellent AI researchers but they haven't built many (any?) AI products people can use.

teruakohatu• 3 years ago

Google researchers invented Transformers and Deepmind pioneered reinforcement learning.

10xDev• 3 years ago

Deepmind did not pioneer reinforcement learning. In fact it wasn’t even the first place to use neural networks for feature representation in RL. That was achieved with Backgammon in the 90s.

jorgemf• 3 years ago

Board games have been used in AI since the beginning. They provide a good environment as we know the rules and control them. Also as everybody uses them it is easier to compare different algorithms. Most advances in AI were done in board games. Most probably chatGPT uses lot of those things you think are irrelevant in board games (reinforcement learning for fine tuning the responses with human feedback, same algorithms used in board games).

mtlmtlmtlmtl7• 3 years ago

If chess was interesting enough for Ken Thompson to work on, that's good enough for me.

largepeepee2• 3 years ago

Games offer clear perimeters and have lotsa depth to them.

Humans are also trained in games when young, and even into adulthood, think war games that are held semi-annually between countries.

These are just board games with increased stakes and additional variables.

blitzar• 3 years ago

They also do work with the UK National Health Service.

I have heard (high level sources, unconfirmed publicly) the YouTube algorithm that promotes open mouth creator with $$$$ signs thumbnails is based on their ground breaking research from this collaboration.

aix1• 3 years ago

Not since 2018: https://www.theverge.com/2018/11/13/18091774/google-deepmind...

mellosouls• 3 years ago

"board games"

ie. Chess and Go. Go a couple of thousand years old, and Chess in particular a core element of AI research history.

Language model research is cool, but you should perhaps consider expanding your horizons beyond the latest headlines in AI.

codeflo2• 3 years ago

“The heart of” can mean many things, is this runnable or not? Also, they should open source the weights as well so that their claims can finally be verified independently.

Y_Y• 3 years ago

Totally agree. I don't even know what benefit they'd get at this point from keeping some parts locked up.

Anyway if you want something runnable Leela has a nice reimplementation: https://github.com/leela-zero/leela-zero

mourner• 3 years ago

I'd suggest KataGo, which is much stronger and more actively developed than Leela Zero https://github.com/lightvector/KataGo

d4rkp4ttern• 3 years ago

And let us not forget those secret random seeds!

codeflo2• 3 years ago

Is this a joke I don't get? Because it's a bit unlikely there's something interesting in random seeds.

Synaesthesia4• 3 years ago

Please do the same for Alphastar, the starcraft AI, that would be great.

notTooFarGone• 3 years ago

This would probably be against some Blizzard ToS

Synaesthesia4• 3 years ago

I thought that they had an open API, and I also thought we as players and AI enthusiasts would benefit somehow from this venture, it was all rather exciting. But the research was all behind closed doors, and nothing much came of it except for some impressive demonstrations.

knicholes• 3 years ago

I talked to the guy who led AlphaStar at Blizzcon. I asked if I could get the weights. He said that balance changes and map changes makes keeping the model updated prohibitive.

swyx2• 3 years ago

sure but even frozen outdated weights would be very instructive to study

runnerup• 3 years ago

I personally hope these types of AI's make it into future games as bots to play against in single-player, and also as an agent to run some of the background stuff while you're focusing on other things.

For example, in Starcraft, when you A-move a large part of your army across the map, an AlphaStar-like AI could decide which units to prioritize it to attack once it reaches a group of enemy units, or what formation to move in (automatically keeping colossus in back, observer ahead, balls of marines split apart into smaller clumps), or when your Terran army encounters a significant force along the way, the tanks in back could auto-siege / auto-unsiege.

Synaesthesia4• 3 years ago

That was one of the things I was hoping for, better ai, but also there are many unanswered questions in Starcraft. Like which race is the "best", what is an optimal strategy, and I thought these questions could be explored more.

mtlmtlmtlmtl7• 3 years ago

Obligatory note that AlphaZero has long since been surpassed by its independently developed cousin Leela Chess Zero(which is also open source btw), and also Stockfish has more than caught up and remains competitive with MCTS engines.

bsaul• 3 years ago

Did stockfish use any tech from alphazero & co ? or has it managed to catch up keeping itself "pure" ?

FartyMcFarter2• 3 years ago

They use neural networks since Stockfish 12, but not in the same way that AlphaZero did:

https://www.chessprogramming.org/Stockfish_NNUE

thom• 3 years ago

Stockfish now uses a neural network in some situations. Last I checked, it uses the classical, fast, tactical search and evaluation for active, tactical positions, and uses the network for quieter, more strategic positions.

mtlmtlmtlmtl7• 3 years ago

Stockfish has added neural networks for evaluation, much simpler than the large networks in MCTS engines, and efficiently CPU computable. The technique actually originated in a Shogi engine and was later ported to mainline Stockfish, so there's no direct inheritance from AlphaZero in that sense.

The search algorithms have been incrementally improved, but still follow the same Alpha Beta and a heap of heuristics approach.

londons_explore2• 3 years ago

Worth noting that while AlphaGo and AlphaZero are incredible achievements, the amount of actual code to implement them isn't very much.

If you have the research paper, someone in the field could reimplement them in a few days.

Then there is the large compute cost for training them to produce the trained weights.

So, opensourcing these bits of work without the weights isn't as major a thing as you might imagine.

tiagod• 3 years ago

Then there is the large compute cost for training them to produce the trained weights.

And as far as I understand, the training code is where the secret sauce lies.

londons_explore2• 3 years ago

generally no...

Secret sauce is in the ML compiler and accelerator used, but all those improvements simply lower the cost of training a model. You could still do it on a regular GPU, it would just take you more time.

In the case of Google, they probably used TPU chips that you can't get direct 'bare metal' access to anyway, so none of that code would have helped.

The actual optimizer used and parameters (like the learning rate schedule) is normally published in the research paper.

Yuioup• 3 years ago

It's already been done:

https://github.com/lightvector/KataGo

ta_tunestub• 3 years ago

Yep, there are many reimplementations. Here is a reimplementation that swaps out a neural net with a GBDT to address compute costs:

https://github.com/cgreer/alpha-zero-boosted

woah• 3 years ago

How does the performance of this version compare?

rhdunn• 3 years ago

There is Leela Zero (https://github.com/leela-zero/leela-zero) for Go and lc0/Leela Chess (https://github.com/orgs/LeelaChessZero/repositories) for Chess, where both provide trained weights. The Leela Chess project specifically have been working for a long time on training and refining the weights for Chess, as well as providing the code -- they allow you to see the history and performance over time for the various trained models.

CSMastermind• 3 years ago

If I'm not mistaken Stockfish has it's own neural network implementation as well correct?

sebstefan• 3 years ago

I'm moderately into chess but I have never heard of Leela

I'm shocked to discover it's been rated higher than AlphaZero & Komodo and just slightly below Stockfish

ArtWomb2• 3 years ago

Models can be massive, but also totally doable. Just to put things in perspective: ProcMaze solving using DeepMind MCTX converges <1M steps. Whereas a physically based agent such as HalfCheetah may require >100M steps to learn to run. Q-learning Pac-Man on snapdragon chromeos is ~1hr for 1000 epochs ;)

mensetmanusman3• 3 years ago

If only there existed a distributed way to incentivize calculation of AI weightings while also providing a currency to encourage scale…

eternalban• 3 years ago

not an entirely bad idea.

lern_too_spel2• 3 years ago

As many people have pointed out before, the distributed currency part adds energy waste. BOINC accomplishes the same without the waste.

brilee2• 3 years ago

If you have the research paper, someone in the field could reimplement them in a few days.

Hi I did this while I was at Google Brain and it took our team of three more like a year. The "reimplementation" part took 3 months or so and the rest of the time was literally trying to debug and figure out all of the subtleties that were not quite mentioned in the paper. See https://openreview.net/forum?id=H1eerhIpLV

westurner3• 3 years ago

Replication crisis: https://en.wikipedia.org/wiki/Replication_crisis :

The replication crisis (also called the replicability crisis and the reproducibility crisis) is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibility of empirical results is an essential part of the scientific method,[2] such failures undermine the credibility of theories building on them and potentially call into question substantial parts of scientific knowledge.

People should publish automated tests. How does a performance-optimizer know that they haven't changed the output of there are no known-good inputs and outputs documented as executable tests? Pytest-hypothesis seems like a nice compact way to specify tests.

AlphaZero: https://en.wikipedia.org/wiki/AlphaZero

GH topic "AlphaZero" https://github.com/topics/alphazero

I believe ther are one or more JAX implementations of AlphaZero?

Though there's not yet a quantum-inference-based self-play (AlphaZero) algorithm?

TIL about the modified snow plow problem is a variation on TSP, and there are already quantum algos capable of optimally solving TSP.

swyx2• 3 years ago

alltime classic Hacker News moment: "heh someone in the field could write this in a few days" "Hi its me 3 of us literally work at google brain and it took us a year"

abraxas• 3 years ago

How come you weren't able to just get it from DeepMind given that they are a subsidiary of Google? Is there a lot of red tape involved in exchanging IP like that?

rcme• 3 years ago

One thing that's not appreciated by many who haven't tried to implement a NN is how subtle bugs can be. When you look at code for a NN, it's generally pretty simple. However, what happens when your code doesn't produce the output you were expecting? When that happens, it can be very difficult and time consuming to find the subtle issue with your code.

xcombelle• 3 years ago

actually it is not the original code of alphazero , the original code was tensorflow/TPU https://en.wikipedia.org/wiki/AlphaGo_Zero#Training and this code is in JAX

nmca• 3 years ago

Note that JAX was created after the success of Alpha(Go|Zero), so the claim that this powered those papers is not accurate. It is a DM MCTS library though.

banach• 3 years ago

...and announce that they are changing name to OpenMind?

nohat• 3 years ago

I feel like this headline is deceptive. This is a Jax reimplementation, and it was released a year ago. It is a cool library though. The basic operation of muzero is very simple, but training it efficiently is tricky.

fnbr• 3 years ago

Yeah. DeepMind has released various MCTS implementations (eg in OpenSpiel). Not the same as releasing AlphaZero.

Recursing• 3 years ago

Interesting approach to private variables https://github.com/deepmind/mctx/blob/577fc77a3cda1b796e277e...

FartyMcFarter2• 3 years ago

Is there a better way to do it in Python?

brilee2• 3 years ago

So the naive MCTS implementation in Python is ridiculously inefficient. Of course, you could reimplement it in C++ but this then requires you to use the C wrappers of Tensorflow/JAX to do the MCTS/neural network interop.

I came up with a nifty implementation in Python that outperforms the naive impl by 30x, allowing a pure python MCTS/NN interop implementation. See https://www.moderndescartes.com/essays/deep_dive_mcts/

MCTX comes up with an even niftier implementation in JAX that runs the entire MCTS algorithm on the TPU. This is quite a feat because tree search is typically a heavily pointer based algorithm. It uses the object pool pattern described in https://gameprogrammingpatterns.com/object-pool.html to serialize all of the nodes of the search tree into one flat array (which is how it manages to fit into JAX formalisms). I suspect it's not a particularly efficient use of the TPU, but it does cut out all of the CPU-TPU round trip latency, which I'm sure more than compensates.

cgreerrun2• 3 years ago

I came up with a nifty implementation in Python that outperforms the naive impl by 30x, allowing a pure python MCTS/NN interop implementation. See https://www.moderndescartes.com/essays/deep_dive_mcts/

Great post!

Chasing pointers in the MCTS tree is definitely a slow approach. Although typically there are ~ 900 "considerations" per move for alphazero. I've found getting value/policy predictions from a neural network (or GBDT[1]) for the node expansions during those considerations is at least an order of magnitude slower than the MCTS tree-hopping logic.

[1] https://github.com/cgreer/alpha-zero-boosted

DeepMind has open-sourced the heart of AlphaGo and AlphaZero

Related Stories