The neural network of the Stockfish chess engine

https://cp4space.hatsya.com/2021/01/08/the-neural-network-of-the-stockfish-chess-engine/

by c1ccccc1 • 6 years ago

355 91 6 years ago

cp4space.hatsya.com

knuthsat2• 6 years ago

This is very nice. Reminds me a lot of tricks used for simple linear models. And it seems to work, given that Leela Chess Zero is losing with quite a gap.

Most of the times one would learn a model by just changing a single feature and then doing the whole sum made no sense.

A good example is learning a sequence of decisions, where each decision might have a cost associated to it, you can then say that the current decision depends on a previous one and vary the previous one to learn to recover from errors. If previous decision was bad, then you'd still like to make the best decision for current state.).

So even if your training data does not have this error-recovery example, you can just iterate through all previous decisions and make the model learn to recover.

An optimization in that case would be to just not redo the whole sum (for computing the decision function of a linear model).

kohlerm6• 6 years ago

"Leela Chess Zero is losing with quite a gap" Check https://tcec-chess.com/ ATM Leela is in front again. That being said it is not clear which approach is better e.g. a very smart but relatively slow evaluation function (Leela) or SFs approach to use a relatively dumb eval function with a more sophisticated search. It is pretty clear that Leelas search can be improved (Check https://github.com/dje-dev/Ceres for example)

kohlerm6• 6 years ago

That being said, assuming you can fully saturate multiple GPUs then the SF approach has the disadvantage that it cannot use the performance improvements for GPUs/TPUs which still are growing fast, whereas CPU performance only grows slowly these days.

knuthsat2• 6 years ago

Looks like the tournament is still ongoing. I was referring to the last few.

https://en.wikipedia.org/wiki/TCEC_Season_18

https://en.wikipedia.org/wiki/TCEC_Season_19

Fragoel2• 6 years ago

I don't know a lot about chess but I have one question: isn't chess a solved game (in the sense that given a board state we always can compute the right move)? why use a neural network that can introduce, a very small percentage of the time, mistakes? I guess it is for performance reasons?

elcomet2• 6 years ago

Chess is definitely not a solved game. The end game is solved though, when you have only 7 pieces left on the board.

See https://en.wikipedia.org/wiki/Endgame_tablebase#Computer_che...

mvanaltvorst2• 6 years ago

You can imagine playing chess as a tree. You start with a game state, and every possible move is an edge to a new game state. Unfortunately, the amount of game states grows approximately exponentially (e.g. for every game state there are approximately 40 moves to play, thus every extra move multiplies #(game states) by 40, approximately). This neural engine is a trick so that Stockfish does not have to simulate all 40 moves every game states. The neural net outputs the moves that are most likely to be strong moves, and Stockfish will only consider those moves. Of course, this is a very basic explanation and there are many more optimisations Stockfish uses, though the ultimate goal of almost every optimisation is to reduce the amount of simulation Stockfish has to do.

V-26• 6 years ago

"This neural engine is a trick so that Stockfish does not have to simulate all 40 moves every game states."

This optimization always done by chess engines, it's called pruning (they'd be quite crippled without it).

Maybe the neural network component is now in charge of it, but it's not a new thing.

bbojan• 6 years ago

> The neural net outputs the moves that are most likely to be strong moves, and Stockfish will only consider those moves.

I don't think this is correct. According to the article, they are only using the neural network for evaluating the value (strength) of the positions. The search is still the same.

I believe Leela works in the manner you described.

blt• 6 years ago

From the article:

it’s a simple feedforward network with... a single scalar output to give a numerical score for the position, indicating how favourable it is for the player about to move.

The search algorithm proposes moves, computes the board state resulting from them, and evaluates each state.

This may seem like a trivial distinction, but it's important. The Stockfish method requires knowing the rules of the game; networks that output a move directly do not.

dan-robertson5• 6 years ago

Chess is not a solved game (though I think the stalemate rules make it finite, there are far too many positions). All computer chess programs have to try to determine if positions will be good without playing out all possible ways the game could go. Usually this is some combination of search strategies and scoring positions (the article describes how stockfish scores positions).

V-26• 6 years ago

I agree it's finite, but why do you think it's specifically owed to the stalemate rule, of all things?

zone4114• 6 years ago

No, it's not close to being solved. I made a new estimate recently at there are about 8.7E+45 unique positions: https://github.com/lechmazur/ChessCounter. You don't need to analyze all of them to generate the proof but it's still far beyond our computational and storage capacities to solve chess.

altvali2• 6 years ago

For comparison, there are about 10E+49 atoms making up our planet.

aliceryhl• 6 years ago

No, it is not solved. To solve it, you would need to investigate all possible continuations of the game, all the way until the game ends, but there are so many continuations that this is impossible.

nullc• 6 years ago

A game could be solved even if its state space was too large to enumerate, for example if you could show that there was some latent structure that let you prove things about large parts of the state space without completely exploring it.

This is, for example, why we have a concrete number for the number of legal go positions even though there are far too many to enumerate. https://en.wikipedia.org/wiki/Go_and_mathematics#Legal_posit...

The rules of chess are so irregular that it seems likely that any latent structure would be tremendously complex, but you never know what some clever new analysis technique will do.

tspduck• 6 years ago

I have to disagree with the others. Since AlphaZero I think chess is closed to being solved. Technically it is not, but practically is is.

Chess is a hard game to solve completely, and I think one reason is that there are many states in chess where there is not one superior move, but only a probable optimal move, in the sense that the game-tree for winning against the move is smaller than other moves. Then the agent/player has to guess what the opponents strategy is given that move, and that depends on the opponent.

I plays are truly creative, and its results speak for itself.

From wiki: "In AlphaZero's chess match against Stockfish 8 (2016 TCEC world champion), each program was given one minute per move. Stockfish was allocated 64 threads and a hash size of 1 GB,[1] a setting that Stockfish's Tord Romstad later criticized as suboptimal. AlphaZero was trained on chess for a total of nine hours before the match. During the match, AlphaZero ran on a single machine with four application-specific TPUs. In 100 games from the normal starting position, AlphaZero won 25 games as White, won 3 as Black, and drew the remaining 72. In a series of twelve, 100-game matches (of unspecified time or resource constraints) against Stockfish starting from the 12 most popular human openings, AlphaZero won 290, drew 886 and lost 24."

source: https://en.wikipedia.org/wiki/AlphaZero#Chess

hiq3• 6 years ago

there are many states in chess where there is not one superior move, but only a probable optimal move

I don't think this statement is true given [0], but maybe I understood you wrong.

[0] https://en.wikipedia.org/wiki/Zermelo%27s_theorem_(game_theo...

zone4114• 6 years ago

You're making a jump from "AlphaZero plays better than the best programs of its time" to "chess is close to being solved" but that's not justified without much more evidence. There are still a lot of improvements and new ideas added to the top programs and even if they reach a plateau, you'd have to show that increasing time per move 100x doesn't result in a significant improvement in playing strength. We don't know what would be the Elo difference between a 32-piece tablebase and the best existing program but I don't think anybody would bet that it's less than 200 Elo.

Here is a graph of Stockfish's progress since the end of 2015: https://camo.githubusercontent.com/f169f774996346ad146f96f74.... Self-play exaggerates strength differences but it's been a very good rate of progress, even without hardware improvements in the meantime.

kohlerm6• 6 years ago

No it is not solved. E.g. SFs recent inclusion of a NN has shown that it can beat Leela (which should be superior to AlphaZero).

hiq3• 6 years ago

https://en.wikipedia.org/wiki/Solving_chess

danielbarla2• 6 years ago

Chess is estimated to have 10^120 possible games. Observable universe is estimated to have between 10^78 to 10^82 atoms. So, unless you had some way of dramatically compressing or simplifying this search space, you'd be needing to store 38 bits+ information, if you had every atom in the observable universe as your memory. Suffice to say, this has not yet been achieved.

kevincox2• 6 years ago

You don't need to consider every state individually to solve a game. For example if you have a game on a 100x100 board and the only goal is to move your piece to the other side you can know you have found the optimal solution if you can do it in 99 moves (assuming that you move 1 per turn).

In the chess scenario if you can prove that there is a set of moves where you win no matter what the other colour does then you have solved the game. You don't need to consider every possible game, just the games reachable within this set of moves.

Solving chess would be proving that "From starting position black can win every time" (or the same for white). You don't need to prove how to win from every possible board state.

altvali2• 6 years ago

You don't need to care about possible games, but game states, what's the best result and the move to achieve it in each. Another user computed an upper bound at 8.7E+45 positions: https://github.com/lechmazur/ChessCounter . I pointed out that our planet has about 1E+50 atoms. And we can further compress the database by perhaps two orders of magnitude if we don't store symmetric positions or positions close to mate. A Kardashev 2 civilization could play perfect chess.

T-hawk• 6 years ago

You're correct, in that given a board state the right move can always be exhaustively computed, because the game has no randomness or hidden information. But enough computing power doesn't exist on the planet to iterate through all the possible states.

The word you're looking for is "deterministic", rather than solved. Solved is usually used to mean the exhaustive computation has been done. Deterministic would mean it could be if enough computing power existed, but for chess it doesn't.

LittlePeter• 6 years ago

Leela played stockfish 200 games and won with 106 - 94 [1]. Not sure which version of stockfish was used.

Some of the Leela-Stockfish games are analyzed by agadmator on YouTube [2].

[1] https://www.chess.com/news/view/13th-computer-chess-champion...

[2] https://www.youtube.com/watch?v=YtXZjKItuC8

zone4114• 6 years ago

This match was played before the NNUE version of Stockfish was introduced. Stockfish NNUE beat LC0 in TCEC season 19: https://www.chessprogramming.org/TCEC_Season_19

thomasahle2• 6 years ago

The chess.com version is the old Stockfish from mid 2020. The NUE architecture was only put in place around August. If you see https://github.com/glinscott/fishtest/wiki/Regression-Tests you'll notice that gave a very significant (100+ ELO) boost.

There is a current tournament going on at tcec-chess.com/ which stockfish has been leading so far, but I see Leela has just caught up in the head to head.

Of course Leela also keeps evolving.

kohlerm6• 6 years ago

FYI Leela is in the lead ATM

dmurray2• 6 years ago

It's difficult to organise Leela-Stockfish as a fair fight, because they run on different hardware (CPU vs GPU) and both get substantial improvements by playing on better hardware.

Traditionally this wasn't a big problem as every engine was more or less optimised for a fast Intel CPU with a moderate to large amount of RAM. The organisers would decree the specs of the championship hardware some time in advance. Now, (at least for TCEC, the other major engine tournament) they pick two hardware configurations, one CPU-heavy and one GPU-heavy, and give each team the choice.

How do you balance those? It's been suggested you should make them equal in terms of watts of power, or dollar cost to buy, but neither of those are obviously best. In practice the TCEC organisers pick something close to what they picked last time but shade it against the winning engines, making the contest more even. Chess.com likely do something similar though they're less rigorous about the details.

criddell• 6 years ago

It's been suggested you should make them equal in terms of watts of power, or dollar cost to buy, but neither of those are obviously best.

Do they give these computers a rating like they do human players?

dan-robertson5• 6 years ago

It seems that the strategy is to use the neural network to score various moves and then a search strategy to try to find moves that result in a favourable score. And this post focuses on some of the technical engineering details to design such a scoring network. In particular the scoring is split into two parts: 1. A matrix multiplication by a sparse input vector to get a dense representation of a position, and 2. Some nonlinear and further layers after this first step. And it seems that step 1 is considered more expensive.

The way this is made cheap is by making it incremental: given some board s and output of the first layer b + Ws, it is cheap to compute b + Wt where a t is a board that is similar to s (the difference is W(t-s) but the vector t-s is 0 in almost every element.)

This motivates some of the engineering choices like using integers instead of floats. If you used floats then this incremental update wouldn’t work.

It seems to me that a lot of the smarts of stockfish will be in the search algorithm getting good results, but I don’t know if that just requires a bit of parallelism (surely some kind of work-stealing scheduler) and brute force or if it mostly relies on some more clever strategies. And maybe I’m wrong and the key is really in the scoring of positions.

eutectic2• 6 years ago

I don't think the integer weights are neccessary for sparsity; they are just faster because they allow for low precision.

Of course floats aren't strictly associative so you wouldn't get bitwise equivalence between the incremental and non-incremental updates, but I don't see how that would matter in this context.

dan-robertson5• 6 years ago

The point is that integer weights allow for incremental updates to be correct. If you used floats then they would drift away from correct as you applied more incremental updates. And neural networks can be quite sensitive to small errors

thomasahle2• 6 years ago

It seems that the strategy is to use the neural network to score various moves and then a search strategy to try to find moves that result in a favourable score

The neural net is only for scoring positions, not moves. Check the article on Stockfish NNUE at the chess programming wiki: https://www.chessprogramming.org/Stockfish_NNUE

There is a LOT of cleverness built into the stockfish search as well.

In fact, while neural networks have now won the position evolution game, it is still an open discussion in the chess programming community if the Alpha Zero / Leela search (NN + Monte Carlo) is as good as Stockfish's PVS.

ogogmad• 6 years ago

PVS?

pcwelder• 6 years ago

Using a previous Stockfish scorer they trained an NN without any labelling effort. This is also similar to how unsupervised translation is done in some methods. They start from word->word dictionary results and iteratively train lang1->lang2 and lang2->lang1 models feeding on each other's output.

brilee• 6 years ago

Ironically, a lot of the tricks Stockfish is using here are reminiscent of tricks that were used in the original AlphaGo and later discarded in AlphaGoZero.

In particular, the AlphaGo paper mentioned four neural networks of significance:

- a policy network trained on human pro games. - a RL-enhanced policy network improving on the original SL-trained policy network. - a value network trained on games generated by the RL-enhanced policy network - a cheap policy network trained on human pro games, used only for rapid rollout simulations.

The cheap rollout policy network was discarded because DeepMind found that a "slow evaluations of the right positions" was better than "rapid evaluations of questionable positions". The independently trained value network was discarded because co-training a value and policy head on a shared trunk saved a significant amount of compute, and helped regularize both objectives against each other. The RL-enhanced policy network was discarded in favor of training the policy network to directly replicate MCTS search statistics.

The depth and branching factor in chess and Go are different, so I won't say the solutions ought to be the same, but it's interesting nonetheless to see the original AlphaGo ideas be resurrected in this form.

The incremental updates are also related to Zobrist Hashing, which the Stockfish authors are certainly aware of.

mattalex• 6 years ago

It's also different because Stockfish uses Alpha-beta treesearch instead of MCTS: MCTS tries to deal with an explosion in search-space by only sampling very small parts of the searchspace and relying on a very good heuristic to guid that search process. In this case it is crucial to find the most relevant subset of the tree to explore it, so spending more time on your policy makes sense.

Alpha-beta pruning however always explores the entire searchtree systematically (up to a certain depth using itd) and prunes the searchtree by discarding bad moves. In this case you don't need as good of an evaluation function because you search the entire tree anyways. Rather you need the function to be fast, as it is evaluated on many more states.

In general AB-pruning only needs the heuristic for estimating the tail-end of the tree and for sorting the states based on usefulness, while MCTS uses all the above plus guiding the whole search process. Spending tons of time on the heuristic is not useful as even the optimal search order can only double your searchdepth. Don't get me wrong that still a lot (especially considering exponential blowup) but MCTS can surpass this depth easily. The disadvantage is that MCTS loses a lot of the guarantees of AB-pruning and tends to "play down to his opponent" when trained using self-play because the exploration order is entirely determined by the policy.

glinscott• 6 years ago

If anyone wants to experiment with training these nets, it's a great way to get exposed to a nice mix of chess and machine learning.

There are two trainers currently, the original one, which runs on CPU: https://github.com/nodchip/Stockfish, and a pytorch one which runs on GPU: https://github.com/glinscott/nnue-pytorch.

The SF Discord is where all of the discussion/development is happening: https://discord.gg/KGfhSJd.

Right now there is a lot of experimentation to try adjusting the network architecture. The current leading approach is a much larger net which takes in attack information per square (eg. is this piece attacked by more pieces than it's defended by?). That network is a little slower, but the additional information seems to be enough to be stronger than the current architecture.

Btw, the original Shogi developers really did something amazing. The nodchip trainer is all custom code, and trains extremely strong nets. There are all sorts of subtle tricks embedded in there as well that led to stronger nets. Not to mention, getting the quantization (float32 -> int16/int8) working gracefully is a huge challenge.

pk2200• 6 years ago

Just wanted to say thanks for many years of fantastic work on both Stockfish and Leela. The computer chess community owes you a huge debt of gratitude!

EvgeniyZh2• 6 years ago

Interesting how before A0 it was mainly "search matters the most", with crazy low branching factors to get deeper. It seems that humans were just better in search heuristics than in evaluation ones.

spiantino2• 6 years ago

Great writeup!

One thing I don't understand is why it would be smarter to augment the inputs with some of the missing board information - particularly the availability of castling. Even though this network is a quick-and-dirty evaluation, seems like there's room for a few additional bits in the input and being able to castle very much changes the evaluation of some common positions.

mattnewton• 6 years ago

I am not an expert (and don’t have any idea what I am talking about), but wouldn’t this be captured the same way other future positional advantages are when evaluating the next “level” of possible board positions? Ie, the fact that a piece can later give check, or castle, or anything else if you first move it here is a function of that next move’s resulting position score.

This network is just to signal to the final search algorithm how good the board looks in a given spot on tree of possible moves.

spiantino2• 6 years ago

Yes, the overall algorithm will do something much more precise and comprehensive, so it's not like this scoring function needs to be perfect. Still, it matters in a quick evaluation so I'm surprised it isn't in there.

A few examples come to mind where, for instance, white has a bishop and knight attacking f7 and it's black to move. If black castles nothing interesting happens and white is likely overextended. But if black cannot castle white will win a rook.

I can make up a similar situation with en passant, and this does come up, but a lot less frequently.

So anyway, surprised you wouldn't toss a few bits in there, at least the 2 for ability to castle by each player.

zetazzed• 6 years ago

The gap between white and black piece performance is massive in these top engines if I'm reading it right. LCZero won 0/ lost 4 as black and won 24 / lost zero as white (with lots of draws)? I had no idea the split was so big now. Do human tournaments look like this too these days? (From https://tcec-chess.com/)

dsjoerg• 6 years ago

Overwhelmingly the result is a draw. White can win sometimes, and wins for black are very rare. It's similar for top grandmasters, but not quite as stark.

bonzini3• 6 years ago

Not really---similar for sure, but not as bad for black. Consider that computer chess tournaments do not start with the pieces in their initial positions. The first 5-15 moves are predetermined by human master players to achieve positions that are imbalanced while not having a side with a clearer advantage. Otherwise it would be even more of a draw fest.

This makes things a bit worse for black, typically, because playing for the win as black means making some positional concession (that white can exploit after successfully defending) in exchange for the initiative and a chance for an attack.

In human play, black has a little more ability than white to steer the game towards their opening preparation[1]. If they want to win, they will try to get have a position that they know in and out from (computer-assisted) home preparation. But there are still a lot of draws in classical (many hours per game) chess.

[1] Chess openings that white chooses are typically called "attack" (King's Indian Attack) or "game" (Scotch Game), while those that black chooses are called "defense" (Sicilian defense). You'll find that there are a lot more "defenses" that black can choose from than "attacks".

tayo42• 6 years ago

if they want to win, they will try

Why would someone not want to win?

nl3• 6 years ago

Consider that computer chess tournaments do not start with the pieces in their initial positions. The first 5-15 moves are predetermined by human master players to achieve positions that are imbalanced while not having a side with a clearer advantage.

I've never heard this before, and it seems strange! Do you have a link to tornament rules with this?

Computer chess is traditionally really strong in openings because their opening book is just database lookups. What is getting a human to play the same opening book supposed to achieve?

EvgeniyZh2• 6 years ago

Current divp uses very unbalanced openings for engines to be able to score. Using starting position/more optimal openings inevitably leads to draw. So it's actually exactly the opposite -- the advantage is so low and draw tendencies are so strong that we need to force position where white has advantage to be able to compare two strong engines

billiam• 6 years ago

Analyses like this are a great indication that the main effect of refining neural networks around chess will make neural networks more exciting and ultimately make chess more boring.

gelert• 6 years ago

Why do you believe that neural networks being better at chess would make it more boring? Genuinely I just don't follow the logic.

nl3• 6 years ago

This is a really good article.

Lots of pieces about neural network design skip over the design of the representation in the input stage, which is one of the key design issues when building a custom neural network.

I love how much depth this articles goes into about that representation.

The neural network of the Stockfish chess engine

Related Stories