Distilling Stockfish: Training a ResNet/ViT Hybrid Model on 1 Billion Chess Positions

Project Overview: Fitting Stockfish into a Neural Network

Stockfish has long been the gold standard of chess engines, with its powerful value function built on deep search and hand-crafted optimizations. A developer recently shared an intriguing project on Reddit: "distilling" Stockfish's value function into a ResNet/ViT hybrid neural network, trained on 1 billion board positions from the Gigafish dataset.

Even more noteworthy, the author released the complete dataset of approximately 3.9 billion positions, hosted on Hugging Face (gigafish-3.8b-d10). Built from 37 months of Lichess games, this dataset is exceptionally large for an open-source project in this space — providing a solid foundation for researchers looking to reproduce or extend the experiments.

Core Idea: Approximating the Search Tree at Fixed Depth

The theoretical motivation here is genuinely interesting. The central question the author tackles is: at a fixed search depth, a value function is essentially approximating the entire search tree beneath it.

In other words, when Stockfish evaluates a position at a fixed depth — depth 10, as indicated by the "d10" in the dataset name — the score it produces is a compressed judgment over all possible continuations at that depth. If a neural network can mimic these fixed-depth evaluations while running far faster than Stockfish performing an equivalent search, then that network has real potential to compete with NNUE in the trade-off between speed and playing strength.

NNUE (Efficiently Updatable Neural Network) is the efficient, lightweight evaluation architecture used in modern Stockfish. Renowned for its blazing inference speed, it was a key driver of Stockfish's leap in playing strength. The author's goal is to explore an alternative path — using distillation to produce an equally efficient evaluator.

Why Keep the Depth Fixed

Fixed depth is the critical design constraint of the entire experiment. If training data were drawn from evaluations at varying search depths, the labels themselves would be inconsistent in "quality," and the network would end up learning a muddled target. By standardizing on depth 10, the network learns a well-defined, measurable approximation task: replace a depth-10 tree search with a single forward pass. This makes "is it actually faster than real search?" a directly testable proposition.

---

Knowledge Distillation is a deep learning technique for transferring the "knowledge" of a complex model into a lighter one, systematically formalized by Hinton et al. in 2015. The core idea is to have a small "student" network directly mimic the outputs of a large "teacher" model, rather than learning from raw labels. In this project, Stockfish's depth-10 search acts as the teacher, with its evaluation scores (in centipawns) serving as soft labels to supervise training. The advantage is that scores produced by deep search carry far richer information than simple win/loss labels — for instance, +0.3 and +2.5 both represent a White advantage, but to vastly different degrees. Neural networks can learn much finer-grained positional judgment from this continuous signal, nuance that is largely lost when training only on game outcomes.

NNUE (Efficiently Updatable Neural Network) is the evaluation architecture introduced to Stockfish in 2020, originally adapted from Japanese Shogi programs. Its design centers on incremental updates: since each move typically changes only a handful of piece positions, NNUE's input layer is specially designed so that only the affected portions need to be recomputed, rather than redoing a full forward pass for the entire position. This makes NNUE inference on CPU extremely fast — typically on the order of microseconds. Its introduction boosted Stockfish's playing strength by roughly 100 Elo over purely hand-crafted evaluation, widely regarded as one of the most important technical advances in traditional engines in recent years. Whether a general-purpose neural network trained via distillation can compete on inference speed with this architecture — purpose-built for incremental updates — is the key challenge this project faces.

---

Architecture Choices: CNN vs. ViT

In exploring network architectures, the author shares a genuinely valuable empirical observation.

Vision Transformers (ViT) are very slow at understanding the chessboard. A chessboard is a highly structured 8×8 grid, and the relationships between pieces carry strong geometric properties. ViT has no built-in assumptions about this kind of local spatial structure — it has to learn concepts like "adjacent squares" and "diagonal attacks" from scratch through data, leading to very slow convergence early in training.

CNNs, by contrast, perform more effectively thanks to their inherent geometric inductive biases. Convolutional kernels are naturally suited to capturing local spatial patterns, allowing the model to quickly build an understanding of board geometry early in training — a property that aligns closely with the nature of chess.

The Hybrid Architecture Delivers the Best Results

That said, the author's final conclusion isn't a simple either/or. Experiments showed that combining CNN and ViT yielded the best results. The logic is straightforward: the CNN efficiently extracts local geometric features, giving the model a fast and robust spatial foundation; the Transformer excels at modeling global, long-range piece relationships — such as pins and coordination that span the entire board. The two are complementary, preserving CNN's training efficiency while gaining the attention mechanism's ability to grasp the overall position.

This finding echoes a broader trend in computer vision: on structured tasks with sufficient data, pure Transformers aren't always optimal, and inductive biases still carry real value.

---

Inductive Bias refers to the prior assumptions about data structure that a model encodes before any learning takes place. CNNs carry inductive biases of "translation equivariance" and "locality" — that features at neighboring positions are correlated, and that the same pattern should be recognized equally regardless of where it appears. This is particularly relevant for chess: the rule that a bishop attacks diagonally should be recognized as the same geometric relationship no matter which corner of the board it occurs in. Transformers carry almost no such spatial priors and must learn these relationships entirely from data and attention, leaving them at a disadvantage with limited data or early in training. Hybrid architectures exploit the complementarity of both — the CNN first encodes the raw board into feature maps rich with spatial structure, and the Transformer then models global piece interactions on top of that, with a clean division of labor that balances efficiency and expressive power.

---

Significance and Highlights

The value of this project extends well beyond the chess model itself.

From an open-source contribution standpoint, a dataset of 3.9 billion positions — drawn from real Lichess games and annotated at a uniform search depth — is a rare public resource for chess machine learning research. It lowers the barrier to entry for follow-up researchers and makes it easier to study the question of how well neural networks can approximate fixed-depth search.

From a methodological standpoint, the framing of "value function as search tree approximation" offers a clean framework for understanding the relationship between evaluation functions and search. It also speaks to a broader question: to what extent can expensive explicit search be replaced by a single neural network inference pass?

From an engineering standpoint, the empirical evidence on CNN vs. ViT performance differences on chess tasks is worth noting for anyone working with grid-structured, geometrically rich data — don't blindly chase Transformers; inductive biases remain powerful tools in the right context.

It's worth noting that the author's current write-up focuses primarily on design rationale and dataset release. Quantitative results — such as a direct playing-strength comparison between this model and NNUE — are still pending further validation. Developers interested in this direction can get started by accessing the dataset directly on Hugging Face and reproducing the experiments themselves.