When an LLM Makes a Bad Chess Coach

Handing a game to a chatbot and asking it to explain where the student went wrong — that's the naive idea many chess coaches and parents have. But the author of the open-source project Second ran a set of comparison experiments, and the results were pretty disheartening: give Gemma 4 E4B a fact about a blunder and ask it to write a coaching comment, and it called the simplest beginner's mating pattern a "queenside back-rank trap," then invented a "discovered attack" out of thin air.

This isn't a harmless error in a chat window. It gets printed on a piece of paper, handed to a ten-year-old, whose parents are paying for "accuracy."

Second's solution is blunt: the language model only writes words; all chess judgments go to code. Stockfish computes the facts, python-chess translates the facts into natural-language sentences, Gemma only phrases things within the bounds of those sentences, and a final verifier written in pure Python checks every line — every piece, every square, every "mate in N," every tactical term must be traceable to the fact table.

The Division of Labor Between Facts and Wording

The pipeline works like this: a PGN or a Lichess/Chess.com username comes in, Stockfish analyzes each position at depth 14, and python-chess determines what actually happened — which piece is undefended, who's attacking it, what the opponent's best reply is, whether it's a fork, whether there's a forced mate, how much material the better move wins. These get written into a "fact list," for example:

Played 19. Rxe5 (blunder). After the move your d6 queen is attacked by the g6 queen and is undefended. The opponent's best reply is Qxd6. The better move was Qxe7, winning the e7 knight for free.

This is what Gemma gets. It writes a title, a two- or three-sentence explanation, a question that guides the student to find the answer themselves, a rule of thumb, plus a summary, homework, and a note to parents. Then the verifier steps in: tactical terms (fork, pin, sacrifice, back rank, etc.) are only allowed if they appeared in the facts; assertions like "you lost your queen" or "the f4 knight is hanging" must correspond to pieces that were actually lost or undefended in the facts; the parent paragraph can't contain chess notation, and if Hindi is specified it must actually be in Devanagari. If anything fails, Gemma gets a specific list of errors and rewrites; after three failures, the system falls back to neutral text assembled directly from the facts.

The cost of this design is nearly zero, because all models run on the coach's own laptop.

Stockfish is currently the strongest open-source chess engine, using NNUE (Efficiently Updatable Neural Network) evaluation, and the UCI protocol is the standard interface for communicating with external programs. depth 14 means the search tree is about 14 plies deep, a balance between accuracy and time for amateur game review: too shallow misses tactics, too deep takes far too long on an ordinary laptop.

PGN (Portable Game Notation) is the plain-text standard for recording all moves in a game, and most chess sites support exporting it; Lichess and Chess.com provide APIs to batch-pull a student's game history by username. python-chess is the most mature chess library in the Python ecosystem, handling legality checks, position simulation, and move generation — the logic in this article for "determining which piece is undefended, whether it's a fork" is built on these underlying capabilities.

The Numbers from 29 Real Blunders

The author sampled 16 real games in the 1000–1700 rating range from the Lichess open database (CC0), took each side's biggest blunder, for 29 moments total, and ran three approaches on the same machine with the same model:

Metric	Bare Gemma	Gemma + Facts	Second (with verification)
Gives a specific move	14/29	29/29	29/29
Names Stockfish's top alternative	0/29	29/29	29/29
Mentions an illegal move	7/29	0/29	0/29
Makes a claim the engine doesn't confirm	18/29	0/29	0/29

A 4B model on its own didn't name the engine's top move once in 29 tries, and mentioned moves that don't exist 7 times. But once structured facts were fed in, it got all 29 right.

The author admits the result surprised him: the fact list does the real work, not the model. In testing, all of Gemma's first-pass answers with facts passed the verifier, and the verification layer was never triggered even once. He kept it anyway — "seatbelts are for the day the traction fails" — and it lets him say "no fabricated content will reach the student" rather than "probably not."

A 4B Model Running on an Ordinary Laptop

The hardware is an Intel Core Ultra 5 laptop with 16GB of RAM and no discrete GPU, running gemma4:e4b-it-qat (6.1GB) in Ollama. By default Ollama ignores the integrated GPU; setting OLLAMA_IGPU_ENABLE=1 to enable Vulkan raised prompt processing from about 31 tokens/s to about 315 tokens/s. A full game review takes about 90 seconds, of which Stockfish takes about 20 seconds and each comment takes about 12 seconds. A complete review consumes 2,000–3,000 prompt tokens and about 550 output tokens, all free.

The public demo is deployed on Render's free tier, where memory can't hold Gemma, so the container runs Stockfish and offloads commenting to Gemma 4 26B A4B on Google AI Studio. Because they're from the same open model family, the prompt, verifier, and model behavior stay consistent — only where the weights run differs. A 21-move game returns in under 30 seconds, and all comments pass verification on the first try.

Ollama is a common tool for running open-source LLMs locally; by default it uses only the CPU because it can't easily determine integrated GPU availability; setting OLLAMA_IGPU_ENABLE=1 offloads inference to the iGPU through the Vulkan compute interface. Vulkan is a cross-vendor graphics and general-purpose compute API, benefiting both Intel and AMD integrated GPUs. The qat in the model name stands for Quantization-Aware Training, which simulates low-precision arithmetic during training so the model loses less accuracy when quantized to 4 bits — hence a 4B-parameter model taking only about 6.1GB of memory.

The distinction between prompt tokens and output tokens is crucial: prompt processing is parallelizable and batchable, while output is generated token by token serially, so enabling the iGPU speeds up prompts nearly tenfold while each comment still takes about 12 seconds. The A4B in Gemma 4 26B A4B means that under a MoE (Mixture of Experts) architecture, only about 4B parameters are activated per token, so a large model can still keep inference costs low.

A Deliberately Narrow Product

The target user is specific: chess coaches in India. After Gukesh won the world championship, local chess training exploded, and many coaches are strong amateurs and part-time college students — itself a side-hustle market. Lessons are paid; the after-lesson work isn't: reviewing each student's games, finding turning points, writing something both the kid and the parents are willing to read. Twenty students playing two games a week each is 40 games; at 5 minutes each, that's over 3 hours of unpaid labor per week.

Second turns those 40 games into a background task on a laptop. The product is positioned as GPL open source and permanently self-hostable, and what's paid for is the part coaches don't want to do themselves: one-click install, the academy's name on printouts, weekly automatic import of students' online games, technical support — priced at about 299 rupees per month (about $3.15). This price works only because the marginal cost is zero: the model runs on the coach's own machine, so the 40th game costs no more than the first, and no account, API key, or international credit card is needed.

Data compliance is another consideration. Students are minors, and India's Digital Personal Data Protection Act, 2023 has extra requirements for children's data, including verifiable parental consent. Running locally means game records and names never leave the machine, there's no third-party processor, no cross-border transfer, and nothing extra to explain to parents. Add to that the notoriously bad Wi-Fi at tournament venues, and offline capability isn't a gimmick.

On licensing, Gemma 4 uses Apache 2.0, so building a paid product on it is legal; the same prompt set can run on Gemma on a laptop, Ollama Cloud, or Google AI Studio, and when the machine is upgraded you swap in a bigger model. The evaluation was possible because the Lichess database is CC0, and every comment and every score is public in eval/results.json and links back to the original game.

India's Digital Personal Data Protection Act, 2023 (DPDP Act) classifies data of users under 18 as children's data, requiring verifiable parental consent before processing, and prohibits behavioral tracking and targeted advertising aimed at children. For educational tools, this means that if student game records and names are uploaded to a third-party cloud service, the coach may become a data processor with obligations to inform, retain consent, and ensure security; when data stays entirely on a local machine, these obligations are greatly simplified.

GPL (GNU General Public License) requires source code disclosure when distributing derivative works, which fits the "permanently self-hostable" positioning but doesn't mean you can't charge — what's charged for is installation, operations, and support services, not the software license itself. Apache 2.0 is more permissive, allowing closed-source distribution and commercial use as long as copyright and license notices are retained, which is exactly the legal basis for building a paid product with Gemma.

A Few Prompt Lessons Worth Stealing

• Tell the model what matters most. Early comments were all correct and all bland ("watch out for blunders!"). In one game the facts noted an impending Nxc2+ knight fork attacking both king and rook, yet Gemma talked about the c2 pawn. The fix was to have the code pick a KEY FACT for each moment and require the explanation to lead with it.
• Be clear about who the reader is. The first version of the Hindi parent letter opened with "प्रिय Aarav" — addressed to the child. Now the prompt explicitly asks for it to be written to the parent, referring to the child in the third person, and to point out one thing to practice this week.
• Just don't give the child's name to the model. Gemma writes [STUDENT], and Second substitutes it afterward. It doesn't matter for local runs, but it means the hosted demo never sends a child's name to anyone.

The verifier itself had bugs at first. On the first sample game it reported "caught 4 fabricated details," and the author checked each one and found they were all his own fault: the summary said the student "castled on move 5" and had an accuracy of "75.1%," both of which actually existed in the fact table given to Gemma — the summary verifier just only recognized key moments. Worse, the entire summary fell back to an English template, when the parent wanted Hindi. There were two fixes: verify the summary against the actual fact table shown to the model, and make fallbacks per-field so only the broken sentence gets replaced.

The Open-Source Puzzle and an Honest Boundary

Second's argument is: the hard part has long been open source. The world's strongest chess engine, Stockfish, is free software, and a 4B language model can't play chess and doesn't need to — chess analysis has already been solved by open source; the model just needs to talk. This project is essentially a thin layer of glue between two open-source projects.

Product validation is still very preliminary. The first user was a friend of the author's who's a decent player; after trying it, he thought it was "pretty good for coaches." The author doesn't treat that as an accuracy review, just as the first positive feedback from outside the build process. The next step is putting the tool into a coach's weekly routine: reviewing students' games, using those questions in class, and seeing which part actually saves prep time. Until then, the 259-rupee paid plan is just a proposal.

For developers trying to put LLMs into educational settings, the value of this project isn't chess. It demonstrates a division of labor: determinism goes to code and the engine, expression goes to the language model, and correctness goes to a verification layer that can say "no" — and a counterintuitive finding that when the facts are good enough, a 4B model can be more reliable than you'd think.

One-line Summary

Give chess judgments to the engine, wording to a small model, and correctness to a verifier — even a 4B model can write reliable commentary.

Paragraph Summary

The open-source project Second uses a set of comparison experiments to show that bare Gemma 4 E4B, across 29 real blunders, got Stockfish's top move right 0 times, gave illegal moves 7 times, and made claims the engine didn't confirm 18 times. Its solution splits responsibilities: Stockfish analyzes positions at depth 14, python-chess organizes the results into a structured "fact list," Gemma only phrases language within that list, and a pure-Python verifier checks every piece, square, mate, and tactical term, sending failures back for rewriting and falling back to template text after three failures. With facts fed in, the same model got the right move 29/29. The whole system completes a game review in about 90 seconds on a laptop with no discrete GPU, targets Indian chess coaches, and monetizes through GPL self-hosting plus low-cost value-added services.