Paper Deep Dive · Episode 9: Quantum Code Testing, Spatial Mentalizing, and Tool Retrieval Reliability

It's time for our weekly paper deep dive again. Li Bo, I was scrolling through arXiv this week and wow, there's a lot. Quantum code, spatial reasoning, agent tool retrieval — pretty scattered directions. It looks scattered, but there's actually a hidden thread. Several papers this week are asking the same question: does the model know what it doesn't know? Does it know whether its own output is reliable? That's an interesting angle. Let's follow that thread then, starting with the one that surprised me the most — quantum code testing. Okay, paper one, QTestBench. What it's trying to solve is: can large models write unit tests for quantum programs? And note — not writing quantum code, writing tests. Wait, isn't writing tests easier than writing code? Why is quantum so hard? That's exactly the point. Quantum programs are probabilistic, they depend on numerical tolerances, and the criteria for judgment are domain-specific checkers. So a test that's completely correct in format might reach a conclusion that's the exact opposite of the ground truth. So a test can run successfully but that doesn't mean it can actually distinguish a correct implementation from a wrong one. Right — they treat the tests themselves as the object being evaluated. They ran eight PennyLane challenges and ten Qiskit tasks. The generated tests almost all execute, but their actual bug-catching ability varies a lot. How much variation are we talking about? The best generator got test quality of only 0.71 on QHack, and it drops to 0.34 on Qiskit. So 'it runs' seriously overestimates the real quality of the test. I really agree with that conclusion. We've stepped in that pitfall doing AI product evaluations too — the task gets completed, but that doesn't mean it understood the task. And they also found that naturally occurring wrong solutions are too rare, so they use synthetic mutants to get stable estimates. That's the same idea as when you don't have enough data for evaluations, so you generate data. Let's sum up the value of this paper: evaluating AI-generated tests has to look at discriminative power, not just whether they run. So what's number two? Number two, the structured automaton paper. It's about semantic parsing, where the model generates token by token, but legality depends on references and scopes that only unfold during the generation process. Isn't that just constrained decoding? Don't let it generate illegal tokens. You're right, but here's the key point. Local masking can only prevent what's illegal right now. It doesn't guarantee there's still enough length left to complete a legal prefix. So the model might move along smoothly at first, then get halfway and realize there's no path forward. Exactly. They propose accepted-path decoding, where every step guarantees there's still a path to the end. And it doesn't undo tokens that have already been generated. No undoing, and it still guarantees legality? How do they do that? It prioritizes following the token the model wants to pick, as long as that token can still be completed. Only when that path is blocked does it fall back to the next best. When length is tight, it beats local masking by 5.1 points on DRS. That approach is really engineering-minded. Like navigation — instead of recalculating after you take a wrong turn, it keeps an exit open for every path. That analogy is pretty accurate. They also found that invalid outputs actually cluster on the same set of inputs. When training data is scarce, none of eight samples is legal, at 10.2%, and only 0.4% when data is complete. So the problem isn't sampling luck, it's that the input difficulty distribution is uneven. Let's sum this one up: structured constraints need to be tackled from decoding, post-processing, and training all at once. Number three, I think this is the most stunning one this week. SpatialASM — teaching vision-language models to actively shift perspectives when thinking. Actively shifting perspectives? Explain that. When people look at an image, they really do rotate it in their head and look from another angle. Right, and they call this active spatial mentalizing. Current vision-language models are basically passive viewers — they don't re-observe on their own. They use a new perspective generator that anchors imagined viewpoints to real visual evidence. So it's teaching the model when to shift perspective, and how to shift it? Exactly — two kinds of supervision. One is explicit trajectories, teaching it when to invoke perspective planning; the other is perspective-taking tasks, reinforcing perspective-conditioned scene imagination. And the results? Improvements across all four model backbones? Yes, average accuracy improved on seven spatial benchmarks, and their original general vision abilities didn't drop. That's key — specialized enhancement can't come at the cost of general capability. So number three and number two are actually two sides of the same thing — one handles output legality, the other handles reasoning reliability. Right, and they both point to the same trend: the model needs metacognition about its own generation process. And number four ties this hidden thread together the tightest. Is that the one about whether models should abstain from answering? The title alone grabbed me. That's the one. They use a score — correct answers gain points, wrong answers lose points, abstaining is neutral. Then on Qwen3-14B they compare two approaches: modifying the model itself, versus adding an answer selector. And the results? Which wins? The selector wins. On TriviaQA, coverage went from 20.2% to 30.9%, and the error rate among valid answers dropped from 23% to 14.5%. Coverage went up and error rate went down — both metrics improving at the same time, that's not easy. Right, but the important part comes after. The reinforcement learning approach didn't bring stable improvements. And on NQ-Open, although the selector beats confidence filtering, it's still worse than just not answering at all. So reliable abstention doesn't hold up when you switch datasets. I really appreciate that honest conclusion. Number five is about this kind of problem too — agent tool retrieval. Right now retrieval only looks at semantic relevance, without caring whether the tool happens to be down at the moment. Isn't that the most frustrating thing about running agents in production? The semantically best-matching tool turns out to have a dead endpoint, and the whole pipeline collapses. So they built RATED — semantic matching plus a lightweight contextual bandit that uses real-time execution feedback to rerank candidates. The core is fast adaptation to runtime failures without retraining the model. The word 'bandit' sounds a bit intimidating — it's really just adjusting preferences through trial and error? Right, it's dynamically tuning weights. And the results are solid. On real queries, MRR improved 52.2% relative. Under severe failures, the wrong-tool selection rate went from 86.2% down to 2.9%, with less than 1 millisecond of added latency. 86% down to 3% — that number is almost exaggerated. It's like pulling a system that constantly crashes into something basically usable. Right, and they also open-sourced the EcoAgent benchmark — 5,172 real tools, over 210,000 query pairs. For people doing agent work, that's a direct asset. Let's wrap this one up: tool retrieval has to factor in runtime reliability. Static semantic similarity just isn't enough anymore. Let's finish with something lighter. There's a paper where language models play the Wikipedia game — you can only jump through article links to reach a target page. That game is so addictive for humans. And the results — are the models better than people? No model significantly beats humans. The best ones, like o3mini, GPT-4o, QwQ, and DeepSeek-R1, just tie with humans. And rankings shift across different languages and topics. That conclusion is actually quite sobering. Retrieval navigation might not be solvable by just scaling parameters — it might need a different paradigm. Right, and they built a huge local Wikipedia graph that doesn't depend on third-party APIs. That in itself is a practical piece of infrastructure. So from everything we've talked about today, these threads actually converge on one judgment. Right — this batch of research from the second half of 2026 isn't about making models bigger anymore. It's about making models trustworthy, verifiable, and aware of their own boundaries. Tests need to discriminate, decoding needs to guarantee legality, reasoning needs to shift perspectives, answers need to know when to abstain, tools need to avoid risks. In the end, it's all the same thing. It's the model's self-awareness. Its estimate of its own abilities might matter more than gaining another 10 points on any single capability. I'm holding onto that judgment. That's it for today — see you next time.