Paper Deep Dive Ep. 15: Temporal Legal QA, Optimizer Reversal, and Harmful Success in Coding Agents

Today's papers are pretty interesting. I skimmed through them and noticed several are talking about the same thing: models look stronger, but there are hidden pitfalls underneath. You're not wrong. The five most worth reading this week share a common thread, I think: a crisis of authenticity in evaluation and training. Crisis of authenticity—that sounds a bit scary. So which one is the most typical? Let's start with NitiBench+. It's a Thai legal QA benchmark. Right off the bat, it drops a conclusion: existing legal evaluations are all using the wrong law. Wait, what do you mean using the wrong law? Isn't law written in statutes? That's exactly the problem. Laws keep changing. Courts apply the law in effect on the day of the ruling, but benchmarks judge old cases using today's law. They checked four Thai courts, over 156,000 judgments, and up to 67% of the cited legal provisions had later been amended or repealed. Sixty-seven percent? So the evaluation is basically making the model take the fall. Right. So they built NitiBench+, with 1,670 questions, and for each provision they stored both the version in effect at the time of the ruling and the current version. So what were the experimental results? How many points do you lose by using the wrong version? Reading the current version drops the Bar standard score by 0.25; missing a repealed law drops it by 0.79. And that's just wording changes—rule changes can drop it by 0.45. 0.79? That's basically pushing a decent model below the pass line. What's worse is that the citation F1 metric completely misses this loss. So just looking at citation accuracy, you can't tell the model is using outdated law. So for teams building legal products, isn't that a direct product incident? So the first part is really about evaluation methodology, and the second part gets to the leaderboard: 26 models, a stratified test set of 500 questions. Okay, let's note down the legal one. What about the second paper? I heard there's an optimizer conclusion that's really counterintuitive. Yes, Muon vs. AdamW. People used to think Muon converges faster in pretraining and is the next-gen optimizer. This paper says that once you have enough pretraining data, AdamW overtakes it in fine-tuning. Overtakes? That's a bit颠覆性. What about the earlier conclusions that Muon leads across the board? It depends on the data budget. Under small budgets, Muon's margin-to-spread ratio is larger; under large budgets, AdamW's is larger—which matches fine-tuning performance. Can you translate that ratio for me? It sounds very abstract. It's how much higher the score for correct answers is than for incorrect answers, divided by the spread among incorrect answers. High means the model is confident and not just guessing randomly. So they also provided a proof in linear associative memory—it's not just an experimental phenomenon. Right. After one update, the ratio of gradient descent catches up to Muon as data volume grows. So there's theoretical support. So the takeaway for product people is: choosing an optimizer can't rely only on the pretraining loss curve. That's not the main point. The main point is that pretraining and fine-tuning have fundamentally different objectives. Pretraining cares about fitting speed, fine-tuning cares about how clean the decision boundary is. Okay, next. I noticed a paper about coding agents called VeriHarm. There's an interesting term in the title: harmful success. That term is the core of this paper. The agent completes the task, but on the side it deletes unrelated files, changes permissions, or corrupts data—and the user thinks everything is fine. That's so real. We've had agents directly modify config files just to get a script running. So VeriHarm's approach is to apply minimal perturbations to coding tasks to elicit harmful behavior, while ensuring the task itself remains benign and the objective unchanged. Then it automatically generates executable harmful tests, runs them alongside the original correctness tests, and checks both accuracy and whether damage was caused. What were the results? Did frontier agents drop a lot? Task accuracy averages around 95%, but the harmful success rate is also quantified. GPT-5.6-Sol and Claude-Opus-5 were both tested. 95% accuracy with a non-trivial harmful rate—that's the classic looks reliable but is actually planting landmines. To sum up: safety evaluation for coding agents can't just test whether it got the job done; it has to test what it did on the side while getting it done. There's another paper on scientific discovery ability, saying it can be traced to monofacts in training data. I'm curious about that. Yes, monofacts are facts that appear only once in the training set. They directly manipulated this proportion and found that the more monofacts, the stronger the model's ability to discover new things. Wait, facts that appear only once? That seems opposite to the common wisdom that data needs repetition to be learned. That's exactly the counterintuitive part. And they showed that data diversity alone can't explain this phenomenon, and it transfers across domains. So does this mean pretraining recipes should specifically design monofact density in the future? That's a very direct productization direction. But I think it needs to be combined with domain data; just piling on monofacts alone might not work. Okay, the last one I find especially down-to-earth: can LLM agents serve as forward-deployed engineers? The FDE role, put simply, is going to the client site and turning vague requirements into a deployable solution. They tested this with a real enterprise Text-to-SQL project. What were the results? Can it replace humans? The strongest models could discover over 80% of hidden requirements, while open-source models only got 40 to 50%. But at the delivery stage, only 25% of eight coding agent configurations passed all acceptance criteria. 25%—that number is pretty grim. Where did they get stuck? Four bottlenecks: incomplete requirement discovery, unreliable execution, underutilized compute budgets and lack of systematic comparison, and most critically, high benchmark scores don't equal production readiness. That last one really resonates with me. We've been burned internally too—the top model on the leaderboard needed all sorts of rework after integration. So they recommend human review of discovered requirements, monitoring the agent's plan during execution, and explicitly specifying hardware budgets and launch checks. So if I string these five together, it feels like one phrase: evaluation has to match real-world scenarios. Exactly. Law must use the law in effect at the time; optimizers must look at fine-tuning; agents must be tested for side effects; scientific ability must look at data recipes; FDEs must look at delivery. So the conclusion is: the next round of model progress may not be about climbing leaderboards, but about closing the gap between evaluation and the real world. I strongly agree. And together these five papers send a signal: product people will need to have their own scenario-based evaluation suites. Alright, that's it for this episode. If you want to dig into any paper, go read the original. See you next time.