Paper Deep Dive · Ep. 11: Agents' Geometry Blind Spot and the World Model Memory Revolution

So I went through this week's batch of papers, and one thing really jumped out at me. Everyone's giving agents remedial lessons, and they're all the same subject. You're not wrong. On the surface this week's papers are about proteins, CAD, robot navigation. But really they're all asking one thing: does the agent actually understand the world it's operating in? Then let's not dance around it. Start me on the one that makes the point clearest. ProteinGeoBench. Let me give you the punchline first: frontier models are great at looking things up, and terrible at doing geometry. Hold on. Looking things up and doing geometry — aren't those both things it should be good at? That's exactly the point. Out of 131 protein structure tasks, three of the four frontier models were better at metadata lookup than numerical computation. The fourth was the reverse. So is the score gap big, or are they all bunched together? Three of them are squeezed between 0.785 and 0.816 — statistically you can't tell who's stronger. The fourth drops to 0.707, and that's the real gap. So leaderboard rankings are kind of a crapshoot to begin with. Right, and that's the most interesting part. If you only run each task once, in 23.9% of resamples the top spot changes hands. Let me write that number down. So change the eval budget, and the champion changes. And one more detail: numerical tasks are only 28 of them, but of the 12 tasks all four models failed, half were numerical. Numerical is the hard bone. If you swap in your own command-line agent, how much does the score move? It swings from minus 0.071 to plus 0.086. So the eval harness itself is quietly shaping the conclusion. So what's the practical value here? It can't just be 'don't trust leaderboards.' Flip it around and it's a product opportunity. For a structural biology agent, numerical computation has to be its own tool, verified separately. Don't expect the model to do mental math. Got it — being able to look it up doesn't mean it can compute it right. Okay, next up, AnyGoal — I really want to talk about this one. It hits a soft spot in world models. States and actions seen in expert trajectories — separately, they're familiar. Combine them and it falls apart. Isn't that exactly what we dread most in product work? Fine on each piece, then it breaks when you chain them. Exactly the same thing. And the authors emphasize these combinations are still within the training distribution — not totally unseen new stuff. So within the training set's range, it's already getting confused? Yep. Across multiple environments and several world model families, this recomposition failure shows up consistently. It's not a quirk of one model. So does this mean today's world models are learning memorization, not dynamics? You've nailed it. It's more like it memorized the trajectories than extracted the rules. Change the goal and it's exposed. So the conclusion is, if world models are going to be actually useful, compositional generalization has to be the core metric they're trained on. Right. Otherwise planning is a castle in the air. Okay, let's talk MemT-1.0 — I was pretty excited after reading this one myself. The name alone sounds like it's gunning for retrieval augmentation's lunch. It's aiming bigger. It makes memory part of the model's computation, instead of bolting on a vector store. Wait, so what does it use for memory? It can't still just be stuffing things into context. Gated DeltaNet-2 handles token-level memory updates, plus a Mixture-of-Storage to expand capacity. Basically the model carries its own notebook. And the results? What I care about most is cost over long histories. At 128K history length, the Nano version's end-to-end inference is about 1.86x faster, and persistent storage is 66x smaller than a full KV cache. That storage compression ratio is brutal. More importantly, it internalizes retrieved experience into the model, and agent task success rates go up too. That's where it pulls away from RAG. So it's stronger than retrieval augmentation on average, while general capability basically holds? The material says it beats parametric memory baselines on average, ties with retrieval augmentation, and largely keeps general ability. A pragmatic step. Speaking of which, the CAD paper has the same root problem. Looks like it runs, but physically it's a mess. FunCADBench, 46 mechanism families. The core trick is FunCAD-Harness — it translates vague requirements into physics tests tailored to each candidate design. That's clever. Different mechanisms shouldn't be validated with the same yardstick anyway. Right, it drives iteration with simulation feedback. It catches mechanical failures you can't see with the naked eye, and functional success rate is clearly higher than the standard simulation baseline. Then WildAWM — building training environments for computer-use agents. I like this angle. It fuses GUI, MCP services, and command line into one executable environment. And it's driven entirely by open models — no closed frontier models. How many tasks, and how hard? 341 long-horizon environment-task pairs. Some take hundreds of steps to solve. Even GPT-6-Astra only solves 60%. And it specifically tackles three old problems: whether a task is solvable, whether it's hard enough, and whether the verifier is reliable. Using execution-quality gates — heterogeneous rollouts plus cross-judge consistency, plus counterfactual probes. It's quite engineering-heavy. Let me tie today's five papers together. From proteins to CAD to computer use, they're all saying one thing. Go on. Agents can look up, look at, and talk. But the moment they have to compute on their own, combine on their own, be physically validated — the blind spot shows. And one more thing: evals have to change too. Single-run scores can't be trusted — compositional generalization and cost efficiency are the real metrics. Internalized memory, I'd bet, becomes one of the main lines of agent architecture next year.