Why More Data Can't Replace Prior Structure: A Deep Interview with Alexander Mattick

There's a prevailing belief in the machine learning community: given enough data and compute, a model can learn anything. But Alexander Mattick, a researcher at the Technical University of Nuremberg in Germany (known as "Zixag" on Yannick Kilcher's Discord), systematically challenges this view in a deep-dive episode of the MLST podcast. Starting from the mathematical history of inference, he covers energy-based models, diffusion models, deep learning theory, constrained reinforcement learning, and world models — with a consistent throughline: information is expensive, and prior structure (constraints) is often more valuable than piling on more data.

Information Is Expensive: Why Constraints Are Indispensable

Mattick's position is unambiguous: "Information can be very expensive — in fact, I would say most information is extremely expensive." If you abandon constraints entirely and refuse to encode any prior knowledge, you face the problem of "having to relearn everything from scratch."

This directly challenges two of Rich Sutton's famous arguments — "Reward is Enough" and "The Bitter Lesson." Sutton argues that you can design an appropriate reward function for any task and don't need to bother with complex structures. Mattick concedes this holds "in the limit" — theoretically, you can define a reward function for almost any problem. But he considers this an "extremely wasteful worldview."

He offers a compelling example: in the appendix of the OpenAI Five (Dota-playing AI) paper, there's a massive table with reward function properties on the left and a column of seemingly arbitrary weight numbers in the middle. "If you told me there was a way to actually derive those numbers, that's impossible." More importantly, those reward functions actually hide a large number of constraints — for instance, "getting a negative reward for staying on the wrong path too long" is essentially a constraint, just crammed into the reward function framework.

The Evolution of Inference: From Rejection Sampling to Flow Matching

The technical backbone of the interview is a lucid history of inference methods. Mattick frames this trajectory as a process of continuously "amortizing" expensive sampling costs:

• Variational Inference: Fits data using parameterized normalized densities (e.g., Gaussians), but is limited by the density form — faced with a bimodal distribution, it can only "pile mass in the middle."
• Rejection Sampling and MCMC: Works without knowing the density itself, relying only on sampling and rejection, but suffers from the curse of dimensionality in high dimensions and is extremely slow.
• Energy-Based Models (EBMs): Directly derived from MCMC, parameterizing unnormalized densities with an energy function. Training is cheap (no complex integrals required), but sampling is expensive — the costliest part is executed repeatedly at inference time rather than amortized during training.
• Diffusion Models: Introduce "infinite latent variables," reformulating the problem as a stochastic differential equation, replacing accept-reject steps with a vector field — at the cost of an added time dimension.
• Flow Matching / Continuous Normalizing Flows: Frames finding the density as a pure optimization problem, eliminating sampling entirely, with only invertibility as the price to pay.

Mattick's key insight is: "Most of deep learning is actually doing inference." Whether it's the autoregressive factorization of language models or regression with mean squared error (which is fundamentally the negative log-likelihood under a Gaussian with fixed variance), these are all different forms of inference.

Amortization is a central concept here, worth unpacking on its own. In statistical inference, "amortization" refers to pre-paying the expensive computation that would otherwise be repeated at every inference step — pushing it into the training phase so that the marginal cost at inference time is dramatically reduced. The problem with energy-based models is precisely that they don't amortize: every time you want to sample from the model, you must run a full MCMC chain, which is extremely costly. Diffusion models and flow matching, by contrast, train a neural network to "memorize" how to generate data from noise, absorbing the cost of sampling once during training so that inference requires only a forward pass. This perspective also explains why autoregressive decoding in language models is fundamentally inference: it uses the trained parameterized distribution to perform amortized posterior sampling over sequences, one step at a time.

Are Energy-Based Models Obsolete?

This is the most contentious part of the interview. Yann LeCun has long championed energy-based models, arguing they are "not generative" and therefore superior. Mattick pushes back directly: "I don't quite understand this claim that energy-based models are not generative… fundamentally they're still generative, just very expensive, which is why nobody uses them that way."

His verdict is sharper still: "I think energy-based models are not really useful anymore, at least not the way they used to be." Exceptions exist — physical settings where energy functions arise naturally, or cases where you only need a "ratio" (whether one point is more likely than another) rather than an absolute score. But in the vast majority of problems, if you want to generate, diffusion models or flow matching are better; if you want self-supervised learning, energy-based models are too broad a formulation to be informative.

The same "branding" critique extends to JEPA. Mattick notes: "There were JEPA-like models before JEPA." He even states bluntly that if DINO were released today, people would call it a JEPA architecture, because the term's definition is too open-ended. On the contrastive vs. non-contrastive learning debate, he agrees with LeCun's direction — contrastiveness alone is insufficient to learn useful models, and the right move is toward mapping to compact latent spaces (as in variational autoencoders and their successors).

JEPA (Joint-Embedding Predictive Architecture) is the self-supervised learning framework Yann LeCun's team has been championing in recent years. The core idea is to make predictions in latent space rather than pixel space: given one part of an input, the model predicts the representation of another part rather than its raw content, avoiding the resource waste of pixel-level reconstruction in generative models. LeCun argues this is more principled than contrastive learning because it doesn't rely on negative samples, while being more efficient than generative models. Mattick's critique targets the term's blurry boundaries: JEPA's definition is broad enough to encompass nearly any model that makes predictions in embedding space — including earlier work like DINO and BarlowTwins — making it more of a retroactive naming framework than a genuine architectural innovation. This echoes his criticism of "world models": when a term's definition is broad enough, its discriminative power approaches zero, and may actually obscure the technical distinctions that truly matter.

Deep Learning Theory: Parameter View vs. Function View

Faced with the proliferating landscape of "deep learning theory," Mattick broadly divides it into two camps. The parameter view treats neural networks as large nonlinear optimization problems, focusing on convergence, global optima, spline regions, and the like. The function view (e.g., neural tangent kernel theory, mean-field theory) treats networks as objects in function space; at the large-scale limit, changes to individual parameters become meaningless — what matters is the evolution of the overall "field." This is analogous to describing water flow with macroscopic properties rather than tracking individual particles.

He leans toward the function view because it better explains the "overparameterization" phenomenon: a billion-parameter function is a closer approximation to the true function space than a fifty-parameter one. But he's candid: the standard for truly good deep learning theory should be "making correct predictions that are falsifiable," and the field's current problem is that "we can't even tell whether there's an implementation bug in a follow-up paper" — he himself once spent three months tracking down an implementation bug. That alone says something about the theory's maturity.

On the manifold hypothesis, his stance is equally cautious: "I don't think it's right and I don't think it's wrong — I think even if it is right, it's not a useful way to think about things." He is skeptical of the robustness of interpretability techniques, believing many rest on silent simplifying assumptions.

The Neural Tangent Kernel (NTK) is the representative theoretical tool of the function view. Its central result is that when neural network width tends to infinity, the network's behavior under gradient descent training is equivalent to kernel regression with the NTK as the kernel function — parameters barely move ("lazy training"), and learning is entirely determined by the function-space structure at initialization. The significance is that it maps a difficult nonlinear optimization problem onto a linearized function space, making properties like convergence precisely characterizable. However, the main critique of NTK theory follows from the same move: real networks are not infinitely wide, and feature learning is precisely the source of deep learning's power — yet the NTK framework allows no feature learning. Mattick's preference for the function view is partly because it can accommodate feature learning as a reality, rather than eliminating the most interesting phenomena through limiting assumptions as NTK does.

Constrained Reinforcement Learning and the Real Limits of World Models

As Mattick's doctoral research focus, constrained reinforcement learning (Constrained RL) embodies his conviction that constraints beat hyperparameter tuning. Compared to adjusting weights inside a reward function, explicitly modeling constraints offers two key advantages: communicability (engineers find it easier to discuss constraints than weights) and composability (adding a new constraint doesn't force you to retune all other weights, because Lagrange multipliers adjust automatically).

He emphasizes that deep RL cannot provide L-infinity (worst-case) guarantees — because everything is sample-based, and sampling is fundamentally incompatible with absolute worst-case error bounds. A controller that is perfect in expectation but fails catastrophically once is indistinguishable from one that always drifts slightly, in terms of expected performance. This is the fatal flaw for real-world deployment.

On the buzzy concept of "world models," Mattick sees it as equally a product of branding: "There were world models before world models." Schmidhuber's original world model paper is essentially model-based reinforcement learning. He proposes a practical distinction: if data collection is inside the loop, call it model-based RL; if it's outside the loop, call it a world model.

His take on the real-world viability of world models is cool and sharp. Against LeCun's claim that "with a perfect world model you don't need RL," he counters: "I can give you a perfect chess world model and you still can't beat Magnus Carlsen." There is a gap between being able to predict the future and being able to infer the correct action. The more fundamental issue is reliability requirements in the real world: "If you go to BMW and say my robot gets it right 90% of the time and wrong 10% of the time, they'll say I'm not paying for that." These methods are "set up to look beautiful in demos," but "how do you know how many times it failed?"

Constrained Reinforcement Learning is mathematically grounded in the Constrained Markov Decision Process (CMDP), which extends the standard MDP with additional cost functions, requiring the policy to maximize cumulative reward while keeping the expected cumulative cost for each constraint below a given threshold. The standard solution method is Lagrangian relaxation: constraints are incorporated into the objective via multipliers, converting the constrained problem into an unconstrained dual optimization, with Lagrange multipliers adjusting automatically during training to satisfy the constraints. This is exactly the source of the "composability" Mattick describes — adding a new constraint only requires introducing a new multiplier, and existing multipliers re-equilibrate during optimization without requiring manual retuning of all reward weights. Yet he simultaneously points out that sample-based RL can, by its nature, only provide guarantees in expectation, not in the worst case (L-infinity norm), which is a fundamental limitation in safety-critical deployment scenarios.

Constraints as Freedom

The interview invokes a quote attributed to Leonardo da Vinci — constraints give us freedom. Mattick analogizes levels of abstraction to control systems: lower levels provide guarantees (invariants), upon which upper levels build capabilities. This is also why hierarchical reinforcement learning has never truly succeeded — lower levels cannot provide sufficient guarantees to upper levels. The difference between a good abstraction and a bad one ultimately comes down to usefulness: we only see the abstractions that succeeded, while many potentially valuable ones were simply forgotten because the right person never picked them up.

The value of this interview lies in its refusal to fixate on praising or dismissing any particular model, and its persistent pursuit of a more fundamental question: in an era of exploding data and compute, do we still need human-encoded structure and knowledge? Mattick's answer is clear and firm — yes, and very much so.