Machine Learning Beyond Statistical Consistency: A New Formal Account of Hybrid AI

Statistical consistency has long been regarded as the gold standard for convergence in machine learning, but a new paper proposes a stricter definition: true "learning" requires not only that estimates converge to the true probability, but also that the machine can derive this convergence within a formal system. This framework not only redraws the boundaries of machine learning but also offers a unified explanation for the success of modern hybrid AI.

From Computation to Derivation: Redefining "Learning"

The paper's core claim is that a machine has learned a function only when it successfully computes that function—that is, when the value it returns for each input has an appropriate derivation within a formal system. This definition elevates learning from "outputting the correct answer" to "provably correct."

The authors extend Gödel's representability theorem—which originally equates computation with derivation—and apply it to empirical targets. The extension proceeds in two directions: the theorem is generalized to computable real numbers, and data are incorporated into the system as "empirical definitions" that record observed outcomes. This means the formal system no longer deals only with purely logical objects but can accommodate empirical records from the real world, providing a foundation for subsequent convergence proofs.

The cleverness of this approach lies in the fact that it does not abandon statisticality but rather places statisticality under the framework of derivation. The machine must not only "compute correctly" but also "articulate why it is correct."

Gödel's representability theorem is a cornerstone result in mathematical logic. It shows that in a sufficiently strong formal system (such as Peano arithmetic), every computable function has a formula that "represents" it—that is, for every input-output pair, the system can prove that the corresponding formula holds. This theorem establishes a profound equivalence between computation and derivation: whatever is computationally realizable is logically representable. The paper follows this line of thinking by extending "representability" from purely logical objects to computable real numbers and empirical data, thereby providing a formal basis for "learning as derivation." Understanding this theorem is the key prerequisite for grasping how the paper incorporates statistical learning into a derivational framework.

Two Conditions: Truth and Assurance

The paper proves that a machine learns a true probability if and only if two conditions are satisfied.

Truth Condition: The machine's estimate converges to the true probability. This is precisely the standard definition of statistical consistency and the central concern of current machine learning theory.

Assurance Condition: The machine appropriately derives its own convergence within the system. In other words, convergence cannot merely be an empirical phenomenon observed after the fact; it must be a conclusion deducible within the system.

Learning "goes beyond" statistical consistency precisely because of the second condition. Mere convergence without the ability to derive convergence from the records does not constitute true learning under the paper's framework.

Why the i.i.d. Assumption Is Insufficient

A natural question is whether global assumptions (such as the independent and identically distributed assumption) can provide the required derivation. The paper's answer is no. Using PAC learning as an example, the authors show that such global assumptions can only lead to circular reasoning—the assumption itself already presupposes the convergence to be proved and cannot constitute a valid derivation within the system.

This criticism strikes at the soft underbelly of statistical learning theory: the i.i.d. assumption is an imposed constraint on the data-generating process, not a conclusion derived from data records. Under the paper's framework, it cannot satisfy the assurance condition.

PAC learning (Probably Approximately Correct learning) is a computational learning theory framework proposed by Valiant in 1984. Its core idea is that a learning algorithm selects a hypothesis from a hypothesis space such that, with high probability (probably), it achieves low error (approximately correct) on unseen data. The PAC framework typically relies on the i.i.d. assumption to guarantee generalization bounds. The paper's key criticism is that PAC's convergence guarantee is a conclusion about future data deduced from an assumption (i.i.d.), rather than a derivation of convergence itself from observed data records. In other words, i.i.d. is imposed as a premise on the data-generating process, and the system does not genuinely "derive" convergence internally, so it cannot satisfy the assurance condition. This criticism reveals the circular structure between premises and conclusions in statistical learning theory.

The Exception of Finite Ideal Corpora

The paper also presents a positive result: for the probabilities of sentences in a finite ideal corpus, both conditions are satisfied. This provides a concrete formal example of the success of hybrid AI—in controlled, idealized settings, convergence can be derived from the records.

The Dual Meaning of Hybrid AI

The paper calls modern AI "hybrid" and explains this at two levels.

Hybridity at the task level: Modern AI simultaneously performs two types of tasks—probabilistic tasks (such as predicting the next word) and deductive tasks (such as proving and coding). The former relies on statistical regularities, while the latter relies on logical derivation; the two are fundamentally different in nature yet coexist within the same system.

Hybridity at the architectural level: Modern AI systems often consist of two components—a statistical proposer and a deductive verifier—that work together on proving and coding tasks. The proposer generates candidates, and the verifier checks their correctness, forming a loop between statistics and deduction.

The paper's key contribution is to explain both forms of hybridity within a single formal system. If the convergence of the statistical proposer satisfies both the truth and assurance conditions, then its outputs can be incorporated by the deductive verifier into the derivation chain; conversely, the verifier's deductive process also provides within-system constraints on the proposer.

An Explanation for the Success of Modern AI

The paper suggests that modern AI may be effective precisely because it operates in a domain that satisfies both conditions. A real machine can only approximate the true probability—provided that an ideal machine can learn it exactly, which holds only when the truth and assurance conditions are satisfied in that domain.

This view offers a new perspective for understanding the emergent capabilities of large models: success does not come merely from statistical fitting driven by scale expansion, but because the system has, in some sense, learned a derivable structure. Of course, this explanation remains suggestive, and the paper itself positions it as a "clue" rather than a definitive conclusion.

From an engineering perspective, the framework offers inspiration for hybrid architecture design: the coupling of statistical and deductive modules should not be a simple pipeline concatenation but should aim for the proposer's outputs to be acceptable to the verifier at the formal level, thereby forming a derivable loop. This may be closer to the essence of "learning" than merely improving statistical accuracy.

Conclusion

The value of this paper lies not in denying statistical consistency but in pointing out that it is insufficient. By extending Gödel's representability theorem to the empirical domain, the authors set a higher bar for machine learning: convergence must be derivable. The dual hybridity of hybrid AI—in tasks and architecture—receives a unified explanation under this framework. For researchers concerned with the theoretical foundations of AI, this is a direction worth following closely.

Background Supplement

The division of labor between statistical proposers and deductive verifiers is already concretely manifested in contemporary AI systems. For example, in program synthesis and theorem proving, language models generate candidate proof steps or code snippets (proposing), while formal verifiers (such as the Lean and Coq proof assistants) check their validity step by step (verifying). This architecture combines the breadth of probabilistic generation with the rigor of symbolic reasoning, avoiding the hallucination problems of purely statistical methods while also alleviating the search bottlenecks of purely deductive systems. The paper's contribution is to provide a theoretical characterization of this engineering practice: for the proposer's outputs to be accepted by the verifier, their convergence must be derivable within the formal system, not merely empirically accurate.