Breaking Through the Reproducibility Crisis: Replacing Re-execution with Evidence Chains to Verify Code Results

The Long-Standing Reproducibility Problem Plaguing Research and Engineering

In academic papers, technical blogs, and even open-source projects, authors routinely claim "my code produced these results." But for reviewers, how can they trust such claims? Traditionally, there has been only one brute-force approach — pull down the code, set up the environment, and rerun everything from scratch. This is not only time-consuming and labor-intensive but also frequently fails due to dependency versions, random seeds, hardware differences, and other issues that prevent reproducing the original author's results — leading to endless back-and-forth disputes.

This predicament is nothing new. A 2016 survey by Nature of 1,576 scientists revealed that over 70% of researchers had tried and failed to reproduce others' experiments, and more than 50% couldn't even reproduce their own results. This phenomenon, known as the "Replication Crisis," initially drew attention in psychology and biomedicine, but as computational science has permeated every discipline, irreproducibility at the software and data analysis level has become an even more pervasive problem. Organizations like ACM and IEEE have published guidance frameworks for computational experiment reproducibility, and some top conferences (such as NeurIPS and ICML) now require authors to submit code and reproducibility checklists — though enforcement remains inconsistent.

Recently, a Show HN project that made it to the Hacker News front page attempted to tackle this pain point head-on. Its core proposition cuts right to the heart of the matter: Prove your code produced your claims without making reviewers rerun it. This seemingly paradoxical goal actually touches on an increasingly critical issue in computational science and software engineering — Reproducibility.

Why Re-execution Verification Is Failing

The Four Root Causes of the Reproducibility Crisis

The assumption that "rerunning equals verification" is becoming increasingly untenable. The reasons are multifaceted:

• Environment Drift: Subtle differences in Python packages, CUDA versions, and OS-level libraries can cause identical code to produce different outputs. Environment Drift is a classic challenge in software engineering — even patch-level version upgrades (e.g., NumPy 1.24.0 to 1.24.1) can alter floating-point precision or sorting algorithm stability. CUDA toolchain version differences are particularly pronounced: different versions of NVIDIA's cuDNN library use different optimization strategies for convolution operations, causing the same neural network to produce different intermediate results on identical inputs. Containerization technologies (such as Docker) mitigate this problem but cannot fully eliminate it, since GPU drivers inside the container still depend on the host machine's version.

• Prohibitive Computational Cost: Training a large model can take days or even weeks and cost thousands of dollars in compute. Asking reviewers to "rerun it" is simply not economically feasible.

• Non-deterministic Computation: GPU floating-point non-determinism and improper random seed management can make results impossible to reproduce exactly. This non-determinism isn't a design flaw — it's a byproduct of performance optimization. When modern GPUs perform Parallel Reduction operations, the order of floating-point additions depends on thread scheduling, and floating-point addition doesn't satisfy the associative property — meaning (a+b)+c and a+(b+c) can yield different results under finite precision. For example, when summing a tensor with a million elements, different thread scheduling orders can cause the final result to differ in the last few significant digits. NVIDIA provides environment variables like CUBLAS_WORKSPACE_CONFIG to force deterministic execution, but this typically incurs a 10%-30% performance penalty and is rarely adopted in large-scale training.

• Inaccessible Data: Many results depend on proprietary or massive datasets that reviewers simply cannot obtain.

For these reasons, the widely discussed "reproducibility crisis" has erupted in academia — a large number of published computational experiment results cannot be independently verified. The same problem exists in engineering: how can you ensure that a CI pipeline artifact or a performance benchmark report genuinely came from the code under review, rather than from manually modified data after the fact?

"Proving" with Evidence Chains Instead of "Rerunning"

From Trust to Verifiable Computation

This project's shift in thinking is highly instructive: rather than having reviewers bear enormous costs to reproduce results, let authors provide a verifiable evidence chain. The philosophy behind this aligns with the rising "Verifiable Computation" movement.

Verifiable Computation is an active research area in cryptography and theoretical computer science. Its core idea is that a computationally limited Verifier can efficiently check whether a powerful Prover's claimed computation result is correct, with verification costs far lower than re-executing the computation itself. This concept is closely related to Zero-Knowledge Proofs and Interactive Proof Systems. In recent years, zero-knowledge proof systems such as zkSNARKs and zkSTARKs have been widely adopted in the blockchain space, with their core capability being precisely "verifying the correctness of computational results without re-executing the computation." While the computational overhead of these cryptographic schemes is still too high for scientific computing scenarios, they represent the theoretical direction of the ultimate solution.

The core technical approaches include the following:

Provenance Tracking

Comprehensively recording the inputs, environment, dependency hashes, and execution steps of a code run to form a tamper-proof provenance record. Reviewers can examine this record to confirm that "the output was indeed produced by this code in a specific environment."

The concept of provenance tracking originated from Data Provenance research in the database field, aiming to answer the fundamental question: "Where did this data come from?" In scientific computing, the W3C's PROV standard provides a data model for describing relationships between Entities, Activities, and Agents. In practice, tools like noWorkflow and ReproZip can automatically capture system calls, file I/O, and library dependencies during Python script execution, generating a complete execution graph. The Linux kernel's eBPF mechanism and audit subsystem (auditd) also provide low-level support for system-level provenance, making even non-invasive (no source code modification required) provenance collection possible.

Cryptographic Commitment

By computing hashes of the code, input data, and output results and binding them with signatures, any post-hoc tampering can be detected.

Cryptographic commitment schemes typically consist of two phases: the Commit Phase and the Reveal Phase. In this scenario, the author computes cryptographic hashes (e.g., SHA-256) of the source code and input data before execution, then similarly hashes the output results after the run completes. The code hash, input hash, output hash, and timestamp are then digitally signed together. More advanced schemes may use a Merkle Tree to organize numerous intermediate states into a hash tree, allowing verifiers to efficiently check any intermediate step without downloading all the data. Blockchain-based timestamping services (such as OpenTimestamps) can further provide non-repudiable temporal anchors, proving that a particular commitment existed before a given point in time.

Deterministic Execution Environments

Using containerization, locked dependency versions, fixed random seeds, and other techniques to build a trusted execution sandbox, with execution traces submitted as evidence.

In other words, the focus of verification shifts from "recomputing results" to "verifying the integrity of a proof" — the latter costs far less, often requiring just a few seconds of hash verification rather than hours of rerunning.

Applicable Scenarios and Real-World Limitations

Four High-Value Use Cases

These types of tools provide significant value in the following scenarios:

1. Academic Paper Review: Reviewers don't need to set up complex environments to confirm that the tables and charts submitted by authors genuinely came from their code.
2. AI Model Benchmarking: When training costs can reach millions of dollars, replacing rerunning with evidence chains dramatically lowers the verification barrier.
3. Performance Claims in Open-Source Projects: When project maintainers claim "30% performance improvement," they can attach verifiable benchmark evidence to bolster community trust.
4. Enterprise Compliance Auditing: When data processing pipeline outputs must meet compliance requirements, provenance records can serve as audit documentation.

Boundaries That Must Be Acknowledged

"Proving that results were produced by the code" is not the same as "proving the code itself is correct." These are two distinct levels of the problem:

• This approach can prove "this code did indeed produce this number";
• But it cannot prove "the logic of this code is correct," nor can it prevent authors from deliberately writing flawed code that happens to produce self-consistent results.

Furthermore, the credibility of the evidence chain is highly dependent on whether the proof generation process itself is trustworthy. If authors can forge provenance records, the entire mechanism collapses. Therefore, how to anchor proof generation to a trusted execution foundation that cannot be manipulated by the author is the key challenge for deploying such solutions.

Currently, there are two main technical paths to address this trust foundation problem. The first is based on Trusted Hardware, such as Intel SGX (Software Guard Extensions) and AMD SEV (Secure Encrypted Virtualization), which create hardware-protected isolated regions (Enclaves) at the CPU level — even if the operating system is compromised, computation inside the enclave cannot be tampered with. SGX supports "Remote Attestation," allowing remote verifiers to confirm that code was indeed executed within genuine SGX hardware. The second path is based on trusted third-party CI environments, such as GitHub Actions and GitLab CI — when build logs and artifact hashes are generated and signed by a trusted third-party platform, their credibility is significantly higher than evidence self-submitted by the author. GitHub's recently launched Artifact Attestation feature is precisely this approach put into practice.

The Bigger Picture: Trust Infrastructure for Scientific Computing

Judging by its quiet debut on Show HN, this project is still in its early stages, and community discussion has not yet fully developed. But the direction it points to is exactly the proposition the entire computational science ecosystem is exploring — how to build scalable trust mechanisms for "claims derived from computation."

Whether it's experiment tracking platforms like Weights & Biases and MLflow, the Jupyter ecosystem's efforts toward reproducible Notebooks, or academia's promotion of "Reproducibility Badges" — all are trying to answer the same question. Weights & Biases (W&B) and MLflow are currently the two most mainstream experiment tracking platforms in the machine learning field. W&B provides cloud-hosted experiment logging that automatically captures hyperparameters, model metrics, system resource usage, and code versions, and has been widely adopted by organizations like OpenAI and DeepMind. MLflow, open-sourced by Databricks, offers full lifecycle management capabilities including experiment tracking, model registry, and deployment. However, these platforms primarily solve "experiment management" rather than "verifiable proof" — they record data self-reported by experimenters but do not guarantee that this data hasn't been tampered with. DVC (Data Version Control) takes a data version management approach, using Git-like methods to version-control and hash-verify data and models. Academia's Reproducibility Badges (such as ACM's Artifacts Evaluated badges) grant reproducibility certification through independent verification by peer volunteers, but this process still relies on manual rerunning and has limited scalability.

The idea of "replacing rerunning with proofs" offers a pragmatic and economical complementary perspective: in an era of increasingly expensive compute and ever-larger models, the efficiency of verification itself is a scarce resource.

Conclusion

The proposition "Prove your code produced your claims without making reviewers rerun it" captures a real pain point in contemporary research and engineering collaboration. It reminds us that as computational scale grows exponentially, the naive verification approach of "rerun everything yourself" is going bankrupt. Future trust may increasingly be built on verifiable evidence chains rather than redundant labor.

For developers and researchers, tools like these are worth continued attention — they could become indispensable "trust infrastructure" in the next generation of scientific computing and AI engineering. Whether they will truly be accepted by the community depends on the credibility and usability of the proof mechanism, and whether it can seamlessly integrate with existing workflows. There is still a long road ahead, but the direction is undoubtedly right.