No Backpropagation Needed? A 0.8B Model Outperforming 2B on ARC-Challenge Sparks Debate

A Claim That Challenges Conventional Wisdom

Backpropagation is the cornerstone of modern deep learning — nearly every large language model is trained within this gradient descent framework. But a technical post on Reddit raises a provocative question: what if we don't actually need backpropagation?

The poster claims to have trained a model with just 0.8B (800 million) parameters that achieves 42.15% on the ARC-Challenge benchmark — not only surpassing the untuned Qwen3.5-2B base model, but significantly outperforming a solution using 25,000 examples of SFT (supervised fine-tuning) distillation. The results were reportedly validated on an NVIDIA L4 GPU.

If this result holds up, it touches on several thought-provoking questions about the current AI training paradigm: Has model scale been overemphasized? Is there still enormous room to optimize the training methodology itself?

The Weight of ARC-Challenge

To appreciate what this result means, we first need to understand the difficulty of the ARC-Challenge benchmark. ARC (AI2 Reasoning Challenge), released by the Allen Institute for AI, is a scientific reasoning QA dataset. The Challenge subset specifically filters for problems that neither traditional retrieval methods nor simple word co-occurrence algorithms can solve — emphasizing genuine reasoning ability over memorization.

On this benchmark, parameter count generally correlates with performance: larger models tend to reason better. That's why the claim of a 0.8B model outperforming a 2B-class model directly challenges the intuition that "scale equals performance." The poster specifically highlights the comparison against a "25k SFT distillation" approach, implying that the method also has a data efficiency advantage.

Why the SFT Distillation Comparison Matters

Supervised fine-tuning distillation is currently the mainstream approach for giving small models stronger capabilities: a small model is trained on data generated by a large model, "imitating" the large model's outputs. This method requires significant labeled or synthetic data (such as the 25,000 examples mentioned in the post). If an alternative training method can achieve better results with fewer resources, the appeal for resource-constrained researchers and companies is obvious.

The ARC dataset was released in 2018 by the Allen Institute for AI and contains approximately 8,000 standardized science exam questions for U.S. grades 3–9. The Easy subset can be answered by traditional IR systems or word co-occurrence models, while the Challenge subset specifically selects questions that both types of systems get wrong — approximately 1,172 test questions in total. Since all questions are multiple choice (four options, with a 25% random baseline), a score of 42.15% indicates significantly above-chance reasoning ability, though it remains far below current top-tier large models (GPT-4-class models can exceed 90% on this test). It's worth noting that ARC-Challenge has gradually shifted from being a "hard" benchmark to a "baseline" benchmark over the past few years — many mainstream models with 7B+ parameters already easily surpass 70%. So 42.15% for a 0.8B model is competitive but not groundbreaking, which is important context when calibrating the original post's claims.

Knowledge Distillation was systematically formalized by Hinton et al. in 2015, with the core idea of having a small model (student) learn from the soft labels or full output distribution of a large model (teacher), rather than just hard labels. In the LLM era, this paradigm has evolved into "data distillation": using a teacher model (such as GPT-4 or LLaMA-3-70B) to mass-generate high-quality instruction-response pairs, then fine-tuning the student model via supervised learning. The "25k SFT distillation" in the post refers to fine-tuning using 25,000 such synthetic examples. This approach isn't cheap — generating 25,000 high-quality examples requires calling large model APIs or self-hosting a large model, and the backpropagation fine-tuning itself demands substantial GPU compute and memory. So if an alternative method can outperform distillation with lower data and compute investment, its practical value would be significant, especially for academic labs and smaller companies.

If Not Backpropagation, Then What?

The central mystery the post's title raises is an alternative training path that bypasses backpropagation. While the original content doesn't go into technical details, this direction is not a new concept in academia. Several lines of research have already tried to challenge backpropagation's dominance:

• Forward-Forward Algorithm: Proposed by Geoffrey Hinton, it replaces backpropagation with two forward passes — one on positive samples and one on negative samples.
• Evolution Strategies and Gradient-Free Optimization: Parameters are updated through sampling and selection rather than gradients.
• Local Learning Rules: Each layer updates based on local information, avoiding global gradient propagation.

What these approaches share is a desire to: reduce memory usage, support more flexible hardware, and potentially align more closely with biological neural system learning mechanisms. Completing training validation on an NVIDIA L4 — a card positioned for inference with relatively limited VRAM — directly echoes the potential resource efficiency advantages of backpropagation-free methods.

Backpropagation is fundamentally an automated application of the chain rule: it computes the partial derivative of the loss function with respect to each parameter, propagating error signals backward through the network layer by layer, then updating weights via gradient descent. This mechanism relies heavily on storing the global computation graph (i.e., caching activation values), which consumes substantial VRAM during large model training — typically 3–4x or more compared to inference. Geoffrey Hinton's Forward-Forward algorithm, proposed in 2022, introduces a contrastive learning objective between "good data" and "bad data," allowing each layer to independently update its parameters based on a local objective function, fundamentally eliminating the need for gradient backpropagation. The potential advantages of such methods extend beyond memory efficiency — they also naturally support inter-layer parallel computation and streaming training, making them theoretically better suited for edge devices or heterogeneous hardware. However, the current academic consensus is that these methods still struggle to match backpropagation's convergence efficiency and final performance on large-scale language model tasks, which is precisely why the original post's claim is so striking.

Staying Cautious: Unverified Questions Behind the Claim

Faced with a claim lacking a complete paper and open-source code, a cautious approach is warranted. Several key questions cannot be answered from the original post:

Evaluation rigor. Is the 42.15% figure from a 0-shot or few-shot setting? Is the evaluation script standard? Different evaluation configurations can lead to wildly different results, and direct comparisons require unified experimental conditions.

Fairness of comparison. "Stock Qwen3.5-2B" refers to a completely untuned base model. Comparing a purpose-trained 0.8B model against an untuned 2B model creates an inherently unequal baseline. A more compelling comparison would be a side-by-side evaluation with equivalent training investment.

Reproducibility. What truly advances the field is results that can be independently reproduced by others. Until the code, weights, and complete methodology are made public, 42.15% should be treated as a noteworthy preliminary signal — not a definitive conclusion.

Implications for the Small Model Trajectory

Setting aside the debate over a single data point, this post reflects a growing trend: sustained industry interest in "small but capable" models. Beyond the arms race of hundred-billion-parameter models, squeezing stronger capabilities from fewer parameters and lower training costs is becoming a compelling parallel track.

Whether or not backpropagation can truly be replaced, the value of this kind of exploration lies in forcing us to reexamine assumptions we take for granted. If there is fundamental optimization headroom in training paradigms themselves, then the current development path — centered on throwing compute at problems — may not be the only answer.

For readers following this direction, it's worth watching whether this project will eventually release a technical report or open-source implementation. Only when the methodology is transparent and results are reproducible can this 0.8B model's score evolve from "an attention-grabbing claim" into "a credible breakthrough."