New Insights into LLM Prediction Errors: Recall Fidelity Score Reveals How Internal Knowledge Precision Affects Forecasting

Large language models (LLMs), when recalling past events, often retain only coarse-grained associations while losing specific details—a phenomenon known as "approximate recall" that has been widely observed. But a key question has not been sufficiently studied: how does this loss of precision in internal knowledge propagate to the model's ability to predict future events? A recent paper proposes a new metric called the "Recall Fidelity Score" to quantify this propagation mechanism, and gives an affirmative answer: the higher the local recall fidelity, the smaller the prediction error.

Approximate Recall: A Pervasive Flaw in LLM Knowledge Precision

The paper opens with a vivid example. When asked "Did Anna Sawai win the Outstanding Lead Actress in a Drama Series award at the 76th Emmy Awards?" the LLM correctly answers "yes" but incorrectly records the event date as September 22 instead of the correct September 15. This pattern of "right in general, wrong in detail" is a typical manifestation of approximate recall: the model retains the occurrence and outcome of an event but loses precise information such as time and location.

Prior research has mostly evaluated LLM forecasting ability through prediction markets but has not delved into the connection between internal knowledge fidelity and forecasting performance. This paper fills that gap by treating recall precision as a new lens for diagnosing prediction errors.

Approximate recall resembles the concept of "gist memory" in cognitive science: when humans recall past events, they typically retain the core meaning of an episode while specific details like times and numbers become blurred or even reconstructed. This phenomenon in LLMs partly stems from the compression inherent in parametric memory—vast amounts of facts from training data are encoded into limited parameters, and high-frequency, structured information (such as "a certain actor won an award") is more stably retained than low-frequency, precise values (such as exact dates). As a result, models tend to produce semantically plausible but potentially inaccurate answers during recall, and often do not proactively flag this uncertainty.

Recall Fidelity Score: Quantifying Knowledge Precision

To systematically study this issue, the authors introduce the "Recall Fidelity Score." This metric measures the precision of a model's internal knowledge when recalling semantically similar historical events. Specifically, it focuses on whether the model accurately reproduces event details (such as dates and numerical values), rather than merely judging whether the event outcome is correct.

The design logic is: if a model has high-fidelity memory of past events semantically similar to a target event, it may exhibit lower error when predicting future similar events. In other words, recall precision can serve as a diagnostic signal of forecasting ability.

Prediction markets are mechanisms in which participants bet real money on binary or categorical outcomes, and market prices can be interpreted as the crowd's estimate of event probabilities. Contracts on platforms like Kalshi and Polymarket "resolve" once the event is revealed, and their historical price series and final outcomes constitute natural labeled data for evaluating forecast quality. Researchers commonly use the Brier score or log loss to measure forecast error; the former is the mean squared difference between predicted probabilities and true outcomes (0 or 1), with lower values indicating more accurate forecasts. This paper benchmarks against resolved markets and compares model predictions with deviations from market prices, leveraging precisely this kind of quantifiable error measurement framework.

Empirical Findings: Local Recall Fidelity Predicts Forecast Error

The paper conducts an empirical analysis on resolved historical prediction market data. Results show that around semantically similar historical events, higher local recall fidelity predicts lower forecast error on resolved markets. At the same time, this signal also predicts smaller deviations of model predictions from open market prices—that is, predictions closer to market consensus.

This finding has dual significance. Theoretically, it reveals the transmission path between internal knowledge precision and forecasting ability: detail loss caused by approximate recall directly weakens the model's judgment of future events. Practically, the Recall Fidelity Score can serve as a diagnostic tool to help researchers and developers identify sources of model prediction error and thereby improve knowledge injection or retrieval-augmented mechanisms in a targeted way.

Implications for LLM Forecasting Applications

This research has direct implications for application scenarios that rely on LLMs for event forecasting. For example, in forecasting in finance, politics, and other domains, if a model has vague memories of details of similar historical events, its predictions may systematically deviate. By evaluating recall fidelity, one can to some extent anticipate the reliability of a model's forecasts.

Moreover, this metric provides a new dimension for model evaluation. Traditional forecast accuracy assessments often focus only on whether outcomes are right or wrong, whereas the Recall Fidelity Score focuses on whether the knowledge base supporting predictions is solid. Combining the two can more comprehensively characterize a model's capability boundaries.

The paper does not propose specific improvement methods but points to a direction: enhancing a model's recall precision for details may indirectly improve its forecasting performance. This has reference value for research on techniques such as retrieval-augmented generation (RAG) and knowledge editing.

Retrieval-augmented generation (RAG) compensates for the shortcomings of parametric memory by retrieving relevant documents from external knowledge bases at inference time and injecting them into the context; knowledge editing directly modifies internal model parameters to update or correct specific facts. The two target precisely the two root causes of approximate recall: the former bypasses parametric memory and relies on external precise text, while the latter attempts to make the parameters themselves carry more accurate information. The Recall Fidelity Score can serve as an auxiliary metric for evaluating the effectiveness of these two types of methods—if a certain retrieval strategy or editing operation improves the model's precision in reproducing details of similar historical events, a corresponding decrease in its prediction error should theoretically also be observed.

Limitations and Future Directions

This research is currently based on prediction market data, and whether its conclusions generalize to other types of forecasting tasks remains to be verified. The calculation of the Recall Fidelity Score depends on defining semantically similar events, and this process itself may introduce noise. Future work could explore how to integrate this metric into model training or inference pipelines, and how to use it to dynamically adjust a model's confidence in its own knowledge.

Overall, this paper elevates "approximate recall" from an observed phenomenon to a quantifiable diagnostic tool, offering new ideas for understanding and improving LLM forecasting ability.