Recall Fidelity Score
An evaluation metric for quantifying the precision of knowledge internal to an LLM. It measures how accurately the model reproduces details such as dates and numerical values when recalling semantically similar historical events, and can serve as a signal for diagnosing predictive capability and the sources of prediction error.
Timeline (last 90 days)
A recent paper proposes a new metric called the "Recall Fidelity Score" to quantify how internal knowledge precision loss propagates to predictive capability.
The Recall Fidelity Score measures how accurately a model restores event details (such as dates and numerical values) when recalling semantically similar historical events.
The paper conducts empirical analysis on resolved historical prediction market data, and the results show that higher local recall fidelity around semantically similar historical events predicts lower prediction error on resolved markets.
The paper finds that high local recall fidelity also predicts smaller deviation of model forecasts from open market prices, i.e., predictions closer to market consensus.
The paper does not propose a specific improvement method, but notes that improving a model's recall precision for details may indirectly improve predictive performance.
The study's conclusions are based on prediction market data, and whether they generalize to other types of prediction tasks still needs verification.
The computation of the Recall Fidelity Score depends on defining semantically similar events, and this process itself may contain noise.
All Facts (7)
A recent paper proposes a new metric called the "Recall Fidelity Score" to quantify how internal knowledge precision loss propagates to predictive capability.
50%UnverifiedThe Recall Fidelity Score measures how accurately a model restores event details (such as dates and numerical values) when recalling semantically similar historical events.
50%UnverifiedThe paper conducts empirical analysis on resolved historical prediction market data, and the results show that higher local recall fidelity around semantically similar historical events predicts lower prediction error on resolved markets.
50%UnverifiedThe paper finds that high local recall fidelity also predicts smaller deviation of model forecasts from open market prices, i.e., predictions closer to market consensus.
50%UnverifiedThe paper does not propose a specific improvement method, but notes that improving a model's recall precision for details may indirectly improve predictive performance.
50%UnverifiedThe study's conclusions are based on prediction market data, and whether they generalize to other types of prediction tasks still needs verification.
50%UnverifiedThe computation of the Recall Fidelity Score depends on defining semantically similar events, and this process itself may contain noise.
50%