FinDendro
A financial backtesting benchmark for evaluating the temporal self-knowledge of large language models. It contains dated facts on U.S. macroeconomics, stock prices and Federal Reserve decisions, and uses behavioral measurement to test the consistency between model knowledge boundaries and self-reports.
Timeline (last 90 days)
The FinDendro benchmark contains 2,226 dated U.S. facts from 1999 to 2026, covering three categories of content: monthly macroeconomic series, end-of-month stock prices, and Federal Reserve interest rate decisions.
Each fact in FinDendro is expanded into four types of prompts: direct question-and-answer value questions, Recall questions and Predict questions that pair candidate values with "I know/I'm guessing" declarations (without explicit dates), and Predict questions with a historical statement date.
The experiments designed control groups for persistence and dominance results, keeping the accuracy of random guessing at around 0.27.
On FinDendro's controlled facts, GPT-5 answered 76% correctly but claimed to know only 38%.
Most other frontier models on FinDendro likewise show a bias in which self-reports are systematically lower than actual performance.
Models' knowledge claims can extend beyond the measured knowledge boundary, reaching as far as seven quarters afterward.
When the prompt explicitly declares a historical date and asks the model to answer from a past perspective, most of the model's claims of "foreknowledge" disappear, but accuracy does not change accordingly.
The direction the researchers give is behavioral measurement: not relying on the model's self-description, but actually measuring through controlled question-answering at which time points and on which types of facts the model has reliable knowledge, and then choosing the backtest interval accordingly.
All Facts (8)
The FinDendro benchmark contains 2,226 dated U.S. facts from 1999 to 2026, covering three categories of content: monthly macroeconomic series, end-of-month stock prices, and Federal Reserve interest rate decisions.
50%UnverifiedEach fact in FinDendro is expanded into four types of prompts: direct question-and-answer value questions, Recall questions and Predict questions that pair candidate values with "I know/I'm guessing" declarations (without explicit dates), and Predict questions with a historical statement date.
50%UnverifiedThe experiments designed control groups for persistence and dominance results, keeping the accuracy of random guessing at around 0.27.
50%UnverifiedOn FinDendro's controlled facts, GPT-5 answered 76% correctly but claimed to know only 38%.
50%UnverifiedMost other frontier models on FinDendro likewise show a bias in which self-reports are systematically lower than actual performance.
50%UnverifiedModels' knowledge claims can extend beyond the measured knowledge boundary, reaching as far as seven quarters afterward.
50%UnverifiedWhen the prompt explicitly declares a historical date and asks the model to answer from a past perspective, most of the model's claims of "foreknowledge" disappear, but accuracy does not change accordingly.
50%UnverifiedThe direction the researchers give is behavioral measurement: not relying on the model's self-description, but actually measuring through controlled question-answering at which time points and on which types of facts the model has reliable knowledge, and then choosing the backtest interval accordingly.
50%