LLMs Don't Know What They Know: The Temporal Boundary Dilemma in Financial Backtesting

Using large language models for financial backtesting rests on the premise that the model knows nothing beforehand about the outcomes it is meant to predict. Otherwise, the "excess returns" from a backtest are nothing more than an exam the model has already memorized the answers to. How do you confirm this premise holds? The industry typically relies on three approaches: trusting the knowledge cutoff dates declared by vendors, directly asking the model what it knows, or instructing the model to "pretend to stand at some point in the past."

A study focused on the financial backtesting scenario systematically tested these three approaches using a new benchmark called FinDendro, and the conclusions are rather discouraging: on most frontier models, the model's self-assessment significantly underestimates its own knowledge—it knows far more than it admits.

FinDendro: Turning "the Model's Knowledge Boundary" into a Measurable Object

The FinDendro benchmark built by the research team contains 2,226 dated U.S. facts spanning 1999 to 2026, covering three categories of content: monthly macroeconomic series, month-end stock prices, and Federal Reserve interest rate decisions. These happen to be the most commonly used data types in quantitative backtesting.

Each fact is expanded into four types of prompts: a direct question-and-answer value question; Recall and Predict questions that pair candidate values with "I know / I'm guessing" declarations (without explicit dates); and Predict questions with historical declaration dates. To rule out the possibility of lucky guesses, the experiment designed control groups targeting persistent and dominant outcomes, keeping the accuracy of random guessing at around 0.27.

The key to this design is that it strips "whether the model knows" away from subjective statements and turns it into comparable behavioral data—what the model says doesn't matter; what matters is how many it answers correctly under controlled conditions.

Knowledge Cutoff Date is the upper time limit of training data declared by model vendors in their documentation, typically phrased as "the model's knowledge cuts off at a certain year and month." The industry widely treats this as a hard boundary of model capability: in theory, a model should know nothing about events after the cutoff date, making it safe to backtest in periods after that date. However, this assumption has two pitfalls. First, the cutoff date is a rough declaration made by vendors based on the temporal distribution of training corpora, not a precise annotation of each individual fact—a small amount of content later than the declared date may be mixed into the corpus. Second, the model's self-assessment of "what it knows" is itself unreliable; even if a fact is later than the cutoff date, the model may actually possess it due to training data leakage or parametric memorization. The FinDendro study approaches from this second pitfall, using behavioral measurement in place of documentation declarations to examine the degree of misalignment between the declared boundary and the true capability boundary.

Core Finding: Self-Reports Severely Underestimate Actual Knowledge

The most striking number in the experimental results comes from GPT-5: it answered 76% of controlled facts correctly but only claimed to know 38%. More subtly, among the facts it explicitly stated it "didn't know," 61% were still answered correctly. According to the paper's data, most other frontier models exhibit the same directional bias—self-reports are systematically lower than actual performance.

This means that the seemingly intuitive verification method of "asking the model whether it knows" can provide almost no safety guarantee in financial backtesting scenarios.

The study also found that knowledge claims can extend beyond the measured knowledge boundary by up to seven quarters. In other words, the model will claim to possess relevant information during periods in which it actually no longer has reliable knowledge. For practitioners who need to precisely define the start and end points of a backtest, a seven-quarter deviation is enough to contaminate a supposedly "clean" backtest window with training data leakage.

Metacognition in cognitive science refers to an individual's ability to monitor and evaluate their own cognitive processes. Applied to large language models, it refers to the model's ability to self-judge "what I know, what I don't know, and how confident I am." Ideal metacognition should make the model's self-reports highly consistent with actual performance: answering correctly when it says it knows, and answering incorrectly when it says it doesn't. FinDendro's Recall and Predict questions are designed precisely to measure this consistency—pairing candidate values with "I know / I'm guessing" declarations, then comparing against actual accuracy. The results show that frontier models have systematic biases in metacognition: self-reports are significantly lower than actual performance, meaning the model "underestimates" rather than "overestimates" its knowledge. This is a bias in the opposite direction from the "overconfidence" commonly seen in hallucination research, indicating that the model's access to its internal knowledge state is neither precise nor uniformly biased, and cannot be simply summarized with the word "calibration."

Declared Dates: Confidence Eliminated, But Not Knowledge

A counterintuitive result: when prompts explicitly declare a historical date and ask the model to answer from a past perspective, most of the model's claims of "prior knowledge" disappear—but accuracy does not change accordingly.

This reveals the inherent flaw of instruction-based control—what the model changes is only its self-presentation at the wording level; the underlying factual memory remains active. Asking the model to "pretend not to know" does not make it actually not know. Similarly, vendor-declared knowledge cutoff dates are insufficient as proof of leak-free backtesting: the existence of knowledge claims beyond the measured boundary demonstrates a systematic misalignment between the declared boundary and the true capability boundary.

The conclusion from combining all three is clear: backtesting within the model's training window cannot obtain "leak-free" certification through self-reports, instruction constraints, or official cutoff dates.

Instruction-based Control refers to using prompts to require a model to follow a certain role or constraint, such as "Please pretend you are standing in January 2020 and answer only using information available at that time." This method is widely adopted in backtesting scenarios, with the implicit assumption that the model can follow instructions to block subsequent knowledge acquired during training. But FinDendro's results show that instructions change the model's wording and self-presentation, not the factual memory stored in its underlying parameters. After being told a historical date, the model reduces claims like "I knew this in advance," yet still answers those "future" facts with the same accuracy. This phenomenon is consistent with the "role-playing does not change capability" conclusion in LLM research: prompts can adjust output style but cannot truly sever already-encoded knowledge. For backtesting practitioners, this means "making the model pretend to go back in time" does not constitute leak protection; true isolation must come from the training data level or from post-hoc behavioral measurement.

Direct Implications for Quantitative Practice

This study raises a methodological question for teams using LLMs for strategy backtesting. Traditional backtest leakage prevention focuses mainly on the data itself—ensuring that timestamps for prices and financial data are correct and avoiding look-ahead functions. But the model's internal memory constitutes a new type of leakage source: it does not appear in the data pipeline, nor can it be found through code review.

The direction proposed by the researchers is behavioral measurement: rather than relying on the model's self-reports, use controlled question-answering to actually measure at which points in time and on which types of facts the model possesses reliable knowledge, then select backtest intervals accordingly. FinDendro provides both a benchmark dataset and a process for selecting backtest periods, with the idea of treating the "knowledge boundary" as a parameter that must be calibrated through experiments, rather than a configuration item that can be looked up in documentation.

For practitioners, one actionable insight is: construct small-scale, date-anchored question-answering tests for specific asset classes and macroeconomic indicators, and use measured accuracy rather than vendor declarations to define credible backtest windows. Such tests are inexpensive but can prevent treating a period whose answers the model has already "seen" as out-of-sample validation.

Look-ahead Bias is the most classic type of error in backtesting, referring to using information that was not available at the time in decisions made at a given point, thereby inflating backtest returns. Traditional prevention focuses on the data pipeline: ensuring every price and financial indicator carries the correct timestamp and avoiding mixing future data into historical cross-sections. What FinDendro reveals is a new type of look-ahead bias—the leakage source is not in the data files but in the model parameters. Historical facts that the model "read" during training silently influence its judgments during backtesting, undetectable by code review or data validation. Therefore, preventing this type of bias requires treating the model itself as a data source that needs auditing: first measure the model's mastery of relevant facts within the target period, then set the backtest window outside the measured knowledge boundary. This shares the same logic as traditional look-ahead bias prevention—decisions may only use information available at the time—except that the object of inspection is extended from the data table to the model's interior.

A Deeper Issue

This study touches on a fundamental limitation of LLM metacognitive ability. The model's reports about its own knowledge state are unreliable—not only imprecise, but consistently biased in one direction: toward underestimation. This forms an interesting contrast with the common "model hallucination" narrative: models fabricate in factual question-answering, and models are modest in self-assessment—both indicate that models lack accurate access to their internal knowledge states.

For domains like finance that are extremely sensitive to causality and timeliness, the impact of this limitation is amplified. The point of backtesting is to use history to test how a strategy performs in an unknown future; once history itself is memorized by the model, the entire validation logic loses its fulcrum. The behavioral measurement path advocated by the researchers is essentially rebuilding an externally verifiable method of knowledge auditing on the premise that models cannot self-report.

As model capabilities continue to expand and training corpora increasingly cover more recent data, this auditing process will likely shift from an optional best practice to essential infrastructure.

One-line Summary

FinDendro benchmark reveals that LLM self-reports systematically underestimate knowledge, and all three backtest leak-prevention methods fail.

Paragraph Summary

The research team built the FinDendro benchmark using 2,226 dated U.S. macroeconomic, stock price, and Fed decision facts from 1999 to 2026 to systematically test three common methods for confirming that a model is "uninformed in advance" in financial backtesting. Results show: GPT-5 answered 76% of controlled facts correctly but only claimed to know 38%, and answered 61% of facts it claimed not to know correctly—self-reports systematically underestimate actual knowledge; knowledge claims can exceed the measured boundary by up to seven quarters; explicitly declaring historical dates eliminates most claims of prior knowledge but does not change accuracy, showing that instructions change only wording, not memory; vendor cutoff dates likewise cannot guarantee leak-free backtesting. The study advocates abandoning reliance on self-reports and documentation in favor of controlled question-answering behavioral measurement to calibrate the model's temporal knowledge boundary, and selecting credible backtest windows accordingly.