Are AI-Generated Quantitative Factors Reliable? Good and Bad Cases and Core Pitfalls

In the field of quantitative investing, using large language models (LLMs) to automatically generate factor hypotheses has become a cutting-edge exploration. The RD-AGENT project systematically dissects what AI-generated factors actually look like through multiple real-world campaigns (covering A-shares, futures, and cryptocurrency markets, totaling 290 factors). As the conclusion of the first chapter of this series, based on cross-validation across multiple independent campaigns, this chapter presents a core judgment: AI's coding ability is far more honest than its financial intuition. Code bugs are explicit—if it doesn't run, it throws an error; but logical emptiness is implicit—no matter how hollow the hypothesis, the backtest still spits out a number, and that number then enters the leaderboard to participate in selection.

I. Decent Factors: Data Boundaries Determine the Upper Limit of AI's Taste

In the futures panel, the factors that truly possess industrial logic are concentrated on two columns of specialty data provided by humans: basis (spot-futures price spread) and open interest. For example, based on term structure hypotheses—deep discounts reflecting delivery pressure or pessimistic expectations—AI can construct basis z-scores for mean reversion or directional exposure. Such factors deserve recognition, but the way they are produced warrants caution: the data boundaries are drawn by humans. Humans put basis and oi into the panel, and only then does AI have material to think in that direction; without these columns, its hypothesis space degenerates into pure price-volume.

This phenomenon recurs across multiple campaigns: the hypotheses AI proposes with "the most financial flavor" almost always circle around the specialty data that humans have prepared. According to the analytical consensus of Bilibili UP hosts and YouTube channels, the upper limit of AI's research taste is determined by the quality of the data panel. What it does is "efficient search within a given material space," not "opening up new material spaces."

A Counterexample Note: A +18% Illusion

In the basis/term structure direction, the factory once produced a carry strategy with an annualized +18%—the most eye-catching number across all campaigns. Subsequent auditing revealed this was a double adjustment bug: returns were deducted twice at roll dates during panel construction. After correction and recalculation: +3.18%, t=0.66, not significant. From "nearly deployable" to "statistical noise," there was only one audit of data conventions in between.

Before data conventions are verified, the numbers that look most real are often the most dangerous. The reason +18% almost fooled everyone is precisely because it "looked too reasonable"—term structure carry is supposed to have returns, and the bug-amplified number fell within expectations.

Double adjustment refers to applying the same price adjustment twice when constructing a return panel. Futures contracts have a roll problem: when splicing continuous contracts, roll gains/losses need to be handled, and a common approach is to use adjusted price series. If roll adjustment is already done once when generating continuous prices, and then the spread is deducted again on roll dates when calculating daily returns, the return on the roll day will be corrected twice. Such bugs do not throw errors; instead, they amplify the apparent performance of carry-type factors in direction—because term structure factors are expected to have returns around roll dates, and the amplified number happens to fall within a "reasonable range," making it extremely difficult to identify with the naked eye. The identification method is to compare the return distribution on roll days versus non-roll days, or to directly recalculate using unadjusted prices for cross-checking.

II. Mediocre Factors: Three Characteristics of Pseudo-Logical Hypotheses

A large number of mediocre factors exist in the middle of the leaderboard. Represented by turn_of_month_rank (turn-of-month effect) and overlay_reversal_liquidity (reversal × liquidity overlay), their self-reported excess returns are as high as +26.5% and +14.9%, respectively. But such factors often possess the three characteristics of pseudo-logical hypotheses:

1. Excessive degrees of freedom: Two known phenomena (short-term reversal, liquidity premium) multiplied or intersected, with parameters (lookback window, filter threshold, quantile boundary) all adjustable. The larger the parameter space, the easier it is for the backtest to find a good-looking cell.
2. Good story but vague mechanism: "Illiquid stocks have stronger reversal because arbitrage capital can't get in"—sounds reasonable, but it can simultaneously explain the opposite result ("illiquid stocks have stronger momentum because information diffuses slowly"). A mechanism that can explain any result equals no mechanism at all.
3. Many evaluations: turn_of_month_rank underwent 15 evaluations and 14 versions. Repeatedly looping back to it is itself a signal that "the number was ground out through many attempts."

Compare with a positive example: after two rounds of cryptocurrency campaigns, the true signals confirmed by independent verification were vol_20d (volatility) and reversal_5d (5-day reversal). These are classic phenomena that have been repeatedly validated in the literature for decades. AI looped around a huge circle, and what ultimately survived were still textbook factors. This is not a coincidence: textbook factors have undergone decades, dozens of markets, and countless researchers' "multiple testing," while newly created overlay factors within the loop have only undergone this one selection on the leaderboard.

For novel combination factors generated by LLMs, the default assumption is "product of parameter search"; the burden of proof lies with the factor. Classic phenomena that have withstood cross-market, cross-era testing naturally have a higher prior.

Multiple testing is the most easily overlooked statistical trap in quantitative research. When researchers repeatedly try different parameters, different combinations, and different filters on the same dataset, even if the true signal is zero, they will almost inevitably find a "seemingly significant" result. Assuming each independent test has a 5% probability of producing a false positive at the 5% significance level, after 20 independent attempts, the probability of at least one false positive appearing is about 64%. The leaderboard is essentially a multiple testing generator: it displays the maximum value among all attempts, not the expected value. Therefore, the number of evaluations itself is the factor's "hidden cost," and Bonferroni or FDR correction should be applied when using t-statistics, or offset through out-of-sample/pre-registered validation.

III. Failed Factors: The Asymmetry Between Explicit and Implicit Errors

Failures in the loop fall into two types, with vastly different pedagogical value.

Non-running type: code errors (alignment errors, missing columns, boundary overflow). AI handles this type very well—reading error messages, locating, auto-fixing, rerunning—engineering-wise impeccable.

Implicit failure type (far more dangerous): code runs, backtest numbers are perfect, but logically it uses future data (look-ahead bias) or convention misalignment. The factory itself has no warning mechanism whatsoever—it looks exactly the same as good factors on the leaderboard, and can only be exposed through independent verification. All 136 experiments in the first round of futures were eliminated under pre-registered validation, and some of them belonged to this category.

Code errors are explicit, with immediate feedback, and AI can self-repair; logical errors are implicit, with no feedback, and AI is completely unaware—if humans don't verify, no one knows. The loop accelerates the death of the first type of error while accelerating the reproduction of the second type.

Look-ahead bias refers to using information that could only be obtained after a certain point in time when calculating a factor value at that point. Common forms include: calculating signals using the closing price of the day but executing at the opening of the day, using data before financial report publication but aligning by financial report period, and standardizing using full-sample statistics. The terrifying thing about look-ahead bias is that it almost always improves backtest results—because the factor "knows the future in advance." It does not trigger code errors, nor does it leave abnormal traces in single-factor metrics; it can only be discovered through strict timestamp auditing, line-by-line checking of data alignment logic, or using walk-forward validation.

IV. An Honest Portrait of the AI Factor Factory

Combining the independent verification results from multiple campaigns, a clear evaluation framework can be given:

• Code implementation: Strong. High quality, self-repairable, output speed unmatched by humans.
• Hypothesis generation: Medium. Efficient search within given data materials, rediscovery of classic phenomena is reliable, novel hypotheses are mostly high-degree-of-freedom recombinations.
• Data conventions: Weak. Problems like double adjustment and benchmark artifacts are completely imperceptible to AI.
• Result credibility: No built-in mechanism. The leaderboard is a multiple testing generator, and the top rank is untrustworthy.
• The real bottleneck: Not "whether AI can think of factors," but "whether humans will verify."

290 factors, self-reported excess returns peaking at +26.5%, zero survivors under independent verification—this conclusion reveals AI's true position in quantitative research. It is not a tool to replace human research taste, but rather amplifies human responsibility in data preparation and verification. If the loop automatically filters high-degree-of-freedom pseudo-logical hypotheses at the generation stage, checkable features include: number of adjustable parameters, whether the hypothesis can be falsified by the opposite result, and whether it cites specialty data columns.

Pre-registration originates from clinical trial methodology: before viewing results, first lock down in writing the hypothesis, sample period, evaluation metrics, and significance threshold, then run the data. Its core function is to cut off the retreat path of "adjusting conventions after seeing results," thereby eliminating false positives caused by multiple testing. In quantitative factor research, pre-registration means that factor logic, parameters, and backtest plans must be validated in one shot on independent data, with no repeated fine-tuning within the loop. All 136 experiments in the first round of futures were eliminated under pre-registered validation precisely because pre-registration blocked the path of "trying a few more times will eventually hit the right one," exposing the inflated nature of self-reported excess returns within the loop.

One-line Summary

AI's coding ability is far stronger than its financial intuition; 290 factors with zero survivors under independent verification; the bottleneck is whether humans will verify.

Paragraph Summary

The RD-AGENT project tested the true level of LLM-automatically-generated factors through multiple real-world campaigns (A-shares, futures, cryptocurrency, totaling 290 factors). The core conclusion is: AI's coding ability is far more honest than its financial intuition—code bugs are explicit and fixable, while logical emptiness is implicit with no feedback. Decent factors cluster near specialty data provided by humans (basis, open interest), indicating that data boundaries determine the upper limit of AI's research taste; mediocre factors possess three pseudo-logical characteristics: excessive degrees of freedom, unfalsifiable mechanisms, and too many evaluations; among failed factors, implicit logical errors (look-ahead bias, convention misalignment) are indistinguishable from good factors on the leaderboard and can only be exposed through independent verification. Across 290 factors, self-reported excess returns peaked at +26.5%, with zero survivors under independent verification. The real bottleneck is not whether AI can think of factors, but whether humans will verify.