Terminal X's GPT-6.1 Matches Senior Investor Judgment 88.6% of the Time

Article Content

Finance and investment has always held AI to a high bar for trust—after all, these decisions involve real money and tolerate very little error. A recent set of figures from Terminal X is worth noting: its Eval Agent, when assessing whether a portfolio complies with a fund mandate (investment authorization), reaches an 88.6% agreement rate with senior investors' judgments and captures 25% more deviations than the baseline method. The model behind it is GPT-6.1 Sol.

How the evaluation agent works: 54 criteria and deviation detection

Terminal X's Eval Agent does not simply ask a large model to "take a look" at a portfolio and give a conclusion. It uses an evaluation rubric containing 54 criteria, checking the fund's investment mandate item by item—covering dimensions such as asset class restrictions, concentration constraints, sector exposure, and liquidity requirements—and scores the output in a structured way.

The value of this approach lies in interpretability. Traditionally, compliance and investment research teams rely on senior investors' experience to judge whether a position has "crossed the line"; a rubric-based evaluation breaks tacit experience into reproducible checks, where the model only needs to give a judgment on each item and finally aggregate them into an overall assessment. An 88.6% agreement rate means that in the vast majority of cases, the Eval Agent's conclusions overlap with human experts' judgments.

More interesting is the 25% figure: compared with the baseline method, the Eval Agent captured one quarter more deviations. In investment compliance scenarios, the cost of missing a deviation can be far higher than a false positive—the former means risk exposure goes unidentified, while the latter only means one more manual review.

Investment mandate and deviations

An investment mandate is a set of binding investment rules agreed between a fund and its investors. It usually specifies investable asset classes, single-position limits, sector or regional exposure limits, liquidity requirements, leverage and derivatives usage, and so on. Compliance and investment research teams need to continuously monitor whether the portfolio stays within these constraints. A so-called "deviation" (or breach) means that the portfolio's actual holdings exceed the boundary allowed by the mandate in some dimension—for example, a sector weight exceeds the limit, a prohibited asset class is held, or the proportion of liquid assets is insufficient. Deviations can be divided into hard violations and soft warnings; the former must be remediated, while the latter must be recorded and explained. The difficulty of deviation detection lies in the possible coupling among constraints—adjusting one position simultaneously affects concentration, sector exposure, and liquidity indicators, so item-by-item checking must be accompanied by overall judgment. This is also why rubric-based structured evaluation is more suitable for compliance scenarios than a single holistic score.

What GPT-6.1 Sol brings

The original article attributes this improvement to the capabilities of GPT-6.1 Sol. Given the nature of the evaluation task, this is not surprising: judging item by item against 54 criteria requires the model to maintain attention to detail over a long context, understand the precise meaning of financial terms, and make trade-offs that converge with human experts on cases with ambiguous boundaries. Such tasks place high demands on the model's reasoning stability and instruction-following ability; mere language fluency is meaningless.

A reasonable inference is that GPT-6.1 Sol's progress in following structured rubrics is the main source of the 25% improvement in deviation capture rate. However, the original article does not disclose exactly what model or rule system the baseline method is, so the reference point for this comparison needs to be interpreted cautiously—if the baseline is an earlier model or a pure rule engine, the 25% gap says more about the generational difference between "new-generation models vs. old methods" than about an advantage unique to GPT-6.1 Sol.

Rubric following and long-context reasoning

Rubric following means that the model judges item by item according to a pre-given set of sub-criteria rather than freely generating conclusions. This ability is fundamentally different from ordinary question answering: the model needs to simultaneously maintain memory of multiple rules within one reasoning process, locate details in the input text, and judge priorities when rules intersect. When evaluating 54 criteria item by item, the input often contains lengthy portfolio holdings tables and mandate documents, and the model must not lose key constraints over a long context—this tests long-context attention and the stability of instruction following. The precision of financial terminology further amplifies the difficulty—"concentration" and "exposure," "liquidity" and "tradability" have different meanings in different contexts, and misreading them directly leads to judgment bias. Therefore, the performance bottleneck of evaluation agents usually lies not in language generation but in the consistency and reproducibility of structured reasoning.

The real meaning of the 88.6% agreement rate

Viewed in the context of investment decisions, 88.6% is both impressive and draws a clear boundary of capability.

The impressive part: investment mandate evaluation has long been regarded as a high-threshold task requiring senior experience. That an evaluation agent can agree with expert judgment in nearly 90% of cases shows that the rubric-based evaluation framework has indeed captured the main logic of expert decision-making. For compliance teams with limited staff, this means a large number of routine checks can be handed to the Agent for initial screening, with humans handling only the remaining 11.4% of disputed cases.

The boundary is equally clear: the remaining roughly 11.4% of disagreements may precisely be the trickiest cases—situations with ambiguous boundaries, involving interactions among multiple constraints, or requiring judgment combined with market context. Fully handing these cases to the model is currently unrealistic. The reasonable positioning is "an amplifier of expert judgment" rather than a replacement: the Agent is responsible for coverage and consistency, while humans are responsible for difficult rulings.

Interpreting agreement-rate metrics and baseline comparisons

Agreement rate measures the proportion of cases where the agent's judgment overlaps with human expert judgment, but it does not distinguish the direction of agreement: both the model and the expert judging "compliant" and both judging "deviation" count in the numerator. Therefore, a high agreement rate may mask the ability to identify the minority class (deviations), which is why the supplementary metric of "capturing 25% more deviations" is more valuable—it directly reflects the reduction in missed reports. On the other hand, the composition of the baseline determines the meaning of the comparison: if the baseline is a rule engine or an early model, the 25% gap reflects generational progress; if the baseline is an equally capable strong model with a different prompting strategy, it better demonstrates the contribution of rubric design and model adaptation. When evaluating such claims, one should also pay attention to sample composition, inter-expert agreement (human experts may also disagree with one another), and whether there is selection bias—all of these affect the transferability of the numbers.

Implications for AI evaluation products

The methodology of Terminal X's case is worth referencing for other teams building vertical evaluation products. It does not pursue end-to-end black-box judgment; instead, it solidifies domain knowledge into 54 rubrics and lets the model operate within a structured framework. This "rubric + strong model" combination may be easier to implement than pure end-to-end solutions in scenarios with high requirements for interpretability, such as compliance, auditing, and investment research.

At the same time, it should be noted that the validity of the 88.6% figure depends on the quality of the rubric itself and the clarity of the fund mandate. The complexity of mandates varies greatly across funds, and the original article does not answer whether this agreement rate can be reproduced on more complex mandates. For practitioners concerned with AI implementation in finance, this is both a signal worth tracking and a claim that requires independent verification.