Turning Long Text into Risk Features in Seconds: A New LLM Block-Level Feature Engineering Framework

Why Is the Goldmine in Long Text So Hard to Mine?

Industrial risk control systems face an awkward reality: the models that actually make decisions are built on structured data, because they are fast, stable, and easy to maintain. But the truly valuable information is often buried in lengthy text—contract clauses, user feedback, transaction descriptions, ticket records.

To put this information to use, there are only two paths. The first is manual feature engineering, which is costly, time-consuming, and depends on expert experience. The second is to call a large language model directly during online inference, but latency and cost make this unsustainable—practically infeasible in risk control scenarios.

!

The approach proposed in this paper is straightforward: let the LLM "compile" long text into executable programs during the offline stage, and only run the programs online without touching the LLM. This is the core logic of LLM-BlockFE.

How Does LLM-BlockFE Work?

Core Idea: Turning Feature Engineering into Program Search

The traditional approach has the LLM directly output feature descriptions or values, but the problem is that the model must be called at every inference, which is unsustainable. LLM-BlockFE takes a different angle: it has the LLM generate code blocks—executable programs.

Specifically, the framework builds feature programs through incremental appending of immutable code blocks. Once a code block is written, it is never modified, and the entire program takes shape step by step like assembling building blocks. The quality of candidate features is evaluated by a downstream model—this effectively provides an objective feedback signal for the search process.

The advantage of this approach is that after the search ends, the program is frozen and deployed. During online inference, only this program needs to be executed to extract structured features, completely bypassing LLM calls.

What is an executable feature program? In traditional machine learning pipelines, features are typically manually written fixed code or configurations, whereas "program search" refers to letting a search algorithm automatically compose this feature extraction logic within a code space. Each "code block" generated by LLM-BlockFE is an immutable, independently executable mini-program—such as a regex extraction, a statistical aggregation of a text segment, or a type conversion. Multiple code blocks are concatenated in sequence to form a complete feature extraction program. Since code blocks cannot be modified, the search process is equivalent to continuously appending new blocks, making the program state clear and traceable; after the search ends, it is solidified into ordinary code and deployed, requiring no LLM at runtime, with inference overhead on par with hand-written feature engineering.

Solving the "Myopia" Problem of Greedy Search

Feature engineering search has a classic challenge: greedy strategies easily fall into local optima. You choose the code block that looks best at each step, but after a few steps you may realize you have gone in the wrong direction and cannot turn back.

LLM-BlockFE addresses this with two key designs:

Block-level rollback mechanism. Rather than overturning the entire path and starting over, it allows rollback at the code block granularity, combined with depth-calibrated credit allocation to determine which block is responsible for the current performance.

Multi-trajectory interleaved search. Multiple independent search trajectories are advanced simultaneously, each with its own exploration direction. More importantly, different trajectories share fixed direction descriptions, reducing redundant exploration. This is like having multiple search "clones" working in parallel, but each knows what the others are looking for, avoiding collisions.

Combining these two points preserves search flexibility while controlling computational overhead.

Greedy search and local optima. Greedy search is a step-by-step decision strategy: at each step, it selects the action that appears optimal under the current state, without considering the impact of that choice on subsequent steps. Its advantages are simple implementation and low per-step computation; its disadvantage is the inability to backtrack—once a step is chosen incorrectly, subsequent steps can only continue on the erroneous foundation, easily getting stuck at a local optimum rather than the global optimum. In feature engineering search, each added feature transformation is equivalent to taking a step; a greedy strategy may lock onto a direction too early and miss better combinations that require a "detour" to reach. LLM-BlockFE's block-level rollback and multi-trajectory parallelism essentially introduce a degree of backtracking and exploration capability into the search to alleviate this inherent flaw.

Experimental Results: Both AUC and KS Improve

The paper validated on four datasets—two public datasets plus two private datasets, covering different scenarios.

In the full data comparison, relative to the strongest baseline, LLM-BlockFE achieves an absolute AUC improvement between 0.0069 and 0.0358 on each dataset. In risk control scenarios, achieving absolute AUC improvements of this magnitude is already quite remarkable.

More noteworthy is the online performance. The paper mentions post-deployment monitoring across five deployed financial risk control applications, with an absolute KS improvement between 0.02 and 1.56 percentage points, compared against existing manually designed strategies. This is not just good-looking offline metrics, but improvement in real business scenarios.

KS is a discrimination metric in risk control; improving by more than one percentage point represents a very tangible gain for strategy iteration.

The meaning of AUC and KS. AUC (Area Under the ROC Curve) measures a model's ability to rank positive and negative samples: given a randomly chosen positive sample and a negative sample, the probability that the model assigns a higher score to the positive sample is the AUC; a value of 0.5 means equivalent to random guessing, and 1 means perfect discrimination. The KS (Kolmogorov-Smirnov) statistic takes the maximum difference between the cumulative distribution curves of positive and negative samples, reflecting the strongest discrimination the model can achieve at a certain threshold. Both are core offline metrics in risk control and credit scoring. They are very sensitive to small improvements: in mature risk control models, an AUC improvement of 0.01 often implies a substantial change in bad debt rate or approval rate, so the absolute improvements of 0.0069 to 0.0358 in this paper are by no means negligible numbers.

What Does This Framework Actually Solve?

Back to the original question: industrial risk control needs fast and accurate models, and long text contains information that cannot be used.

The value of LLM-BlockFE is not in using an LLM, but in using the LLM in the right place—the offline feature construction stage. It "solidifies" the LLM's language understanding capability into executable programs, allowing online inference to return to efficient structured models.

From an engineering perspective, this "offline compilation, online execution" approach is actually quite instructive. In many scenarios, we do not need to call large models in real time; rather, we need to distill the capabilities of large models into the system. Feature engineering is just one entry point.

The paper also leaves some open questions: balancing search efficiency and program complexity, adaptability to different downstream models, and the strategy for choosing code block granularity. But the overall direction is clear—let the LLM do what it is good at, and let traditional models do what they are good at.

One-line Summary

LLM-BlockFE lets an LLM compile long text into executable feature programs offline, improving risk control metrics without calling the LLM online.

Paragraph Summary

In industrial risk control, structured data models make decisions, but high-value information is often hidden in long texts such as contracts and tickets. Manual feature engineering is costly, and calling large models online is limited by latency and cost. LLM-BlockFE proposes an offline-compile, online-execute approach: the LLM searches for executable feature extraction programs during the offline stage by incrementally appending immutable code blocks, and online only runs the program without calling the LLM. To mitigate the local optima problem of greedy search, it introduces block-level rollback and credit allocation mechanisms, and uses multi-trajectory interleaved search to share exploration directions and control overhead. On four datasets, absolute AUC improves by 0.0069 to 0.0358, and KS improves by 0.02 to 1.56 percentage points across five deployed financial risk control applications.