TAPES: Task-Aligned Joint Predictive Pre-Training for Transaction Sequences

In online commerce and digital financial services, transaction event sequences underpin a diverse array of predictive tasks. The same set of transaction records can give rise to semantically distinct prediction targets—numeric values, changes, relationships to historical events, and so on. These targets impose different constraints on historical representations, but the predictability of a target itself does not guarantee its usefulness for downstream tasks. Moreover, per-target prediction losses do not explicitly model the conditional co-occurrence relationships among targets. To address these challenges, researchers propose TAPES—a task-aligned joint predictive pre-training framework for transaction sequences.

Problem Background: The Dilemma of Choosing Prediction Targets

The core idea of transaction sequence pre-training is to construct prediction targets from historical transactions and let the model learn useful representations by learning these targets. However, a fundamental question is often overlooked: which prediction targets are truly beneficial for downstream tasks?

A single transaction can yield multiple prediction signals, such as the absolute value of the transaction amount, the amount change between adjacent transactions, or the type of relationship between the current transaction and historical events. These targets differ significantly in semantics and impose different constraints on the encoder. A target may be easy to predict yet irrelevant to downstream tasks, or may even introduce noise.

Another overlooked dimension is the dependency relationships among targets. Traditional approaches compute a prediction loss for each target separately, implicitly assuming targets are independent. But in real transaction scenarios, different targets often exhibit conditional co-occurrence—for example, the direction of an amount change may be highly correlated with the type of transaction relationship. Per-target losses cannot capture this joint structure, limiting the quality of representation learning.

Pretext tasks in pre-training refer to artificially constructed supervision signals on unlabeled data; the model learns general representations by completing these tasks. In natural language processing, masked language modeling is a typical example; in transaction sequences, predicting the next amount or relationship type falls into this category. The key distinction is that downstream task performance is the ultimate test of pre-training targets, while the target's own accuracy is only an intermediate metric. This distinction forms the theoretical basis of TAPES's design logic—it argues against using predictability as a proxy for target quality, because low-entropy targets may only require the model to memorize shallow statistical patterns and may not produce abstractions useful for downstream tasks.

TAPES's Core Design: Selection and Joint Modeling

TAPES's approach proceeds in two steps: first select the target set, then perform joint pre-training.

Downstream-Utility-Based Target Set Selection

TAPES first constructs a finite set of candidate target sets from permissible field-operator combinations. Each candidate set represents a group of prediction tasks. The key lies in the selection criterion: rather than looking at target predictability, it looks at the downstream utility of representations acquired through short-horizon joint pre-training.

Specifically, TAPES runs short-horizon joint pre-training trials for each candidate target set under a matched budget, then evaluates the resulting representations on downstream tasks. This process directly aligns target selection with the ultimate objective, avoiding the "easy-to-predict but useless" trap. After selection, the encoder is reinitialized to ensure fairness.

Short-horizon pre-training trials are a low-cost proxy evaluation strategy: instead of fully training each candidate configuration, a coarse representation is obtained with a small number of steps, and its performance on downstream tasks is measured as a ranking criterion. Its effectiveness relies on an empirical assumption—that the relative merits of short-horizon training are highly correlated with those of full training. This idea has analogous applications in neural architecture search and meta-learning; the core value is compressing the selection cost from "full training × number of candidates" to "short-horizon training × number of candidates." The cost is that short-horizon signals may be noisy, so control measures such as matched budgets and reinitialization are essential to ensure fair comparison.

Joint Likelihood with Shared Discrete Latent Variables

After the target set is selected, TAPES performs pre-training via a joint likelihood parameterized by shared discrete latent variables. This design explicitly models the conditional co-occurrence relationships among targets: the latent variables capture the dependency structure among targets, enabling the model to learn richer representations than per-target prediction.

After pre-training, the model is adapted to downstream supervised tasks. The entire pipeline can be summarized as: construct candidate sets → short-horizon trials to evaluate downstream utility → select target set → reinitialize → joint pre-training → downstream fine-tuning.

Discrete latent variable models assume that observed data are generated by a set of discrete categories that cannot be directly observed (i.e., hidden states); each observation corresponds to a hidden state, and different observations become dependent through shared hidden states. In TAPES, the latent variables act as "common factors" among multiple prediction targets: given the latent variable value, the targets are conditionally independent; after marginalizing out the latent variable, conditional co-occurrence structure emerges among the targets. This parameterization resembles the role of latent variables in mixture models and topic models; its advantage is that it characterizes complex joint distributions with a small number of discrete states while remaining computationally tractable. Compared with modeling each target separately, shared latent variables force the encoder to learn representations that can simultaneously explain multiple targets, thereby increasing information density.

Experimental Validation: Real Risk Scenarios and Public Benchmarks

According to the paper, TAPES was evaluated on a real-world transaction risk scenario and multiple public financial benchmarks, covering diverse downstream tasks. Experimental results show that TAPES outperforms representative baseline methods.

Ablation studies further verify the contributions of two core components: the prediction target selection mechanism and the joint modeling strategy. This suggests that the "quality" of target selection and the "structure" of joint modeling together constitute the source of performance improvement, and neither can be dispensed with.

Interestingly, TAPES's evaluation logic itself has methodological significance: it does not rely on target predictability as a proxy metric but directly measures downstream utility through short-horizon pre-training trials. This "begin-with-the-end-in-mind" selection approach may be transferable to other sequence pre-training scenarios.

Implications for Transaction Sequence Modeling

TAPES's work reveals an underappreciated aspect of transaction sequence pre-training: the construction and selection of prediction targets. In scenarios such as financial risk control and e-commerce behavior prediction, practitioners often focus on model architecture and training techniques but rarely systematically examine the question of "what to make the model predict."

The joint modeling approach is also worth learning from. Multiple prediction signals in transaction data are naturally correlated; characterizing this correlation with shared latent variables is closer to the data generation process than modeling them independently. For teams building transaction foundation models, TAPES provides an actionable design paradigm: first define the candidate target space, then screen with low-cost trials, and finally pre-train with structured joint objectives.

Of course, the effectiveness of this method is currently demonstrated mainly within the experimental scope reported in the paper. The computational overhead of target set selection, the way the candidate space is constructed, and transferability across different transaction domains remain issues worthy of attention in future work.

One-line Summary

TAPES screens prediction targets by the downstream utility of short-horizon pre-training and jointly models inter-target dependencies with shared latent variables.

Paragraph Summary

TAPES is a task-aligned joint predictive pre-training framework for transaction sequences, aimed at solving two problems: prediction target selection should not rely on the target's own predictability but should directly align with downstream utility; and per-target losses ignore conditional co-occurrence relationships among targets. Methodologically, TAPES first constructs candidate target sets, evaluates the downstream performance of representations obtained from each set through short-horizon joint pre-training trials, selects the target set accordingly, and reinitializes the encoder; it then pre-trains with a joint likelihood parameterized by shared discrete latent variables, explicitly modeling target dependencies. On a real transaction risk scenario and public financial benchmarks, TAPES outperforms representative baselines, and ablation studies verify the contributions of the two components: target selection and joint modeling.