TS2Vec Becomes a Foundation Model: 1.6M-Parameter Frozen Encoder Approaches SOTA

The field of time series classification is undergoing a paradigm shift from "training a separate model for each dataset" to "one encoder reused across all datasets." However, foundation models often rely on attention mechanisms and complex tokenizer designs. A new study demonstrates that the classic TS2Vec self-supervised method, through three key modifications, can transform into a compact and efficient foundation model that approaches current state-of-the-art performance on multiple benchmarks.

Why Turn TS2Vec into a Foundation Model?

TS2Vec is a time series self-supervised method proposed in 2022 that performs self-supervised pretraining independently on each dataset followed by linear probing, achieving strong performance. But it has a fundamental limitation: it must be retrained for every new dataset and cannot be reused.

Meanwhile, the field has shifted toward foundation models—pretrain once on a large-scale corpus, then freeze the encoder to adapt to downstream tasks. This raises the question: can TS2Vec's appealing characteristics—compact convolutional architecture, direct processing of raw time series samples, and acceptance of variable-length inputs without resampling—be preserved under the "pretrain only once" paradigm?

According to the experiments in this study, the answer is yes. The authors substantially transformed TS2Vec into a foundation model through three modifications, naming it TS2Vec-FM.

Time Series Foundation Models (TSFM) represent an emerging paradigm in time series analysis in recent years. Traditional methods typically train a separate model for each dataset, whereas TSFM draws on the foundation model approach from natural language processing and computer vision: pretrain a general-purpose encoder on a large-scale, diverse corpus, then transfer the pretrained model to various downstream tasks through a frozen encoder plus lightweight downstream adaptation (such as linear probing). The advantage of this paradigm lies in significantly reducing the training cost of downstream tasks while improving generalization capability. Time series foundation models face unique challenges including: variable time series lengths, inconsistent sampling frequencies, lack of a unified pretraining corpus, and how to design encoder architectures that require no resampling. Current mainstream TSFMs are mostly based on Transformer architectures, relying on attention mechanisms and patch tokenizers, which also bring high computational complexity and parameter counts. The TS2Vec-FM research explores, in this context, achieving similar or even better performance with a more compact convolutional architecture.

Three Key Modifications: From Specialized Model to General Foundation Model

The design of TS2Vec-FM is extremely restrained, requiring only three adjustments:

First, pretraining on synthetic corpus. Instead of relying on target datasets, it uses a synthetically generated time series corpus for one-time self-supervised pretraining, thereby obtaining transferable general representations.

Second, dual-view input design. The encoder simultaneously receives the raw amplitude view and the instance-normalized shape view, each injected through a small stem network. This way the model can capture absolute amplitude information while focusing on morphological patterns, avoiding information loss caused by normalization.

Third, distillation to fuse into a single encoder. A expert encoder is trained for each view, and then their capabilities are merged into a single encoder through distillation. At inference time, only a single encoder forward pass is needed, keeping deployment simple.

The resulting TS2Vec-FM is a dilated convolutional network with only 1.6M parameters.

Instance Normalization is a common feature normalization technique, often used in time series analysis to eliminate amplitude and offset differences between different samples. Specifically, for each time series sample, instance normalization subtracts the sample's mean and divides by its standard deviation, standardizing each sample's distribution to zero mean and unit variance. The benefit is that the model can focus on the morphological patterns of the time series (such as trends, cycles, and abrupt changes) without being affected by absolute amplitude. However, instance normalization also loses amplitude information, which may be a key discriminative feature in certain classification tasks. TS2Vec-FM's dual-view design is precisely aimed at resolving this contradiction: the raw amplitude view preserves absolute amplitude information, while the instance-normalized shape view focuses on morphological patterns; the two views are injected into the encoder through different stem networks, allowing the model to obtain both types of information simultaneously. This design is relatively novel in time series foundation models, balancing the pros and cons of normalization.

Knowledge Distillation is a model compression and knowledge transfer technique originally proposed by Hinton et al. Its core idea is to have a student model learn from the outputs of one or more teacher models, thereby transferring the teacher models' knowledge to the student model. In TS2Vec-FM, the authors train a separate expert encoder as a teacher for each view (raw amplitude view and instance-normalized shape view), then fuse the capabilities of these expert encoders into a single student encoder through distillation. This way, at inference time only one encoder needs to be run, maintaining deployment simplicity while gaining the advantages of multi-view fusion. Distillation is typically implemented by minimizing the difference between the student model's and teacher models' outputs (such as KL divergence or mean squared error). This expert distillation strategy is still rare in time series foundation models; it effectively combines multi-view learning with model compression, providing a new idea for the design of compact foundation models.

Performance: Frozen Encoder Approaches SOTA

Under the setting of a frozen encoder with only linear probing, TS2Vec-FM achieves 83.3% accuracy on UCR-128 and 75.3% on UEA-27. Compared with the current state-of-the-art MantisV2, the gap is only 0.3 percentage points. More importantly, it surpasses TS2Vec trained separately on each target dataset.

This means that a convolutional model pretrained once with an extremely small parameter count has caught up with carefully designed attention-based foundation models in terms of generalization capability.

Linear Probing is a standard protocol for evaluating the representation quality of pretrained models. After pretraining is complete, all parameters of the encoder are frozen, and only a simple linear classifier (typically a linear layer or logistic regression) is trained on its output features, then performance is evaluated on downstream tasks. The advantage of linear probing is that it directly measures the generality and transferability of the features extracted by the pretrained encoder, unaffected by complex downstream fine-tuning strategies. If an encoder can achieve performance close to or even surpassing fully supervised training under linear probing, it indicates that the representations it has learned are already very powerful. In the TS2Vec-FM experiments, the authors adopt the linear probing protocol to evaluate the frozen encoder's performance on the UCR-128 and UEA-27 benchmarks, and compare with the current state-of-the-art attention-based foundation model MantisV2. The results show a difference of only 0.3 percentage points, and it surpasses the original TS2Vec trained separately on each dataset. This fully demonstrates the effectiveness of TS2Vec-FM's pretrained representations.

Design Philosophy: Near-Optimal Without Attention

The authors particularly emphasize in the original text that the design differences between TS2Vec-FM and other foundation models are the most interesting part: it does not use attention mechanisms, does not use patch tokenizers, and does not rely on intermediate layer feature extraction. It achieves near-optimal performance with only an "old-school" convolutional encoder and a two-layer MLP stem.

This provides an important reminder for time series foundation model research: architectural complexity is not a necessary condition for performance. The natural adaptability of compact convolutional networks to variable-length time series inputs, combined with a reasonable pretraining strategy, is sufficient to support an efficient foundation model. For scenarios with limited resources or requiring low-latency inference, the design approach of TS2Vec-FM is particularly worth learning from.

One-line Summary

TS2Vec becomes a 1.6M-parameter foundation model through three modifications, with a frozen encoder approaching SOTA.

Paragraph Summary

This article introduces a study that transforms the classic time series self-supervised method TS2Vec into a foundation model. TS2Vec originally required separate pretraining on each dataset and could not be reused. The researchers constructed TS2Vec-FM through three key modifications—one-time pretraining on a synthetic corpus, a dual-view input design (raw amplitude view and instance-normalized shape view), and fusing two expert encoders into a single encoder through distillation. The model is a dilated convolutional network with only 1.6M parameters. Under the setting of a frozen encoder with only linear probing, it achieves 83.3% and 75.3% accuracy on the UCR-128 and UEA-27 benchmarks respectively, only 0.3 percentage points behind the current state-of-the-art MantisV2, and surpassing the original TS2Vec trained separately on each target dataset. The research emphasizes that without attention mechanisms, patch tokenizers, or intermediate layer feature extraction, near-optimal performance can be achieved with just a convolutional encoder and a two-layer MLP stem, providing a compact and efficient new approach for time series foundation model design.