Universal Patterns in Cross-Modal ML: Toward a Unified Taxonomy for Text, Image, and Audio

Introduction: When Technical Names Obscure the Essence

In machine learning research, an interesting phenomenon recurs: the same underlying technique is often given different names across modalities (e.g., text, image, audio). For example, "tokenisation" in text corresponds to "patch embedding" in images and "frame-level feature extraction" in audio; "masked language modeling (MLM)" in text and "masked autoencoders (MAE)" in images share the same idea; diffusion models originated in image generation and are now used for audio synthesis. These recurring patterns raise a core question: Is there a cross-modal taxonomy that organizes ML techniques by underlying patterns rather than by modality?

Recently, a Reddit discussion thread posed this question. The poster observed that many techniques differ only in name while sharing the same essence. He hoped to find a pattern-based classification system to more efficiently categorize research papers and reference existing work. This article starts from this observation, combines community discussion with existing research, and explores the possibility of cross-modal pattern classification.

Common Cross-Modal Recurring Patterns

1. Discretization and Encoding: The Universality of Tokenisation

In NLP, text is split into tokens; in vision, images are divided into patches (e.g., ViT); in audio, waveforms are framed or converted to spectrograms. Despite different forms, they all serve the same purpose: transforming continuous or high-dimensional data into discrete or low-dimensional representations. This pattern can be called "discretization encoding".

ViT (Vision Transformer) is a typical representative of this pattern. It splits an image into fixed-size non-overlapping patches (e.g., 16×16 pixels), each linearly projected and fed as a "token" in a sequence to a Transformer. This design makes image data formally isomorphic to text sequences, allowing direct reuse of the Transformer architecture designed for NLP without convolutional inductive bias. ViT's success shows that modality differences can be dissolved through a unified discretization-embedding pipeline, which is why "patch embedding" is often regarded as the visual equivalent of tokenisation.

2. Masking and Reconstruction: The Core of Self-Supervised Learning

Masked language modeling (MLM) learns representations by predicting masked tokens; masked autoencoders (MAE) reconstruct occluded image patches; audio also has masked modeling (e.g., audio masked autoencoders). These techniques share the "mask-predict" paradigm, a general strategy in self-supervised learning.

MAE (Masked Autoencoder) was proposed by He et al. in 2021. It divides an image into patches, randomly masks a high proportion (e.g., 75%), feeds only visible patches to the encoder, and a lightweight decoder reconstructs the masked pixels. This is conceptually homologous to BERT's MLM: both construct self-supervised signals by "masking part of the input and predicting the missing part". The key difference lies in masking ratio and reconstruction target—MLM predicts discrete tokens, MAE reconstructs continuous pixels, but the shared core assumption is: completing the reconstruction task forces the model to learn semantic representations of the data. Audio masked modeling (e.g., Audio-MAE) further transfers this paradigm to spectrograms.

3. Generation and Prediction: From Next Word to Next Frame

Autoregressive generation manifests as "next word prediction" in text, "next frame prediction" in video or image sequences, and "next sample point prediction" in audio. Diffusion models take a different path, generating data through gradual denoising, but also cross modality boundaries.

The core idea of Diffusion Models is to define a forward noising process that gradually corrupts data into pure noise, then train a neural network to learn the reverse denoising process, thereby gradually recovering data from noise. Unlike autoregressive models that generate item by item in sequence order, diffusion models iteratively refine the entire data in continuous space, naturally suited for continuous signals like images and audio. DDPM (Denoising Diffusion Probabilistic Models) established this framework, and subsequent Latent Diffusion Models (LDM) moved the diffusion process to a compressed latent space to reduce computational cost. The cross-modal generality of this paradigm is strong evidence for the idea of "classifying by pattern rather than modality".

Existing Classification Frameworks and Community Views

Academic Attempts

Currently, academia has not formed a unified cross-modal taxonomy. Common classifications are still modality- or task-oriented (e.g., NLP, CV, speech). However, some surveys have begun to focus on cross-modal general techniques, such as self-supervised learning and generative models. Some researchers propose classification by "learning paradigm" (supervised, self-supervised, reinforcement learning) or by "data structure" (sequence, grid, graph).

Self-Supervised Learning (SSL) refers to a learning paradigm that constructs supervision signals from the data itself without relying on manual labels, and is the most representative category of cross-modal general techniques. Its mainstream methods can be grouped into two types: generative/reconstructive (e.g., MLM, MAE), which learn representations by predicting masked parts; and contrastive (e.g., SimCLR, MoCo), which learn invariant representations by pulling positive pairs closer and pushing negative pairs apart. SSL has become an important dimension for cross-modal classification because its methodology is highly isomorphic across text, image, audio, and graph data, allowing researchers to transfer design ideas across modalities without being limited by task labels.

Insights from Community Discussion

In the Reddit discussion, some users pointed out that pattern recurrence stems from similarity in data characteristics: text, images, and audio can all be viewed as sequence or grid data, so similar techniques naturally apply. Others mentioned that the success of the Transformer architecture exemplifies this unity: it was originally used for text, then extended to images (ViT), audio (AST), and multimodal.

Possibility of Building a Unified Taxonomy

Dimension 1: Data Representation and Preprocessing

• Discretization: tokenisation (text), patchification (image), frameization (audio)
• Embedding: word embeddings, image patch embeddings, audio spectrogram embeddings

Dimension 2: Learning Objectives

• Self-supervised: masked prediction, contrastive learning, next word/frame prediction
• Generative: autoregressive, diffusion, GAN

Dimension 3: Architectural Paradigms

• Sequence models: RNN, Transformer
• Convolutional models: CNN (image, audio)
• Graph models: GNN (for relational data)

Dimension 4: Task Types

• Classification: sentiment analysis, image classification, speech recognition
• Generation: text generation, image generation, audio synthesis

This multi-dimensional taxonomy can help researchers quickly locate similar cross-modal work. For example, if studying text masked modeling, one can also refer to visual MAE and audio masked autoencoders, promoting knowledge transfer.

Practical Advice: How to Use Pattern Classification to Organize Research

1. Establish pattern tags: Label each paper with its underlying pattern (e.g., "masked self-supervision", "diffusion generation") rather than just modality tags.
2. Cross-modal retrieval: Use pattern tags to search and discover similar methods across modalities.
3. Focus on general architectures: Transformer, diffusion models, etc., have proven cross-modal effectiveness and can be prioritized.
4. Participate in community discussions: Forums like Reddit's MachineLearning often have in-depth discussions on cross-modal patterns.

Conclusion

Cross-modal recurring patterns indeed exist and stem from the essential commonality of data and learning tasks. Although there is no universally accepted unified taxonomy yet, by organizing along multiple dimensions (representation, objective, architecture, task), researchers can build their own mental models to more efficiently organize literature and inspire cross-domain innovation. In the future, with the rise of multimodal learning, this pattern-based perspective will become increasingly important.

This article is based on Reddit discussions and public research, aiming to provide a thinking framework for cross-modal technique classification.