Do Shared Latent Spaces Really Share? The Compatibility Truth About Motion VAEs

Shared latent space is a common assumption in multimodal learning: as long as different modalities share parameters, dimensions, or priors, their latent representations can be directly swapped and used interchangeably. But a recent study on motion VAEs systematically challenges this assumption.

Three Meanings of "Shared Latent Space" Have Been Conflated

The paper points out that researchers often conflate three distinct things: recoverable correspondence, direct compatibility, and decoder reuse.

• Recoverable correspondence: After linear calibration, the pairing relationships between instances can be recovered from latent coordinates.
• Direct compatibility: Without any post-processing, latent representations from different schemas can be directly swapped and used interchangeably.
• Decoder reuse: The same decoder can simultaneously handle latent encodings from different schemas.

Sharing a backbone network or a prior only guarantees consistency at the parameter level; it does not determine how corresponding instances are expressed in latent coordinates. This means the word "shared" points to vastly different capabilities depending on the context.

Motion VAE refers to a class of methods that apply Variational Autoencoders (VAEs) to human motion sequence modeling. A VAE consists of an encoder and a decoder: the encoder maps an input motion sequence to distribution parameters in a low-dimensional latent space (typically mean and variance), while the decoder reconstructs motion from latent vectors. Its training objective is to maximize the Evidence Lower Bound (ELBO), balancing reconstruction error against a regularization term on the latent distribution. In motion generation tasks, the VAE's latent space is often treated as a compact semantic representation of motion, used for interpolation, editing, or cross-modal retrieval. Because motion data has strong temporal dependencies and joint correlations, motion VAE encoders typically employ Transformers or graph convolutional networks, while decoders must output coherent skeletal pose sequences. The "multi-schema motion VAE" discussed in this paper refers to a single model that handles multiple motion representation specifications under a shared backbone and decoder architecture.

Experimental Design: Controlled Testing of Multi-Schema Motion VAEs

The study constructed a controlled multi-schema motion VAE on the HumanML3D dataset. A schema can be understood as a different representation or encoding specification of the same motion data. The model shares a backbone and decoder architecture, and in theory should produce interchangeable latent representations.

Evaluation metrics include raw Recall@1 (measuring direct retrieval capability) and paired token cosine similarity (measuring latent coordinate consistency). These two metrics directly reflect whether the latent space "can be used directly," not whether it "can be aligned."

Recall@1 (Top-1 recall) is a commonly used metric in retrieval tasks. Given a query sample, the model retrieves the most similar sample from a candidate pool; if the top-ranked result is the correct paired sample, it counts as a hit. Recall@1 is the proportion of hits to total queries. In this paper's context, raw Recall@1 measures: without any linear calibration or post-processing, when directly using the latent vector of an instance in schema A to retrieve the corresponding instance in schema B, the probability of hitting the correct pair on the first attempt. The lower this metric, the less compatible the latent coordinates of the two schemas are. The complementary paired token cosine similarity computes the cosine similarity between latent vectors of corresponding instances, ranging from -1 to 1, where values closer to 1 indicate more consistent latent representations. Together, these two metrics characterize the "direct usability" of the latent space, rather than merely its "alignability."

Striking Finding: Direct Compatibility Under Shared Architecture Is Nearly Zero

The experimental results were unexpected. Even with identical inputs, between latent representations produced by different schemas:

• Raw Recall@1 was only 5.9%–17.7%
• Paired token cosine similarity was only 0.221–0.484

In other words, in a theoretically "shared" latent space, the coordinates of corresponding instances almost completely fail to match. But if linear calibration is allowed, correspondences can be recovered almost perfectly. This shows that the information exists in the latent space—it just isn't organized in a directly interchangeable form.

Cycle consistency as a training constraint can further enhance recoverability, but its improvement on direct retrieval and token consistency is nearly zero. The constraint makes the space "easier to interpret," but does not make the coordinates of different schemas "actively align."

Cycle consistency is a training constraint that requires no paired data, commonly used in cross-domain mapping or alignment tasks. Its core idea is: if a latent representation from schema A is mapped to schema B and then mapped back to schema A, the original representation should be recoverable; and vice versa. In this paper's multi-schema motion VAE, the cycle consistency constraint encourages the latent spaces of different schemas to maintain structural consistency under transformation, thereby improving the recoverability of latent representations. However, cycle consistency does not directly enforce that corresponding instances have the same coordinates in their respective latent spaces. Therefore, it makes the space "easier to linearly interpret" but cannot make the coordinates of different schemas actively align. This distinction explains why cycle consistency can improve recoverable correspondence but offers almost no improvement in direct compatibility.

A Small Amount of Paired Data Brings Dramatic Improvement

The most practically valuable finding in the study comes from the addition of paired data. Just 64 paired clips are enough to boost raw Recall@1 to 88.4%–90.3% and token cosine similarity to approximately 0.82.

This number means that under the test setting, most post-hoc alignment becomes unnecessary. 64 paired samples are negligible relative to the entire dataset, but what they provide are cross-schema "anchors" that directly tell the model which latent coordinates should correspond.

But the researchers also caution that even with substantially improved direct compatibility, crossed decoding remains noticeably worse than native reconstruction. Latent representations can be directly retrieved, but that doesn't mean decoding quality has kept up.

Paired clips refer to paired representations of the same motion segment under different schemas. For example, the same action sequence encoded in two specifications—joint rotation angles and joint positions—forms one-to-one sample pairs. In this paper, fine-tuning or alignment training with just 64 paired clips is enough for the model to learn to map corresponding instances across schemas to nearby latent coordinates. These paired samples serve as "anchors," providing direct cross-schema supervision signals to the model, thereby substantially improving raw Recall@1 and token cosine similarity. This finding demonstrates that the leverage effect of a small amount of paired data far exceeds simply adding more unpaired data or relying on architectural sharing.

Narrowing Marginal Differences Does Not Equal Schema Invariance

Another counterintuitive conclusion of the paper is that reducing the measured marginal discrepancy does not mean achieving schema invariance.

A common practice is to use MMD or similar metrics to measure whether the latent distributions of two schemas are close—the closer they are, the better the sharing is assumed to be. But experiments show that marginal distribution alignment and direct compatibility of latent coordinates are two independent dimensions. Distributions looking the same does not mean corresponding instances land in the same position.

This reminds researchers that when evaluating shared representations, they cannot rely solely on distribution-level aggregate metrics—pair-level validation must be introduced.

Implications for Multimodal Representation Learning

This study also has reference value for multimodal work beyond motion generation:

1. Shared parameters do not equal shared semantics. Architectural sharing is a necessary but not sufficient condition.
2. Recoverability and direct compatibility need to be evaluated separately. The former can be achieved through post-processing; the latter is the hallmark of "true sharing."
3. A small amount of paired supervision has significant leverage. In the test setting, 64 paired samples can change the organization of the latent space.
4. Marginal distribution alignment metrics can be misleading. When evaluating sharing quality, pair-level retrieval and token consistency metrics should be included.

For teams building cross-modal retrieval, motion transfer, or multi-schema generation systems, the most direct actionable advice is: do not assume that a shared architecture automatically brings a shared latent space. Validate with paired data, and explicitly introduce a small amount of paired supervision when necessary.

One-line Summary

Shared architecture does not guarantee directly compatible latent spaces; just 64 paired samples can dramatically improve cross-schema retrieval.

Paragraph Summary

This paper systematically questions the common assumption of "shared latent space" in multimodal learning. The study constructs a multi-schema motion VAE on HumanML3D and finds that even with a shared backbone and decoder, the raw Recall@1 between latent representations of different schemas is only 5.9%-17.7%, with paired token cosine similarity of only 0.221-0.484—direct compatibility is nearly zero. However, linear calibration can almost perfectly recover correspondences, indicating that information exists but is not organized into an interchangeable form. Cycle consistency improves recoverability but offers no improvement in direct compatibility. Just 64 paired clips can boost Recall@1 to 88.4%-90.3% and token cosine to approximately 0.82, making post-hoc alignment unnecessary. The study also points out that reducing marginal distribution differences does not equal achieving schema invariance, and evaluating shared representations requires pair-level validation.