Why Does JEPA Underperform MLM in Longitudinal EHR Representation Learning?

Pretraining on electronic health records (EHR) has long been dominated by masked language modeling (MLM), but a new study on MIMIC-IV throws cold water on the emerging Joint Embedding Predictive Architecture (JEPA): all three JEPA variants trained stably without representation collapse, yet their downstream performance fell uniformly short of standard MLM. What design dilemmas lie behind this?

Why MLM Remains the Mainstream for EHR Pretraining

Masked language modeling (MLM) is the de facto standard pretraining objective for Transformer encoders handling longitudinal electronic health records. Its logic is intuitive: randomly mask some events and have the model reconstruct them from context. This approach has been validated on large EHR sequences and has therefore become the initialization choice for most clinical prediction models.

But the paper points out that reconstructing individual events may overemphasize noisy or low-information observations. EHR data is rife with routine labs, repeated records, and missing codes, and event-by-event reconstruction makes the model waste capacity on these low-value signals. More importantly, MLM does not explicitly encourage the model to learn a patient's high-level state—such as overall disease progression or risk stratification—which are precisely the abstractions most needed for downstream clinical tasks.

The Appeal of JEPA: Predicting in Latent Space

The Joint Embedding Predictive Architecture (JEPA) offers a different path: instead of reconstructing raw inputs, it makes predictions in the latent representation space. After seeing partial context, the model predicts the embedding of the masked portion in representation space rather than the raw tokens. This idea has demonstrated strong performance in images, video, and audio because it forces the model to focus on predictable abstract features rather than pixel- or token-level details.

Adapting JEPA to EHR sequence representation could theoretically bypass MLM's noise sensitivity and directly learn high-level representations of patient state. The research team therefore built three JEPA variants and conducted an empirical evaluation on MIMIC-IV.

Representation collapse is a typical failure mode of joint embedding architectures in self-supervised learning: the encoder maps all inputs to nearly identical representations (constant output or a low-rank subspace), making the prediction loss appear low while the learned features lack discriminability. To prevent collapse, JEPA-style methods typically employ asymmetric architectures (e.g., momentum encoders, stop-gradient) or regularization terms (e.g., VICReg's variance-covariance constraints). The paper emphasizes that the three JEPA variants "did not exhibit representation collapse" to rule out this common confounding factor, thereby attributing the downstream performance gap to objective design rather than training failure.

Experimental Results: Stable Training, Yet Uniformly Behind

According to the paper, all three JEPA formulations trained stably without representation collapse—that is, the models did not degenerate into outputting constants or low-rank representations. This is itself a positive signal, indicating that JEPA's architectural design is feasible on EHR data.

However, the uniform shortfall in downstream performance breaks this optimistic expectation. Across multiple evaluation tasks, none of the JEPA variants surpassed standard MLM. This result is not caused by training instability but points to a more fundamental problem: stable latent-space prediction alone is insufficient to achieve competitive EHR representation learning.

Key Findings on Target Size and Heterogeneity

The study further decomposes the sources of JEPA prediction error. A counterintuitive finding is that target size, rather than target heterogeneity, drives up prediction error. This means that when the predicted latent target covers a larger range or coarser granularity, prediction difficulty rises, while whether the target itself is diverse has relatively limited impact.

But the more critical conclusion is that even reducing target size cannot recover downstream performance. This rules out the simple idea that "adjusting prediction granularity can fix it" and suggests that JEPA's bottleneck on EHR lies not in the magnitude of prediction error but in whether the predicted latent target itself is clinically meaningful.

Target size refers to the time span or aggregation granularity covered by the predicted latent target—for example, predicting the patient state embedding for the next day versus the next week; target heterogeneity refers to the degree of diversity among different samples or dimensions within the target set. In the JEPA framework, the target is generated by the encoder for the masked portion, and its size directly affects the difficulty and information density of the prediction task: larger and coarser targets typically yield higher prediction error. By separating the effects of the two through controlled variables, the study finds that error is mainly driven by target size, which provides empirical evidence for subsequent adjustment of prediction granularity, but also shows that reducing target size alone cannot resolve the semantic-level bottleneck.

Core Challenge: Defining Clinically Meaningful Latent Targets

The paper lands on an open question: how to define clinically meaningful latent targets for EHR? In the image domain, latent-space prediction works because the visual world contains learnable physical regularities and semantic structure. EHR sequences are different—the high-level abstraction of patient state itself lacks a universally accepted, computable representation definition. Concepts such as disease progression, comorbidity burden, and treatment response are clinically meaningful but difficult to translate into dense latent targets suitable for JEPA prediction.

This also explains why reducing target size is of no avail: the problem is not prediction accuracy but what is being predicted. If the latent target itself cannot capture clinical relevance, then no amount of stable training will translate into downstream gains.

Implications for EHR Representation Learning

This empirical study conveys two messages. First, JEPA is not a plug-and-play replacement; transferring success from the visual domain to EHR requires rethinking objective design. Second, MLM's continued dominance is justified—although it overemphasizes individual events, reconstructing raw signals at least provides an information-dense training signal that requires no manual definition of high-level concepts.

For practitioners, MLM remains a reliable baseline for longitudinal EHR pretraining in the short term. For researchers, the real breakthrough may lie not in changing the prediction target but in designing intermediate representations that bridge clinical semantics and latent-space prediction. Until such a representation is found, JEPA's potential on EHR remains theoretical.