Can Large Language Models Predict Patient Survival? The Survprompt Framework Has Answers

When Patients Ask AI About Prognosis, Can LLMs Be Trusted?

Survival analysis is a cornerstone of medical risk assessment. By leveraging patient covariates, it estimates "time-to-event" outcomes — such as expected survival time following a cancer diagnosis. As consumer-facing applications like ChatGPT become increasingly mainstream, more and more patients are turning directly to large language models (LLMs) for prognostic guidance after receiving a diagnosis.

The question is: how accurate are the survival predictions these models provide? Until now, there has been no rigorous evaluation of this question. A recent arXiv paper, Large Language Models are Approximate Survival Estimators, tackles this head-on — and the findings are striking. State-of-the-art LLMs, operating in a zero-shot setting, can produce prognostic estimates that are surprisingly close to those of specialized models.

rss source: Large Language Models are Approximate Survival Estimators

Survprompt: Turning Medical Records into "Clinical Vignettes"

The research team introduces a framework called Survprompt. The core idea is straightforward yet clever: structured patient covariates (such as age, tumor stage, and lab values) are converted into free-text "clinical vignettes," which are then fed to a pretrained LLM. The model is asked to predict survival in a zero-shot fashion — no task-specific training required.

The significance of this approach lies in how it bypasses the constraints of traditional survival models, which require dedicated training and structured data formats. Instead, it draws on knowledge the LLM has already absorbed from vast amounts of medical literature and clinical text. In essence, it tests how much prognostic signal a general-purpose model can extract from a narrative clinical description — without ever being explicitly trained for survival prediction.

Rigorous Benchmarking on Two Pan-Cancer Cohorts

To evaluate Survprompt in a meaningful way, the researchers benchmarked it on two large multi-institutional pan-cancer cohorts:

MSK-CHORD: A large, publicly available cancer cohort dataset
Providence St. Joseph Health Network cohort: A newly curated dataset built using an LLM-based medical information extraction framework

The comparison was made against traditional survival analysis models, with Random Survival Forests (RSF) serving as the primary baseline — one of the most widely used methods in survival prediction. Two key metrics were used for evaluation:

Censored Mean Absolute Error (cMAE): Measures the deviation between predicted and actual survival times
Concordance index (c-index): Measures the model's ability to distinguish between high- and low-risk patients

The study also conducted feature ablations to identify which variables most influenced the LLM's predictions.

Random Survival Forests (RSF) is an ensemble learning method derived from Leo Breiman's random forest algorithm, specifically designed for survival analysis tasks involving censored data. While traditional random forests aggregate predictions from multiple decision trees for classification or regression, RSF trees split nodes by maximizing survival differences between child nodes — typically using the log-rank test statistic. Each tree outputs a cumulative hazard function, and the ensemble prediction is the average across all trees. RSF naturally handles high-dimensional features, nonlinear relationships, and feature interactions without requiring parametric assumptions about survival time distributions (such as the proportional hazards assumption in Cox models). This makes it a consistently strong performer on real-world oncology datasets and a go-to baseline in survival prediction benchmarks.

Censoring is a fundamental concept in survival analysis. It refers to situations where the target event (e.g., death) has not yet occurred by the end of the study period or when a patient is lost to follow-up — meaning we only know the patient was alive up to a certain point, not their exact event time. Properly accounting for censored data is what distinguishes survival analysis from ordinary regression. The cMAE (censored mean absolute error) applies a statistical correction to censored samples on top of traditional MAE, enabling fairer model evaluation.

Key Findings: Surprisingly Accurate, but Poor at Risk Stratification

The results can be broken into two parts: the encouraging and the concerning.

The Upside: Highly Competitive cMAE

Frontier LLMs performed surprisingly well on individual survival time prediction as measured by cMAE. Using GPT-5.6-Sol as a representative example, the paper shows that across multiple cancer types, its cMAE was within 10% of state-of-the-art RSF models trained specifically for the task. On prostate cancer predictions in MSK-CHORD, the LLM's cMAE was actually lower than RSF — meaning its predictions were more accurate in absolute terms.

Perhaps more tellingly, feature ablation experiments revealed that the LLM's prioritization of clinical variables closely mirrored that of specialized survival models. This suggests the model isn't guessing randomly — it is genuinely capturing the clinical factors most relevant to prognosis.

The Downside: Weak Risk Stratification

The other side of the picture is equally clear. LLM performance varied inconsistently across cancer types and institutions. More critically, LLMs demonstrated poor ability to distinguish high-risk from low-risk patients (low c-index).

This is a clinically significant limitation. The value of survival prediction lies not only in estimating how long a patient might survive, but in ranking patients by risk — identifying who needs more aggressive treatment versus who can be safely monitored. A low c-index means LLMs are not reliable at this task.

The concordance index (c-index, also known as Harrell's C-statistic) is the primary metric for evaluating a survival model's discriminative ability. It can be understood as: "For any two patients, what is the probability that the model correctly identifies which one will experience the event first?" A c-index of 0.5 equals random guessing; 1.0 indicates perfect ranking. In clinical practice, a c-index below 0.6 is generally considered insufficient discriminative ability.

Risk stratification matters because oncology decisions are inherently comparative. Two patients with stage III lung cancer may have vastly different outcomes depending on molecular subtype and performance status — and their treatment plans should differ accordingly. A model that can only output "average survival is about 18 months" but cannot distinguish individual risk profiles has limited utility in personalized medicine. This is why the researchers report both cMAE and c-index: the former assesses absolute prediction accuracy, the latter assesses relative ranking ability. Together, they provide a complete picture of a model's practical value.

What This Means for Clinical Applications

The signal from this research is nuanced. On one hand, it demonstrates that zero-shot LLMs — without any task-specific training — can generate surprisingly accurate prognostic estimates. This opens up possibilities for democratizing medical AI: patients and primary care providers might use such tools to get a preliminary reference point.

On the other hand, inconsistent performance across cancer types and institutions, combined with weak risk stratification ability, presents a significant barrier to clinical deployment. Deploying a tool that achieves low average error but poor patient ranking directly in clinical decision-making carries obvious risks.

The word "Approximate" in the paper's title is well chosen: LLMs are approximate survival estimators. They can produce numbers that point in roughly the right direction, but they are not yet ready to replace rigorously trained and validated specialized models. For practitioners thinking carefully about the boundaries of AI in medicine, this work is both an encouragement and a caution.