ContextAdapt

The arXiv paper "ContextAdapt" investigates whether large language models can appropriately adjust how they apply values across different professional contexts while remaining consistent when contextual changes do not alter professional norms.

The ContextAdapt evaluation framework covers three categories of values—honesty, autonomy, and confidentiality—spanning four high-risk professional domains: healthcare, law, finance, and national security.

ContextAdapt constructs a two-dimensional "value × domain" matrix based on primary professional and regulatory documents, and develops two types of test scenarios covering default professional rules and recognized exceptions.

The evaluation subjects of the ContextAdapt study cover 12 mainstream large language models.

The study examines not only what actions models recommend, but also what justifications they provide.

On action appropriateness, the average score across all models reached 95.6%.

In providing correct, domain-specific justifications, performance across different models ranged from 25.6% all the way to 76.9%.

Factorial experiments show that explicitly naming the professional domain and changing the role the model plays both had limited influence on model behavior.

Varying the stakes of a scenario causes severe but localized failures in models, causing them to change their responses even when the underlying professional obligation remains unchanged.

Perceived severity becomes a trigger for models to favor disclosing information, a phenomenon observed in both honesty and confidentiality scenarios.

Before deploying LLMs in high-stakes professional settings, the underlying model reasoning may be unreliable beneath a surface of high accuracy, making rigorous testing of contextual adaptability indispensable.