Frozen LLMs + Dual Projection Heads: A New System 1 Fast-Inference Baseline for Medical Multiple-Choice Questions

Frozen LLMs with only projection heads trained achieve 46% accuracy on medical multiple-choice via a single forward pass—a cheap baseline.
This article introduces an arXiv study that attempts to solve medical exam multiple-choice questions with a System 1-style "fast thinking" architecture: freeze three encoders (Qwen3.5-0.8B, Granite-4.2-3B, Qwen3-8B), train only two small projection heads and a learnable temperature parameter, and use contrastive loss to map questions and options into a shared space, selecting the nearest as the answer. On 3,842 held-out MedMCQA questions, raw last-token vectors yielded about 33% accuracy; adding a "one-word summary" prompt and lowering the head learning rate raised this to 41.7% and 42.5%, with Qwen3-8B reaching 46.0%. Shuffling question-option pairings dropped accuracy back to random level, proving the model genuinely establishes question-option associations. The authors explicitly position it as a cheap, fast baseline, still clearly behind Med-PaLM 2's 72.3%, and proactively disclose limitations such as validation gains being about 1.7 to 2 times test gains and the effect of multi-layer reading being non-attributable.
Can Medical Multiple-Choice Questions Also Use "Fast Thinking"?
Multiple-choice questions on medical exams have long served as a touchstone for testing the breadth of a model's knowledge. The conventional approach relies on a large language model performing a complete forward inference pass, which is costly and high-latency. A study from arXiv proposes a System 1-style decision model: freeze the language model, train only two small projection heads, map the question and options into a shared space, and let the nearest one win—a single forward pass yields the answer.
The study reproduces this recipe from scratch on the MedMCQA dataset and systematically evaluates how far it can go on exam questions that require medical knowledge. The core motivation is: for such questions, the vector similarity between the correct option and the question is often no higher than that of incorrect options—is "fast thinking" that relies solely on representational similarity sufficient? The authors validate this across three frozen models.
The cognitive metaphor of System 1 and System 2 originates from Daniel Kahneman's dual-system theory: System 1 refers to fast, automatic, intuitive thinking, while System 2 corresponds to slow, deliberate analytical reasoning that requires working memory. In the context of artificial intelligence, especially large language models, this pair of concepts has been borrowed as an architectural design metaphor—System 1 typically refers to a shallow decision path with a single forward pass, low latency, and low cost; System 2 refers to expensive but more reliable reasoning mechanisms such as multiple sampling, chain-of-thought reasoning, tool calling, or search. The "fast thinking" model discussed in this paper corresponds to the former: frozen backbone, training only lightweight projection heads, and an answer from a single encoding pass. The significance of this distinction is that it suggests system designers can deploy compute in tiers according to task difficulty, rather than invoking the heaviest reasoning pipeline for every input.
Method Breakdown: Training Only Two Heads and One Temperature
The model architecture is extremely lightweight. A frozen encoder (using Qwen3.5-0.8B, Granite-4.2-3B, and Qwen3-8B respectively) encodes the question and each option into vectors separately. Two small trainable projection heads project the two types of vectors into the same space, and a learnable temperature parameter is used to compute similarity, with the closest option being the predicted answer.
The training objective uses contrastive loss, combining in-batch negatives with hard negatives, and applies masking to duplicate answers. Throughout the process, the encoder parameters are completely frozen, and the number of trainable parameters is compressed to an extremely small scale, making both training and inference costs far lower than fine-tuning the entire model.
An easily overlooked detail is prompt design. The study wraps each text segment in a "one-word summary" prompt and lowers the learning rate of the projection heads—these two changes brought significant improvements.
Contrastive loss with in-batch negatives and hard negatives are core mechanisms in contrastive learning. The goal of contrastive loss is to pull positive sample pairs closer in the embedding space and push negative pairs farther apart. In a multiple-choice setting, the correct option and the question form a positive pair, and the remaining options naturally become negatives. In-batch negatives refer to reusing the positives of other samples within the same batch as negatives for the current sample, thereby greatly expanding the number of negatives without increasing encoding cost. Hard negatives refer to options that are highly similar to the question but are not the correct answer—they lie near the decision boundary and are more informative for training. If only random negatives are used, the model tends to learn a decision boundary that is too simple; introducing hard negatives forces the projection heads to learn finer semantic distinctions. This paper combines both and masks duplicate answers to prevent the same answer from being mistakenly treated as a negative due to repeated appearance, thereby contaminating the training signal.
Experimental Results: From One-Third to Nearly Half
On 3,842 held-out questions (random guessing accuracy is 25.0%), the two small encoders using raw last-token vectors achieved accuracies of 33.1% and 33.0%, respectively—roughly equivalent to answering one out of every three questions correctly.
After introducing the one-word summary prompt and lowering the head learning rate, the accuracy of the two small encoders improved to 41.7% and 42.5%, and Qwen3-8B reached 46.0% ±0.2 (three random seeds). This magnitude of improvement shows that even with a frozen backbone, the way representations are organized has a huge impact on the final decision.
To verify that the projection heads indeed utilize question information, the study shuffled the pairing of questions and options, and accuracy immediately fell back to random level. This constitutes a clean control experiment: the model is not exploiting prior frequencies of the options themselves, but genuinely establishing an association between the two sides of the text.

Gap with Med-PaLM: Positioned as a Cheap Baseline
The authors explicitly acknowledge that this system is not a competitive solution. Flan-PaLM and Med-PaLM 2 report accuracies of 57.6% and 72.3% on MedMCQA under different protocols. In comparison, the best result of 46.0% is still a clear distance away, and the evaluation protocols are inconsistent, so direct comparison requires caution.
The value of this work lies in providing a cheap, fast baseline: a frozen backbone means no need to bear the cost of fine-tuning a large model, and a single forward pass means low latency. For resource-constrained deployment scenarios or pipelines that need to quickly screen answers, this System 1 architecture has practical significance.
MedMCQA and the Med-PaLM series are two key references in medical question-answering evaluation. MedMCQA is a large-scale multiple-choice dataset derived from Indian medical entrance exams, covering a wide range of subjects such as anatomy, pharmacology, and pathology; each question typically provides four options, so random guessing accuracy is 25%. The dataset is widely used as a benchmark for testing the coverage of a model's medical knowledge. Med-PaLM is a series of language models for the medical domain introduced by Google; its first generation was based on Flan-PaLM with medical instruction fine-tuning, and Med-PaLM 2 further introduced expert feedback and more refined alignment strategies, reporting results significantly higher than general models on benchmarks such as MedQA and MedMCQA. It should be noted that different studies vary greatly in protocols such as the number of few-shot examples, option ordering, and whether chain-of-thought prompting is used, so accuracy numbers across papers cannot be directly compared horizontally—the authors of this paper also clearly point this out.
Limitations Acknowledged by the Authors
The study remains restrained regarding the boundaries of evidence. The gain on the validation set is about 1.7 to 2 times the gain on the test set, a gap that suggests some degree of overfitting or distribution shift; readers should not directly extrapolate validation set numbers to real exam scenarios.
Another variable that could not be isolated is the effect of reading multiple encoder layers, which is mixed with other changes (prompting, learning rate adjustment) and cannot be attributed separately. This means the question of "whether multi-layer reading is critical" remains unresolved, and future research needs to design ablation experiments to clarify it.
Overfitting and distribution shift are two common reasons for the gap between validation and test gains. Overfitting refers to the optimistic bias in performance estimates after repeatedly tuning parameters on the validation set, because validation set information indirectly participates in model selection; distribution shift refers to systematic differences between the validation set and the test set in question sources, subject composition, or difficulty distribution, causing validation performance to fail to represent real scenarios. In this paper, the validation gain is about 1.7 to 2 times the test gain, a ratio that suggests both may be present: hyperparameters such as prompts and learning rate were selected on the validation set, so validation numbers are naturally inflated; and if the held-out test set differs from the validation set in question types, the gap widens further. A prudent approach is to treat validation numbers as upper-bound estimates and re-evaluate on an independent and identically distributed test set, or use cross-validation to reduce selection bias.
Implications for Deploying Medical AI
This work reflects a pragmatic direction: in high-stakes domains such as medical question answering, not every step requires heavyweight reasoning. For well-formatted multiple-choice questions, a lightweight System 1 decision layer can serve as a fast initial screen, leaving complex reasoning to the more expensive System 2 module.
At the same time, the number 46.0% also delineates a clear capability boundary—frozen representations plus shallow projections are not yet sufficient to replace a complete reasoning system with medical knowledge. The researchers' cautious attitude toward their own conclusions, including proactively disclosing the validation-test gain gap and acknowledging confounding variables, is precisely the most worthwhile part of this kind of baseline research to learn from.
Future directions worth watching include whether multi-layer feature fusion can narrow the gap with Med-PaLM, and whether hard-negative mining strategies still have room for optimization on terminology-dense options in the medical domain.
Related articles

Anthropic Goes Public: AI Bubble or Buying Opportunity?
Anthropic lists on Nasdaq, opening about a third above its IPO price. We break down the $81.3B raise, $1.5T offering valuation, and $2T market valuation, plus Claude's growth and $567M quarterly operating loss.

Why the FBI's AI Hacker Warning Hinges on a Single Word from Microsoft
FBI says AI tool Artex attacked 2,250+ organizations, but the citation trail leads to Microsoft's wording of "attempted sign-ins." The gap between attempts and breaches reveals the attack's true scale.

Fable 5.2 Powers the Creation of Getsumen Express: A Browser-Based Lunar and Martian Exploration Game
A developer used Fable 5.2 to build Getsumen Express, a browser game where players explore the Moon and Mars, scan real landmarks, shoot aliens, and ride an interstellar train. Built on Three.js and WebGPU for about $730, it showcases AI-assisted development.