Q64: Freeze the LLM Backbone, Jointly Adapt the Readout Network for Efficiency

In scenarios where multiple questions share a document, recomputing the document representation for every question is a significant waste in current LLM inference. A technical report from rss proposes Q64—a candidate scoring system with a straightforward idea: completely freeze the 64-layer Qwen3.8-27B text backbone, reuse its hybrid prefix states and intermediate document representations, then jointly train a residual adapter, a four-layer decoder, and a readout network conditioned on candidate answers.

Two Opportunities: Reusing Computation and Relearning Scoring

The original article breaks down repeated decision-making over shared documents into two independent opportunities. First, reuse document computation: when the same document is read repeatedly by multiple questions, the document-side representation can be computed only once. Second, adapt the scoring method: how question representations score candidate answers is itself learnable and need not follow the backbone's original readout logic.

Q64 bets on both. Freezing the backbone ensures consistency of document representations and makes reuse possible; jointly training the residual adapter and readout network reorganizes the "question—candidate" scoring relationship on top of the frozen representations. This combination is a system-level design, not an improvement to some isolated component—the original article specifically emphasizes this point in its conclusion.

Prefix State Reuse: In Transformer-type models, when processing the same document, the key-value cache (KV Cache) and prefix hidden states produced at each layer depend only on the document itself, independent of subsequent questions. When multiple questions share the same document, if the backbone is frozen, these states can be computed once and directly reused in subsequent requests, avoiding repeated forward propagation for each question. This is the main source of Q64's latency benefit, and it explains why the benefit only appears in "one document, multiple questions" workloads—single-question requests cannot amortize the initial computation cost.

Candidate-Conditioned Readout: The traditional approach uses dot products or simple concatenation between question representations and candidate answer representations for scoring. Candidate-conditioned readout instead takes the candidate answer as an additional input, allowing the readout network to dynamically generate scoring weights and thereby relearn the "question—candidate" interaction on top of the frozen backbone. It does not change document representations, only the final decision layer, so it has few trainable parameters and minimal interference with backbone knowledge.

88.66M Parameters and 8,192 Training Views

The final adaptation stage updates only 88.66 million parameters out of the 88.66M-scale totals, i.e., 88.66 million parameters, with training data consisting of 8,192 curated views. Relative to the 27B backbone, this trainable scale is quite restrained, indicating that the authors intended to concentrate changes in the readout and adaptation layers without touching the backbone's knowledge.

The design exploration process leveraged ZiYor—a self-evolving AI system. The authors used it to explore and refine Q64's design, then independently specified the scoring method, implemented the system, and evaluated accuracy and latency. This "AI-assisted design + human implementation and verification" workflow is itself a working approach worth noting in current systems research.

Residual Adapter: A lightweight fine-tuning structure that typically inserts a small trainable module alongside frozen layers, with its output added back to the original layer output in a residual manner. It does not modify backbone weights, only learning offsets to the frozen representations, thus adapting to new tasks while preserving the original model's knowledge. In Q64, the residual adapter is jointly trained with the four-layer decoder and readout network, collectively handling the mapping from frozen document representations to candidate scoring.

Self-evolving AI system ZiYor: Refers to AI systems capable of exploring, generating, and optimizing design solutions through their own iteration. In this article, the authors used ZiYor to assist in searching Q64's architecture and training configurations, but the final scoring method, system implementation, and evaluation were still completed and verified by humans. This division of labor—"AI explores the design space + humans implement and verify"—is a collaborative paradigm gradually emerging in current systems research, whose value lies in expanding design search coverage rather than replacing engineering judgment.

Accuracy: 74.32% — Progress and Gap

The evaluation set is a fixed 5,112 samples, mixing ARC-Challenge, ANLI-R3, and stratified subsets of MMLU-Pro and CMMLU. Joint adaptation raises the four-benchmark macro-average accuracy from 66.46% to 74.32%—a notable improvement.

But the original article does not shy away from the shortcoming: this figure remains below the native 27B baseline's 75.68% and below AnyJev Qwen3-32B L0's 76.68%. In other words, Q64 does not surpass the unmodified native model in accuracy, let alone larger comparison configurations. The original article characterizes this as evidence supporting an accuracy—latency trade-off, rather than advantages from backbone reduction or isolated components.

Latency: 2.237 Seconds vs. 56.872 Seconds

Latency data comes from a separate synthetic workload: 4,096 document tokens and 50 questions. Q64's median cold-request latency is 2.237 seconds, versus 5.383 seconds for the native baseline and 56.872 seconds for the measured batched AnyJev configuration. Under the setting of multiple questions sharing long documents, the benefit from reusing document computation is on an order-of-magnitude level.

The original article also provides an important boundary condition: short single-question requests do not share this advantage. This is intuitive—when a document is read only once, prefix reuse is irrelevant, and the adaptation layer may instead introduce additional overhead. Therefore Q64's applicability has a clear precondition: "same document, multiple questions" repeated decision-making workloads.

Median Cold-Request Latency: Cold requests refer to requests that do not rely on existing caches and must process the document from scratch. The report uses the median rather than the mean to reduce the influence of occasional long-tail requests on the overall metric and more robustly reflect typical experience. Q64's 2.237 seconds corresponds to a synthetic workload of 4,096 document tokens and 50 questions, where document-side computation is reused, so the latency amortized per question is significantly lower than the native baseline's 5.383 seconds from per-question recomputation; the batched AnyJev configuration's 56.872 seconds reflects serial overhead without prefix reuse.

Evidence Boundaries and Open Questions

The authors express their conclusions with considerable restraint. Public benchmarks were only used in early development stages, and validation on real documents with multiple questions remains open. In other words, current results are built on a fixed evaluation set and synthetic workloads, and there is still a validation gap before reaching real document scenarios in production environments.

From an engineering perspective, Q64's value lies not in topping some leaderboard, but in providing a reusable system paradigm: freeze the backbone, reuse prefix states, perform joint adaptation at the readout layer, and trade a small number of trainable parameters for latency benefits in multi-question scenarios. Whether this paradigm can maintain the same trade-off curve on real document distributions is the most noteworthy question for follow-up work.

One-line Summary

Q64 freezes a 27B backbone, reuses document prefix states, and trains only 88.66M parameters to gain latency benefits in multi-question scenarios, but accuracy does not exceed the native baseline.

Paragraph Summary

Q64 is a candidate scoring system for scenarios where multiple questions share a document. It freezes the 64-layer Qwen3.8-27B text backbone, reuses hybrid prefix states and intermediate document representations, and jointly trains only a residual adapter, a four-layer decoder, and a candidate-conditioned readout network. The final adaptation stage updates 88.66 million parameters using 8,192 curated training views, with design exploration assisted by the self-evolving AI system ZiYor. On a fixed evaluation set of 5,112 samples, the four-benchmark macro-average accuracy rises from 66.46% to 74.32%, but remains below the native 27B's 75.68% and AnyJev Qwen3-32B L0's 76.68%. For latency, under a synthetic workload of 4,096 document tokens and 50 questions, median cold-request latency is 2.237 seconds, versus 5.383 seconds for the native baseline and 56.872 seconds for the batched AnyJev configuration. The authors explicitly limit their conclusion to an accuracy—latency trade-off and note that short single-question requests do not share the latency advantage, with multi-question validation on real documents still open.