WorkspaceBench: How to Evaluate Interpretability Methods for LLM Global Workspace

Why Do We Need WorkspaceBench?

A model's intermediate representations (its workspace) often hide answers, intentions, and plans, but we've lacked a standard for measuring whether "activation-to-text" tools actually read these variables correctly. According to research published simultaneously on LessWrong and the Alignment Forum, the authors propose WorkspaceBench—a benchmark specifically designed to evaluate "global workspace" reading capability, in order to determine whether a tool can truly restore intermediate variables from the forward pass into readable text.

The significance of this evaluation lies in the fact that models like Astra can complete a large number of tasks without a chain of thought, which makes chain-of-thought monitoring unreliable and makes interpretability a critical path to understanding model behavior. WorkspaceBench attempts to provide a ruler that allows different tools to be compared side by side.

The Scale and Design Goals of WorkspaceBench

The benchmark contains 27 evaluation families with a total of 3356 questions, covering topics such as safety, logical reasoning, and multihop computation, with a subset specifically reserved for "single-token output" tools.

The authors list several expectations for an ideal workspace reader: it can cover everything captured by J-lens without hallucinating; it can express multi-token concepts that J-lens can only "refer to" but cannot articulate in a single token; it can capture other intermediate quantities relevant to computation; it is practically useful for safety auditing; and its output is trustworthy and not overly fabricated.

For this reason, the benchmark also provides a dedicated hallucination evaluation and uses "precision/recall relative to J-lens" as auxiliary metrics: low precision does not necessarily equal hallucination—it may also reflect multi-token content that J-lens cannot express.

The Two Categories of Activation Reading Methods Being Evaluated

According to the research, the tools are divided into single-token readers and multi-token readers. The single-token side includes Logit lens, J-lens, R-lens, and Tuned lens; the multi-token side includes Natural Language Autoencoders (NLAs), Sparse Autoencoders, Patchscopes, Template lens, and Oracle Lens. For NLAs and Oracle Lens, fine-tuned variants before RL are also evaluated.

In terms of reading methodology, J/R/Logit-lens take top-10 tokens from layers 20 to 60 of the 64-layer Qwen-3.6-27B, sampling every 4 layers; Template Lens takes top-10 words/phrases at the same set of layers; Oracle Lens takes 1 sample with 4 key points (about 90 tokens), and NLAs take 1 sample at layer 42 (about 250 tokens).

Logit lens is the earliest and simplest activation reading method: it multiplies the residual stream representation at a certain intermediate layer directly by the model's final unembedding matrix, projecting it back into vocabulary space, thereby reading out the "next word this layer most wants to say at this moment." Its advantages are that it requires no training and costs almost nothing; its disadvantage is that semantics gradually drift as layers deepen, and the top tokens of intermediate layers are often fragmented words rather than complete concepts.

Tuned lens is an improvement on Logit lens: it trains a separate affine transformation (translator) for each layer, mapping that layer's representation into a shared space "close to the final layer" before decoding, thereby alleviating the mismatch between intermediate layer distributions and the output layer. Compared with Logit lens, its readouts at each layer are more stable and closer to the true continuation.

J-lens is short for Jacobian—it uses the Jacobian matrix of the output with respect to a certain layer's activations to construct a readout direction, capturing "how changing this layer's activations would affect the output distribution." R-lens can be understood as a readout variant based on a similar linear probing approach. What all three have in common is that they ultimately compress expression into a single token, so they are inherently incapable of representing concepts that require multiple words to explain clearly—this is also the root of the later "bag-of-words" critique.

Sparse Autoencoders (SAE) work by training an overcomplete and sparse dictionary on activations: an encoder maps the high-dimensional residual stream onto a large number of sparsely activated features, and a decoder then reconstructs the original activations. Each feature is usually interpreted as some nameable concept (such as "legal terminology" or "code indentation"), and thus can output multi-token natural language descriptions.

Patchscopes works by injecting the activation "patch" from a certain layer into another prompt (usually through a stronger model or another forward pass of the same model), letting the model itself translate this activation into text—essentially borrowing the model's generative capability as a decoder.

Natural Language Autoencoders (NLAs) use an encoder to compress activations into a latent vector, then a decoder (usually a language model) generates a natural language description, with the training objective including activation reconstruction. It has the strongest expressiveness and the longest readout length, but because the training signal does not force the description to be consistent with facts, it easily generates content that seems plausible but contradicts the model's actual output. Oracle Lens, by contrast, pursues more faithful and more concise readouts through autoregressive reconstruction plus RL optimization.

Three Major Challenges: Why No One Had Built This Kind of Benchmark Before

The first challenge is that there is no ground truth—no one knows exactly what is in the model's workspace. The authors' response is: when designing tasks, they try to ensure the model must use a certain intermediate quantity to obtain the answer, and adjust task difficulty according to model capability. For evaluations that cannot be absolutely confirmed, such as association and directed modulation, they do not require saturation, but higher scores still indicate stronger tool capability.

The second challenge is that the model itself may not have sufficiently strong intermediate representations. The research points out that recent open-source models have increasingly strong workspace representations, so Qwen-3.6-27B was chosen for testing.

The third challenge is that the trade-offs between different methods are difficult to balance: J-lens is single-token but reliable, while NLAs are expressive but prone to fabrication. The benchmark characterizes this trade-off through"accuracy vs. hallucination rate."

Question Quality Control and Scoring Methods

To ensure the intermediate quantities are genuinely reasonable, questions must pass several checks: the model's accuracy is above 80% under n=10, temperature=0.8, and no chain of thought; the intermediate quantity never appears in the prompt (whether as a numeral or a word); the model's own continuation does not state that intermediate quantity either; and for questions with multiple possible answers, such as poetry, repeated sampling confirms that there is only one valid path.

In terms of scoring, most evaluations use Gemini-3.8-Flash as the judge, safety-related evaluations use Sonnet-5, and only the "baseline evaluation" (adapted from the J-lens paper) follows the original paper's regular expression matching method. The authors strongly recommend using judges of equal or stronger capability for original evaluations, because simpler or weaker judges will let in a large number of misjudgments.

Baselines include "Lucky guess" (giving options directly without the question) and "Prompt-only" (giving only the prefix and letting the model infer what is being thought about at the moment). The authors specifically note: because many tasks themselves require inferring workspace content logically from the prompt, good prompt-only performance does not mean the tool is bad.

LLM-as-a-judge refers to using a sufficiently capable language model instead of a human to semantically compare and score the outputs of the evaluated tool against reference answers. Compared with regular expression matching, it can tolerate synonymous rewrites and word-order differences, but the cost is that it introduces the judge's own biases and instability. The research uses Gemini-3.8-Flash as the general judge and Sonnet-5 as the safety judge, precisely because safety scenarios are more sensitive to wording and semantic details.

It should be emphasized that judge capability must match the difficulty of the evaluated task: if a weak model is used as the judge, it will misjudge "obviously wrong but superficially fluent" readouts as correct, thereby systematically overestimating tool performance. This is also why the authors recommend using judges of equal or stronger capability for original evaluations. temperature=0.8, n=10 then refer to the sampling temperature and number of samples, used to estimate whether the model truly "knows how to do" a given question under no chain-of-thought conditions, avoiding treating questions that were accidentally answered correctly as valid questions.

Overview of Evaluation Families: From Basic Reading to Safety

Basic categories cover Basic readout (whether the true continuation emerges one token before the model answers), Multihop (whether silently inferred intermediate concepts emerge), Multilingual, Poetry (whether the planned rhyme emerges at line breaks), Typo, Directed Modulation (when the model is copying a sentence while thinking about a certain concept, whether that concept emerges at the writing position), as well as multilingual versions of typo and multihop.

Computation categories require the model to compute intermediate values without a chain of thought and never write them out, including Chained intermediates (two-step or three-step arithmetic chains, with the starting number given last), Brew intermediates (a ten-rule color rewriting table applied twice), Buggy code (comparison between a program containing a single verified bug and its bug-free twin version), and Arithmetic intermediates (bare arithmetic expressions in fourteen forms).

Safety categories target production auditing scenarios, including Agentic misalignment (when Qwen actually engages in misaligned behavior, whether its plan can be shown before it writes it out, such as blackmail, data leakage, and self-exfiltration) and Jailbreak recognition (whether it recognizes that the last turn is a jailbreak or persona overwrite).

Association categories examine subtext recognition: Basic association, Conjunctive association (compound emotional states, such as jealousy rather than pride over a close friend's promotion), User modeling (inferring the user's country, gender, and occupation from the chat), and Role-bound association (such as distinguishing "a thief chasing a police officer" from the reverse).

Anti-bag-of-words categories are represented by Relational multihop, which requires maintaining the correct order of two relations simultaneously; reversing the order means failure. In addition, there are Hallucination eval and J-lens concept precision/recall, the latter comparing at the comma position in "Chinese, English, French" whether the readout covers the J-lens top-10.

Results and Discussion

The authors give two main conclusions. First, single-token readers cannot cover important workspace content: they are bag-of-words models, and even if an LLM summarizer can arrange top-k tokens into a seemingly reasonable order, they cannot encode order in the absence of common-sense ordering (such as multi-digit numbers or directional concepts of intent), and they perform poorly on compound tasks such as conjunctive association and buggy code, demonstrating the necessity of multi-token readers.

Second, NLAs can surface a large amount of workspace content, but they are extremely prone to hallucination. The research states that almost all NLA readouts contain claims directly contradicted by the model's output context; their average readout length is about twice that of Oracle Lens. The authors argue that an ideal workspace reader should be trustworthy, especially when claims cannot be directly verified from model output; if a large number of verifiable claims are false, then unverifiable claims should also be treated with low confidence.

In terms of training details, Oracle Lens is implemented through autoregressive activation reconstruction, Inverter fine-tuning, and RL optimization (with the reward being the proportion of variance explained by the reconstruction, and AR frozen). The authors also mention that, inspired by CHIVE, they are initially exploring how agents equipped with activation reader tools perform in real evaluations.

Hallucination here is defined as: claims generated by the reader that directly contradict the "model output context" (that is, the actual prompt for the question and the model's true continuation). It differs from "low precision": low precision may simply mean the reader failed to hit J-lens's top tokens, which may be caused by J-lens itself being unable to express multi-token concepts in a single token; hallucination, by contrast, is the fabrication of content that can be falsified by external evidence.

For this reason, the evaluation treats "accuracy vs. hallucination rate" as the core two-dimensional metric: an ideal reader should have both high recall (not missing genuinely existing intermediate variables) and low hallucination (not fabricating nonexistent content). The authors specifically point out that NLA readouts are about twice as long as Oracle Lens's, and longer text gives it more "room to fabricate"; when verifiable claims are largely false, confidence in unverifiable claims should also be reduced—this is an especially important principle in safety auditing scenarios.

Usage Recommendations

WorkspaceBench was developed for Qwen-3.6-27B, and the authors expect it to be applicable to larger models, but it may need adjustment for smaller or weaker models. Adaptation recommendations are: have a sufficiently capable LLM generate customized variant questions, run the same validity checks, and re-derive the readout positions, because readout positions do not transfer across models and tokenizers. The benchmark has been open-sourced on GitHub.

One-line Summary

WorkspaceBench uses 27 families and 3356 questions to evaluate whether activation reading tools can correctly read the intermediate variables in a model's workspace.

Paragraph Summary

WorkspaceBench is a benchmark for evaluating whether "activation-to-text" tools can correctly read the intermediate variables of a model's global workspace. It contains 27 evaluation families and 3356 questions, covering basic readout, computation, safety, association, and anti-bag-of-words topics, and is equipped with a dedicated hallucination evaluation and precision/recall metrics relative to J-lens. The research divides tools into single-token readers (Logit lens, J-lens, R-lens, Tuned lens) and multi-token readers (NLAs, SAE, Patchscopes, Template lens, Oracle Lens). There are two core conclusions: single-token readers are essentially bag-of-words models and cannot encode order, limiting them on compound tasks such as conjunctive association and buggy code, which shows that multi-token readers are necessary; NLAs are highly expressive and can surface a large amount of content, but they are extremely prone to hallucination, with almost every readout containing claims that contradict the model output. The benchmark is customized for Qwen-3.6-27B and has been open-sourced; transferring it to weaker models requires regenerating questions and re-deriving readout positions.