jlens-gguf: Bringing Jacobian Lens Visualization and Steering to llama.cpp Local LLMs

Making the Inner World of GGUF Models Visible

As large language models grow ever more capable, understanding how they work internally has become a central topic in AI interpretability research. Anthropic's Jacobian Lens method, introduced in their "Global Workspace" research, offers a new way to observe and guide the internal representations of language models. Notably, the name "Global Workspace" draws from cognitive science — specifically Global Workspace Theory (GWT), proposed by Bernard Baars in 1988, which posits that consciousness arises from broadcasting information to a shared "workspace" accessible across the brain. Anthropic draws an analogy between this framework and the residual stream in Transformers: the residual stream acts as a shared channel for information passing between layers, structurally similar to GWT's global broadcast mechanism. This cross-disciplinary perspective has opened new conceptual territory for interpretability research. Until now, however, nearly all related projects have been built around the HuggingFace and PyTorch ecosystem — leaving the GGUF format and llama.cpp engine, beloved by local inference users, largely untouched.

The open-source project jlens-gguf aims to fill that gap. Inspired by Anthropic's paper and code, its developer has built an interactive Jacobian Lens visualization and real-time steering tool specifically for GGUF models and llama.cpp. This means hobbyists and researchers running local LLMs can now explore the internal structure of a model's "thinking" process firsthand.

What Is the Jacobian Lens?

The Core Idea from Anthropic's Research

The Jacobian Lens concept originates from Anthropic's interpretability research. Mathematically, the Jacobian matrix is a tool for describing partial derivatives of multi-variable functions — it measures how sensitive outputs are to local changes in inputs. It's worth noting that Jacobian matrices have a long history in deep learning well beyond interpretability: they are a core tool for computing gradients in backpropagation, since the chain rule is essentially a product of multiple Jacobian matrices. The key insight for interpretability is that a neural network's hidden states are points in a high-dimensional continuous space, and the Jacobian matrix describes the linear approximation of the mapping between this space and the output vocabulary probability space — providing a rigorous mathematical basis for "projecting activations into readable tokens."

To understand how the Jacobian Lens works, you also need to understand the residual stream in Transformers. In a standard Transformer architecture, each layer's output doesn't overwrite its input; instead, it adds the layer's computation to the input vector via a residual connection. This means there is a "trunk stream" running through the entire network, with each layer (attention, feed-forward) writing its results additively into this trunk. This design not only aids gradient flow and training stability, but also provides a natural analytical framework for interpretability — every layer's activations are additive contributions to the same vector space, making cross-layer comparison and directional analysis possible. The Jacobian Lens works by computing the Jacobian of each layer's residual stream activations with respect to the token embedding space, effectively "translating" the state at each node in this trunk stream into a readable vocabulary probability distribution.

In the context of neural networks, the Jacobian matrix can reveal how each layer's activations respond to small changes in input. Anthropic applied this tool to interpretability research by computing the Jacobian of each layer's activations with respect to the token embedding space, "projecting" high-dimensional hidden states back into readable vocabulary probability distributions. This allows researchers to approximately describe, in natural language, what a given layer of the model is "thinking."

This approach follows in the tradition of earlier methods like the "logit lens," but there's an interesting methodological progression worth knowing: Logit Lens, proposed by nostalgebraist in 2020, was among the earliest methods to directly decode intermediate layer activations into vocabulary distributions, revealing how Transformers gradually "form an answer" across layers. Tuned Lens later improved projection quality by training affine probes. The Jacobian Lens goes further by incorporating first-order gradient information, enabling more precise capture of specific semantic directions in activation space. Together, these methods form the "internal representation decoding" paradigm in Transformer interpretability research — and the Jacobian Lens is currently its most refined instantiation.

Using this lens, researchers can not only "see" which internal directions are activated when a model processes a concept, but also intervene on those directions. For readers unfamiliar with academic papers, YouTuber Matthew Berman has produced an explanatory video that introduces this concept more intuitively. The core value is this: the model is no longer a pure black box, but an observable and steerable dynamic system.

From Observation to Real-Time Steering

The significance of the Jacobian Lens goes beyond mere observation. Once certain semantic directions are identified, it becomes possible to modulate the model's internal activations during inference — this is the essence of model steering.

Model steering is an experimental technique in AI interpretability and alignment research. Its core idea is to directly modify intermediate layer activation vectors during inference, influencing model output behavior without changing the weights. This line of research has a clear academic lineage: Anthropic's 2023 "Steering Language Models" work and Turner et al.'s "Activation Addition" paper were among the first to systematically validate this approach, finding that adding directional vectors to specific layers' residual streams can reliably shift a model's output style and content tendencies. Subsequently, Zou et al.'s "Representation Engineering (RepE)" framework systematized this idea further, introducing adversarial probes and control vector mechanisms. Together, these works established the important finding that LLMs contain linearly decodable semantic directions in their internal representations.

Unlike traditional prompt engineering or fine-tuning, steering techniques operate directly on the model's "internal language," enabling more precise behavioral adjustments. Steering can be used to amplify or suppress specific behaviors — and even to perform what's known as abliteration. This technique was systematized by open-source community researchers in 2024: by collecting the difference in intermediate layer activations between a model's responses to "harmful requests" versus "normal requests," using PCA or mean differences to identify direction vectors strongly correlated with "refusal" behavior, and then subtracting that directional component from each layer's activations during inference, it becomes possible to reduce the model's tendency to refuse responses. This technique sparked wide discussion in the open-source community: as a research tool, it reveals that safety behaviors in models are encoded to some extent through specific activation directions — itself an important finding for alignment research — while also opening new possibilities for model alignment, behavior customization, and security research.

Core Capabilities of jlens-gguf

Native GGUF Server

One of the most notable aspects of jlens-gguf is its built-in native GGUF server. To understand the significance of this design, some background on GGUF's evolution is helpful: GGUF was not llama.cpp's first format. Its predecessor, the GGML format (early 2023), hardcoded metadata into the code, meaning every model architecture update required modifying the loader — extremely poor extensibility. GGUF (GPT-Generated Unified Format) was introduced in August 2023 with a "self-describing" design philosophy: all model metadata (architecture type, hyperparameters, tokenizer vocabulary, quantization type, etc.) is stored as key-value pairs in the file header, bundling model weights, hyperparameters, tokenizer, and all other necessary information into a single file. Loaders can correctly parse the file without needing prior knowledge of the model structure.

GGUF's native support for quantization has also been key to its rapid adoption. Quantization is the technique of compressing model weights from high-precision floats (e.g., FP32, BF16) to low-bit integers (e.g., INT8, INT4, or even INT3), dramatically reducing memory footprint and inference compute at the cost of some precision. llama.cpp supports a range of quantization levels from Q2_K to Q8_0, letting users balance hardware constraints against accuracy, while the GGUF file header records quantization type information so the loader can automatically select the appropriate dequantization path. This self-describing feature makes supporting new model architectures much easier, and is a core reason GGUF quickly became the community standard. llama.cpp itself, the open-source C/C++ inference engine started by Georgi Gerganov, uses highly optimized CPU inference and quantization to run billion-parameter models on consumer hardware — together, these two have become the de facto standard for running open-source LLMs locally.

The jlens-gguf server is built on llama.cpp and kept in git sync with its codebase, so it stays current with upstream engine updates. Through this native server, users can not only observe a model's internal state but also perform j-space swapping, abliteration, and real-time steering operations.

jlens-gguf offers two operating modes:

Full Mode: Uses the native GGUF server, enabling both internal observation and real-time model steering.
Observation Mode: Attaches to a running llama-server instance for observation only — steering is not available in this mode.

This design balances flexibility with deep control, accommodating users with different needs.

Support for Both Dense and MoE Architectures

jlens-gguf supports both dense models and Mixture of Experts (MoE) models. Understanding the value of this support requires appreciating the unique interpretability challenges posed by MoE: Mixture of Experts models maintain multiple "expert" sub-networks in each feed-forward layer, using a router mechanism to dynamically select a small number of experts for each input token. This design allows models to dramatically increase parameter counts while keeping inference compute low, but the sparse activation pattern also presents additional challenges for interpretability research. In dense models, every token passes through the same feed-forward layers, so activation paths are deterministic. In MoE, the same concept may activate completely different expert combinations in different contexts — rather than tracking a uniform feed-forward layer, you must trace a dynamically routed expert network. Applying the Jacobian Lens to MoE requires additional handling for the non-differentiability of routing decisions (typically approximated via straight-through estimators) and the effects of unbalanced expert load on gradient computation — significantly more technically demanding than dense models.

As MoE architectures become increasingly common in open-source LLMs — such as the Qwen, Mixtral, and DeepSeek series — native MoE support in tooling is especially critical. This means researchers can apply Jacobian Lens analysis directly to the most mainstream open-source models available today.

Resource Requirements and Practical Considerations

Extra Memory Overhead of About 1/8 the Model Size

Using the Jacobian Lens comes with a cost. According to the developer, the lens's memory requirements scale linearly with model size — roughly 1/8 of the model's size.

For a model of approximately 160GB (such as the UD-Q3_K_XL quantized version of Qwen3.5-397B), running the lens requires roughly 20GB of additional memory. This overhead technically comes from the intermediate gradient storage required for Jacobian matrix computation — computing the Jacobian of each layer's activations with respect to the token embedding space requires maintaining a set of intermediate gradient vectors for each possible projection direction in the vocabulary. The matrix dimensions are directly related to the model's hidden size. Specifically, for a model with hidden dimension $d$, the Jacobian matrix for each layer is of size vocabulary size $V$ times $d$, though in practice low-rank approximations or retaining only the Top-K projection directions are commonly used to control memory overhead. Since hidden size scales approximately linearly with model size, the linear relationship between memory usage and model scale holds intuitively as well. For users running large models locally, this is an additional overhead that needs to be planned for in advance — especially in scenarios where hardware resources are already near their limits.

Positioning Within the Local Inference Ecosystem

The emergence of jlens-gguf reflects the growing demand for interpretability tools within the local inference ecosystem. In the past, deep model interpretability research typically required cloud GPUs and a PyTorch environment, putting it out of reach for most enthusiasts. The GGUF and llama.cpp combination has been a major driver of large model deployment on consumer hardware, with a community far larger than the academic research circle. Bringing the Jacobian Lens into this ecosystem means more people have the opportunity to explore model internals firsthand, rather than just working with model outputs. This trend toward "democratizing research tools" is gradually closing the gap between the academic frontier and everyday developers.

Open-Source Spirit and a Human-AI Collaborative Development Model

The developer candidly notes that this project was completed with AI assistance — using a large language model to tackle core problems, while applying rigorous human oversight, code review, and multiple rounds of testing. This "AI-generated, human-supervised" development model is itself a microcosm of the evolving state of software engineering: humans set the goals, review quality, and bear responsibility, while AI handles the heavy lifting of implementation.

The project is fully open-source on GitHub (igorbarshteyn/jlens-gguf) and explicitly credits Anthropic's original research as well as llama.cpp and GGML as foundational building blocks. This approach of rapidly building tools on the shoulders of giants is a vivid demonstration of the open-source community at its best.

Conclusion: A New Entry Point for Local LLM Interpretability

jlens-gguf fills the gap in the GGUF/llama.cpp ecosystem when it comes to Jacobian Lens tooling, bringing interpretability capabilities previously confined to HuggingFace and PyTorch to a broad audience of local inference users. It supports internal model observation and real-time steering, is compatible with both Dense and MoE architectures, and stays in sync with llama.cpp updates.

While the additional memory overhead and the limitation that steering is only available in full mode are trade-offs users should be aware of, jlens-gguf is a free, open-source project that offers a rare entry point into interpretability research for local LLM deployments. For developers interested in model internals, alignment research, or behavioral customization, this is undoubtedly a new tool worth trying out.