VisKG-LM: Compiling Knowledge Graphs into Visual Memory to Solve the Repeated Encoding Problem in QA

A Long-Overlooked Efficiency Black Hole

In knowledge graph-augmented question answering systems, there's an inefficiency that has been almost universally accepted: every time a model scores a "question–candidate answer" pair, it re-encodes the retrieved subgraph from scratch. This happens across every training epoch, every random seed, and every evaluation run — even though the underlying knowledge graph never changes.

The conventional approach uses graph neural networks (GNNs) to encode retrieved subgraphs, then fuses them with a language model along the online inference path. This means the same subgraphs are repeatedly and redundantly recomputed, resulting in enormous computational waste. A new arXiv paper introducing VisKG-LM challenges this inertia with a straightforward question: if the knowledge graph doesn't change, why not compile it once and access it as read-only memory afterward?

A Graph Neural Network (GNN) is a neural architecture designed specifically for graph-structured data. Unlike Transformers, which process sequences, or CNNs, which process grids, GNNs use a "message passing" mechanism where each node aggregates information from its neighbors, learning node representations that incorporate local topological structure. In knowledge graph QA, the standard approach is to first retrieve a relevant subgraph based on the question, run multi-round message propagation with a GNN, and then fuse the resulting graph representation with the language model's text representation for scoring. The computational cost of this process is especially pronounced during training: each training sample requires a full, independent round of message propagation over its subgraph, and the same subgraphs are processed repeatedly across multiple training epochs — creating massive redundant computation.

The Core Idea: "Drawing" the Graph as an Image

VisKG-LM's key innovation is decoupling graph encoding from language reasoning. Instead of running a GNN repeatedly at inference time, it performs a one-time, clever transformation of each candidate answer's subgraph.

The process has two steps:

Step 1: Serialize into Relation-Labeled Paths

First, the system serializes the retrieved candidate subgraph into Relation-Labeled Paths — decomposing the graph structure into a series of paths, each annotated with relation labels. This is a way of linearizing the semantic information contained in the graph.

Relation-Labeled Paths are a standardized representation that converts graph structures into linear sequences. Concretely, a path in a knowledge graph consists of an alternating chain of "entity–relation–entity–relation–…–entity" triples — for example, "Aspirin → used to treat → Headache → is a symptom of → Neurological Disease." The advantage of serializing graphs into such paths is direct compatibility with language model text inputs. The trade-off is losing the graph's branching structure: crossing nodes and parallel relations between multiple paths lose their spatial association once unfolded into independent sequences. VisKG-LM's key insight is precisely this: simple textual path linearization is insufficient to fully preserve graph topology, so a two-dimensional image is needed to explicitly encode that structure.

Step 2: Render into Structure-Preserving Images

The second step is where things get interesting: VisKG-LM renders these paths into an image, with a two-dimensional layout that deliberately preserves the branching structure of the paths. In other words, the graph's topological information isn't flattened into text — it's encoded into the spatial layout of an image, where the global composition reflects the overall structure and local details preserve specific relations.

Each such image only needs to be encoded once, offline, then cached for subsequent reuse. This fundamentally eliminates the repeated graph propagation computation during the online phase.

Inference-Time Division of Labor: Understand Text First, Then Consult Memory

VisKG-LM's inference-stage design reflects a clear ordering of information intake. The language model first contextualizes the question and candidate answers using text alone. Only at its final layer does it consult the cached visual memory — reading both the global layout and local relational details of the image.

This arrangement means graph knowledge only enters the model after the text has been thoroughly understood. This "text-first, graph-later" mechanism prevents graph information from interfering prematurely with language reasoning, and lets the cached visual memory play the role of a genuine "read-only reference" rather than a deeply entangled fusion target.

Experimental Results: Small Parameter Count, Big Impact

The paper validates the approach on three mainstream benchmarks: CommonsenseQA, OpenBookQA, and MedQA-USMLE.

Compared to the classic GreaseLM baseline, VisKG-LM improves by 1.2, 0.8, and 4.3 points, respectively. The especially significant gain on MedQA-USMLE suggests that the visual memory mechanism holds a stronger advantage in scenarios requiring complex relational reasoning.

More noteworthy is the efficiency comparison: VisKG-LM uses only approximately 400M online parameters yet matches or exceeds GraphVis, a visual-language model with 7B parameters. A nearly 20× parameter gap, yet comparable or superior performance — a strong demonstration of the cost-effectiveness of the "offline compilation + cached reuse" approach.

What Does Visual Memory Actually Add?

To prove that the performance gains don't simply come from path textualization, the paper includes a key ablation: a text-only model that receives the same Relation-Labeled Path text but no image.

The results show VisKG-LM leads this text-only baseline by 4.2, 6.5, and 5.1 points on the three benchmarks, respectively. This demonstrates that the full visual memory interface provides value beyond simply converting paths to text — the two-dimensional image layout itself carries additional, useful structural information.

GreaseLM is a classic baseline in knowledge graph QA. Its core idea is to perform bidirectional fusion between a language model (such as RoBERTa) and a GNN at every layer, deeply interweaving text and graph representations throughout the inference process. CommonsenseQA, OpenBookQA, and MedQA-USMLE are three standard benchmarks with distinct focuses: CommonsenseQA tests everyday commonsense reasoning, OpenBookQA emphasizes combining elementary science knowledge with external memory, and MedQA-USMLE is drawn from the United States Medical Licensing Examination, requiring multi-step reasoning over complex medical concept relationships. The larger 4.3-point gain on MedQA-USMLE is partly attributable to the high relational density and long reasoning chains in medical knowledge graphs — exactly the scenarios where the 2D visual layout's ability to preserve multi-path crossing structures proves most valuable.

Implications for RAG and Knowledge Graph Applications

The "compiled visual memory" approach proposed by VisKG-LM offers an alternative path to online graph propagation, with implications that may extend well beyond knowledge graph QA.

It reveals a broadly applicable engineering insight: for static, unchanging knowledge sources, rather than re-encoding at every inference step, precompute the encoding and cache it as reusable memory. This aligns well with growing interest in caching and precomputation within the RAG (Retrieval-Augmented Generation) space.

Rendering structured data as images and leveraging vision models' spatial understanding to read them is also a novel interface design idea. Of course, this remains an arXiv preprint — its generalization ability, performance on larger-scale knowledge graphs, and the cost of image rendering itself all await further validation. But it at least demonstrates that knowledge graphs don't need to be re-encoded from scratch every time: "compile once, reuse everywhere" is viable.

RAG (Retrieval-Augmented Generation) is a paradigm that dynamically retrieves relevant information from external knowledge bases at inference time and feeds it to the language model. It is widely used to reduce hallucinations in large models and extend knowledge currency. Traditional RAG requires real-time retrieval and encoding at every query — when the knowledge base is large and relatively stable, this similarly produces large amounts of redundant computation. In recent years, both academia and engineering practice have explored "pre-computing document embeddings and caching the index" as an optimization. VisKG-LM's "offline compiled image memory" can be seen as an extension of this idea to structured knowledge — the key difference being that what's cached is not a vector embedding but a visualization that preserves topological semantics, which can be read directly by a multimodal model's visual channel. This provides a structured knowledge access method that doesn't rely on traditional vector retrieval.