Reusing LLM Inference Across Users: Exploring the Feasibility of Knowledge Graph Caching

A Wasteful Problem of Repeated Computation

A Reddit user raised a thought-provoking observation: if 1,000 users ask essentially the same question in different wordings, we may end up paying for 1,000 separate generations—even though a large portion of the underlying knowledge and reasoning is redundant.

Take the most classic machine learning concept as an example:

• "Explain gradient descent"
• "How does gradient descent work?"
• "Teach me gradient descent from a mathematical perspective"
• "Why does gradient descent converge?"

These requests are not identical, but there is a significant amount of reusable structure shared among them. Traditional inference services treat each request as a completely independent task, generating from scratch—an enormous waste of compute at scale. When we consider the pricing model of mainstream LLM APIs—charged per input and output token—the economic cost of this redundant computation becomes even more intuitive: with a GPT-4-class model, a single detailed technical explanation might consume thousands of output tokens. Multiply that by tens of millions of daily active users, and the cost of redundant inference is staggering.

The Core Idea: Caching Knowledge and Reasoning Structures, Not Just Final Answers

The key insight of this idea is that it goes beyond caching the final text response. The author wants to cache and reuse the knowledge itself and the underlying reasoning structures.

Shared Knowledge Graph Design

In this design, the system maintains a persistent knowledge graph. Each node might contain information like:

Concept: Gradient Descent
Related concepts: Optimization / Derivatives / Convexity / Learning Rate
Knowledge: ...
Reasoning structure: ...
Sources: ...
Confidence: ...
Last verified: ...
Reuse count: ...

When a new user request arrives, the system first performs semantic/intent matching to determine whether the graph already contains reusable nodes. If a match exists and is fresh, it's reused directly; if partially available, a retrieval update is triggered; only when content is missing or outdated does the system initiate crawling or regeneration.

This semantic matching is typically implemented using vector embedding technology: both user queries and graph nodes are mapped into a high-dimensional vector space, and semantically similar nodes are quickly located via metrics like cosine similarity. Unlike traditional exact string matching, this approach can recognize that "how does gradient descent work" and "explain gradient descent" point to the same knowledge node, dramatically improving cache hit rates.

Separation of the Personalization Layer

An interesting aspect of this design is the decoupling of "shared knowledge" from "personalized expression." The author envisions an independent user preference graph:

mathematical_depth = high
verbosity = high
code_preference = high
preferred_language = Python
preferred_framework = NumPy
domain_interest = ML

The underlying knowledge is shared, but the final presentation to users is personalized. For gradient descent, one user might want a brief explanation while another wants a full mathematical derivation with code implementation—the shared reasoning skeleton stays the same, only the final "rendering" layer differs.

This "content-expression" separation architecture is not unfamiliar in software engineering—it resembles the MVC (Model-View-Controller) pattern or the separation of content and styling in frontend development. But in the LLM context, the dimensions of "expression" are far richer than CSS styles—they involve abstraction level, terminology choice, example style, level of detail, and other jointly adjusted dimensions. This means the "rendering" layer itself still requires significant model inference capability, just far less than generating a complete answer from scratch.

The Economic Assumptions Behind LLM Inference Reuse

The author uses a simple comparison to summarize the core hypothesis:

• Traditional approach: N users × expensive generation
• Reuse approach: One shared expensive computation + N × relatively cheap retrieval/personalization + occasional crawling/update costs

In other words, extending the principle of "compute once, reuse many times" from low-level caching to the semantic and reasoning layers. Theoretically, if knowledge nodes can be reused at high frequency, marginal costs drop significantly.

The economic logic behind this can be understood with more concrete numbers: in current mainstream large model inference costs, GPU computation (primarily matrix multiplication and attention mechanism calculations) accounts for the vast majority. A typical GPT-4-class inference on an A100 GPU might consume tens of milliseconds to several seconds of compute time, while a vector database retrieval typically completes in milliseconds—a cost difference of two to three orders of magnitude. Therefore, even if building and maintaining the reuse system has significant overhead, as long as knowledge node reuse frequency is high enough (i.e., the knowledge falls into the "high-frequency query" category), the overall ROI can still be positive. But for long-tail queries—those rarely repeated, highly personalized questions—the reuse system's benefit approaches zero or even turns negative.

Comparison with Existing LLM Optimization Techniques

The author is admirably transparent, proactively listing a series of mature technologies that might already cover this idea, and seeking community critique. This list itself serves as an excellent index for understanding LLM inference optimization:

Existing Related Technologies

• Semantic Caching: Caches final answers for semantically similar requests. Semantic caching is a caching strategy based on semantic similarity rather than exact string matching. Traditional caching relies on exact key-value matching, while semantic caching converts user queries into vector embeddings and searches for existing responses to similar requests in vector space. Open-source projects like GPTCache have already implemented this approach—before a query reaches the LLM, it checks whether there's a semantically close enough historical response that can be returned directly, completely avoiding one LLM call.

• KV-Cache / Prefix Caching: Reuses attention computations for shared prefixes. KV-Cache is a core optimization in Transformer architecture inference: during autoregressive generation, each new token requires computing the Key and Value matrices for all preceding tokens in the attention mechanism. KV-Cache stores these intermediate results to avoid redundant computation. Prefix caching extends this further—when multiple requests share the same system prompt or context prefix, this portion of the KV-Cache can be shared across requests. Mainstream inference frameworks like vLLM already natively support automatic prefix caching, significantly reducing time-to-first-token latency and GPU memory usage.

• RAG / GraphRAG: Retrieval-Augmented Generation, with GraphRAG organizing knowledge in graph structures. Traditional RAG splits documents into text chunks stored in vector databases, retrieving the most relevant fragments via semantic similarity to feed to the LLM for answer generation. GraphRAG is an improved approach proposed by Microsoft Research in 2024 that uses LLMs to automatically extract entities and relationships from documents to build a knowledge graph, then applies community detection algorithms to partition the graph into hierarchical community structures with generated summaries, enabling better answers to complex questions requiring cross-document synthesis. However, the LLM call costs during the indexing phase are extremely high, and graph maintenance faces challenges like entity disambiguation and relationship staleness.

• Speculative Decoding: Uses smaller models to accelerate generation. This is a lossless acceleration technique that doesn't alter the model's output distribution. The core idea borrows from CPU branch prediction: a draft model with far fewer parameters quickly generates several candidate tokens, then the target large model verifies these tokens in parallel. Since Transformers in verification mode can process multiple tokens at once, this is far more efficient than token-by-token generation. Google DeepMind systematically proposed this method in 2023, typically achieving 2-3x inference speedup. Subsequent variants like Medusa and Eagle further reduced the dependency on separate draft models.

• Reasoning Trace Reuse: This is a relatively cutting-edge research direction that has become even more important with the emergence of reasoning models like OpenAI o1 and DeepSeek-R1. These models generate lengthy intermediate reasoning steps through Chain-of-Thought, consuming thousands or even tens of thousands of tokens per inference. If sub-steps within these reasoning traces could be cached and reused, costs could theoretically be reduced dramatically. But the challenge is that reasoning traces are far less composable than knowledge fragments—a reasoning step's validity often depends heavily on the context and assumptions established by preceding steps. No mature industrial-grade solution exists yet.

• Multi-Agent Shared Memory: In multi-agent collaboration frameworks (such as AutoGen, CrewAI, etc.), different agents exchange intermediate results and knowledge through shared memory pools to avoid redundant reasoning. This is architecturally very similar to the cross-user knowledge reuse discussed here, just extending the application scenario from single complex task agent collaboration to cross-user, cross-temporal knowledge services.

The Key Distinction

The author astutely points out: semantic caching targets final answers, while what he describes is a persistent graph of reusable knowledge/reasoning substructures. Different requests can reuse overlapping parts, with generation triggered only for genuinely new content.

This does touch on a subtle boundary. GraphRAG focuses on knowledge organization at the retrieval level, while the author goes further, wanting to reuse the "reasoning process" itself—which is precisely the most technically uncertain part. An analogy helps illustrate: GraphRAG is like a well-organized library that efficiently locates relevant books for readers; what this author wants is more like caching "the reading comprehension process"—not just storing the raw knowledge, but also storing the thought path of how to derive B from A, so the next reader needing a similar derivation can directly reuse that path.

Engineering Challenges Facing Reasoning Reuse

Setting aside the novelty of the concept, this vision contains several engineering challenges genuinely worth discussing:

First, can reasoning processes be safely reused? Intermediate reasoning is often highly dependent on the specific prompt and context. The same reasoning chain about gradient descent may need completely different expansion paths in "explaining to a beginner" versus "explaining to a math student" contexts. Hard reuse could introduce factual errors or logical discontinuities. The root of this problem is that LLM reasoning is not formalized logical deduction—it's closer to "probabilistic language generation under specific contextual conditions." Two seemingly identical reasoning steps may have completely different implicit premises and target audiences. In formal mathematical proofs, a lemma can be safely reused across different theorem proofs because its preconditions are explicit; but in LLM reasoning traces, many preconditions are implicit and embedded in context, making the determination of "safe reuse" itself a hard problem.

Second, graphs or vectors? Whether to use graph structures or embedding vector stores to represent these reusable units is an open question. Graphs excel at expressing explicit relationships but are expensive to build and maintain; vector retrieval is flexible but lacks structured semantics. In practice, the industry is exploring hybrid approaches: using vector embeddings for coarse-grained fast recall, then graph structures for fine-grained relationship reasoning and context verification. Graph databases like Neo4j have begun natively supporting vector indexes, while vector databases like Pinecone are experimenting with metadata filtering to simulate some graph query capabilities. The ultimate technology choice likely depends on the characteristics of the knowledge domain—highly structured domains (like medicine or law) are better suited to graph structures, while open-domain knowledge may be better served by vector approaches.

Third, where is the bottleneck? The author himself asks: is the real bottleneck retrieval, verification, context construction, or final generation? In many RAG systems, the quality of retrieval and context construction is often what determines success or failure, not generation itself. This observation is repeatedly validated in engineering practice: many RAG system failures are not due to insufficient LLM generation capability, but because retrieval returned irrelevant fragments, information in the context window was poorly organized, or critical information was truncated. If the retrieval and verification steps of a reasoning reuse system themselves require extensive LLM calls (e.g., determining whether a cached reasoning fragment is still valid in the current context), the savings from reuse may be offset by these additional costs.

Fourth, freshness and verification. The "last verified" and "confidence" fields in knowledge nodes expose a harsh reality: the cost of maintaining a continuously fresh, trustworthy knowledge graph may erode the savings gained from reuse. Taking Google's Knowledge Graph and Wikidata as examples, they have each invested hundreds of person-years of engineering resources to handle entity disambiguation, relationship updates, and quality control. In AI-driven knowledge graphs, automated extraction reduces manual annotation costs, but entity recognition accuracy typically ranges from 85-95%, with relationship extraction being even lower, causing errors to continuously accumulate in the graph. Moreover, knowledge timeliness varies enormously: mathematical theorems almost never become outdated, while technology framework APIs might update quarterly, and Python library best practices may change annually. A differentiated freshness strategy is itself a complex engineering system requiring careful design.

Conclusion: The Direction Has Value, but Reasoning Reuse Is the Greatest Challenge

Objectively speaking, most individual components of this vision already have counterparts in existing research—semantic caching, GraphRAG, KV-Cache reuse, reasoning trace reuse, and so on, are all working toward this direction. What hasn't been perfectly solved is unifying them into a "persistent, reusable reasoning substructure knowledge graph" with a personalization expression layer on top.

The author's value lies not in "inventing something new," but in re-examining the compute waste problem in LLM services from a systems architecture perspective, and honestly comparing against existing technologies to find the boundaries. For anyone looking to go deeper into LLM inference optimization, the technology checklist and sharp questions he raised serve as an excellent learning roadmap.

The conclusion is perhaps this: this direction is technically partially feasible, but the most enticing part—"reusing reasoning processes"—is precisely the most fragile and context-dependent. The real engineering value likely lies in knowledge-layer reuse (closer to GraphRAG) rather than reasoning-layer reuse. From an industry practice perspective, the most pragmatic path today is probably layered optimization: at the infrastructure layer, use KV-Cache and prefix caching to reduce redundant computation; at the knowledge layer, use GraphRAG to organize and reuse factual knowledge; at the application layer, use semantic caching to intercept highly similar requests—while continuing to explore "reasoning trace reuse" as a long-term research direction, awaiting the next breakthrough in Transformer architecture or reasoning methodology.