Open-Source AI Engine StateFrame: Fit More Agents on a GPU and Say Goodbye to Context Bloat

Most AI agents rely on chat-style conversations to maintain context. As interaction turns accumulate, the context window keeps expanding, driving up GPU memory usage and slowing down inference, until the agent is forced into context compaction—which often means information loss. A developer shared StateFrame, an open-source project built over five weeks, on Reddit, aiming to change this from the ground up: store everything about the agent as state, feed the model only what it truly needs at each step, and keep the context window flat.

Core Idea: Replace Conversation History with State

StateFrame's design philosophy is straightforward—an agent doesn't need to remember the entire conversation; it only needs the state required for the current decision at each step. In a traditional chat-style agent, all historical messages are stuffed into the context window, so the more steps, the bigger the window and the higher the management cost. StateFrame instead keeps all information in external state and injects only what is relevant to the current step at each inference.

According to the author, the direct result is that the context window becomes "flat" and predictable: the load at step 300 is comparable to step 3, with no growth over time. For a long-running agent task, this means the resource consumption curve shifts from continuously climbing to a horizontal line.

Context Window refers to the maximum number of tokens a large language model can receive in a single inference, including the system prompt, conversation history, tool outputs, and all other inputs. It is determined by the model architecture (e.g., the complexity of the attention mechanism) and is a hard upper limit. When an agent keeps stuffing the full conversation history into the window, the token count grows linearly with turns, which not only increases GPU memory usage (the KV Cache expands accordingly) but also slows inference as the sequence lengthens. More critically, once it approaches the limit, the agent must compact or discard history, causing information loss. StateFrame's "flat context" targets exactly this bottleneck: through external state management, the input length per inference stays constant, thereby bypassing the capacity constraints of the context window.

Three Practical Benefits of Flat Context

More Agents per GPU

Because each agent's context window size is stable and predictable, memory allocation no longer needs to reserve headroom for "history that might balloon." According to the author, this directly increases the number of agents a single GPU can host. For teams self-hosting multi-agent systems, this is a tangible cost optimization.

Self-Hosted LLMs Become Truly Usable

The author notes that many people self-hosting LLMs are constrained by running out of context space, especially in long-task scenarios. A flat window means the model won't "blow up" its context due to accumulated history, while inference speed stays stable—because it doesn't need to process an ever-lengthening input sequence.

No Compaction, No Lost Clues

Traditional history-based agents often resort to compaction when the context nears its limit—summarizing or truncating old conversations—which easily causes the agent to lose critical information. StateFrame stores everything in state, removing compaction from the mechanism entirely, which the author believes prevents the agent from "losing the thread."

KV Cache is a key optimization in Transformer inference: during autoregressive generation, the model caches the Key and Value matrices of already-processed tokens to avoid redundant computation. However, the cache size is proportional to context length, and long contexts consume substantial GPU memory. In multi-agent concurrent scenarios, each agent maintains its own KV Cache, multiplying memory pressure. StateFrame keeps each agent's context length stable at a small value, so the KV Cache no longer expands with steps, allowing more agent instances to reside on a single card simultaneously. This is the technical root of the author's emphasis on "fitting more agents on a GPU"—what is saved is not model weight memory, but dynamic cache and activation overhead.

Context Compaction is a common fallback strategy for long-conversation agents. Typical approaches include: sliding-window truncation of old messages, using an LLM to summarize history, or discarding segments by importance. While these methods can temporarily free up space, they introduce irreversible information loss—details that were summarized may be exactly the key basis for later decisions, and truncation directly erases early constraints. StateFrame removes compaction at the architectural level: all raw information is persisted in external state storage, and the model retrieves it on demand at each step rather than passively receiving compacted history. This is akin to the difference between a database and a cache—state is the complete source of truth, and context is merely the working set of the current query. As a result, the agent won't "lose the thread" due to compaction, but the trade-off is the need to design reliable state retrieval and update mechanisms.

Open Source and Future Plans

The project is open-sourced on GitHub under the name StateFrame, and the author provides the source code for developers to try. In the post, he admits that during five weeks of development he "ran Foundry's servers into the ground" and teases that more features are on the way, some of which "I haven't seen on other agents."

One detail: a main motivation for the author's Reddit post was to get feedback—he says outright that "it's really hard to get anyone to see this project." The cold-start dilemma of open-source projects is evident here. For developers interested in agent runtime architecture, StateFrame offers a technical path different from mainstream chat-style approaches, and its actual performance and scalability deserve further validation under real workloads.

Foundry in this context refers to the cloud GPU/compute platform the author used during development (such as Modal, RunPod, or similar services) for remote training and inference testing. The author's remark about "running Foundry's servers into the ground" implies a large number of high-frequency experimental iterations over five weeks, which also indirectly shows that StateFrame's validation relies on repeated tuning under real workloads. Developers looking to reproduce or evaluate the project should note that its performance may be closely tied to specific hardware, model scale, and state storage backend, and the out-of-the-box experience may not match the author's test environment.

One-line Summary

Open-source project StateFrame replaces conversation history with external state, keeping the agent's context window flat and preventing bloat.

Paragraph Summary

StateFrame is an open-source agent runtime architecture built by a developer over five weeks. Its core idea is to store all agent information as external state and inject only what is needed for the current decision at each step, rather than accumulating the full conversation history. This keeps the context window "flat"—the load at step 300 is comparable to step 3, turning the resource consumption curve from continuously climbing into a horizontal line. This yields three benefits: more agents per GPU, self-hosted LLMs that don't exhaust context on long tasks, and no compaction so no information loss. The project is open-sourced on GitHub; the author admits promotion is difficult and teases more features still in development, with real-world performance awaiting validation under real workloads.