Superfluid: A Local LLM Inference Server Built for Multi-Agent Workloads

When Local AI Goes from 'One Person Chatting' to 'Eight Agents Fighting Over One GPU'

The use cases for local large models are undergoing a structural shift. In the past, local inference needs were simple — a user opens a terminal, chats back and forth with a large model, one request at a time. Now the situation is entirely different: one agent sorts emails overnight, one monitors health data, one handles price comparison shopping, and the user wants to jump in with questions at any moment. Eight agents plus one human, all sharing a single laptop's compute.

The problem is that today's mainstream local inference servers — llama-server, Ollama, mlx_lm.server — are all designed for data center rack throughput. There's no priority scheduling, no checkpoint recovery. Your question queues up behind whatever batch is already running; restart the process and all sessions vanish together.

Superfluid tries to solve this problem. It's an Apache-2.0 open-source local LLM inference server, positioned around single-machine multi-agent inference.

Interactive Requests Shouldn't Queue Behind Batch Jobs

Superfluid's most central design decision is at the scheduling layer: interactive requests preempt batch work in the next scheduling cycle, rather than queuing behind it.

This difference is extremely visible under real workloads. The Superfluid team ran eight agents each processing a 7.5k token document, then issued a chat request at the second mark — Superfluid answered in 0.36 seconds, while the same model running on llama-server hadn't even emitted its first token after more than 10 seconds. A 30x gap in first-token latency.

This isn't a model-level optimization — it's a trade-off at the serving architecture level. Data center inference servers pursue maximum total throughput, and sacrificing the responsiveness of individual requests is acceptable; but in multi-agent scenarios on personal devices, the human user's interactive requests must take priority, otherwise the "local AI assistant" experience simply doesn't hold up. According to official test data, with eight agents running, chat requests have a first-token latency of 1.5 seconds, whereas llama-server needs 291 seconds, Ollama needs 380 seconds, and mlx_lm.server needs 612 seconds.

Time to First Token (TTFT) is the key metric for measuring interactive inference experience — the interval between when a client sends a request and when it receives the first generated token. It consists of two parts: queueing wait time and the prefill computation time for the prompt. Under multi-agent concurrency, the GPU is usually saturated by batch processing of long tasks; if a newly arrived interactive request uses first-come-first-served (FCFS) or continuous batching, it gets stuffed into the tail of the current or next batch, waiting tens of seconds or even minutes. This is the root cause of the extremely long TTFT in solutions like llama-server — they pursue total throughput, not fairness and responsiveness for individual requests. Preemptive scheduling, by contrast, allows high-priority requests to interrupt an in-progress batch and immediately obtain compute in that scheduling cycle, at the cost of some batch processing efficiency. For the load structure on personal devices — where "humans can't wait, agents can" — this trade-off is reasonable.

Every Token Is Logged Before It's Sent

Superfluid's second key design centers on fault tolerance: every token is written to a log before being sent to the client. This means that after the inference process is killed or the server restarts, sessions can continue from the checkpoint without losing data.

Official measured results: after an inference worker is killed mid-stream, Superfluid recovers service within 1.8 seconds; Ollama also recovers in 1.8 seconds but the stream is truncated; llama-server and mlx_lm.server simply don't have this capability.

For a local environment that needs to run multiple agents 24/7, "restart equals lost sessions" is fatal. Superfluid puts persistence on the core path rather than treating it as an optional plugin.

Write-Ahead Logging (WAL) is a classic mechanism from the database world for guaranteeing durability and crash consistency: before actually applying a change, the operation is first appended to a log. Superfluid brings this idea to token streaming — every token is written to disk as a log entry first, then sent to the client. This way, even if the worker crashes, the recovery process can replay the log to know how far generation had progressed, and continue from the checkpoint. By contrast, in ordinary streaming inference, tokens exist only in a memory buffer; once the process dies, the buffer content is lost with it, and the client only receives a truncated stream. WAL incurs additional disk write overhead, but this can be pushed quite low through batched flushing (fsync strategies). For multi-agent environments that need to run 24/7 and whose session state can't be rebuilt, the recoverability bought by this overhead is often worth it.

One API Set Covering Three Runtimes

Superfluid simultaneously supports three API protocols — OpenAI, Anthropic, and Ollama — and underneath connects to three inference runtimes: llama.cpp for GGUF models, MLX for MLX directories, and the BaseRT engine for .base model packages.

This means a single server instance can directly take over requests from all kinds of clients — Claude Code, Codex, Cline, OpenCode, Ollama CLI, and more:

superfluid launch claude --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

It also has a shared prefix caching optimization: when 32 requests share the same 4k-token system prompt, Superfluid achieves a 97% cache hit rate, on par with Ollama, while mlx_lm.server is at 0%. In scenarios where multiple agents share the same system prompt, this optimization directly eliminates redundant prefill computation.

Prefix caching is an optimization targeting the prefill stage of large models. Since in Transformer inference the computation results for the first several tokens of a prompt depend only on those tokens themselves, they can be reused across multiple requests. In multi-agent scenarios, all agents often share the same long system prompt (for example, 4k tokens of tool descriptions and role definitions). If each request recomputes this prefix, a large amount of compute is wasted. Prefix caching stores this KV cache in VRAM or memory, and on a hit, skips its prefill entirely. The higher the hit rate, the greater the savings; Ollama and Superfluid both reach 97% in this scenario, while the request-by-request mlx_lm.server has a 0% hit rate, meaning it recomputes the entire system prompt every time. For devices running multiple agents concurrently, prefix caching is practically a necessity.

Stitching Multiple Devices into One Inference Machine

Superfluid has built-in multi-device inference. One machine acts as the head node, and others join via token authentication:

superfluid fleet token
superfluid serve unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M --http 0.0.0.0:8453 --fleet-listen 0.0.0.0:8454

After a node executes superfluid node join, it automatically discovers the head node, pulls the model, and serves over a token-encrypted link. The scheduling strategy is whole-session placement — an entire session is placed on a single node, and only sampled tokens are transmitted back to the head node. If a node dies mid-way, the session is replaced, and tokens are not delivered twice.

Different nodes can run different runtimes: BaseRT on a Mac, llama.cpp on the adjacent Linux machine — completely transparent to upper-layer clients.

Whole-session placement is a distributed inference scheduling strategy: all requests of a session are pinned to the same node, rather than splitting a single request across multiple machines for tensor parallelism or pipeline parallelism. Its benefit is that a node only needs to maintain that session's KV cache locally, and inter-node communication is limited to transmitting sampled tokens back, so bandwidth requirements are extremely low — and it avoids the high latency of per-token cross-node synchronization. The cost is coarser load balancing granularity and the fact that a single-node failure interrupts an entire session, requiring additional session migration or replacement mechanisms as a fallback. Superfluid chose this strategy precisely because its target scenario — multiple personal devices interconnected over ordinary networks — can't bear high-bandwidth cross-machine tensor synchronization, whereas whole-session placement still works stably under low bandwidth.

The Boundaries of Performance Trade-offs

One point that needs to be viewed objectively: Superfluid's serving layer itself introduces no additional overhead. Under single requests and eight-concurrent requests, the tokens/s of llama.cpp called through Superfluid matches that of using llama-server directly. Superfluid batches MLX requests, making it more efficient than mlx_lm.server's request-by-request processing.

In other words, Superfluid's gains come mainly from scheduling, caching, and fault tolerance — not from computation acceleration. In single-agent, low-concurrency scenarios, it differs little from existing solutions; only when you're running multiple agents simultaneously, or need long-term stable operation, do the architectural differences become apparent.

If you want to verify these numbers on your own machine, installation and startup take just two commands:

curl -fsSL https://superfluid.sh/install.sh | sh
superfluid serve unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

The project is at github.com/basecompute/superfluid, supporting macOS (Apple silicon) and Linux (x86-64, arm64).