Superfluid
An open-source local LLM inference server designed specifically for single-machine multi-agent inference scenarios, supporting preemptive scheduling, write-ahead log fault tolerance, and a unified API across multiple runtimes.
Timeline (last 90 days)
Superfluid is a local LLM inference server under the Apache-2.0 open-source license, with its core positioning being single-machine multi-agent inference.
Superfluid uses preemptive scheduling, and interactive requests preempt batch work in the next scheduling cycle rather than queuing behind batch tasks.
In a test scenario where eight agents each process 7.5k-token documents and a chat request is issued in the second second, Superfluid answers in 0.36 seconds while llama-server outputs no first token for more than 10 seconds, a first-token latency gap of about 30 times.
Superfluid uses a write-ahead log (WAL) mechanism for each token: the token is first written to the log and then sent to the client, enabling the session to continue from the checkpoint after the inference process is killed or the server is restarted.
Official measurements: after the inference worker is killed midway through streaming output, Superfluid restores service within 1.8 seconds and keeps the stream intact; Ollama also restores within 1.8 seconds but the stream is truncated; llama-server and mlx_lm.server have no checkpoint recovery capability.
Superfluid simultaneously supports three API protocols—OpenAI, Anthropic, and Ollama—and underneath connects to three runtimes: llama.cpp (GGUF models), MLX (MLX directories), and the BaseRT engine (.base model packages).
Superfluid can directly take over requests from clients such as Claude Code, Codex, Cline, OpenCode, and the Ollama CLI.
When 32 requests share the same 4k-token system prompt, Superfluid's prefix cache hit rate reaches 97%, on par with Ollama, while mlx_lm.server is 0%.
Superfluid has built-in multi-device inference capability: one machine acts as the head node, other machines join through token authentication, and they automatically discover the head node, pull models, and serve through encrypted links.
Superfluid's multi-device scheduling uses a whole-session assignment strategy: a session is placed entirely on one node, with only sampled tokens sent back to the head node; if a node goes down midway, the session is replaced and tokens are not delivered twice.
4 more timeline events
All Facts (14)
Superfluid is a local LLM inference server under the Apache-2.0 open-source license, with its core positioning being single-machine multi-agent inference.
50%UnverifiedSuperfluid uses preemptive scheduling, and interactive requests preempt batch work in the next scheduling cycle rather than queuing behind batch tasks.
50%UnverifiedIn a test scenario where eight agents each process 7.5k-token documents and a chat request is issued in the second second, Superfluid answers in 0.36 seconds while llama-server outputs no first token for more than 10 seconds, a first-token latency gap of about 30 times.
50%UnverifiedSuperfluid uses a write-ahead log (WAL) mechanism for each token: the token is first written to the log and then sent to the client, enabling the session to continue from the checkpoint after the inference process is killed or the server is restarted.
50%UnverifiedOfficial measurements: after the inference worker is killed midway through streaming output, Superfluid restores service within 1.8 seconds and keeps the stream intact; Ollama also restores within 1.8 seconds but the stream is truncated; llama-server and mlx_lm.server have no checkpoint recovery capability.
50%UnverifiedSuperfluid simultaneously supports three API protocols—OpenAI, Anthropic, and Ollama—and underneath connects to three runtimes: llama.cpp (GGUF models), MLX (MLX directories), and the BaseRT engine (.base model packages).
50%UnverifiedSuperfluid can directly take over requests from clients such as Claude Code, Codex, Cline, OpenCode, and the Ollama CLI.
50%UnverifiedWhen 32 requests share the same 4k-token system prompt, Superfluid's prefix cache hit rate reaches 97%, on par with Ollama, while mlx_lm.server is 0%.
50%UnverifiedSuperfluid has built-in multi-device inference capability: one machine acts as the head node, other machines join through token authentication, and they automatically discover the head node, pull models, and serve through encrypted links.
50%UnverifiedSuperfluid's multi-device scheduling uses a whole-session assignment strategy: a session is placed entirely on one node, with only sampled tokens sent back to the head node; if a node goes down midway, the session is replaced and tokens are not delivered twice.
50%UnverifiedSuperfluid multi-device supports different nodes running different runtimes (e.g., BaseRT on Mac, llama.cpp on Linux), transparent to upper-layer clients.
50%UnverifiedSuperfluid's serving layer introduces no additional overhead: under single-request and eight-concurrent-request conditions, the tokens/s of llama.cpp called through Superfluid is on par with using llama-server directly.
50%UnverifiedSuperfluid batches MLX requests, which is more efficient than mlx_lm.server's one-by-one processing.
50%UnverifiedSuperfluid's gains mainly come from scheduling, caching, and fault tolerance rather than compute acceleration; in single-agent, low-concurrency scenarios it differs little from existing solutions.
50%