llama-swap
A model memory scheduling tool designed for use with llama.cpp, supporting on-demand dynamic loading and automatic unloading of models to enable efficient memory management across multiple models.
Timeline (last 90 days)
This scheme locks each backend by host and port, using cross-process flock, with the lock granularity being a single turn: acquired at the start of the turn and released at the end.
The final configuration is model.single_session_lock: true, and the gate locks per backend rather than per model, because the GPU loads only one model at a time; when a second GPU channel appears, the key will become model-scoped.
Frequent agent cron jobs must be paused, because llama-swap burns a 420-second health check timeout per request for dead members, and the cron agent will error out mid-run.
systemctl stop strata will also stop llama-swap, because llama-swap carries Requires=strata.service; and llama-swap cannot load GGUF members while Strata is running, because Strata always occupies VRAM; the HyperQwen window can only bypass the proxy and connect directly to :18020, at which point :8080 is offline.
In actual testing, llama.cpp combined with llama-swap can achieve 75–80 tokens per second, and the author considers it faster and easier to use than Ollama.
llama-swap is a lightweight HTTP proxy built on top of llama.cpp, responsible for the dynamic loading and unloading scheduling of multiple models.
All Facts (6)
This scheme locks each backend by host and port, using cross-process flock, with the lock granularity being a single turn: acquired at the start of the turn and released at the end.
50%UnverifiedThe final configuration is model.single_session_lock: true, and the gate locks per backend rather than per model, because the GPU loads only one model at a time; when a second GPU channel appears, the key will become model-scoped.
50%UnverifiedFrequent agent cron jobs must be paused, because llama-swap burns a 420-second health check timeout per request for dead members, and the cron agent will error out mid-run.
50%Unverifiedsystemctl stop strata will also stop llama-swap, because llama-swap carries Requires=strata.service; and llama-swap cannot load GGUF members while Strata is running, because Strata always occupies VRAM; the HyperQwen window can only bypass the proxy and connect directly to :18020, at which point :8080 is offline.
50%UnverifiedIn actual testing, llama.cpp combined with llama-swap can achieve 75–80 tokens per second, and the author considers it faster and easier to use than Ollama.
50%Unverifiedllama-swap is a lightweight HTTP proxy built on top of llama.cpp, responsible for the dynamic loading and unloading scheduling of multiple models.
50%