Running Multi-Agent on a Single 3090: Cross-Process Locks Keep Model Exclusive Without Collisions

One 3090, one Qwen backend, multiple Agent sessions online at once — this is the reality for many local AI enthusiasts. But when chat sessions, cron jobs, and sub-agents all hit the same inference engine simultaneously, the KV cache gets trampled by concurrent requests and generation quality collapses. A developer on rss shared his solution: use a cross-process file lock to create a "single-lane" mechanism for Agent turns, and use real showdown tests to catch lock-path bugs that unit tests structurally cannot find.

The Essence of the Problem: One Model, Multiple Processes

llama-swap sits behind port :8080 with a Qwen model running on a single 3090. The chat channel, the scheduled task runner, and the sub-agent tool each spawn independent processes and fire requests at this engine at the same time. When two turns land simultaneously, the KV cache in use gets corrupted.

The core contradiction is that a session is not a single process. In-process locks cannot protect against cross-process contention, and the problem comes from exactly three different processes. So the developer designed the lock as a cross-process flock, keyed per backend by host and port — this is the single most critical design decision in the whole scheme.

KV cache (Key-Value Cache) is the core in-memory structure during LLM inference. When a Transformer generates each token, it needs to compute attention over all historical tokens. To avoid redundant computation, the inference engine caches the Key and Value tensors from each layer's attention mechanism. For long-context conversations, this cache can consume several GB of VRAM. When two requests hit the same inference engine at once, the engine tries to reuse or extend the same KV cache, but the histories of the two sessions are completely different — the cache indices and sequence positions conflict, causing the model to read garbled context, output quality to plummet, and even produce gibberish or repetition. This is why a single-GPU backend is best off serving only one generating request at a time.

flock is the file-locking mechanism provided by Linux (the flock(2) syscall). Unlike traditional POSIX record locks, it binds the lock to the open file descriptor. This means when a process exits normally or crashes, the kernel automatically releases all locks held by that descriptor, with no cleanup logic required. Another property of flock is that its locks are "advisory" — only processes that also flock to contend for the same path will mutually exclude each other; other processes are not blocked even if they read or write that file. This explains why the lock path must be consistent across all processes later on: only when all processes contend for the same real path does the gate actually take effect; when paths are inconsistent, each process locks its own path, appearing to be mutually exclusive while actually everyone locks their own thing.

Hold the Lock Per Turn, Not Per Session

The scope of lock holding determines the success or failure of the scheme. If you hold the lock per session, a long-running chat session would permanently occupy the 3090 and all cron jobs would starve. So the granularity is a single turn: acquire at the start of a turn, release at the end. A quiet session is just a quiet session — only the turn currently generating holds the lane.

The second design decision is cleanup. flock lives in the file descriptor, and the kernel releases the lock automatically when the process dies — this means no expired-lock reaper, no heartbeat files, no logic to decide whether a lock from 20 minutes ago is dead. The release path can crash arbitrarily and the gate still reopens. A small sidecar file next to the lock records pid, session, model, and timestamp so the gate can be inspected when it looks stuck.

Three Hook Points, Fail-Open Everywhere

The gate hooks into three places in the Agent loop:

Turn construction — after the runtime restores the primary model, it immediately attempts to acquire the lock; if the gate is busy, the turn is sent down the actual fallback chain (starting with the cloud model).

Fallback traversal — skip chain entries pointing to a busy local backend, preventing a session whose primary config is cloud from falling back to a saturated engine.

Turn release — the release logic sits in the outermost finally of the turn runner, so whether it completes, errors, or is cancelled, it releases in the same place.

The new failure reason local_backend_busy renders as "local model in use by another session," making deflection look like deflection rather than a provider failure.

Fail-open is the bottom line: this is a performance guard, and Agent availability never depends on it. A gate error doesn't block turns; with no configuration, behavior is exactly the same as before; any non-local request skips the gate entirely. The worst case is two turns sharing the GPU — which is exactly the world without the gate.

Fail-open is a system design principle: when a guard component itself errors or becomes unavailable, the system chooses to let requests through rather than reject them, prioritizing the availability of the main flow. Its counterpart is fail-closed, which blocks on error — safer but prone to cascading failures. Here the gate's role is "performance guard," not "security boundary" — when enabled it only affects VRAM contention, and when it errors or is unconfigured, the entire Agent's behavior should be exactly the same as before. Betting availability on the guard means a single lock-file anomaly, path resolution error, or permission problem would freeze all Agent turns outright, a cost far greater than the quality loss from briefly sharing the GPU. A typical fail-open implementation wraps the guard logic in try/catch, returns "allow" on exception, and also chooses to let through when the guard returns an uncertain result.

All Tests Green, Yet the Real Showdown Flops

The test suite passed on its first run: 17 gate tests covering mutual exclusion, busy skipping, fail-open, and real cross-process flock scenarios, plus 74 fallback and turn-lifecycle regression tests.

But the real validation came from a "showdown" of two live CLI sessions plus one cron turn, which on the first try caught a bug the test suite structurally could not find. flock only excludes processes that compute the same lock path, and unit tests run in a single process using a hand-picked home directory — any internally consistent path passes the tests.

In actual operation, the profile-home stripping logic resolved the wrong root, so the lock and sidecar landed under a per-profile directory tree, while the canonical shared directory stayed empty. Only the showdown printing the lock file's real location exposed this. Even sneakier: the two CLI sessions still "worked" in deflecting, because they belonged to the same profile and agreed on the same wrong path. But sessions come from every profile on the machine, and the shared root exists precisely so all sessions land on the same lock — a gate only some processes can see is two gates.

After the commit fixing the root, the second showdown had the lock and sidecar at the canonical path, B deflected to the cloud model, and A released cleanly — pass. The third showdown added the probe I really wanted: after A announces completion the lock file is idle, and a fresh acquisition proves the local model is immediately reacquirable.

profile-home refers to the independent home directory that an Agent framework allocates to each configuration profile. Such frameworks typically support multiple profiles, each with its own configuration, session history, and cache directory, isolated from one another. But the inference backend is a shared resource, so the framework additionally maintains a "canonical shared directory," whose very purpose is to make sessions from all profiles land on the same lock. The problem is that if path resolution mistakenly picks the profile-specific home instead of the canonical root, different profiles compute different lock paths. Two sessions belonging to the same profile are still mutually exclusive because their paths agree (appearing to "work normally"), while cross-profile sessions each lock their own path and are unaware of each other — the gate is quietly cloned into multiple gates, with the actual gate count equal to the number of contending profiles, and the guard is effectively useless. This kind of bug is naturally invisible in single-process tests and only surfaces when real multi-profile processes run simultaneously.

Automatic Return Trip: Recovery Without Intervention

A fallback that never returns is an expensive way to run a single model, so the return trip has its own check. The primary runtime restores at the top of every turn, the gate only deflects colliding turns, and the next scheduled turn tries the local model first again.

The real validation was an overnight cron turn: 00:05 used the local model, 00:11 deflected to the cloud while the showdown held the gate, 00:17 returned to the local model, all without any intervention.

The final config is model.single_session_lock: true, with the gateway restarted twice after the fix and all channels reconnecting fine. The gate locks per backend rather than per model, because the GPU loads only one model at a time; when a second GPU channel appears, the key will become model-scoped.

Advice for Similar Setups

If you run anything behind a single model server that spawns concurrent sessions, this shape is worth referencing. Spend your effort on two real processes doing real turns — a single-process test suite passes no matter where the lock lands. Always fail-open, because a gate that can deadlock your Agent is worse than the contention it was meant to prevent.

One 3090, one model at a time.