mini-vllm-rs

To preserve validation, the original design, after adding the Active and Draining states, tracks released followers until their termination.

In the self-developed Rust LLM inference engine mini-vllm-rs, after prefix caching is enabled, when two completely identical requests arrive simultaneously at a cold cache, possibly not a single token can be reused, and both perform full prefill.

In mini-vllm-rs, after introducing incremental publishing, each follower request recovers 480 of 493 prompt tokens.

In the initial single-flight implementation of mini-vllm-rs, the AI directly deleted the flight record when releasing followers, causing the coordinator to fail to recognise a follower's ID when it later completed and notified the coordinator; the author refused to remove the unknown-request check because doing so would mask termination notifications carrying genuinely invalid request IDs.

Testing with a 4-bit quantized Qwen2.5 7B model on an M2 Mac mini with 16 GB unified memory, throughput dropped from 16.8 tokens/s without batching to 8.7 tokens/s with batching.

Based on this workload, the author set 4 rows as the default kernel-switching threshold.

Continuous batching counts rows by token position rather than by request count; in one batch, the 120 prefill tokens of a new request and the 4 verification tokens of an existing request were merged into 124 rows, thereby crossing the threshold and selecting GEMM.

At the first token where divergence occurred, the two leading scores of the target model for tokens 476 and 323 swapped order between serial and batched processing, reversing the greedy decoding selection.