Why Continuous Batching Slowed Down My LLM Inference Engine: A Metal Kernel Investigation

When adding optimizations to his Rust-based macOS LLM inference engine mini-vllm-rs, the author assumed continuous batching would be the next easy performance win. Pack multiple requests together, reuse model weights across more tokens, watch throughput climb — it seemed like a sure thing. Instead, the opposite happened.

On an M2 Mac mini with 16GB unified memory, the author tested two concurrent requests using a 4-bit quantized Qwen2.5 7B model, each generating 512 tokens with speculative decoding disabled. Instead of the expected throughput boost, performance nearly halved: from 16.8 tokens/s without batching down to 8.7 tokens/s with batching.

This investigation reveals an easy-to-overlook truth: a "correct" decision at the scheduler level can be implemented in a slower way at the backend execution layer — and more insidiously, that difference can quietly change the text being generated.

Two Rows of Input, Two Different Kernels

The clue was hiding in Candle, the backend framework being used. Its quantized Metal execution path contains a seemingly minor check: when the input matrix has exactly one row, it takes the GEMV (matrix-vector multiplication) path; once there's more than one row, it switches to GEMM (matrix-matrix multiplication), processing multiple input vectors together.

// Candle 0.11.0: candle-core/src/quantized/metal.rs, line 372
if src_shape.dim(D::Minus2)? == 1 {
    return self.fwd_mv(self_shape, storage, layout);
}
// Line 414 calls into call_quantized_matmul_mm_t

This is where the problem lies: batching two requests means the input goes from one row to two, and that tiny change is enough to switch to an entirely different compute kernel. GEMM can theoretically reuse weights across rows, but at such a small batch size, the overhead of this implementation far outweighs the benefit of that reuse.

In other words, the scheduler did exactly the right thing — it merged the requests — but the backend chose a slower execution path.

GEMV (General Matrix-Vector multiplication) and GEMM (General Matrix-Matrix multiplication) are two fundamental operations in linear algebra libraries, corresponding to very different compute patterns in deep learning inference. GEMV handles the multiplication of a matrix with a single column vector, well-suited for batch size 1: it simply dot-products each row of the weight matrix with the input vector, with a linear and predictable memory access pattern. GEMM arranges multiple input vectors into a matrix and processes them all at once, theoretically achieving much higher hardware utilization through matrix tiling and cache reuse.

However, this benefit depends on a sufficiently large batch size. With very few matrix rows, the tiling, kernel launch, and intermediate buffer overhead of GEMM can actually exceed the savings from reuse. On GPU backends like Apple Metal, the dequantization step for quantized weights amplifies this gap further — GEMV can dequantize weights on the fly during reads, while GEMM's tiling logic makes this process considerably more complex. The intuition that "bigger batches are always faster" often breaks down at very small batch sizes (typically fewer than 4–8 rows), requiring empirical measurement rather than theory to determine the kernel switching threshold.

Row-by-Row GEMV to the Rescue

Once the root cause was identified, the author added a fallback: split small-batch inputs into individual rows, run GEMV on each row separately, then concatenate the results.

The effect was immediate. At batch size 2, this fallback brought throughput back to 16.25 tokens/s, essentially matching the unbatched baseline of 16.15 tokens/s. But the real question was: as batch size grows, does row-by-row GEMV continue to win?

Benchmarking gave the answer:

Concurrent Requests	Unbatched	Batched GEMM	Batched Row-by-Row GEMV	GEMV/GEMM Throughput Ratio
3	17.41	13.05	17.52	134.3%
4	17.22	16.55	18.07	109.2%
5	17.44	19.92	18.06	90.7%

Data shown as aggregate output throughput (tokens/s)

At 5 concurrent requests, the ratio drops below 100% and GEMM regains the lead — by that point, the cost of dispatching individual rows and concatenating results exceeds the cost of a single GEMM call. Based on this workload, the author set 4 rows as the default kernel switching threshold.

Problem solved? Not quite.

Same Prompt, Different Story

The real surprise came later. When testing speculative decoding with Qwen2.5 0.5B as the draft model proposing 4 tokens at a time, the author noticed: the overall draft acceptance rate was 32.9% for serial processing but 37.1% for concurrent processing. Same requests, same models, same proposal strategy, both using greedy decoding — why would the results differ?

To make the numerical issue more concrete, the author had Qwen write a sci-fi scene: a mechanic and an archivist investigating a distress signal in an abandoned lunar greenhouse. Both runs used greedy decoding; the only difference was whether prefill went through GEMM or row-by-row GEMV.

The first 597 characters were identical, after which the content diverged:

Prefill Kernel	Output After Divergence
GEMM	"Mara moved toward the second door, where an emergency terminal stood…"
GEMV	"Mara's flashlight beam cut through the dimness, illuminating rows of withered hydroponic trays…"

Both continuations are coherent, consistent with the characters and setting, with no signs of model breakdown. That's precisely what makes this suspicious — the culprit is an extremely subtle numerical difference.

Speculative decoding traces provided stronger evidence. At the first token where divergence occurred, the draft model proposed the same token ID in both runs, but the target model's top two scores swapped order:

Execution Mode	Token 476	Token 323	Selected
Serial	25.571232	25.565775	476
Continuous Batching	25.549618	25.562220	323

This vanishingly small difference was enough to flip the greedy decoding choice. Every subsequent token unfolded along a different prefix, naturally changing the draft acceptance rate along with it.

Greedy decoding is the simplest text generation strategy: at each step, select the token with the highest score in the model's output distribution. Because there's no random sampling, it's generally considered deterministic — the same input should produce the same output. But "deterministic" requires perfectly identical numerical results, and floating-point arithmetic doesn't guarantee that.

Floating-point addition and multiplication aren't associative: (a + b) + c and a + (b + c) may produce different results under IEEE 754. Matrix multiplication involves extensive floating-point accumulation, and GEMM and GEMV have different internal summation orders; the timing of dequantization for quantized weights also differs. This ultimately causes the same weight matrix applied to the same input vector to yield activation values that differ by an infinitesimal amount — but potentially enough to flip the sign. When two candidate tokens have very similar logit scores, this tiny difference is sufficient to change the greedy selection, triggering a butterfly effect through autoregressive generation that sends all subsequent tokens down a different branch.

The Other Request Already Finished — So Why Is It Still Having an Effect?

The most surprising part was this: when the token selection was flipped, the other request had already finished. So why was it still influencing results?

The key is that the "rows" here count token positions, not requests. A certain early batch combined 120 prefill tokens from a new request with 4 verification tokens from an existing request. Those 124 rows crossed the threshold, so GEMM was chosen over GEMV, introducing a tiny numerical difference into the cached state.

By the time the output actually diverged, the other request had completed, and the remaining request's four-token verification had returned to the GEMV path. But switching back to GEMV doesn't undo the differences already written to the KV cache.

In other words, if you only look at the iteration where the text diverges, you'll never find the real cause. To verify this explanation, the author forced all relevant inputs to go through GEMV in both execution modes; in the shortened comparison, output hashes and acceptance totals matched. This doesn't prove which kernel is more accurate, but it supports the causal chain described above.

The KV cache (Key-Value Cache) is a core optimization in Transformer inference. During self-attention, each token generation requires computing attention against all previous tokens in the sequence, and the Key and Value vectors for those historical tokens remain unchanged in subsequent steps. The KV cache stores these intermediate results to avoid redundant computation, reducing the generation phase's computational complexity from O(n²) to O(n).

However, the KV cache also makes numerical errors persistent. In continuous batching, multiple requests share a single forward pass, and the Key/Value tensors written to the cache by a given batch carry the floating-point errors introduced by that batch's specific execution path (e.g., GEMM instead of GEMV). Once written to the cache, these values are never modified; subsequent steps read them into attention computations, incorporating the error into all token generation that depends on that history. Even after the batch that introduced the error has ended and that request has completed, the "contamination" persists through the cached state — which is why debugging these issues requires tracing back to when the cache was written, not just when the output diverged.

Takeaways from This Investigation

This experience distills to one sentence: a batching decision changed kernel selection, and its effects lived on in the cached state long after the batch itself had changed again.

For developers building inference engines, a few lessons are worth keeping in mind:

• "Merging requests" at the scheduling layer and "selecting a kernel" at the execution layer are two separate things. Getting the former right doesn't guarantee the latter is efficient — kernel switching thresholds must be measured against real workloads, not assumed.
• GEMM doesn't necessarily beat GEMV at small batch sizes. Kernel switching costs should be determined by benchmark data, not intuition.
• Greedy decoding does not mean results are deterministic and reproducible. Floating-point differences from different execution paths are enough to flip token selection and create long-range "memory effects" through the KV cache.
• When debugging numerical issues, don't just look at the moment of divergence — trace back to how the cached state was contaminated.

The author's longer post also documents how a generation loop gradually evolved into a service engine with clear boundaries between scheduling, model execution, and request state — as well as where AI helped with architecture and optimization, and where it needed to be questioned. For readers who want to go deeper into inference engine engineering, this investigation provides a rare first-hand perspective.