8x RTX 2080Ti 22G Local Deployment of GLM-5.3-Flash Benchmarked: TP2 PP4 Hybrid Parallel Strategy Explained

Running large models on consumer-grade or prosumer GPUs is rarely about whether the VRAM fits — the real pain point is multi-GPU coordination efficiency. A Bilibili content creator deployed GLM-5.3-Flash locally on eight modded 2080Ti 22G GPUs, achieved solid benchmark numbers through a custom parallel strategy, and along the way offered unusually candid insight into the performance pitfalls that commonly trip up multi-GPU inference.

Hardware Setup and Benchmark Results

The system consists of eight RTX 2080Ti cards with 22GB of VRAM each — a popular modded variant in China's second-hand GPU market that doubles the original 11GB VRAM at relatively low cost, creating a substantial total VRAM pool. After a non-trivial amount of tuning, this platform achieves approximately 30 tokens per second for short-text inference with GLM-5.3-Flash.

For a mid-size model running locally, 30 tok/s is comfortably within the range of fluid interactive use. The creator specifically noted that this result was only achieved after "considerable tweaking," implying that out-of-the-box default configurations fall well short of the system's potential.

Parallel Strategy: Why It's Not "One Card Working, Seven Cards Watching"

The most common failure mode in multi-GPU deployments is severe load imbalance — one card pegged at 100% utilization while the rest sit nearly idle. The creator observed that each card in this setup maintains meaningful utilization with a relatively even power distribution, which is directly attributable to the choice of parallel strategy.

The specific configuration is TP2 PP4: GPUs are grouped in pairs for Tensor Parallelism (TP), forming four groups total, with Pipeline Parallelism (PP) applied across groups for layer-wise partitioning. This hybrid design is deliberately targeted — in a two-card TP topology, the communication overhead of tensor parallelism stays manageable, since high-frequency all-reduce communication only occurs between the two cards within a group. Meanwhile, pipeline parallelism across groups avoids the communication explosion that pure tensor parallelism suffers at higher GPU counts.

In the creator's own words, this approach is "faster overall than pure pipeline parallelism." It's a pragmatic engineering trade-off: pure TP becomes communication-bottlenecked as card count grows, while pure PP tends to suffer from pipeline bubbles that hurt utilization. TP2 PP4 attempts to find the sweet spot between both extremes.

Tensor Parallelism (TP) splits a single layer's weight matrices along a chosen dimension and distributes the computation across multiple GPUs. Each forward pass requires an All-Reduce communication step to aggregate results. This allows each GPU to store only a fraction of the weights, but the cost is per-layer cross-GPU synchronization at very high frequency. As the number of TP GPUs increases, the data volume and latency of All-Reduce grow linearly or worse — making TP most practical in environments with very high inter-GPU bandwidth (such as NVLink), or when TP is constrained to just two cards as in this case.

Pipeline Parallelism (PP) partitions the model vertically by layer — the first few layers run on the first GPU group, the remaining layers on subsequent groups, with data flowing through like an assembly line. PP requires far less communication than TP (only activations need to pass between adjacent groups), but introduces "pipeline bubbles": downstream groups sit idle waiting for upstream outputs, reducing overall GPU utilization. The TP2 PP4 hybrid is precisely designed to balance communication overhead against pipeline bubble inefficiency.

Prefill vs. Decode: Where Long-Context Performance Actually Breaks

The system's behavior on long-context inputs reveals the deeper characteristics of multi-GPU inference. The creator used a 33K-token Qwen3.8-Flash technical report as the test input (joking that when you need a long document, what better option is there than a research PDF).

The results break down across two phases:

• Prefilling phase: drops to under 200 tok/s on long context (~180 tok/s)
• Decoding phase: long-context output runs at approximately 18 tok/s

There's a key technical observation here: prefilling is far more communication-intensive than decoding. When ingesting 33K tokens of context at once, prefilling must compute attention across the entire input in parallel, sharply increasing cross-GPU communication pressure. Decoding, by contrast, generates one token at a time — communication demands are smaller, and 18 tok/s, while not fast, stays within an acceptable threshold for human-facing output.

Prefilling and decoding have fundamentally different computational characteristics. During prefilling, the model must compute self-attention across all input tokens simultaneously — computation scales quadratically with sequence length, making it a classic compute-bound operation. In a multi-GPU setup, intermediate activations must be exchanged frequently, and communication pressure scales linearly with sequence length. Decoding, by contrast, generates one new token per step while reusing KV Cache entries — it's memory-bandwidth-bound, with relatively fixed and modest cross-GPU communication volume. This is why the same hardware and parallel strategy yields a nearly 10× gap between long-context prefill speed (~180 tok/s) and decode speed (~18 tok/s): the two operations hit entirely different bottlenecks.

An Important Caveat on Speculative Decoding

Many practitioners' first instinct would be to apply speculative decoding or MTP (Multi-Token Prediction) to accelerate the decode phase — but the creator explicitly emphasized that no speculative decoding methods were used in this testing.

The reasoning is targeted: in a deeply layer-partitioned topology like PP4, speculative decoding does not provide meaningful speedup. Speculative decoding's benefits depend on a small draft model rapidly generating candidate tokens and the main model verifying them in batch. But when the model is split across multiple GPU groups and cross-group verification carries high communication cost, that overhead consumes whatever gains the batch verification offers. This explains why naively applying speculative decoding tricks in deep pipeline scenarios may yield zero benefit.

Speculative Decoding works by using a much smaller "draft model" to rapidly generate several candidate tokens in sequence, then having the main model verify the entire batch in a single parallel pass — transforming serial token-by-token generation into batch verification, theoretically delivering significant throughput gains without quality loss. MTP (Multi-Token Prediction) is a related variant that trains a model to predict multiple future tokens simultaneously. Both approaches work well in single-GPU or shallow-TP settings, but their effectiveness depends on the verification step incurring low additional communication cost. In a PP4 deep-partition topology, each verification pass requires activations to traverse all four GPU group stages, and the cross-group communication latency effectively cancels out the parallel verification gains — reducing the speedup to essentially zero.

Takeaways from This Deployment

The most valuable aspect of this benchmark isn't the raw token speed numbers — it's the clarity with which it illustrates several core engineering decisions in multi-GPU local deployment:

First, parallel topology must match hardware communication structure. TP2 PP4 works here precisely because it confines high-frequency communication to within two-card groups, rather than forcing all GPUs into global synchronization.

Second, prefilling and decoding are two fundamentally different performance regimes. When evaluating an inference system, short-text speed, long-context prefill speed, and long-context decode speed each tell a different story — no single number captures the complete picture.

Third, optimization techniques are context-dependent. Speculative decoding delivers meaningful gains on single GPUs or in shallow parallel configurations, but can fail completely under deep pipeline partitioning — blindly applying "universal optimizations" doesn't guarantee improvement.

For enthusiasts looking to build a local LLM inference platform from affordable second-hand GPUs, this 8× RTX 2080Ti 22G + TP2 PP4 setup offers a quantifiable real-world reference point: 30 tok/s on short text, 18 tok/s decode on 33K long context, and 180 tok/s prefill — the true ceiling of this hardware after careful tuning.