V100 GPU Showdown: 4×16G PCIe vs. 2×32G SXM — Which Gives You More Bang for Your Buck?

Hardware Background: Two V100 Deployment Approaches

NVIDIA's V100, launched in 2017 on the Volta architecture, was the flagship AI compute product for data centers at the time. Its standout innovation was the introduction of Tensor Cores — dedicated hardware units designed for mixed-precision matrix multiplication — delivering peak throughput of 125 TFLOPS at FP16, more than five times that of the previous Pascal generation. Before the A100 arrived, the V100 was the de facto standard for deep learning training and inference, widely deployed across AWS, Azure, Google Cloud, and other major cloud platforms. Today, its second-hand price has fallen from a peak of roughly $10,000 USD per card to just a few hundred to a couple thousand RMB, making it a viable hardware option for individuals and small teams running local LLM inference.

As both SXM2 and PCIe V100s have flooded the used market, two distinct deployment strategies have emerged. One approach is to buy native PCIe 16G V100s and stack them (e.g., four cards) to expand VRAM and total compute. The other is to use the higher-memory 32G SXM variant with an adapter board to fit a standard PCIe slot, betting on larger single-card VRAM and higher memory bandwidth.

It's worth understanding the fundamental engineering differences between these two form factors. SXM2 (Socket for HPC with Mezzanine 2) is NVIDIA's high-density interconnect standard designed specifically for data center servers, allowing GPUs to connect directly through a dedicated slot and communicate via NVLink 2.0 with a theoretical bidirectional bandwidth of 300 GB/s — nearly 10× the one-way bandwidth of PCIe 3.0 x16 (~16 GB/s). Used SXM cards on the market typically come from retired DGX/HGX servers and require third-party adapter boards to fit standard PCIe slots. After the conversion, NVLink functionality is generally unavailable, leaving only the larger VRAM capacity and a slightly higher TDP as the key advantages.

One important detail: the native PCIe 16G V100 has a TDP capped at around 250W, while the SXM variant runs at 300W. That 50W difference translates into a perceptible compute gap during sustained, high-load inference. The tests in this article use OneCat VLLM 1.2.1 as the inference framework, running the Qwen3.6 R7B FP8 model.

About OneCat VLLM: vLLM is a high-performance LLM inference engine developed at UC Berkeley. Its core innovation, PagedAttention, draws from OS virtual memory management concepts — dividing the KV Cache into fixed-size "pages" for dynamic allocation, fundamentally eliminating memory fragmentation and over-allocation waste, and pushing VRAM utilization close to 100%. OneCat VLLM 1.2.1 is a community-customized build optimized for specific hardware environments, with compatibility patches for older-generation GPUs on top of the official vLLM codebase.

About FP8 Quantization: FP8 compresses each model parameter from 16-bit to 8-bit storage, roughly halving VRAM consumption with minimal quality degradation. It's important to note that the Volta architecture underlying the V100 does not have native FP8 hardware acceleration (that feature is natively supported on Hopper-architecture H100s), so FP8's inference speedup on V100 is limited — the primary benefit is memory savings.

About Qwen3 R7B: Qwen3 is Alibaba Cloud's third-generation open-source Qianwen series, spanning model sizes from 0.6B to 235B parameters. R7B is a reasoning-optimized variant with approximately 7B active parameters, featuring enhanced chain-of-thought capabilities suited for logical reasoning and complex Q&A tasks.

Prefill Speed: Tensor Parallelism Falls Short of Linear Scaling

Intuitively, four 16G cards should outperform two 32G cards in both total compute and total VRAM, yielding nearly double the performance. The actual results tell a more nuanced story.

Although vLLM supports Tensor Parallelism — splitting the model across multiple GPUs for collaborative computation — the prefill (preprocessing) benchmark shows that 4×V100 16G does not deliver twice the prefill throughput of 2×V100 32G.

Tensor parallelism works by horizontally splitting each neural network layer's weight matrix across multiple GPUs. In a Transformer attention layer, for example, multi-head attention weights are evenly distributed across cards; each card independently computes its local matrix multiplications, then synchronizes results via an All-Reduce operation. This All-Reduce step is the core communication bottleneck of tensor parallelism: every forward pass requires synchronizing intermediate activations across all participating GPUs, and the communication volume scales linearly with the degree of parallelism. In an NVLink environment, 300 GB/s bidirectional bandwidth keeps this overhead manageable. When cards are connected via PCIe adapters, however, inter-GPU communication is forced through the PCIe bus (~16 GB/s one-way) or even the CPU, dramatically worsening both latency and bandwidth — and this penalty is compounded with every additional card added to the parallel group.

The analysis points to two contributing factors:

• Power ceiling: The native PCIe 16G V100 is capped at 250W per card, limiting each card's compute headroom;
• Bandwidth bottleneck: The prefill stage is highly sensitive to memory bandwidth and inter-GPU communication bandwidth. The additional communication overhead introduced by multi-card tensor parallelism cancels out a significant portion of the raw compute gains from stacking more cards.

Performance Convergence at Long Context Lengths

An intriguing observation emerged during testing: once the context length exceeded roughly 30,000 tokens, the prefill speed curves for both configurations began to converge. In other words, the prefill performance gap between 4×16G and 2×32G narrows significantly at long context lengths.

Whether this is coincidence or a sign that bandwidth bottlenecks dominate both configurations at long contexts remains an open question. From a technical standpoint, this behavior is closely tied to the memory access patterns of the KV Cache. The KV Cache (Key-Value Cache) is a core optimization in Transformer inference — the model caches the attention key-value pairs for each processed token to avoid redundant computation, but this cache grows linearly with context length. For a typical 7B-scale model, the KV Cache at FP16 precision can occupy tens to hundreds of MB per thousand tokens. As context approaches 30,000 tokens, KV Cache read/write pressure surges, and memory bandwidth becomes the dominant constraint. Since both configurations converge toward similar effective bandwidth in this regime, their performance curves naturally align.

Decode Speed: The Cliff Drop at Long Contexts

If the prefill comparison was roughly even, the decode (output generation) speed comparison reveals a more pronounced problem.

The test data clearly shows the 2×32G SXM adapter configuration splitting the 4×16G native PCIe configuration into two distinct performance zones:

• Under 20,000 tokens: Output speed reaches approximately 45 tokens/second;
• Over 30,000 tokens: Output speed drops sharply to approximately 20 tokens/second.

This cliff-edge behavior has a clear physical explanation. The decode phase is a textbook memory-bandwidth-bound operation — generating each new token requires reading the full KV Cache of all previous tokens from VRAM to compute attention weights. The computation itself is minimal, but the data movement is enormous. GPU compute units sit largely idle waiting for data transfers, and overall throughput is entirely limited by memory bandwidth rather than compute. Once context exceeds roughly 30,000 tokens, each card in the 4×16G setup holds only a fraction of the full KV Cache. Generating every new token requires fetching KV Cache fragments from other GPUs over the PCIe bus, and PCIe communication latency — several to tens of times higher than NVLink — is amplified repeatedly at each decode step. The result is a hard speed drop from 45 tokens/second to 20 tokens/second. In the 2×32G setup, each card holds more VRAM, the KV Cache is more concentrated, and cross-card communication is less frequent — yielding more stable performance at long context lengths.

This cliff-like degradation means users processing long documents or extended conversations will experience a noticeable slowdown in generation speed. The charts here put a concrete visual on what was previously described only qualitatively, showing exactly where the critical threshold lies.

Caveats: Single-Run Limitations

To the credit of the analysis, it's explicitly noted that this decode speed drop was observed in a single test run, making it impossible to determine whether it's a consistent pattern or an outlier. A recommended follow-up would be to run five repeated measurements and average the results, using statistical rigor to confirm whether the degradation is significant and reproducible. This kind of intellectual honesty is rare in hardware benchmarking content — a single benchmark run is easily distorted by transient factors like system state, cache behavior, and thermals, and GPU dynamic frequency scaling (Boost Clock varying with temperature) alone can introduce several percentage points of variance under otherwise identical conditions.

So Which Configuration Actually Wins on Value?

Pulling together both datasets, some tentative conclusions emerge — though no absolute verdict is possible:

• Short-to-medium context (under 20,000 tokens): Output speeds are comparable; the 4×16G setup has a slight edge from raw compute stacking, but TDP limits and communication overhead prevent anywhere near a 2× improvement;
• Long context (over 30,000 tokens): Prefill speeds converge, but output speeds show a clear degradation for 4×16G. The 32G large-VRAM solution handles long KV Caches within a single card, avoiding cross-card reads and maintaining more stable performance;
• Cost considerations: Native PCIe 16G V100 prices have fallen substantially, but the total cost of four cards isn't necessarily lower than two 32G SXM cards. You also need to factor in motherboard PCIe slot availability, PSU capacity (4×250W vs. 2×300W — the former draws higher peak total power), and thermal management.

There's no single "best value" answer — it depends entirely on your use case. If your workload is mostly short-text tasks and budget is the primary constraint, 4×16G is a workable choice. If you need to reliably handle long-context inference, the 2×32G configuration's VRAM advantage should take priority.

Conclusion and What's Next

This comparison focused on a single model — Qwen3.6 R7B FP8 — but it surfaces an important insight: stacking more GPUs does not equal linear performance scaling. TDP walls, bandwidth bottlenecks, and inter-GPU communication overhead all meaningfully drag down real-world throughput. For anyone building a local V100 inference cluster, understanding these constraints is far more practically valuable than fixating on total VRAM numbers alone.

The plan is to follow up with a comparison using the larger Qwen3.6 35B A3 FP8 model, a MoE (Mixture of Experts) architecture. MoE models activate only a subset of expert sub-networks on each forward pass (approximately 3B active parameters in the 35B A3 variant), dramatically reducing per-token compute despite the large total parameter count. Unlike dense models, however, different tokens are dynamically routed to expert networks distributed across different GPUs, introducing an All-to-All communication pattern — each token may need to be sent to any GPU holding the corresponding expert weights. This topology is far harder to optimize than the All-Reduce pattern in dense models and poses an even stiffer challenge to PCIe bandwidth. Load imbalance (some experts are heavily routed while others sit idle) and VRAM fragmentation are also more pronounced. The differences between the two hardware configurations — in VRAM usage and expert routing communication — are likely to be even more stark in that scenario, and well worth watching.