Unverified50% confidenceFactExact time
Tensor Parallelism horizontally shards the weight matrices of individual layers across multiple GPUs and aggregates results via AllReduce communication; it has lower latency but places extremely high demands on inter-GPU interconnect bandwidth (such as NVLink).
1
Sources
50%
Confidence
Long-term
Relevance
9/21/2026
First Seen
Sources
Related Entities
Related Claims
Unverified块调度建议将张量并行（TP）限制在NVLink带宽最高的GPU子集内（同一个最小块），数据并行（DP）跨块分布，流水线并行（PP）沿拓扑层次展开76% similarUnverifiedThe core goal of NVLink is to break through the bandwidth limitations of the traditional PCIe bus and allow multiple GPUs to communicate directly at extremely high speeds.75% similarUnverifiedThe communication volume of the Ring-AllReduce algorithm is independent of the number of GPUs; the total amount of data sent and received by each node is fixed at 2(N-1)/N times the data size, making it theoretically the bandwidth-optimal AllReduce implementation.73% similarUnverified将所有请求路由到同一个服务商可命中温热缓存，若请求被负载均衡到不同GPU节点则缓存命中率会大幅下降，因各节点的KV Cache相互独立69% similarUnverifiedMoE架构中每个Token需通过All-to-All通讯路由到对应专家GPU，随集群规模增大通讯开销呈近似平方级增长69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/939691API
curl https://kongchang.com/api/v1/knowledge/claims/939691MCP
get_claim(id=939691)