Tensor parallelism splits the weight matrix of the same layer and places it across multiple machines, requiring one All-Reduce communication per layer computed, is extremely sensitive to bandwidth and latency, and is usually only practical under high-speed interconnects such as NVLink.