num_workers
A parameter of the PyTorch DataLoader that determines the number of subprocesses used to read and preprocess data in parallel, affecting data-loading throughput.
Timeline (last 90 days)
In this case, the real bottleneck of H100 training was non-GPU overhead in the data pipeline, including repeatedly downloading the dataset from S3, the base model not being cached, and single-threaded preprocessing with num_workers set too low.
After fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
In the PyTorch DataLoader, num_workers determines how many independent subprocesses are used to read and preprocess data in parallel, and a low value prevents data preparation from overlapping with GPU computation.
With QLoRA, GPU computation density is relatively low, making data pipeline shortcomings more likely to surface, so that the impact of num_workers, caching and preprocessing parallelism on end-to-end time is further amplified.
All Facts (4)
In this case, the real bottleneck of H100 training was non-GPU overhead in the data pipeline, including repeatedly downloading the dataset from S3, the base model not being cached, and single-threaded preprocessing with num_workers set too low.
50%UnverifiedAfter fixing the non-compute overhead (configuring persistent storage, migrating data to local NVMe, caching the base model and tokenizer, and increasing num_workers), the single-run time dropped from 9.5 hours to 2 hours 40 minutes, nearly a 4x speedup compared with the RTX 4070's 11 hours.
50%UnverifiedIn the PyTorch DataLoader, num_workers determines how many independent subprocesses are used to read and preprocess data in parallel, and a low value prevents data preparation from overlapping with GPU computation.
50%UnverifiedWith QLoRA, GPU computation density is relatively low, making data pipeline shortcomings more likely to surface, so that the impact of num_workers, caching and preprocessing parallelism on end-to-end time is further amplified.
50%