Zephon
A project implementing elastic deterministic data loading for TorchTitan, introducing a globally ordered sample index and a dynamic mapping mechanism to ensure consistent data ordering when the degree of parallelism changes.
Timeline (last 90 days)
The official PyTorch blog published a technical article introducing the implementation of Elastically Deterministic Data Loading for TorchTitan through the Zephon project.
Zephon introduces a globally ordered sample index mechanism at the data loading layer; each training sample has a unique logical sequence number in the entire dataset, and the data loader maps the global sequence to each rank and worker process according to the current parallel configuration.
When the degree of parallelism changes, Zephon's dynamic mapping logic ensures that each sample is still consumed only once, and the global order of consumption remains unchanged.
Zephon's globally ordered sample index abstracts data progress as a globally monotonically increasing logical cursor; a checkpoint only needs to record the global sequence number N currently consumed, and upon recovery any new parallel configuration can continue from that sequence number.
Zephon's integration with TorchTitan is mainly concentrated in the data loading pipeline wrapper layer, with almost zero intrusion into model code; users only need to enable Zephon's loading strategy in the data configuration.
All Facts (5)
Zephon introduces a globally ordered sample index mechanism at the data loading layer; each training sample has a unique logical sequence number in the entire dataset, and the data loader maps the global sequence to each rank and worker process according to the current parallel configuration.
50%UnverifiedWhen the degree of parallelism changes, Zephon's dynamic mapping logic ensures that each sample is still consumed only once, and the global order of consumption remains unchanged.
50%UnverifiedZephon's globally ordered sample index abstracts data progress as a globally monotonically increasing logical cursor; a checkpoint only needs to record the global sequence number N currently consumed, and upon recovery any new parallel configuration can continue from that sequence number.
50%UnverifiedZephon's integration with TorchTitan is mainly concentrated in the data loading pipeline wrapper layer, with almost zero intrusion into model code; users only need to enable Zephon's loading strategy in the data configuration.
50%UnverifiedThe official PyTorch blog published a technical article introducing the implementation of Elastically Deterministic Data Loading for TorchTitan through the Zephon project.
50%