Elastic Training
An advanced fault-tolerance capability that allows distributed training jobs to continue running when the number of resources changes, requiring the training framework to support dynamic adjustment of parallelism and global batch size.
Timeline (last 90 days)
NVIDIA's automated recovery approach shares the same goal as elastic training—reducing manual intervention and shortening downtime—but elastic training places higher demands on framework modification.
Elastic training means that a training task can dynamically add or remove computing resources at runtime, which triggers reinitialization of the process group (rendezvous), causing the global rank count and topology to change.
All Facts (2)
NVIDIA's automated recovery approach shares the same goal as elastic training—reducing manual intervention and shortening downtime—but elastic training places higher demands on framework modification.
50%UnverifiedElastic training means that a training task can dynamically add or remove computing resources at runtime, which triggers reinitialization of the process group (rendezvous), causing the global rank count and topology to change.
50%