Training
The process of using massive amounts of data to let a model learn its parameters. It typically requires extremely high floating-point compute and memory bandwidth, and is one of the key scenarios driving demand for AI chip compute.
Timeline (last 90 days)
Sending a chat message does not immediately retrain the model; training repeatedly adjusts model parameters through large amounts of data, takes weeks or even months, and is usually done offline; inference, by contrast, uses the already-trained fixed parameters to process new input.
Context, memory and training are three different concepts: context is the information available for this reply, memory is information that has been selected and saved, and training is the process by which the model learns underlying patterns.
Training refers to repeatedly iterating model parameters with large-scale data, an offline, long-cycle task; inference is the forward computation process by which the model responds to each user request after going live, a real-time, highly concurrent online task.
Training can be completed concentrated in a few supercomputing clusters, but inference is an ongoing expense, and lowering the cost per token of inference helps seize the initiative in AI product gross margins.
Training prioritizes maximizing throughput, whereas inference is extremely sensitive to latency and faces unpredictable traffic fluctuations.
The inference market is far more sensitive to performance-per-watt than the training market.
The lifecycle of deep learning is typically divided into two phases: Training and Inference.
All Facts (7)
Sending a chat message does not immediately retrain the model; training repeatedly adjusts model parameters through large amounts of data, takes weeks or even months, and is usually done offline; inference, by contrast, uses the already-trained fixed parameters to process new input.
50%UnverifiedContext, memory and training are three different concepts: context is the information available for this reply, memory is information that has been selected and saved, and training is the process by which the model learns underlying patterns.
50%UnverifiedTraining refers to repeatedly iterating model parameters with large-scale data, an offline, long-cycle task; inference is the forward computation process by which the model responds to each user request after going live, a real-time, highly concurrent online task.
50%UnverifiedTraining can be completed concentrated in a few supercomputing clusters, but inference is an ongoing expense, and lowering the cost per token of inference helps seize the initiative in AI product gross margins.
50%UnverifiedTraining prioritizes maximizing throughput, whereas inference is extremely sensitive to latency and faces unpredictable traffic fluctuations.
50%UnverifiedThe inference market is far more sensitive to performance-per-watt than the training market.
50%UnverifiedThe lifecycle of deep learning is typically divided into two phases: Training and Inference.
50%