Inference Efficiency
An efficiency metric for the inference phase of large models, typically measured by the number of tokens generated per unit time (throughput) and first-token latency. It is in tension with inference cost and can be optimized through techniques such as distillation, quantization, and speculative decoding.
Timeline (last 90 days)
For large models, "inference efficiency" usually refers to the number of tokens that can be generated per unit time (throughput) and the first-token latency.
Sample efficiency and inference efficiency are two independent metrics; CoPE saves interactions during training and maintains single-step actions during inference, with no additional overhead in either.
All Facts (2)
For large models, "inference efficiency" usually refers to the number of tokens that can be generated per unit time (throughput) and the first-token latency.
50%UnverifiedSample efficiency and inference efficiency are two independent metrics; CoPE saves interactions during training and maintains single-step actions during inference, with no additional overhead in either.
50%