Speculative execution
A request strategy that balances latency and cost: after the first request is sent, if it has not returned within a set time limit, a second request is issued in parallel.
Timeline (last 90 days)
Hedged request is a compromise approach: after the first request is sent, if it does not return within a set time limit, a second request is sent in parallel, balancing latency and cost.
Speculative execution refers to sending requests to both large and small models simultaneously, accepting the small model's result if quality meets the standard and discarding the large model's result; however, this introduces additional computational resource waste and is only worth considering in latency-extremely-sensitive scenarios.
All Facts (2)
Hedged request is a compromise approach: after the first request is sent, if it does not return within a set time limit, a second request is sent in parallel, balancing latency and cost.
50%UnverifiedSpeculative execution refers to sending requests to both large and small models simultaneously, accepting the small model's result if quality meets the standard and discarding the large model's result; however, this introduces additional computational resource waste and is only worth considering in latency-extremely-sensitive scenarios.
50%