Assistant Prefill
A feature supported by certain LLM APIs that allows the prefix of an assistant reply to be pre-filled in the request, causing the model to continue generation from the specified prefix. Commonly used in scenarios such as resuming generation after a streaming interruption.
Timeline (last 90 days)
Anthropic's Claude API supports assistant prefill, implemented by setting the role of the last message in the messages array to assistant with no stop_reason.
Using prefill for continuation does not guarantee that the content will be fully consistent with the original sampling path, because the random seed and KV cache state are already different.
All Facts (2)
Anthropic's Claude API supports assistant prefill, implemented by setting the role of the last message in the messages array to assistant with no stop_reason.
50%UnverifiedUsing prefill for continuation does not guarantee that the content will be fully consistent with the original sampling path, because the random seed and KV cache state are already different.
50%