Realtime API
A real-time interaction interface provided by OpenAI, built on a bidirectional streaming channel over persistent WebSocket connections, supporting low-latency scenarios such as voice conversation and real-time transcription.
Timeline (last 90 days)
v3.35.1 fixes the byte reading issue of the beta Realtime feature under websockets 13, corresponding to PR #4102, with commit hash 5766349.
Before upgrading, it is recommended to verify message sending and receiving on Realtime connections in a test environment, confirm the byte reading path works properly, and then push to production.
Projects using the beta Realtime feature and running on websockets 13 should upgrade to v3.35.1 first.
A sideband connection is an event channel in the Realtime API that is independent of the main WebSocket session and is used to send outbound control-type events such as transcription text and tool-call status.
v3.35.0 added typed Live sideband events to the Realtime API (PR #4099).
Developers using realtime audio streams or binary frames are advised to upgrade to v3.35.0 and run regression tests.
The modalities parameter of the Realtime API is used to declare the input/output modality types supported by the current session, such as text and audio.
v3.19.2 clarified the definition of the modality array in the Realtime API (#3954).
The 2024 DevDay introduced capabilities including the Realtime API and Vision Fine-tuning.
OpenAI's Realtime API, ElevenLabs' low-latency mode, and Google's Live API all emphasize low-latency voice capabilities.
All Facts (12)
GPT-Realtime-2支持function calling（函数调用），允许模型在听到用户说话的同时决定调用哪个函数，无需等待完整的文本转录
85%UnverifiedA sideband connection is an event channel in the Realtime API that is independent of the main WebSocket session and is used to send outbound control-type events such as transcription text and tool-call status.
70%UnverifiedThe Realtime API launched by OpenAI at the end of 2024 supports WebRTC as a transport layer, with latency typically under 200ms.
70%UnverifiedBefore upgrading, it is recommended to verify message sending and receiving on Realtime connections in a test environment, confirm the byte reading path works properly, and then push to production.
50%Unverifiedv3.35.1 fixes the byte reading issue of the beta Realtime feature under websockets 13, corresponding to PR #4102, with commit hash 5766349.
50%UnverifiedProjects using the beta Realtime feature and running on websockets 13 should upgrade to v3.35.1 first.
50%UnverifiedDevelopers using realtime audio streams or binary frames are advised to upgrade to v3.35.0 and run regression tests.
50%Unverifiedv3.19.2 clarified the definition of the modality array in the Realtime API (#3954).
50%UnverifiedThe modalities parameter of the Realtime API is used to declare the input/output modality types supported by the current session, such as text and audio.
50%UnverifiedOpenAI's Realtime API, ElevenLabs' low-latency mode, and Google's Live API all emphasize low-latency voice capabilities.
50%Unverifiedv3.35.0 added typed Live sideband events to the Realtime API (PR #4099).
50%UnverifiedThe 2024 DevDay introduced capabilities including the Realtime API and Vision Fine-tuning.
50%