OpenAI Realtime API: What has happened in the year since the beta

Nearly a year after the developer preview was introduced, OpenAI released the GA version (General Availability) of the Realtime API in August 2025. The Realtime API is a multimodal interface that allows audio and text data to be exchanged directly with a language model with very low latency. The Developer Day in October 2025, a few months after the GA launch, brought additional innovations, including new tools, price changes, and a smaller, faster model variant.

Marius Obert

Marius Obert enjoys building prototypes with the latest cloud technologies and loves talking about them even more. He started his career in UI development in sunny California. During this time, he learned to love web technologies such as JavaScript in general and the entire Node.js ecosystem in particular.

With the GA version, OpenAI significantly expands the possibilities for interaction with AI agents. In a blog post, the company presents application examples from partners such as Zillow, T-Mobile, StubHub, Oscar Health, and Lemonade, which illustrate the diversity of use cases. The application fields go beyond classic language dialogues and enable so-called "hands-free interactions," where users can flexibly combine text, voice, and visual inputs. The Realtime API is not exclusively designed as a voice-to-voice solution, but as a multimodal system that accepts text, audio, and images equally as input. Voice interaction thus represents a complementary communication channel alongside other forms of use.

Technical Innovations in the General Release

Compared to the Developer Preview, the GA version of the Realtime API contains numerous technical extensions and improvements in the areas of model architecture, integration, and usability. A central change is the introduction of a mini variant of the model, which enables more cost-effective and faster applications. The OpenAI website shows the differences between gpt-realtime and gpt-realtime-mini.

Heise Conference: enterJS Integrate AI

When is Voice AI useful in web applications, and when is it not? Marius Oberts's talk at the online themed day enterJS Integrate AI on April 28, 2026. Early bird tickets and group discounts in the online ticket shop.

The audio quality has been significantly revised: the generated speech sounds more natural and expressive, with finer intonations, smoother pauses, and better adaptation to conversation flows. OpenAI has introduced two new voices for this: "Cedar" and "Marin."

The provider has also improved the ability to follow complex instructions. The model responds more precisely to system and developer prompts, and it can read texts exactly, reproduce alphanumeric sequences correctly, and switch fluently between languages. Benchmark tests such as the Big Bench Audio Evaluation indicate an increase in accuracy from around 65 percent in the beta version to over 82 percent in the GA version:

A significant improvement concerns the model's enhanced rhetorical capabilities. The Realtime API can now make conversation flows smoother, more natural, and more context-aware by better interpreting pauses, intonations, and conversational dynamics. In this context, OpenAI introduces the Conversation Idle Timeouts function. If the model detects no input over a defined period, it can automatically output follow-up sentences such as "Are you still there?" to maintain the conversation flow and signal an active conversation to the user.

Furthermore, during longer or asynchronous function calls, the model responds with intermediate texts, such as "I'm still waiting for the result," to bridge waiting times communicatively and keep the dialogue lively. These additions contribute to making the interaction more natural and consistent – especially in use cases where real-time feedback and spoken intermediate outputs are crucial for the user experience.

In addition, the OpenAI Realtime API now supports the Session Initiation Protocol (SIP) in addition to WebSocket and WebRTC, which facilitates direct integration into telephony and contact center systems.

For developers, the provider has revised the structures of the event ("Event") and message items to simplify debugging and error handling. This model also now has EU data residency to comply with European data protection requirements.

Pre-built tools such as Web-Search or Code Interpreter are not yet integrated, so users have to rebuild them themselves. Via the implementation of the MCP server (Model Context Protocol), it is possible to integrate external tools into the agent logic. This allows the capabilities of an agent to be extended relatively easily and integrated into existing tools of larger applications. These changes collectively increase the robustness, flexibility, and practical applicability of the API.

Model Overview

The following table provides an overview of the key model parameters and pricing structures for gpt-realtime and gpt-realtime-mini.

Table 1: Comparison of the core parameters of both models; cost per million tokens in US dollars ($)

Improved Developer Experience with the Agents SDK

In parallel with the GA release, OpenAI has also further developed the developer experience. The Agents SDK, introduced in early 2025, now supports the Realtime API and simplifies the creation of interactive, multimodal agents.

It offers standardized integrations for various platforms, including:

• Twilio: Via the Media Streams API, live audio from phone calls can be directly forwarded to the Realtime API. The SDK handles technical details such as latency management and audio formatting.
• Cloudflare Workers: A customized transport layer enables real-time interaction even in environments without native WebSockets.
• Vercel AI SDK: An adapter allows the connection of various models, which is advantageous for multi-agent applications and model mixing.

These tools reduce infrastructure and integration effort and allow developers to focus on logic and user experience. In productive environments, this improves reliability and scalability without having to delve deep into streaming or audio architectures.

In practical use, this allows, for example, a voicebot to be realized with minimal code that dynamically retrieves data from an external API during a live phone call – without manual work on WebSocket or network protocols. With the Agents SDK, programmers can elegantly write the entire implementation, which without the Agents SDK might require, for example, 275 lines, in less than 100 lines of JavaScript code, representing a saving of more than 60 percent. The following listing shows an example:

import Fastify from 'fastify';
import dotenv from 'dotenv';
import fastifyFormBody from '@fastify/formbody';
import fastifyWs from '@fastify/websocket';
import {
   RealtimeAgent,
   RealtimeSession,
   backgroundResult,
   tool,
} from '@openai/agents/realtime';
import { TwilioRealtimeTransportLayer } from '@openai/agents-extensions';
import { z } from 'zod';

// Lese Umgebungsvariablen aus einer .env-Datei
dotenv.config();

const { OPENAI_API_KEY } = process.env;
if (!OPENAI_API_KEY) {
   console.error('Fehlende Umgebungsvariable. Bitte definieren Sie diese in einer .env-Datei.');
   process.exit(1);
}

// Initialisiere Fastify-Server
const fastify = Fastify();
fastify.register(fastifyFormBody);
fastify.register(fastifyWs);

// Tool zum Bewerten von Reisepreisen – entspricht der ursprünglichen Funktionalität
const bewerteReisePreisTool = tool({
   name: 'bewerte_reise_preis',
   description: 'Falls der Nutzer nach dem Preis der Reise fragt, nutze diese Funktion, um den Preis zu berechnen.',
   parameters: z.object({
       start: z.string().describe('Ort, von dem die Reise startet'),
       ziel: z.string().describe('Ort, an dem die Reise endet'),
       verkehrsmittel: z.string().describe('Verkehrsmittel für die Reise'),
       dauer: z.number().describe('Dauer der Reise in Minuten'),
   }),
   execute: async ({ start, ziel, verkehrsmittel, dauer }) => {
       const preis = Math.round(Math.random() * 10 * dauer);
       console.log(`Der zufällige Preis für die Reise von ${start} nach ${ziel} mit ${verkehrsmittel} beträgt ${preis}€`);
       return backgroundResult(`Der Preis für die Reise von ${start} nach ${ziel} mit ${verkehrsmittel} (Dauer: ${dauer} Minuten) beträgt ${preis}€.`);
   },
});

// Erstelle den Agent mit deutschen Anweisungen
const agent = new RealtimeAgent({
   name: 'Reise-Assistent',
   instructions: 'Du bist ein hilfsbereiter Sprachassistent, der Fragen kurz und bündig beantwortet, ohne viel zu reden. Du hilfst hauptsächlich bei Reisefragen und Preisbewertungen.',
   tools: [bewerteReisePreisTool],
});

// Route für eingehende Anrufe von Twilio
fastify.all(
   '/incoming-call',
   async (request, reply) => {
       const twimlResponse = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
   <Say language="de-De">Viel Spaß beim Testen des Sprachassistenten!</Say>
   <Connect>
       <Stream url="wss://${request.headers.host}/media-stream"/>
   </Connect>
</Response>`;
       reply.type('text/xml').send(twimlResponse);
   },
);

// WebSocket route für die WebSocket-Verbindung zwischen Twilio und OpenAI
fastify.register(async (scopedFastify) => {
   scopedFastify.get(
       '/media-stream',
       { websocket: true },
       async (connection) => {
           console.log('Anrufer verbunden');

           const twilioTransportLayer = new TwilioRealtimeTransportLayer({
               twilioWebSocket: connection,
           });

           const session = new RealtimeSession(agent, {
               transport: twilioTransportLayer,
               model: 'gpt-realtime'
           });

           // Verbinde zur OpenAI Realtime API
           await session.connect({
               apiKey: OPENAI_API_KEY,
           });
           console.log('Verbunden mit der OpenAI Realtime API');

       },
   );
});

fastify.listen({ port: 3000 }, (err) => {
   if (err) {
       console.error(err);
       process.exit(1);
   }
   console.log(`Server gestartet`);
});

Listing 1: Example of a compact phone assistant with the Agents SDK

The listing demonstrates how a voice assistant with real-time audio can be implemented in a few lines with the Agents SDK. A Fastify server processes incoming calls and audio streams via Twilio, while the provided transport layer establishes the connection to the OpenAI Realtime API. A tool for evaluating travel prices is integrated via zod validation. The agent responds to German-language instructions and provides short, concise answers.

Processing occurs without media breaks and with minimal latency: speech is recognized, processed, and the response is synthesized directly. The entire logic remains bundled in a compact, scalable server process and can be implemented with low maintenance.

Changes During the Transition from Beta to GA

In addition to functional enhancements, there were also structural changes in the API design:

• Removal of the Beta header OpenAI-Beta: realtime=v1; requests are made without a special header.
• Introduction of short-lived API keys that are generated centrally
• Removal of the Temperature parameter for controlling model creativity
• New endpoints and optimized event structure for WebRTC and WebSocket streams
• New URL schema for WebRTC initialization
• Revised event names and item formats (e.g., response.text.delta became response.output_text.delta).
• New item events .added and .done for finer synchronization

Limitations and Known Boundaries

Despite the significant progress of the OpenAI Realtime API, there are still some key limitations that developers should consider when planning their applications.

The system is based on a Large Language Model (LLM) that is fundamentally designed to interact actively. This characteristic leads to the model tending to react proactively to conversational situations – even when this is not explicitly desired.

Even with clear instructions such as "Please only respond when addressed by name" or "Just listen," the model often participates unprompted in group conversations, for example, when a short speaking pause occurs during a conference call.

Since every input in the Realtime API is answered with an output almost immediately, passive notetaker scenarios, where a system merely listens and summarizes the conversation later, can only be realized to a limited extent or not at all. For such use cases, manual transcription using classic speech-to-text methods remains a viable option.

Limitations are also currently apparent in multi-user situations. The model often cannot clearly identify to whom a statement is directed, which can lead to misunderstandings in conversations with multiple participants.

Another point concerns the cost structure of productive applications. Despite the significant price reductions through the mini model, the computational effort for high-quality real-time audio processing remains a crucial factor that should be evaluated individually.

For orientation: in an average conversation with about 140 words per minute and participant, approximately 5,600 tokens (or 0.0056 million) are generated per direction in 30 minutes – i.e., for input and output each.

Based on this, the following estimated costs arise for both models:

Table 2: Estimation of costs per million tokens by model in US dollars ($)

The prices of both models are significantly lower compared to the beta version. The mini model, in particular, is very cost-effective and reduces prices by 90 percent. This opens up new, cost-sensitive application areas. Nevertheless, this variant should also be carefully tested for scenarios that require high audio processing performance over extended periods – for example, continuous protocol interrogations or meeting summaries where a model "silently" transcribes over a long time.

In general, developers must carefully weigh which model type best meets the requirements of their application and whether real-time capabilities truly offer added value. However, for various modern use cases where immediate, multimodal interaction and bidirectional voice interfaces are crucial, the Realtime API is now practically applicable and opens up new possibilities. For classic, purely textual tasks, the standard chat model often suffices.

Competition and Outlook

In the area of multimodal real-time interaction, OpenAI is currently the leading provider. Comparable developments exist at Google (Gemini Live), Amazon (Nova Sonic), and Alibaba (Qwen3 Omni), but these are not yet available or documented to a similar extent. The OpenAI Realtime API is also available via Microsoft Azure and can be used there at the same price as directly from OpenAI due to the close strategic partnership between Microsoft and OpenAI.

Furthermore, since July 2025, Microsoft has offered a more abstracted interface with the Voice Live API, which includes additional functions such as video streaming of digital avatars via WebRTC. The Voice Live API can be operated in the Pro, Basic, and Lite categories and can be combined with models from OpenAI as well as Microsoft's own models: Voice Live Pro includes GPT-4o-Realtime, GPT-4o, and GPT-4.1, among others; Voice Live Basic offers smaller variants such as GPT-4o-Mini-Realtime, GPT-4o Mini, and GPT-4.1 Mini; while Voice Live Lite relies on equivalent models such as GPT-4.1 Nano or Phi.

A comparison of the current pricing models shows the following values:

• Google Gemini 2.5 Flash Native Audio (Live API) charges approximately $3 per million input tokens for audio and video, and around $12 per million output tokens, including all processing and reasoning tokens.
• Amazon Nova Sonic costs $3.40 per million input speech tokens and $13.60 per million output speech tokens.
• Alibaba Qwen3 Omni is open source. There is currently no publicly available information on costs.
• Microsoft Azure: Via Azure, the Realtime API is available at the same prices as directly from OpenAI, currently $64 (input) and $32 (output) per million audio tokens for the standard model, and $20 and $10 per million audio tokens for the mini variant; when using the more abstracted Azure Voice Live API, prices are higher due to additional abstraction and platform functions, at approximately $88/$44 per million audio tokens (standard) and $22/$11 per million audio tokens (mini), with the same underlying model.

While other providers are still experimenting with early speech-to-speech models, OpenAI is already providing a stable platform with a mature SDK and comprehensive developer support with the Realtime API. Price differences are explained, among other things, by differences in the performance and objectives of the underlying models. However, it should be considered that models from different providers are not necessarily directly comparable.

With the GA version of the Realtime API, a stable and consolidated range of functions is now available that goes beyond early speech-to-speech experiments. By supporting multiple input formats – voice, text, and image – and with low end-to-end latency, application scenarios can be implemented that are only partially feasible with classic "speech-to-text à LLM à text-to-speech" architectures. At the same time, with the further development of the SDK, the clearer API structure, and the expanded integration options, hurdles for the implementation of such systems have been reduced.

Thus, the current version of the Realtime API covers additional use cases and makes it easier for developers to transition from experimental prototypes to production-ready, interactive real-time applications.

(mki)