Voice and Multimodal Agents: Architecture and Latency

One number dominates voice-agent talk: respond in about 200 milliseconds, because that is how fast people answer each other. It is the most quoted target in the material we read. The number is real, but it is the mean of a wide distribution. In our assessment, production failures usually sit elsewhere: the agent speaks at the wrong moment, confirms something the backend never did, or places a call nobody agreed to receive. The evidence below points the same way; we found no controlled study linking latency to caller abandonment.

Our thesis is that a voice architecture is a decision about where three things are decided: when the user's turn has ended, what is true, and what may be said aloud. Pipelines, speech-to-speech models and full-duplex systems differ mainly in who holds those decisions. Latency is the symptom you see; the placement of control is the cause you can change.

This guide follows the evidence from conversational timing to transport, guardrails, consent law and evaluation, builds NKK models, and says where evidence is thin. The market moved fast in 2026 (GPT-Live reached general availability on 10 September ^[1], Gemini 3.8 Live on 15 September ^[2]), so vendor facts are dated 13 October 2026.

The 200 ms figure is a mean, not a deadline

The cleanest human data comes from Stivers and colleagues, who measured answers to yes/no questions in informal conversation in ten languages. The mean response offset was +208 ms and the median +100 ms. Language means ran from Japanese +7 ms to Danish +469 ms, which is 261 ms above the mean by our arithmetic (the paper says within about 250 ms) ^[3]. Heldner and Edlund, analysing three corpora, found turn-taking less precise than often claimed. Overlaps were about 40% of between-speaker intervals; 55-59% of all intervals (gaps plus overlaps) were shorter than 200 ms, and more than 40% were longer ^[4].

People plan replies while still listening: a Max Planck press release on Levinson's review says preparing a word takes 600 ms or more ^[5]. Speed also carries meaning: response speed predicted felt connection, at a scale under 250 ms ^[6].

Do not turn this into a specification. ITU-T G.114 sets 400 ms as a one-way limit for network planning and calls delays under 150 ms mouth-to-ear essentially transparent for most applications; neither is an AI target ^[7]. Vendor numbers are no firmer. Pipecat says the round trip is 500-800 ms "in most cases", with no stated conditions ^[8]. OpenAI's cookbook measured 1.51 s in one run and says a single run is not a benchmark ^[9].

Human timing versus agent claims

Key figures

• +208 ms

  mean response offset to yes/no questions across ten languages

  Source[3]
• ~40%

  of between-speaker intervals are overlaps, not gaps

  Source[4]
• 400 ms

  ITU-T G.114 one-way planning limit, not an AI target

  Source[7]
• 1.51 s

  latency in one cookbook run, not a benchmark

  Source[9]

We found no controlled study showing that any latency threshold makes callers hang up or distrust an AI agent. Our working view, as opinion: variance and mistimed interruption cost more than the gap between 400 and 700 ms. Humans overlap and backchannel constantly ^[4]; an agent that hears every sound as silence or interruption feels wrong.

Three architectures, one decision: where the turn is decided

OpenAI's voice documentation names three options ^[10]. A chained pipeline (speech-to-text, agent workflow, text-to-speech) is positioned for cases where "each stage needs to be visible or replaceable". The Realtime API uses one model to interpret audio, decide and answer in speech. GPT-Live is a full-duplex voice model that manages the conversation while a backend handles reasoning and tools ^[11][12]. Google's Live API runs over a stateful WebSocket ^[13], and Amazon's Nova 2 Sonic over bidirectional streaming ^[14]. The research model Moshi shows the full-duplex idea, with a reported 160 ms theoretical and 200 ms practical latency that cannot be compared with hosted-API figures ^[15].

Three voice architectures, compared on control

NKK framework

Criterion	Chained pipeline	Turn-based speech-to-speech	Full duplex with delegation
Who decides the turn	Your code: VAD, endpointer or turn-detector model	The model, with server-side VAD options	The voice model, listening and speaking at once
What you can inspect before audio plays	Transcript and text answer, so a policy check can sit before TTS	Tool calls and conversation items; no separate text answer to gate	The delegate's result; the spoken wording is the voice model's
Reasoning and tools	Any model you choose	One model reasons and speaks	A backend agent reasons; the voice model narrates
Interruption	You implement it	Provider handles it, differently per transport	Native, but backend work can keep running
Examples (October 2026)	Open-source frameworks plus STT and TTS services	gpt-realtime-2.1, Gemini Live API, Nova 2 Sonic	GPT-Live 1; Moshi in research
Best fit	Regulated or auditable calls	General assistants where naturalness matters	Open conversation with heavy backend work

Who decides the turn

Chained pipeline

Your code: VAD, endpointer or turn-detector model

Turn-based speech-to-speech

The model, with server-side VAD options

Full duplex with delegation

The voice model, listening and speaking at once

What you can inspect before audio plays

Chained pipeline

Transcript and text answer, so a policy check can sit before TTS

Turn-based speech-to-speech

Tool calls and conversation items; no separate text answer to gate

Full duplex with delegation

The delegate's result; the spoken wording is the voice model's

Reasoning and tools

Chained pipeline

Any model you choose

Turn-based speech-to-speech

One model reasons and speaks

Full duplex with delegation

A backend agent reasons; the voice model narrates

Interruption

Chained pipeline

You implement it

Turn-based speech-to-speech

Provider handles it, differently per transport

Full duplex with delegation

Native, but backend work can keep running

Examples (October 2026)

Chained pipeline

Open-source frameworks plus STT and TTS services

Turn-based speech-to-speech

gpt-realtime-2.1, Gemini Live API, Nova 2 Sonic

Full duplex with delegation

GPT-Live 1; Moshi in research

Best fit

Chained pipeline

Regulated or auditable calls

Turn-based speech-to-speech

General assistants where naturalness matters

Full duplex with delegation

Open conversation with heavy backend work

The useful question is not "cascade or end-to-end" but which party holds each decision, because that determines what you can check before a sentence is spoken.

What the 2026 benchmarks say about the trade-off

The strongest recent evidence is EVA-Bench, which scored 12 systems (7 cascade, 3 speech-to-speech, 2 hybrid) on 213 enterprise scenarios ^[20]. Accuracy was roughly at parity: the best cascade (Nova-3 + GPT-5.4 + Sonic 3) scored 0.504 on accuracy pass@1 and the best speech-to-speech system (GPT-Realtime-1.5) 0.467. Experience separated more by architecture: turn-taking scores were 0.82-0.83 for the three speech-to-speech systems and 0.28-0.58 for the seven cascades. The two hybrids scored 0.019 and 0.417, below the cascade range, so the split is not clean. Within cascades, accuracy and speed traded off: the three most accurate had mean tool-call turn latencies above 5 s. No system exceeded 0.5 on both accuracy and experience pass@1.

Reliability is the larger gap, and it does not depend on architecture. The top-accuracy cascade reached 0.809 at pass@k (some attempt succeeds) but 0.217 at pass^k (every attempt succeeds). The best speech-to-speech system fell from 0.710 to 0.283. A different cascade had the highest pass^k of any cascade, 0.269. The median pass@k to pass^k gap across all 12 systems was 0.44 ^[20].

EVA-Bench: peak versus reliable accuracy

Chart

• Top-accuracy cascade, pass@1
  0.5
• Top-accuracy cascade, pass@k (peak)
  0.81
• Top-accuracy cascade, pass^k (every attempt)
  0.22
• Best speech-to-speech, pass@1
  0.47
• Best speech-to-speech, pass@k (peak)
  0.71
• Best speech-to-speech, pass^k (every attempt)
  0.28

Values in score (0-1).

tau-Voice points the same way from a different angle. On 278 tasks, a text agent (GPT-5, reasoning) reached 85%, while voice agents reached 31-51% in clean conditions and 26-38% under realistic noise and accents ^[21].

tau-Voice: the text-to-voice capability gap

Key figures

• 85%

  text agent baseline (GPT-5, reasoning), pass@1 on 278 tasks

  Source[21]
• 31-51%

  voice agents, clean conditions

  Source[21]
• 26-38%

  voice agents, realistic noise and accents

  Source[21]

The paper reports voice agents keeping 30-45% of text capability under realistic noise ^[21]; against clean conditions the ratio is higher, about 36-60% by our arithmetic. Treat this with care: we could not confirm a venue for tau-Voice, the systems were tested months ago, and EVA-Bench's accent test used one French accent and one voice ^[20]. A 2025 benchmark found ASR-LLM pipelines ahead of end-to-end models on agentic tasks ^[22]; different benchmarks make a direct comparison impossible, but the parity in EVA-Bench is consistent with a narrowing. X-Talk claims a sub-second cascade but reports no measurements in its abstract ^[23].

The most interesting response is a hybrid. RelayS2S lets a duplex fast path draft a short prefix while a cascaded path writes the continuation. It reports P90 first-chunk latency of 81 ms (excluding TTS and network) against 1,006 ms for the cascaded baseline, keeping 99% of text quality ^[24]: one paper, first-chunk text latency, not voice-to-voice. The paper says nothing about inspectability, so the hybrid's place on the map below is a hypothesis.

NKK Control-Naturalness Map

NKK framework Illustrative

FluidTurn-taking naturalnessStilted

Natural but opaque

Fluid conversation, little you can check before audio plays. Suits low-stakes assistance.

Natural and controllable

The target for regulated calls. In 2026 evidence no single system clears 0.5 on both accuracy and experience.

Neither

Slow and unchecked. A design to avoid.

Controllable but stilted

Pipelines with silence-based endpointing: auditable, but they cut users off or wait too long.

Pipeline

Speech-to-speech

Full duplex

Hybrid (unproven)

OpaqueInspectability and policy controlEvery stage checkable

Natural but opaque

Fluid conversation, little you can check before audio plays. Suits low-stakes assistance.

Natural and controllable

The target for regulated calls. In 2026 evidence no single system clears 0.5 on both accuracy and experience.

Neither

Slow and unchecked. A design to avoid.

Controllable but stilted

Pipelines with silence-based endpointing: auditable, but they cut users off or wait too long.

Place your call type first. A collections or claims call needs the top-right; accept an inspectable pipeline until a hybrid proves itself on your recordings. An internal help assistant can sit top-left. Re-score on your own test set whenever the model changes.

Transport and telephony: where milliseconds go before the model starts

WebRTC carries media as RTP with SRTP mandatory ^[25]; WebSocket is TCP-based ^[26]. OpenAI recommends WebRTC over WebSocket for browser and mobile clients "for more consistent performance" ^[27]. Microsoft says WebSockets "aren't recommended for real-time audio streaming because they have higher latency than WebRTC" ^[28]. We found no independent head-to-head latency measurement, so the usual explanation, TCP head-of-line blocking, is protocol reasoning, not a measured result. WebRTC for client apps and WebSocket for trusted server links remains a sound default.

Browsers should not hold your API key. OpenAI issues an ephemeral client secret from your server ^[29][27]; Google offers ephemeral tokens (Preview) ^[30].

Telephony changes the physics. WebRTC endpoints must implement Opus and G.711 ^[31], and the phone leg is narrowband: Twilio's bidirectional streams take mu-law audio at 8 kHz ^[32], whatever the model's native rate (Gemini Live takes 16 kHz in ^[13]). SIP is the signalling layer that sets up the call ^[33]. OpenAI's SIP integration announces each call to your server by webhook ^[34].

Transport shows up in interruption behaviour. Over WebSocket the client must stop playback and send a truncate event; over WebRTC and SIP the server truncates unplayed audio itself ^[35], though the SIP guide still tells the bridge to manage queued audio and interruptions ^[34], so verify who owns buffered audio. On Twilio the application sends a clear message to empty buffered audio, and mark messages report what actually played ^[32]. Whoever owns playback knows what the caller actually heard; get it wrong and the history records sentences the caller never heard. Session limits differ by platform and are listed in the platform table below.

Transport options for a voice agent

NKK framework

Criterion	WebRTC	WebSocket	SIP trunk
Media	RTP with mandatory SRTP; JSON events on a data channel	TCP-based; you handle audio events	SRTP media over UDP; signalling on TLS port 5061
Audio format	Negotiated by SDP; Opus and G.711 required	Your choice: PCM on Gemini, mu-law 8 kHz on Twilio	Telephone-grade narrowband
Who truncates on interruption	Server (OpenAI)	Your client or bridge	Server (OpenAI); bridge still manages queued audio, verify
Best use	Browser and mobile clients	Server-to-server and telephony bridges	Inbound and outbound phone calls
Watch for	App server in the critical path if it brokers the SDP	Head-of-line blocking on lossy links (protocol reasoning)	Webhook latency; transfer and hangup handling

Media

WebRTC

RTP with mandatory SRTP; JSON events on a data channel

WebSocket

TCP-based; you handle audio events

SIP trunk

SRTP media over UDP; signalling on TLS port 5061

Audio format

WebRTC

Negotiated by SDP; Opus and G.711 required

WebSocket

Your choice: PCM on Gemini, mu-law 8 kHz on Twilio

SIP trunk

Telephone-grade narrowband

Who truncates on interruption

WebRTC

Server (OpenAI)

WebSocket

Your client or bridge

SIP trunk

Server (OpenAI); bridge still manages queued audio, verify

Best use

WebRTC

Browser and mobile clients

WebSocket

Server-to-server and telephony bridges

SIP trunk

Inbound and outbound phone calls

Watch for

WebRTC

App server in the critical path if it brokers the SDP

WebSocket

Head-of-line blocking on lossy links (protocol reasoning)

SIP trunk

Webhook latency; transfer and hangup handling

Turn-taking is a model problem: VAD, endpointing and interruption

The simplest approach is silence-based voice activity detection (VAD). OpenAI's server_vad chunks audio by periods of silence, and a shorter silence duration detects turns sooner ^[17]. Gemini's guide recommends 500-800 ms of silence; 100-200 ms can split one utterance into fragments, and 2,000 ms or more raises perceived latency ^[19]. Skantze and Irfan report that silence-based endpointing causes unnatural pauses and interruptions; predictive models (TurnGPT with Voice Activity Projection) significantly reduced response delays and interruptions in a study of 39 adults on a robot, not on a phone line ^[36].

Next is meaning-aware endpointing: OpenAI's semantic_vad waits longer when speech trails off ^[17], and LiveKit recommends its turn-detector model as the default ^[37]. Deepgram's Flux can signal end of turn early so the LLM starts before the user finishes, at the cost of 50-70% more LLM calls; its "~260ms" figure is a vendor claim without conditions ^[38]. Using such a signal for eager generation, with cancel-on-resume if the caller keeps talking, is what we count as rung 3. Above that sit models that decide themselves, which is what a full-duplex model does ^[39]. A survey by Lu and colleagues warns that "full-duplex" names many different capabilities ^[40].

NKK Turn-Control Ladder

NKK framework

1. 1

   Level 1: 1. Silence timer

   VAD plus a fixed silence window decides the user has finished. Fast to build; cuts people off mid-thought or waits too long.

   • OpenAI server_vad
   • Gemini automatic VAD
2. 2

   Level 2: 2. Meaning-aware endpointing

   A classifier on the words, or STT-integrated end-of-turn, decides. Waits when speech trails off.

   • OpenAI semantic_vad
   • LiveKit turn-detector model
   • Deepgram Flux end-of-turn
3. 3

   Level 3: 3. Predictive turn-taking

   Predicts turn ends from dialogue context, and starts generating early with cancel-on-resume.

   • TurnGPT with Voice Activity Projection
   • Eager generation with cancel-on-resume
4. 4

   Level 4: 4. In-model full duplex

   The model listens and speaks at once and emits its own listen/speak decisions. Most natural; least inspectable.

   • GPT-Live
   • Moshi (research)

Choose the lowest rung that meets your tolerance for interruption, then write that rung's interruption policy. Set the tolerance per call type by weighing the cost of cutting a caller off against the cost of a pause: a caller dictating a long reference number suffers more from a cut-off, a caller asking a yes-or-no question more from a pause. The ladder breaks because rungs are not strictly ordered by quality: Full-Duplex-Bench v1.5 found two divergent strategies among five agents, responsive and floor-holding ^[41], which NKK reads as a trade-off, not a ranking. A compliance script that must be read in full may want floor-holding. Benchmarks here are young: Full-Duplex-Bench covers pause handling, backchanneling, turn-taking and interruption management ^[42].

Teams often skip interruption policy. Gemini cancels and discards ongoing generation, and the client must clear its audio queue ^[19]. LiveKit stops speaking, truncates history to what the user heard, and offers an adaptive mode that separates real interruptions from backchannel ^[37]. EchoChain, a small-team preprint, found that no system exceeded a 50% pass rate when a user interrupted mid-response and the task state had to be updated; failures fell 40.2% in a half-duplex control ^[43].

A latency budget you can manage

Vendor figures measure different things. ElevenLabs states that its 75 ms is model inference only ^[44]; Deepgram's ~260 ms is end-of-turn detection ^[38]; RelayS2S's 81 ms excludes TTS and network ^[24]. None is the caller's wait.

The NKK Latency Ledger splits time to first audio into stages and records each stage's owner, timestamps and main lever. Endpointing and backend work are yours to control; model and TTS first-chunk times belong mostly to vendors.

NKK Latency Ledger

NKK framework

Criterion	Owner	Measured between	Main levers
Capture and uplink	You and the network	Mic capture to server receipt, p50/p95 by network type	WebRTC for clients; region placement; direct client connection with ephemeral tokens
Endpointing wait	You, by configuration	End of caller speech to turn-end decision	Meaning-aware detection; shorter silence with a resume policy; eager generation
STT finalisation	Vendor	Turn end to final transcript	Streaming STT with integrated end-of-turn
Model first token	Vendor	Request to first token	Lower reasoning effort where reliable; concise context
Tool or delegated work	You	Tool call start to result	Concurrent lookups; speculative lookups from partial transcripts; concise results; spoken acknowledgement
TTS first audio	Vendor	First text chunk to first audio byte	Stream LLM output into TTS; avoid chunk-schedule stalls
Downlink and playback	You and the network	Server send to speaker output	Jitter handling; codec; narrowband on phone legs

Capture and uplink

Owner

You and the network

Measured between

Mic capture to server receipt, p50/p95 by network type

Main levers

WebRTC for clients; region placement; direct client connection with ephemeral tokens

Endpointing wait

Owner

You, by configuration

Measured between

End of caller speech to turn-end decision

Main levers

Meaning-aware detection; shorter silence with a resume policy; eager generation

STT finalisation

Owner

Vendor

Measured between

Turn end to final transcript

Main levers

Streaming STT with integrated end-of-turn

Model first token

Owner

Vendor

Measured between

Request to first token

Main levers

Lower reasoning effort where reliable; concise context

Tool or delegated work

Owner

You

Measured between

Tool call start to result

Main levers

Concurrent lookups; speculative lookups from partial transcripts; concise results; spoken acknowledgement

TTS first audio

Owner

Vendor

Measured between

First text chunk to first audio byte

Main levers

Stream LLM output into TTS; avoid chunk-schedule stalls

Downlink and playback

Owner

You and the network

Measured between

Server send to speaker output

Main levers

Jitter handling; codec; narrowband on phone legs

Illustrative sub-second cascade budget

NKK framework Illustrative

• Capture and uplink
  50 ms
• Endpointing wait
  250 ms
• STT finalisation
  100 ms
• Model first token
  300 ms
• TTS first audio
  150 ms
• Downlink and playback
  100 ms

The bar chart is a target to negotiate, not a finding; the vendor claim of 500-800 ms ^[8] and the cookbook's 1.51 s run ^[9] bracket it. In NKK's view, tool and delegated work is the stage most likely to dominate; we found no data on which stage most often overruns. OpenAI's delegation guide suggests lower reasoning effort, concise tool results, concurrent and speculative lookups, and stage-level measurement ^[12].

Vision and documents: keep the voice layer thin

"Multimodal" on 2026 voice platforms mostly means voice plus still images or low-rate frames. OpenAI's Realtime guide adds images as input_image content parts in a user message, not streamed video ^[35], and Gemini Live caps frames at one per second ^[19]. The table summarises each platform ^[16][11][14].

Input modalities and session limits by platform, October 2026

Comparison

Criterion	Audio	Still images	Video or frames	Session limit
OpenAI gpt-realtime-2.1	Yes	Yes, as conversation items	Not listed	60 minutes
OpenAI GPT-Live 1	Yes (full duplex)	Unsupported	Unsupported	Not found in the pages read
Gemini Live API (guide does not name model versions)	Yes	JPEG/PNG frames	Up to 1 frame per second	15 min audio-only; 2 min with video, extendable
Amazon Nova 2 Sonic	Yes	Not stated in the guide	Not stated in the guide	8 minutes per connection, renewable

OpenAI gpt-realtime-2.1

Audio

Yes

Still images

Yes, as conversation items

Video or frames

Not listed

Session limit

60 minutes

OpenAI GPT-Live 1

Audio

Yes (full duplex)

Still images

Unsupported

Video or frames

Unsupported

Session limit

Not found in the pages read

Gemini Live API (guide does not name model versions)

Audio

Yes

Still images

JPEG/PNG frames

Video or frames

Up to 1 frame per second

Session limit

15 min audio-only; 2 min with video, extendable

Amazon Nova 2 Sonic

Audio

Yes

Still images

Not stated in the guide

Video or frames

Not stated in the guide

Session limit

8 minutes per connection, renewable

For documents and images we found no benchmark inside voice sessions, so the evidence is architectural. GPT-Live's design sends reasoning and tool use to a backend ^[12], and Nova 2 Sonic can keep speaking while tools run asynchronously ^[14]. NKK's view: keep the voice layer thin and hand documents, images and long-running work to a backend that returns a short, speakable result, with an acknowledgement in between. Gaze and gestures help coordinate human turn transitions ^[45]; we found nothing showing visual cues improve machine turn timing.

Guardrails for a channel you cannot unsay

A text agent can hold an answer until a check passes; spoken words are gone once played. Guardrails become a question of placement: the more irreversible the effect, the earlier the check.

OpenAI's delegation guide is blunt about the action layer: enforce permissions and required confirmations in application code, report an action complete only after the tool confirms, and check whether the original action happened before retrying ^[12]. For injection, OWASP's LLM01 says "it is unclear if there are fool-proof methods of prevention", and recommends least privilege, output validation and human approval for high-risk actions ^[46].

Audio-specific evidence exists but is dated. A 2024 study found text jailbreaks transferred poorly to GPT-4o's voice mode, yet a narrative-style attack raised average success across six scenarios from 0.033 to 0.778 ^[47]; a 2026 paper reports 98.26% for narrative audio attacks on Gemini 2.0 Flash ^[48]. Both are lab results on older models, and we found no primary-source report of a production voice-agent injection incident. GPT-Live's system card says the voice models have no code execution, with harder work delegated ^[39], so delegation is also a security boundary.

NKK Irreversibility Gates

NKK framework

1. 1. Entry gate

   Tell the caller it is an AI before anything else, state recording and processing, and get consent where the law requires it. Cheap to build, costly to retrofit.

   EU Art 50(1); recording law
2. 2. Input gate

   Treat transcripts and retrieved content as untrusted. Do not identify speakers by voice without a lawful basis.

   OWASP LLM01; EDPB
3. 3. Action gate

   Permissions, confirmations and idempotency live in application code, never in the prompt. Reconcile in-flight work on barge-in.

   Delegation guide
4. 4. Output gate

   For commitments, payments and medical or legal statements, approve the wording before TTS. A design idea: we found no verified setting or optimum for transcript-based checks that run during audio generation.

   Pipeline advantage
5. 5. Record gate

   Decide what is stored, for how long and who may replay it, including whether audio is kept at all.

   GDPR; wiretap law

NKK's view is that each call type gets an irreversibility class. Class 1, informational answers: gates 1, 2 and 5. Class 2, reversible actions: add gate 3. Class 3, irreversible or financial actions: gate 3 with explicit confirmation before the tool runs. Class 4, regulated commitments or statements: all five gates, and an architecture that exposes a text answer before TTS, which in practice means a pipeline or a hybrid you have proven. That is why a pipeline earns its latency cost on commitments.

One legal limit shapes guardrails directly: the AI Act prohibits inferring emotions of people in workplaces and education institutions, except for medical or safety reasons ^[51]. Voice-emotion analytics aimed at your own contact-centre staff needs legal review before it is designed in.

Consent, disclosure and recording: design to the strictest rule

This section is design guidance, not legal advice. In the EU, Article 50(1) obliges providers to design systems that interact directly with people so that those people are told it is AI, unless obvious, at the latest at first interaction (Article 50(5)) ^[49]. A company that builds its own agent on a vendor model may well count as the provider of that system; confirm with counsel. It applies from 2 August 2026 ^[52]. The Commission's guidelines list AI-enabled voice assistants as in scope, suggest an explicit spoken statement at the beginning of voice or telephony interactions with reminders in longer ones, and say tones alone are not sufficient ^[53]. Guidelines are interpretive, not binding law. Only the Article 50(2) machine-readable marking duty has a grace period, to 2 December 2026, for generative systems placed on the market before 2 August 2026 ^[54].

In the US, the FCC ruled in February 2024 that calls using AI-generated human voices are artificial or prerecorded voice under the TCPA, needing the called party's prior express consent to initiate absent an emergency or exemption ^[55]. An August 2024 FCC proposal would require disclosure at the start of each AI-voice call; we found no final rule as of 13 October 2026, though we could not open the docket page to confirm ^[56]. Recording is a separate regime. Federal wiretap law has a one-party consent exception unless the purpose is criminal or tortious ^[57]; California requires consent of all parties to a confidential communication, with fines up to $2,500 per violation ^[58]. We read California's text on a statute mirror; verify it on the official site.

Under GDPR, the European Data Protection Board states that using voice to identify a user is biometric processing, and that only explicit consent seems applicable as the Article 9 exemption ^[50]. That guidance was written for smart-speaker-style assistants before the AI Act, so applying it to enterprise voice agents is an extrapolation. We did not verify Illinois, Texas, Utah or UK rules.

Dates that shape voice-agent compliance

Timeline

1. 2024-02-02
   FCC TCPA ruling on AI voices

   Adopted 2 February, released 8 February: AI-generated human voices are artificial or prerecorded voice under the TCPA.

   Source[55]U.S. Federal Communications Commission, 2024
2. 2024-08-07
   FCC proposes AI-call disclosure

   A proposal, not a rule: disclose AI-generated voice at the start of each call. No final rule found as of 13 October 2026.

   Source[56]U.S. Federal Communications Commission, 2024
3. 2026-07-20
   Commission Article 50 guidelines

   Name AI-enabled voice assistants as in scope and suggest a spoken statement at the start.

   Source[53]European Commission, 2026
4. 2026-08-02
   Article 50 applies

   AI disclosure duties apply from this date.

   Source[59]European Commission (AI Act Service Desk), unda
5. 2026-12-02
   Article 50(2) grace period ends

   Machine-readable marking for generative systems placed on the market before 2 August 2026.

   Source[54]European Commission (AI Act Service Desk), unda

Disclosure and consent gate

NKK framework

0 of 5 answered Reset

AI disclosure

When does the caller learn it is an AI?

1. Not stated, or only implied by the voice2. A line in a menu or terms page3. Spoken at the start of every call, repeated after long interruptions or role changes

Recording consent

How do you handle jurisdictions that need all-party consent?

1. One notice for every caller, no legal review2. Notice at the start, jurisdiction not checked3. Strictest-rule notice plus consent capture, reviewed by counsel

Outbound AI-voice calls

Do you hold prior express consent before initiating?

1. Unknown2. Consent captured, evidence not retained3. Consent and exemption basis logged per number

Voice identification

Do you identify speakers by voiceprint?

1. Yes, with no separate basis2. Yes, under a general privacy notice3. No, or only with explicit consent recorded

Vendor data use

What may your AI vendor do with call audio?

1. Not reviewed2. Contract reviewed, retention unverified3. Contract and settings verified: retention, training use and regional endpoints

Answer all 5 dimensions to see your score and where to start.

Evaluate the call, not the transcript

OpenAI's cookbook harness steps through three modes: crawl (synthetic single-turn speech), walk (saved caller recordings, optionally with noise and telephony conditions) and run (an independent simulated caller in continuous multi-turn conversation). It reports P50 and P90 with sample counts and repeats runs because voice models are nondeterministic ^[9]. A natural-sounding confirmation, it warns, does not prove the right backend actions happened.

Perturb the audio. VoiceBench exists because earlier evaluations focused on clean speech; it adds speaker, environment and content variation ^[60]. In EVA-Bench, accent cut cascade task completion by a mean 10 points, and accent plus noise cut it by a mean 19 points, while speech-to-speech stayed within 5 points ^[20]. Test timing behaviours too: Full-Duplex-Bench v1.5 covers interruption, backchannel, talking to others and background speech ^[41]; EchoChain tests state updates after interruption ^[43]. A survey notes that evaluation here is fragmented ^[61]; expect to assemble your own suite.

“Test both the conversation and the completed task.”

NKK Voice Evaluation Ladder

NKK framework

1. 1

   Level 1: 1. Crawl

   Synthetic single-turn utterances. Checks tool selection and wording cheaply, in CI.

   • Deterministic checks on tool calls
   • Does not test timing or noise
2. 2

   Level 2: 2. Walk

   Recorded or re-synthesised callers with noise, accents and telephony codecs. Checks robustness.

   • Accent and noise perturbations
   • 8 kHz mu-law replay
3. 3

   Level 3: 3. Run

   A simulated caller in continuous multi-turn conversation, with interruptions. Checks timing, state and recovery.

   • P50/P95 time to first audio
   • Interruption and backchannel cases
4. 4

   Level 4: 4. Production loop

   Real failed calls, consented and redacted, become new test cases weekly. Report pass^k, not only pass@k.

   • Human listening review
   • Failure taxonomy feeding the suite

Report three families of metrics separately: task (did the right thing happen), timing (time to first audio, interruption handling, silence during delegation) and experience (human listening). Gate releases on task completion at pass^k.

What to do next

1. Pick the architecture per call type with the Control-Naturalness Map. Start regulated calls on a pipeline and prove hybrids on your own recordings.
2. Define time to first audio now; measure p50 and p95 from day one, acknowledgements separately.
3. Use WebRTC for client apps and a SIP or WebSocket bridge for telephony. Verify codecs and who truncates history on barge-in.
4. Write the interruption policy: cancel or reconcile in-flight tool calls, and check state before retrying.
5. Enforce permissions and confirmations in code; put output checks before speech for irreversible statements.
6. Write the disclosure and consent script with counsel, scored on the gate above, before the first live call.
7. Build the crawl-walk-run suite with accent, noise and telephony perturbations, and plan for vendor churn: model names changed within months in 2026.

Conclusion

The hard parts of voice agents are turn-taking, reliability and consent, not speech quality. Your architecture settles who decides the turn, who verifies the truth and who may speak. Measure the caller's wait honestly, put gates where effects cannot be undone, and test the completed task, not the transcript.