AI voice agents: When to build and when to skip

AI voice agents handle phone calls, voice commands, and spoken interactions autonomously. Unlike text-based AI agents that process typed input, voice agents add two demanding constraints: sub-second latency and natural-sounding speech. Get either wrong and the conversation feels broken.

The voice pipeline

Every voice agent follows the same three-stage pipeline:

Stage 1: Speech-to-text (STT)

The user speaks. The audio is captured and converted to text. This stage takes 100-300ms depending on the provider and whether you use streaming.

Key providers: Deepgram (fastest, 100-200ms streaming), Google Speech-to-Text, Azure Speech Services, AssemblyAI, and Whisper (open source, higher latency).

Use streaming STT. Instead of waiting for the user to finish speaking, process audio in chunks as it arrives. Deepgram's streaming API starts returning partial transcripts within 100ms.

Stage 2: LLM reasoning

The transcribed text goes to the LLM. The LLM generates a response based on the conversation context, tools, and instructions. This is the slowest stage - typically 200-500ms for the first tokens.

Use streaming LLM responses. Start sending text to the TTS engine as soon as the first tokens arrive, not after the full response is generated. This overlaps Stage 2 and Stage 3.

Stage 3: Text-to-speech (TTS)

The LLM's text response is converted to speech audio. Modern TTS engines produce natural-sounding speech with emotion and pacing.

Key providers: ElevenLabs (highest quality), Cartesia (low latency), PlayHT, Azure Neural Voices, and Google WaveNet.

Stream TTS output. Start playing audio from the first sentence while later sentences are still being generated.

Total pipeline latency

The optimized pipeline achieves latency comparable to human conversation pauses (300-700ms between turns).

McKinsey's generative AI research found that voice AI deployments produced a 14% improvement in issue resolution per hour and a 9% reduction in average handle time. Both gains came almost entirely from cutting pipeline latency.

Handling interruptions (barge-in)

Humans interrupt each other constantly. A voice agent must handle interruptions gracefully:

1. Detect the interruption: monitor audio input while the agent is speaking. When the user starts talking, the agent should stop.
2. Stop playback: immediately cease current TTS output.
3. Process the interruption: send the new user speech through the STT pipeline.
4. Discard unspoken text: if the LLM generated text that wasn't spoken yet, decide whether to discard it or save it.
5. Respond to the new input: generate a response that acknowledges the context shift.

Turn-taking

Knowing when the user has finished speaking is surprisingly difficult. Silence alone isn't a reliable signal - people pause mid-sentence to think.

Approaches to end-of-turn detection:

• Silence duration: wait for 500-700ms of silence. Simple but causes awkward pauses.

• Prosodic cues: detect falling pitch and slowing tempo that signal sentence endings. More natural but harder to implement.

• Semantic analysis: use the STT transcript to predict whether the sentence is complete. Most accurate but adds latency.

• Hybrid: combine silence detection with semantic completeness checking. Best results in practice.

The ideal system adapts its turn-taking behavior to the conversation. During rapid exchanges, use shorter silence thresholds. During complex questions, wait longer.

RaftLabs has found in production builds that a 600ms silence threshold works well for appointment booking but feels awkward in sales qualification calls, where users pause to think. Each conversation type needs a tuned threshold. There's no single value that works for all use cases.

"Turn-taking in conversational AI is harder than it looks. You need to model not just silence but intent -- whether the user has finished their thought or is just pausing. Getting this wrong is the fastest way to make users abandon the call." -- Monica Lam, Professor of Computer Science at Stanford, speaking at INTERSPEECH 2023

Voice quality and personality

The voice your agent uses defines its personality. Consider:

Voice selection: choose a voice that matches your brand and use case. A healthcare appointment system needs a calm, reassuring voice. A sales agent needs an energetic, warm voice.

Pacing and prosody: control speaking speed, pause duration, and emphasis. Slower for important information. Faster for routine confirmations. Pauses before key numbers.

Emotional range: modern TTS engines support emotional modulation. The agent should sound empathetic during complaints, confident during problem resolution, and warm during greetings.

Consistency: use the same voice across all interactions. Switching voices between calls breaks trust.

RaftLabs has seen clients lose trust in AI agents that solved every problem but sounded robotic doing it. At 10,000 calls per day, a poor voice choice compounds fast. The voice is the product -- it's what customers remember.

"Customers forgive many things. A bad voice is not one of them. Research shows that perceived warmth and naturalness in a voice agent correlates directly with task completion rates and customer satisfaction scores." -- Clifford Nass, co-author of "Wired for Speech" and Professor at Stanford University

Production considerations

Call recording and compliance

Record all calls with appropriate disclosure. Many jurisdictions require informing callers they're speaking with AI. Build the disclosure into the greeting: "Hi, this is an AI assistant from [Company]. How can I help you?"

Fallback to human

Always provide a path to a human agent. "Let me connect you with a team member" should be triggered by: low confidence in understanding, user frustration signals, complex requests beyond the agent's scope, or explicit user request.

Telephony integration

Voice agents connect to phone systems via SIP trunking or WebRTC. Providers like Twilio, Vonage, and Retell handle the telephony layer. Your AI agent handles the conversation logic while the telephony provider handles the transport.

Cost per call

• STT: $0.005-0.02 per minute

• LLM: $0.01-0.10 per call (depending on conversation length and model)

• TTS: $0.01-0.05 per minute

• Telephony: $0.01-0.05 per minute

• Total: $0.05-0.25 per minute, or $0.25-1.50 for a typical 5-minute call

Compare to human agent cost of $1-3 per minute (fully loaded), and the economics are compelling for high-volume use cases. Gartner predicts conversational AI will reduce contact center agent labor costs by $80 billion by 2026 - and voice is the channel where the bulk of that shift happens. This is why AI customer service agents using voice are growing faster than text-only alternatives.

Where voice AI works today

A December 2024 Gartner survey found 85% of customer service leaders plan to explore or pilot customer-facing conversational GenAI in 2025, with 44% specifically exploring voicebots. Adoption is moving from experimentation to production deployment.

High-confidence use cases:

• Appointment scheduling and confirmation

• Order status and tracking

• Restaurant reservations

• Payment reminders

• Survey collection

• After-hours call handling

Emerging use cases:

• Technical support with troubleshooting flows

• Sales qualification calls

• Insurance claims intake

• Healthcare symptom triage

Not ready yet:

• Emotionally sensitive calls (collections, bad news delivery)

• Complex negotiations

• Unstructured conversations with no clear goal

Gartner predicts agentic AI will autonomously resolve 80% of common customer service issues without human intervention by 2029 - including voice interactions. The "not ready yet" list will shrink significantly over the next 24 months.

Voice AI is advancing fast. The latency and quality gaps are closing each quarter. For structured, high-volume conversations, it is production-ready today — and voice AI development is more accessible than most teams assume.

At RaftLabs, we have built production voice agents handling thousands of daily calls across hospitality and fintech. The pattern that works: start with a structured use case (appointment booking, order status), nail sub-500ms latency, then expand scope. Our AI agent development team ships voice-capable agents in 12-week sprints, with latency optimization as a core engineering focus from day one.