What is an AI voice agent?

An AI voice agent is a telephony system that answers or places a real phone call, transcribes the caller, reasons with a language model and tools, then speaks back — until policy says transfer, voicemail or hangup. It is not a chatbot with a microphone.

How is an AI voice agent different from a chatbot?

A chatbot owns text. An AI calling agent owns a SIP session: answer time, RTP, barge-in, STT lag, TTS pacing and a transfer the PBX can actually execute. Browser-only demos fail the first time a trunk returns 487 or a caller interrupts.

What is VAD in an AI voice agent?

Voice activity detection decides when the caller is speaking. Combined with barge-in it stops TTS, cancels the in-flight LLM turn and starts listening. That is call-state, not a smarter prompt.

How does STT work on a live SIP call?

Audio is streamed from the media bridge into speech-to-text. Partial transcripts feed the agent before the caller finishes the sentence. Batch record-then-transcribe is too slow for conversation. Production work in this practice has used streaming STT such as Whisper on a VoIP/RTP path.

Can the agent speak with a natural voice?

Yes. Streaming neural TTS is paced back onto the RTP stream so the caller hears a continuous voice, not a dumped clip. Voice, rate and language follow the tenant. Public production reviews of this practice include neural TTS such as Inworld on the same pipeline as Whisper STT.

Should we use Vapi or Retell instead of owning SIP?

Hosted agent UIs are useful for a prototype. Production still needs a SIP/PBX backbone for trunks, queues, recording, transfer and policy. This case study is that backbone with the model inside the agent runtime, not owning hangup.

How does an engagement like this start?

Send the current PBX, inbound vs outbound, languages, the one workflow that must complete on the first call, CRM/API list, recording constraints, and where humans take over. The first reply is an architecture review.

AI Voice Agent Case Study | SIP, VAD, STT, LLM & Production Handoff

Q: Does this AI voice agent support multiple languages?

Language is a routing key: STT locale, TTS voice, barge-in timing and DTMF fallback. This case is the SIP agent architecture. Hebrew, Yiddish and English IVR proof lives on the multilingual AI voice and IVR case study — a related product, not the same stack.

Industry: AI calling / conversational telephony
Type: Inbound + outbound AI voice agent on SIP
Role: Voice architecture + media path + agent runtime
Speech path: VAD · streaming STT · LLM + tools · natural TTS
Stack: FreeSWITCH / Asterisk · SIP · NestJS · CRM APIs
Proof: Live PSTN path; model requests, PBX executes

The problem

The brief was an AI voice agent that could take inbound DIDs and outbound campaign calls until policy said transfer, voicemail or hangup. A chatbot with a microphone was out of scope. Browser-only demos were out of scope.

Constraints

Delivery stayed inside the client's stack, tenancy, compliance and media-ownership boundaries. Where a choice was forced (suite vs owned plane, BYOC vs CPaaS, local vs cloud models), the architecture section below records the trade-off rather than a marketing rewrite.

Most failures in this class are not prompts. They are 487s, barge-in that never fires, 800ms STT lag, a TTS clip dumped onto RTP, and a model that tries to transfer a call it does not own.

Constraints that actually mattered

Real SIP / PSTN, not WebRTC-only
Caller can interrupt the agent (VAD + barge-in)
Streaming speech-to-text, not record-then-transcribe
Tool calls into CRM, calendar and tickets — after identity
Natural voice back on the same media path
Transfer and hangup stay on the PBX, not in the LLM

Business requirements

One production path: the phone rings, the agent answers in under a second, the conversation feels turn-based, and the system either completes a job or hands off cleanly. Appointment booking, lead qualification, customer support and call routing are workflows on that path — not a second product.

The solution

The PBX answers SIP. A media bridge turns RTP into audio frames. VAD decides who is speaking. Streaming STT feeds an agent runtime. The runtime calls tools and RAG. Neural TTS is paced back to the caller. Transfer, hangup, recording and DNC stay on the call-control plane.

Keep the language model inside the agent runtime. The model may request “transfer to sales.” The PBX executes that after policy checks. That is the difference between an AI calling agent and a demo that sounds good until the first live trunk.

The same rules are written as architecture notes in How to build an AI voice agent with SIP. This case study is the shipped platform those notes describe.

How the system flows

The PBX owns the call. Audio becomes frames, speech becomes text, the agent runtime decides — TTS streams back without giving the model SIP control.

flowchart TD
  Call[Inbound or outbound SIP] --> PBX[FreeSWITCH or Asterisk]
  PBX --> RTP[RTP to media bridge]
  RTP --> VAD[VAD barge-in]
  VAD --> STT[Streaming STT]
  STT --> LLM[LLM tools RAG]
  LLM --> TTS[Neural TTS stream]
  TTS --> RTP
  LLM -->|transfer request| PBX

Live call media path — model requests, PBX executes hangup and transfer.

Architecture

Phone call in, natural voice out, with speech and intelligence as services on the media path — not as the owner of SIP.

flowchart TB
  SIP[SBC SIP trunk] --> PBX[PBX]
  PBX --> Media[Audio media bridge]
  Media --> VAD[VAD]
  Media --> STT[STT]
  STT --> Agent[LLM agent runtime]
  Agent --> APIs[CRM calendar tickets]
  Agent --> TTS[TTS]
  TTS --> Media

Production AI voice agent stack.

Key components

Telephony

FreeSWITCH or Asterisk answers, parks, records and transfers. The AI agent is a destination on the dialplan, with a human queue as fallback.

Media bridge

RTP to audio frames and back. Clock, codec and barge-in live here — not in the prompt.

Agent runtime

NestJS / Node holds state, tools and RAG. OpenAI, Claude or Gemini reason; they do not own SIP.

Speech pair

Streaming STT in, natural voice TTS out. Language, voice and endpointing are configuration, not improvisation.

VAD and barge-in

Voice activity detection is how the platform knows the caller started talking. Without it the agent talks over the customer, or waits forever after a pause.

Barge-in — stop TTS immediately, cancel the in-flight LLM turn, start listening.
Endpointing — too aggressive cuts slow or accented speech; too loose creates dead air.
State — VAD is wired to call-state on the media bridge, not to a “be polite” instruction in the prompt.

If the caller cannot interrupt, it is not a production AI voice agent. It is a prompt playing a file.

Streaming STT

Speech-to-text on a live call has to be streaming. Batch “record then transcribe” is too slow for turn-taking. Partial hypotheses go to the agent so it can start planning before the sentence ends.

Language and possible mid-call switch
Numeric / phone-number biasing so digits are not guessed as words
A confidence floor — below it, repeat or offer keypad fallback
Transcript logging as a compliance decision, not a debug default

Public production work in this practice documents Whisper STT on a VoIP / RTP voice-AI pipeline (alongside Anthropic LLM). This page does not add engines that were not part of that documented path.

STT lag is a product bug. An 800ms stall on a live trunk feels like the line dropped. Metrics on the bridge include STT lag, not only “did the model answer.”

Language support

Language is not a prompt line. It is a routing key that selects STT locale, TTS voice, barge-in timing and whether DTMF is the fallback.

Locale on the session, not guessed every turn
STT language matched to that locale (and a switch policy if the caller changes language)
TTS voice in the same language, with slower read-back for names, IDs and numbers
Barge-in / silence tuned so endpointing does not clip the locale you just selected

This case is the SIP agent architecture. Hebrew, Yiddish and English telephone IVR — recorded prompts, PIN, registration — is documented on the related Hebrew & Yiddish IVR case. That is a sibling product, not a claim that this LLM agent was that IVR.

Natural voice TTS

The caller judges the product by how it sounds. Streaming neural TTS is paced onto the RTP stream so the voice is continuous. Dumping a four-second clip without packet pacing sounds like a broken robot.

Stream the first sentence as soon as it is ready
Match voice, speaking rate and language to the tenant
Confirm names, IDs and phone numbers slowly rather than fluent-guessing them
Stop playback on barge-in; do not finish the paragraph over the caller

Public production reviews of this practice include Inworld TTS on the same VoIP / Whisper / Anthropic pipeline. Natural voice here means that class of neural TTS on a real media path — not a hosted demo widget.

Engineering challenges

SIP vs demo — inbound DID and outbound campaign on a trunk, not a browser sandbox.
Barge-in — VAD plus cancel of TTS and the LLM turn.
Latency budget — STT lag, time-to-first-byte TTS, dead air over 1.5s.
Tool safety — CRM writes only after identity; irreversible telephony stays off the model.
Language — locale, STT, TTS and barge-in as one key.
Handoff — warm transfer with context, not a dead-air drop onto a queue.

How those were solved

Call control stays on FreeSWITCH or Asterisk. The agent runtime requests actions. Policy checks transfer, hangup, recording and DNC before the PBX executes them. Conversation state (workflow step, slots, retry count, whether a human is already ringing) lives outside the model so a timeout does not lose the call.

The same split is why hosted “Vapi or Retell” UIs are the wrong first question. Layer 1 is the voice. Layer 2 is the SIP backbone. This platform is Layer 2 with a tool-using agent attached. See Vapi or Retell is the wrong question.

My role

Voice architecture + SIP media path + agent runtime. Architecture, media bridge, STT/TTS wiring, tool APIs, transfer policy and latency work. Confidential client names, DIDs and recordings stay off this page.

Voice architectureSIP media pathVAD / barge-inSTT / TTS wiringAgent runtimeTool APIsTransfer policyLatency work

Technology stack

FreeSWITCH / AsteriskSIPVADSTTLLM + toolsRAGOpenAIClaudeGeminiNatural voice TTSNestJS / NodeCRM APIs

These technologies are the documented stack for this platform. Whisper STT, Anthropic LLM and Inworld TTS appear in public production reviews of the same voice-AI pipeline class. This page does not add engines that were not part of that work.

Production considerations

Real SIP trunk testing — not only web RTC demos.
Tool calls and hangup authority defined per tenant.
Recording and PCI boundaries for payment over voice if enabled.

Engineering outcomes

No invented conversion rates or uptime claims. What this system actually established:

A live PSTN / SIP path for inbound and outbound AI calling.
VAD barge-in, streaming STT and natural voice TTS on one media bridge.
Tool calls (CRM, calendar, tickets) without giving SIP to the LLM.
Warm transfer to human queues when policy says the agent is done.

Screenshots and diagrams

Representative AI voice agent operations console for SIP calling, STT and natural voice — Product surface used on this site for the AI voice agent control plane — representative UI, not a client screenshot.

Architecture on this page is the HTML diagram above. The same pipeline is explained in How to build an AI voice agent with SIP and contrasted with menu IVR in Traditional IVRs are dying.

Architectural insight

An AI voice agent is a telephony product that happens to use a language model. Media, state and policy come first. The model is second. If transfer and hangup live in the prompt, the first 487 or the first interruption will prove it.

FAQ — buyer & architecture questions

Ten common questions about this case study, fit, and engagement.

Documented barge-in, multilingual fallback, tool use or telephony integration—not a demo webhook only.

No. Asterisk/FreeSWITCH handles media; AI orchestration sits beside call control.

Yes. Narrow IVR replacement or receptionist is a common first phase.

Retention and redaction policies are set per project; we implement access control and storage limits.

Architecture can route providers; latency and data residency drive the choice.

Call flows, CRM, languages, compliance rules and example calls that fail today.

Share the current stack (PBX, CRM, cloud), the failure or goal in one paragraph, peak call volume, carriers, and any deadline. Screenshots, a pcap, or a short Loom beat a 40-page RFP. Use the project brief

Yes. Delivery is remote-first from Delhi with scheduled overlap for EU, UK, US and APAC stand-ups. Production changes use written runbooks, rollback steps and agreed maintenance windows.

Yes. Many engagements begin with a 1–2 week SIP trace review, tenant-isolation audit, or architecture assessment. If the fit is good, scope expands from evidence—not from a generic sales deck.

Founders, CTOs, telecom leads, MSPs and product teams building or fixing UCaaS, CCaaS, CRM+voice, AI voice, or vertical SaaS—not buyers who only need seats on a mass-market suite.

Building an AI voice agent on SIP?

Share the workflow, current PBX, languages and where humans take over. The first reply is whether the gap is media, STT, the model or call control.

Discuss Your Project