CONFIDENTIAL PROJECT

Production AI Voice Agent on SIP

Architecture and delivery of an AI voice agent connected to real phone calls: SIP answer, streaming speech, LLM tools, TTS and human transfer — with the model outside call control.

FreeSWITCH / AsteriskSIPSTTLLM + toolsRAGOpenAIClaudeGeminiTTSNestJS / NodeCRM APIs

Confidential client names are omitted. Interfaces are representative product UIs and architecture from systems designed in this practice.

AI voice agent architecture from SIP trunk to STT, LLM tools and TTS

Project overview

The brief was inbound and campaign calls that a model could handle until policy said transfer, voicemail or hangup. Demos that only work in a browser were out of scope.

Business challenge

Most failures are not prompts. They are 487s, barge-in, 800ms STT lag, and a model that tries to transfer a call it does not own.

Solution architecture

PBX answers SIP. A media bridge streams audio to STT. An agent runtime calls tools. TTS returns to the caller. Transfer and hangup stay on the call-control plane.

Keep the language model inside the agent runtime. The model may request “transfer to sales.” The PBX executes that after policy checks.

Production AI Voice Agent on SIP
Caller / PSTNSIP INVITE
SBC / trunkEdge
STTStreaming
Human queueWarm transfer
The model requests actions. The PBX performs them.

Technology stack

FreeSWITCH / AsteriskSIPSTTLLM + toolsRAGOpenAIClaudeGeminiTTSNestJS / NodeCRM APIs

Key features

Real SIP path

Inbound DID and outbound campaign, not WebRTC-only.

Barge-in / VAD

Caller can interrupt the agent.

Tool-using agent

CRM, calendar, tickets — after identity.

Human handoff

Queue transfer with context, not a dead air drop.

My role

Voice architectureSIP media pathAgent runtimeTool APIsTransfer policyLatency work

Implementation

Same architectural rules documented in the AI-voice SIP article on this site: media, state and policy first; model second.

Results / capabilities

  • Live PSTN path
  • Transfer to human queues
  • Tool calls without giving SIP to the LLM

Building something in this class?

Share the workflow, current stack and constraints. The first reply is an architecture-minded review.

Discuss Your Project