How to Build an AI Voice Agent with SIP

Most “AI voice” demos are a chatbot with a microphone. A production agent is a telephony system that happens to use a language model. The difference shows up the first time a caller interrupts, a SIP trunk returns 487, STT lags by 800ms, or the model tries to transfer a call it does not control.

This article is the architecture I use when connecting an AI agent to real phone calls: inbound DID, outbound campaign, or a queue that can hand off to a human.

What the caller actually experiences

From the caller’s side the product is simple: the phone rings, someone answers in under a second, the conversation feels turn-based, and the system either completes a job or transfers cleanly.

From the platform side that experience is a pipeline:

1. A SIP INVITE arrives at the PBX or SBC.
2. The call is answered and media (RTP / SRTP / WebRTC) starts flowing.
3. Audio is streamed to speech-to-text with voice-activity detection.
4. Partial transcripts feed an agent that already knows the tenant, caller, and workflow.
5. The agent may call business APIs, then return text for speech synthesis.
6. TTS audio is paced back onto the media path without clipping the caller.
7. If confidence drops, policy says transfer, voicemail, callback, or hangup — not “the model will figure it out.”

Reference architecture

Caller / PSTN / WebRTC
        │
        ▼
   SBC / SIP trunk
        │
        ▼
 PBX (FreeSWITCH / Asterisk / FusionPBX)
   • answer, park, transfer, record
   • tenant / DID / queue routing
        │
        ▼
 Realtime media bridge
   • RTP ↔ audio frames
   • barge-in / VAD / jitter buffer
        │
        ├──► STT (streaming)
        ├──► Agent runtime (LLM + tools + state)
        ├──► TTS (streaming)
        └──► Business APIs (CRM, calendar, orders, tickets)
        │
        ▼
 Call-control plane (not the LLM)
   • transfer, hangup, recording, DNC, emergency

Keep the language model inside the agent runtime. Do not let it own SIP. The model can request “transfer to sales queue”; the PBX executes that request after policy checks.

1. Telephony edge: SIP, DID and the PBX

Start with a boring, correct SIP stack. Inbound numbers terminate on an SBC or carrier trunk, then land on FreeSWITCH, Asterisk or FusionPBX. That layer already knows how to:

• authenticate trunks and registrations
• route DIDs to a tenant, IVR, queue or agent
• handle early media, re-INVITE, hold and codec negotiation
• record, park, blind/attended transfer and conference
• fail over when a carrier or media node dies

The AI agent is just another destination. A typical pattern is: DID → tenant dialplan → “AI queue” → media websocket / ESL / AMI / ARI originator that attaches the realtime bridge. If the agent is unavailable, the same dialplan should already have a human queue or voicemail. That fallback is a telephony rule, not a prompt.

WebRTC callers (softphone, click-to-call) join the same architecture after DTLS-SRTP and ICE succeed. Do not build a second “browser-only” agent that cannot later take PSTN traffic.

2. Media path: the part demos skip

Signaling can look perfect while audio is unusable. The bridge has to convert RTP into a stable stream the STT engine accepts, and convert TTS frames back to the codec the far end negotiated (often PCMU/PCMA on trunks, Opus on WebRTC).

Production issues that show up here:

• One-way audio from NAT, wrong ext-rtp-ip, missing ICE/TURN, or AWS security groups that allow SIP but not RTP.
• Clock drift and pacing — dumping a 4-second TTS clip onto the RTP stream without packet pacing sounds like a broken robot.
• Barge-in — if the caller speaks over the agent, you must stop TTS, cancel the in-flight LLM turn, and start listening. That is VAD plus call-state, not a smarter prompt.
• Silence — too-aggressive end-of-turn detection cuts Indian, Arabic or elderly speech; too-loose detection makes the agent wait forever.

Treat the media bridge as its own service with metrics: RTP packet loss, jitter, STT lag, TTS time-to-first-byte, barge-in count, and “dead air” over 1.5s.

3. Speech in: streaming STT

Batch “record then transcribe” is too slow for conversation. Use streaming STT and send partial hypotheses to the agent runtime so it can start planning a response before the caller finishes the sentence.

Configure, do not improvise:

• language and possible language switch mid-call
• numeric / phone-number biasing so “nine four one two…” is not guessed as words
• punctuation and endpointing tuned per locale
• a confidence floor — below it, ask the caller to repeat or offer a keypad fallback

STT is also where PII appears first. If you record or log transcripts, that is a compliance decision, not a logging convenience.

4. Agent runtime: LLM, tools and state

The model should see a compact working set: system policy, tenant prompt, current workflow step, retrieved CRM facts, and the last few turns. It should not see the entire company handbook on every token.

Give it tools, not freedom:

• lookup_customer, create_ticket, book_slot, get_order_status
• transfer_call, schedule_callback, end_call — which the control plane authorizes
• read-only first; writes only after confirmation when the workflow requires it

Keep conversation state outside the model: workflow id, collected slots (name, phone, date), retry counts, and whether a human is already ringing. When the LLM times out or returns garbage, the state machine still knows what to do next.

Timeouts matter. A 6-second model wait on a live call feels like the line dropped. Cap generation, stream TTS as soon as the first sentence is ready, and have a spoken filler policy that does not ramble.

5. Speech out: TTS back onto the call

Streaming TTS is what makes the agent feel present. Generate and play incrementally. Match voice, speaking rate and language to the tenant. For names, IDs and phone numbers, prefer a slower, confirmed read-back rather than a fluent guess.

After playback, return to listen mode with a clear turn-taking rule. If the caller is silent, a single prompt (“I can help with booking, status or a callback — what do you need?”) is better than three stacked questions.

Telephony controls that must stay outside the model

The LLM is a bad place to put anything irreversible or regulated:

• hangup and transfer (especially to emergency, collections, or a paying queue)
• recording on/off and announcement legalities
• authentication — PIN, OTP, ANI allow-list, or CRM match before account details
• DNC / consent for outbound
• rate limits and fraud (short-duration international hops, traffic pumping)

A practical split: the agent emits an intent (TRANSFER sales_queue). The PBX and application check hours, queue length, tenant config and recording policy, then execute or refuse with a spoken reason.

Business integrations

An agent that cannot touch the business is a talking FAQ. Wire it to the same systems a human agent would use:

• CRM / lead record and last interaction
• calendar or booking inventory
• order, policy, ticket or shipment status
• identity and account flags (VIP, overdue, already-open ticket)
• after-call: disposition, transcript summary, next action

Do the write path carefully. Collect, confirm, then commit. “I have Tuesday at 4pm, shall I book it?” is a workflow step, not a personality trait.

Latency budget

A usable voice turn usually needs to stay near one second of “thinking silence” after the caller stops, and well under that for barge-in stop. Rough budget:

• endpointing / VAD: 200–400ms
• STT finalization: 100–300ms (partials already sent)
• LLM first tokens: 200–500ms
• TTS first audio: 150–300ms

If any one hop is a 2-second round trip, the rest of the stack cannot save the conversation. Stream every hop. Skip extra retrieval when the workflow slot is already filled. Cache tenant prompts. Put the media bridge close to the PBX; put STT/TTS close to the bridge.

Failure handling before you launch

Design the unhappy paths on paper:

• STT down → short apology + DTMF menu or human queue
• LLM timeout → repeat last confirmed state, do not invent a new question
• API 500 → “I can’t see that record right now; I can take a callback.”
• caller abuse / loop → max turns then transfer
• low confidence on a phone number or amount → digit-by-digit confirm
• agent process crash → PBX still owns the call and can re-invite a human

Log correlation ids across SIP Call-ID, media session, STT stream and agent turn. Without that, production debugging is guesswork.

A sane build order

1. Get a SIP call answering with stable two-way audio and a recorded greeting. No AI yet.
2. Attach streaming STT and print transcripts. Tune silence and barge-in.
3. Play canned TTS so pacing and codecs are proven.
4. Add a deterministic state machine (book / status / transfer) with no LLM.
5. Drop the LLM in as the NLU + reply generator behind that state machine.
6. Add one live tool (CRM lookup), then a write tool with confirmation.
7. Only then: multi-language, outbound campaigns, and multi-tenant prompts.

Teams that start at step 5 spend weeks debugging “the model” when the real bug is RTP, endpointing, or a transfer that never hits the queue.

What to send if you want this built

A useful kickoff is: current PBX (FreeSWITCH, Asterisk, FusionPBX, hosted), inbound vs outbound, languages, the one workflow that must complete on the first call, CRM/API list, recording/compliance constraints, and where humans should take over. From there the architecture is specific instead of generic.

Related reading