Building Enterprise AI Voice Agents with LiveKit, Pipecat and FreeSWITCH: Architecture, Integration and Development

Category: AI Voice & Telecom Engineering

By UnifiedPBX Engineering Team | VoIP, PBX, UCaaS, CCaaS and AI voice infrastructure

How to combine real-time media infrastructure, conversational AI pipelines and carrier-grade telephony to build scalable, multi-tenant AI voice platforms.

The development of AI-powered voice applications is changing how businesses communicate with customers. Traditional IVR systems, rule-based call routing and conventional contact-center solutions are increasingly being complemented by conversational AI agents capable of understanding speech, responding naturally, handling interruptions and interacting with enterprise systems.

However, building a production-ready AI voice agent involves much more than connecting a speech-to-text API to a large language model.

A complete solution requires real-time audio streaming, media transport, SIP and PSTN connectivity, speech processing, conversational orchestration, session management, monitoring, security and scalable backend infrastructure.

Three technologies are particularly relevant when designing these systems:

• LiveKit: Real-time WebRTC infrastructure and media routing.
• Pipecat: A framework for orchestrating real-time conversational AI pipelines.
• FreeSWITCH: A powerful telephony and media server for SIP, PSTN, PBX and contact-center applications.

These technologies solve different engineering problems. When integrated correctly, they can form the foundation of enterprise-grade AI voice-agent platforms.

At UnifiedPBX, we work across VoIP, PBX, UCaaS, CCaaS, AI voice automation and multi-tenant communication platforms. This article explores the architecture, integration considerations and engineering decisions involved in building production-ready systems using these technologies.

1. Understanding the Architecture of an AI Voice Agent

A modern AI voice agent typically consists of several independent components working together.

When a customer speaks, the system must capture the audio, transport it to an AI processing pipeline, recognize the speech, understand the intent, generate a response and return synthesized audio with minimal delay.

A typical interaction follows this sequence:

1. A customer connects through a browser, mobile application, SIP phone or PSTN telephone number.
2. The media infrastructure receives and transports audio.
3. The audio processing pipeline performs speech recognition and conversation management.
4. An LLM processes the user's request and determines the response or action.
5. A text-to-speech service generates audio.
6. The response is streamed back to the customer.
7. The backend records events, updates conversation state and triggers business workflows.

This architecture requires multiple technologies rather than a single framework.

The important engineering decision is to select the right technology for each responsibility.

2. What Is LiveKit? Real-Time Media Infrastructure for AI Voice Applications

LiveKit is an open-source real-time communication platform built around WebRTC.

It provides a Selective Forwarding Unit (SFU) that routes media between participants in real-time communication sessions.

In AI voice applications, LiveKit can provide the media infrastructure required for browser-based voice agents, audio rooms and real-time communication between users and AI participants.

Key LiveKit capabilities

• WebRTC-based audio and video communication.
• Real-time media routing through an SFU.
• Rooms and participant management.
• Audio and video track management.
• Secure participant authentication using access tokens.
• Self-hosted deployment and managed cloud options.
• Integration with AI voice-agent frameworks.
• Horizontal scaling for distributed deployments.

Why use LiveKit for AI voice agents?

Consider a customer interacting with an AI assistant through a website.

The browser captures microphone audio and publishes it into a LiveKit room. An AI agent joins the same room, receives the customer's audio and processes it through an AI pipeline.

The agent then publishes synthesized speech back into the room.

This architecture avoids building the complete WebRTC media layer from scratch.

LiveKit deployment options

Organizations can choose between:

LiveKit Cloud

A managed infrastructure option where the provider handles the LiveKit server infrastructure.

Self-hosted LiveKit Community Edition

Organizations deploy and manage the open-source LiveKit server on their own infrastructure.

Self-hosting provides control over deployment, networking, configuration and infrastructure ownership. However, production operation also requires responsibility for TLS, firewall rules, media ports, monitoring, backups and capacity planning.

For organizations building their own communication platforms, self-hosted LiveKit can be an important infrastructure component.

Official documentation: https://docs.livekit.io/

3. What Is Pipecat? A Framework for Real-Time Conversational AI

Pipecat is an open-source Python framework designed to build real-time voice and multimodal conversational AI applications.

Rather than implementing every audio-processing and AI integration manually, developers can compose a conversation using reusable processing components.

Pipecat uses a frame-based pipeline architecture.

A typical voice-agent pipeline can contain:

• Audio input transport.
• Voice activity detection (VAD).
• Speech-to-text processing.
• Conversation context management.
• LLM inference.
• Function calling and external tools.
• Text-to-speech generation.
• Audio output transport.

Understanding the Pipecat pipeline

A simplified conversational pipeline looks like this:

User Audio
    |
    v
Input Transport
    |
    v
Voice Activity Detection
    |
    v
Speech-to-Text
    |
    v
Conversation Context
    |
    v
Large Language Model
    |
    v
Text-to-Speech
    |
    v
Output Transport
    |
    v
User Audio Response

Each component performs a specific processing function.

The pipeline architecture makes it possible to change speech providers, LLMs, transports and other processing components without redesigning the entire application.

Important Pipecat capabilities

• Frame-based real-time processing.
• Modular speech and AI service integrations.
• WebRTC and WebSocket transports.
• LiveKit transport integration.
• Voice activity detection.
• Turn detection and interruption handling.
• LLM context and function calling.
• Text-to-speech and speech-to-text integration.
• Multimodal conversational workflows.

Why Pipecat matters

Different AI applications require different processing pipelines.

For example:

• A customer-support agent may require speech recognition, knowledge retrieval and tool execution.
• A booking agent may require calendar integration and structured conversation flows.
• A contact-center agent may require call transfers, customer identification and CRM integration.
• A healthcare appointment assistant may require controlled workflows and appropriate data protection.

Pipecat provides a framework for composing these processing stages into an application-specific conversational system.

Official documentation: https://docs.pipecat.ai/

4. LiveKit vs Pipecat: Understanding Their Different Responsibilities

LiveKit and Pipecat are not direct replacements for each other.

They operate at different architectural layers.

A practical architecture can use LiveKit to manage real-time media while Pipecat processes audio and coordinates the AI conversation.

This separation makes it possible to select media infrastructure and AI orchestration independently.

5. Where Does FreeSWITCH Fit in an AI Voice Architecture?

LiveKit and Pipecat do not eliminate the need for traditional telephony infrastructure in applications that must communicate with telephone networks.

This is where FreeSWITCH becomes important.

FreeSWITCH is an open-source telephony platform capable of handling SIP signalling, RTP media, call routing, IVR, conferencing and other communication functions.

It is widely used in PBX, UCaaS, CCaaS, SIP trunking and communication service provider environments.

FreeSWITCH responsibilities

• SIP registration and signalling.
• Inbound and outbound call routing.
• SIP trunk integration.
• RTP media handling.
• IVR and dialplan execution.
• Call recording.
• Conference management.
• Call queues and extensions.
• Event Socket Library (ESL) integration.
• Integration with external applications.

Why FreeSWITCH remains relevant

A browser-based AI voice agent and a telephone-based AI voice agent have different connectivity requirements.

A browser application can connect directly through WebRTC.

A PSTN customer, however, calls through a telephone number provided by a carrier or SIP trunk provider.

FreeSWITCH can provide the telephony edge for these calls and connect them to an AI media and processing environment.

For example:

PSTN Customer
     |
     v
SIP Trunk Provider
     |
     v
FreeSWITCH
     |
     v
Media Bridge / SIP Integration
     |
     v
LiveKit Room
     |
     v
AI Voice Agent
     |
     v
STT / LLM / TTS

The exact media bridge depends on the selected integration. LiveKit provides a SIP server for bringing SIP participants into LiveKit rooms, while other architectures can use custom media gateways or application-controlled bridging.

It is important not to assume that FreeSWITCH automatically connects to a LiveKit room. SIP routing, media compatibility, codec negotiation and call-control integration must be designed and tested.

6. Combining LiveKit, Pipecat and FreeSWITCH

An enterprise voice-agent platform may need to support several communication channels simultaneously.

For example:

• Website voice assistant.
• Mobile application voice assistant.
• SIP desk phones.
• Inbound PSTN calls.
• Outbound automated calling.
• Contact-center agent assistance.
• AI-powered IVR.

Using separate application logic for every communication channel creates unnecessary complexity.

A better architecture separates media transport, telephony connectivity, AI processing and application management.

Reference architecture

                 CUSTOMER CHANNELS
                         |
          +--------------+--------------+
          |              |              |
        Browser        Mobile         PSTN/SIP
          |              |              |
          v              v              v
       WebRTC         WebRTC        FreeSWITCH
          |              |              |
          +--------------+--------------+
                         |
                         v
                    LIVEKIT SFU
                         |
                         v
                 PIPECAT AI RUNTIME
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
            STT         LLM         TTS
             |           |           |
             +-----------+-----------+
                         |
                         v
                  AGENT RESPONSE
                         |
                         v
                 CUSTOMER CHANNEL

The application control plane operates alongside this media and AI processing architecture.

It manages tenants, users, agents, permissions, conversation configurations, usage, integrations and business workflows.

Why this separation is important

A scalable platform should not force all responsibilities into a single backend service.

For example:

• LiveKit manages real-time media sessions.
• FreeSWITCH manages traditional telephony.
• Pipecat manages conversational AI processing.
• The application backend manages business logic and tenant configuration.

This allows individual components to scale independently.

7. Designing the Go Backend and Python AI Runtime

A common enterprise architecture uses a Go backend for application services and a separate Python runtime for AI processing.

This is particularly useful when the main platform is written in Go but requires Python-based AI frameworks such as Pipecat.

The two services communicate through well-defined APIs and asynchronous events.

Go backend responsibilities

A Go backend can manage:

• Authentication and authorization.
• Multi-tenant account management.
• Agent configuration.
• Conversation templates.
• SIP and PBX configuration.
• Call lifecycle management.
• Usage metering.
• Billing and subscriptions.
• Webhooks and integrations.
• API access and administration.

Python AI runtime responsibilities

A separate Python runtime can manage:

• Pipecat pipelines.
• Speech recognition.
• LLM interactions.
• Text-to-speech generation.
• Audio streaming.
• Conversation context.
• Tool execution.
• Interruption handling.
• Agent session processing.

Communication between services

The services can communicate through:

REST APIs

Suitable for configuration, session creation and administrative operations.

WebSockets or real-time messaging

Useful for live events, status updates and interactive control.

RabbitMQ

Useful for asynchronous tasks, event processing and distributed workflows.

Redis

Useful for transient state, caching, session coordination and selected real-time workloads.

PostgreSQL

Suitable for persistent tenant, agent, configuration, conversation and usage data.

A key design principle is to avoid using a message queue for every real-time audio frame. Audio should follow an appropriate low-latency media transport, while messaging systems handle control events and asynchronous business operations.

8. Multi-Tenant AI Voice Agent Architecture

Building a single AI voice assistant is relatively straightforward compared with operating a multi-tenant SaaS voice platform.

An enterprise platform may serve hundreds or thousands of customers, each with separate:

• Agents.
• Knowledge bases.
• Phone numbers.
• SIP trunks.
• Conversation templates.
• Credentials.
• Usage limits.
• Billing policies.
• Access permissions.

The platform must enforce tenant isolation across every layer.

Example tenant structure

EAIBAP Platform
      |
      +-- Tenant A
      |      |
      |      +-- Agent 1
      |      +-- Agent 2
      |      +-- Knowledge Base
      |      +-- Phone Numbers
      |
      +-- Tenant B
      |      |
      |      +-- Agent 1
      |      +-- Agent 2
      |      +-- Knowledge Base
      |      +-- Phone Numbers
      |
      +-- Tenant C
             |
             +-- Agents
             +-- Workflows
             +-- Integrations

Each conversation must be associated with the correct tenant, agent and session.

The backend should validate tenant ownership before issuing LiveKit access tokens or dispatching an AI agent.

Scaling considerations

For larger deployments, the architecture may require:

• Multiple AI worker instances.
• Distributed session management.
• Queue-based job dispatch.
• Tenant-aware resource allocation.
• Per-tenant concurrency limits.
• Horizontal scaling.
• Centralized logs and metrics.
• Failure recovery and retry handling.

LiveKit supports distributed deployments using Redis for coordination between nodes. Redis should be configured according to the deployment's requirements rather than treated as a universal shared cache.

9. Building Natural Voice Conversations: Latency and Interruption Handling

Low latency is one of the most important requirements of conversational voice AI.

A system may be technically functional but still provide a poor user experience if every response takes several seconds.

A typical voice interaction involves several processing stages:

1. Audio capture.
2. Network transmission.
3. Voice activity detection.
4. Speech recognition.
5. LLM inference.
6. Tool execution, when required.
7. Speech synthesis.
8. Audio transmission.

The combined delay determines the perceived responsiveness.

Important optimization areas

Streaming speech recognition

Process partial speech results instead of waiting for the entire utterance whenever the selected provider supports it.

Streaming LLM responses

Start generating output as soon as sufficient conversational context is available.

Streaming text-to-speech

Begin audio generation before the entire response has been completed.

Voice activity detection

Identify when a user starts and stops speaking.

Barge-in and interruption handling

When a customer interrupts an AI response, the system should stop or cancel the current response, update conversation state and prioritize the new user input.

Echo cancellation and audio processing

Browser and telephony audio paths have different acoustic and codec requirements. These should be tested separately.

Telephone audio considerations

Traditional telephone networks frequently use narrowband audio such as G.711 at 8 kHz.

AI speech services may use different sample rates and audio formats.

A production integration must correctly handle:

• Codec negotiation.
• Sample-rate conversion.
• Mono and channel configuration.
• Audio frame sizes.
• RTP timing.
• Packet loss and jitter.
• Audio buffering.

A WebRTC browser test does not automatically prove that the same agent will perform correctly on an 8 kHz SIP telephone call.

Both environments need independent testing.

10. AI Model and Voice Service Integration

An enterprise AI voice platform should avoid unnecessary dependence on a single AI provider.

The architecture should allow different providers to be selected according to language support, latency, quality, reliability and cost.

Speech-to-text

Examples include:

• Deepgram.
• Google Cloud Speech-to-Text.
• Azure Speech.
• OpenAI speech recognition services.

Large language models

Examples include:

• OpenAI models.
• Anthropic Claude.
• Google Gemini.
• Self-hosted or open-weight language models.

Text-to-speech

Examples include:

• Cartesia.
• ElevenLabs.
• Azure Neural TTS.
• Google Cloud Text-to-Speech.

Provider selection should be based on actual workload requirements and measured performance.

Cost considerations

A production voice agent has several cost components:

• Speech-to-text processing.
• LLM input and output tokens.
• Text-to-speech generation.
• Real-time media infrastructure.
• Telephony minutes.
• Storage and recording.
• Monitoring and observability.
• Infrastructure and network usage.

A realistic cost-per-minute model should include all these components.

It is not sufficient to calculate only the LLM cost.

11. Self-Hosted LiveKit vs Managed Infrastructure

Organizations need to decide whether to operate their own media infrastructure or use a managed service.

Self-hosting can be attractive for organizations requiring infrastructure control, custom networking or specific data-management arrangements.

Managed services can reduce operational responsibility and simplify early-stage development.

The correct choice depends on traffic, team expertise, operational requirements, geographic distribution and total cost of ownership.

Official self-hosting documentation: https://docs.livekit.io/transport/self-hosting/

12. Why EAIBAP Is an Interesting Reference Architecture

EAIBAP represents an example of an application-level platform designed around multi-tenant AI voice-agent management.

Its architecture separates application management from the underlying communication infrastructure.

The current implementation uses:

• Go for the application backend.
• PostgreSQL for persistent data.
• Redis for application state and caching.
• RabbitMQ for asynchronous messaging.
• FreeSWITCH as the configured telephony adapter.
• LiveKit as a separately deployed media-plane component.

The LiveKit deployment uses its own Redis instance for LiveKit-specific state, separate from the application's Redis service.

This separation is important because application caching and distributed media coordination have different operational responsibilities.

EAIBAP architecture direction

                  EAIBAP PORTAL
                        |
                        v
                  GO API BACKEND
                        |
             +----------+----------+
             |          |          |
             v          v          v
         PostgreSQL   Redis     RabbitMQ
                        |
                        v
                MEDIA / TELEPHONY
                        |
             +----------+----------+
             |                     |
             v                     v
          LiveKit              FreeSWITCH
             |                     |
             +----------+----------+
                        |
                        v
                AI AGENT RUNTIME
                        |
                        v
                 STT / LLM / TTS

A Pipecat-based Python runtime can be evaluated as a separate AI processing service and connected through the supported LiveKit transport.

This is an architectural integration direction, not a claim that Pipecat has already been deployed in EAIBAP production.

Why this architecture matters

The goal is not to build another single-purpose voice bot.

The objective is to establish a reusable platform where organizations can configure, operate and scale multiple AI voice agents across different business applications.

Possible applications include:

• Customer support automation.
• Appointment booking.
• Hotel and travel reservations.
• Healthcare appointment assistance.
• Lead qualification.
• Sales automation.
• Order-status enquiries.
• Enterprise helpdesks.
• Contact-center automation.
• Automated outbound calling.

The platform architecture should allow each tenant to configure its agents, workflows, integrations and communication channels without requiring a separate application deployment.

13. When Should You Hire a LiveKit Developer, Pipecat Developer or FreeSWITCH Consultant?

These technologies require different areas of engineering expertise.

Choosing the right developer or consultant depends on the actual problem.

Hire a LiveKit developer when you need:

• Self-hosted LiveKit deployment.
• WebRTC application development.
• Real-time audio and video infrastructure.
• LiveKit room and participant management.
• SIP integration with LiveKit.
• Media server scaling and monitoring.
• Real-time AI agent connectivity.

Hire a Pipecat developer when you need:

• Custom real-time AI voice pipelines.
• STT, LLM and TTS integration.
• Conversational agent development.
• LiveKit transport integration.
• Barge-in and interruption handling.
• Function calling and external tools.
• Custom multimodal AI workflows.

Hire a FreeSWITCH consultant when you need:

• SIP and RTP integration.
• Multi-tenant PBX development.
• SIP trunking.
• IVR and call routing.
• FreeSWITCH ESL integration.
• Contact-center solutions.
• AI-powered telephony integration.

When you need all three

For an enterprise platform connecting browser users, telephone networks and AI agents, the engineering work may span all three technologies.

In such projects, architecture design is particularly important.

The media layer, telephony layer, AI runtime and business backend must be designed to work together without introducing unnecessary latency, duplicate session management or operational complexity.

14. How UnifiedPBX Can Help

UnifiedPBX provides software engineering and consulting services across VoIP, PBX, UCaaS, CCaaS and AI-powered communication platforms.

Our engineering focus includes:

LiveKit Development and Consulting

• Self-hosted LiveKit infrastructure.
• WebRTC application integration.
• Media-plane architecture.
• SIP and telephony integration.
• Real-time communication systems.

Pipecat and AI Voice Agent Development

• Real-time conversational pipelines.
• STT, LLM and TTS integration.
• Voice-agent orchestration.
• Custom business workflows.
• AI service integration.

FreeSWITCH Development and Consulting

• Multi-tenant PBX architecture.
• SIP trunk and carrier integration.
• IVR and call routing.
• ESL-based application development.
• Contact-center and communication platforms.

Enterprise AI Voice Platform Engineering

• Multi-tenant SaaS architecture.
• Go and Python backend integration.
• Redis and RabbitMQ-based distributed processing.
• PostgreSQL data architecture.
• Monitoring, deployment and infrastructure engineering.

Whether you are building a new AI voice-agent product, modernizing an existing PBX, integrating WebRTC with SIP or developing an enterprise conversational AI platform, we can help evaluate the architecture and implement the required components.

Looking for a LiveKit developer, Pipecat developer or FreeSWITCH consultant?

Contact UnifiedPBX to discuss your architecture, integration requirements and development scope.

Enquiry: contact form · ../index.html

Frequently Asked Questions

1. What is the difference between LiveKit and Pipecat?

LiveKit provides real-time communication and media infrastructure, while Pipecat provides a framework for building and orchestrating conversational AI pipelines. They can be integrated to build real-time AI voice agents.

2. Can Pipecat work with LiveKit?

Yes. Pipecat provides a LiveKit transport that allows audio and other real-time communication data to flow between LiveKit rooms and Pipecat pipelines.

3. Can FreeSWITCH integrate with LiveKit?

Yes. SIP-based telephony can be connected to LiveKit using its SIP infrastructure or a suitable media-bridging architecture. The integration requires appropriate SIP signalling, media routing and codec configuration.

4. Do I need FreeSWITCH if I already use LiveKit?

Not necessarily. If your application only requires browser-based WebRTC communication, LiveKit may be sufficient. FreeSWITCH becomes relevant when you need traditional SIP telephony, PBX functions, carrier connectivity or advanced call routing.

5. Do I need Pipecat to build a LiveKit AI voice agent?

No. LiveKit provides its own agent framework, LiveKit Agents. Pipecat is an alternative framework for orchestrating conversational AI pipelines. The choice depends on programming language, architecture and application requirements.

6. Can a Go backend work with Pipecat?

Yes. A Go backend can manage application APIs, tenant configuration, authentication and business workflows while a separate Python service runs Pipecat pipelines. They can communicate through APIs, events and messaging systems.

7. Can LiveKit be self-hosted?

Yes. LiveKit provides an open-source server that organizations can deploy on their own infrastructure. Production deployments require appropriate networking, TLS, firewall configuration, media-port accessibility and operational monitoring.

8. Can these technologies support multi-tenant AI voice platforms?

Yes, when the application is designed with tenant isolation, session ownership, access control, media resource management, concurrency limits and appropriate scaling mechanisms.

9. What is the difference between a traditional IVR and an AI voice agent?

A traditional IVR usually follows predefined menus and routing rules. An AI voice agent can understand natural-language requests, maintain conversational context, invoke external tools and generate dynamic responses.

10. How can I hire a LiveKit, Pipecat or FreeSWITCH developer?

Use the contact form with your application requirements, preferred infrastructure, expected call volume and integration needs. An initial technical assessment can help establish the architecture and development scope.

Need a LiveKit developer, Pipecat consultant or FreeSWITCH integration partner? Send your architecture brief via our contact form — we respond with scope, timeline and integration options.

Conclusion

The next generation of enterprise communication platforms will increasingly combine real-time media infrastructure, conversational AI and traditional telephony.

LiveKit, Pipecat and FreeSWITCH each address a different part of this architecture.

LiveKit provides the real-time media foundation. Pipecat enables modular conversational AI processing. FreeSWITCH connects applications to traditional telephony infrastructure.

When combined with a scalable application backend, these technologies can support sophisticated AI-powered communication platforms.

The engineering challenge is not simply connecting APIs. It is designing a reliable, low-latency, secure and scalable system that can operate in real production environments.

UnifiedPBX helps businesses design and develop these communication systems, from SIP and PBX infrastructure to real-time AI voice agents and multi-tenant SaaS platforms.

If you are planning a LiveKit, Pipecat, FreeSWITCH or enterprise AI voice-agent project, contact UnifiedPBX to discuss your requirements.

Author: UnifiedPBX Engineering Team\

Website: ../index.html\

Article focus: LiveKit development, Pipecat development, FreeSWITCH integration and enterprise AI voice-agent architecture.