Agent Runtime Guide

AI voice agents are transforming how businesses interact with customers, providing natural, conversational experiences through voice interfaces. VideoSDK's Agent Runtime feature offers a powerful no-code/low-code interface that enables you to build sophisticated AI voice agents without extensive programming knowledge.

Prerequisites

Before you begin, ensure you have:

• VideoSDK Account: Visit VideoSDK Dashboard to sign up for a free account and access the AI Agent builder.

Step-By-Step Guide

Step 1: Create a New Agent

1

Create a New Agent

1. In the dashboard, navigate to AI Agent > Agents or visit Agents Dashboard.
2. You'll see the AI Agent > Agents section in the dashboard.
3. To create a voice agent, click on Agents in the sidebar.

Step 2: Click Add New Agent

2

Click `Add New Agent`

This is where you'll start creating your voice agent. If no agent has been created yet, you'll see a Add New Agent button. If agents already exist, you'll see a list of all AI voice agents, and you can click the button in the top right corner to create a new agent.

Step 3: Configure Agent Details

3

Configure Agent Details

This is where you can define your AI voice agent's persona and behavior:

• Agent Name: Set a descriptive name for your agent (e.g., "AI Interviewer").
• System Prompt: Define the agent's role, personality, and behavior guidelines.
• Welcome Message: Set the message that plays when the agent joins a conversation.
• Closing Message: Set the message that plays when the agent leaves a conversation.

Step 4: Configure the Pipeline

4

Configure the Pipeline

The pipeline is the core engine of your voice agent, processing audio through speech recognition, AI reasoning, and text-to-speech. VideoSDK offers two pipeline options: Realtime Pipeline and Cascading Pipeline.

Realtime Pipeline

The Realtime Pipeline provides direct speech-to-speech processing with minimal latency, ideal for natural, conversational interactions.

Example: Adding Gemini Realtime Model

1. Add your Gemini API key in the pipeline configuration or at Realtime Integrations.
2. To get your API key, visit Gemini API Keys.

Available models:

• gemini-2.0-flash-live-001
• gemini-2.0-flash
• gemini-2.5-flash-native-audio-preview-09-2025
• gemini-2.5-flash-native-audio

Cascading Pipeline

The Cascading Pipeline processes audio through distinct stages (STT → LLM → TTS), providing maximum control over each component.

Configure your providers for STT Integrations, LLM Integrations and TTS Integrations.

Example: Adding Deepgram STT

• Get API Key at: Deepgram Console

Available models:

• flux-general-en
• nova-2 or nova-2-general (for non-English transcriptions)
• nova-3 or nova-3-general
• base

Step 5: Knowledge Base Integration

5

Knowledge Base Integration

Upload a knowledge base to provide context and domain expertise to your voice agent. This dramatically improves answer accuracy and enables your agent to handle specialized queries.

• Navigate to the Knowledge Base tab in your agent configuration.
• Upload documents, FAQs, or product sheets that contain relevant information.
• The agent will use this knowledge to provide more accurate and contextual responses.

Step 6: Configure Telephony Settings

6

Configure Telephony Settings

Configure telephony settings to enable your agent to handle phone calls:

• Agent Type: Set the type of agent (inbound, outbound, or both).
• Inbound Gateways: Set up gateways to receive incoming calls.
• Outbound Gateways: Set up gateways to make outbound calls.
• Routing Rules: Create rules to map phone numbers to your agent.
• Calling Settings: Configure call handling preferences and behavior.

This configuration is essential for call center automation, platform integration, and smooth agent orchestration.

Step 7: Test Your Voice Agent

7

Test Your Voice Agent

You can interact with the agent directly from the dashboard before connecting it to production channels:

1. Visit Agents Dashboard.
2. Locate your agent in the list and click the Test button in the top-right corner.
3. Use the built-in simulator to speak with the agent in real time, view live transcripts, and fine-tune prompts based on the conversation.

Step 8: Connect Voice Agent

8

Connect Voice Agent

Once your agent is configured, you can connect it to various platforms and devices:

• Web: Integrate your agent into web applications.
• Mobile: Connect to iOS and Android mobile apps.
• Telephony: Deploy to phone systems for voice calls.
• IoT Devices: Connect to Internet of Things devices.

Next Steps

Congratulations! You've successfully created your AI voice agent. Here are the next steps:

• Test Your Agent: Use the built-in test simulator to verify your agent's behavior and responses.
• Deploy to Production: Connect your agent to production environments and real user interactions.
• Monitor Performance: Track agent performance, user satisfaction, and conversation quality.
• Iterate and Improve: Refine your agent's prompts, knowledge base, and configuration based on real-world usage.

Keep refining your agent's configuration to build a powerful voice AI solution tailored to your specific business needs.

Integrations

Connect with JavaScript

Core language of the web

Connect with React

UI library for building interactive web apps.

Connect with React-Native

Cross-platform mobile app JS framework.

Connect with flutter

Cross-platform apps from one codebase.

Connect with iOS

Mobile apps for Apple devices.