IoT SDK + AI Agent Example

You can pair the IoT SDK with a VideoSDK AI Agent so a device joins the same meeting as the agent. The device streams its audio to the agent, and the agent listens, processes the audio, and responds in real time. This gives the device two-way voice interaction.

What You Can Build

Smart Home & Lifestyle

• Adaptive Climate Control: a thermostat that reports the temperature and takes spoken instructions to adjust it.

• Interactive Home Chef: a kitchen device that suggests recipes and guides the user through them by voice.

• Wellness Coach: a wearable that monitors heart rate and gives spoken prompts, such as a breathing exercise when the rate spikes.

Education & Entertainment

• Child-Friendly Storyteller: a toy that reads stories aloud, changes character voices, and answers a child's questions.

• Interactive Tutor: a classroom device that answers students' questions aloud and adjusts its explanations to each student.

Any device that can join a meeting alongside an AI Agent can support similar two-way voice use cases.

Demo: IoT SDK in Conversation with AI Agent

---

Prerequisites

Before you get started, ensure you have the following:

• Python: Python version >= 3.11
• Video SDK Developer Account (Not having one, follow Video SDK Dashboard)

---

Step 1: Setup for AI Agent

1

Setup for AI Agent

On the AI Agent side, configure it to join the meeting as a participant. It listens to the device's audio, processes it, and generates responses in real time, handling speech recognition, reasoning, and synthesis for low-latency two-way interaction between the device and the meeting.

def make_context() -> JobContext:
    room_options = RoomOptions(
        room_id="YOUR_MEETING_ID",
        name="VideoSDK Cascaded Agent",
        playground=True
    )

return JobContext(room_options=room_options)

caution

Make sure you keep the same meetingID in the RoomOptions here, which you’ll also use on the IoT device side later.

See the full setup in the AI Agent Quick Start Guide

Step 2: Setup for IoT SDK

2

Setup IoT SDK

Now it’s time to configure the IoT SDK on your ESP device. This step enables the device to capture and publish audio streams into the meeting, as well as receive and play responses from the AI Agent. With this setup, your device transitions from being a simple endpoint to an active communication node.

char *token = "Generated - token";
init_config_t init_cfg = {
    .meetingID = "meeting-id",     
    .token = token,
    .displayName = "ESP32-Device",
    .participantId = "your participant id",   // "" or NULL => a random id is generated
    .audioCodec = AUDIO_CODEC_OPUS,
    .videoCodec = VIDEO_CODEC_NONE,           // audio-only agent
};

result_t init_result = init(&init_cfg);
printf("Result: %d\n", init_result);

caution

Ensure that the meetingID configured on the IoT device is identical to the meetingID defined in the AI Agent’s RoomOptions.

See the full setup in the IoT SDK Quick Start Guide

Step 3: Connect the IoT Device and AI Agent

3

Connect the IoT Device and AI Agent

Run the two sides together:

1. The AI Agent joins the meeting first.

if __name__ == "__main__":
    # Start the AI Agent, which will join the meeting as a participant
    job = WorkerJob(entrypoint=start_session, jobctx=make_context) 
    job.start()

2. The IoT device then joins the same meeting and starts publishing and subscribing to audio.

  // joining the User through IoT - device once AI - Agent joins the meeting.
  result_t result_publish = startPublishAudio(); 
  result_t result_subscribe = startSubscribeAudio();
  printf("Result Publish:%d\n", result_publish);
  printf("Result Subscribe:%d\n", result_subscribe); 

Supported hardware

Two-way voice needs a board with a speaker so the device can play the agent's replies. Without one, startSubscribeAudio() returns DEVICE_NOT_SUPPORTED, so the device can send audio to the agent but cannot play its replies. See Supported Microcontrollers.

3. The device and the AI Agent now exchange audio in both directions in real time.