Skip to main content
Version: Next

IoT Quick Start

The IoT SDK enables real-time audio and video conferencing by connecting devices directly to the VideoSDK server. Devices can join meetings, publish audio and video or sensor data, and subscribe to other participants' streams.

Prerequisites​

Before you get started, ensure you have the following:

Install VideoSDK's IoT component​

Step 1: Setup for esp-idf​

1
Setup for esp-idf

Install and configure the ESP-IDF environment on your system to prepare the toolchain, dependencies, and base project structure for development. For more information visit ESP-IDF

# Tools
brew install cmake ninja dfu-util ccache git wget flex bison gperf
brew install openssl libffi

# Create Project directory
mkdir esp
cd esp

# Now clone esp-idf from github
git clone -b v5.4.2 https://github.com/espressif/esp-idf.git
cd esp-idf

# Set Python env path and install ESP-IDF tools
./install.sh

# Export ESP-IDF environment variables
. ./export.sh
# or
source export.sh

# Create an esp-idf project
cd ~/esp
idf.py create-project your-project-name
cd your-project-name

Step 2: Add IoT SDK component​

2
Add IoT SDK component

The IoT SDK is published on the ESP Component Registry as videosdk/iot-sdk. Create a file named idf_component.yml in your main folder and add it as a managed dependency. ESP-IDF downloads it on the next build, so there is nothing to clone.

main/idf_component.yml
dependencies:
  videosdk/iot-sdk: '*'
  idf:
    version: ">=5.4.2,<=5.4.4"
  mdns: "*"
  # Wi-Fi/network helper (example_connect). Bundled with ESP-IDF,
  # resolved from $IDF_PATH, not a registry component.
  protocol_examples_common:
    path: ${IDF_PATH}/examples/common_components/protocol_examples_common

Step 3: Update your CmakeLists.txt​

3
Update your CmakeLists.txt

This ensures that your project links correctly with the required IoT SDK and ESP-IDF components. Replace your-project-name.c with the name of your source file. For more information visit CMake

main/CMakeLists.txt
idf_component_register(
    SRCS "your-project-name.c"
    INCLUDE_DIRS "."
    REQUIRES mbedtls json esp_netif fatfs vfs esp_common esp_timer esp_lcd nvs_flash
)

# libsrtp pointer-type mismatch on Xtensa with ESP-IDF 5.4.2+: uint32_t is
# 'long unsigned int' there, while srtp.c uses 'unsigned int'. Same width,
# but the compiler treats the pointers as incompatible.
idf_component_get_property(srtp_lib sepfy__srtp COMPONENT_LIB)
target_compile_options(${srtp_lib} PRIVATE -Wno-incompatible-pointer-types)

target_compile_options(${COMPONENT_LIB} PRIVATE "-Wno-format")
warning

The two target_compile_options lines are required. Without the libsrtp option the project fails to build on ESP-IDF 5.4.2+.

Step 4: Create Partitions Table​

4
Create Partitions Table

Create a file named partitions.csv in your project folder. This file defines the flash memory layout for your device. The same layout is used on both supported boards. For more information visit Partitions Table

partitions.csv
# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 4M,

Step 5: Add board configurations​

5
Add board configurations

Create a file named sdkconfig.defaults in your project folder. These values are tuned so the SDK has enough memory and CPU. The SDK runs audio, video and the network stack at the same time on the ESP32-S3. For more information sdkconfig

sdkconfig.defaults
# Pick your board first, under menuconfig -> "SET Microcontroller"
# (ESP32-S3-Korvo-2 or the XIAO ESP32-S3)
CONFIG_IDF_TARGET="esp32s3"
CONFIG_APP_RETRIEVE_LEN_ELF_SHA=16

# Flash: this board has 8 MB, and QIO (4-line) reads run code from flash roughly
# twice as fast as the default 2-line mode. A custom partition table is used to
# fit the app.
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
CONFIG_ESPTOOLPY_FLASHMODE_QIO=y
CONFIG_PARTITION_TABLE_CUSTOM=y

CONFIG_EXAMPLE_CONNECT_IPV6=n
CONFIG_ESP_PHY_REDUCE_TX_POWER=y

# PSRAM (external RAM), REQUIRED. The SDK's audio/video buffers and the WiFi
# network buffers live here so the small, fast internal RAM stays free for
# time-critical work. Running PSRAM at 80 MHz roughly doubles its bandwidth,
# which speeds up the whole audio + video + network path.
CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_SPIRAM_USE_MALLOC=y

# Keep WiFi/network buffers and task stacks in PSRAM so enough internal RAM stays
# free while a call is active. Without these, the device can fail to connect or
# reset once audio and video are both streaming.
CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY=y

# Reserve a slice of fast internal RAM for the WiFi driver and encryption, which
# cannot run from PSRAM. This is what keeps outgoing audio/video packets flowing
# when the network is busy. Do not lower these; raising the reserve does not help
# if you are short on RAM, reduce what you allocate instead.
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=1024
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=16384

# Run at full 240 MHz and build size-optimized (-Os). -Os is noticeably faster
# than the debug default for the SDK's audio/video/network code, not just smaller.
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
CONFIG_COMPILER_OPTIMIZATION_SIZE=y

# 1 ms scheduler tick: smoother time-sharing between the audio, video, and
# network tasks, which reduces jitter in the call.
CONFIG_FREERTOS_HZ=1000

# Task stacks sized for the SDK's signaling and media work.
CONFIG_ESP_SYSTEM_EVENT_TASK_STACK_SIZE=3072
CONFIG_ESP_MAIN_TASK_STACK_SIZE=10240
CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU1=n
CONFIG_ESP_IPC_TASK_STACK_SIZE=2048

# WiFi + TCP/IP buffers tuned so the video stream does not starve the network
# stack and drop outgoing packets.
CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=16
CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=48
CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM=32
CONFIG_LWIP_IPV6_AUTOCONFIG=y
CONFIG_LWIP_IPV6_DHCP6=y
CONFIG_LWIP_TCP_SND_BUF_DEFAULT=5744
CONFIG_LWIP_TCP_WND_DEFAULT=5744
# Larger inbound UDP queue so received audio/video is not dropped while the
# outgoing stream keeps the network busy.
CONFIG_LWIP_UDP_RECVMBOX_SIZE=64

# Logging. A normal build is quiet, lifecycle messages, warnings, and errors
# only. This line compiles the SDK's debug logs into the image (a few KB of
# flash) WITHOUT turning them on, so you can enable them later at
# menuconfig -> "VideoSDK Logging" -> Debug, or from code with
# videosdk_set_log_mode(VIDEOSDK_LOG_DEBUG). Leave it in, or debug mode prints
# nothing extra.
CONFIG_LOG_MAXIMUM_LEVEL_DEBUG=y

# TLS / DTLS, required to reach the VideoSDK signaling server over wss:// and
# for the encrypted WebRTC media transport. The certificate bundle verifies the
# server's certificate.
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
CONFIG_MBEDTLS_SSL_PROTO_DTLS=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y
CONFIG_PTHREAD_TASK_STACK_SIZE_DEFAULT=8192

# Camera (used by the video send path). The common sensors are enabled so the
# driver auto-detects whichever one is fitted on your board.
CONFIG_OV2640_SUPPORT=y
CONFIG_OV3660_SUPPORT=y
CONFIG_OV5640_SUPPORT=y
CONFIG_CAMERA_DMA_BUFFER_SIZE_MAX=32768
warning

sdkconfig.defaults is read only on the first build, when no sdkconfig file exists yet. If you have already built the project once, delete the generated sdkconfig before rebuilding; otherwise these values are silently ignored.

Step 6: Update your-project-main.c​

6
Update your-project-main.c

Write your project logic inside your-project-name.c. This is where you initialize the IoT SDK, join the meeting, and start streaming.

The token and meeting ID are read from menuconfig (Step 7)

main/your-project-name.c
#include <stdio.h>
#include <string.h>
#include "esp_event.h"
#include "esp_log.h"
#include "esp_mac.h"
#include "esp_netif.h"
#include "esp_system.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"   // vTaskDelay, pdMS_TO_TICKS
#include "mdns.h"
#include "nvs_flash.h"
#include "protocol_examples_common.h"
#include "sdkconfig.h"
#include "videosdk.h"

static const char *TAG = "IOT-SDK";

// Set your token under menuconfig -> "VideoSDK Configuration".
// Not const: init_config_t.token is a char*. The SDK copies what you pass, so
// this buffer is not retained after init() returns.
char *token = CONFIG_VIDEOSDK_TOKEN;

void app_main(void)
{
    static char deviceid[32] = {0};
    uint8_t mac[8] = {0};

    ESP_LOGI(TAG, "[APP] Startup..");
    ESP_LOGI(TAG, "[APP] IDF version: %s", esp_get_idf_version());

    // Log verbosity, chosen in menuconfig -> "VideoSDK Logging".
    // Call it before init() so it covers the join.
#if CONFIG_VIDEOSDK_LOG_MODE_DEBUG
    videosdk_set_log_mode(VIDEOSDK_LOG_DEBUG);
#else
    videosdk_set_log_mode(VIDEOSDK_LOG_NORMAL);
#endif

    ESP_ERROR_CHECK(nvs_flash_init());
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    ESP_ERROR_CHECK(example_connect());

    // A stable participant id derived from this board's Wi-Fi MAC.
    if (esp_read_mac(mac, ESP_MAC_WIFI_STA) == ESP_OK) {
        sprintf(deviceid, "esp32-%02x%02x%02x%02x%02x%02x",
                mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
        ESP_LOGI(TAG, "Device ID: %s", deviceid);
    }

    if (token[0] == '\0' || CONFIG_VIDEOSDK_MEETING_ID[0] == '\0') {
        ESP_LOGE(TAG, "Token / meeting ID not set. Run 'idf.py menuconfig'"
                      " -> VideoSDK Configuration.");
        return;
    }

    init_config_t init_cfg = {
        .meetingID     = CONFIG_VIDEOSDK_MEETING_ID,
        .token         = token,
        .displayName   = "ESP32S3-Device",   // shown to other participants
        .participantId = deviceid,           
        .audioCodec    = AUDIO_CODEC_PCMA,   // PCMA, PCMU or OPUS
        .videoCodec    = VIDEO_CODEC_JPEG,   // VIDEO_CODEC_NONE for audio-only
    };

    result_t init_result = init(&init_cfg);
    ESP_LOGI(TAG, "init: %d", init_result);
    if (init_result != RESULT_OK) {
        return;
    }

    // Send the microphone and the camera into the meeting.
    // Both supported boards can publish.
    result_t publish_audio = startPublishAudio();
    ESP_LOGI(TAG, "startPublishAudio: %d", publish_audio);

    result_t publish_video = startPublishVideo();
    ESP_LOGI(TAG, "startPublishVideo: %d", publish_video);

    // ------------------------------------------------------------------
    // RECEIVING (playing remote audio / showing remote video).
    //
    // Needs a board with a speaker and a display; on boards without them the
    // subscribe calls return DEVICE_NOT_SUPPORTED (3004). See the
    // Supported Microcontrollers page for board capabilities.
    //
    // To use them, select the Korvo-2 under
    //   menuconfig -> "SET Microcontroller" -> "Audio hardware board"
    // and uncomment:
    //
    // result_t subscribe_audio = startSubscribeAudio();
    // ESP_LOGI(TAG, "startSubscribeAudio: %d", subscribe_audio);
    //
    // result_t subscribe_video = startSubscribeVideo();
    // ESP_LOGI(TAG, "startSubscribeVideo: %d", subscribe_video);
    // ------------------------------------------------------------------

    // Keep the session active for a defined duration (adjust as per your application use case)
    vTaskDelay(pdMS_TO_TICKS(100000));

    // leave() stops every stream that was started.
    result_t result_leave = leave();
    ESP_LOGI(TAG, "leave: %d", result_leave);

    // Keep main loop alive
    while (1) {
        vTaskDelay(pdMS_TO_TICKS(10));
    }
}

Step 7: Build & Flash Project​

7
Build & Flash Project

Configure, build, and flash the firmware onto your ESP32-S3 board.

warning

Set the target to esp32s3. Any other target will fail to build; the SDK supports the ESP32-S3 only.

# 1. Set your board as the target
idf.py set-target esp32s3

# 2. Configure the project
idf.py menuconfig

# 3. Build the project
idf.py build

# 4. Flash it to your board and open the monitor
idf.py -p <PORT> flash monitor

Inside idf.py menuconfig, set the following. Press S to save and Enter to confirm, then Esc or q to exit.

a. Example Connection Configuration
       |-> WIFI SSID              <!-- your WiFi name -->
       |-> WIFI Password          <!-- your WiFi password -->

b. VideoSDK Configuration
       |-> Auth token (JWT)       <!-- paste your VideoSDK token -->
       |-> Meeting / room ID      <!-- the meeting you want to join -->

c. SET Microcontroller
       |-> Audio hardware board
             |-> ESP32-S3-XIAO      (default)
             |-> ESP32-S3-Korvo-2
       |-> Speaker output volume (0-100)   <!-- board-dependent; see Supported Microcontrollers -->

d. VideoSDK Logging
       |-> Log verbosity
             |-> Normal (default)
             |-> Debug

e. Partition table                   <!-- already set by sdkconfig.defaults -->
       |-> Custom partition table CSV

f. Serial flasher config             <!-- already set by sdkconfig.defaults -->
       |-> Flash size: 8 MB

Step 8: Connecting with VideoSDK Client Applications​

8
Connecting with VideoSDK Client Applications

Integrate IoT devices with VideoSDK client applications. Ensure that both the IoT device and client apps connect to the same room for communication.

JavaScript
React
React Native
Android
Flutter
iOS
tip

You can checkout the complete quick start example here.

tip

Prefer a ready-made project? The component ships two runnable examples you can scaffold in one command:

idf.py create-project-from-example "videosdk/iot-sdk=0.3.2:audio_call"
idf.py create-project-from-example "videosdk/iot-sdk=0.3.2:video_call"

Got a Question? Ask us on discord

Last updated on by karan07080
Previous
IoT + AI Agent
Next

Was this helpful?