Version: 1.x.x

Optimize Video Tracks - React Native

To optimize the viewing experience, it's essential to fine-tune the video tracks used during calls.

For an enhanced fine-tuning experience, we've introduced the capability to provide a custom video track for a participant's media before and during the meeting.

Custom Video Track
Custom Screen Share Track

note

This page uses the following asynchronous methods. Refer to their API reference for the errors each Promise may reject with, and handle these rejections appropriately based on your use case.

Custom Video Track

This feature allows you to incorporate custom video encoder configurations, choose optimization modes (focusing on motion, text or detail of the video), specify a modern video quality profile (bitrateMode), and control simulcast behavior (maxLayer), and apply background removal and video filters from external libraries (e.g., videosdk-media-processor), then send these modifications to other participants.

Once you set a custom video track created using createCameraVideoTrack(), VideoSDK remembers its settings and automatically reuses them whenever the webcam is turned back on. To change the track configuration, call disableWebcam(), then pass a new track created using createCameraVideoTrack() to enableWebcam(). VideoSDK preserves only tracks created using createCameraVideoTrack(). Filters applied through external libraries such as videosdk-media-processor are not preserved and must be re-applied each time the webcam is re-enabled. Read more in Track configuration is preserved.

`How to Create a Custom Video Track ?`

You can create a Custom Video Track using createCameraVideoTrack() method of @videosdk.live/react-native-sdk.
This method can be used to create video track by specifying parameters like video resolution, camera facing mode, and the quality controls (bitrateMode and maxLayer).
You can choose video resolution from the below mentioned list of values for the encoder config:

Encoder Config	Resolution	Frame Rate	Optimized (kbps)	Balanced (kbps)	High Quality (kbps)
h144p_w192p	192x144	15 fps	60	100	150
h240p_w320p	320x240	15 fps	80	150	300
h480p_w640p	640x480	25 fps	300	500	700
h720p_w960p	960x720	30 fps	800	1200	1600
h1080p_w1440p	1440x1080	30 fps	1600	2200	2700
h90p_w160p	160x90	15 fps	60	100	150
h360p_w640p	640x360	20 fps	200	400	600
h720p_w1280p	1280x720	30 fps	1000	1500	2000

Example

import { createCameraVideoTrack , Constants} from "@videosdk.live/react-native-sdk";

let customTrack = await createCameraVideoTrack({
  // It will be the id of the camera from which the video should be captured.
  cameraId: "camera-id", // OPTIONAL

  // This parameter will be discussed in the next step.
  optimizationMode: "motion", // "text" | "detail",  Default : "motion"

  // This will accept the resolution (height x width) of video you want to capture.
  encoderConfig: "h720p_w960p", // "h360p_w640p" | "h480p_w640p" ... // Default : "h720p_w1280p"

  // For Mobile browser It will specify whether to use front or back camera for the video track.
  facingMode: "environment", // "user",  Default : "environment"

  // This parameter will be discussed in the next step.
  multiStream: true, // false,  Default : true

  // Optional: This controls the video quality and bandwidth usage.
  bitrateMode: Constants.BitrateMode.HIGH_QUALITY, // "BANDWIDTH_OPTIMIZED" | "BALANCED" , Default : BALANCED

  // Optional: This specifies the maximum number of simulcast layers (maxLayer) to publish.
  maxLayer: 2, // 3 , Default: 3

  // Optional: This specifies the video codec used to encode this track.
  codec: Constants.VideoCodec.VP8 // "H264" | "VP9" | "AV1" , Default : VP8
});

caution

The behavior of custom track configurations is influenced by the capabilities of the device. For example, if you set the encoder configuration to 1080p but the webcam only supports 720p, the encoder configuration will automatically adjust to the highest resolution that the device can handle, which in this case is 720p.

What is `optimizationMode`?

This parameter specifies the optimization mode for the video track being generated.
motion : This type of track focuses more on motion video such as webcam video, movies or video games.
- It will degrade resolution in order to maintain frame rate.
text : This type of track focuses on significant sharp edges and areas of consistent color that can change frequently such as presentations or web pages with text content.
- It will degrade frame rate in order to maintain resolution.
detail : This type of track focuses more on the details of the video such as, presentations, painting or line art.
- It will degrade frame rate in order to maintain resolution.

What is `multiStream`?

This parameter specifies whether the stream should send multiple resolution layers or a single resolution layer. It is true by default.

Refer to the Simulcast (Adaptive Bitrate) guide for the layers a track publishes and for how participants select the layer they receive.

What is `BitrateMode`?

BitrateMode is the key setting for video quality. It lets you decide whether to prioritize sharp details, save internet data, or find a good balance between the two.

You can choose from three distinct modes:

bandwidth_optimized: Prioritizes lower bandwidth usage over video quality. Ideal for users with poor or unstable network conditions.
balanced: The default setting. It provides a smart compromise between clear video and efficient bandwidth consumption, suitable for most use cases.
high_quality: Prioritizes video quality over bandwidth usage. Use this when high-fidelity video is essential and viewers are expected to have strong network connections.

What is `maxLayer`?

This parameter caps how many layers a track publishes, and accepts 2 or 3. It applies only when multiStream is true, and defaults to 3.

Refer to the Simulcast (Adaptive Bitrate) guide for what each value publishes.

What is `codec`?

This parameter specifies the video codec used to encode your stream. VideoSDK supports VP8 (default), H264, VP9, and AV1, set through Constants.VideoCodec.
Currently, VideoSDK doesn't support multiStream for H264, VP9, and AV1. multiStream defaults to true, so set multiStream: false when you create an H264, VP9, or AV1 track. Otherwise VideoSDK sets it to false for you and emits the ERROR_MULTISTREAM_NOT_SUPPORTED error event.

Refer to the Video Codecs guide to compare the four codecs, to check which codec a meeting is actually using, and to handle the cases where a codec is not supported.

`How to Setup a Custom Video Track ?`

The custom track can be configured both before and after the meeting is initialized. Following are the methods that help in doing so:

1. Setting up a Custom Track during the initialization of a meeting

If you are enabling the webcam (webcamEnabled: true) in the config of MeetingProvider and wish to use custom tracks from the start of the meeting, you can pass a custom track in the config as demonstrated below.

caution

Custom Track will not apply on the webcamEnabled: false configuration.

Example

import {
  createCameraVideoTrack,
  MeetingProvider,
  Constants,
} from "@videosdk.live/react-native-sdk";

function App() {
  const getTrack = async () => {
    const track = await createCameraVideoTrack({
      optimizationMode: "motion",
      encoderConfig: "h720p_w960p",
      facingMode: "user",
      multiStream: true,
      bitrateMode: Constants.BitrateMode.HIGH_QUALITY,
      maxLayer: 2,
      codec: Constants.VideoCodec.VP8
    });
    setCustomTrack(track);
  };

  let [customTrack, setCustomTrack] = useState();

  useEffect(() => {
    getTrack();
  }, []);

  return (
    customTrack && (
      <MeetingProvider
        config={{
          meetingId,
          micEnabled: true,
          //If true, it will use the passed custom track to turn webcam on
          webcamEnabled: true,
          //Pass the custom video track here
          customCameraVideoTrack: customTrack,
        }}
        token={token}
      >
        <MeetingView />
      </MeetingProvider>
    )
  );
}

2. Setting up a Custom Track with methods

To switch tracks during the meeting, you need to pass the MediaStream in the enableWebcam() or toggleWebcam() method of useMeeting.

tip

Make sure to call the disableWebcam() method before you create a new track as it may lead to unexpected behavior.

Example

import {
  createCameraVideoTrack,
  useMeeting,
  Constants,
} from "@videosdk.live/react-native-sdk";
import { TouchableOpacity, Text } from "react-native";

const MeetingControls = () => {
  const { localWebcamOn, enableWebcam, disableWebcam, toggleWebcam } =
    useMeeting();

  const handleToggleWebcam = async () => {
    if (localWebcamOn) {
      try {
        await toggleWebcam();
      } catch (err) {
        console.error("toggleWebcam failed:", err);
      }
    } else {
      let customTrack = await createCameraVideoTrack({
        optimizationMode: "motion",
        encoderConfig: "h720p_w960p",
        facingMode: "user",
        multiStream: true,
        bitrateMode: Constants.BitrateMode.HIGH_QUALITY,
        maxLayer: 2,
        codec: Constants.VideoCodec.VP8
      });

      try {
        await toggleWebcam(customTrack);
      } catch (err) {
        console.error("toggleWebcam failed:", err);
      }
    }
  };

  const handleEnableWebcam = async () => {
    if (localWebcamOn) {
      try {
        await disableWebcam();
      } catch (err) {
        console.error("disableWebcam failed:", err);
      }
    } else {
      let customTrack = await createCameraVideoTrack({
        optimizationMode: "motion",
        encoderConfig: "h720p_w960p",
        facingMode: "user",
        multiStream: true,
        bitrateMode: Constants.BitrateMode.HIGH_QUALITY,
        maxLayer: 2,
        codec: Constants.VideoCodec.VP8
      });

      try {
        await enableWebcam(customTrack);
      } catch (err) {
        console.error("enableWebcam failed:", err);
      }
    }
  };

  return (
    <>
      <TouchableOpacity
        onPress={() => {
          handleToggleWebcam();
        }}
      >
        <Text>Toggle Webcam</Text>
      </TouchableOpacity>

      <TouchableOpacity
        onPress={() => {
          handleEnableWebcam();
        }}
      >
        <Text>Enable Webcam</Text>
      </TouchableOpacity>
    </>
  );
};

info

Using custom video tracks is not just limited to the video tracks created using the createCameraVideoTrack method. You can use any MediaStream object as a replacement including a custom canvas track created by you.

`Which configuration is suitable for the Device ?`

In this section, the focus is on understanding participant size wise encoder(Resolution) and multiStream configuration.

This feature enables the customization of screenshare streams with enhanced optimization modes and predefined encoder configuration (resolution + FPS) for specific use cases, which can then be sent to other participants.

`How to Create Custom Screen Share Track ?`

You can create a Video Track using createScreenShareVideoTrack() method of @videosdk.live/react-native-sdk.
This method enables the creation of a video track with different encoding parameters and optimization modes.

Example

import { createScreenShareVideoTrack } from "@videosdk.live/react-native-sdk";

let customShareTrack = await createScreenShareVideoTrack({
  optimizationMode: "motion", // "text" | "detail",  Default : "motion"

  // This will accept the  height & FPS of video you want to capture.
  encoderConfig: "h720p_15fps", //  `h360p_30fps` | `h1080p_30fps` // Default : `h720p_15fps`
});

You can learn about optimizationMode from here

`How to Setup Custom Screen Share Track ?`

In order to switch tracks during the meeting, you have to pass the MediaStream in the enableScreenShare() or toggleScreenShare() method of useMeeting.

note

Make sure to call the disableScreenShare() method before you create a new track as it may lead to unexpected behavior.

Example

import {
  createScreenShareVideoTrack,
  useMeeting,
} from "@videosdk.live/react-native-sdk";
import { TouchableOpacity, Text } from "react-native";

const MeetingControls = () => {
  const {
    localScreenShareOn,
    enableScreenShare,
    disableScreenShare,
    toggleScreenShare,
  } = useMeeting();

  const handleToggleScreenShare = async () => {
    if (localScreenShareOn) {
      try {
        await toggleScreenShare();
      } catch (err) {
        console.error("toggleScreenShare failed:", err);
      }
    } else {
      let customTrack = await createScreenShareVideoTrack({
        optimizationMode: "motion",
        encoderConfig: "h720p_15fps",
      });

      try {
        await toggleScreenShare(customTrack);
      } catch (err) {
        console.error("toggleScreenShare failed:", err);
      }
    }
  };

  const handleEnableScreenShare = async () => {
    if (localScreenShareOn) {
      try {
        await disableScreenShare();
      } catch (err) {
        console.error("disableScreenShare failed:", err);
      }
    }

    let customTrack = await createScreenShareVideoTrack({
      optimizationMode: "motion",
      encoderConfig: "h720p_15fps",
    });

    try {
      await enableScreenShare(customTrack);
    } catch (err) {
      console.error("enableScreenShare failed:", err);
    }
  };

  return (
    <>
      <TouchableOpacity
        onPress={() => {
          handleToggleWebcam();
        }}
      >
        <Text>Toggle ScreenShare</Text>
      </TouchableOpacity>

      <TouchableOpacity
        onPress={() => {
          handleEnableWebcam();
        }}
      >
        <Text>Enable ScreenShare</Text>
      </TouchableOpacity>
    </>
  );
};

API Reference

The API references for all the methods and events utilised in this guide are provided below.

Got a Question? Ask us on discord

Custom Video Track​

How to Create a Custom Video Track ?​

Example​

What is optimizationMode?​

What is multiStream?​

What is BitrateMode?​

What is maxLayer?​

What is codec?​

How to Setup a Custom Video Track ?​

1. Setting up a Custom Track during the initialization of a meeting​

Example​

2. Setting up a Custom Track with methods​

Example​

Which configuration is suitable for the Device ?​

Custom Screen Share Track​

How to Create Custom Screen Share Track ?​

Example​

How to Setup Custom Screen Share Track ?​

Example​

API Reference​

Custom Video Track

`How to Create a Custom Video Track ?`

Example

What is `optimizationMode`?

What is `multiStream`?

What is `BitrateMode`?

What is `maxLayer`?

What is `codec`?

`How to Setup a Custom Video Track ?`

1. Setting up a Custom Track during the initialization of a meeting

Example

2. Setting up a Custom Track with methods

Example

`Which configuration is suitable for the Device ?`

Custom Screen Share Track

`How to Create Custom Screen Share Track ?`

Example

`How to Setup Custom Screen Share Track ?`

Example

API Reference