Optimize Video Tracks - Javascript

To optimize the viewing experience, it's essential to fine-tune the video tracks used during calls.

For an enhanced fine-tuning experience, we've introduced the capability to provide a custom video track for a participant's media before and during the meeting.

1. Custom Video Track
2. Custom Screen Share Track

Custom Video Track

This feature allows you to incorporate custom video encoder configurations, choose optimization modes (focusing on motion, text or detail of the video), specify a modern video quality profile (bitrateMode), and control simulcast behavior (maxLayer), and apply background removal and video filters from external libraries (e.g., videosdk-media-processor), then send these modifications to other participants.

How to Create a Custom Video Track ?

• You can create a Custom Video Track using createCameraVideoTrack().
• This method can be used to create video track by specifying parameters like video resolution, camera facing mode, and the quality controls (bitrateMode and maxLayer).
• You can choose video resolution from the below mentioned list of values for the encoder config:

Encoder Config	Resolution	Frame Rate	Optimized (kbps)	Balanced (kbps)	High Quality (kbps)
h90p_w160p	160x90	15 fps	60	100	150
h180p_w320p	320x180	15 fps	100	150	250
h216p_w384p	384x216	15	120	200	350
h360p_w640p	640x360	20	200	400	600
h540p_w960p	960x540	25	350	550	800
h720p_w1280p	1280x720	30	1000	1500	2000
h1080p_w1920p	1920x1080	30	1600	2200	2700
h1440p_w2560p	2560x1440	30	5000	5500	6000
h2160p_w3840p	3840x2160	30	8000	8500	9000
h120p_w160p	160x120	15	60	100	150
h180p_w240p	240x180	15	100	150	250
h240p_w320p	320x240	15	120	200	350
h360p_w480p	480x360	20	200	400	600
h480p_w640p	640×480	25	300	500	700
h540p_w720p	720×540	30	350	550	800
h720p_w960p	960×720	30	900	1300	1600
h1080p_w1440p	1440×1080	30	1600	2200	2700
h1440p_w1920p	1920×1440	30	3500	4000	4500

note

Above mentioned encoder configurations are valid for both, landscape as well as portrait mode.

Example

let customTrack = await VideoSDK.createCameraVideoTrack({
  // It will be the id of the camera from which the video should be captured.
  cameraId:"camera-id", // OPTIONAL

  // We will discuss this parameter in next step.
  optimizationMode: "motion", // "text" | "detail",  Default : "motion"

  // This will accept the resolution (height x width) of video you want to capture.
  encoderConfig: "h540p_w960p", // "h360p_w640p" | "h720p_w960p" ... // Default : "h720p_w1280p"

  // For Mobile browser It will specify whether to use front or back camera for the video track.
  facingMode: "environment", // "front",  Default : "environment"

  // We will discuss this parameter in next step.
  multiStream:true, // false,  Default : true

  // Optional: This controls the video quality and bandwidth usage.
  bitrateMode: VideoSDK.Constants.BitrateMode.HIGH_QUALITY, // "BANDWIDTH_OPTIMIZED" | "BALANCED" , Default : BALANCED

  // Optional: This specifies the maximum number of simulcast layers (maxLayer) to publish.
  maxLayer: 2 // 3 , Default: 3

  // Optional: This Specifies the codec that we have to pass on the meeting.
  codec: VideoSDK.Constants.VideoCodec.H264 // "VP8" | "VP9" | "AV1" , Default : VP8 
});

caution

The behavior of custom track configurations is influenced by the capabilities of the device. For example, if you set the encoder configuration to 1080p but the webcam only supports 720p, the encoder configuration will automatically adjust to the highest resolution that the device can handle, which in this case is 720p.

What is optimizationMode?

• This parameter specifies the optimization mode for the video track being generated.

• motion : This type of track focuses more on motion video such as webcam video, movies or video games.

  • It will degrade resolution in order to maintain frame rate.

• text : This type of track focuses on significant sharp edges and areas of consistent color that can change frequently such as presentations or web pages with text content.

  • It will degrade frame rate in order to maintain resolution.

• detail : This type of track focuses more on the details of the video such as, presentations, painting or line art.

  • It will degrade frame rate in order to maintain resolution.

What is multiStream?

• This parameter specifies whether the stream should send multiple resolution layers or a single resolution layer.

The multiStream : true configuration indicates that VideoSDK, by default, sends multiple resolution video streams to the server. For example, if a user's device capability is 720p, VideoSDK sends streams in 720p, 360p, and 180p resolution. This enables VideoSDK to deliver the appropriate stream to each participant based on their network bandwidth.

Setting multiStream : false restricts VideoSDK to send only one stream, helping to maintain quality by focusing on a single resolution.

danger

The setQuality parameter will not have any effect if multiStream is set to false.

What is BitrateMode?

BitrateMode is the key setting for video quality. It lets you decide whether to prioritize sharp details, save internet data, or find a good balance between the two.

You can choose from three distinct modes:

• bandwidth_optimized: Prioritizes lower bandwidth usage over video quality. Ideal for users with poor or unstable network conditions.
• balanced: The default setting. It provides a smart compromise between clear video and efficient bandwidth consumption, suitable for most use cases.
• high_quality: Prioritizes video quality over bandwidth usage. Use this when high-fidelity video is essential and viewers are expected to have strong network connections.

What is maxLayer?

maxLayer is an advanced parameter that gives you direct control over simulcast, which is the process of sending multiple versions (or "layers") of the same video stream at different resolutions and qualities simultaneously.

• maxLayer : 3 (Default): Publishes all available layers (e.g., high, medium, and low quality). This provides the maximum adaptability for viewers on diverse and unpredictable networks.

• maxLayer : 2: Intelligently publishes only the highest and lowest quality layers, skipping any in the middle. This is useful for providing a high-quality option and a low-bandwidth fallback without the overhead of a third stream.

  • If a device is capable of producing a h720p_w960p stream, setting maxLayer : 2 will result in two streams being sent: the highest quality (h720p_w960p) and the lowest quality (e.g. h180p_w240p), ensuring a fallback for poor connections.

info

To use the maxLayer parameter, multiStream must be set to true. The multiStream flag enables simulcasting (sending multiple quality streams), while maxLayer specifies how many streams to send.

note

• If the video width or height is 960 or higher, it will create 3 layers.
• If it’s 480 or higher and less than 960, it will create 2 layers.
• If it’s below 480, it will create only 1 layer.
  • For example, if you set maxLayer to 3 but your video is 320×180, it will only create 1 layer.

What is codec?

note

The codec parameter is supported from version 0.10.2 onward.

codec specifies the video codec used to encode your stream when you join the meeting. A codec determines how your video is compressed before it's sent to other participants, and each one strikes a different balance between video quality, CPU usage, browser/device support, and multiStream (simulcast) capability. VideoSDK supports four codecs — VP8 (default), H264, VP9, and AV1 — set via VideoSDK.Constants.VideoCodec.

Which codec should I use?

Codec	Quality / Compression	CPU	Support	multiStream	Recommended Use
VP8	Good	Low	Very Broad	✅	Default choice for maximum compatibility
H.264	Good	Low	Broad	✅	Mobile and enterprise environments
VP9	Better	Medium-High	Modern Browsers	❌	Improved video quality on modern devices
AV1	Best	High*	Growing	❌	Highest video quality where supported

• Hardware acceleration can significantly reduce CPU usage.

caution

Currently, VP9 and AV1 do not support multiStream. If you pass VP9 or AV1 with multiStream: true, VideoSDK will automatically set multiStream to false and emit the ERROR_MULTISTREAM_NOT_SUPPORTED error event.

danger

AV1 has limited browser support — in our testing it failed on some versions of Safari (macOS) and mobile web (Chrome and Safari on iPhone). Test it on your target browsers and devices before using it.

Codec fallback behaviour

VideoSDK automatically handles codec compatibility, so an unsupported codec never interrupts a meeting.

• When video cannot be sent using the provided codec

  If your browser or device does not support the provided codec, VideoSDK automatically changes the codec to VP8 and emits the ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED error event.

  meeting.on("error", (error) => {
    if (error.code === "ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED") {
      // VideoSDK switched the codec to VP8 because the provided codec is not supported
    }
  });

• When video cannot be received using the provided codec

  If a participant's device or browser does not support the codec currently being used in the meeting, the participant receives the CODEC_NOT_SUPPORTED error through the meeting's onError callback.

  meeting.on("error", (error) => {
    if (error.code === "CODEC_NOT_SUPPORTED") {
      // Current codec is not supported on this participant's device/browser
    }
  });

  VideoSDK then switches all participants to VP8, and every participant receives the codec-changed event through the Meeting class.

  meeting.on("codec-changed", (data) => {
    console.log("Current codec:", data.currentCodec);
    console.log("Unsupported remote codec:", data.unsupportedRemoteCodec);
    console.log("Kind:", data.kind);
    console.log("Caused by participant:", data.causedBy);
    console.log("Previous codec:", data.previousCodec);
  });

info

This automatic codec switching is controlled by the codecSwitchEnabled flag in initMeeting() (default: true). If you set it to false, VideoSDK will not switch the codec to VP8 when a codec is unsupported, and affected participants may see a black screen instead of the video.

How to Setup a Custom Video Track ?

The custom track can be configured both before and after the meeting is initialized. Following are the methods that help in doing so:

1. Setting up a Custom Track during the initialization of a meeting
2. Setting up a Custom Track with methods

1. Setting up a Custom Track during the initialization of a meeting

If you are enabling the webcam (webcamEnabled: true) in the config of the initMeeting method and wish to use custom tracks from the start of the meeting, you can pass a custom track in the config as demonstrated below.

caution

Custom Track will not apply on webcamEnabled: false configuration.

Example

let customVideoTrack = await VideoSDK.createCameraVideoTrack({
  optimizationMode: "motion",
  encoderConfig: "h540p_w960p",
  facingMode: "environment",
  multiStream: true,
  bitrateMode: VideoSDK.Constants.BitrateMode.HIGH_QUALITY,
  maxLayer: 2,
  codec: VideoSDK.Constants.VideoCodec.H264,
});

// Meeting Init
meeting = VideoSDK.initMeeting({
  meetingId: meetingId, // required
  name: name, // required
  //If true, it will use the passed custom track to turn webcam on
  webcamEnabled: true,
  //Pass the custom video track here
  customCameraVideoTrack: customVideoTrack,
});

2. Setting up a Custom Track with methods

To switch tracks during the meeting, you need to pass the MediaStream in the enableWebcam() method of the meeting object.

tip

Make sure to call the disableWebcam() method before you create a new track as it may lead to unexpected behavior.

Example

let meeting;

// Initialize Meeting
meeting = VideoSDK.initMeeting({
  // ...
});

const enableWebcamBtn = document.getElementById("enableWebcamBtn");
enableWebcamBtn.addEventListener("click", async () => {
  let customVideoTrack = await VideoSDK.createCameraVideoTrack({
    optimizationMode: "motion",
    encoderConfig: "h540p_w960p",
    facingMode: "environment",
    multiStream: true,
    bitrateMode: VideoSDK.Constants.BitrateMode.HIGH_QUALITY,
    maxLayer: 2,
    codec: VideoSDK.Constants.VideoCodec.H264,
  });

  // Enable Webcam in Meeting
  meeting?.enableWebcam(customVideoTrack);
});

info

Using custom video tracks is not just limited to the video tracks created using the createCameraVideoTrack method. You can use any MediaStream object as a replacement which can include a custom canvas track created by you.

Which Configuration is suitable for me ?

In this section, the focus is on understanding participant size and platform-wise encoder(Resolution) and multiStream configuration.

1. For Desktop Browser

2. For Mobile Browser

3. For Desktop + Mobile Browser

Custom Screen Share Track

This feature enables the customization of screenshare streams with enhanced optimization modes and predefined encoder configuration (resolution + FPS) for specific use cases, which can then be sent to other participants.

How to Create a Custom Screen Share Track ?

• You can create a Video Track using the createScreenShareVideoTrack() method.
• This method enables the creation of a video track with different encoding parameters and optimization modes.

Example

let customShareTrack = await VideoSDK.createScreenShareVideoTrack({
  optimizationMode: "motion", // "text" | "detail",  Default : "motion"

  // This will accept the  height & FPS of video you want to capture.
  encoderConfig: "h720p_15fps", //  `h360p_30fps` | `h1080p_30fps` // Default : `h720p_15fps`
});

You can learn about optimizationMode from here What is optimizationMode ?

How to Setup a Custom Screen Share Track ?

In order to switch tracks during the meeting, you have to pass the MediaStream in the **enableScreenShare() method of the meeting object.

note

Make sure to call disableScreenShare() before you create a new track as it may lead to unexpected behavior.

Example

let meeting;

// Initialize Meeting
meeting = VideoSDK.initMeeting({
  // ...
});

const enableScreenShareBtn = document.getElementById("enableScreenShareBtn");
enableScreenShareBtn.addEventListener("click", async () => {
  let customShareTrack = await VideoSDK.createScreenShareVideoTrack({
    optimizationMode: "motion",
    encoderConfig: "h720p_15fps",
  });

  meeting?.enableScreenShare(customShareTrack);
});

API Reference

The API references for all the methods and events utilized in this guide are provided below.

• Custom Video Track
• Custom Screen Share Track
• Codec-changed