Skip to main content
Version: 1.x.x

Optimize Video Tracks - Android

While optimizing for the best viewing experience, it is necessary to fine-tune the video tracks that are being used during the calls.

For the best fine-tuning experience, we have introduced the ability to pass a custom video track for the participant's media before and during the meeting.

  1. Custom Video Track
  2. Custom Screen Share Track

Custom Video Track​

This feature gives you powerful tools to deliver the best possible video stream for varying network conditions. You can specify a modern video quality profile (bitrateMode) and control simulcast behavior (maxLayer).

The SDK also supports legacy optimization modes (such as focusing on motion, text, or detail) and allows for the integration of background removal and video filters from external SDKs (e.g., Banuba) to further enhance the video.

How to Create a Custom Video Track ?​

  • You can create a Video Track using createCameraVideoTrack() method of VideoSDK.
  • This method can be used to create video track by specifying parameters like video resolution, camera facing mode, and the quality controls (bitrateMode and maxLayer).
  • You can choose video resolution from the below mentioned list of values for the encoder config:
ConfigResolutionFrame RateOptimized (kbps)Balanced (kbps)High Quality (kbps)
h144p_w176p176x14415 fps60100150
h240p_w320p320x24015 fps80150300
h360p_w640p360x64025 fps200400800
h480p_w640p640x48025 fps3006001000
h480p_w720p720x48030 fps4007001100
h720p_w960p960x72030 fps80013001800
h720p_w1280p1280x72030 fps100016002400
h1080p_w1440p1440x108030 fps200025003500

Example​

val videoCustomTrack: CustomStreamTrack = VideoSDK.createCameraVideoTrack(

  // This will accept the resolution (height x width) of video you want to capture.
  "h720p_w960p", // "h480p_w6400p" | "h720p_w1280p" ... // Default : "h480p_w720p"

  // It will specify whether to use front or back camera for the video track.
  "front", "back",  Default : "front"

  // We will discuss this parameter in next step.
  CustomStreamTrack.VideoMode.MOTION, // CustomStreamTrack.VideoMode.TEXT, CustomStreamTrack.VideoMode.DETAIL ,  Default : CustomStreamTrack.VideoMode.MOTION

  // multiStream - we will discuss this parameter in next step.
  false, // true

  // Pass Context
  this,

  // Optional: Specify a specific camera device.
  videoDeviceInfo, // Default: null

  // Optional: This specifies the maximum number of simulcast layers (maxLayer) to publish.
  2, // 3 |  Default: 3

  // Optional: This controls the video quality and bandwidth usage.
  BitrateMode.HIGH_QUALITY, // .BANDWIDTH_OPTIMIZED | .BALANCED Default: .BALANCED 

  // Optional: This specifies the codec to use for the meeting.
  VideoCodec.H264 // VideoCodec.VP8 | VideoCodec.VP9 | VideoCodec.AV1 , Default : VideoCodec.VP8
  )
caution

The capabilities of the device have a significant impact on how custom track configurations behave. Assuming a case where you set encoder configuration to 1080p but the camera only supports 720p, then encoder configuration will automatically switch to the highest resolution that the device can handle, which is 720p.

What is multiStream?​
  • It will specify if the stream should send multiple resolution layers or single resolution layer.

multiStream : true By default, VideoSDK sends multiple resolution video streams to the server (whether you are using custom video track or not), For instance, user device capability is 720p, so VideoSDK sends 720p along with 360p and 144p streams. This allows VideoSDK to deliver the appropriate stream to each participant based on their network bandwidth.

Multi Stream False

multiStream : false If you want to restrict the VideoSDK to send only one stream to maintain quality, you can set multiStream to false.

Multi Stream False

danger

setQuality would not have any effect if multiStream is set to false.

What is BitrateMode?​

BitrateMode is the key setting for video quality. It lets you decide whether to prioritize sharp details, save internet data, or find a good balance between the two.

You can choose from three distinct modes:

  • BitrateMode.BANDWIDTH_OPTIMIZED: Prioritizes lower bandwidth usage over video quality. Ideal for users with poor or unstable network conditions.
  • BitrateMode.BALANCED: The default setting. It provides a smart compromise between clear video and efficient bandwidth consumption, suitable for most use cases.
  • BitrateMode.HIGH_QUALITY: Prioritizes video quality over bandwidth usage. Use this when high-fidelity video is essential and viewers are expected to have strong network connections.
What is MaxLayer?​

MaxLayer is an advanced parameter that gives you direct control over simulcast, which is the process of sending multiple versions (or "layers") of the same video stream at different resolutions and qualities simultaneously.

  • maxLayer = 3 (Default): Publishes all available layers (e.g., high, medium, and low quality). This provides the maximum adaptability for viewers on diverse and unpredictable networks.
  • maxLayer = 2: Intelligently publishes only the highest and lowest quality layers, skipping any in the middle. This is useful for providing a high-quality option and a low-bandwidth fallback without the overhead of a third stream.
    • If a device is capable of producing a h720p_w960p stream, setting maxLayer = 2 will result in two streams being sent: the highest quality (h720p_w960p) and the lowest quality (e.g. h144p_w176p), ensuring a fallback for poor connections.
info

To use the maxLayer parameter, multiStream must be set to true. The multiStream flag enables simulcasting (sending multiple quality streams), while maxLayer specifies how many streams to send.

note
  • If the video width or height is 960 or higher, it will create 3 layers.
  • If it’s 360 or higher and less than 960, it will create 2 layers.
  • If it’s below 360, it will create only 1 layer.
    • For example, if you set MaxLayer to 3 but your video is 320×240, it will only create 1 layer.
What is optimizationMode?​

  • It will specify the optimization mode for the video track being generated.

  • motion : This type of track should more focus on motion video. For example, webcam video, movies or video games.

    • This type of track will degrade resolution in order to maintain frame rate.

  • text : This type of track should more focus on significant sharp edges and areas of consistent color that can change frequently. For example, presentations or web pages with text content.

    • This type of track will degrade frame rate in order to maintain resolution.

  • detail : This type of track should more focus on details of the video. For example, presentations, painting or line art.

    • This type of track will degrade frame rate in order to maintain resolution.
What is codec?​
note

The codec parameter is supported from version 1.3.0 onward.

codec specifies the video codec used to encode your stream when you join the meeting. A codec determines how your video is compressed before it's sent to other participants, and each one strikes a different balance between video quality, CPU usage, device support, and multiStream (simulcast) capability. VideoSDK supports four codecs — VP8 (default), H264, VP9, and AV1 — set via VideoCodec.

Which codec should I use?​
CodecQuality / CompressionCPUSupportmultiStreamRecommended Use
VP8GoodLowVery Broad✅Default choice for maximum compatibility
H.264GoodLowBroad❌Mobile and enterprise environments
VP9BetterMedium-HighModern Devices❌Improved video quality on modern devices
AV1BestHigh*Growing❌Highest video quality where supported
  • Hardware acceleration can significantly reduce CPU usage.
caution

Currently, H264, VP9, and AV1 do not support multiStream. If you pass H264, VP9, or AV1 with multiStream: true, VideoSDK will automatically set multiStream to false and emit the ERROR_MULTISTREAM_NOT_SUPPORTED error event.

danger

AV1 has limited device support — in our testing it failed on some versions of Safari (macOS) and mobile web (Chrome and Safari on iPhone). Test it on your target browsers and devices before using it.

Codec fallback behaviour​

VideoSDK automatically handles codec compatibility, so an unsupported codec never interrupts a meeting.

  • When video cannot be sent using the provided codec

    If your device does not support the provided codec, VideoSDK automatically changes the codec to VP8 and emits the ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED error event.

    private val meetingEventListener: MeetingEventListener = object : MeetingEventListener() {
      //..
      override fun onError(error: JSONObject) {
      try {
          val errorCodes: JSONObject = VideoSDK.getErrorCodes()
          val code = error.getInt("code")
          if (code == errorCodes.getInt("ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED")) {
            // VideoSDK switched the codec to VP8 because the provided codec is not supported
          }
      } catch (e: Exception) {
              e.printStackTrace()
          }
      }
    }

  • When video cannot be received using the provided codec

    If a participant's device does not support the codec currently being used in the meeting, the participant receives the CODEC_NOT_SUPPORTED error through the meeting's onError callback.

    private val meetingEventListener: MeetingEventListener = object : MeetingEventListener() {
      //..
      override fun onError(error: JSONObject) {
      try {
          val errorCodes: JSONObject = VideoSDK.getErrorCodes()
          val code = error.getInt("code")
          if (code == errorCodes.getInt("CODEC_NOT_SUPPORTED")) {
            // VideoSDK switched the codec to VP8 because the provided codec is not supported
          }
      } catch (e: Exception) {
              e.printStackTrace()
          }
      }
    }

    VideoSDK then switches all participants to VP8, and every participant receives the codec-changed event through the Meeting class.

    private val meetingEventListener: MeetingEventListener = object : MeetingEventListener() {
        // ...
        override fun onCodecChanged(info: CodecChangeInfo) {
          Log.d("TAG", "Current codec: " + info.currentCodec)
          Log.d("TAG", "Unsupported remote codec: " + info.unsupportedRemoteCodec)
          Log.d("TAG", "Media kind: " + info.kind)
          Log.d("TAG", "Participant that caused the codec switch: " + info.causedBy)
          Log.d("TAG", "Previous codec: " + info.previousCodec)
        }
    };
info

This automatic codec switching is controlled by the codecSwitchEnabled flag in initMeeting() (default: true). If you set it to false, VideoSDK will not switch the codec to VP8 when a codec is unsupported, and affected participants may see a black screen instead of the video.

What is observer?​
  • If you want to use video filter from external SDK(e.g., Banuba) then pass instance of CapturerObserver in this parameter.
note

For Banuba integraion with VideoSDK, please visit Banuba Intergation with VideoSDK

How to Setup a Custom Video Track ?​

The custom track can be set up both before and after the initialization of the meeting.

  1. Setting up a Custom Track during the initialization of a meeting
  2. Setting up a Custom Track with methods
1. Setting up a Custom Track during the initialization of a meeting​

If you are passing webcamEnabled: true in the initMeeting of VideoSDK and want to use custom tracks from start of the meeting, you can pass custom track in the initMeeting as shown below.

caution

Custom Track will not apply on webcamEnabled: false configuration.

Example​
override fun onCreate(savedInstanceState: Bundle?) {
  //..

  val customTracks: MutableMap<String, CustomStreamTrack> = HashMap()
  val videoCustomTrack: CustomStreamTrack = 
    VideoSDK.createCameraVideoTrack(
    "h720p_w960p",
     "front", 
     CustomStreamTrack.VideoMode.MOTION, 
      true, 
      this,
      2, 
      BitrateMode.HIGH_QUALITY,
      VideoCodec.H264)

  customTracks["video"] = videoCustomTrack  //Key must be "video"

  // create a new meeting instance
  val meeting = VideoSDK.initMeeting(
    this@MainActivity, meetingId, participantName,
    //MicEnabled
    true,
    //WebcamEnabled , If true, it will use the passed custom track to turn webcam on
    true,
    // ParticipantId
    null,
    // Mode
    null,
    // MultiStream
    false,
    //Pass the custom tracks here
    customTracks,
    //MetaData
    null
  )
}

2. Setting up a Custom Track with methods​

In order to switch tracks during the meeting, you have to pass the CustomStreamTrack in the enableWebcam() method of Meeting.

tip

Make sure to call disableWebcam() before you create a new track as it may lead to unexpected behavior.

Example​
 val videoCustomTrack: CustomStreamTrack = 
    VideoSDK.createCameraVideoTrack(
    "h720p_w960p",
     "front", 
     CustomStreamTrack.VideoMode.MOTION, 
      true, 
      this,
      2, 
      BitrateMode.HIGH_QUALITY,
      VideoCodec.H264)

meeting!!.enableWebcam(customStreamTrack)

Which Configuration is suitable for Device ?​

In this section, we will understand participant size wise encoder(Resolution) and multiStream configuration.

Custom Screen Share Track​

This feature can be used to customise screenshare streams with enhanced optimization mode and predefined encoderConfig(Resolution + FPS) for specific use-case and send it to other participants.

How to Create Custom Screen Share Track ?​

  • You can create a Screen Share track using createScreenShareVideoTrack() method of VideoSDK.
  • This method can be used to create video track using different encoding parameters and optimization mode.

Example​

// data is received from onActivityResult method.
VideoSDK.createScreenShareVideoTrack(

  // This will accept the  height & FPS of video you want to capture.
  "h720p_15fps", //  `h360p_30fps` | `h1080p_30fps` // Default : `h720p_15fps`

  // It is Intent received from onActivityResult when user provide permission for ScreenShare.
  data,

  // Pass Conext
  this)

  //Callback to this listener will be made when track is ready with CustomTrack as parameter
  {
    track ->
    meeting!!.enableScreenShare(track)
  }

How to Setup Custom Screen Share Track ?​

In order to switch tracks during the meeting, you have to pass the CustomStreamTrack in the enableScreenShare() method of Meeting.

note

Make sure to call disableScreenShare() before you create a new track as it may lead to unexpected behavior.

Example​
@TargetApi(21)
private fun askPermissionForScreenShare() {
  val mediaProjectionManager = application.getSystemService<Any>(
      Context.MEDIA_PROJECTION_SERVICE
  ) as MediaProjectionManager
  startActivityForResult(
      mediaProjectionManager.createScreenCaptureIntent(), CAPTURE_PERMISSION_REQUEST_CODE
  )
}

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
  super.onActivityResult(requestCode, resultCode, data)
  if (requestCode != CAPTURE_PERMISSION_REQUEST_CODE) return
  if (resultCode == RESULT_OK) {
    VideoSDK.createScreenShareVideoTrack("h720p_15fps", data, this) { track ->
        meeting!!.enableScreenShare(track)
    }
  }
}

API Reference​

The API references for all the methods and events utilised in this guide are provided below.

Got a Question? Ask us on discord

Last updated on by pavan901
Previous
Optimize Audio Track
Next

Was this helpful?