Video Processor - Android

VideoSDK lets you add custom effects to your camera video before it's sent during video calls.

Understanding Video Processing

Video processing involves manipulating raw video data (frames) captured by your camera. You can write your own processor to change the camera video in rooms.

tip

For auto-framing, low-light correction, a stylized look or a color grade, you don't need your own processor. Use the SDK's built-in camera effects, which you set with VideoSDK.setVideoEffects(). See Video Effects.

The VideoFrameProcessor Interface

The VideoFrameProcessor interface defines a way for your code to interact with the video stream. It essentially lets you intercept each frame before it's sent and apply your processing logic.

Key Function: onFrameCaptured

Whenever a new video frame is captured by your camera, the onFrameCaptured function is called. This function provides you with the raw frame data, allowing you to modify it as needed.

• The frame is a videosdk.org.webrtc.VideoFrame with an I420 buffer.
• onFrameCaptured runs on the camera thread for every frame. Keep the work shorter than one frame interval, or frames are dropped.
• A processor costs CPU even when it returns the frame unchanged.
• Only camera frames pass through the processor. Screen-share frames don't.
• The processor receives no frames while camera effects run on Pipeline.GRAPH. Use either a processor or camera effects, not both.

How it Works:

1. Implement the Interface: Create a class that implements the VideoFrameProcessor interface. Within this class, define the onFrameCaptured function.

2. Process the Frame: In the onFrameCaptured function, access the raw video frame data. Apply your desired effects or manipulations to this data.

3. Return the Processed Frame: After processing the frame, return the modified version to VideoSDK. This processed frame will then be transmitted during the video call. You can return the same frame or a new one. The SDK releases both the frame it passed in and the frame you return, so don't release either of them, and don't keep them after onFrameCaptured returns. If onFrameCaptured throws, the original frame is sent.

4. Apply the Processor: To apply a processor, pass an object of your class as a parameter to the setVideoProcessor() method of the Room class. It works before join() too, so a pre-join preview is processed. Only one processor runs at a time, so setting another one replaces it. It stays applied until you remove it or leave the room. Set it again for the next room.

5. Remove the Processor: To stop applying the selected video processor, use the removeVideoProcessor() method of the Room class.

import live.videosdk.rtc.android.listeners.VideoFrameProcessor
import videosdk.org.webrtc.VideoFrame

class MyVideoProcessor : VideoFrameProcessor {
    override fun onFrameCaptured(videoFrame: VideoFrame): VideoFrame {
        // Read or change the frame here, then return it or a new frame.
        // Don't release either frame: the SDK releases both.
        return videoFrame
    }
}

// Apply the processor
room.setVideoProcessor(MyVideoProcessor())

// Remove it when you no longer need it
room.removeVideoProcessor()

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoProcessor()
• removeVideoProcessor()