Video Processor - Flutter

VideoSDK allows you to add custom effects to your video stream before it's transmitted during video calls. This lets you enhance your video in creative ways or achieve specific functionalities.

Understanding Video Processing

Video processing involves manipulating raw video data (frames) captured by your camera. You can create a native processor for Android to change each camera frame before it is sent during rooms.

info

• Custom video processors are supported only on Android. On iOS, room.setVideoProcessor() takes the file path or URL of an image and replaces the camera background with it, on iOS 15 or later. Learn more in Background image on iOS.
• For background blur and the other built-in video effects, you do not need a processor. They work on both Android and iOS.

The VideoFrameProcessor Interface

The VideoFrameProcessor interface, from the package live.videosdk.rtc.android.listeners, defines a way for your code to interact with the video stream. It essentially lets you intercept each camera frame before it's sent and apply your processing logic.

Key Function: onFrameCaptured

This is the heart of video processing with VideoSDK. Whenever a new video frame is captured by your camera, the onFrameCaptured function is called. This function provides you with the frame as a videosdk.org.webrtc.VideoFrame, allowing you to modify it as needed.

How it Works:

1. Implement the Interface: In your Android app, create a class that implements the VideoFrameProcessor interface. Within this class, define the onFrameCaptured function.

2. Process the Frame: In the onFrameCaptured function, access the raw video frame data. Apply your desired effects or manipulations to this data.

3. Return the Processed Frame: After processing the frame, return the modified version to VideoSDK. This processed frame will then be transmitted during the video call.

4. Register Your Processor: Create instances of all the processors you plan to use during the room and register them with the registerVideoProcessor() method of the native VideoSDK class (live.videosdk.videosdk.VideoSDK) during your app's initialization.

5. Apply the Processor: Select the desired processor and apply it with the setVideoProcessor() method of the Room, before or after you join.

6. Remove the Processor: Stop applying the selected video processor with the removeVideoProcessor() method of the Room, allowing you to switch to a different effect or processor as needed.

To write and register a processor, follow Video Processor Android Setup.

Built-in Video Effects

VideoSDK.setVideoEffects() applies built-in effects to the camera video you send: framing, low-light correction, style, color filter and background blur. It works on Android and iOS, before and after you join a room. Send the whole configuration each time, because an effect you leave out goes back to its default.

final Map<String, dynamic>? applied = await VideoSDK.setVideoEffects(
  const VideoEffectsConfig(
    lighting: VideoEffectsLighting(enabled: true),
    background: VideoEffectsBackground(mode: VideoEffectsBackgroundMode.BLUR),
  ),
);

if (applied?['applied'] != true) {
  print("Effects not running: ${applied?['reason']}");
}

• On Android, framing needs com.google.mlkit:face-detection:16.1.7, and background blur needs com.google.mlkit:segmentation-selfie:16.0.0-beta6 in your app's build.gradle. On iOS, background blur needs iOS 15 or later.
• On the default VideoEffectsPipeline.LEGACY, the built-in effects and a processor do not run together: the one you apply last replaces the other. room.removeVideoProcessor() also removes the built-in effects.

To learn more about the built-in effects, see Video Effects.

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoProcessor()
• removeVideoProcessor()
• registerVideoProcessor()
• setVideoEffects()