Video Processor Android Setup - Flutter

• VideoSDK allows you to add unique effects to your video stream before transmission during video calls with custom native video processing code. This guide offers step-by-step instructions for creating and implementing a custom video processor for frame manipulation on Android.

• The VideoFrameProcessor interface and the VideoFrame class come with the SDK, so you do not need to add any dependency. If you added io.github.webrtc-sdk:android to your build.gradle for a processor written for v4.0.1, remove it.

Step-by-Step Guide: Integrating Native Video Processor in Flutter

Step 1: Create Your Own Processor Code

• First, create a class that implements the VideoFrameProcessor interface. In this class, define the onFrameCaptured function, which will contain the logic for manipulating video frames according to your needs. After applying the desired effect, make sure to return the processed frame.
• The SDK passes each frame in I420 format. It releases both the frame it passes in and the frame you return, so do not call release() on either, and do not keep a frame to return it again later.
• onFrameCaptured is called on the camera thread for every frame. Keep the work short, because a slow processor drops frames.

Kotlin

YourOwnBackgroundProcessor.kt

package com.yourOwnPackage

import live.videosdk.rtc.android.listeners.VideoFrameProcessor
import videosdk.org.webrtc.VideoFrame

class YourOwnBackgroundProcessor : VideoFrameProcessor {
    // Override the onFrameCaptured function to access VideoFrames
    override fun onFrameCaptured(videoFrame: VideoFrame): VideoFrame {
        // Function to manipulate frame as per requirement
        return applyEffect(videoFrame)
    }

    private fun applyEffect(videoFrame: VideoFrame): VideoFrame {
        // Add your custom code here to apply the effect and return the processed VideoFrame
        return videoFrame
    }
}

Java

YourOwnBackgroundProcessor.java

package com.yourOwnPackage;

import live.videosdk.rtc.android.listeners.VideoFrameProcessor;
import videosdk.org.webrtc.VideoFrame;

public class YourOwnBackgroundProcessor implements VideoFrameProcessor {
    //Override the onFrameCaptured function to get access to the VideoFrames
    @Override
    public VideoFrame onFrameCaptured(VideoFrame videoFrame) {
        //Function to manipulate frame as per requirement.
        VideoFrame processedVideoFrame = applyEffect(videoFrame);
        return processedVideoFrame;
    }
    private VideoFrame applyEffect(VideoFrame videoFrame) {
        // Add your custom code here to apply the effect and return the processed VideoFrame
        return videoFrame;
    }
}

Step 2: Register your Processor

• In your MainActivity, register your custom video processor using the registerVideoProcessor() method from the VideoSDK class during app initialization. Provide a string representing the unique processor name along with an instance of your processor to complete the registration. This processor name will be used later to apply the effects offered by your processor.

Kotlin

MainActivity.kt

package com.example.example

import io.flutter.embedding.android.FlutterActivity
import android.os.Bundle
import com.yourOwnPackage.YourOwnBackgroundProcessor
import live.videosdk.videosdk.VideoSDK

class MainActivity : FlutterActivity() {

    private lateinit var bgProcessor: YourOwnBackgroundProcessor

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        //Create an instance of your Processor 
        bgProcessor = YourOwnBackgroundProcessor()
        //Create an instance of the VideoSDK class
        val videoSDK = VideoSDK.getInstance()
        //Register your processor with a unique name and its instance
        videoSDK.registerVideoProcessor("ProcessorName", bgProcessor)
    }
}

Java

MainActivity.java

package com.example.example;

import io.flutter.embedding.android.FlutterActivity;
import android.os.Bundle;
import com.yourOwnPackage.YourOwnBackgroundProcessor;
import live.videosdk.videosdk.VideoSDK;

public class MainActivity extends FlutterActivity {

    private YourOwnBackgroundProcessor bgProcessor;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        //Create an instance of your Processor 
        bgProcessor = new YourOwnBackgroundProcessor();
        //Create an instance of the VideoSDK class
        VideoSDK videoSDK = VideoSDK.getInstance();
        //Register your processor with a unique name and its instance
        videoSDK.registerVideoProcessor("ProcessorName", bgProcessor);
    }
}

Step 3: Apply the Processor

• Once you have registered the processor during initialization, you can use it throughout the entire app lifecycle. To apply the effect provided by the processor, use the setVideoProcessor() method of the Room. This method requires the name of the processor that was used during registration. A name that was never registered makes the call throw.
• It works before and after you join, because the room exists from VideoSDK.createRoom().
• If you have registered multiple processors and wish to switch between them, simply call the setVideoProcessor() method with the name of the desired processor.
• Applying a processor replaces the built-in effects set with VideoSDK.setVideoEffects() on the default VideoEffectsPipeline.LEGACY.

import 'package:flutter/material.dart';
import 'package:videosdk/videosdk.dart';

class RoomScreen extends StatefulWidget {
  //Existing configuration
}

class _RoomScreenState extends State<RoomScreen> {
  late Room _room;

  @override
  void initState() {
    super.initState();
    //Existing configuration
  }

  @override
  Widget build(BuildContext context) {
    return Column(
      children:[
        // To apply the effect from the processor, provide the name used during registration.
        ElevatedButton(
          onPressed: () async {
            await _room.setVideoProcessor("ProcessorName");
          },
          child: const Text("Apply Video Processor"),
        ),
      ]
    );
  }
}

Step 4: Remove the Processor

• You can remove the processor when you no longer need the effect. To do this, use the removeVideoProcessor() method of the Room to eliminate the effect provided by the processor. This method will remove the effect of the currently active video processor.

import 'package:flutter/material.dart';
import 'package:videosdk/videosdk.dart';

class RoomScreen extends StatefulWidget {
  //Existing configuration
}

class _RoomScreenState extends State<RoomScreen> {
  late Room _room;

  @override
  void initState() {
    super.initState();
    //Existing configuration
  }

  @override
  Widget build(BuildContext context) {
    return Column(
      children:[
        // To remove the effect, remove the active processor.
        ElevatedButton(
          onPressed: () async {
            await _room.removeVideoProcessor();
          },
          child: const Text("Remove Video Processor"),
        ),
      ]
    );
  }
}

API Reference

The API references for all the methods utilized in this guide are provided below.

• registerVideoProcessor()
• setVideoProcessor()
• removeVideoProcessor()