Video Processor Android Setup - React Native

With a custom video processor, your own native code changes each camera frame before it is sent. This guide offers step-by-step instructions for creating and implementing a custom video processor for frame manipulation on Android.

Installation

• To create a custom video processor, you only need the VideoSDK React Native SDK. The native processor classes ship with it, so no other package is needed. Install it in your React Native app using either npm or yarn.

NPM

npm install "@videosdk.live/react-native-sdk@2.0.0"

YARN

yarn add "@videosdk.live/react-native-sdk@2.0.0"

• Complete the SDK's Android setup. React Native links the SDK automatically and your app's native code can use its processor classes, so you do not need to add a Gradle dependency.

Step-by-Step Guide: Integrating Native Video Processor in React Native

Step 1: Create Your Own Processor Code

• First, create a class that implements the VideoFrameProcessor interface from live.videosdk.rtc.android.listeners. Implement its onFrameCaptured() method, which receives each camera frame before it is encoded.
• Add your custom logic in onFrameCaptured() to apply transformations or effects to the frame. Once the effect has been applied, return the processed frame.

Kotlin

YourOwnBackgroundProcessor.kt

package com.yourOwnPackage

import live.videosdk.rtc.android.listeners.VideoFrameProcessor
import videosdk.org.webrtc.VideoFrame

class YourOwnBackgroundProcessor : VideoFrameProcessor {
  // Called for every camera frame, on the camera thread
  override fun onFrameCaptured(videoFrame: VideoFrame): VideoFrame {
    // Add your custom code here to apply the effect and return the processed VideoFrame.
    // Returning videoFrame sends the frame unchanged.
    return videoFrame
  }
}

Java

YourOwnBackgroundProcessor.java

package com.yourOwnPackage;

import live.videosdk.rtc.android.listeners.VideoFrameProcessor;
import videosdk.org.webrtc.VideoFrame;

public class YourOwnBackgroundProcessor implements VideoFrameProcessor {

    // Called for every camera frame, on the camera thread
    @Override
    public VideoFrame onFrameCaptured(VideoFrame videoFrame) {
        // Add your custom code here to apply the effect and return the processed VideoFrame.
        // Returning videoFrame sends the frame unchanged.
        return videoFrame;
    }
}

• onFrameCaptured() runs on the camera thread for every frame. Keep the work shorter than one frame interval, or frames are dropped.
• Each frame is an I420 VideoFrame. Import it from videosdk.org.webrtc, not from org.webrtc.
• Return videoFrame unchanged, or a new VideoFrame with its own buffer. If your effect rotates the picture, set the rotation on the frame you return.
• The SDK releases both the frame it passes in and the frame you return. Do not release them yourself, and do not keep a reference to them after the call.

Step 2: Register your Processor

• Next, create a class VideoEffectModule that extends ReactContextBaseJavaModule. This class will be used in your React Native module to interact with the native Android code.
• In this class, register your custom video processor using the register() method of VideoProcessorRegistry from live.videosdk.rncore. Provide a String representing the unique processor name along with an instance of your processor to complete the registration. This processor name will be used later to apply the effects offered by your processor. Registering another processor under the same name replaces the first one.

Kotlin

VideoEffectModule.kt

package com.yourOwnPackage

import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.bridge.ReactContextBaseJavaModule
import com.facebook.react.bridge.ReactMethod
import live.videosdk.rncore.VideoProcessorRegistry

class VideoEffectModule(reactContext: ReactApplicationContext) :
  ReactContextBaseJavaModule(reactContext) {

  override fun getName(): String = NAME

  @ReactMethod
  fun registerProcessor(processorName: String) {
    // Create an instance of your Processor
    val bgProcessor = YourOwnBackgroundProcessor()
    // Register your processor with a unique name and its instance
    VideoProcessorRegistry.register(processorName, bgProcessor)
  }

  companion object {
    const val NAME = "VideoEffectModule"
  }
}

Java

VideoEffectModule.java

package com.yourOwnPackage;

import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import live.videosdk.rncore.VideoProcessorRegistry;

public class VideoEffectModule extends ReactContextBaseJavaModule {

    public static final String NAME = "VideoEffectModule";

    public VideoEffectModule(ReactApplicationContext reactContext) {
        super(reactContext);
    }

    @Override
    public String getName() {
        return NAME;
    }

    @ReactMethod
    public void registerProcessor(String processorName) {
        // Create an instance of your Processor
        YourOwnBackgroundProcessor bgProcessor = new YourOwnBackgroundProcessor();
        // Register your processor with a unique name and its instance
        VideoProcessorRegistry.register(processorName, bgProcessor);
    }
}

• Add VideoEffectModule to a ReactPackage, and add that package in MainApplication, as shown in the React Native guide to registering an Android native module.

• Now, in your React Native app, you can register the processor using the module. Call it once, for example when your app starts.

app.js

import { NativeModules } from "react-native";

const { VideoEffectModule } = NativeModules;

function registerProcessor() {
  VideoEffectModule.registerProcessor("ProcessorName");
}

Step 3: Apply the Processor

• Once you have registered the processor, you can use it throughout the entire app lifecycle. To apply the effect provided by the processor, use the setVideoProcessor() method of the useRoom hook. This function requires the name of the processor that was used during registration, and returns a Promise.
• Call it after join(). Before that, no room exists and the call rejects. It also rejects with ERROR_INVALID_PARAMETER (3073) when no processor is registered under that name.
• Applying a processor replaces the one that is already applied.

app.js

import { useRoom } from "@videosdk.live/react-native-sdk";

const { setVideoProcessor } = useRoom();

async function applyProcessor() {
  try {
    await setVideoProcessor("ProcessorName");
  } catch (error) {
    console.warn("Video processor:", error.code, error.message);
  }
}

note

The processor receives no frames while video effects run with pipeline: "GRAPH" in setVideoEffects().

Step 4: Remove the Processor

• You can remove the processor when you no longer need the effect. To do this, use the removeVideoProcessor() method of the useRoom hook. It removes the processor that is currently applied, and returns a Promise. Like setVideoProcessor(), it rejects before join().
• The processor applies to your whole app, not to one room. It stays applied after you leave, so your next room uses it too. Remove it before you leave if you do not want that.

app.js

import { useRoom } from "@videosdk.live/react-native-sdk";

const { removeVideoProcessor } = useRoom();

async function removeProcessor() {
  try {
    await removeVideoProcessor();
  } catch (error) {
    console.warn("Video processor:", error.code, error.message);
  }
}