How to integrate Banuba SDK with Android Video SDK?

The Banuba SDK is a software development kit that enables developers to integrate advanced augmented reality (AR) features, such as real-time face filters and effects, into their apps.

To integrate the Banuba SDK with Android Video SDK, follow these steps:

Step 1: Get Banuba Token

• To begin working with the Banuba SDK in your project, you'll need Banuba's client token.
• You can obtain this token by following the instructions in Banuba's documentation at banuba.com

Step 2: Add Banuba SDK dependencies

• Add custom maven repo to your settings.gradle.kts:

settings.gradle.kts

dependencyResolutionManagement {
    repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
    repositories {
        google()
        mavenCentral()
        maven {
            name = "GitHubPackages"
            url = uri("https://maven.pkg.github.com/sdk-banuba/banuba-sdk-android")
            credentials {
                username = "sdk-banuba"
                password = "<BANUBA_GITHUB_TOKEN>"
            }
        }
    }
}

• Replace <BANUBA_GITHUB_TOKEN> with the GitHub token for Banuba's Maven repository. Get it from Banuba, along with your client token.

• Add Banuba SDK and VideoSDK dependencies in app/build.gradle.kts file.

app/build.gradle.kts

dependencies {
    // Banuba SDK dependencies
    implementation("com.banuba.sdk:banuba_sdk:1+")
    // Video SDK dependencies
    implementation("live.videosdk:rtc-android-sdk:3.0.0")
    // other app dependencies
}

• For the rest of the VideoSDK setup, such as the permissions your app must declare, see Integrate Android SDK.

Step 3: Initialization of Banuba SDK

1. Create a MainApplication class that extends android.app.Application.

2. Within the MainApplication class, create a KEY field to store the token you generated from Banuba. This token will be used to initialize the BanubaSdkManager.

MainApplication.kt

class MainApplication : Application() {
    override fun onCreate() {
        super.onCreate()

        //...
        val key = "Banuba Client Token"
        BanubaSdkManager.initialize(applicationContext, key)
    }

    override fun onTerminate() {
        BanubaSdkManager.deinitialize()
        super.onTerminate()
    }
}

Step 4: Add effects into your project

• Create an effects folder under the assets directory in your project and add the filters you want to use. You can download test filters from here Effects

Step 5: Integration of BanubaProcessor

• The BanubaProcessor class will implement the VideoFrameProcessor interface from VideoSDK.
• VideoSDK calls onFrameCaptured() for every camera frame. Change the VideoFrame there to apply the effect.
• Import VideoFrame, JavaI420Buffer, EglBase and SurfaceTextureHelper from the videosdk.org.webrtc package. The frame you receive has an I420 buffer.
• VideoSDK releases both the frame it passes in and the frame you return. So BanubaProcessor returns a new VideoFrame each time, with its own reference to Banuba's latest output, and sends the original frame until Banuba has produced one.

BanubaProcessor.kt

import android.util.Size
import com.banuba.sdk.effect_player.CameraOrientation
import com.banuba.sdk.frame.FramePixelBufferFormat
import com.banuba.sdk.input.StreamInput
import com.banuba.sdk.output.FrameOutput
import com.banuba.sdk.player.Player
import com.banuba.sdk.types.FrameData
import com.banuba.sdk.types.FullImageData
import live.videosdk.rtc.android.listeners.VideoFrameProcessor
import videosdk.org.webrtc.EglBase
import videosdk.org.webrtc.JavaI420Buffer
import videosdk.org.webrtc.SurfaceTextureHelper
import videosdk.org.webrtc.VideoFrame

class BanubaProcessor : VideoFrameProcessor {
    private var mPlayer: Player = Player()
    private var mFrameOutput: FrameOutput? = null
    private val eglContext: EglBase.Context = EglBase.create().eglBaseContext
    private var mSurfaceTextureHelper: SurfaceTextureHelper? = null
    // latest frame from Banuba: written on the handler thread, read on the camera thread
    private val frameLock = Any()
    private var processedFrame: VideoFrame? = null
    private val streamInput: StreamInput = StreamInput()

    init {
        mSurfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglContext, false)
    }

    // set a visual effect in the Banuba Player
    fun setEffect(effect: String) {
        mPlayer.loadAsync("effects/$effect")
        mPlayer.play()
    }

    // remove any applied effects
    fun removeEffect() {
        mPlayer.loadAsync("")
    }

    fun removePlayer() {
        // close the player when it's no longer needed
        mPlayer.close()
    }

    // method to initialize the frame output for video processing
    private fun initOutput(): VideoFrame? {
        val handler = mSurfaceTextureHelper!!.handler
        mFrameOutput = FrameOutput { iOutput, framePixelBuffer ->
            handler.post {
                val i420Buffer = JavaI420Buffer.wrap(
                    framePixelBuffer.width,
                    framePixelBuffer.height,
                    framePixelBuffer.getPlane(0),
                    framePixelBuffer.getBytesPerRowOfPlane(0),
                    framePixelBuffer.getPlane(1),
                    framePixelBuffer.getBytesPerRowOfPlane(1),
                    framePixelBuffer.getPlane(2),
                    framePixelBuffer.getBytesPerRowOfPlane(2),
                    null
                )
                synchronized(frameLock) {
                    processedFrame?.release()
                    processedFrame = VideoFrame(i420Buffer, 0, System.nanoTime())
                }
            }
        }
        mFrameOutput!!.setFormat(FramePixelBufferFormat.I420_BT601_FULL)

        handler.postDelayed({
            mFrameOutput?.close()
        }, 2000)

        // attach the FrameOutput to the player for rendering
        mPlayer.use(mFrameOutput)

        // VideoSDK releases the frame you return, so give it a new frame with its own reference
        synchronized(frameLock) {
            val frame = processedFrame ?: return null
            frame.buffer.retain()
            return VideoFrame(frame.buffer, frame.rotation, frame.timestampNs)
        }
    }

    // called whenever a new video frame is captured
    override fun onFrameCaptured(videoFrame: VideoFrame): VideoFrame {
        // Determine the camera orientation based on the rotation of the video frame
        val cameraOrientation = CameraOrientation.values()[(videoFrame.rotation / 90) % 4]
        val orientation = FullImageData.Orientation(cameraOrientation, true, 0)
        val frameData = FrameData.create() ?: return videoFrame
        val i420Buffer = videoFrame.buffer.toI420() ?: return videoFrame

        // create FullImageData using the frame data and orientation
        val fullImageData = FullImageData(
            Size(i420Buffer.width, i420Buffer.height),
            i420Buffer.dataY,
            i420Buffer.dataU,
            i420Buffer.dataV,
            i420Buffer.strideY,
            i420Buffer.strideU,
            i420Buffer.strideV,
            1, 1, 1,
            orientation
        )

        frameData.addFullImg(fullImageData)
        // releases only the reference toI420() added; VideoSDK releases videoFrame
        i420Buffer.release()
        streamInput.push(frameData, videoFrame.timestampNs)

        // attach the StreamInput to the player for further processing
        mPlayer.use(streamInput)

        // return the processed frame, or the original one until Banuba has produced one
        return initOutput() ?: videoFrame
    }
}

Step 6: Use filters in the Room

• First, create an instance of BanubaProcessor class and pass it to the setVideoProcessor() method of the Room class.
• Next, use the setEffect method on the instance of BanubaProcessor class to apply the desired effect.
• To apply video filters as soon as the room is joined, you can call setEffect method in onRoomJoined event. onRoomJoined can fire again, for example after a reconnect, so create the processor once and reuse it.
• To stop sending the filtered video, call room.removeVideoProcessor(). The processor stays applied until you remove it or leave the room.

MainActivity.kt

// create the processor once and reuse it
private val banubaProcessor by lazy { BanubaProcessor() }

private val roomEventListener = object : RoomEventListener() {
    override fun onRoomJoined(switchRoomId: String?) {
        room.setVideoProcessor(banubaProcessor)
        //here you can pass any effect that you want to apply
        banubaProcessor.setEffect("PineappleGlasses")
    }
}

tip

To learn how video processors work in VideoSDK, see Video Processor.