Video Processor - iOS

VideoSDK allows you to add custom effects to your video stream before it's transmitted during video calls.

Understanding Video Processing

Video processing involves manipulating the raw video data (frames) captured by your camera. VideoSDK provides a tool called VideoSDKVideoProcessor to achieve this.

The VideoSDKVideoProcessor Protocol

VideoSDKVideoProcessor protocol defines a way for your code to interact with the video stream. It essentially lets you intercept each frame before it's sent and apply your processing logic.

Key Function: onFrameReceived(pixelBuffer:)

Whenever a new video frame is captured by your camera, the onFrameReceived(pixelBuffer:) function is called. It gives you the frame as a BGRA CVPixelBuffer (kCVPixelFormatType_32BGRA), allowing you to modify it as needed.

• The function runs once for every camera frame, on a background queue. Keep the work fast, because a slow processor lowers the frame rate.
• The protocol also has onFrameReceived(frame:), which works on an RTCVideoFrame. Your app can't create an RTCVideoFrame, so implement onFrameReceived(pixelBuffer:). Implement only one of the two.

How it Works:

1. Implement the Protocol: You create a class that implements the VideoSDKVideoProcessor protocol. Within this class, you define the onFrameReceived(pixelBuffer:) function.

2. Process the Frame: Inside the onFrameReceived(pixelBuffer:) function, you access the frame's pixel buffer. You can then apply your desired effects or manipulations to it, for example with Core Image or Vision.

3. Return the Processed Frame: After processing the frame, return the processed CVPixelBuffer back to VideoSDK. This processed frame will then be transmitted during the video call. If you return nil, the frame is dropped and not sent, so return the input buffer when you have nothing to change.

Example: Grayscale Video

This example demonstrates a simple CustomVideoProcessor class that converts the video to grayscale:

class CustomVideoProcessor: VideoSDKVideoProcessor {

  func onFrameReceived(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    // Convert the frame to grayscale (replace with your desired processing)
    let processedBuffer = convertToGrayscale(pixelBuffer)
    return processedBuffer
  }

  private func convertToGrayscale(_ pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    // Implement grayscale conversion logic here
    // ... (grayscale conversion logic)
    return pixelBuffer
  }
}

Setting the Video Processor

Once you have your custom processor class, use the setVideoProcessor method of the Room class to activate it. This tells VideoSDK to use your processor for processing the video stream. setVideoProcessor is not async, so you call it without await.

Example:

class RoomViewController: UIViewController {
  var room: Room?

  @IBAction func setProcessorTapped(_ sender: Any) {
    let processor = CustomVideoProcessor()
    room?.setVideoProcessor(processor: processor)
  }
}

Removing the Processor

To stop applying the video effects, call removeVideoProcessor(). This will disable your custom processor and revert to the original video stream. Passing nil to setVideoProcessor does the same.

room?.removeVideoProcessor()

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoProcessor()
• removeVideoProcessor()
• VideoSDKVideoProcessor