Video Processor - iOS

A video processor changes the frames of your camera before they are sent to other participants, for example to replace the background. Set it with setVideoProcessor(processor:) of the Room class. To remove it, call removeVideoProcessor(), or pass nil to setVideoProcessor(processor:). The processor stays set until you remove it, also for later rooms.

For step-by-step guides, see Video Processor and Virtual Background.

VideoSDKVideoProcessor

protocol VideoSDKVideoProcessor

Conform to it to write your own processor. Implement onFrameReceived(pixelBuffer:).

onFrameReceived(pixelBuffer:)

func onFrameReceived(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer?

• Called for every camera frame before it is sent, on a background queue. Return quickly, because a slow processor lowers the frame rate.
• pixelBuffer is a BGRA buffer (kCVPixelFormatType_32BGRA).
• Return the processed buffer. It is scaled to the size of the video that is sent.
• Return nil to drop the frame, so it is not sent. To send a frame unchanged, return pixelBuffer.

Example

final class MyVideoProcessor: VideoSDKVideoProcessor {

    func onFrameReceived(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
        // change the frame here, for example with Core Image, then return it
        return pixelBuffer
    }
}

extension RoomViewController {
    func turnOnMyProcessor() {
        room?.setVideoProcessor(processor: MyVideoProcessor())
    }
}

---

onFrameReceived(frame:)

func onFrameReceived(frame: RTCVideoFrame) -> RTCVideoFrame?

• Works on an RTCVideoFrame instead of a CVPixelBuffer. Your app can't create an RTCVideoFrame, so implement onFrameReceived(pixelBuffer:) instead.
• Implement only one of the two methods.

---

VideoSDKBackgroundProcessor

class VideoSDKBackgroundProcessor: VideoSDKVideoProcessor

A processor that replaces the background behind the person with an image.

init(backgroundSource:)

init(backgroundSource: URL)

• Creates the processor and starts to download the image at backgroundSource.
• While the processor is set and has no image yet, no video is sent from your camera. If the download fails, no video is sent until a new image loads. Create the processor before you set it, and use an image URL that loads quickly.
• The image is stretched to the size of the video, so use an image with the same aspect ratio as your video.

Parameters

• backgroundSource: URL - The URL of the background image.

Example

extension RoomViewController {
    func turnOnVirtualBackground() {
        let backgroundSource = URL(string: "https://cdn.videosdk.live/virtual-background/cloud.jpeg")!
        let processor = VideoSDKBackgroundProcessor(backgroundSource: backgroundSource)
        room?.setVideoProcessor(processor: processor)
    }

    func turnOffVirtualBackground() {
        room?.removeVideoProcessor()
    }
}

---

changeBackground()

func changeBackground(backgroundSource: URL)

• Downloads a new background image and uses it once it has loaded. The current background stays until then.
• If the download fails, the background is removed, and no video is sent until a new image loads.

Parameters

• backgroundSource: URL - The URL of the new background image.

Example

func showSanFranciscoBackground(with processor: VideoSDKBackgroundProcessor) {
    let backgroundSource = URL(string: "https://cdn.videosdk.live/virtual-background/san-fran.jpeg")!
    processor.changeBackground(backgroundSource: backgroundSource)
}