Background Blur - iOS

Background blur hides what is behind you during video calls.

This guide explains how to achieve background blur in your iOS video calling app using the VideoSDKVideoProcessor protocol.

How Background Blur Works

Background blur relies on a process called person segmentation. This involves separating you (the foreground) from the background in the video frame. Once the segmentation is done, the background is blurred while your image remains clear.

Implementation Steps

Here's a breakdown of how to implement background blur in your app:

1. Process Video Frames with onFrameReceived(pixelBuffer:)

Within the onFrameReceived(pixelBuffer:) function provided by the VideoSDKVideoProcessor protocol, you'll perform the core background blur processing for each video frame:

• Get the Frame: VideoSDK passes each camera frame as a BGRA CVPixelBuffer, which is ready for processing.

• Segment the Person (iOS 15.0+): (If your app supports iOS 15.0 or later) Use Vision's person segmentation request (VNGeneratePersonSegmentationRequest) to separate you (the foreground) from the background.

• Blur the Background: Create a blurred version of the background to achieve the desired effect.

• Composite the Image: Combine the blurred background with your clear image using a compositing technique.

• Return the Processed Frame: Return the processed CVPixelBuffer for transmission. If you return nil, the frame is dropped, so return the original buffer when the background can't be blurred.

Code Example:

This code demonstrates a basic implementation of background blur using VideoSDKVideoProcessor

Swift

import CoreImage
import CoreImage.CIFilterBuiltins
import Vision
import VideoSDKRTC

// class conforming to VideoSDKVideoProcessor protocol.
class MyVideoProcessor: VideoSDKVideoProcessor {

    // reuse one context for every frame
    private let ciContext = CIContext(options: nil)

    func onFrameReceived(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {

        // 1. Get the Frame: the frame arrives as a BGRA CVPixelBuffer

        // 2. Segment the Person (iOS 15.0+): Separate foreground (you) from background
        if #available(iOS 15.0, *) {
            let request = VNGeneratePersonSegmentationRequest()
            request.qualityLevel = .balanced // Use balanced for smooth video experience
            request.outputPixelFormat = kCVPixelFormatType_OneComponent8
            let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])

            do {
                try requestHandler.perform([request])
                guard let result = request.results?.first else {
                    return pixelBuffer
                }

                let maskPixelBuffer = result.pixelBuffer

                // 3. Blur the Background and 4. Composite the Image
                if let compositedPixelBuffer = compositeImage(originalPixelBuffer: pixelBuffer, maskPixelBuffer: maskPixelBuffer) {

                    // 5. Return the Processed Frame
                    return compositedPixelBuffer
                }
            } catch {
                print("Error performing person segmentation request: \(error)")
            }
        }
        // send the original frame when the background can't be blurred
        return pixelBuffer
    }

    func compositeImage(originalPixelBuffer: CVPixelBuffer, maskPixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
        //original image
        let ciImage = CIImage(cvPixelBuffer: originalPixelBuffer)
        //masked image
        let maskCIImage = CIImage(cvPixelBuffer: maskPixelBuffer)

        //scale masked image
        let maskScaleX = ciImage.extent.width / maskCIImage.extent.width
        let maskScaleY = ciImage.extent.height / maskCIImage.extent.height
        let maskScaled = maskCIImage.transformed(by: CGAffineTransform(scaleX: maskScaleX, y: maskScaleY))

        //apply blur to the original image
        let blurredImage = ciImage.clampedToExtent()
            .applyingGaussianBlur(sigma: 2.0) //tweak sigma to adjust blur amount
            .cropped(to: ciImage.extent)

        let blendFilter = CIFilter.blendWithMask()
        blendFilter.inputImage = ciImage
        blendFilter.backgroundImage = blurredImage
        blendFilter.maskImage = maskScaled

        guard let blendedImage = blendFilter.outputImage else {
            return nil
        }

        var outputPixelBuffer: CVPixelBuffer?
        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
             kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
        let width = CVPixelBufferGetWidth(originalPixelBuffer)
        let height = CVPixelBufferGetHeight(originalPixelBuffer)

        let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, attrs, &outputPixelBuffer)

        guard status == kCVReturnSuccess, let buffer = outputPixelBuffer else {
            return nil
        }
        ciContext.render(blendedImage, to: buffer)
        return buffer
    }
}

2. Activating Background Blur

Once you've created your MyVideoProcessor class, you can enable background blur in your video calls using the setVideoProcessor method of the Room class. To turn the blur off, call removeVideoProcessor(). This essentially tells VideoSDK to utilize your custom processor for processing the video stream before transmission.

Swift

class RoomViewController: UIViewController {

    var room: Room?

    // Indicates whether the background is currently blurred
    var isBackgroundBlurred: Bool = false

    // Button action that toggles the blur effects  for the local participant
    @IBAction func toggleBackgroundBlur(_ sender: Any) {

        // If the background is already blurred, remove the video processor
        guard isBackgroundBlurred == false else {
            room?.removeVideoProcessor()
            self.isBackgroundBlurred = false
            return
        }

        // Otherwise, set the video processor to enable background blur
        let processor = MyVideoProcessor()
        room?.setVideoProcessor(processor: processor)
        self.isBackgroundBlurred = true
    }
}

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoProcessor()
• removeVideoProcessor()
• VideoSDKVideoProcessor