Picture-in-Picture Mode - iOS

Overview

Picture-in-picture (PiP) is a commonly used feature in video conferencing software, enabling users to simultaneously engage in a video conference and perform other tasks on their device. With PiP, you can keep the video conference window open, resize it to a smaller size, and continue working on other tasks while still seeing and hearing the other participants in the conference. This feature proves beneficial when you need to take notes, send an email, or look up information during the conference.

This guide explains the steps to implement the Picture-in-Picture (PiP) feature in a SwiftUI iOS application using VideoSDK, without relying on any third-party packages.

Prerequisites

To implement this feature, you need an app that uses VideoSDK. Build one with the SwiftUI Quick Start, or clone this Quick Start Repository and run the app. The repository uses iOS SDK 3.x, so set its videosdk-rtc-ios-spm package to Exact Version 4.0.1 and apply the changes in the Migration Guide first. Once the app is running, we will start integrating PiP mode in the iOS app.

Info.plist Configuration

<key>NSCameraUsageDescription</key>
<string>Camera permission description</string>
<key>NSMicrophoneUsageDescription</key>
<string>Microphone permission description</string>

Xcode Capabilities

Enable the following capabilities in Xcode:

1. Go to your target settings.
2. Select the "Signing & Capabilities" tab.
3. Click the "+" button to add capabilities.
4. Add Background Modes

For Background Modes, ensure the following options are checked:

• Audio, AirPlay, and Picture in Picture
• Voice over IP

Adding a PiP Toggle Button and Functions in MeetingViewController

Ensure that a button is added to the UI in MeetingViewController to enable and disable the PiP (Picture-in-Picture) feature.

Swift

MeetingView.swift

Button {
    meetingViewController.checkPiPStatus()
    
    if meetingViewController.isPiPActive {
        meetingViewController.stopPiP()
    } else {
        meetingViewController.startPiP()
    }
} label: {
    Text(meetingViewController.isPiPActive ? "Disable PiP" : "Enable PiP")
        .foregroundStyle(Color.white)
        .font(.caption)
        .padding()
        .background(RoundedRectangle(cornerRadius: 25).fill(Color.indigo))
}

Inside the cloned project, in MeetingViewController.swift, we need to add functions to handle PiP mode activation and manage foreground/background states.

Key Functions:

• startPiP(): Initializes and starts PiP mode.
• stopPiP(): Stops PiP mode and restores the main UI.
• checkPiPStatus(): Updates isPiPActive from the system PiP window, for example after the user closes it.
• handleEnterBackground(): Ensures PiP remains active when the app goes into the background.
• handleEnterForeground(): Reconfigures PiP when the app returns to the foreground.

You can also copy the file from here

Swift

MeetingViewController.swift

func startPiP() {
    if pipVideoCallViewController == nil {
        pipVideoCallViewController = PiPVideoCallViewController(meetingViewController: self)
    }
    pipVideoCallViewController?.startPiP()
}
func stopPiP() {
    pipVideoCallViewController?.stopPiP()
    isPiPActive = false
}
func checkPiPStatus() {
    isPiPActive = pipVideoCallViewController?.pipController?.isPictureInPictureActive ?? false
}

Implementing PiP Mode

PiP implementation consists of the following four key files:

• PiPContainerView.swift
• PiPVideoCallViewController.swift
• PiPFrameProcessor.swift
• PiPVideoView.swift

You can copy these files from github pip mode example and add them to your project.

note

With the 4.x SDK, check these points in the files you copy:

• Remove import WebRTC. RTCVideoTrack, RTCVideoFrame and RTCVideoRenderer now come from VideoSDKRTC.
• Create the room with one try VideoSDK.createRoom(roomId:token:...) call; the token is a parameter, with no separate config step. The call returns a Room. Call room and participant actions, such as join(), muteMic(), enableCam(), end() and setQuality(_:), with try await inside a Task. Refer to the Migration Guide for the full list of changes.
• In PiPFrameProcessor, read each frame's pixel buffer from frame.buffer, as shown in the PiPFrameProcessor.swift section.

Architecture Diagram

The diagram below shows the PiP workflow:

When the user taps "Enable PiP", the PiPVideoCallViewController fetches video tracks from VideoSDK and sends them to PiPContainerView, which manages the local and remote video views. These tracks are processed by PiPFrameProcessor, and the frames are rendered using AVSampleBufferDisplayLayer within PiPVideoView for real-time video display.

File Breakdown

Each of the four files serves a specific purpose in enabling PiP mode.

PiPVideoCallViewController.swift (PiP Video Call Manager)

When the user taps "Enable PiP", PiPVideoCallViewController fetches the video tracks from the VideoSDK room and sends them to the PiPContainerView. It also manages the AVPictureInPictureControllerDelegate state.

• Key Responsibilities:
  • Initializes a PiPContainerView that contains both local and remote video views.
  • Creates an AVPictureInPictureController to handle PiP behavior.
  • Updates video tracks dynamically based on participants' activity.
• How It Connects:
• Receives video tracks from the VideoSDK room and updates PiPContainerView.
• Uses PiPContainerView to display the video.
• Calls updateVideoTracks() whenever video state changes.

PiPContainerView.swift (Container for PiP Video Views)

• Purpose: Acts as a container for the local and remote video views inside PiP mode. You can customize PiPContainerView.swift as per your requirement.
• Key Responsibilities:
  • Manages two PiPVideoView instances: localVideoView and remoteVideoView.
  • Updates video tracks when they change.
  • Arranges views properly using constraints.
• How It Connects:
• Receives video tracks from PiPVideoCallViewController.
• Calls addVideoTrack() on PiPVideoView to render video frames.
• Uses PiPFrameProcessor inside PiPVideoView to process and render frames.

PiPVideoView.swift (Individual Video View)

• Purpose: PiPVideoView.swift Handles rendering of an individual video track (either local or remote).
• Key Responsibilities:
  • Uses AVSampleBufferDisplayLayer to display video frames.
  • Receives an RTCVideoTrack and processes frames using PiPFrameProcessor.
  • Ensures real-time rendering of the participant's video.
• How It Connects:
• Receives an RTCVideoTrack from PiPContainerView.
• Passes the video track to PiPFrameProcessor for frame processing.

PiPFrameProcessor.swift (Frame Processing)

• Purpose: PiPFrameProcessor.swift Converts VideoSDK's RTC video frames into a format that can be displayed in AVSampleBufferDisplayLayer.
• Key Responsibilities:
  • Reads the CVPixelBuffer of each RTCVideoFrame and wraps it in a CMSampleBuffer for rendering.
  • Handles frame rate and performance optimizations.
  • Uses a pixel buffer pool for efficient memory management.
• How It Connects:
• Receives video frames from PiPVideoView.
• Converts and renders frames into AVSampleBufferDisplayLayer

In 4.x, RTCVideoFrame is a struct and its buffer is an enum. VideoSDK delivers video frames as .cvPixelBuffer, so read the pixel buffer with a switch instead of casting frame.buffer:

Swift

PiPFrameProcessor.swift

private func getPixelBuffer(from frame: RTCVideoFrame) -> CVPixelBuffer? {
    switch frame.buffer {
    case .cvPixelBuffer(let pixelBuffer):
        return pixelBuffer
    case .i420(let i420Buffer):
        return createPixelBuffer(from: i420Buffer, width: Int(frame.width), height: Int(frame.height))
    @unknown default:
        return nil
    }
}

Because a frame is a struct, compare frames by timestampNs, for example self.frameBuffer?.timestampNs != frame.timestampNs, instead of with !==.

Final Output

We have successfully implemented Picture-in-Picture (PiP) Mode for an iOS application using VideoSDK without any third-party packages. You can customize the view and optimize it as needed.

tip

Stuck anywhere? Check out this example code on GitHub. The example uses iOS SDK 3.x; the Migration Guide lists what changed in 4.0.0.