Video Codecs - iOS

A video codec compresses your camera video so it can be sent efficiently to other participants. VideoSDK supports four codecs, VP8 (the default), H264, VP9, and AV1, and each one balances picture quality, processing power, and support across browsers and devices in its own way. This guide helps you pick the right one and set it with the codec parameter of createCameraVideoTrack().

How a Codec Works

Uncompressed camera video is far too large to send in real time, so a codec compresses it. It keeps the detail viewers notice most and stores each frame largely as the changes from the one before, which makes the video much smaller. Your device compresses the video with the codec you choose, and each participant's device uses the same codec to decompress it. VP9 and AV1 compress more efficiently than VP8 and H264, so they can deliver sharper video over the same connection.

Choosing a Codec

VP8 is a reliable starting point for most apps. The comparison below shows how the codecs differ, and each codec's section describes where it works best.

Comparing the four codecs

Codec	Quality / Compression	CPU	Support	Use it when
VP8	Good	Low	Very Broad	You want a dependable default
H264	Good	Low	Broad	Your devices or media pipeline work best with H.264
VP9	Better	Medium-High	Modern Devices	You want better picture quality on modern devices
AV1	Best	High*	Growing	You want the most efficient compression on devices you've tested

* Hardware acceleration can significantly reduce CPU usage.

VP8

• Google released VP8 as an open-source codec in 2010. It runs on the widest range of browsers and devices.
• It is the default codec in VideoSDK and gives good video quality with low CPU use.
• VideoSDK supports multiStream for VP8, so it can publish several quality layers and send a lighter layer to participants viewing small tiles or on slower connections.
• It's also VideoSDK's fallback codec. If your device can't encode the codec you chose, or another participant can't decode it, VideoSDK switches your video to VP8 so the call keeps running.

H264

• H264, also known as AVC, was published in 2003 by the ITU-T and ISO/IEC MPEG. Browsers support it alongside VP8.
• It gives good video quality with low CPU use. On iPhone and iPad, VideoSDK encodes H264 with the device's hardware encoder, which keeps the processing load light.
• Currently, VideoSDK doesn't support multiStream for H264, so it publishes H264 video as a single quality layer.
• It's a good fit when you want to use the hardware encoder on iPhone and iPad, or when your media pipeline works best with H.264.

VP9

• Google released VP9 in 2013 as the successor to VP8, and it compresses video more efficiently than VP8.
• It may need more processing power than older codecs to encode and decode.
• Currently, VideoSDK doesn't support multiStream for VP9, so it publishes VP9 video as a single quality layer and maxLayer has no effect.
• It works well for one-to-few calls on modern devices where picture quality matters most.

AV1

• The Alliance for Open Media finalized AV1 in 2018 as the successor to VP9. It has the most efficient compression of the four codecs, with excellent video quality.
• As the newest of the four, AV1 is still gaining hardware support and may need more processing power to encode and decode than older codecs. Most of that work happens on the device that sends the video, because decoding is lighter than encoding.
• Currently, VideoSDK doesn't support multiStream for AV1, so it publishes AV1 video as a single quality layer.
• It's a good fit when you want the most efficient compression on devices you've tested.

caution

AV1 device support is still growing, and older devices may not be able to encode or decode it. Test it on your target devices before using it in production.

Setting the Codec

• codec is an optional parameter of the createCameraVideoTrack() method. When you don't pass it, VideoSDK encodes the track with .VP8.
• Pass the codec as a VideoCodec value: .VP8, .H264, .VP9, or .AV1.
• createCameraVideoTrack() is async throws, so call it with try await.
• To use the track, pass it to createRoom() as customCameraTrack, which applies it when camEnabled is true, or pass it to enableCam(customCameraTrack:) to turn the camera on while in the room. If the camera is already on, enableCam(customCameraTrack:) throws ERROR_ALREADY_IN_REQUESTED_STATE, so use changeCam(customCameraTrack:) instead, as shown below. Refer to How to Setup a Custom Video Track for full examples.
• The codec is set when you create the track. VideoSDK reuses that track's settings, codec included, when you turn the camera off and on again, unless it has already switched your video to .VP8.
• To change the codec while in the room, create a new track with the codec you want and pass it to changeCam(customCameraTrack:). If the camera is on, VideoSDK turns it off and on again with the new track, so the new codec applies at once. If the camera is off, it turns on with the new track.
• If this device can't send the codec you chose, VideoSDK sends .VP8 instead and tells you through onError(error:), as described in How VideoSDK Handles Codec Compatibility.
• enableCam(customCameraTrack:) and changeCam(customCameraTrack:) are async throws, so call them with try await inside a Task.
• For the other parameters of createCameraVideoTrack(), refer to the Optimize Video Track guide.

Example

Swift

// The camera is off
Task {
    do {
        let videoMediaTrack = try await VideoSDK.createCameraVideoTrack(
            encoderConfig: .h720p_w1280p,
            facingMode: .front,
            multiStream: false,
            codec: .VP9
        )

        try await room?.enableCam(customCameraTrack: videoMediaTrack)
    } catch {
        print("Failed to enable the camera: \(error)")
    }
}

To switch to another codec while the camera is on, for example VP9:

Swift

Task {
    do {
        let videoMediaTrack = try await VideoSDK.createCameraVideoTrack(
            encoderConfig: .h720p_w1280p,
            facingMode: .front,
            multiStream: false,
            codec: .VP9
        )

        try await room?.changeCam(customCameraTrack: videoMediaTrack)
    } catch {
        print("Failed to change the codec: \(error)")
    }
}

Codec and multiStream

• multiStream (simulcast) publishes several quality layers of your video, so VideoSDK can send a lighter layer to participants viewing small tiles or on slower connections.

caution

Currently, VideoSDK doesn't support multiStream for H264, VP9, and AV1. multiStream defaults to true, so set multiStream: false when you create an H264, VP9, or AV1 track. Otherwise VideoSDK sets it to false for you and reports ERROR_MULTISTREAM_NOT_SUPPORTED (code 3046) through onError(error:).

• maxLayer requires multiStream: true, so maxLayer has no effect with H264, VP9, or AV1. Refer to Optimize Video Track for what maxLayer controls.

How VideoSDK Handles Codec Compatibility

VideoSDK falls back to .VP8 so an unsupported codec does not stop your video, unless the device can encode neither codec.

When your device cannot send the codec

• If your device doesn't support the codec you passed, VideoSDK automatically switches your camera track to .VP8 so your video keeps flowing, and reports ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED (code 3045) through the onError(error:) method of RoomEventListener. You can use this as a signal to pick a different codec for that kind of device.
• If the device can encode neither the codec you passed nor .VP8, VideoSDK reports ERROR_NO_SUPPORTED_VIDEO_CODEC (code 3052) through onError(error:) and no video is sent.

Example

Swift

extension RoomViewController: RoomEventListener {
    func onError(error: VideoSDKError) {
        if error == .ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED {
            // VideoSDK switched the codec to VP8 because the provided codec is not supported
        }
    }
}

When a participant cannot receive the codec

• If a participant's device or browser can't decode the codec in use, that participant receives CODEC_NOT_SUPPORTED (code 4051) through onError(error:).
• VideoSDK then switches to .VP8 every participant whose video that participant can't decode, so everyone can keep seeing each other. Those participants receive the onCodecChanged(info:) event of RoomEventListener, and their video stays on .VP8 for the rest of the room, even after that participant leaves.

Example

Swift

extension RoomViewController: RoomEventListener {
    func onError(error: VideoSDKError) {
        if error == .CODEC_NOT_SUPPORTED {
            // Your device can't decode the codec another participant is sending
        }
    }

    func onCodecChanged(info: CodecChangeInfo) {
        // VideoSDK switched your own video to VP8 because another participant can't decode your codec
        print("Current codec:", info.currentCodec.rawValue)
        print("Unsupported codec:", info.unsupportedRemoteCodec.rawValue)
        print("Media kind:", info.kind)
        print("Participant that caused the codec switch:", info.causedBy?.displayName ?? "unknown")
        print("Previous codec:", info.previousCodec.rawValue)
    }
}

info

The codecSwitchEnabled parameter of createRoom() controls this switch and defaults to true. If you set it to false, VideoSDK keeps the current codec and does not call onCodecChanged(info:). Participants who can decode that codec keep seeing the video as usual, while a participant who can't may see a black screen.

For each error's code and message, refer to the Error Events guide.

Checking Which Codec Is in Use

• VideoSDK may switch a participant's video to .VP8 in a room to keep the call running, so the codec in use isn't always the one that was requested.
• For your own camera, start from the codec you set on the track. ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED and onCodecChanged(info:) tell you when VideoSDK switches it to .VP8. info.currentCodec holds the codec in use after the switch.
• For a remote participant, the getVideoStats() method of the Participant class returns an array of statistics dictionaries ([[String: Any]]), and each entry has a codec field with the codec's MIME type, such as "video/VP8". The first entry's codec tells you what that participant's video is encoded with. It returns nil or an empty array when no statistics are available. Refer to the Understanding Call Quality guide for the other statistics it returns.

Example

Swift

// A remote participant's camera
if let participant = room?.participants[participantId],
   let stats = participant.getVideoStats() as? [[String: Any]],
   let codec = stats.first?["codec"] as? String {
    print("Receiving:", codec)
}

note

These statistics are a snapshot. To follow a codec change as it happens, listen for the onCodecChanged(info:) event instead of polling.

tip

For the smoothest experience, test your codec choice on the lowest-specification device you support, with participants joining from every platform your app runs on. This helps you find any device that can't use your codec, which would make VideoSDK switch video to VP8.

API Reference

The API references for all the methods and events utilized in this guide are provided below.

• VideoCodec
• createCameraVideoTrack()
• createRoom()
• enableCam()
• changeCam()
• onCodecChanged
• CodecChangeInfo
• getVideoStats()