Video Codecs - iOS

A video codec compresses your camera video so it can be sent efficiently to other participants. VideoSDK supports four codecs, VP8 (the default), H264, VP9, and AV1, and each one balances picture quality, processing power, and support across browsers and devices in its own way. This guide helps you pick the right one and set it with the codec parameter of createCameraVideoTrack().

How a Codec Works

Uncompressed camera video is far too large to send in real time, so a codec compresses it. It keeps the detail viewers notice most and stores each frame largely as the changes from the one before, which makes the video much smaller. Your device compresses the video with the codec you choose, and each participant's device uses the same codec to decompress it. VP9 and AV1 compress more efficiently than VP8 and H264, so they can deliver sharper video over the same connection.

Choosing a Codec

VP8 is a reliable starting point for most apps. The comparison below shows how the codecs differ, and each codec's section describes where it works best.

Comparing the four codecs

Codec	Quality / Compression	CPU	Support	Use it when
VP8	Good	Low	Very Broad	You want a dependable default
H264	Good	Low	Broad	Your devices or media pipeline work best with H.264
VP9	Better	Medium-High	Modern Devices	You want better picture quality on modern devices
AV1	Best	High*	Growing	You want the most efficient compression on devices you've tested

* Hardware acceleration can significantly reduce CPU usage.

VP8

• Google released VP8 as an open-source codec in 2010 as part of the WebM project. The WebRTC standard requires browsers to support it, so it runs on the widest range of browsers and devices.
• It is the default codec in VideoSDK and gives good video quality with low CPU use.
• VideoSDK supports multiStream for VP8, so it can publish several quality layers and send a lighter layer to participants viewing small tiles or on slower connections.
• It's also VideoSDK's fallback codec. If your device can't encode the codec you chose, or another participant can't decode it, VideoSDK switches your video to VP8 so the call keeps running.

H264

• H264, also known as AVC, was published in 2003 by the ITU-T and ISO/IEC MPEG. The WebRTC standard requires browsers to support it alongside VP8.
• It gives good video quality with low CPU use. On iPhone and iPad, VideoSDK encodes H264 with the device's hardware encoder, which keeps the processing load light.
• Currently, VideoSDK doesn't support multiStream for H264, so it publishes H264 video as a single quality layer.
• It's a good fit when you want to use the hardware encoder on iPhone and iPad, or when your media pipeline works best with H.264.

VP9

• Google released VP9 in 2013 as the successor to VP8, and it compresses video more efficiently than VP8.
• It may need more processing power than older codecs to encode and decode.
• Currently, VideoSDK doesn't support multiStream for VP9, so it publishes VP9 video as a single quality layer and maxLayer has no effect.
• It works well for one-to-few calls on modern devices where picture quality matters most.

AV1

• The Alliance for Open Media finalized AV1 in 2018 as the successor to VP9. It has the most efficient compression of the four codecs, with excellent video quality.
• As the newest of the four, AV1 is still gaining hardware support and may need more processing power to encode and decode than older codecs. Most of that work happens on the device that sends the video, because decoding is lighter than encoding.
• Currently, VideoSDK doesn't support multiStream for AV1, so it publishes AV1 video as a single quality layer.
• It's a good fit when you want the most efficient compression on devices you've tested.

caution

AV1 device support is still growing, and older devices may not be able to encode or decode it. Test it on your target devices before using it in production.

Setting the Codec

• codec is an optional parameter of the createCameraVideoTrack() method. When you don't pass it, VideoSDK encodes the track with .VP8.
• Pass the codec as a VideoCodec value: .VP8, .H264, .VP9, or .AV1.
• To use the track, pass it to initMeeting() as customCameraVideoStream, which applies it when webcamEnabled is true, or pass it to enableWebcam(customVideoStream:) during the meeting. Refer to How to Setup a Custom Video Track for full examples.
• The codec is set when you create the track. From version 3.1.0 onward, VideoSDK reuses that track's settings, codec included, when you turn the webcam off and on again, unless it has already switched your video to .VP8. On earlier versions, create a new track with the same codec and pass it to enableWebcam(customVideoStream:) each time you turn the webcam back on.
• To change the codec during the meeting, call disableWebcam(), then create a new track with the codec you want and pass it to enableWebcam(customVideoStream:).
• For the other parameters of createCameraVideoTrack(), refer to the Optimize Video Track guide.

Example

Swift

guard let videoMediaTrack = try? VideoSDK.createCameraVideoTrack(
    encoderConfig: .h720p_w1280p,
    facingMode: .front,
    multiStream: false,
    codec: .VP9
) else { return }

meeting?.enableWebcam(customVideoStream: videoMediaTrack)

Codec and multiStream

• multiStream (simulcast) publishes several quality layers of your video, so VideoSDK can send a lighter layer to participants viewing small tiles or on slower connections.

caution

Currently, VideoSDK doesn't support multiStream for H264, VP9, and AV1. multiStream defaults to true, so set multiStream: false when you create an H264, VP9, or AV1 track. Otherwise VideoSDK sets it to false for you and reports the ERROR_MULTISTREAM_NOT_SUPPORTED error.

• maxLayer requires multiStream: true, so maxLayer has no effect with H264, VP9, or AV1. Refer to Optimize Video Track for what maxLayer controls.

How VideoSDK Handles Codec Compatibility

VideoSDK automatically handles codec compatibility, so an unsupported codec never interrupts a meeting.

When your device cannot send the codec

• If your device doesn't support the codec you passed, VideoSDK automatically switches your camera track to .VP8 so your video keeps flowing, and reports ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED through the onError(error:) method of MeetingEventListener. You can use this as a signal to pick a different codec for that kind of device.
• If the device can encode neither the codec you passed nor .VP8, VideoSDK reports ERROR_NO_SUPPORTED_VIDEO_CODEC through onError(error:) and no video is sent.

Example

Swift

extension MeetingViewController: MeetingEventListener {
    func onError(error: VideoSDKError) {
        if error == .ERROR_VIDEO_PRODUCE_CODEC_NOT_SUPPORTED {
            // VideoSDK switched the codec to VP8 because the provided codec is not supported
        }
    }
}

When a participant cannot receive the codec

• If a participant's device or browser can't decode the codec in use, that participant receives CODEC_NOT_SUPPORTED through onError(error:).
• VideoSDK then switches to .VP8 every participant whose video that participant can't decode, so everyone can keep seeing each other. Those participants receive the onCodecChanged(info:) event of MeetingEventListener, and their video stays on .VP8 for the rest of the meeting, even after that participant leaves.

Example

Swift

extension MeetingViewController: MeetingEventListener {
    func onError(error: VideoSDKError) {
        if error == .CODEC_NOT_SUPPORTED {
            // Your device can't decode the codec another participant is sending
        }
    }

    func onCodecChanged(info: CodecChangeInfo) {
        // VideoSDK switched your own video to VP8 because another participant can't decode your codec
        print("Current codec:", info.currentCodec.rawValue)
        print("Unsupported codec:", info.unsupportedCodec.rawValue)
        print("Media kind:", info.kind)
        print("Participant that caused the codec switch:", info.causedBy?.displayName ?? "unknown")
        print("Previous codec:", info.previousCodec.rawValue)
    }
}

info

The codecSwitchEnabled parameter of initMeeting() controls this switch and defaults to true. If you set it to false, VideoSDK keeps the current codec and does not call onCodecChanged(info:). Participants who can decode that codec keep seeing the video as usual, while a participant who can't may see a black screen.

For each error's code and message, refer to the Error Events guide.

Checking Which Codec Is in Use

• VideoSDK may switch a participant's video to .VP8 during a meeting to keep the call running, so the codec in use isn't always the one that was requested.
• The getVideoStats() method of the Participant class returns an array of statistics dictionaries ([[String: Any]]), and each entry has a codec field. The first entry's codec tells you what that participant's video is encoded with. It returns nil when no statistics are available. Refer to the Understanding Call Quality guide for the other statistics it returns.

Example

Swift

// Use meeting?.localParticipant for your own video
if let participant = meeting?.participants[participantId],
   let stats = participant.getVideoStats() as? [[String: Any]],
   let codec = stats.first?["codec"] as? String {
    print("Codec in use:", codec)
}

note

These statistics are a snapshot. To follow a codec change as it happens, listen for the onCodecChanged(info:) event instead of polling.

tip

For the smoothest experience, test your codec choice on the lowest-specification device you support, with participants joining from every platform your app runs on. This helps you find any device that can't use your codec, which would make VideoSDK switch video to VP8.

API Reference

The API references for all the methods and events utilized in this guide are provided below.

• VideoCodec
• initMeeting()
• enableWebcam()
• disableWebcam()
• onCodecChanged
• CodecChangeInfo
• getVideoStats()