Video Effects - Android

The Android SDK can apply effects to your camera video before it is sent. Everyone in the room sees the effects, and so does your own preview.

The SDK offers four effects:

Effect	Setting	What it does
Auto-framing	Framing	Crops and zooms the picture to keep a subject in the center. You provide the subject's position.
Low-light correction	Lighting	Brightens dark parts of the picture and leaves bright parts alone.
Stylized look	Style	Applies a look to the whole picture: POSTERIZE or VIVID.
Color grade	Filter	Grades the colors of the finished picture: WARM, COOL or MONO.

The effects always run in this order: framing, then lighting, then style, then the color grade.

info

The SDK has no built-in virtual background, and none of these effects replaces or blurs the background. To process camera frames with your own code or a third-party library, use a video processor instead.

How it works

You describe the effects with a VideoEffectsConfig and pass it to VideoSDK.setVideoEffects().

• Effects run only with pipeline = VideoEffectsConfig.Pipeline.GRAPH. The default, Pipeline.LEGACY, sends the camera video unchanged.
• Each effect is off until you turn it on.
• The effects run on the device's GPU. If the GPU effects can't start on the device, onError reports 3080 (ERROR_VIDEO_EFFECTS_GRAPH_UNAVAILABLE) once and the camera stays on Pipeline.LEGACY, so the video is sent without effects. Once the camera is running, getAppliedVideoEffectsConfig() tells you which pipeline is in use (Step 3).

Step 1: Turn effects on before joining

Call VideoSDK.setVideoEffects() after VideoSDK.initialize() and before you join. The effects apply when the camera starts.

val config = VideoEffectsConfig(
    pipeline = VideoEffectsConfig.Pipeline.GRAPH,
    lighting = VideoEffectsConfig.Lighting(enabled = true, strength = 0.8f),
    filter = VideoEffectsConfig.Filter(lut = VideoEffectsConfig.Lut.WARM, strength = 0.6f),
)

val applied: JSONObject = VideoSDK.setVideoEffects(config)
val reason = applied.optString("reason")
// before the camera starts, a GRAPH request is held with a "pending:" reason
if (!applied.optBoolean("applied") && !reason.startsWith("pending:")) {
    // the GPU effects couldn't start; onError reports 3080 and the video is sent without effects
    Log.w("VideoEffects", "GPU effects not applied: $reason")
}

Before any camera has started, a GRAPH request returns applied false with a reason that starts with pending:. It is kept, and the next camera start runs it.

The settings of each effect:

• Lighting(enabled, strength): strength runs from 0.0 to 1.0 and defaults to 1.0.
• Style(preset, strength, levels): preset is StylePreset.OFF, POSTERIZE or VIVID. strength runs from 0.0 to 1.0. levels sets how many color steps POSTERIZE uses, an odd number from 3 to 17, and defaults to 5.
• Filter(lut, strength): lut is Lut.OFF, WARM, COOL or MONO. strength runs from 0.0 to 1.0.
• Framing(enabled, maxZoom): maxZoom is the closest the crop can zoom in, from 1.0 to 4.0, and defaults to 2.0. See Auto-framing.

A strength of 0.0 is the same as turning that effect off.

Step 2: Change effects while in the room

Call room.setVideoEffects() with a new configuration.

• A change to the effect settings, such as a new color grade or strength, applies straight away.
• A change of pipeline, from LEGACY to GRAPH or back, applies the next time the camera starts. Turn the camera off and on again, or switch cameras, to apply it while in the room.

// switch from the warm grade to a vivid look
room.setVideoEffects(
    VideoEffectsConfig(
        pipeline = VideoEffectsConfig.Pipeline.GRAPH,
        lighting = VideoEffectsConfig.Lighting(enabled = true, strength = 0.8f),
        style = VideoEffectsConfig.Style(preset = VideoEffectsConfig.StylePreset.VIVID, strength = 0.7f),
    )
)

To turn every effect off without restarting the camera, keep Pipeline.GRAPH and leave each effect off:

room.setVideoEffects(VideoEffectsConfig(pipeline = VideoEffectsConfig.Pipeline.GRAPH))

Each call replaces the whole configuration, so include every effect you want to keep.

Step 3: Check which effects are running

getAppliedVideoEffectsConfig() returns the configuration in force, which can differ from what you asked for. When applied is false, reason explains why.

val current: JSONObject = room.getAppliedVideoEffectsConfig()
Log.d("VideoEffects", "applied=${current.optBoolean("applied")} pipeline=${current.optString("pipeline")}")

VideoSDK.getAppliedVideoEffectsConfig() returns the same, also outside a room.

Showing your own video

With Pipeline.GRAPH, your own preview shows the picture with effects, the same picture others receive.

• When you show your camera stream with VideoView.addTrack(), the view shows the picture with effects.
• VideoView.attachLocalPreview() shows the same preview in a view that has no track, and detachLocalPreview() stops it. It does nothing unless effects are running.
• Your preview can be shown in one view at a time. A second view that shows it takes over from the first.
• localParticipant.captureImage() returns the picture with effects too.

Auto-framing

Auto-framing keeps a subject in the center of the picture. The SDK doesn't detect faces or people itself, so you tell it where the subject is with VideoSDK.publishFramingSubject().

• Turn it on with Framing(enabled = true) and Pipeline.GRAPH.
• Pass the subject's box as fractions of the video frame, from 0.0 to 1.0: the center (cx, cy), the width and the height. A smaller box zooms in further, up to maxZoom.
• Publish a new box each time your detector finds the subject, not once. A width or height of 0 or less means there is no subject, and the picture eases back to the full frame.
• Until you publish a box, the video is sent unchanged.
• You can call it from any thread.

VideoSDK.setVideoEffects(
    VideoEffectsConfig(
        pipeline = VideoEffectsConfig.Pipeline.GRAPH,
        framing = VideoEffectsConfig.Framing(enabled = true, maxZoom = 2.0f),
    )
)

// call this for every face box your detector finds
fun onFaceDetected(box: RectF, frameWidth: Int, frameHeight: Int) {
    VideoSDK.publishFramingSubject(
        cx = box.centerX() / frameWidth,
        cy = box.centerY() / frameHeight,
        width = box.width() / frameWidth,
        height = box.height() / frameHeight,
    )
}

Video effects and video processors

Camera effects and a video processor set with room.setVideoProcessor() don't run together. While effects run on Pipeline.GRAPH, an installed video processor receives no frames. Use one or the other.

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoEffects()
• getAppliedVideoEffectsConfig()
• publishFramingSubject()
• room.setVideoEffects()
• room.getAppliedVideoEffectsConfig()
• setVideoProcessor()