Video Effects - Flutter

Hosts in Mode.SEND_AND_RECV can apply these effects to the camera video they send to the live stream.

The SDK can apply effects to your camera video before it is sent. Everyone in the room sees the effects, and so does your own preview.

Effect	Setting	What it does
Auto-framing	framing	Crops and zooms the picture to keep you in the center.
Low-light correction	lighting	Brightens the dark parts of the picture and leaves bright parts alone.
Stylized look	style	Applies POSTERIZE or VIVID to the whole picture.
Color grade	filter	Applies WARM, COOL or MONO colors.
Background blur	background	Keeps you sharp and blurs everything behind you.

Step 1: Add the Android libraries

On Android, background blur and auto-framing on the default pipeline use ML Kit, which the SDK does not include. Add the libraries for the effects you use to your app's android/app/build.gradle.kts:

android/app/build.gradle.kts

dependencies {
    // Auto-framing
    implementation("com.google.mlkit:face-detection:16.1.7")
    // Background blur
    implementation("com.google.mlkit:segmentation-selfie:16.0.0-beta6")
}

In a Groovy android/app/build.gradle, write implementation "com.google.mlkit:face-detection:16.1.7" and implementation "com.google.mlkit:segmentation-selfie:16.0.0-beta6" instead.

Without a library, the other effects still run, and the SDK reports why framing or blur is off. iOS needs no extra setup.

Step 2: Turn effects on

Describe the effects with a VideoEffectsConfig, and pass it to VideoSDK.setVideoEffects(). You can call it before or after you join the room. The effects start when your camera starts.

import 'package:flutter/foundation.dart';
import 'package:videosdk/videosdk.dart';

Future<void> turnOnEffects() async {
  try {
    final Map<String, dynamic>? applied = await VideoSDK.setVideoEffects(
      const VideoEffectsConfig(
        lighting: VideoEffectsLighting(enabled: true, strength: 0.8),
        filter: VideoEffectsFilter(lut: VideoEffectsLut.WARM, strength: 0.6),
        background: VideoEffectsBackground(
          mode: VideoEffectsBackgroundMode.BLUR,
          blurStrength: 0.7,
        ),
      ),
    );

    if (applied?['applied'] != true) {
      debugPrint("Effects are not running: ${applied?['reason']}");
    }
  } on VideoSDKError catch (error) {
    debugPrint("Could not set the effects: ${error.code} ${error.name}");
  }
}

• Each effect is off until you turn it on.
• strength and blurStrength run from 0.0 to 1.0.
• In a room, room.setVideoEffects() does the same as VideoSDK.setVideoEffects().

The Video Effects Config reference lists every setting.

Step 3: Change effects

Each call replaces the whole configuration. An effect you leave out goes back to its default, so keep the effects in your app's state and build the full configuration from it every time.

import 'package:flutter/foundation.dart';
import 'package:videosdk/videosdk.dart';

class EffectsController {
  EffectsController(this.room);

  final Room room;
  bool blur = false;
  VideoEffectsLut lut = VideoEffectsLut.OFF;

  VideoEffectsConfig get _config => VideoEffectsConfig(
        filter: VideoEffectsFilter(lut: lut),
        background: VideoEffectsBackground(
          mode: blur
              ? VideoEffectsBackgroundMode.BLUR
              : VideoEffectsBackgroundMode.OFF,
        ),
      );

  Future<void> setBlur(bool on) async {
    blur = on;
    await _apply();
  }

  Future<void> setColorGrade(VideoEffectsLut value) async {
    lut = value;
    await _apply();
  }

  Future<void> _apply() async {
    try {
      await room.setVideoEffects(_config);
    } on VideoSDKError catch (error) {
      debugPrint("Could not change the effects: ${error.code} ${error.name}");
    }
  }
}

• A change applies on the next frame.
• Wait for each call to complete before you make the next one.
• To turn every effect off, pass const VideoEffectsConfig().

Step 4: Check which effects are running

setVideoEffects() returns the effects that are running, and getAppliedVideoEffectsConfig() returns the same at any time. They can differ from what you asked for.

• applied is false when the effects are not running, and reason says why.
• engine is cpu, gpu or none.
• While your camera is off, applied is false. The SDK keeps the configuration and applies it when the camera starts.

final Map<String, dynamic>? effects = await room.getAppliedVideoEffectsConfig();
debugPrint("Engine: ${effects?['engine']}, running: ${effects?['applied']}");

Choose the pipeline

pipeline picks the engine that runs the effects.

	VideoEffectsPipeline.LEGACY (default)	VideoEffectsPipeline.GRAPH
Runs on	The CPU	The GPU
Auto-framing follows	The largest face the SDK detects	The subject you publish with publishFramingSubject()
Background blur	Yes	No

• On a device whose GPU cannot run the effects, GRAPH returns applied as false with a reason, and your camera video is sent without effects.
• You can change the pipeline at any time. On Android, a change while your camera is on restarts the camera, on the same camera. On iOS, it applies on the next frame.

Auto-framing on the GRAPH pipeline

On GRAPH, the SDK does not look for faces itself. Run your own face or person detection, and tell the SDK where the subject is with VideoSDK.publishFramingSubject().

• cx and cy are the center of the subject, and width and height are its size. Each is a fraction from 0 to 1 of the video that the SDK sends, not of the raw camera image.
• Publish a new box each time your detector finds the subject. The SDK keeps the last box until you publish another.
• Pass 0 as width and height when the subject is lost, and the picture zooms back out to the full frame.

import 'dart:ui';

import 'package:videosdk/videosdk.dart';

Future<void> startFraming() async {
  await VideoSDK.setVideoEffects(
    const VideoEffectsConfig(
      pipeline: VideoEffectsPipeline.GRAPH,
      framing: VideoEffectsFraming(enabled: true, maxZoom: 2.0),
    ),
  );
}

// Call this each time your detector finds the face in a frame of `frameSize`.
Future<void> onFaceDetected(Rect face, Size frameSize) async {
  await VideoSDK.publishFramingSubject(
    cx: face.center.dx / frameSize.width,
    cy: face.center.dy / frameSize.height,
    width: face.width / frameSize.width,
    height: face.height / frameSize.height,
  );
}

Background image on iOS

On iOS, room.setVideoProcessor() replaces your camera background with an image. Pass the image's file path or URL as the processor name, and call room.removeVideoProcessor() to remove it. Android has no built-in background image.

import 'dart:io';

import 'package:videosdk/videosdk.dart';

Future<void> showOfficeBackground(Room room) async {
  if (Platform.isIOS) {
    await room.setVideoProcessor(
      "https://example.com/backgrounds/office.jpg",
    );
  }
}

Effects and video processors

The effects on VideoEffectsPipeline.LEGACY and a video processor run one at a time. On Android, that is a processor you registered yourself. On iOS, it is the background image. The one you apply last replaces the other, and room.removeVideoProcessor() removes both. To learn about processors, see Video Processor.

API Reference

The API references for all the methods utilized in this guide are provided below.

• setVideoEffects()
• getAppliedVideoEffectsConfig()
• publishFramingSubject()
• room.setVideoEffects()
• room.getAppliedVideoEffectsConfig()
• room.setVideoProcessor()
• room.removeVideoProcessor()
• VideoEffectsConfig