Skip to main content
Version: 1.x.x

Virtual Background - React

Virtual backgrounds enhance your live stream experience by allowing you to replace your physical background with a digital image or apply a blur effect. This is ideal for creators and streamers who want to maintain privacy, reduce distractions, or add a personal touch to their broadcast. You can use a preloaded background image or supply your own via a CDN.

note

This page uses the following asynchronous methods. Refer to their API reference for the errors each Promise may reject with, and handle these rejections appropriately based on your use case.

important
  • This feature is supported on desktop browsers only.
Virtual Background

Install VideoSDK Virtual Background Package​

npm install --save "@videosdk.live/videosdk-virtual-background-web"
note

The @videosdk.live/videosdk-media-processor-web package no longer receives new versions. Use @videosdk.live/videosdk-virtual-background-web instead.

Instantiate Virtual Background Processor​

After installing the library, create one instance of VirtualBackgroundProcessor and reuse it for the whole live stream. Create it outside the component, so the loaded model is kept across renders.

// Import package
import { VirtualBackgroundProcessor, VBModel, BlurIntensity } from "@videosdk.live/videosdk-virtual-background-web";

// Instantiate VirtualBackgroundProcessor Class once, outside the component
const videoProcessor = new VirtualBackgroundProcessor();

const MeetingView = () => {
  const handleStartVirtualBackground = () => {};

  const handleStopVirtualBackground = () => {};

  const handleDisposeVirtualBackground = () => {};

  const handleChangeConfig = () => {};

  return (
    <>
      <button onClick={handleStartVirtualBackground}>
        Start Virtual Background
      </button>
      <button onClick={handleChangeConfig}>Change Virtual Background</button>
      <button onClick={handleStopVirtualBackground}>
        Stop Virtual Background
      </button>
      <button onClick={handleDisposeVirtualBackground}>
        Dispose Virtual Background
      </button>
    </>
  );
};

Checking Graphics Support​

Before you show the virtual background option, call getDeviceInfo() to check whether the host's device can run it smoothly. It tells you:

  • isWebGL2Supported: Indicates whether the browser supports WebGL2, which is required for virtual background. If false, virtual background cannot start, so hide the option or ask the host to update their browser.
  • isGPUAccelerated: Indicates whether the browser uses the device's GPU for graphics processing. If false, video may lag, so ask the host to enable hardware acceleration in their browser settings or hide the option.
  • isSoftwareRendered: Indicates whether the browser uses the CPU instead of the GPU for graphics processing. If true, virtual background may run slowly.
  • renderer and vendor: Provide the name of the graphics renderer and its vendor, respectively. These values may be null if the browser does not expose this information.
info

getDeviceInfo() is available from version 1.0.0 and above.

import { getDeviceInfo } from "@videosdk.live/videosdk-virtual-background-web";

// Show virtual background only when the device can run it smoothly
const device = getDeviceInfo();
const showVirtualBackground = device.isWebGL2Supported && device.isGPUAccelerated;

Using Virtual Background​

Initialize the processor with a segmentation model, process the camera stream, and pass the processed stream to VideoSDK.

Initializing​

You can select a model during initialization. If no model is specified, VBModel.landscape is used by default.

  • VBModel.landscape → Uses Selfie Segmenter Landscape.
  • VBModel.multiClass → Uses Multi-Class Selfie Segmentation with a 256 × 256 input.
  • VBModel.multiClassLite → Uses a quantized version of the Multi-Class Selfie Segmentation model with a 128 × 256 input, requiring less processing per frame than VBModel.multiClass.
  • VBModel.meet → Uses MediaPipe Meet Segmentation.
info

Model selection is supported from version 0.1.0. VBModel.multiClassLite is available from version 1.0.0.

The init() method resolves when the selected model has finished loading. If the model is already loaded, calling init() with the same model resolves without loading it again.

The method rejects if WebGL 2 is unavailable, the specified model is not a valid VBModel value, or the model fails to load. When WebGL 2 is unavailable, the error includes the getDeviceInfo() result in err.deviceInfo, which you can use to inspect the browser's graphics capabilities.

const MeetingView = () => {
  const handleStartVirtualBackground = async () => {
    // Initialize processor with model selection
    try {
      await videoProcessor.init(VBModel.landscape); // or VBModel.multiClass | VBModel.multiClassLite | VBModel.meet
    } catch (err) {
      console.error("init failed:", err);
    }
  };

  return <>...</>;
};

Choosing a Model for the Device​

Use measurePerformance() before calling start() to measure a model's processing time on the host's device. The method returns the measured frame time and an isDegraded flag, which is true when the frame time exceeds the threshold for the hardware used to run the model.

  • Result: Returns model, hardwareType (gpu or cpu), frameTimeMs, thresholdMs, and isDegraded.
  • Default model: If no model is specified, measurePerformance() measures the currently loaded model, or VBModel.landscape if no model is loaded. It returns null for VBModel.meet, which does not support performance measurement.
  • Execution constraints: The method rejects if the processor is running or starting. Call it before start() or after stop(). If start() is called while performance measurement is in progress, the measurement rejects with an AbortError.
info

measurePerformance() is available from version 1.0.0 and above.

const prepareVirtualBackground = async () => {
  try {
    // Time the multiclass-lite model on this device
    const result = await videoProcessor.measurePerformance(VBModel.multiClassLite);
    const model = result?.isDegraded ? VBModel.landscape : VBModel.multiClassLite;
    await videoProcessor.init(model);
  } catch (err) {
    console.error("Virtual background setup failed:", err);
  }
};

Starting the Processor​

To process a video stream, pass the camera MediaStream and a config to start(). The processor returns the processed MediaStream.

  1. MediaStream : MediaStream is essentially a video stream captured from the camera. Use createCameraVideoTrack() to create a MediaStream.
  2. config : This object accepts type, imageUrl, and blurIntensity properties.
    • type : Accepts filter type image or blur.
    • imageUrl : The path of the image, which will be ignored when the type is blur.
    • blurIntensity : The intensity of the blur effect. Accepts a value of BlurIntensity enum (low, mid, or high). The default value is mid. This property will be ignored when the type is image.

The start() method uses the model loaded by init(). If no model is loaded, it uses VBModel.landscape by default.

The method rejects if the model fails to load. It also rejects with an AbortError if a later start(), stop(), or dispose() call cancels it.

note

If you intend to display a specific background image, make sure it is uploaded on a CDN.

info

Blur intensity support is available from version 0.1.1 and above.

import { useMeeting, createCameraVideoTrack } from "@videosdk.live/react-sdk";
import { VirtualBackgroundProcessor, VBModel, BlurIntensity } from "@videosdk.live/videosdk-virtual-background-web";

const MeetingView = () => {
  const { changeWebcam } = useMeeting({});

  const handleStartVirtualBackground = async () => {
    // Configuration for starting processor
    const config = {
      type: "image", // "blur"
      imageUrl: "https://cdn.videosdk.live/virtual-background/cloud.jpeg",
      // blurIntensity: BlurIntensity.mid, // "low" | "mid" | "high" (only applicable if type is "blur")
      // Here is a list of background images you can use for your project.
      // imageUrl: "https://cdn.videosdk.live/virtual-background/beach.jpeg",
      // imageUrl: "https://cdn.videosdk.live/virtual-background/san-fran.jpeg",
      // imageUrl: "https://cdn.videosdk.live/virtual-background/paper-wall.jpeg",
    };

    try {
      // Getting stream from webcam
      const stream = await createCameraVideoTrack({});
      const processedStream = await videoProcessor.start(stream, config);
    } catch (err) {
      // A newer start(), stop(), or dispose() call cancelled this one
      if (err.name === "AbortError") return;
      console.error("start failed:", err);
    }
  };
  return <>...</>;
};

Passing Processed Stream to VideoSDK​

Once you have the processed stream, you can pass it to functions like enableWebcam(), changeWebcam(), or toggleWebcam() to apply the virtual background effect during your live stream.

const MeetingView = () => {
  const { changeWebcam } = useMeeting({});

  const handleStartVirtualBackground = async () => {
    // Configuration for starting processor
    const config = {
      type: "image", // "blur"
      imageUrl: "https://cdn.videosdk.live/virtual-background/cloud.jpeg",
      // blurIntensity: BlurIntensity.mid, // "low" | "mid" | "high" (only applicable if type is "blur")
    };

    try {
      // Getting stream from webcam
      const stream = await createCameraVideoTrack({});
      const processedStream = await videoProcessor.start(stream, config);

      await changeWebcam(processedStream);
    } catch (err) {
      if (err.name === "AbortError") return;
      console.error("Virtual background failed:", err);
    }
  };
  return <>...</>;
};

Updating Video Processor Configuration​

To change the background while the processor is running, call updateProcessorConfig() with the new filter type or image.

const MeetingView = () => {
  const handleChangeConfig = async () => {
    const config = {
      type: "image", // "blur"
      imageUrl: "https://cdn.videosdk.live/virtual-background/cloud.jpeg",
      // blurIntensity: BlurIntensity.mid, // "low" | "mid" | "high" (only applicable if type is "blur")
    };

    try {
      await videoProcessor.updateProcessorConfig(config);
    } catch (err) {
      console.error("updateProcessorConfig failed:", err);
    }
  };
  return <>...</>;
};

Switching Models While Running​

Call init() with another model while the processor is running to switch between VBModel.landscape, VBModel.multiClass, and VBModel.multiClassLite. The host's video keeps streaming to the audience during the switch, and the processed stream stays the same, so there is no need to call changeWebcam() again.

  • When init() calls overlap, the latest call takes effect. Earlier calls reject with an AbortError.
  • Switching to or from VBModel.meet while the processor is running throws an error. Call stop(), then init() with the new model, and then start().
const MeetingView = () => {
  const handleSwitchModel = async (model) => {
    try {
      await videoProcessor.init(model);
    } catch (err) {
      // A newer init() call replaced this one
      if (err.name === "AbortError") return;
      console.error("init failed:", err);
    }
  };

  return (
    <>
      <button onClick={() => handleSwitchModel(VBModel.landscape)}>landscape</button>
      <button onClick={() => handleSwitchModel(VBModel.multiClassLite)}>multiclass-lite</button>
    </>
  );
};

Handling Performance Degradation​

Use on() to register a handler for the performance-degraded event. The processor emits this event while it is running when virtual background performance falls below the expected level. The cause field identifies the reported cause:

  • model: The selected model is too demanding for the host's device. Switch to VBModel.landscape or VBModel.meet.
  • app: The application is keeping the main thread busy.
  • system: The host's computer is under heavy system load. This cause is reported in Chrome and Edge only.

The event also carries model, hardwareType, frameTimeMs, and thresholdMs. It is emitted at most once for each cause and model, and it is not emitted for VBModel.meet. A handler registered after the event was emitted receives the latest result. Use off() to remove the handler.

info

on(), off(), and the performance-degraded event are available from version 1.0.0 and above.

import { useEffect } from "react";

const MeetingView = () => {
  useEffect(() => {
    const onPerformanceDegraded = async (info) => {
      if (info.cause !== "model" || info.model === VBModel.landscape) return;
      try {
        // Switch to the lighter model on this device
        await videoProcessor.init(VBModel.landscape);
      } catch (err) {
        if (err.name !== "AbortError") console.error("init failed:", err);
      }
    };

    videoProcessor.on("performance-degraded", onPerformanceDegraded);
    return () => videoProcessor.off("performance-degraded", onPerformanceDegraded);
  }, []);

  return <>...</>;
};

Stopping Virtual Background Processor​

Use the stop() method to turn off the virtual background. After stopping the processor, pass a new camera stream to changeWebcam() so the audience sees the original video.

If a start() call is still loading, stop() cancels it, and that start() call rejects with an AbortError.

const MeetingView = () => {
  const { changeWebcam } = useMeeting({});

  const handleStopVirtualBackground = async () => {
    await videoProcessor.stop();

    // Pass webcam MediaStream in VideoSDK `changeWebcam` method
    try {
      const stream = await createCameraVideoTrack({});
      await changeWebcam(stream);
    } catch (err) {
      console.error("changeWebcam failed:", err);
    }
  };
  return <>...</>;
};

Disposing Virtual Background Processor​

Use the dispose() method when you no longer need the virtual background, for example when the live stream ends. It frees the memory and graphics resources used by the model, and removes the handlers registered with on().

If an init() or start() call is still in progress, dispose() cancels it, and that call rejects with an AbortError. You can initialize and start the same processor again after calling dispose().

const MeetingView = () => {
  const handleDisposeVirtualBackground = () => {
    videoProcessor.dispose();
  };

  return <>...</>;
};

Extras​

Want your live stream to start with virtual background already applied? Here’s how to pass the processed stream right at the beginning.

import { useState, useEffect } from "react";
import {
  MeetingProvider,
  createCameraVideoTrack,
} from "@videosdk.live/react-sdk";
import { VirtualBackgroundProcessor, VBModel, BlurIntensity } from "@videosdk.live/videosdk-virtual-background-web";

const [mediastream, setMediaStream] = useState(null);

// Instantiate VirtualBackgroundProcessor Class
const videoProcessor = new VirtualBackgroundProcessor();

useEffect(() => {
  const setup = async () => {
    const stream = await createCameraVideoTrack({});

    if (!videoProcessor.ready) {
      await videoProcessor.init(VBModel.landscape); // or VBModel.multiClass | VBModel.multiClassLite | VBModel.meet
    }
    const processedStream = await videoProcessor.start(stream, {
      type: "image", // "blur"
      imageUrl: `https://cdn.videosdk.live/virtual-background/cloud.jpeg`,
      // blurIntensity: BlurIntensity.mid, // "low" | "mid" | "high" (only applicable if type is "blur")
    });
    setMediaStream(processedStream);
  };
  setup();
}, []);

return (
  mediastream && (
    <MeetingProvider
      config={{
        meetingId,
        micEnabled: true,
        webcamEnabled: true,
        name: "TestUser",
        customCameraVideoTrack: mediastream, // Pass processed MediaStream in VideoSDK
      }}
      //...
    >
      <></>
    </MeetingProvider>
  )
);

Migrate from Version 0.1.x​

Version 1.0.0 adds the VBModel.multiClassLite model, the measurePerformance() and getDeviceInfo() methods, and the performance-degraded event. If you are upgrading from version 0.1.x, review the following changes.

  • Update the package: Install the new version with npm install @videosdk.live/videosdk-virtual-background-web@1.0.0.
  • Handle AbortError rejections: A call can now be cancelled by a later call. When init() calls overlap, only the latest one takes effect and the earlier ones reject with an AbortError. A start() call also rejects with an AbortError when a later start(), stop(), or dispose() call cancels it. In your catch blocks, check err.name and skip error handling for AbortError.
  • Use only VBModel values: init() and measurePerformance() now reject a model name that is not a VBModel value. In version 0.1.x, an unknown name fell back to VBModel.landscape.
  • Handle unsupported browsers: init() rejects when the browser does not support WebGL 2, and err.deviceInfo contains the getDeviceInfo() result. Call getDeviceInfo() first to show the virtual background option only to hosts whose devices can run it smoothly.
  • Switch models without restarting: While the processor is running, call init() to switch between VBModel.landscape, VBModel.multiClass, and VBModel.multiClassLite. To switch to or from VBModel.meet, call stop(), then init(), and then start().
  • Reuse the processor after dispose(): In version 0.1.x, you had to create a new instance after dispose(). In version 1.0.0, you can initialize and start the same instance again. dispose() also removes the handlers registered with on().
  • Update your Content Security Policy: VBModel.multiClassLite downloads its model from cdn.videosdk.live. If your application sets a Content Security Policy, add https://cdn.videosdk.live to connect-src.

API Reference​

The API references for all the methods and events utilized in this guide are provided below.

Got a Question? Ask us on discord

Last updated on by ishabodiwala
Previous
Virtual Gifts
Next

Was this helpful?

Not helpfulHelpful