Live Captioning for Livestreams - Android

Live captioning turns the hosts' speech into text while the livestream runs. Start and stop it with startTranscription() and stopTranscription() on Room, and show the text from onTranscriptionText to every viewer.

startTranscription()

Call startTranscription() to start live captions for the whole live stream. It takes an optional TranscriptionConfig with a webhookURL for the finished transcript, a SummaryConfig for an AI summary, and a modelConfig. Pass nothing to use the defaults.

stopTranscription()

Call stopTranscription() to stop live captions for the live stream.

Both methods are suspend functions. Call them from a coroutine and catch VideoSDKException, which they throw when the request fails. Called before the live stream is joined, or while it is reconnecting, they do nothing and report 3035 (ERROR_ACTION_PERFORMED_BEFORE_ROOM_JOINED) or 3065 (ERROR_ROOM_RECONNECTING) on onError instead of throwing.

Example

import android.os.Bundle
import android.util.Log
import android.widget.Button
import androidx.appcompat.app.AppCompatActivity
import androidx.lifecycle.lifecycleScope
import kotlinx.coroutines.launch
import live.videosdk.rtc.android.Room
import live.videosdk.rtc.android.VideoSDK
import live.videosdk.rtc.android.lib.VideoSDKException
import live.videosdk.rtc.android.lib.transcription.SummaryConfig
import live.videosdk.rtc.android.lib.transcription.TranscriptionConfig

class LiveStreamActivity : AppCompatActivity() {

    private lateinit var liveStream: Room

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_live_stream)

        // Create the room (replace with actual room ID and token)
        liveStream = VideoSDK.createRoom(roomId = "<ROOM_ID>", token = "<TOKEN>")

        val startCaptionsButton: Button = findViewById(R.id.btn_start_captions)
        val stopCaptionsButton: Button = findViewById(R.id.btn_stop_captions)

        startCaptionsButton.setOnClickListener {
            startRealtimeTranscription()
        }

        stopCaptionsButton.setOnClickListener {
            stopRealtimeTranscription()
        }
    }

    private fun startRealtimeTranscription() {
        val webhookUrl = "https://www.example.com"

        val summaryConfig = SummaryConfig(
            true,
            "Write summary in sections like Title, Agenda, Speakers, Action Items, Outlines, Notes and Summary"
        )

        val transcriptionConfig = TranscriptionConfig(
            webhookURL = webhookUrl,
            summary = summaryConfig
        )

        lifecycleScope.launch {
            try {
                liveStream.startTranscription(transcriptionConfig)
            } catch (e: VideoSDKException) {
                Log.e("VideoSDK", "Start captions failed: ${e.code} ${e.name}")
            }
        }
    }

    private fun stopRealtimeTranscription() {
        lifecycleScope.launch {
            try {
                liveStream.stopTranscription()
            } catch (e: VideoSDKException) {
                Log.e("VideoSDK", "Stop captions failed: ${e.code} ${e.name}")
            }
        }
    }
}

Events associated with live captioning

• All participants-including all the hosts and audience members receive onTranscriptionStateChanged with the captioning state, as a TranscriptionState. liveStream.transcriptionState holds the latest one.
• All participants-including all the hosts and audience members receive onTranscriptionText with the latest caption whenever a host speaks. The same line can arrive more than once as the service refines it: getType() is TranscriptionType.REALTIME until the line settles and then TranscriptionType.FULL_SENTENCE, so keep only the FULL_SENTENCE ones. getType() is null for a type this SDK version doesn't know. getTimestamp() returns a Long, and 0 when the service sent no timestamp.

import live.videosdk.rtc.android.lib.transcription.TranscriptionState
import live.videosdk.rtc.android.lib.transcription.TranscriptionText
import live.videosdk.rtc.android.lib.transcription.TranscriptionType
import live.videosdk.rtc.android.listeners.RoomEventListener

// Inside LiveStreamActivity
private val roomEventListener: RoomEventListener = object : RoomEventListener() {
  override fun onTranscriptionStateChanged(state: TranscriptionState) {
    when (state) {
        TranscriptionState.TRANSCRIPTION_STARTING -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is starting")
        TranscriptionState.TRANSCRIPTION_STARTED -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is started")
        TranscriptionState.TRANSCRIPTION_STOPPING -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is stopping")
        TranscriptionState.TRANSCRIPTION_STOPPED -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is stopped")
    }
  }

  override fun onTranscriptionText(data: TranscriptionText) {
    // keep only settled lines: `REALTIME` lines are previews that a later line replaces
    if (data.getType() != TranscriptionType.FULL_SENTENCE) return

    val participantName: String = data.getParticipantName()
    val text: String = data.getText()
    val timestamp: Long = data.getTimestamp()

    Log.d("onTranscriptionText", "$participantName: $text $timestamp")
  }
}

override fun onCreate(savedInstanceState: Bundle?) {
  //...

  // add listener to room
  liveStream.addEventListener(roomEventListener)
}

Live translation

To show the hosts' speech translated into each viewer's own language, see Live Translation.

API Reference

The API references for all the methods utilized in this guide are provided below.

• startTranscription()
• stopTranscription()
• onTranscriptionStateChanged
• onTranscriptionText