Realtime Transcription - Android

Realtime transcription turns the speech in a room into text while the room runs. This guide shows how to use the startTranscription() and stopTranscription() methods of the Room class.

You can also set up webhooks for realtime transcription.

Integrating Realtime Transcription Feature

The image above shows:

1. Start Transcription: The SDK Client initiates real-time transcription using the startTranscription method.

2. Resource Acquisition: The VideoSDK server prepares transcription for the room.

   • If the request is denied, startTranscription() throws a VideoSDKException.
   • If the request is successful, the server sends an onTranscriptionStateChanged event with the state TranscriptionState.TRANSCRIPTION_STARTED to the client, indicating that transcription has begun.

3. Transcription Data: As transcription progresses, the client receives an onTranscriptionText event with a TranscriptionText: the text, the speaker's ID and name, a timestamp and a type.

   • getType() returns a TranscriptionType?: REALTIME while the service may still revise a line, FULL_SENTENCE once it has settled, or null for a type this SDK version does not know. Add a line to a transcript only when it is FULL_SENTENCE.
   • getTimestamp() returns the service's timestamp for the line as a Long, or 0 when the service sent none.

4. Stop Transcription: The client stops transcription with stopTranscription().

   • The server then sends an onTranscriptionStateChanged event with the state TranscriptionState.TRANSCRIPTION_STOPPED to confirm that transcription has ended.

Step 1: Configure Realtime Transcription

• In this step, we set up the configuration for realtime transcription. TranscriptionConfig takes three optional arguments: webhookURL, the URL where the webhooks will be received; summary, a SummaryConfig for the AI summary; and modelConfig, a JSONObject of model settings. Each defaults to null.

// Webhook URL where, webhooks are received
val webhookUrl = "YOUR_WEBHOOK_URL"

// Realtime Transcription Configuration
val summaryConfig = SummaryConfig(
    true,
    "Write summary in sections like Title, Agenda, Speakers, Action Items, Outlines, Notes and Summary"
)
val transcription = TranscriptionConfig(
    webhookURL = webhookUrl,
    summary = summaryConfig
)

Step 2: Listen for the transcription events

• Add these callbacks to your RoomEventListener.
• Every participant receives the text of every speaker. The same text can arrive more than once as the service refines it.

private val roomEventListener: RoomEventListener = object : RoomEventListener() {
  override fun onTranscriptionStateChanged(state: TranscriptionState) {
    when (state) {
        TranscriptionState.TRANSCRIPTION_STARTING -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is starting")
        TranscriptionState.TRANSCRIPTION_STARTED -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is started")
        TranscriptionState.TRANSCRIPTION_STOPPING -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is stopping")
        TranscriptionState.TRANSCRIPTION_STOPPED -> Log.d("onTranscriptionStateChanged", "Realtime Transcription is stopped")
    }
  }

  override fun onTranscriptionText(data: TranscriptionText) {
    // keep only settled lines: `REALTIME` lines are previews that a later line replaces
    if (data.getType() != TranscriptionType.FULL_SENTENCE) return

    val participantName: String = data.getParticipantName()
    val text: String = data.getText()
    val timestamp: Long = data.getTimestamp()

    Log.d("onTranscriptionText", "$participantName: $text $timestamp")
  }
}

override fun onCreate(savedInstanceState: Bundle?) {
  //...

  // add listener to room
  room!!.addEventListener(roomEventListener)
}

Step 3: Start realtime transcription

• Initiate the realtime transcription using the startTranscription() method.

fun startRealtimeTranscription() {
    // Webhook URL where, webhooks are received
    val webhookUrl = "YOUR_WEBHOOK_URL"

    // Realtime Transcription Configuration
    val summaryConfig = SummaryConfig(
        true,
        "Write summary in sections like Title, Agenda, Speakers, Action Items, Outlines, Notes and Summary"
    )
    val transcription = TranscriptionConfig(
        webhookURL = webhookUrl,
        summary = summaryConfig
    )

    lifecycleScope.launch {
        try {
            room!!.startTranscription(transcription)
        } catch (e: VideoSDKException) {
            Log.e("VideoSDK", "Start transcription failed: ${e.code} ${e.name}")
        }
    }
}

note

• startTranscription() and stopTranscription() are suspend functions. Call them from a coroutine, such as lifecycleScope.launch (from androidx.lifecycle:lifecycle-runtime-ktx), and catch VideoSDKException, which they throw when the request fails.
• They return before transcription has started or stopped. Wait for onTranscriptionStateChanged to report TRANSCRIPTION_STARTED or TRANSCRIPTION_STOPPED.
• Called before the room is joined, or while it is reconnecting, they do nothing and report 3035 (ERROR_ACTION_PERFORMED_BEFORE_ROOM_JOINED) or 3065 (ERROR_ROOM_RECONNECTING) on onError instead of throwing.

Step 4: Stop realtime transcription

• Terminate the realtime transcription using the stopTranscription() method.

fun stopRealtimeTranscription() {
    lifecycleScope.launch {
        try {
            room!!.stopTranscription()
        } catch (e: VideoSDKException) {
            Log.e("VideoSDK", "Stop transcription failed: ${e.code} ${e.name}")
        }
    }
}

important

You can access a summary of your realtime transcription using the Fetch Realtime Transcription API.

Example

• The following code snippet allows you to start and stop realtime transcription with just a click. When you click the "Start Realtime Transcription" button, it begins realtime transcription. Clicking the "Stop Realtime Transcription" button ends the realtime transcription.

import android.os.Bundle
import android.util.Log
import android.widget.Button
import androidx.appcompat.app.AppCompatActivity
import androidx.lifecycle.lifecycleScope
import kotlinx.coroutines.launch
import live.videosdk.rtc.android.Room
import live.videosdk.rtc.android.lib.VideoSDKException
import live.videosdk.rtc.android.lib.transcription.SummaryConfig
import live.videosdk.rtc.android.lib.transcription.TranscriptionConfig
import live.videosdk.rtc.android.lib.transcription.TranscriptionState
import live.videosdk.rtc.android.lib.transcription.TranscriptionText
import live.videosdk.rtc.android.lib.transcription.TranscriptionType
import live.videosdk.rtc.android.listeners.RoomEventListener

class RoomActivity : AppCompatActivity() {
    private lateinit var room: Room

    // add roomEventListener to handle transcription events
    private val roomEventListener = object : RoomEventListener() {
        override fun onTranscriptionStateChanged(state: TranscriptionState) {
            Log.d("RoomActivity", "Transcription state: $state")
        }

        override fun onTranscriptionText(data: TranscriptionText) {
            if (data.getType() != TranscriptionType.FULL_SENTENCE) return
            val participantName = data.getParticipantName()
            val text = data.getText()
            val timestamp = data.getTimestamp()

            Log.d("RoomActivity", "$participantName: $text $timestamp")
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_room)

        // create the room
        //...

        val startButton: Button = findViewById(R.id.startTranscriptionButton)
        val stopButton: Button = findViewById(R.id.stopTranscriptionButton)

        startButton.setOnClickListener { startRealtimeTranscription() }
        stopButton.setOnClickListener { stopRealtimeTranscription() }

        // add listener to room
        room.addEventListener(roomEventListener)
    }

    private fun startRealtimeTranscription() {
        // Configuration for realtime transcription
        val summaryConfig = SummaryConfig(
            true,
            "Write summary in sections like Title, Agenda, Speakers, Action Items, Outlines, Notes and Summary"
        )
        val transcriptionConfig = TranscriptionConfig(
            summary = summaryConfig
        )

        // Call to start transcription with the specified configuration
        lifecycleScope.launch {
            try {
                room.startTranscription(transcriptionConfig)
            } catch (e: VideoSDKException) {
                Log.e("RoomActivity", "Start transcription failed: ${e.code} ${e.name}")
            }
        }
    }

    private fun stopRealtimeTranscription() {
        // Call to stop transcription
        lifecycleScope.launch {
            try {
                room.stopTranscription()
            } catch (e: VideoSDKException) {
                Log.e("RoomActivity", "Stop transcription failed: ${e.code} ${e.name}")
            }
        }
    }
}

Realtime Translation

To show the speech in the room translated into each participant's own language, see Realtime Translation.

API Reference

The API references for all the methods utilized in this guide are provided below.

• startTranscription()
• stopTranscription()
• onTranscriptionStateChanged
• onTranscriptionText