Realtime Translation - Android

Realtime translation turns the speech in a room into text, translated into each participant's own language, while the room runs. This guide explains how to use the startTranslation(), stopTranslation() and changeTranslationLanguage() methods of the Room class.

You set a participant's translation language in VideoSDK.createRoom(). Once translation is running, you can change it with changeTranslationLanguage().

note

The translation language is always a language code, both in the translationLanguage parameter of VideoSDK.createRoom() and in changeTranslationLanguage(). Only the codes listed in Supported Languages return translated text. If you pass an unsupported code, no translated text arrives.

Integrating Realtime Translation Feature

1. Start Translation: You start realtime translation with the startTranslation() method. The onTranslationStateChanged event then reports TRANSLATION_STARTING, followed by TRANSLATION_STARTED once translation is running.

2. Translation Data: As translation progresses, the onTranslationText event delivers a TranslationText object with the translated text, the participant who spoke it, the timestamp and the language.

3. Change Translation Language: A participant can switch to another language while translation is running with changeTranslationLanguage(). The onTranslationLanguageChanged event then reports the new language.

4. Stop Translation: When you stop translation with stopTranslation(), the onTranslationStateChanged event reports TRANSLATION_STOPPING, followed by TRANSLATION_STOPPED.

Step 1: Configure the translation languages

• Set the participant's translation language and speaking language when you create the room with VideoSDK.createRoom().

val room = VideoSDK.createRoom(
    roomId = roomId,
    token = token,
    participantName = "John Doe",
    translationLanguage = "hi",
    speakingLanguage = "en",
)

• translationLanguage: The language the participant receives translated text in. onTranslationText delivers text only in this language, so a participant without a translation language receives no translated text until they call changeTranslationLanguage().

• speakingLanguage: The language the participant speaks for the whole session. When it is not set, the participant's translationLanguage is also used as their spoken language. Refer to Speaking Language and Translation Language for how the two work together.

Step 2: Start realtime translation

• Start realtime translation for the room with the startTranslation() method, once the room is joined.

lifecycleScope.launch {
    try {
        room.startTranslation()
    } catch (e: VideoSDKException) {
        Log.e("VideoSDK", "Start translation failed: ${e.code} ${e.name}")
    }
}

• One translation runs for the whole room. A participant who joins while translation is already running receives TRANSLATION_STARTED too, before its own onRoomJoined. Add your RoomEventListener before you call join(), or you miss it.

Step 3: Change the translation language

• Switch to another language with the changeTranslationLanguage() method while translation is running. It changes the language for this participant only.
• If the server refuses the change, for example because translation has not started yet, it throws a VideoSDKException with code 4044 (CHANGE_TRANSLATION_LANGUAGE_FAILED).

lifecycleScope.launch {
    try {
        room.changeTranslationLanguage("fr")
    } catch (e: VideoSDKException) {
        Log.e("VideoSDK", "Language change failed: ${e.code} ${e.name}")
    }
}

• The new language applies once onTranslationLanguageChanged reports it for this participant. Until then, onTranslationText keeps delivering the old language.

Step 4: Stop realtime translation

• Stop realtime translation with the stopTranslation() method. It stops translation for every participant in the room.

lifecycleScope.launch {
    try {
        room.stopTranslation()
    } catch (e: VideoSDKException) {
        Log.e("VideoSDK", "Stop translation failed: ${e.code} ${e.name}")
    }
}

note

• startTranslation(), stopTranslation() and changeTranslationLanguage() are suspend functions. Call them from a coroutine, such as lifecycleScope.launch, and catch VideoSDKException, which they throw when the request fails.
• Called before the room is joined, or while it is reconnecting, they do nothing and report 3035 (ERROR_ACTION_PERFORMED_BEFORE_ROOM_JOINED) or 3065 (ERROR_ROOM_RECONNECTING) on onError instead of throwing.

Step 5: Listen for the translation events

• Add these callbacks to your RoomEventListener.

private val roomEventListener = object : RoomEventListener() {
    override fun onTranslationStateChanged(state: TranslationState) {
        when (state) {
            TranslationState.TRANSLATION_STARTING -> Log.d("Translation", "Realtime Translation is starting")
            TranslationState.TRANSLATION_STARTED -> Log.d("Translation", "Realtime Translation is started")
            TranslationState.TRANSLATION_STOPPING -> Log.d("Translation", "Realtime Translation is stopping")
            TranslationState.TRANSLATION_STOPPED -> Log.d("Translation", "Realtime Translation is stopped")
        }
    }

    override fun onTranslationText(data: TranslationText) {
        Log.d("Translation", "${data.getParticipantName()} (${data.getLanguage()}): ${data.getText()}")
    }

    override fun onTranslationLanguageChanged(participantId: String, language: String) {
        Log.d("Translation", "$participantId now receives translations in $language")
    }
}

• onTranslationStateChanged delivers a TranslationState: TRANSLATION_STARTING, TRANSLATION_STARTED, TRANSLATION_STOPPING or TRANSLATION_STOPPED.

• onTranslationText delivers a TranslationText with getText(), getParticipantId(), getParticipantName(), getParticipantLanguage() (the language the speaker spoke), getLanguage() (the language the text was translated into), getTimestamp() and getType(). It fires only for text in this participant's translation language.

• getType() is TranslationType.PARTIAL while the service is still revising a line, TranslationType.FULL once the line is final, or null for a type this SDK version does not know. A new PARTIAL line replaces the previous one, so add a line to a transcript only when it is FULL.

• onTranslationLanguageChanged fires when any participant changes their translation language, including you. Compare participantId with room.localParticipant.id to react only to your own change.

Example

• The following code starts and stops realtime translation, and switches the language, with a click. Partial lines replace each other, and full lines are kept.

class RoomActivity : AppCompatActivity() {
    private lateinit var room: Room
    private lateinit var captionView: TextView
    private val finishedLines = mutableListOf<String>()

    private val roomEventListener = object : RoomEventListener() {
        override fun onTranslationStateChanged(state: TranslationState) {
            Log.d("Translation", "Translation state: $state")
        }

        override fun onTranslationText(data: TranslationText) {
            val line = "${data.getParticipantName()}: ${data.getText()}"
            if (data.getType() == TranslationType.FULL) {
                finishedLines.add(line)
                captionView.text = finishedLines.takeLast(3).joinToString("\n")
            } else {
                // a partial line replaces the previous partial line
                captionView.text = (finishedLines.takeLast(2) + line).joinToString("\n")
            }
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_room)
        captionView = findViewById(R.id.captionView)

        // create the room with a translationLanguage and join it
        // ...

        room.addEventListener(roomEventListener)

        findViewById<Button>(R.id.btnStartTranslation).setOnClickListener { runTranslationAction { room.startTranslation() } }
        findViewById<Button>(R.id.btnStopTranslation).setOnClickListener { runTranslationAction { room.stopTranslation() } }
        findViewById<Button>(R.id.btnFrench).setOnClickListener { runTranslationAction { room.changeTranslationLanguage("fr") } }
    }

    private fun runTranslationAction(action: suspend () -> Unit) {
        lifecycleScope.launch {
            try {
                action()
            } catch (e: VideoSDKException) {
                Log.e("Translation", "Translation request failed: ${e.code} ${e.name}")
            }
        }
    }
}

Speaking Language and Translation Language

speakingLanguage sets the language a participant speaks, while translationLanguage sets the language they receive translated text in. The two examples below show how translation works with and without a speaking language.

Example 1: Without speakingLanguage

Both participants set only their translation language, so each participant's translation language is also their spoken language.

// On Participant 1's device (Spanish)
val room1 = VideoSDK.createRoom(
    roomId = roomId,
    token = token,
    participantName = "Participant 1",
    translationLanguage = "es",
)

// On Participant 2's device (German)
val room2 = VideoSDK.createRoom(
    roomId = roomId,
    token = token,
    participantName = "Participant 2",
    translationLanguage = "de",
)

Speaker	Speaking Language (Input)	Receiver	Translated Output
Participant 1	Spanish (es)	Participant 2	German (de)
Participant 2	German (de)	Participant 1	Spanish (es)

Here each participant must speak their own translation language, because no speaking language is set.

Example 2: With speakingLanguage on one participant

Here only Participant 1 sets a speakingLanguage. That fixes the language of their speech, while Participant 2's translation language is still also their spoken language.

// On Participant 1's device (speaks English, receives Spanish)
val room1 = VideoSDK.createRoom(
    roomId = roomId,
    token = token,
    participantName = "Participant 1",
    translationLanguage = "es",
    speakingLanguage = "en",
)

// On Participant 2's device (no speakingLanguage, receives German)
val room2 = VideoSDK.createRoom(
    roomId = roomId,
    token = token,
    participantName = "Participant 2",
    translationLanguage = "de",
)

Speaker	Speaking Language (Input)	Receiver	Translated Output
Participant 1	English (en)	Participant 2	German (de)
Participant 2	German (de)	Participant 1	Spanish (es)

Key takeaways

• When speakingLanguage is not set, the participant's translation language is also their speaking language.
• When only one participant sets a speakingLanguage, that participant's input language is fixed, while the others keep using their translation language as their input language.
• Setting speakingLanguage gives you more control, especially in rooms with many languages.

Supported Languages

These are the supported language codes:

• Multilingual (Spanish + English): multi
• Bulgarian: bg
• Catalan: ca
• Chinese (Mandarin, Simplified): zh, zh-CN, zh-Hans
• Chinese (Mandarin, Traditional): zh-TW, zh-Hant
• Chinese (Cantonese, Traditional): zh-HK
• Czech: cs
• Danish: da, da-DK
• Dutch: nl
• English: en, en-US, en-AU, en-GB, en-NZ, en-IN
• Estonian: et
• Finnish: fi
• Flemish: nl-BE
• French: fr, fr-CA
• German: de
• German (Switzerland): de-CH
• Greek: el
• Hindi: hi
• Hungarian: hu
• Indonesian: id
• Italian: it
• Japanese: ja
• Korean: ko, ko-KR
• Latvian: lv
• Lithuanian: lt
• Malay: ms
• Norwegian: no
• Polish: pl
• Portuguese: pt, pt-BR, pt-PT
• Romanian: ro
• Russian: ru
• Slovak: sk
• Spanish: es, es-419
• Swedish: sv, sv-SE
• Thai: th, th-TH
• Turkish: tr
• Ukrainian: uk
• Vietnamese: vi

API Reference

The API references for all the methods and events utilized in this guide are provided below.

• translationLanguage
• speakingLanguage
• startTranslation()
• stopTranslation()
• changeTranslationLanguage()
• onTranslationStateChanged
• onTranslationText
• onTranslationLanguageChanged