Adding multilingual AI dubbing to videos

Log into Smarcat and go to your Smartcat Workspace Home page. Click on the shortcut called Edit and translate videos/subtitles.

Then, choose the source language of your video/subtitle file. You can choose from the 280 different languages supported by Smartcat AI.

Next, upload your file. You can choose a video (MP4, mpeg, avi, mov, 3gp, and other video file types) and subtitle files (VTT, SRT). We will soon add audio file types to this new Subtitle editor. It will take a few minutes for the file to upload and then transcribe, depending on the file size. You’ll see the current step and the progress of the current operation.

After it has processed, click on Preview, which will bring you to the Subtitle Editor.

You can start by reviewing the resulting script. Ensure that the transcription did its job correctly. While the Smartcat AI transcription is 95% correct, there may be times when a proper name is not recognized by the system. Or, if the speaker has an accent, Smartcat AI may not be able to pick up on each inflection, which is why double checking the source text is important. You can preview the source video and ensure that all timecodes are as you want them.

In the Subtitle Editor, you can do the following:

1. Merge cues: this merges the cues that are selected into one.

2. Insert a new cue (before or after): this will insert an empty cue  

3. Delete the whole cue: this will delete a cue

4. Split by line: this will split the lines into two separate cues

After you get the automatic transcription of your video’s audio in Smartcat Subtitle editor, you might want to adjust the cues. When you adjust time stamps, we recommend also making sure that the complete sentence or clause is in one cue to avoid any pauses in the AI dubbing.

In video editing, a cue is a specific marker used to indicate where subtitles, transcripts, or captions should appear in the timeline. These cues ensure that text elements are synchronized accurately with the corresponding audio and visual content.

"Cue per second" (CPS) refers to the number of characters that appear within one second of video. It is a measure used to evaluate the density of text on the screen and ensure readability. A lower CPS ensures that subtitles or captions are displayed at a pace that viewers can comfortably read and understand. It also refers to speed of speech for AI dubbing. Higher CPS might indicate unnaturally fast speed of AI voice.

1. Select Dubbing Voice: Choose the desired dubbing voice from the dropdown menu. In this case, "Rachel (Female 20 CPS)" is selected.

2. Adjust Cue Settings: Modify the cue settings according to your requirements, including line size, lines per cue, cue time, and CPS.

3. Set Min and Max Cue Time: Specify the minimum and maximum duration for each cue to ensure proper synchronization with the video.

Max Line Size

Max Line Size: The maximum number of characters allowed in a subtitle line. This helps maintain readability by preventing lines from being too long.

Max Lines Per Cue: The maximum number of lines that can appear in a single subtitle cue. This ensures that subtitles are not too dense and remain easy to read.

Min and Max Cue Time

Min Cue Time: The minimum duration that a subtitle cue will remain on screen. This ensures that even short subtitles are displayed long enough to be read.

Max Cue Time: The maximum duration that a subtitle cue will remain on screen. This prevents subtitles from staying on screen too long, which could interfere with the reading flow of subsequent subtitles.

Max CPS (characters per second): The maximum number of characters per second. This ensures that subtitles are not too dense, maintaining readability and preventing viewer overload. In this case, the max CPS is set to 20.

After you have finished reviewing script cues in the original language, it’s time to translate your subtitles. To start the translation process, click the Translate button.

That will open the language selector window. Click the Translate button again after you have selected the target language.

You’ll see that the editor has been split into two sections now. Text segments in source language are displayed on the left and the text segments in the target language are displayed on the right, respectively.

Make necessary adjustments to the target language text and time codes as you have already done for the source text cues. It’s important to understand that the translated script inherits the timecodes from the original script and the translated cues will not always match the precise timecodes of the original script.

However, you still can ensure correct positioning with the script structure and with the context of the video with the help of Video Preview.

To preview AI dubbing, follow these steps:

1. Select the AI voice under the Video Preview section.

2. Choose the desired AI voice from the list.

3. Click on the AI voice button, as shown below:

This process works for both original and translated cues. You will hear a preview of the AI voice in the target or source language.

Initiate the AI dubbing process by clicking the Dub video button above the video preview section.

In the following pop-up, specify the subtitle language (the language that will be used for AI dubbing), the source language, and the AI dubbing voice.

Preview the AI voices from the list if needed.

Once all details are entered, click Dub video. Smartcat will automatically generate a fully dubbed video.

Keep track of the dubbing process in the Video Preview section. When it's done, you will see a notification indicating that a new version of the video is available. Click Reload to update your video file in the Smartcat Subtitle editor.

Use the language selector above to switch between language versions of your video.

Once the AI dubbing process is complete, download the resulting fully AI-dubbed video files. These files are ready for distribution across various platforms. You can download the video with AI dubbing for any of your language versions directly by navigating to the file in Drive.

In Drive, you will also see the AI dubbing progress and find all your files in one spot.

If you attempt to dub a video that has already been dubbed, a warning will appear to notify you that proceeding will overwrite the existing dubbed version.

Confirm if you wish to proceed with the new dubbing. If confirmed, the new voice selection and dubbing process will replace the previous version.

Once the new dubbing is complete, review the updated version to ensure it meets your expectations.