Testing AI Voices: How to Measure MOS and WER for Vietnamese Text-to-Speech

Learn how to rigorously evaluate Vietnamese TTS systems. This guide covers MOS and WER methodologies, specific linguistic challenges, and practical optimization tips.

In this article
  1. Why Measure TTS Quality with MOS and WER?
  2. Measuring MOS for Vietnamese Speech
  3. Preparing the Evaluation Dataset
  4. Organizing the Evaluation Session
  5. Calculating WER for Vietnamese TTS
  6. Challenges in Measuring WER for Vietnamese
  7. Implementation Method
  8. Practical Examples and Optimization Tips
  9. The Importance of High-Quality Vietnamese Datasets
  10. Conclusion and Call to Action

In the natural language processing industry, developing a Text-to-Speech (TTS) system is not just about generating audio. To ensure a high-quality user experience, engineers and developers need a set of quantitative metrics to evaluate the naturalness and accuracy of the generated voice. This article provides a detailed guide on the TTS quality measurement process, focusing on two core metrics: Mean Opinion Score (MOS) and Word Error Rate (WER) for the Vietnamese language. It helps you build a scientific basis for optimizing your product.

Why Measure TTS Quality with MOS and WER?

Many developers evaluate AI voices subjectively: they listen to a few sentences and deem them "acceptable." However, in enterprise environments or real-world applications, intuition cannot replace data. When deploying TTS for scenarios such as customer support call centers, technical book reading, or accessibility for the visually impaired, two factors are decisive: naturalness and content accuracy.

  • MOS (Mean Opinion Score): A subjective evaluation of the voice's naturalness, emotion, and listenability.
  • WER (Word Error Rate): An objective evaluation of the accuracy of the spoken content compared to the original text, typically measured by transcribing the audio using Automatic Speech Recognition (ASR).

Combining both metrics provides a comprehensive picture of TTS performance. If MOS is high but WER is low (meaning few errors), the voice is natural but the content might be misunderstood. If WER is high (many errors) but MOS is low, the content is correct but the robotic delivery causes listener fatigue.

Measuring MOS for Vietnamese Speech

MOS is the gold standard for subjective audio quality assessment. According to ITU-T P.800 recommendations, MOS is usually rated on a 5-point scale:

  • 5: Excellent, no perceptible flaws.
  • 4: Good, minor flaws that are not annoying.
  • 3: Fair, noticeable flaws but still acceptable.
  • 2: Bad, obvious flaws that are annoying.
  • 1: Very bad, unintelligible.

Preparing the Evaluation Dataset

To obtain reliable MOS results for Vietnamese, you need a diverse evaluation set. This dataset should include:

  • Sentences of varying lengths (short, medium, long).
  • Different topics (daily conversation, news, technical, medical).
  • Complex sentence structures containing abbreviations or numerical data.

It is recommended to use 30 to 50 sample sentences to ensure representativeness. These sentences are fed into the TTS model to generate audio files.

Organizing the Evaluation Session

You need to invite a group of listeners who are not familiar with the TTS model to avoid bias. The minimum number of evaluators should be 10, but ideally 20-30 to reduce variance.

  1. Listening Environment: Ensure a quiet room and use high-quality headphones or professional speakers.
  2. Random Order: Audio files must be played in a random sequence to prevent evaluators from memorizing the order.
  3. Context Support: Provide evaluators with the original text so they can cross-reference the content while listening.
  4. Scoring: Each listener rates every sentence from 1 to 5.

The final MOS result is the arithmetic mean of all scores. A high-quality Vietnamese TTS model typically achieves an MOS of 4.0 or higher.

Calculating WER for Vietnamese TTS

WER is the most precise quantitative metric for measuring content errors. The formula for WER is:

$$WER = \frac{S + D + I}{N}$$

Where:

  • S: Number of Substitutions
  • D: Number of Deletions
  • I: Number of Insertions
  • N: Total number of words in the reference sentence

Challenges in Measuring WER for Vietnamese

Measuring WER for Vietnamese is more complex than for English due to linguistic characteristics:

  1. Word Segmentation: Vietnamese is a monosyllabic language. Word segmentation significantly impacts WER results. A phrase like "Tôi đi học" might be counted as 3 or 4 words depending on the system's segmentation logic.
  2. Tones and Diacritics: Tone errors can change word meaning, leading to substitution errors even if the audio sounds close to the original.

Implementation Method

To calculate WER automatically, you need a high-accuracy ASR (Speech-to-Text) system. The process is as follows:

  1. Obtain the audio file from the TTS model.
  2. Run the audio file through the ASR system to get the transcribed text.
  3. Compare the transcribed text with the original reference text used for TTS generation.
  4. Use libraries such as jiwer (Python) to calculate the WER.

Important note: The ASR system used to measure WER should be independent of, or more accurate than, the TTS model being tested. If the ASR is weak, the measured WER will be skewed.

Practical Examples and Optimization Tips

Suppose you are developing a TTS for a news reading application. You generate 10 sample sentences.

  • MOS Result: 4.2/5 (Natural and clear voice).
  • WER Result: 12% (Relatively high word error rate).

Root Cause Analysis: Upon detailed inspection, you notice that the high WER is due to substitution errors of words with similar sounds (e.g., "thanh" vs. "sanh") or tone mark deletions.

Actionable Advice:

  • Text Preprocessing: Review your text normalization pipeline before feeding it into the TTS. Ensure abbreviations, numbers, and specific symbols are converted into standard phonetic forms before entering the model.
  • Training Data Optimization: If WER is high for a specific group of words, supplement the training data for that group.
  • Code-Switching Check: If the TTS supports both Vietnamese and English, carefully test sentences with mixed languages. This is often where errors are most frequent.

The Importance of High-Quality Vietnamese Datasets

TTS quality relies heavily on the training data. Owning a large Vietnamese corpus with diverse voices and contexts is a key factor. At AIVISION, we have assembled a large-scale Vietnamese speech corpus comprising over 690,517 hours of recorded audio. This allows our TTS models to deeply understand the specific phonetics and intonations of Vietnamese. This not only helps improve MOS but also minimizes WER caused by pronunciation errors.

Conclusion and Call to Action

Measuring TTS quality with MOS and WER is not a final step but a continuous process throughout the product development lifecycle. By applying standardized measurement methods, you can accurately identify model weaknesses and implement effective optimizations.

If you are looking for a high-accuracy Vietnamese TTS solution with natural voices and rigorous testing, try the s2speech platform. AIVISION provides TTS APIs with multilingual processing capabilities, streaming support, and telephony formats, suitable for needs ranging from enterprise to individual use.

Start free to experience the quality of our AI voices today. If you have any technical questions about the testing process, feel free to Contact our engineering team.

Frequently asked questions

How do MOS and WER differ in TTS evaluation?

MOS is a subjective metric that evaluates the naturalness and listenability of the voice based on human perception. WER is an objective metric that measures the accuracy of the spoken content compared to the original text using speech recognition technology.

How many evaluators are needed for a reliable MOS result?

According to international standards, a minimum of 10 evaluators is necessary, but to achieve high reliability and reduce statistical error, it is recommended to use 20 to 30 independent evaluators.

Why is WER for Vietnamese often harder to measure accurately than English?

Vietnamese is a monosyllabic language with a complex tone system. Word segmentation and intonation handling significantly affect comparison results, making WER calculation require an ASR system with strong Vietnamese support.

Does AIVISION provide TTS measurement tools?

AIVISION focuses on providing high-quality TTS and STT models. However, with our accurate STT API, you can easily automate the WER calculation process for your TTS audio files.

How can I improve the MOS of a TTS model?

To improve MOS, you must focus on increasing the naturalness of the voice. This is achieved by using high-quality training data with diverse contexts and applying modern speech synthesis techniques to simulate natural emotion and rhythm.

Try AIVISION's Vietnamese speech AI

Speech-to-Text, Text-to-Speech and LLM, free.

Start free → Contact

By

Dr. Giang Vo — CEO, AIVISION

Dr. Giang Vo is the CEO of AIVISION, leading the development of s2speech — speech recognition, AI voices and a large language model for Vietnamese — and working with companies at home and abroad on their deployments.

0981 419 967 · giang.vo@aivgroups.com

#Vietnamese TTS#MOS#WER#speech quality#text-to-speech evaluation#AIVISION#s2speech#speech AI

Related articles