Testing AI Voices: How to Measure MOS and WER in Vietnamese
Learn how to evaluate TTS quality in Vietnamese. This guide explains how to calculate MOS scores and Word Error Rate (WER) for accurate, natural AI voices.
In this article
In the field of speech AI, developing a Text-to-Speech (TTS) model is only the first step. The real challenge lies in proving that the synthesized voice is natural, accurate, and suitable for your business context. To achieve this, engineers and product managers must rely on objective quantitative metrics. This article provides a detailed guide on how to measure TTS quality, with a specific focus on two critical indicators: the Vietnamese MOS score and the Word Error Rate (WER).
Why Deep TTS Quality Measurement Matters
Many users evaluate AI voices based on subjective impressions, such as "it sounds good" or "it is clear." However, in an enterprise environment, subjective feelings are insufficient for deployment decisions. A voice may sound harmonious but mispronounce industry-specific terms, or its reading speed might be uneven, causing discomfort for the listener.
Applying standardized measurement criteria helps to:
- Standardize development workflows: Fairly compare the performance of different model versions or configurations.
- Minimize deployment risks: Identify pronunciation errors or clarity issues early, before they reach the live product.
- Optimize user experience: Ensure the voice aligns with communication culture and the specific linguistic characteristics of Vietnamese.
MOS Score: Measuring Voice Naturalness
The Mean Opinion Score (MOS) is the most common subjective metric in the speech industry. It reflects human satisfaction with synthesized speech.
Standard MOS Evaluation Process
To obtain a reliable Vietnamese MOS score, you must follow a rigorous process:
- Data Preparation: Select a representative set of sentences covering various contexts (questioning, announcements, conversation). A set of 100–200 sentences is recommended to ensure statistical relevance.
- Evaluator Selection: Use a group of evaluators who are not involved in the model development process (blind test). Evaluators should be native speakers with normal hearing.
- Scoring Scale: Evaluators rate the audio from 1 to 5, where:
- 5: Perfect, indistinguishable from a human.
- 4: Natural, with minor, negligible errors.
- 3: Acceptable, with clear errors that do not hinder understanding.
- 2: Difficult to listen to, with many pronunciation or pausing errors.
- 1: Very poor, content is unintelligible.
- Calculate the Average: Compute the arithmetic mean of all evaluation scores.
Vietnamese Specific Considerations
Vietnamese is a tonal language with five distinct tones. A TTS voice may have good timbre, but if it gets the tone wrong (for example, reading "bà" as "ba" or "bá"), listeners will find it highly unpleasant, even if the acoustic clarity is high. Therefore, when designing your MOS test set, include words with easily confused tones and phrases with complex final consonants (such as -t, -c, -k), which are significant challenges for TTS models.
WER for TTS: Measuring Content Accuracy
If MOS measures how "good" a voice sounds, WER measures how "correctly" it speaks. In the TTS context, WER is typically calculated by using a robust Speech-to-Text (STT) system to transcribe the synthesized audio, then comparing that transcription against the original ground-truth text.
The WER formula is:
WER = (S + D + I) / N
Where:
- S: Number of Substitutions
- D: Number of Deletions
- I: Number of Insertions
- N: Total number of words in the reference text
How to Perform Effective WER Testing
To measure TTS WER accurately, you need a highly reliable reference STT system. Here are the steps:
- Create a Test Dataset: Include text segments of varying lengths, containing numbers, addresses, proper nouns, and industry terminology.
- Synthesize Speech: Use the TTS model under test to generate the corresponding audio files.
- Transcribe Audio: Use an independent STT system (ideally from a different source than the TTS) to transcribe these audio files.
- Compare and Score: Use string comparison tools (such as Levenshtein distance) to calculate the WER.
Practical Example:
- Original Text: "Hôm nay trời đẹp" (5 words)
- STT Result: "Hôm nay trời đệp"
- Error: 1 substitution (đẹp -> đệp)
- WER = 1/5 = 20%
A low WER (e.g., under 5–10%) indicates that the TTS model pronounces words accurately, minimizing errors that would confuse the STT system. However, note that WER depends on the quality of the reference STT. If the reference STT is weak, the measured WER will be skewed.
MOS vs. WER: A Comparison
| Feature | MOS Score | WER (Word Error Rate) |
|---|---|---|
| Nature | Subjective (Human-in-the-loop) | Objective (Automated) |
| Measures | Naturalness, emotion, rhythm | Word pronunciation accuracy |
| Cost | High (requires human evaluators) | Low (automated) |
| Time | Slow (hours to days) | Fast (minutes) |
| Strengths | Reflects real user experience | Easy to repeat, quick comparison |
| Weaknesses | Hard to standardize, high variance | Depends on reference STT quality |
Practical Advice for Engineering Teams
Based on experience deploying speech solutions for numerous enterprises, here are specific recommendations to optimize your testing workflow:
- Combine Both Metrics: Do not rely solely on MOS or WER. A high-quality TTS system needs both a high MOS (naturalness) and a low WER (accuracy).
- Analyze Errors in Detail: Don't just look at the aggregate WER number. Analyze which words are being mispronounced most frequently. If errors concentrate in medical or financial terms, you may need to supplement training data with domain-specific content or adjust the pronunciation lexicon.
- Test Diverse Contexts: Ensure your test data includes varying speech rates, volume levels, and background noise (if the real-world application involves such conditions).
- Monitor Over Time: TTS quality can change when you update models or adjust parameters. Build an automated testing pipeline to run periodically.
At AIVISION, we understand these challenges deeply. With a large-scale Vietnamese data foundation and experience deploying for hundreds of enterprises in Vietnam and abroad, we have established rigorous quality testing processes for our speech products. AIVISION's TTS models are optimized to ensure accurate pronunciation of Vietnamese tones and natural reading of embedded English words.
Conclusion
Measuring TTS quality is not optional; it is a mandatory requirement for building professional voice applications. Mastering how to calculate the Vietnamese MOS score and TTS WER will help your engineering team make informed decisions, ultimately creating a high-quality user experience.
If you are looking for a Vietnamese TTS solution with high accuracy and flexible customization, explore AIVISION's speech technologies. We are committed to providing tools that help your business communicate more effectively with customers.
Start improving your AI voice quality today. You can learn more about our solutions or Start free to experience the TTS quality directly. If you need detailed technical consultation on testing processes, do not hesitate to Contact our team of experts.
Frequently asked questions
What is an acceptable MOS score for commercial applications?
Generally, a MOS score of 4.0 or higher is considered acceptable for commercial applications. However, for premium products or those with high user experience requirements, the target is often 4.5 or higher.
How can I improve the WER of my TTS system?
To improve WER, ensure your TTS training data is high-quality, recorded in a clean environment, and accurately annotated. Additionally, optimizing the pronunciation lexicon for difficult words and industry-specific terms is crucial.
Can the MOS measurement process be fully automated?
Currently, no automated method can completely replace human subjective perception for evaluating voice naturalness (MOS). However, deep learning models can be trained to predict MOS based on acoustic features, though their accuracy is generally lower than human evaluation.
How does TTS WER differ from STT WER?
STT WER measures the accuracy of a system converting speech to text. TTS WER (in the testing context) measures the pronunciation accuracy of the speech synthesis system by comparing the reference STT's transcription against the original text. The goal of TTS WER is to ensure the TTS pronounces words correctly so that an STT can recognize them accurately.
Does AIVISION provide tools for measuring MOS and WER?
AIVISION focuses on providing high-quality TTS and STT models. Building specific testing workflows is typically done by the client's engineering team using the APIs and technical documentation we provide. You can find more technical insights in our [Blog](/en/blog).
Try AIVISION's Vietnamese speech AI
Speech-to-Text, Text-to-Speech and LLM, free.
Start free → Contact
By
Dr. Giang Vo — CEO, AIVISION
Dr. Giang Vo is the CEO of AIVISION, leading the development of s2speech — speech recognition, AI voices and a large language model for Vietnamese — and working with companies at home and abroad on their deployments.