Automating Voice AI Quality Checks: A Guide to MOS and WER
Learn how to measure TTS quality using MOS and WER. This guide helps businesses quantify voice AI performance for better user experiences.
In this article
- Why Businesses Need to Measure TTS Quality
- Standardized Process for Measuring MOS in Vietnamese
- Step 1: Prepare the Evaluation Dataset
- Step 2: Collect Listener Feedback
- How to Calculate TTS WER Accurately
- The WER Formula
- Notes on Applying WER to Vietnamese
- Comparing MOS and WER: When to Use Which?
- Practical Advice for Technical Teams
- Conclusion
In the era of AI, natural-sounding speech is no longer a binary feature; it is a measure of excellence. Many businesses deploying voice AI quality checks struggle to translate subjective listening impressions into objective metrics. How can you determine if a Text-to-Speech (TTS) voice is truly smooth and accurate? The answer lies in two core technical indicators: Mean Opinion Score (MOS) and Word Error Rate (WER). This guide provides a standardized process for measuring TTS quality, helping you evaluate speech model performance scientifically and practically.
Why Businesses Need to Measure TTS Quality
Developers often rely on personal impressions from listening to a few sample sentences to judge voice quality. However, this approach lacks representativeness and can lead to significant biases when scaled. To ensure a consistent user experience, you need a quantitative measurement framework.
MOS (Mean Opinion Score) evaluates the naturalness and intelligibility of speech. It is a subjective metric that reflects how "human" the AI voice sounds. Conversely, WER (Word Error Rate) measures the accuracy of the spoken content. If the TTS mispronounces words, omits them, or adds extra syllables, the WER increases. Combining both metrics provides a comprehensive view of voice quality.
Standardized Process for Measuring MOS in Vietnamese
MOS is the gold standard for speech naturalness. However, executing MOS testing for Vietnamese requires careful preparation of data and evaluators.
Step 1: Prepare the Evaluation Dataset
You need a set of short sentences that are diverse in context and difficulty. Avoid monotonous inputs. The dataset should include:
- Everyday conversational sentences.
- Technical or financial texts containing many terms.
- Sentences involving Vietnamese-English code-switching.
Step 2: Collect Listener Feedback
MOS requires at least five independent evaluators. Each listener rates every sentence from 1 to 5, where:
- 5: Perfect, indistinguishable from a human.
- 4: Natural, with minor errors that do not affect listening.
- 3: Acceptable, with clear errors but understandable content.
- 2: Poor, with many errors, difficult to hear.
- 1: Unintelligible.
The average of these scores yields the mean MOS. A high-quality TTS voice typically achieves an MOS above 4.0.
How to Calculate TTS WER Accurately
If MOS answers "does it sound good?", WER answers "is it correct?". To calculate TTS WER, you must compare the synthesized audio against the original text.
The WER Formula
$$WER = \frac{S + D + I}{N}$$
Where:
- $S$: Number of Substitutions (wrong words)
- $D$: Number of Deletions (missing words)
- $I$: Number of Insertions (extra words)
- $N$: Total number of words in the original text
For example, if the original text has 10 words, and the TTS substitutes 1, deletes 1, and inserts 1, the WER is $(1+1+1)/10 = 30%$.
Notes on Applying WER to Vietnamese
Vietnamese has specific characteristics regarding word segmentation, particularly with compound words. When measuring TTS quality, ensure that the word segmentation process is consistent between the source text and the speech recognition output (ASR) if you use ASR for cross-checking. However, a more precise method is comparing the TTS engine's input text directly with its phoneme mapping output, or using high-quality ASR to transcribe the audio and compare it against the ground truth.
Comparing MOS and WER: When to Use Which?
| Criteria | MOS (Mean Opinion Score) | WER (Word Error Rate) |
|---|---|---|
| Nature | Subjective (Human evaluation) | Objective (Automated calculation) |
| Measures | Naturalness, tone, prosody | Content accuracy, pronunciation errors |
| Cost | High (Requires many evaluators) | Low (Automated) |
| Time | Slow (Requires listening time) | Fast (Automated scripts) |
| Application | Voice selection, UX evaluation | Terminology checks, content reliability |
Combining both provides the most complete picture. A TTS voice may have a very low WER (correct words) but a low MOS (robotic, hard to listen to), or vice versa.
Practical Advice for Technical Teams
When implementing voice AI quality checks in your enterprise, consider the following factors:
- Diversify Contexts: Do not test only on greetings. Test on realistic scenarios such as banking notifications, medical instructions, or customer support interactions.
- Check Code-Switching: Vietnamese frequently intersperses English words. Ensure your TTS handles English terms within Vietnamese sentences smoothly. This is a common weakness in legacy systems.
- Testing Frequency: Run automated test suites (focusing on WER) after every model update or configuration change. MOS should be conducted quarterly or upon major version releases.
At AIVISION, we understand these challenges. With experience deploying speech AI for hundreds of enterprises in Vietnam and abroad, we provide the aiv-tts-S.1.0 solution, which handles Vietnamese naturally, supports streaming, and offers standard telephony formats (8 kHz G.711). Additionally, AIVISION's voice cloning technology allows you to create voices from 20 seconds to 2 minutes of recorded audio, ensuring consistency and professionalism for your brand.
Conclusion
Measuring TTS quality is not just a technical step but a strategic move to enhance user experience. By applying MOS and WER processes systematically, your business can optimize AI voice performance, reduce errors, and build customer trust.
Do not let voice quality be a bottleneck in your digital transformation journey. Start by evaluating accurately and choosing the right solution.
Contact our technical team for detailed consultation on voice quality check processes. Or you can Start free with AIVISION's speech AI services today to experience the difference. Explore more in-depth knowledge at the Blog of s2speech.com.
Frequently asked questions
What is MOS in TTS evaluation?
MOS (Mean Opinion Score) is an average score from 1 to 5 assigned by human listeners to rate the naturalness of AI speech. Higher scores indicate speech that sounds more like a human.
How is TTS WER calculated?
WER (Word Error Rate) is the ratio of word errors to the total number of words in the reference text. It includes substitutions, deletions, and insertions. A lower WER indicates higher TTS accuracy.
Do I need both MOS and WER?
Yes. MOS evaluates naturalness (subjective), while WER evaluates content accuracy (objective). Combining both provides a comprehensive view of AI voice quality.
How often should I check TTS quality?
You should run automated WER checks after every model update. MOS should be conducted quarterly or before major feature launches to ensure a stable user experience.
Does AIVISION support Vietnamese voice quality checks?
Yes. AIVISION provides Vietnamese TTS solutions with code-switching capabilities and voice cloning, helping businesses easily evaluate and optimize voice quality according to MOS and WER standards.
Try AIVISION's Vietnamese speech AI
Speech-to-Text, Text-to-Speech and LLM, free.
Start free → Contact