Testing Vietnamese Voicebots: WER, Latency, MOS Guide

Deploying an AI voicebot for the Vietnamese market requires more than just API integration. It demands a rigorous testing protocol to ensure a smooth and accurate user experience. Many businesses overlook formal speech AI evaluation, leading to bots that misinterpret context or respond too slowly, ultimately frustrating customers. This guide provides the specific tools and methodology you need to test your voicebot, focusing on three core metrics: Word Error Rate (WER), Latency, and Mean Opinion Score (MOS).

Why Professional Voicebot Performance Measurement Matters

In a business environment, a bot’s incorrect answer can lead to order processing errors or inaccurate legal information. For the Vietnamese language, the challenge lies in its complex tonal system and frequent code-switching between Vietnamese and English, particularly in healthcare, technology, and finance. Therefore, measuring WER (Word Error Rate) is not just a technical figure; it is a direct measure of brand trustworthiness.

Beyond accuracy, response speed dictates user sentiment. If wait times exceed acceptable thresholds, callers feel they are interacting with a rigid machine. Finally, the quality of the synthesized voice (TTS) must be natural enough to avoid discomfort. Below is a detailed guide for each metric.

1. Measuring Word Error Rate (WER): The Accuracy Benchmark

WER represents the percentage of incorrect words relative to the total words in a reference text. The basic formula is: $WER = (S + D + I) / N$, where S is substitutions, D is deletions, I is insertions, and N is the total words in the ground truth text.

Implementation Method

To evaluate speech AI objectively, you need a test dataset with reference transcripts.

1. Prepare Data: Collect 100–500 representative dialogue sentences that reflect your business scenarios. For example, if you are building a voicebot for a hospital, include specialized medical terminology.
2. Run Speech-to-Text (STT): Use AIVISION’s STT API or your existing system to convert speech to text.
3. Compare and Calculate: Use libraries like jiwer (Python) to compare the output text against the reference.

Vietnamese-Specific Considerations

When testing Vietnamese voicebots, pay close attention to errors caused by tones and punctuation. A minor tone error can completely change the meaning of a sentence (e.g., "cần" vs. "cần"). AIVISION, with experience deploying speech AI for hundreds of enterprises in Vietnam and abroad, emphasizes the use of diverse datasets. Specifically, AIVISION’s models are trained on a large-scale Vietnamese speech corpus, enabling them to handle conversations with interspersed English terms effectively—a common issue in modern business environments.

2. Measuring Latency: The Key to User Experience

Voicebot latency is calculated from the moment the user stops speaking (End-of-Speech - EOS) to when the bot begins playing the audio response. High latency makes conversations feel awkward and disjointed.

Components of Latency

To optimize performance, you must break down latency into its individual parts:

• STT Latency: The time taken to process speech into text.
• Logic/NLU Latency: The time the system takes to understand intent and retrieve an answer.
• TTS Latency: The time required to synthesize text into speech.

Technical Recommendations

• Use Streaming: Instead of waiting for a full sentence to be processed, use streaming APIs (WebSocket) for both STT and TTS. AIVISION provides real-time streaming over WebSocket, significantly reducing wait times.
• Continuous Monitoring: Log the timestamp of each event (start of speech, end of speech, text received, audio playback start) to calculate averages and extreme values (P95, P99).
• Target: A responsive voicebot should have an overall latency under 1–2 seconds to create a sense of natural conversation.

3. Measuring MOS (Mean Opinion Score): Assessing Voice Quality

MOS is a scale from 1 to 5 used to evaluate the naturalness and intelligibility of synthesized speech (TTS). While WER measures the accuracy of the "machine ear," MOS measures the experience of the "human ear."

Automating MOS Evaluation

Hiring humans for manual evaluation is costly and slow. Today, dedicated AI models can predict MOS automatically.

1. Select an Evaluation Model: Use machine learning tools designed to predict MOS.
2. Create an Audio Dataset: Run TTS on sample text segments.
3. Run the Prediction Model: Input the audio files into the evaluation model to obtain scores.

Evaluation Criteria

• Naturalness: Does the voice sound like a real human?
• Clarity: Is it easy to hear in noisy environments?
• Contextual Fit: Does the voice suit the brand (e.g., a warm, deep voice for banking, a youthful voice for fashion)?

AIVISION develops the Text-to-Speech model (aiv-tts-S.1.0), which generates natural Vietnamese and English voices. It supports telephony formats (8 kHz G.711) for call centers, ensuring audio quality is optimized even when transmitted over limited-bandwidth communication channels.

Summary of Key Metrics to Check

Metric	Tool/Method	Recommended Target	Business Impact
WER (Word Error Rate)	jiwer library, reference dataset	Under 10–15% (industry-dependent)	Ensures the bot understands customer intent
Latency	Timestamp logging, Streaming APIs	Under 2 seconds	Retains customers, avoids perceived waiting
MOS (Mean Opinion Score)	Automated MOS prediction models	Above 4.0/5.0	Creates a professional, friendly impression

Practical Testing Strategy

Do not limit testing to the lab. Apply "A/B Testing" with small groups of real users.

• Error Testing: Intentionally input sentences with easily confused phonemes or slang to see how the system handles them.
• Multi-Channel Testing: Test on mobile phones, computers, and various IoT devices to ensure stable input audio quality.
• Error Analysis: Whenever WER spikes, examine the specific incorrect words to update your dictionary or fine-tune the model.

AIVISION, with a technology platform built with a focus on Vietnamese, provides easily integrable REST and WebSocket APIs. The system handles code-switching (interspersed English) well, a critical factor for evaluating speech AI in an internationalized business context.

Conclusion

Testing voicebots is not a one-time step but a continuous process. By closely monitoring WER, latency, and MOS, you can detect technical issues early and enhance the user experience. An accurate and fast speech system becomes a significant competitive advantage, helping to automate processes and reduce the workload on your staff.

To start building and testing your own Vietnamese AI voicebot, explore AIVISION’s tools and APIs. We provide a stable development environment and models optimized specifically for the Vietnamese language, helping you deploy and measure performance accurately.

Contact our technical team for detailed advice on the testing process suited to your industry, or Start free today to experience AIVISION’s speech recognition and synthesis quality. You can also read other in-depth articles on our Blog.

Frequently asked questions

What is an acceptable WER for a Vietnamese voicebot?

It depends on the industry, but a WER under 15% is generally considered good for enterprise applications. In Vietnamese environments with many technical terms or interspersed English, WER may fluctuate slightly higher, but it still requires continuous optimization.

How can I measure voicebot latency accurately?

You need to log timestamps for key events: when the user stops speaking, when text is received from STT, when logic processing is complete, and when TTS audio playback starts. The difference between the end of speech and the start of audio playback is the overall latency.

Do I need to hire people to evaluate MOS?

Initially, you should have a small group of real users for subjective evaluation. However, for frequent and automated testing, you should use AI models that predict MOS. Combining both approaches yields the most reliable results.

Does AIVISION support English within Vietnamese sentences?

Yes, AIVISION effectively supports code-switching (interspersed English). The models are trained to recognize and pronounce English words naturally within Vietnamese sentences, making the voicebot suitable for multilingual work environments.

Can I start testing my voicebot on a small scale?

Absolutely. You can start with a small test dataset (e.g., 50–100 sentences) to assess initial performance, then scale up once the system is stable. AIVISION provides easily integrable APIs so you can begin experimenting immediately.

#Vietnamese voicebot testing#WER measurement#speech latency#MOS score#AIVISION#speech-to-text#text-to-speech#voice AI