Measuring Real-World WER for Vietnamese STT: How to Test Accuracy Before Buying an API

When deploying speech AI solutions, measuring WER (Word Error Rate) is not just a technical metric; it is a critical indicator that determines user experience. Many businesses rely on generic advertised figures from providers, only to find their systems underperforming in real-world environments. This guide walks you through a standard process for checking STT accuracy, giving you a scientific basis to evaluate Vietnamese STT APIs before making a purchase or integration decision.

Why WER is the Gold Standard in Speech Recognition?

WER (Word Error Rate) is the ratio of word errors, calculated based on the number of Deletions, Insertions, and Substitutions relative to the total number of words in the reference transcript. The basic formula is:

$$WER = \frac{S + D + I}{N} \times 100%$$

Where N is the total number of words in the reference. However, an average WER often masks significant differences across contexts. A system might achieve a low WER on read speech but see it spike when encountering noise, dialects, or specialized terminology. Therefore, measuring WER on your own data is a mandatory step.

The Difference Between Standard and Real-World Data

Many users make mistakes when comparing WER across platforms using the same public benchmark. In reality, Vietnamese has high phonetic variation, and code-switching (mixing English) is very common in business environments.

• Read Speech: Usually clear pronunciation with no background noise, typical of news or exams. WER here is typically low.
• Free Conversation: Speech with emotions, pauses, and repetitions. WER can increase significantly.
• Specialized Data: Contains medical, financial, or technical terms. If the model isn't trained on this data, word errors will spike.

When evaluating Vietnamese STT APIs, you must clearly define your primary use case to select the appropriate test dataset.

4-Step Process for Professional STT Accuracy Testing

To get reliable STT accuracy check results, you must follow a standardized process. Don't just run a few sample sentences; build your own benchmark.

Step 1: Prepare Reference Data (Ground Truth)

This is the most critical step. You need to collect audio files that represent the actual speech of your target users.

• Length: Minimum 100 to 500 utterances to ensure statistical significance.
• Diversity: Include male and female voices, different age groups, and regional accents (North, South, Central) if your product targets the national market.
• Transcription: Record the script or content to create the most accurate reference transcript possible. This reference must be human-verified.

Step 2: Call the API and Collect Results

Use the provider's API to convert audio files into text. Pay attention to technical parameters:

• Audio Format: Ensure files are in well-supported formats (usually WAV 16kHz or MP3).
• Processing Mode: Choose streaming (real-time) or batch (file) depending on your application.
• Latency: Record response times if real-time experience is a priority.

Step 3: Calculate WER Manually or with Tools

Once you have the machine-generated text (hypothesis) and the reference text, you need to calculate WER. You can use open-source libraries or a simple Python script to compare strings.

Example case:

• Reference: "I want to book a flight to Hanoi"
• Hypothesis: "I want to book a flight to Hanoi" -> WER = 0%
• Hypothesis: "I want to book a flight to Hai Phong" -> 1 substitution. If total 7 words, WER = 1/7 = 14.28%.

When measuring WER for speech to text, break down results by data groups (e.g., short sentences, long sentences, sentences with English) to clearly see the system's strengths and weaknesses.

Step 4: Analyze Errors and Make Decisions

Don't just look at the total number; analyze specific errors:

• Spelling Errors: Due to similar pronunciations (e.g., "anh" vs "an").
• Terminology Errors: Due to the model not recognizing specialized terms.
• Segmentation Errors: Due to missing punctuation or incorrect sentence breaks.

A simple comparison table helps visualize this:

Data Group	Sentences	Avg WER	Comment
Read Speech	100	Low	Suitable for news, audiobooks
Free Conversation	100	Medium	Needs optimization for chatbots
Medical Domain	50	High	Requires specialized training data

Criteria for Evaluating Vietnamese STT APIs Beyond WER

Although WER is the primary metric, when evaluating Vietnamese STT APIs, you need to consider other factors to ensure stability and ease of integration.

Code-Switching Capability (Vietnamese - English)

In Vietnamese business environments, mixing English is very common (e.g., "Send email to the team", "Check project status"). A good STT API must accurately recognize English words within Vietnamese sentences without fragmentation or spelling errors. Test this data group separately when checking STT accuracy.

Response Speed and Stability

• Latency: For real-time applications like translation or voice assistants, low latency is a top priority.
• Uptime: The system must ensure continuous operation without interruptions during peak hours.

Technical Support and Documentation

A good API comes with comprehensive documentation, clear code examples, and a responsive support team. When issues arise, you need the provider to respond promptly to avoid affecting project timelines.

Real-World Example: Applying the Testing Process with AIVISION

AIVISION is a developer of Speech AI solutions focused on Vietnamese, with experience deploying for hundreds of enterprises domestically and internationally. To illustrate the WER measurement process, let's look at how AIVISION approaches quality evaluation.

According to AIVISION's internal measurements on data not used for training, the system achieves an average WER of 10.53% across four Vietnamese test sets. Specifically:

• FLEURS-vi (read speech): 3.60%
• VIVOS (many speakers): 4.71%
• ViMedCSS (medical, with English): 14.10%
• Real Business Meetings: 19.71%

These figures show a clear difference between standard read speech and free speech in work environments. Notably, on the VLSP test set (1,000 utterances, human references), AIVISION achieves a WER of 8.3% and CER of 5.1%. These results are generated from a model trained on a 690,517-hour Vietnamese speech corpus, utilizing a cluster of 24 NVIDIA H200 GPUs.

When you want to evaluate AIVISION's Vietnamese STT API, you can run your own dataset through the API to compare with these figures. This helps you understand the suitability of the technology for your specific business context.

Tips for Choosing an STT Provider

1. Request a Demo on Your Data: Don't just trust generic demos. Send a few real audio files and request a test run.
2. Check Additional Features: Consider features like word timestamps and speaker diarization if needed.
3. Evaluate Hidden Costs: Beyond API fees, consider integration, maintenance, and scaling costs.

AIVISION provides Speech-to-Text services with code-switching handling, word timestamps, and support for multiple languages for translation apps. The system is designed to be optimized for Vietnamese, helping minimize errors in complex contexts. You can explore more in-depth articles on the Blog to stay updated on speech AI knowledge.

Conclusion

Measuring WER for speech to text is not a formal technical step but the foundation for building high-quality speech AI products. By preparing good reference data, calculating WER carefully, and analyzing errors in detail, you will have a solid basis to check STT accuracy and evaluate Vietnamese STT APIs objectively.

Start with small datasets, focusing on your business's real-world context. Don't hesitate to experiment and compare different solutions. If you are looking for a Speech-to-Text solution optimized for Vietnamese, try AIVISION's tools. We are committed to delivering high accuracy and a smooth integration experience for developers.

To get started, you can Start free with AIVISION's services or Contact the technical team for detailed advice on integrating the API into your existing system.

Frequently asked questions

What is WER and why is it important when choosing an STT API?

WER (Word Error Rate) is the word error rate in speech recognition. It is important because it indicates the system's accuracy compared to the reference text. A lower WER means fewer errors, which improves user experience and reduces post-processing costs.

How can I measure WER for speech to text myself?

You need to collect audio files and create a ground truth transcript. Then, call the STT API to get the machine-generated text. Use open-source tools or scripts to compare the two text strings and calculate the error rate (substitutions, insertions, deletions).

What is an acceptable average WER for Vietnamese?

There is no fixed number, but in business environments, a WER under 10-15% is often considered good for automated applications. For read speech, WER can be lower than 5%. However, you should compare with your own real-world data for an accurate assessment.

Does AIVISION support accuracy testing on a company's private data?

Yes. AIVISION encourages customers to test the API on their real-world data to ensure suitability. You can use free trial packages or contact the technical team for support in running benchmarks on your specific data.

How does code-switching (mixing English) affect WER?

Code-switching often increases WER if the model is not well-trained on mixed data. A good STT system like AIVISION handles English words within Vietnamese sentences naturally, helping keep WER stable even in professional conversations.

#WER speech to text#Vietnamese STT accuracy#test STT API#Word Error Rate#speech recognition benchmark#AIVISION#real-world WER#STT integration