Building a Full-Stack Voice AI Web App: Integrating STT, TTS, and LLM in 100 Lines of Code
Learn to build a full-stack voice AI web app by integrating STT, TTS, and LLM APIs. A practical guide with Python and JavaScript code examples.
In this article
- Why Use Speech AI APIs Instead of Training Your Own Models?
- Overview Architecture for a Voice Web App
- Technical API Guide: Backend (Python)
- 1. Setting Up API Configuration
- 2. Handling Speech-to-Text (STT)
- 3. Handling the LLM (aivision-L1.0)
- Technical API Guide: Frontend (JavaScript)
- 1. Recording and Sending Data (JS)
- 2. Playing Audio Response (JS)
- Optimizing User Experience
- Comparison of Integration Methods
- Tips for Developers
- Conclusion
Transforming a static web application into a natural, interactive virtual assistant is often a significant challenge for development teams. Instead of spending months training complex speech recognition models or worrying about high latency, you can integrate voice AI into your product in a matter of hours. This article provides a practical technical API guide to help you build a voice web app using Python voice assistant code and JavaScript in approximately 100 lines.
Why Use Speech AI APIs Instead of Training Your Own Models?
In a fast-paced development environment, training your own speech-to-text (STT) and text-to-speech (TTS) models is costly in terms of both time and infrastructure. AIVISION provides standard REST and WebSocket APIs, allowing you to connect directly to your application.
Key benefits of using APIs from AIVISION:
- Optimized for Vietnamese: The models are trained on a 690,517-hour Vietnamese speech corpus, handling both read speech and natural conversation in real-world environments effectively.
- Multilingual and Code-switching Support: The system smoothly handles conversations that mix Vietnamese and English, a challenge for many general-purpose solutions.
- Low Latency: Supports streaming via WebSocket for both STT and TTS, ensuring a "real-time" experience for users.
Overview Architecture for a Voice Web App
To create a complete voice assistant, the data flow passes through three main stages:
- Input (STT): Capture audio from the microphone, send it to the server, and receive text.
- Processing (LLM): Send the text to the language model to generate a response.
- Output (TTS): Send the response to AIVISION TTS and receive audio to play through the speaker.
Technical API Guide: Backend (Python)
We will use Python with the websockets and requests libraries to handle server-side logic. This is the core of the Python voice assistant code.
1. Setting Up API Configuration
First, create an account on Start free to obtain your API Key.
import os
import json
import asyncio
import websockets
import requests
AIVISION_API_KEY = os.getenv("AIVISION_API_KEY")
STT_WS_URL = "wss://api.s2speech.com/v1/stt"
TTS_WS_URL = "wss://api.s2speech.com/v1/tts"
LLM_API_URL = "https://api.s2speech.com/v1/chat/completions"
2. Handling Speech-to-Text (STT)
We use WebSocket to stream audio from the client to the server.
async def process_stt(ws):
"""Process audio stream from client, return text."""
audio_buffer = b""
while True:
message = await ws.recv()
if message == "stop":
break
# Assume message is bytes of audio chunk
audio_buffer += message
# In a real scenario, you would parse the response
# from the AIVISION WebSocket stream here.
# This is a simplified representation.
return "This is the recognized speech content."
3. Handling the LLM (aivision-L1.0)
Use the OpenAI-compatible API to generate intelligent responses.
def process_llm(user_input):
"""Send text to LLM and receive response."""
headers = {
"Authorization": f"Bearer {AIVISION_API_KEY}",
"Content-Type": "application/json"
}
payload = {
"model": "aivision-L1.0",
"messages": [
{"role": "system", "content": "You are a friendly virtual assistant."},
{"role": "user", "content": user_input}
]
}
response = requests.post(LLM_API_URL, headers=headers, json=payload)
return response.json()['choices'][0]['message']['content']
Technical API Guide: Frontend (JavaScript)
The client-side is responsible for capturing and playing audio. We will use the Web Audio API and WebSocket.
1. Recording and Sending Data (JS)
let audioContext;
let mediaStream;
let processor;
let ws;
function startRecording() {
navigator.mediaDevices.getUserMedia({ audio: true })
.then(stream => {
mediaStream = stream;
audioContext = new AudioContext();
const source = audioContext.createMediaStreamSource(mediaStream);
processor = audioContext.createScriptProcessor(4096, 1, 1);
// Open WebSocket connection to your Backend Server
ws = new WebSocket("ws://localhost:8000/stt");
processor.onaudioprocess = (e) => {
const inputData = e.inputBuffer.getChannelData(0);
// Send raw audio data (Float32Array) to server
if (ws.readyState === WebSocket.OPEN) {
ws.send(new Blob([inputData.buffer], {type: 'audio/f32'}));
}
};
source.connect(processor);
});
}
function stopRecording() {
if (ws) {
ws.send("stop");
ws.close();
}
if (processor) processor.disconnect();
if (mediaStream) mediaStream.getTracks().forEach(track => track.stop());
}
2. Playing Audio Response (JS)
function playAudioResponse(base64Audio) {
// Convert base64 to Blob and play
const byteCharacters = atob(base64Audio);
const byteNumbers = new Array(byteCharacters.length);
for (let i = 0; i < byteCharacters.length; i++) {
byteNumbers[i] = byteCharacters.charCodeAt(i);
}
const byteArray = new Uint8Array(byteNumbers);
const blob = new Blob([byteArray], { type: 'audio/mpeg' });
const audio = new Audio(URL.createObjectURL(blob));
audio.play();
}
Optimizing User Experience
When building a voice web app, latency is a critical factor. Here are some technical tips:
- Use Streaming for TTS: Instead of waiting for the entire audio file to be generated before playing it, use the streaming mode of AIVISION TTS. This allows users to hear the response immediately, creating a natural feel.
- Handle Code-switching: If your application serves users who mix Vietnamese and English (e.g., IT engineers, medical staff), ensure your STT API configuration supports this feature. AIVISION handles English technical terms within Vietnamese sentences effectively.
- Appropriate Audio Formats: For call center or telephony applications, AIVISION TTS supports 8 kHz G.711 (μ-law / A-law) formats, which reduces data size and increases response speed.
Comparison of Integration Methods
| Criteria | Self-trained Model | Using AIVISION API |
|---|---|---|
| Deployment Time | Weeks/Months | Hours |
| GPU Infrastructure Cost | High | Optimized based on usage |
| Vietnamese Accuracy | Depends on data | Pre-optimized (Avg WER 10.53%) |
| Code-switching Support | Difficult to configure | Supported out-of-the-box |
| Maintenance | Complex | Simple |
Note: Accuracy figures are based on AIVISION internal tests on data not used for training.
Tips for Developers
- Start with REST, then Upgrade to WebSocket: If your application does not require absolute real-time performance (e.g., voice notes), use the REST API for STT to simplify code. For conversational assistants, WebSocket is mandatory.
- Manage Audio State: In JavaScript, ensure you cancel old requests when the user starts a new sentence to avoid audio overlap.
- Check Connections: Always have a fallback mechanism if the WebSocket connection drops (e.g., switch to text input).
Conclusion
Integrating voice AI into web applications is no longer a difficult problem when you have specialized API platforms. With the technical API guide above, you have the foundation to create a smooth-running Python voice assistant and JavaScript frontend. AIVISION provides the full toolset from STT, TTS to LLM, allowing you to focus on product experience rather than core technology.
Start today by creating a free account to experience these APIs. You can sign up at Start free or contact the technical team directly for integration support. Explore more in-depth articles at the Blog and don't hesitate to Contact us if you have technical questions.
Frequently asked questions
Do I need a credit card to try AIVISION's APIs?
No. AIVISION provides a completely free trial for every account, with no requirement for a credit card or prepayment.
Does AIVISION support Vietnamese mixed with English?
Yes. AIVISION's Speech-to-Text model is designed to handle conversations that mix Vietnamese and English (code-switching) well, which is particularly useful for technical and medical industries.
Can I use AIVISION's APIs for mobile applications?
Absolutely. AIVISION's REST and WebSocket APIs can be called from any platform, including Android, iOS, and Web.
What is the latency of a voice assistant system using AIVISION APIs?
Latency depends on network speed and the length of the utterance. Thanks to streaming support for both STT and TTS, AIVISION minimizes wait times, providing a near-real-time interaction experience.
How can I improve recognition accuracy in noisy environments?
You should use noise filters on the client side (before sending audio to the server) and ensure a high-quality microphone. AIVISION also optimizes its models for real-world audio conditions, but signal preprocessing always helps improve results.
Try AIVISION's Vietnamese speech AI
Speech-to-Text, Text-to-Speech and LLM, free.
Start free → Contact