Integrating STT, LLM, and TTS APIs into React Web Apps: A Technical Walkthrough

Building web applications that interact naturally via voice is becoming a standard expectation in modern software development. To achieve this, Integrating STT, LLM, and TTS APIs into React Web Apps is a critical step. This article provides a detailed technical guide to help you connect Speech-to-Text (STT), Large Language Models (LLM), and Text-to-Speech (TTS) into your React application smoothly, with a focus on performance optimization and a seamless user experience.

Overview of Voice System Architecture in React

Before diving into the code, it is essential to understand the data flow. A complete voice application typically consists of three main components:

1. Speech-to-Text (STT): Converts audio from the microphone into text.
2. Large Language Model (LLM): Processes context, performs reasoning, and generates responses.
3. Text-to-Speech (TTS): Converts the response text back into audio.

In a React environment, the biggest challenge is managing state and handling streaming data. Instead of waiting for the entire response, you need to display results in real-time to reduce perceived latency.

Step 1: Handling Speech-to-Text with WebSocket

For Speech to Text React implementations, using WebSocket is mandatory to achieve the lowest possible latency. Traditional REST APIs are generally suitable only for pre-recorded audio files and do not meet real-time requirements.

AIVISION provides an STT API that supports streaming over WebSocket, specifically optimized for Vietnamese and code-switching between Vietnamese and English. Below is a basic example of setting up the connection:

const sttSocket = new WebSocket('wss://api.s2speech.com/stt/stream');

sttSocket.onopen = () => {
  console.log('STT Connection established');
};

sttSocket.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (data.type === 'partial') {
  setPartialText(data.text); // Update temporary text
  } else if (data.type === 'final') {
  setFinalText(data.text); // Confirm final text
  handleLLMRequest(data.text);
  }
};

// Send audio chunks (ArrayBuffer) from MediaRecorder
function sendAudioChunk(chunk) {
  if (sttSocket.readyState === WebSocket.OPEN) {
  sttSocket.send(chunk);
  }
}

Technical Tip: Break audio into small chunks (e.g., every 100ms) and send them continuously. This provides the STT engine with sufficient context to accurately recognize specialized terms or proper nouns.

Step 2: Integrating LLM API with Streaming

Once you receive text from the STT, you need to call the LLM API to generate a response.

Using streaming for the LLM is essential so users see a response immediately, rather than waiting for several seconds.

async function handleLLMRequest(userText) {
  setLoader(true);
  setLLMResponse('');
 
  const response = await fetch('https://api.s2speech.com/v1/chat/completions', {
  method: 'POST',
  headers: {
  'Content-Type': 'application/json',
  'Authorization': `Bearer ${API_KEY}`
  },
  body: JSON.stringify({
  model: 'aivision-L1.0',
  messages: [
  { role: 'system', content: 'You are a friendly voice assistant.' },
  { role: 'user', content: userText }
  ],
  stream: true
  })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
  const { done, value } = await reader.read();
  if (done) break;
 
  const chunk = decoder.decode(value);
  // Process SSE format: data: {...}
  const lines = chunk.split('\n');
  lines.forEach(line => {
  if (line.startsWith('data: ')) {
  const data = JSON.parse(line.slice(6));
  if (data.choices[0].delta.content) {
  setLLMResponse(prev => prev + data.choices[0].delta.content);
  }
  }
  });
  }
  setLoader(false);
}

Step 3: TTS in React and Optimizing Audio Experience

Finally, we convert the text from the LLM into speech. TTS in React requires careful handling to avoid audio overlap or high latency.

async function playTTS(text) {
  // Simulate fetching TTS stream
  const response = await fetch('https://api.s2speech.com/tts/stream', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${API_KEY}` },
  body: JSON.stringify({ text: text, voice: 'vi-female-1' })
  });
 
  const audioContext = new AudioContext();
  // Process audio stream and play via AudioContext
  // Note: You must close the old AudioContext before creating a new one to avoid memory leaks
}

Performance Optimization and State Management

When combining all three components, you need efficient state management. Here is a comparison of the processing methods:

Component	Protocol	Advantages	Technical Notes
STT	WebSocket	Real-time, low latency	Requires reconnection logic if network drops
LLM	HTTP (SSE)	Streaming text, easy to debug	Must parse SSE format correctly
TTS	HTTP/WS	Streaming audio	Must manage AudioContext to prevent errors

Practical Tips:

• Debounce for STT: If the user pauses speaking, wait 500ms before sending the "final" command to avoid cutting off sentences.
• AbortController: Always use AbortController when calling LLM or TTS APIs so users can cancel requests if they change their mind.
• Network Error Handling: WebSockets can disconnect. Write logic to automatically reconnect using a backoff strategy.

Conclusion

Integrating STT, LLM, and TTS APIs into React Web Apps is not difficult if you understand streaming architecture and state management. By leveraging modern APIs that support WebSocket and OpenAI-compatible endpoints, you can build voice applications that are smooth, accurate, and natural.

AIVISION has supported hundreds of enterprises in deploying Vietnamese speech solutions with high accuracy and flexible code-switching capabilities. If you are ready to start, Start free with our API services or explore more in-depth articles on our Blog.

Frequently asked questions

Why should I use WebSocket for Speech-to-Text in React?

WebSocket allows for continuous two-way data transmission, which minimizes latency for real-time communication. HTTP REST is only suitable when processing complete, pre-recorded audio files.

How do I handle English words within Vietnamese sentences when using STT?

Use STT APIs that support code-switching. AIVISION's STT model is trained to accurately handle English words interspersed within Vietnamese sentences, ensuring high recognition accuracy.

Do I need to install special libraries to integrate TTS into React?

You can use the native fetch API or axios to call the TTS API. On the browser side, use the Web Audio API or HTML5 Audio to play the audio data received from the server.

How can I reduce latency when displaying LLM responses?

Use streaming mode (SSE) when calling the LLM API. This allows you to display words or sentences as they are received, rather than waiting for the entire response to complete.

Does AIVISION support languages other than Vietnamese?

Yes, in addition to Vietnamese, AIVISION supports 24 other languages, including English, Spanish, Thai, Chinese, Japanese, and Korean, serving multilingual translation applications.

#React voice AI#STT API#LLM integration#TTS API#WebSocket streaming#AIVISION#web speech recognition#real-time AI