While OpenAI delays the release of audio modality for ChatGPT, we decided to share our experience of creating our own application for voice interaction with LLM and integrating it into an interactive booth.
Idea and concept
At the end of February, Bali hosted the Lampu festival, organized along the lines of Burning Man. Participants created their own art objects and installations. My friends from Camp 19:19 and I decided to make the AI Confession Room — a booth where anyone could talk to artificial intelligence.
The scenario was as follows: the user entered the booth, and the system started a new session. The user asked a question, and the AI listened and responded. After the user left, the dialogue was automatically deleted to maintain privacy.
Demo version
To test the idea, I made a simple implementation in Python in one evening:
- Listen to the microphone.
- Recognize the user’s speech using an STT model.
- Generate a response using LLM.
- Voice the response using a TTS model.
- Play the result to the user.
We used OpenAI cloud models — Whisper for speech recognition, GPT-4 for text generation, and TTS for voice synthesis. The speech_recognition library allowed us to assemble a prototype in just a few dozen lines of code.
Main problems
After the first tests, it became clear what needed to be worked on:
- Response delay: 7–8 seconds or longer. Processing speed needed to be optimized.
- Ambient noise: it was difficult to determine the beginning and end of the user’s speech.
Simulation of live communication: in order for the user to be able to interrupt the AI, the microphone had to be constantly on, but then it was necessary to separate the user’s voice from background noise and the AI’s voice.
Feedback: the delay created the impression that the system was freezing.
Each of these problems could be solved either by technical means or circumvented from a UX perspective.
UX booth
We thought about how the user would interact with the booth:
- Identifying a new user: the distance sensor behind the user proved to be more reliable than the weight sensor or camera.
- Microphone control: instead of automatic endpointing, we added a large red button so that the user could start and end the recording themselves.
- Feedback: the screen displayed the current status — listening to the microphone, processing the question, or responding.
Architecture
Arduino (ESP32): tracks the distance sensor and button, transmits data to the backend via HTTP API.
Backend: controls the microphone, interacts with STT, LLM, and TTS models, generates and voices responses.
Web interface: displays the current system status to the user.
Hardware
- ESP32 with Wi-Fi module — $29
- Red button — $5
- Distance sensor — $10
- Laptop for backend — own or donated
- Skype room for cabin space
Speech Recognition
For speech recognition, we used the Whisper cloud model, as local solutions ran slowly on a low-powered laptop. Real-time processing sped up the process, but we were unable to achieve completely real-time performance.
LLM and Prompt Engineering
We tested GPT-3.5, GPT-4, and Claude, but chose GPT-4 for its more interesting and higher-quality responses. Prompt configuration proved to be an important aspect: the conciseness, humor, and style of the responses depended on it.
Text-To-Speech
We voiced the response from LLM using TTS. To minimize delay, token streaming was used: as the text was generated, complete sentences were formed, which were immediately sent to TTS and played back to the user. This reduced the initial delay to ~3 seconds.
UI and Feedback
To prevent users from thinking that the system had frozen, the screen displayed the following statuses: idle, waiting, listening, thinking, speaking.
Results
AI Confession Room ran for four days and attracted hundreds of participants. For just ~$50 on the OpenAI API, we received a lot of positive feedback and gained unique experience. The experiment showed that a voice interface to LLM can be created even with limited resources and difficult external conditions by combining STT, LLM, and TTS.
