LLM + RAG + speech · 2024 – Present
Affective Multimodal Assistant
A multimodal LLM assistant that listens, reads emotional tone from the voice, retrieves context and answers in speech, text or images.
<400 ms
response latency
Most assistants hear the words and ignore how they were said. This project explores an assistant that responds to both.
Architecture
The assistant supports speech and text input with speech, text and DALL·E image output, responding in under 400 ms under typical workloads.
Hearing emotion
An acoustic emotion classifier built on RMS energy, spectral centroid, zero-crossing rate and pitch reached 82% accuracy across a 10-person pilot study, measured with cross-validated experiments.
Retrieval
A lightweight RAG module using OpenAI embeddings improved retrieval relevance by 38% on a curated Q&A set in offline evaluation.
In use
The real-time interface (Flask, the Web Speech API and FFmpeg) sustained 95% uptime across more than 100 interaction sessions.
Next project
Media Bias Detection →
96%
accuracy, 5 classes