ARIA
A multilingual AI receptionist that answers phone calls in the caller's own language, in real time.
ARIA is a phone-based AI voice agent for a medical clinic. Twilio handles incoming calls, audio streams over WebSockets to a Node.js server that acts as a central bridge, processing the caller's audio, reasoning about it, and speaking back naturally, in the caller's own language.
The system supports two swappable architectures: a native audio-in/audio-out pipeline on the Gemini Live API, and a sequential pipeline chaining Deepgram STT, Gemini, and a dual-provider TTS layer, toggled by a single config switch.
30+ languages, auto-detected
Deepgram's nova-2 multilingual model detects the caller's language (normalized to BCP-47) and the entire conversation continues in it. Deepgram Aura 2 for English, Google Cloud TTS for everything else.
Sub-800ms responses
The Gemini Live pipeline speaks natively audio-to-audio with tool calls executed synchronously mid-conversation.
Real-time barge-in
A local ONNX voice-activity detector watches every audio packet. Callers can interrupt mid-sentence and playback cuts instantly.
Stateful appointment booking
A GREETING to COLLECTING_INFO to CONFIRMING to BOOKED/FAILED state machine collects details, checks slot availability, books into Postgres, and sends an SMS confirmation.
Production plumbing
Session locks in Upstash Redis. Full transcripts and final call state logged to Supabase on every hangup.
- 01
Twilio streams continuous 8 kHz µ-law audio packets over WebSockets
- 02
Every packet feeds the local VAD. Speech onset fires an instant barge-in event
- 03
Live mode: audio is upsampled to Gemini Live and its audio response is downsampled back. Sequential mode: Deepgram STT, then Gemini, then language-aware TTS
- 04
Tool calls (check slots, book appointment, send SMS) execute with dynamic state instructions injected before each turn
- 05
On hangup: the Redis session lock releases and the full transcript plus final state log to Supabase