Keep audio moving while inference is busy
Voice Agent Runtime connects voice activity detection, speech recognition, language generation, and speech synthesis through Go and Python services. The central design problem is not just choosing models: slow inference must not block the service from reading incoming audio.
Separate intake from utterance work
AetherRTC bridges browser media to the Go orchestrator. The orchestrator forwards the session ID and source sample rate over a second gRPC hop to Python inference. Audio ingestion and the utterance worker run independently; completed utterances are processed serially to preserve order.
The Silero VAD keeps its model state across utterances and uses speech confirmation, lookback audio, and a maximum utterance duration. The live service waits for 800 ms of silence to close an utterance. That boundary trades faster turn completion against cutting a speaker off too early.
Return sentences, not a fully buffered response
Sentence-level language output feeds Piper TTS without waiting for the whole answer. Session profile information reaches inference, and conversation history is capped at twelve message entries.
Everything runs locally on a laptop with a 4 GB RTX 3050. Silero VAD (ONNX),
faster-whisper base (int8) and Piper (en_US-lessac-medium) run on the CPU.
Only the language model, qwen2.5:3b (Q4_K_M) through Ollama, uses the GPU.
Speech-to-text runs on the whole utterance once the VAD closes it; it does not
stream, and it is set to English only.
What the benchmark measured
A benchmark harness plays twelve pre-recorded utterances into the orchestrator over gRPC, the way AetherRTC does, and times each reply on the client. Three fresh-start runs gave 180 turns, and 179 completed. The median time from end of speech to the first audio byte was 2.07 s (p95 2.52 s, p99 2.67 s), and the three runs agree to within about 2%. Of that, 0.8 s is the VAD silence window, so the pipeline itself accounts for about 1.27 s.
Inside the pipeline, speech-to-text takes 0.67 s at the median on the CPU, the language model's first token arrives after 0.28 s, and Piper synthesizes a sentence in about 0.11 s, roughly 20 times faster than real time. Only the language model uses the GPU, peaking at 2.1 GB of the 4 GB available. The first reply after a cold start takes 4.6 to 7.6 s while the model loads.
The breakdown changes the question. Compute is not the bottleneck: about 40% of the delay is waiting to be sure the speaker has finished, and about a third is CPU transcription. Overlapping transcription with the silence window is a more promising change than a faster model. That is untested.
A short word the VAD missed
Replaying the audio through the real preprocessor and detector 50 times with the service's settings, a 0.48 s "Hello." was detected in 43 trials. The eleven utterances of 1.45 s and longer were detected every time. My first guess was stale model state, since the detector is never reset. Resetting it at every boundary did not help (42 of 50), so that guess was wrong. The same audio is detected or missed depending on how much silence comes before it: the model scores the first frames of the word below the 0.5 threshold, and the detector needs eight consecutive frames above it, while a one-word utterance is only about fifteen frames long. Tolerating a few frames of dip recovered only part of the misses (45 of 50). I have not adopted it, because tuning on one short word would be overfitting. The experiment uses one synthetic voice.
What is still open
Independent ingestion and output are not the same as barge-in: cancelling an in-flight response when new speech begins remains open work. Python's utterance and outbound queues are currently unbounded, so overload behavior needs work before claiming a bounded pipeline.
The numbers above come from a gRPC client, not a browser. Browser capture, WebRTC jitter and playout, real speech, background noise, and concurrent calls are not measured, and the synthetic speech makes recognition accuracy look better than it would be with real callers. A stall after about two exchanges, seen intermittently in browser testing, did not reproduce through the gRPC path, and its cause is not known. Tool calls, durable memory, and RAG are not implemented runtime features.