For the past two months, I’ve been building Voice Agent Runtime, distributed, self-hosted voice agent built across a Go orchestrator and a Python inference engine, connected over a custom gRPC bidirectional stream.
There are countless ways to build voice agents today using third-party black boxes like ElevenLabs, Twilio, or Vapi. But honestly, I had built one in the past sewing all the APIs in to one wrapper and call it as AGENTIC AI. I was naive, and most importantly it only taught me how to control rate limits, figuring out the correct way to initialize the apps for different purpose and how to make it cost effective. So I thought why not make it from scratch and spend two nearly two months now on that.. lol.
Fast forwarding to now, I built a system which is truly local, runs on consumer GPU/CPU and gives a sub ~1.1s for complete workflow of event.
In this blog I will be discussing about the implementation of Voice-Activity-Detection using SileroVAD.
The most complex part of this journey wasn’t the AI models; it was solving the networking and concurrency problems. Specifically, evolving the system from a naive, sequential script into a true-duplex streaming engine, and then teaching it how to listen.
This is the Story of that.
The Walkie-Talkie Problem
When you first build a voice pipeline, you probably have seen something like this:
def handle_session(stream):
for event in stream:
if event.type == "audio":
audio_buffer.append(event.audio)
if event.type == "end_of_utterance":
full_audio = b''.join(audio_buffer)
text = stt.transcribe(full_audio) # BLOCKS
response = llm.generate(text) # BLOCKS
audio_response = tts.synthesize(response)# BLOCKS
yield Event(audio=audio_response) # BLOCKS
audio_buffer.clear()
This was the first and foremost thing i have written and tested against one single recorded audio file, and it worked, life was easy.
But the moment I got to know this is half-duplex the Walkie-Talkie system, I understood this will gonna hurt soon, but the return was invaluable. In the course of making all this things, I learned a lot about FSM, debounce architecture, Read/Write Pump, concurrency, Events and many more…
Coming back to the topic, the above code does some thing like.. the channel gets locked by the OS read/write mechanisms so data doesn't get corrupted. You speak, you wait for the STT, the LLM thinks, the TTS generates the audio, and then you hear the response.
The timeline looks like this:
CLIENT: [SPEAKING............][WAITING........................]
│ │
▼ ▼
PYTHON: [STT][LLM][TTS]
│ │ │
▼ ▼ ▼
BLOCK BLOCK BLOCK
Problem!
Total latency: STT time + LLM time + TTS time + network.
User experience: “Why is it pausing?”
In real-time applications, forcing the user to wait for a 5-second sequential pipeline before they can speak again is a sin. We needed full-duplex like a phone call, where neither party needs to wait for the other.
The Three-Worker Architecture
The power of concurrency comes from threads (Python) and goroutines (Go). But they aren’t magic; they are just names for “an independent little worker inside a program that runs alongside other workers, instead of waiting in line.”
To fix the blocking, I implemented the Producer-Consumer pattern using a thread-safe queue.Queue as a "mailbox" between three distinct workers:
-
read_pump(The Listener): Runs in a background thread. Its only job is to pull audio from the network and stash it. If it sees an utterance boundary, it doesn't process it, rather, it spawns a temporary worker and immediately goes back to listening. -
_run_utterance(The Thinker): A temporary thread spawned per sentence. It does the heavy lifting (STT → streaming LLM → streaming TTS) and drops audio chunks into the mailbox. -
write_pump(The Relay): The main thread. It just loops, pulling chunks out of the mailbox and yielding them back to Go.
Visualized as:
CLIENT: [SPEAKING utterance 1][SPEAKING utterance 2]
│ │
read_pump:[Receiving 1][end!] [Receiving 2][end!]
│ │ │ │
│ └─spawn thread──> │ └─spawn thread──> ...
▼ ▼
Thread 1: [STT][LLM──stream──>][TTS──stream──>]
│ │
▼ ▼
QUEUE: [chunk][chunk] [chunk][chunk]...
Key Insight: While Thread 1 is processing utterance 1, read_pump is already receiving utterance 2. The client can speak at any time.
Problem solved? Not quite.
I have written a complete blog on True Duplex worth checking that out if you are interested, alternatively you can see Here.
The Missing Piece: Boundaries
In the code above, I wrote if event.type == "end_of_utterance". But how does the system actually know when an utterance ends?
Figuring out how to correctly identify and index the boundaries of spoken audio was so complex that I dedicated an entire phase to it. Initially, I relied on the Go orchestrator to send an explicit END_OF_UTTERANCE control signal. But that doesn't scale. A live caller doesn't press a "done" button. If I wanted to plug this into a WebRTC browser client later, the AI backend had to learn how to listen.
I had to build a Voice Activity Detection (VAD) system.
Falling Into the ONNX Trap
I chose Silero VAD, running locally via ONNX runtime to avoid heavy PyTorch dependencies. But here is a lesson I learned the hard way: never trust documentation from memory; trust the actual file.
Most Silero tutorials(or even AI) assume v4 or v5, which expect two separate state tensors for the recurrent neural network: h (hidden state) and c (cell state), both of size 64.
I downloaded the model, wrote my wrapper, and got this:
IndexError: list index out of range
I printed the actual input signature of the file I had downloaded (which was Silero v6):
Name: input | Shape: [None, None] | Type: tensor(float)
Name: state | Shape: [2, None, 128] | Type: tensor(float) <-- Fused!
Name: sr | Shape: [] | Type: tensor(int64)
See that sr in the input? It's the sample rate, which is a scalar value.
The creators had fused the h and c tensors into a single state tensor of size 128 (64 + 64). Furthermore, the sample rate was no longer a 1D array [16000], but a 0-dimensional scalar.
Once I aligned my code to the actual contract, the model worked flawlessly, yielding a single float confidence score per 32ms frames. But a raw probability score isn’t an utterance boundary.
The 4-State Debounce Machine
Raw per-frame speech probability has no concept of an utterance boundary since a state machine sits on top of it. Just trigger on score > 0.5, a cough, a door slam, or a loud exhale will trigger the LLM.
Solution: A 4-state machine:
class VADState(enum.Enum):
SILENCE = "SILENCE"
SPEECH_STARTING = "SPEECH_STARTING"
SPEECH = "SPEECH"
SPEECH_ENDING = "SPEECH_ENDING"
to act as a debounce filter:
SILENCE:Waiting.SPEECH_STARTING:Sound detected, but waiting to see if it lasts longer than 250ms (killing false starts like coughs).SPEECH:Confirmed speech. Accumulating audio for the STT.SPEECH_ENDING:Sound dropped. Waiting to see if the silence lasts longer than 500ms (allowing mid-sentence breaths without cutting the user off).
If a user says, “I’d like to book… [200ms breath] …an appointment,” the state machine hits
SPEECH_ENDINGduring the breath, but because 200ms < 500ms, it seamlessly transitions back toSPEECHinstead of chopping the sentence in half.
Making software without bugs? Not a chance.
The "Lookback Time Machine"
There was a fatal flaw in my initial state machine: VAD is inherently reactive.
It requires 250ms of continuous speech before it transitions to SPEECH. By the time the state machine says "Start recording!", the "H" and "e" in "Hello" are already gone forever. The STT would receive "llo", fail to transcribe it, and the LLM would get confused.
To fix this, I implemented a Lookback Buffer.
Using Python’s collections.deque, I continuously maintain a rolling buffer of the last ~320ms of audio frames, even while in the SILENCE state.
lookback_frame_count = max(1, round(lookback_ms / self.FRAME_DURATION_MS))
self._lookback = collections.deque(maxlen=lookback_frame_count)
When speech is finally confirmed and we move to SPEECH_STARTING, we don't just start recording from that exact millisecond. We reach into the past:
def _handle_silence(self, frame: np.ndarray, is_speech: bool) -> VADCommand:
if is_speech:
self._vad_state = VADState.SPEECH_STARTING
self._pending_frames = list(self._lookback) # <-- Grab the past!
self._pending_frames.append(frame)
Now, when the STT receives the audio, it gets the complete phonetic beginning of the word. We essentially gave the detector a short-term memory.
Evolving for Production
With VAD integrated, the system was autonomous, but the codebase needed to evolve to support it cleanly.
Passing six loose arguments into a spawned thread (threading.Thread(target=_run_utterance, args=(id, audio, prompt, queue, context, event))) gets messy fast. If you want to add latency metrics or cancellation logic, you have to update every thread spawn point.
I refactored the system using Python’s @dataclass to encapsulate the entire life of a phone call into a single SessionContext object:
@dataclass
class SessionContext:
session_id: str
outbound_queue: queue.Queue
grpc_context: grpc.ServicerContext
utterance_done_event: threading.Event = field(default_factory=threading.Event)
system_prompt: str | None = None
Now, every worker just takes ctx. Need to check if the user disconnected? ctx.grpc_context.is_active(). Need to trigger the next sentence? ctx.outbound_queue.put().
I also broke the monolithic read_pump down into distinct handlers (_handle_audio_event, _handle_control_event). This made plugging the VAD into the audio handler completely isolated from the gRPC streaming logic.
The “Two Bosses” Problem
During the transition from Go-triggered boundaries to Python VAD-triggered boundaries, both systems were live. What if the VAD detected the end of a sentence at the exact millisecond Go sent a manual signal?
I solved this with a gate in a new _dispatch_utterance method. It doesn't matter who yells "I'm done!" first-Go or the VAD. The method checks if a thread is already running. If it is, it waits. If a signal fires twice for the same audio, the second one is safely ignored. No double-processing. No garbled audio.
The Final Picture
What started as a naive, blocking script evolved into a highly concurrent, state-managed system.
Today, the read_pump listens continuously. The AudioPreprocessor dynamically resamples incoming audio down to 16kHz. The VADDetector feeds 32ms frames into the 4-state machine, using its lookback buffer to capture the start of words, silently dropping breath pauses. When the user stops talking, it yields a command. The dispatcher checks the gate, grabs the accumulated audio buffer, and spins up a daemon thread.
Meanwhile, the main thread just pulls synthesized TTS audio out of the queue and yields it back the exact millisecond it's generated.
The client speaks, the system listens, thinks, and speaks back. True duplex. No walkie-talkies. Just a clean, continuous stream of bytes.
So yeah, thanks for reading, you are cool, wierd and non-chalant who read my silly blog on a silly project, though it consumed most of my fat(i lost 5 kg lol), and time. I am still making still learning and most importantly still evolving. Each day just a 10% improvement and here I am, a naive API caller to a full architectural driven, Software developer making decision at the system level.
Thanks again. And if you are a Recruiter who has read this, read my resume here, and HIRE ME PLESAE🥺.