I have been building a voice agent runtime for most of this summer. Not the kind where you call an API and get a voice back. The kind where everything (VAD, STT, LLM, TTS) runs locally, on a 4GB laptop GPU, over a custom gRPC bidirectional stream I designed from scratch.
It worked. Sub-1.1 second end-to-end latency, a four-state debounce VAD, true duplex streaming, conversation history across turns. The whole thing.
There was just one problem: it only talked to WAV files on disk.
The Problem
My entire test harness was offline. I would feed the orchestrator a .wav file, it would go through the pipeline, and the response would come back as a .raw file I'd never actually listen to. I was basically measuring "does the pipeline run" rather than "does the AI actually work in a real conversation."
To test it properly, I needed something that would simulate a real phone call. Real-time, bidirectional, no third-party APIs, no Twilio, no SIM card, no phone number.
Then it hit me: WebRTC exists. Browser. Microphone. Works like a phone call. Zero cost. Available everywhere.
I watched the Fireship WebRTC video. Understood the handshake at a high level: SDP offer, SDP answer, ICE candidates, DTLS, SRTP. Looked straightforward enough.
This was the last moment of peace I had for about ten days.
The Plan (It Was Good, Actually)
The architecture I eventually settled on, after a lot of wrong turns, is genuinely clean. Three services, each with exactly one job:
AetherRTC terminates WebRTC from the browser: SDP, ICE, DTLS, SRTP, the works. It decodes G.711 audio to raw PCM and hands it to the Go orchestrator over a second, separate gRPC contract (gateway.proto). The orchestrator translates that into the existing agent.proto stream to Python. Python never learns that a browser exists. AetherRTC never learns what an utterance is.
Two proto contracts. One middle service. Clean boundaries.
The problem was getting here, because I did not start here.
The Part Where I Vibe-Coded and Suffered
My primary goal was always the AI agent. The VAD, the STT, the LLM pipeline. AetherRTC was supposed to be the boring plumbing I got done quickly so I could go back to building the real thing.
So I made a decision I am still paying for emotionally: I opened three AI tools simultaneously and let them write the WebRTC infrastructure while I focused on other things.
One was a Codex app for local checks. One was Claude. And one was Gemini.
Gemini is, to put it gently, not great at systems programming right now. It also named this project "AetherRTC," which is the one positive contribution it made.
The codegen wasn't worthless. Pion WebRTC's API is well-known enough that the skeleton came out roughly right: RTCPeerConnection, SDP negotiation, ICE candidate exchange, the basic OnTrack callback. But the moment it needed to actually think about system design (where the gRPC connection lives, who owns teardown, how the audio channels interact under backpressure), the outputs started quietly introducing bugs that would not surface until several other things were also working.
The bug I am most annoyed about was planted by this phase and found weeks later. But I'll get to that.
Confusion 1: Which Service Is The gRPC Server?
I had spent months building VAR with Go as the gRPC client and Python as the gRPC server. Python listened on :50051, Go dialed in. That's fine; Python happens to sit on a port, Go dials it. Authority over the session still lives entirely in Go.
But when I started thinking about where AetherRTC would connect, I got confused. I initially assumed AetherRTC would connect directly to Python, bypassing Go, reusing agent.proto. That would have been catastrophically wrong: two callers hitting Python's session model simultaneously, each with their own instance of VAD and conversation state, no coordination between them.
The actual shape: Go is a gRPC server to AetherRTC (listens on :50052, waits for connections) and simultaneously a gRPC client to Python (dials :50051 as it always has). Middle-tier service, two roles at once. Normal. The authority over session state stays exactly where it was.
Browser ──WebRTC──> AetherRTC ──gateway.proto──> Orchestrator-Go ──agent.proto──> Python
(server) (client)
(client)
Two separate proto contracts, deliberately different scopes. gateway.proto is dumb, just audio bytes and a sample rate. agent.proto carries AgentProfile, utterance signals, transcripts. AetherRTC never touches agent.proto. Python never sees gateway.proto. Go translates between them.
This sounds obvious in retrospect. It took a full architectural design session to actually nail down.
Bug 1: a sample rate that was never actually being read
First live browser test. Three terminals. Python, Go gateway server, AetherRTC. Browser tab, microphone, click Start Call.
Python logs: Profile received, agent: 'Sarah the Receptionist'
Then silence.
Checked the bridge. Audio was flowing: AetherRTC was logging decoded RTP packets, the stream was open, Go had attached the session. Everything looked fine.
The problem was sitting in main.py, written months earlier, never questioned:
SOURCE_SAMPLE_RATE = 44100
A hardcoded constant. Baked into AudioPreprocessor before the code ever read the incoming ControlSignal.
AetherRTC forces G.711 at 8kHz (that's what Pion negotiates with the browser, that's what we get). But the preprocessor was permanently assuming 44.1kHz, the WAV test harness rate from months ago, no matter what the wire actually said.
We had designed source_sample_rate into gateway.proto specifically to negotiate this at connect time. AetherRTC was sending it correctly. Go was forwarding it correctly. Python was receiving it correctly.
And then completely ignoring it.
# what we had
vad = VADDetector(VAD_MODEL_PATH)
preprocessor = AudioPreprocessor(source_sr=SOURCE_SAMPLE_RATE) # always 44100
# what we needed
if control.type == agent_pb2.ControlSignal.START_SESSION and preprocessor is None:
source_rate = control.source_sample_rate or DEFAULT_SOURCE_SAMPLE_RATE
preprocessor = AudioPreprocessor(source_sr=source_rate)
Defer construction until the actual negotiated rate is known. One change. First real audio finally made it through to VAD.
This is the pattern that kept repeating: the exciting new code was fine. The assumptions baked in months earlier, before a second consumer existed to expose them, were the actual bugs.
Bug 2: garbled audio for a completely different reason
STT started working, but what Whisper transcribed back was nonsense. Fragments. Wrong words. Occasionally total gibberish.
The instinct was to blame the sample rate again, we'd just fixed one, maybe there was another. There wasn't.
AudioPreprocessor.push() resamples every chunk it receives, independently, with no memory of the previous call:
def push(self, raw_bytes: bytes):
samples = np.frombuffer(raw_bytes, dtype=np.int16).astype(np.float32) / 32768.0
if self._source_sr != self.TARGET_SR:
samples = scipy.signal.resample_poly(samples, self._up, self._down)
...
AetherRTC calls this once per RTP packet. G.711 packetizes at 20ms. A polyphase resampling filter has ramp-up and ramp-down edge artifacts at the boundary of whatever block it's given. Feed it a 20ms block, get an artifact roughly every 20 milliseconds. Continuously. For the entire duration of your speech.
The WAV test harness chunked files into 4096-byte blocks, roughly 10x larger. The artifacts were always there, just so diluted against a large clean block that Whisper never cared. Live RTP packets exposed them immediately.
Fix: buffer raw samples across calls, resample in ~100ms blocks instead of per-packet.
def push(self, raw_bytes: bytes):
incoming = np.frombuffer(raw_bytes, dtype=np.int16).astype(np.float32) / 32768.0
self._raw_buffer = np.concatenate([self._raw_buffer, incoming])
while len(self._raw_buffer) >= self._resample_chunk_samples:
block = self._raw_buffer[:self._resample_chunk_samples]
self._raw_buffer = self._raw_buffer[self._resample_chunk_samples:]
resampled = scipy.signal.resample_poly(block, self._up, self._down)
# yield fixed 512-sample frames from resampled output
...
One architectural insight from this: AudioPreprocessor was always designed to carry a _remainder across calls for the VAD frame boundary. It just never needed to carry a resampling buffer across calls before, because the WAV test harness never exposed the case where input chunks were tiny. The change was small. The conceptual shift, that this function needs state across invocations for the resampling and not just for framing, was the actual work.
Bug 3: the pipeline freeze that looked like VAD dying
After 2-3 exchanges, VAD would stop detecting speech. Go would sit in RESPONDING. Nothing in any log looked like an error. You'd say "hello" five times and nothing happened.
This took the longest to find because the symptom pointed at the wrong place.
_read_pump runs on one thread in Python. When it detected a new utterance boundary while the previous one was still being processed, the old code did this:
if not ctx.utterance_done_event.is_set():
logger.warning("VAD boundary while utterance in progress. Waiting.")
ctx.utterance_done_event.wait() # blocks here
While blocked, _read_pump can't call next() on the gRPC stream. gRPC's flow control kicks in the instant a receiver stops reading. Go's send to Python starts blocking. That backpressures AetherRTC's send, which stalls its drain of PCMInboundChan. Once that buffered channel fills to capacity, the packet-receiving goroutine hits its default: branch and silently drops incoming audio.
select {
case session.PCMInboundChan <- pcmBytes:
default:
// silent drop, no log
}
From the outside: VAD stopped working. From the inside: your spoken words were being discarded three layers upstream, before VAD ever saw them. Go was still RESPONDING because it genuinely was, waiting on a Python thread that was waiting on itself.
The fix decoupled reading from dispatching entirely. VAD boundaries push onto a queue non-blocking, a separate dispatcher thread processes them one at a time:
def _utterance_dispatcher(self, ctx: SessionContext) -> None:
while True:
item = ctx.utterance_queue.get()
if item is _SHUTDOWN:
return
self._run_utterance(ctx, item)
def _handle_audio_event(self, ctx, vad, preprocessor, audio):
for frame in preprocessor.push(audio.data):
command = vad.process_frames(frame)
if command == VADCommand.END_OF_UTTERANCE:
self._dispatch_utterance(ctx, vad) # non-blocking, puts on queue
The read loop never stalls again regardless of how far behind inference falls.
Bug 4: the mute gate that never unmuted
By now, audio was actually reaching Python, STT was working, the LLM was responding correctly, TTS was synthesizing, and the AI was speaking back through the browser.
After exactly one exchange, it stopped listening to anything.
This one was genuinely funny once I found it. The outbound playback goroutine had an AgentSpeaking flag to prevent acoustic feedback (speaker output picked up by the mic and sent back as if you'd spoken it). Reasonable idea. The clear condition:
case <-time.After(300 * time.Millisecond):
if len(pcmBuffer) == 0 {
session.AgentSpeaking.Store(false)
}
TTS output is essentially never an exact multiple of the 20ms frame size (320 bytes). After draining all the complete frames, there would always be a handful of leftover bytes sitting in pcmBuffer. Something like 47 bytes of sub-frame audio from the tail of the last sentence.
len(pcmBuffer) == 0 was never true again. AgentSpeaking stayed true permanently. Every microphone packet after the first response was silently discarded by the inbound goroutine:
if session.AgentSpeaking.Load() {
continue // your words, thrown away
}
One line fix:
case <-time.After(300 * time.Millisecond):
session.AgentSpeaking.Store(false) // unconditional
pcmBuffer = pcmBuffer[:0] // clear the remainder too
Stop making the clear conditional on anything. On idle timeout, just clear, always. The sub-frame remainder is under 20ms of audio. You won't notice it's gone. What you will notice is the AI never listening to you again.
The Part The Other AIs Made Worse
At various points during this, I ran suggestions from other tools through the codebase. Some of them were fine. Two were actively harmful.
The DoneChan regression. The inbound goroutine's default: drop, the thing that prevents the capture loop from blocking under backpressure, got replaced with case <-session.DoneChan: return at some point. Sounds reasonable: clean shutdown path, goroutine exits when the call ends. Problem: DoneChan doesn't close until the whole call ends. So under any backpressure, the select has two cases: "send to a full channel" (blocks) and "wait for the whole call to end" (doesn't help mid-call). The goroutine freezes. Microphone capture stops. It looks exactly like Bug 3 again. This regression was silently introduced twice, by two different AI suggestions, on two different days.
A barge-in feature that showed up uninvited. One refactor pass added full barge-in handling: response cancellation flags, Go flushing the audio channel, state machine force-transitions back to ACTIVE. Real feature. Never scoped. Not part of any milestone I was working on. Landed in the same commit as outbound playback, so when things broke (and they broke), there were now two unverified, nontrivial pieces of logic to debug simultaneously instead of one.
The pattern with AI-generated code in this project: skeleton and boilerplate, fine. Anything touching concurrency, backpressure, or teardown ordering needs human review before committing, because the failure mode is always the same: compiles, appears to work in happy-path testing, silently corrupts state under any real load.
Bug 5: the audio that arrived all at once
Even after everything above was fixed, playback sounded wrong. Choppy, sped up, wrong tempo.
WriteSample's Duration field in Pion affects RTP timestamp math, how much to increment the timestamp for the next packet. It does not pace when packets actually get sent onto the wire.
If five TTS chunks are already queued in PCMOutboundChan, this loop:
case pcm := <-session.PCMOutboundChan:
outboundTrack.WriteSample(pionmedia.Sample{Data: ulawData, Duration: 20 * time.Millisecond})
writes all five of them back-to-back, essentially instantly. The browser's jitter buffer receives what should have been 100ms of audio arriving in microseconds. It does its best. It sounds terrible.
Fix: pace writes against a ticker, and re-frame into exact 20ms blocks before writing:
const pcmFrameBytes = 320 // 20ms at 8kHz, 16-bit mono
for len(pcmBuffer) >= pcmFrameBytes {
<-ticker.C // wait for the real 20ms wall-clock tick
frame := pcmBuffer[:pcmFrameBytes]
pcmBuffer = pcmBuffer[pcmFrameBytes:]
outboundTrack.WriteSample(pionmedia.Sample{
Data: codec.EncodeUlaw(frame),
Duration: 20 * time.Millisecond,
})
}
Now each frame arrives when it's actually supposed to, not whenever the goroutine gets scheduled.
What Finally Worked
After ten days:
Terminal 3: [WebRTC session_46367] ICE State: connected
Terminal 2: [Gateway] Session session_46367 attached to inference engine.
Terminal 1: [InferenceEngine] [session_46367] STT: 'Hello, my name is Alita and can I know which time you are free?'
And then, through my laptop speakers, a voice saying: "Hi Alita! Welcome to Smile Dental Clinic. How about tomorrow at 3 PM?"
Sarah the Receptionist. Locally. From a browser microphone. With no external API calls anywhere in the stack.
The full loop: Browser mic -> WebRTC -> AetherRTC -> gRPC -> Orchestrator-Go -> gRPC -> Silero VAD -> Faster-Whisper -> Qwen2.5:3b -> Piper TTS -> gRPC -> AetherRTC -> WebRTC -> browser speaker.
Three services. Two proto contracts. Five distinct bugs that each individually looked like total system failure. About ten days of debugging. Entirely local hardware.
What I Learned
Every assumption baked in before a second consumer exists will eventually be a bug. The sample rate constant. The per-packet resampling. The WAV-file test harness chunk sizes. None of these were bugs when there was only one caller. All of them became bugs the moment a live browser with real RTP packets started hitting the same code.
Real-time audio surfaced bugs that offline tests structurally could not. The resampling artifact, the backpressure stall, the mute gate issue: none of these were possible to hit with two clean pre-recorded WAV files and a deliberate 700ms silence gap between them. Live conversational cadence hits all the edge cases simultaneously.
Vibe-coding works for scaffolding and fails for concurrency. The Pion WebRTC skeleton, the gRPC client structure, the signal/answer flow: all fine from AI codegen. The select statement teardown ordering, the channel backpressure policies, when to clear a flag: all wrong, sometimes subtly, sometimes catastrophically.
Isolating variables is the only way to debug a multi-service pipeline. Every time a fix landed alongside an unrelated change, the next debugging session started with at least two hypotheses instead of one. Every time I confirmed one layer worked before touching the next, the bugs were isolated and fast to find.
What's Left
The pipeline works. The AI responds. Audio plays back in the browser.
What doesn't exist yet: a monitor goroutine for actual barge-in (speaking while the AI speaks), a proper jitter buffer policy for out-of-order RTP packets, TURN server configuration for real-world NAT traversal (right now this only works on the same local network as the AetherRTC server), and the Phase 2 agent work: tool calling, memory, the actual point of building a voice agent runtime rather than a voice agent demo.
But the infrastructure is there. Both directions, both services, end to end, proven by real logs and real ears.
The trauma was real. The vibe-coding regret was real. The ten days were real.
The AI talking back through the browser was also real, and that part was worth it.
Source: voice-agent-runtime and AetherRTC