Why this exists
Voice Agent Runtime (VAR) had been working for months: a Go orchestrator and a Python inference engine, talking over a custom gRPC stream, running speech-to-text, an LLM, and text-to-speech entirely on local hardware. But it only ever spoke to WAV files on disk. There was no way for an actual human to talk to it through a browser.
AetherRTC was supposed to fix that: terminate WebRTC from a browser, decode the audio, hand it to VAR. On paper it sounded like a weekend of plumbing. It took about a week and a half of near-daily debugging, several genuinely wrong mental models, and at least four distinct bugs that each individually made the whole system look completely broken.
This is the record of that week and a half: the architecture, the bugs, and what each one actually taught.
Getting the mental model right, first
Before any code, there was a real conceptual tangle worth naming honestly. I initially assumed AetherRTC would connect directly to Python's inference engine, reusing the existing agent.proto contract Go already used. That's wrong, and it took a full conversation to see why.
"gRPC server" and "gRPC client" are just wire roles: who listens, who dials in. They say nothing about which service is actually in charge. Python is a gRPC server today because it happens to sit on a port waiting for Go to call it. That doesn't make Python the authority over anything; Go still owns the entire session lifecycle.
Once that clicked, the real topology fell out naturally:
Browser <--WebRTC--> AetherRTC <--gRPC--> Orchestrator-Go <--gRPC--> Inference-Python
(gateway.proto) (agent.proto)
Go plays two roles simultaneously: a gRPC server to AetherRTC, and a gRPC client to Python. Nothing unusual about that; it's the ordinary shape of any middle-tier service. And it meant two proto contracts, not one:
agent.proto: AI-aware, carriesAgentProfile, transcripts, VAD signals. Untouched by any of this work.gateway.proto: deliberately dumb. Onlysession_id, raw audio bytes, and a negotiated sample rate. AetherRTC was never allowed to know what an "utterance" is.
service Gateway {
rpc StreamAudio(stream GatewayEvent) returns (stream GatewayEvent);
}
message GatewayEvent {
string session_id = 1;
oneof payload {
AudioChunk audio = 2;
GatewayControl control = 3;
}
}
message GatewayControl {
enum SignalType {
START_SESSION = 0;
END_SESSION = 1;
}
SignalType type = 1;
int32 source_sample_rate = 2;
}
That last field, source_sample_rate, turned out to matter more than it looked like it would.
Milestone 1-3: proving the bridge without a browser
Before touching AetherRTC at all, the plan was to prove Go's side of the bridge worked using a throwaway test client that pretended to be AetherRTC, streaming a WAV file over gateway.proto instead of a live mic. Isolate the layer, verify it, then move to the next one.
The one real design mistake here was in the shutdown logic. My first instinct was to run the inbound relay (AetherRTC to Python) as the main blocking loop, and let its defer clean up the outbound relay (Python to AetherRTC) goroutine. That only handles one of the two ways a call can end: AetherRTC hanging up. If Python's stream died independently (a crash, an aborted inference), the outbound goroutine would die quietly, but nothing would tell the inbound loop to stop. It would keep forwarding audio into a dead connection.
A second-opinion review (run through a different AI in a separate editor, checked against this one) caught it and pointed at the actual existing precedent already sitting in the codebase: readPump, the goroutine reading from Python, was already the one that owned signalDone(). The fix was to run both directions as goroutines reporting to a single coordinating point:
go sess.Run()
outboundDone := make(chan struct{})
go func() {
defer close(outboundDone)
for chunk := range sess.AgentAudioChan {
stream.Send(&gatewaypb.GatewayEvent{
SessionId: sessionID,
Payload: &gatewaypb.GatewayEvent_Audio{Audio: &gatewaypb.AudioChunk{Data: chunk}},
})
}
}()
inboundErr := make(chan error, 1)
go func() {
for {
event, err := stream.Recv()
if err != nil {
inboundErr <- err
return
}
if audio := event.GetAudio(); audio != nil {
sess.SendAudio(audio.Data)
}
}
}()
select {
case <-sess.DoneChan:
case err := <-inboundErr:
}
<-outboundDone
return nil
Whichever side ends first wins the select. No manual CloseSend(), no timing assumption about who finishes first. This shape held up for everything that came after.
The test client proved it: a WAV file streamed through the fake AetherRTC client produced a correct STT to LLM to TTS cycle in Python's logs, with response audio flowing back and landing in an output file. Full round trip through Go, no browser yet.
Bug 1: a sample rate that was never being read
With the bridge proven, AetherRTC's real client code went in: dial Go, send START_SESSION, drain the browser's decoded PCM into the stream. The moment a real browser tab connected, Python's logs lit up correctly. VAD: speech started, then silence, then... Profile received and nothing else. STT never fired.
The bug was almost embarrassingly simple once found. main.py had:
SOURCE_SAMPLE_RATE = 44100
A module-level constant, built into AudioPreprocessor before the code ever read the incoming ControlSignal. AetherRTC forces G.711 at 8kHz (WebRTC negotiates it, Pion's media engine is configured for it), but the preprocessor was permanently assuming 44.1kHz, no matter what the wire actually said. The source_sample_rate field we'd designed into gateway.proto specifically to avoid this was being sent correctly and ignored completely.
The fix: defer constructing the preprocessor until the real negotiated rate is known.
if event.HasField("control"):
control = event.control
if control.type == agent_pb2.ControlSignal.START_SESSION and preprocessor is None:
source_rate = control.source_sample_rate or DEFAULT_SOURCE_SAMPLE_RATE
preprocessor = AudioPreprocessor(source_sr=source_rate)
This is the first of several times in this project where the actual bug wasn't in the exciting new code. It was in an assumption baked in months earlier, before there was a second consumer to expose it.
Bug 2: garbled audio, and the sample rate wasn't the reason this time
Fixing the sample rate got audio flowing, but what Whisper transcribed back was nonsense: fragments, wrong words, occasionally nothing recognizable as speech. The instinct was to blame the sample rate again. It wasn't that.
AudioPreprocessor.push() resamples every chunk it's handed, independently, with no memory of what came before:
if self._source_sr != self.TARGET_SR:
samples = scipy.signal.resample_poly(samples, self._up, self._down)
AetherRTC calls this once per RTP packet, and G.711 packetizes at 20ms. A polyphase resampling filter has ramp-up and ramp-down artifacts at the edges of whatever block it's given. Feed it a 20ms block, get an artifact roughly every 20 milliseconds, continuously, for the entire time someone speaks. The WAV-file test harness never caught this because it chunked files into much larger blocks. The artifacts were there too, just diluted below the point of mattering.
The fix was to buffer raw samples across calls and only resample in larger blocks:
def push(self, raw_bytes: bytes):
incoming = np.frombuffer(raw_bytes, dtype=np.int16).astype(np.float32) / 32768.0
self._raw_buffer = np.concatenate([self._raw_buffer, incoming])
while len(self._raw_buffer) >= self._resample_chunk_samples:
block = self._raw_buffer[:self._resample_chunk_samples]
self._raw_buffer = self._raw_buffer[self._resample_chunk_samples:]
resampled = scipy.signal.resample_poly(block, self._up, self._down)
...
Cutting the artifact frequency down roughly fivefold, from every 20ms to every 100ms, was enough to make STT reliable.
Bug 3: the one that looked like VAD giving up
This was the hardest one to diagnose, because the symptom pointed at the wrong subsystem entirely. After two or three exchanges, VAD would simply stop detecting speech. Go's orchestrator would sit there saying RESPONDING. Nothing in any log looked like an error.
The actual mechanism: _read_pump runs on one thread. When VAD detected a new utterance boundary while the previous one was still generating a response, the code did this:
if not ctx.utterance_done_event.is_set():
ctx.utterance_done_event.wait()
That .wait() blocks the same thread that reads incoming audio. While blocked, it can't call next() on the gRPC stream. gRPC applies flow control the instant a receiver stops reading, so Go's send to Python starts blocking too. That backpressures AetherRTC's own send, which stalls its drain of the inbound channel. Once that channel filled, the packet-receiving goroutine hit its default: drop branch and silently discarded incoming audio, with no log line anywhere, because that branch was never expected to matter much.
From the outside it looked exactly like VAD had stopped working. What had actually happened was that real spoken audio was being thrown away three layers upstream, before VAD ever saw it.
The fix decoupled reading from dispatching entirely. VAD boundaries now push onto a queue instantly, non-blocking, and a separate dedicated thread pulls from that queue one utterance at a time:
def _utterance_dispatcher(self, ctx: SessionContext) -> None:
while True:
item = ctx.utterance_queue.get()
if item is _SHUTDOWN:
return
self._run_utterance(ctx, item)
The read loop never stalls again, no matter how far behind inference falls.
Milestone 5: getting the AI to actually speak back
With inbound audio solid, the last piece was outbound: encoding PCM back to G.711, adding a real outbound WebRTC track, writing it to the browser. This is where the largest number of small, compounding regressions happened, mostly because too many things got changed at once between test runs.
Regression one, twice. The fix for the backpressure bug above, a default: drop instead of a blocking send on the inbound channel, got silently reverted twice during later refactors. Once it was replaced with a case <-session.DoneChan: return, which sounds like a clean shutdown path but isn't one: DoneChan doesn't close until the whole call ends, so under any backpressure it just blocks the microphone-capture loop indefinitely. Both times, the fix was to go back to the drop-with-counter version:
select {
case session.PCMInboundChan <- pcmBytes:
default:
droppedCount++
if droppedCount%50 == 0 {
log.Printf("PCMInboundChan full, dropped %d packets.", droppedCount)
}
}
The mute gate that never unmuted. Once playback worked, the browser's own speakers could feed the AI's voice straight back into the microphone as if a person had said it, a feedback loop. The fix was an AgentSpeaking flag that skipped inbound processing while the agent was talking. Reasonable idea, badly timed clear condition:
case <-time.After(300 * time.Millisecond):
if len(pcmBuffer) == 0 {
session.AgentSpeaking.Store(false)
}
TTS output is essentially never an exact multiple of the 20ms frame size, so a handful of leftover bytes almost always sat in pcmBuffer after the last sentence of a response. len(pcmBuffer) == 0 never became true again. AgentSpeaking stayed true for the rest of the session, and every microphone packet after the first response got silently discarded, which looked, again, like a completely different bug from the outside. The fix was to stop making the clear conditional on anything:
case <-time.After(300 * time.Millisecond):
session.AgentSpeaking.Store(false)
pcmBuffer = pcmBuffer[:0]
Garbled playback, for a different reason than garbled input. Even once audio reached the browser, it sounded wrong, because WriteSample's Duration field only affects RTP timestamp math, not the actual pacing of when packets get sent. If several TTS chunks were already queued, they'd all get written to the wire in a burst rather than one every 20 milliseconds like real-time audio requires. The fix paced writes against a ticker instead of trusting the duration field to do it:
for len(pcmBuffer) >= pcmFrameBytes {
<-ticker.C
frame := pcmBuffer[:pcmFrameBytes]
pcmBuffer = pcmBuffer[pcmFrameBytes:]
outboundTrack.WriteSample(pionmedia.Sample{Data: codec.EncodeUlaw(frame), Duration: 20 * time.Millisecond})
}
Four separate bugs, each capable of looking like total system failure on its own, all hiding in roughly forty lines of code.
What actually made this hard
None of these bugs were individually sophisticated. What made the week and a half hard was that several of them produced nearly identical symptoms. "It just stops working after a couple of exchanges" was the surface behavior of at least three unrelated root causes over the course of this integration: a resampling artifact, a blocking read-pump, and a mute gate with a bad clear condition. Every time a fix landed alongside an unrelated change in the same commit, the next bug took longer to isolate, because there were now two unverified things instead of one.
The pattern that actually worked, every time: revert to the last confirmed-good state, change exactly one thing, retest, read the actual current file contents rather than trust a description of what should be there. More than once, asking to see a pasted file instead of assuming it matched an earlier version surfaced a regression that would have otherwise cost another full debugging cycle.
Where this leaves things
The full loop is closed: a real browser tab, a real microphone, a real spoken response, played back through real speakers, having crossed AetherRTC, Orchestrator-Go, and Inference-Python, with VAD, Whisper, Qwen2.5:3b, and Piper all running locally on a 4GB laptop GPU the entire time.
What's left is verification, not new construction: full lifecycle testing (connect, speak several turns, disconnect, confirm no leaked goroutines, confirm one session_id traces cleanly through all three services' logs), some documentation housekeeping, and then the genuinely deferred work of a monitor goroutine for barge-in, a jitter buffer policy, and eventually the question of whether AetherRTC ever becomes more than a one-user proof of concept.
But the core claim now has evidence behind it, not just architecture diagrams. This runs, end to end, entirely on local hardware, through a browser.