Getting to 'Near Enough' Live Transcription

Chunking instead of streaming

Picked up some freelance backend work this year on a live-events platform (webinars, virtual conference sessions, that kind of thing). Somebody asked for a feature that sounded almost too simple to be worth writing up: let people search the transcript of a session that’s still going on. Someone joins forty minutes late, wants to know what the speaker said about pricing, doesn’t want to scrub a video player or wait for the recording. Type “pricing,” jump to the timestamp, done.

Except the transcription pipeline I inherited had never been asked to do that. It transcribed a session once, after it ended, in one shot. Reasonable design for a recording. Completely wrong shape for “answer this while the thing is still airing.”

I was already halfway to the wrong answer before I’d looked at anything else. Real-time transcription: a live socket into a speech API, words streaming back as people talk, done properly. It’s the version that sounds like the actual engineering answer to “make it live.” I got about an hour into sketching what that would need before I made myself stop and check what we actually had to work with.

What we had didn’t support that: HLS output, not a raw low-latency audio tap, and a fleet of workers that ran to completion and exited, nothing that stayed up for hours on a live connection.

flowchart LR
    subgraph S["Streaming (rejected)"]
        A1[Live audio] -->|persistent socket, new| B1[Speech API, streaming]
        B1 -->|words as spoken| C1[Transcript]
    end
    subgraph K["Chunking (built)"]
        A2[Live audio] -->|ffmpeg cuts 5-min window| B2[Existing async pipeline]
        B2 -->|submit + callback| C2[Transcript]
    end
    S ~~~ K

Streaming means a new persistent connection into the speech API. Chunking means pointing the pipeline that already existed at a moving window.

And for what? A few hundred milliseconds nobody had asked for. The actual requirement, once I looked at what was actually being asked instead of my own mental model of it, was “a transcript should exist within a couple minutes of someone saying the words.” Streaming chases sub-second latency. The ask was minutes.

So I built the right column: chopped a live session into rolling five-minute chunks and ran each one through the transcription pipeline that already existed, aimed at a moving window instead of a whole file. Somewhere in my notes from that week there’s a half-finished architecture diagram for the streaming version I didn’t build, which I keep around as a reminder to nail down what’s actually being asked before opening a design doc.

The sleep-formula bug

The rolling five-minute loop I shipped was quietly slow, and it took me a while to notice.

The first version was exactly what you’d sketch on a whiteboard: cut the next window out of the stream with ffmpeg, submit it to the speech API, sleep five minutes, repeat. It worked. Transcripts showed up. I moved on to the next thing.

What made me come back to it was watching the actual timestamps land: consistently closer to six and a half minutes behind than five. Not broken, just off:

gantt
    dateFormat HH:mm:ss
    axisFormat %M:%S
    section sleep
    naive sleep   :a1, 00:00:00, 300s
    fixed sleep   :b1, 00:00:00, 300s
    section wait
    naive wait    :active, a2, 00:05:00, 90s
    fixed wait    :active, b2, 00:03:30, 90s

Naive sleeps a full window, then waits for the API on top of it (6:30 total). Fixed lets the API’s turnaround overlap the window’s own tail instead of stacking after it (5:00 total).

The session doesn’t pause while the API is thinking. While chunk N sits with the speech API, the session is already generating chunk N+1’s audio in real time, for free. The sleep just needed to know that:

sleepMs = max(0, windowSize - elapsed)

elapsed is how long it’s been since this window started. The loop only sleeps whatever’s left after subtracting what the API’s turnaround already covered, instead of a flat five minutes stacked on top of it regardless.

The bit I actually got a kick out of came later, thinking about what that same line does when the API is having a bad day. If it falls behind and a chunk takes longer than the window itself, elapsed blows past windowSize, the formula clamps to zero, and the loop just stops sleeping and fires the next chunk immediately, catching itself back up without a single line of “are we behind” logic anywhere. One formula, and the backlog handling fell out of it for free. I hadn’t designed that on purpose; I noticed it while trying to convince myself the fix was correct.

There was a second, smaller problem hiding under all this: the worker doing the chunking and the backend receiving the callback are two different processes, so the worker has no direct way to know “did my last chunk actually land” before it dares start the next window. The next window’s start offset depends on the confirmed position, not a guess. I got as far as opening a new file for a status-check endpoint before I remembered the backend already had a claim call for handing out work, built on a plain compare-and-swap. Polling that did the whole job. Chunk still in flight, the claim just fails. Chunk lands, the claim succeeds, and that success is simultaneously “yes, done” and “here’s your next window.” I deleted the new file. The existing claim call already did the job, and it had already been through code review, so there was nothing left to build.

Shipped it behind a setting too, off by default. It’s paid compute per live event, and I’d rather someone turn it on than have it turn itself on for them.

The on-demand gap bug

Then someone asked a smaller question than the loop was built for. Not “keep transcribing this whole session,” but “what did the speaker say around the 12-minute mark, right now.” The rolling loop only knows how to say “I’ll get to everything eventually.” It has no way to answer “give me this one minute, fast.”

The lazy fix was shrinking the rolling loop’s window to sixty seconds too, so it would cycle faster. That made the same drift problem worse. At sixty seconds, the loop gets less audio done per cycle than it spends waiting on the speech API. So it falls further behind every cycle and never catches up. That’s just how a loop works: each window chains onto the last one. A one-off request isn’t a loop, so it doesn’t need to keep pace with anything. It got its own path instead: a single sixty-second job, fired whenever someone asks for it, no cadence to fall behind on. Same sixty-to-ninety-second turnaround as any other chunk. So instead of waiting up to five minutes for the loop to arrive, the answer’s back in about a minute.

Then adjacent requests started leaving a gap that shouldn’t have existed. I reused the rolling loop’s own code for these one-off jobs: figure out which chunks of the session’s stream cover the requested range, hand those straight to the speech API. Seemed free. It wasn’t. Ask for two minutes back to back, and read the transcript straight through, and there’d be a few seconds of dead air right where they meet, every time, in a spot where someone was clearly talking.

Here’s why: the code that decides where ffmpeg cuts never grabs half a segment. It always rounds backward to the last full one. That’s harmless for the rolling loop, since each window just starts wherever the last one actually ended, there’s no separate target it could fall short of. A one-off request doesn’t have that safety net. Its start and end both come straight from the clock. So when two people request adjacent minutes, each one rounds backward from the same shared boundary:

gantt
    dateFormat HH:mm:ss
    axisFormat %M:%S
    section Requested
    minute 3         :r1, 00:03:00, 60s
    minute 4         :r2, 00:04:00, 60s
    section Actual
    minute 3         :a1, 00:03:00, 52s
    gap              :active, g1, 00:03:52, 12s
    minute 4         :a2, 00:04:04, 56s

Requested windows sit exactly adjacent. Actual extraction rounds backward from the shared boundary on both sides, leaving a sliver neither request claims.

Worse, both sides still get marked “done, covered.” A repeat request for either minute sees “already have that” and skips it. You’d catch dead air just by scanning the transcript. You wouldn’t catch a gap hiding inside a range the system already claims is finished. Nothing tells you to go looking for it.

The fix took two moves, not the one-line trick the sleep formula got away with. First: make the end round forward instead of backward, past the boundary into the next segment. Now both edges land outside the requested range, not just the start. Second, clamp it back down: keep only what falls inside the exact requested minute, throw away the rest. A little audio gets transcribed and thrown away on every request. Once I’d seen the alternative, a hole nobody could even ask to fix, the wasted audio was an easy trade.

Neither fix was clever. The first needed me to nail down what was actually being asked before opening a design doc. The second needed me to notice something simple: code built for a loop that chains its own edges together breaks once something else calls it on its own. I needed to catch that before a customer did. It’s been running quietly in production for a few months now, still just answering “what did they say around minute twelve,” a smaller ambition than a real-time pipeline but the actual job the whole time.