How We Built Summaries That Finish Before You Do
Liam Novak
Engineering Lead · Metrik
Inside the system that turns 45 minutes of talk into a crisp summary — in the same moment your call ends.
There's a moment in every demo where we show a summary that appeared before the host clicked 'End call.' People assume it's staged. It isn't — and making it real was one of our hardest engineering problems.
The naive approach is to transcribe the whole meeting, then summarize. That works, but it's slow: a 60-minute meeting takes minutes to transcribe, and summaries lag by minutes — an eternity when someone is already heading to the next call.
Our pipeline is incremental. Audio streams through an ASR model that emits segments as they stabilize. A lightweight stage-1 summarizer keeps a running sketch of decisions, speakers, and topics in memory.
When the call ends, we do a final pass: a stage-2 model reconciles the running sketch with the full transcript, resolves open questions, and extracts action items with owner detection from the speaker diarization.
The whole pipeline adds under 40 seconds of latency on a 60-minute meeting. Speaker diarization runs in parallel streams — one per voice — so the sketch is always current.
The hardest bug was 'phantom owners' — assigning action items to a listener who never spoke. We now weight speaker proximity, sentence form, and past assignment history before committing an owner.
The Monthly Metrik
Enjoyed this? Get one email a month with meeting research, product updates, and a genuinely useful tip.