Skriptly transcription that knows when it is wrong

The pipeline

Audio lands on a Flask API and is processed by serverless GPU workers on Modal running faster-whisper for the words and pyannote for who spoke. Those two models disagree about boundaries by design — one segments on speech, the other on speakers — so a word-level alignment step merges them rather than trusting either.

Long recordings are the hard case. A three-hour call is planned into chunks cut at silences instead of fixed offsets, transcribed in parallel across GPUs, and then stitched — with speaker embeddings clustered across chunk boundaries, so the person who is “Speaker 2” in the first chunk is still Speaker 2 in the ninth.

Fighting the model’s failure modes

Whisper hallucinates in predictable places: silence, music, and its own repetition loops. The pipeline handles each one specifically — voice activity detection before transcription, a compression-ratio filter on the output, and a collapse step for repeated n-grams. The core of that logic is under unit tests, because it is the part where a regression is invisible in a demo.

An LLM correction layer then fixes phonetic errors that are obvious in context — names, product terms, acronyms — and feeds a per-user vocabulary back into the next transcription’s prompt, so a term corrected once is recognised directly the next time.

For clients who cannot let transcripts leave their infrastructure there is a privacy mode: summarisation runs map-reduce over a self-hosted open-weight model instead of a hosted API.

What I learned shipping it

A production capture fault lost remote speakers on headphone setups — the recording sounded fine and was missing half the conversation. It was found by a user, not by me, which is the part worth remembering.

So quality stopped being an impression. Recordings are retained deliberately, and there is an evaluation set measuring word error rate and how often output gets deleted — the two numbers that say whether a change helped. Before that, every tuning decision was a guess with a confident story attached.