Self-hosted Whisper with speaker diarization: a practical setup
Plain Whisper gives you a transcript but no idea who said what. Adding diarization — and running it all locally — is what makes a self-hosted setup genuinely useful for meetings, interviews and calls.
The components
A working self-hosted pipeline needs three pieces:
- Transcription — Whisper (or WhisperX / faster-whisper for speed and word-level timestamps).
- Diarization — a separate model, commonly pyannote, that segments audio by speaker.
- Alignment — stitching speaker segments onto the transcript words so each line is labelled.
The hard parts (where DIY setups break)
- GPU memory (OOM): long files blow up VRAM during alignment. The fix is chunked alignment — process ~10-minute windows with overlap rather than the whole file at once.
- Model wrangling: pyannote needs a Hugging Face token and the right model versions; versions drift and break.
- Speed vs accuracy:
large-v3is accurate but slow on CPU — a GPU (or Apple Silicon MPS) makes it practical.
Output that’s actually usable
Aim for Markdown with YAML frontmatter — speakers, timestamps and topics as metadata — so the result is ready to push into a vector database for RAG, not just a wall of text.
Build it or use a product
You can assemble this yourself (budget a couple of weeks for production reliability), or use a tool that ships the whole pipeline locally. See the best on-prem transcription ranking — Meetily and NoParrot both run Whisper + diarization on your own hardware, and NoParrot adds chunked alignment and vector-DB output out of the box.
Frequently asked questions
Does Whisper do speaker diarization?
Plain Whisper does not — it transcribes speech but doesn't label who is speaking. Diarization needs a separate model (commonly pyannote); WhisperX combines Whisper with diarization and word-level alignment.
What do I need to run Whisper with diarization locally?
Three pieces: a transcription model (Whisper / WhisperX / faster-whisper), a diarization model (e.g. pyannote, which needs a Hugging Face token), and an alignment step that attaches speaker labels to the transcribed words.
Why do self-hosted diarization setups run out of GPU memory?
Long files blow up VRAM during alignment when processed whole. The fix is chunked alignment — process roughly 10-minute windows with overlap instead of the entire file at once.