Local Meeting Scribe: Solving the Cross-Meeting Speaker Diarization Gap

An air-gapped, asynchronous post-meeting intelligence engine on Apple Silicon with persistent speaker voiceprints, acoustic sentiment analysis, and zero cloud token costs.

The Problem Space

Context

Modern product leadership and AI strategy involve hours of dense, high-stakes discussions every week: executive roadmap negotiations, architecture design reviews, and confidential board alignments. While raw automatic speech recognition (ASR) like OpenAI Whisper has solved speech-to-text, turning raw audio into high-conviction decision intelligence presents three fundamental challenges: identity continuity, data privacy & token economics, and true acoustic sentiment.

The Core Friction

1. The Inter-Meeting Identity Reset: Off-the-shelf diarizers (raw PyAnnote pipelines or cloud APIs) treat every meeting as an isolated sandbox. They generate ephemeral, anonymous labels (SPEAKER_00, SPEAKER_01) that scramble between recordings. Re-listening to audio snippets to manually re-label 8 stakeholders after every single meeting creates an unsustainable cognitive tax.
2. The Cloud Privacy Risk & Recurring Token Tax: Sending 2-hour multi-speaker meetings to third-party cloud APIs exposes confidential strategy audio to vendor retention pipelines. Furthermore, feeding massive 30,000–60,000 token raw transcripts into commercial LLMs incurs steep recurring costs that scale linearly with meeting volume.
3. The Real-Time Distraction vs. Deep Post-Meeting Need: Most market tools push for "real-time" streaming transcription—introducing awkward bots lurking on video calls that distract participants and sacrifice diarization accuracy. High-level product strategy does not require real-time captions; it requires deep, asynchronous post-meeting processing ("slow inference") that can thoroughly evaluate what transpired without eating workstation resources during the workday.
4. Shallow Text-Only Summaries Miss Acoustic Sentiment: Evaluating meetings purely on text transcriptions creates severe blind spots. Text strips out vocal cadence, pitch variation, hesitation pauses, interruptions, and acoustic energy. A flat text summary reads "I agree with this roadmap" as full alignment, missing the hesitant cadence and downward vocal inflection that signal deep organizational reservation.

Critical User Journeys (CUJs)

CUJ 1: Persistent Voice Enrollment & Cross-Meeting Speaker Continuity

• Statement: I want to attribute speech to enrolled team members automatically across recurring meetings while avoiding the need to manually identify and re-name anonymous speaker labels every time so that transcripts and action items immediately display correct speaker names with zero manual friction.
• User Scenario: I finish an executive roadmap review with our VP of Engineering and Lead Architect. When I open the local transcription dashboard, the system automatically matches their vocal embeddings against our stored voice library. 90%+ of speaker turns are pre-labeled with high confidence, eliminating manual re-labeling overhead.

CUJ 2: Air-Gapped Privacy, Zero Token Costs & Asynchronous "Slow Inference"

• Statement: I want to process multi-hour confidential meetings offline on Apple Silicon while avoiding expensive third-party cloud LLM token fees and real-time meeting bot distractions so that deep, high-precision batch inference runs reliably in the background without interrupting my workday.
• User Scenario: I drop a 2-hour confidential board recording into the processing queue before heading home. The system schedules the heavy CPU-bound diarization and local LLM analysis for off-hours via a background LaunchAgent daemon, holding off sleep via caffeinate. Zero audio bytes leave the machine, and zero API token costs are incurred.

CUJ 3: Deep Acoustic Sentiment & Conversational Dynamics Analysis

• Statement: I want to evaluate meeting dynamics through raw acoustic vocal signals (cadence, pitch, hesitation, and cross-talk) alongside transcript text while avoiding shallow, text-only summaries so that true stakeholder sentiment and unvoiced reservations are accurately captured.
• User Scenario: During a contentious architectural tradeoff discussion, the acoustic sentiment module flags two segments where an engineering lead's pitch variance dropped and response latency increased before agreeing. The generated post-meeting intelligence highlights this as an area of underlying friction requiring follow-up.

The Ideal Flows

Flow 1: Ingestion, Multi-Stage Execution & One-Click Speaker Enrollment

1. Trigger: User drops an audio file into the local web UI (uv run scribe review) or schedules it via CLI.
2. Action: The engine executes an atomic 5-stage processing pipeline on Apple Silicon:
   • Stage 1 (Ingest): Extracts 16kHz mono audio and validates file headers.
   • Stage 2 (Transcribe): Generates timestamped word-level transcripts via local Whisper.
   • Stage 3 (Diarize): Clusters acoustic speaker turns via PyAnnote.
   • Stage 4 (Merge): Aligns word timestamps with acoustic speaker boundaries.
   • Stage 5 (Identify): Extracts 256-dimensional acoustic embeddings (WeSpeaker) and computes cosine similarity against the local voice library.
3. Automated Response: The UI presents the speaker panel with enrollment suggestions, confidence scores, and an "Enrol Voice" action for any newly recognized team member.
4. Resolution: Future meetings recognize the newly enrolled team member with zero manual configuration.

Flow 2: Asynchronous Post-Meeting Acoustic Sentiment & Synthesis

1. Trigger: Processing completes in the background (or during scheduled overnight execution).
2. Action: The local analysis engine (uv run scribe analyze --llm local) extracts acoustic signals (speech rate, hesitation intervals, speaker overlap) and correlates them with the transcript.
3. Automated Response: The local LLM generates a multi-dimensional briefing: verified speaker-attributed action items, consensus confidence ratings, and acoustic sentiment highlights.
4. Resolution: Product leadership receives actionable meeting intelligence that reflects both what was said and how it was received, with 100% data privacy.

The Solution: Local Meeting Scribe Architecture

graph TD
    subgraph Audio Ingest & Offline Pipeline
        A["Meeting Audio (.mp4, .m4a, .wav)"] --> B["Atomic Stage Orchestrator"]
        B --> C1["Local Whisper (Word-Level ASR)"]
        B --> C2["PyAnnote (Acoustic Diarization)"]
        B --> C3["Acoustic Feature Extractor (Pitch, Cadence, Energy)"]
    end

    subgraph Biometric Identity Engine
        C2 --> D["WeSpeaker Embedding Extractor"]
        D -->|256d Vector| E["Cosine Similarity Matcher"]
        VLib[("Persistent Local Voice Library (SQLite/Vectors)")] <--> E
    end

    subgraph Synthesis & Deep Intelligence
        C1 --> F["Timestamp Alignment & Turn Merging"]
        E --> F
        C3 --> G["Multimodal Sentiment Synthesizer"]
        F --> G
        G --> H["Local LLM Analysis (Ollama / MLX)"]
        H --> I["Structured Named Transcript & Executive Insights"]
    end

Technical Highlights & Engineering Design

1. The "Slow Inference" Philosophy: Depth Over Real-Time Gimmicks

Real-time streaming transcription tools sacrifice diarization precision to meet sub-second latency targets, resulting in misattributed turns and noisy text. Local Meeting Scribe is intentionally engineered for deep asynchronous post-meeting intelligence:

• High-Precision Multi-Pass Clustering: Uses full-context agglomerative clustering across the entire recording rather than sliding window approximations.
• Background Daemon Execution: Integrates with macOS LaunchAgents (~/Library/LaunchAgents) to queue heavy CPU/GPU tasks for off-hours without keeping open terminal windows.
• caffeinate Sleep Prevention: Automatically holds off system idle sleep during active runs and safely releases resources upon completion.

2. Zero-Egress Privacy & Local Token Economics

By eliminating third-party cloud APIs (Whisper API, GPT-4, Claude), the architecture delivers crucial enterprise advantages:

• Zero Audio Egress: Strategic roadmap conversations, executive compensation reviews, and proprietary code architecture discussions never touch external networks.
• Zero Token Cost: Processing 50,000 tokens of raw multi-speaker transcript text per meeting costs $0.00 on local Apple Silicon hardware.

3. Acoustic Sentiment & Meeting Dynamics

Unlike text-only LLM analysis, the acoustic intelligence layer evaluates non-verbal audio features:

• Hesitation & Turn-Taking Latency: Measures the delay between speaker handoffs to identify hesitation or immediate consensus.
• Pitch & Energy Modulation: Detects stress, disengagement, or high conviction across specific agenda topics.
• Interruption & Overlap Ratios: Quantifies conversational dominance and cross-talk to highlight healthy debate versus meeting friction.

4. Persistent Biometric Voice Library (WeSpeaker & ResNet34)

• 256-Dimensional Embeddings: Extracts speaker vectors from clean speech turns using wespeaker-voxceleb-resnet34-LM.
• Multi-Environment Clustering: Enrolling a speaker across multiple meetings naturally clusters varying room acoustics and microphone setups, increasing recognition accuracy over time.

Systems Impact & Value

• Zero Cloud Latency & Zero Token Costs: 100% air-gapped on Apple Silicon with zero marginal cost per meeting.
• Inter-Meeting Continuity: Eliminates 10–15 minutes of manual speaker naming per recording.
• Deep Acoustic Intelligence: Transcends flat text to uncover authentic team alignment, hesitation, and high-conviction commitments.