3. Alternatives to Whisper

Description

Whisper isn't the only STT provider. Google Speech-to-Text, AssemblyAI and Deepgram offer capabilities Whisper doesn't have: speaker diarization, PII detection, real-time streaming, and models optimized for specific domains. In this capsule you'll compare the 4 main providers with working code, comparison tables and a decision tree to pick the right one for your use case.

Why it matters: Choosing the wrong provider can mean paying more for fewer features, or losing critical functionality like diarization. A developer who only knows Whisper is limited. With this capsule, you'll have the judgment to choose among 4 providers based on cost, accuracy, features and use case.

Connection with the project: The Audio Pipeline (capsule 08) uses Whisper by default, but you can extend it with fallback to other providers. The unified transcribe() function you'll build here lets you switch providers with a single parameter.

General Comparison

Comparison table

When is each provider the best option?

Google Speech-to-Text

Setup

pip install google-cloud-speech

# Configure GCP credentials
export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json"

Basic transcription

from google.cloud import speech

def transcribe_google(audio_path: str, language: str = "es-ES") -> str:
    client = speech.SpeechClient()

    with open(audio_path, "rb") as f:
        content = f.read()

    audio = speech.RecognitionAudio(content=content)
    config = speech.RecognitionConfig(
        encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
        sample_rate_hertz=16000,
        language_code=language,
        enable_automatic_punctuation=True,
    )

    response = client.recognize(config=config, audio=audio)
    return " ".join(
        result.alternatives[0].transcript
        for result in response.results
    )

Transcription with diarization

def transcribe_google_with_diarization(
    audio_path: str,
    language: str = "es-ES",
    num_speakers: int = 2
) -> list[dict]:
    client = speech.SpeechClient()

    with open(audio_path, "rb") as f:
        content = f.read()

    audio = speech.RecognitionAudio(content=content)
    diarization_config = speech.SpeakerDiarizationConfig(
        enable_speaker_diarization=True,
        min_speaker_count=num_speakers,
        max_speaker_count=num_speakers,
    )
    config = speech.RecognitionConfig(
        encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
        sample_rate_hertz=16000,
        language_code=language,
        diarization_config=diarization_config,
    )

    response = client.recognize(config=config, audio=audio)
    result = response.results[-1]

    utterances = []
    current_speaker = None
    current_text = []

    for word in result.alternatives[0].words:
        if word.speaker_tag != current_speaker:
            if current_text:
                utterances.append({
                    "speaker": f"Speaker {current_speaker}",
                    "text": " ".join(current_text)
                })
            current_speaker = word.speaker_tag
            current_text = [word.word]
        else:
            current_text.append(word.word)

    if current_text:
        utterances.append({
            "speaker": f"Speaker {current_speaker}",
            "text": " ".join(current_text)
        })

    return utterances

Long audio with Google (>1 min)

For audio longer than 1 minute, Google requires uploading the file to GCS:

from google.cloud import speech, storage

def transcribe_google_long(
    audio_path: str,
    gcs_bucket: str,
    language: str = "es-ES"
) -> str:
    storage_client = storage.Client()
    bucket = storage_client.bucket(gcs_bucket)
    blob_name = f"audio/{Path(audio_path).name}"
    blob = bucket.blob(blob_name)
    blob.upload_from_filename(audio_path)
    gcs_uri = f"gs://{gcs_bucket}/{blob_name}"

    client = speech.SpeechClient()
    audio = speech.RecognitionAudio(uri=gcs_uri)
    config = speech.RecognitionConfig(
        encoding=speech.RecognitionConfig.AudioEncoding.MP3,
        sample_rate_hertz=16000,
        language_code=language,
        enable_automatic_punctuation=True,
    )

    operation = client.long_running_recognize(config=config, audio=audio)
    response = operation.result(timeout=600)

    blob.delete()

    return " ".join(
        result.alternatives[0].transcript
        for result in response.results
    )

Google STT costs

AssemblyAI

Setup

pip install assemblyai

export ASSEMBLYAI_API_KEY="your_key"

Basic transcription

import assemblyai as aai

aai.settings.api_key = os.getenv("ASSEMBLYAI_API_KEY")

def transcribe_assemblyai(audio_path: str, language: str = "es") -> str:
    config = aai.TranscriptionConfig(language_code=language)
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(audio_path, config=config)

    if transcript.status == aai.TranscriptStatus.error:
        raise RuntimeError(f"Transcription error: {transcript.error}")

    return transcript.text

Speaker diarization

def transcribe_assemblyai_diarization(audio_path: str) -> list[dict]:
    config = aai.TranscriptionConfig(
        speaker_labels=True,
        language_code="es"
    )
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(audio_path, config=config)

    return [
        {
            "speaker": utterance.speaker,
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end
        }
        for utterance in transcript.utterances
    ]

PII detection (Personal Information)

def transcribe_assemblyai_redact_pii(audio_path: str) -> dict:
    config = aai.TranscriptionConfig(
        redact_pii=True,
        redact_pii_policies=[
            aai.PIIRedactionPolicy.person_name,
            aai.PIIRedactionPolicy.phone_number,
            aai.PIIRedactionPolicy.email_address,
            aai.PIIRedactionPolicy.credit_card_number,
        ],
        redact_pii_sub=aai.PIISubstitutionPolicy.hash,
    )
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(audio_path, config=config)

    return {
        "text": transcript.text,
        "redacted": True
    }

Automatic summary and analysis

def transcribe_assemblyai_with_analysis(audio_path: str) -> dict:
    config = aai.TranscriptionConfig(
        summarization=True,
        summary_model=aai.SummarizationModel.informative,
        summary_type=aai.SummarizationType.bullets,
        sentiment_analysis=True,
        auto_chapters=True,
    )
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(audio_path, config=config)

    return {
        "text": transcript.text,
        "summary": transcript.summary,
        "sentiment": [
            {
                "text": s.text,
                "sentiment": s.sentiment.value,
                "confidence": s.confidence
            }
            for s in (transcript.sentiment_analysis or [])
        ],
        "chapters": [
            {
                "headline": ch.headline,
                "summary": ch.summary,
                "start": ch.start,
                "end": ch.end
            }
            for ch in (transcript.chapters or [])
        ]
    }

AssemblyAI costs

Deepgram

Setup

pip install deepgram-sdk

export DEEPGRAM_API_KEY="your_key"

Basic transcription

from deepgram import DeepgramClient, PrerecordedOptions, FileSource

def transcribe_deepgram(audio_path: str, language: str = "es") -> str:
    deepgram = DeepgramClient(os.getenv("DEEPGRAM_API_KEY"))

    with open(audio_path, "rb") as f:
        buffer_data = f.read()

    payload: FileSource = {"buffer": buffer_data}
    options = PrerecordedOptions(
        model="nova-2",
        language=language,
        smart_format=True,
        punctuate=True,
    )

    response = deepgram.listen.rest.v("1").transcribe_file(payload, options)
    return response.results.channels[0].alternatives[0].transcript

Diarization with Deepgram

def transcribe_deepgram_diarization(audio_path: str) -> list[dict]:
    deepgram = DeepgramClient(os.getenv("DEEPGRAM_API_KEY"))

    with open(audio_path, "rb") as f:
        buffer_data = f.read()

    payload: FileSource = {"buffer": buffer_data}
    options = PrerecordedOptions(
        model="nova-2",
        language="es",
        smart_format=True,
        diarize=True,
        utterances=True,
    )

    response = deepgram.listen.rest.v("1").transcribe_file(payload, options)

    return [
        {
            "speaker": utt.speaker,
            "text": utt.transcript,
            "start": utt.start,
            "end": utt.end
        }
        for utt in response.results.utterances
    ]

Deepgram costs

Decision Tree

Do you need STT?
│
├── Is it simple transcription with no extra features?
│   ├── Yes → Do you already use OpenAI? → Yes → Whisper
│   │                                     → No → Deepgram (cheaper)
│   └── No → continue ↓
│
├── Do you need to know WHO said WHAT (diarization)?
│   ├── Yes → Do you also need sentiment analysis?
│   │        ├── Yes → AssemblyAI
│   │        └── No → Deepgram (faster and cheaper)
│   └── No → continue ↓
│
├── Do you need to redact personal information (PII)?
│   ├── Yes → AssemblyAI
│   └── No → continue ↓
│
├── Do you need 100+ languages?
│   ├── Yes → Google STT (125+) or AssemblyAI (100+)
│   └── No → continue ↓
│
├── Do you need real-time streaming?
│   ├── Yes → Deepgram (lowest latency) or Google STT
│   └── No → continue ↓
│
├── Do you need automatic translation to English?
│   ├── Yes → Whisper (the only one with built-in translation)
│   └── No → continue ↓
│
└── Is the priority the lowest cost?
    ├── Yes → Deepgram ($0.0043/min)
    └── No → Whisper (integrated OpenAI ecosystem)

Unified Multi-Provider Function

from enum import Enum

class STTProvider(Enum):
    WHISPER = "whisper"
    GOOGLE = "google"
    ASSEMBLYAI = "assemblyai"
    DEEPGRAM = "deepgram"

def transcribe_unified(
    audio_path: str,
    provider: STTProvider = STTProvider.WHISPER,
    language: str = "es",
    diarize: bool = False
) -> dict:
    if provider == STTProvider.WHISPER:
        client = OpenAI()
        with open(audio_path, "rb") as f:
            response = client.audio.transcriptions.create(
                model="whisper-1",
                file=f,
                language=language
            )
        return {"text": response.text, "provider": "whisper"}

    elif provider == STTProvider.ASSEMBLYAI:
        config = aai.TranscriptionConfig(
            language_code=language,
            speaker_labels=diarize
        )
        transcript = aai.Transcriber().transcribe(audio_path, config=config)
        result = {"text": transcript.text, "provider": "assemblyai"}
        if diarize and transcript.utterances:
            result["utterances"] = [
                {"speaker": u.speaker, "text": u.text}
                for u in transcript.utterances
            ]
        return result

    elif provider == STTProvider.DEEPGRAM:
        dg = DeepgramClient(os.getenv("DEEPGRAM_API_KEY"))
        with open(audio_path, "rb") as f:
            payload = {"buffer": f.read()}
        options = PrerecordedOptions(
            model="nova-2",
            language=language,
            diarize=diarize
        )
        response = dg.listen.rest.v("1").transcribe_file(payload, options)
        return {
            "text": response.results.channels[0].alternatives[0].transcript,
            "provider": "deepgram"
        }

    elif provider == STTProvider.GOOGLE:
        return {
            "text": transcribe_google(audio_path, language=f"{language}-ES"),
            "provider": "google"
        }

    raise ValueError(f"Unsupported provider: {provider}")

Transcription with Multi-Provider Fallback

def transcribe_with_fallback(
    audio_path: str,
    providers: list[STTProvider] = None,
    language: str = "es"
) -> dict:
    if providers is None:
        providers = [STTProvider.WHISPER, STTProvider.DEEPGRAM, STTProvider.ASSEMBLYAI]

    errors = []
    for provider in providers:
        try:
            result = transcribe_unified(audio_path, provider=provider, language=language)
            result["fallback_used"] = provider != providers[0]
            result["attempts"] = len(errors) + 1
            return result
        except Exception as e:
            errors.append({"provider": provider.value, "error": str(e)})

    return {
        "error": "All providers failed",
        "details": errors
    }

Troubleshooting

Problem 1: Google STT requires LINEAR16 format

Symptom: Invalid audio format with Google.

Solution: Convert to WAV 16kHz mono:

def prepare_for_google(input_path: str) -> str:
    output_path = Path(input_path).with_suffix(".wav")
    audio = AudioSegment.from_file(input_path)
    audio = audio.set_frame_rate(16000).set_channels(1).set_sample_width(2)
    audio.export(str(output_path), format="wav")
    return str(output_path)

Problem 2: AssemblyAI timeout on long files

Symptom: The transcription takes longer than expected.

Solution: AssemblyAI processes asynchronously. The SDK waits by default, but you can configure polling:

config = aai.TranscriptionConfig(language_code="es")
transcriber = aai.Transcriber()
transcript = transcriber.transcribe(
    audio_path,
    config=config,
)

Problem 3: Deepgram doesn't recognize the language

Symptom: Transcription in the wrong language.

Solution: Specify the language explicitly with the correct code:

options = PrerecordedOptions(
    model="nova-2",
    language="es",
    detect_language=False
)

Exercises

Exercise 1: Provider benchmark

Create a function that transcribes the same audio with Whisper and a second provider, and compares the results.

import time

def benchmark_providers(
    audio_path: str,
    providers: list[STTProvider] = None
) -> list[dict]:
    if providers is None:
        providers = [STTProvider.WHISPER, STTProvider.DEEPGRAM]

    results = []
    for provider in providers:
        start = time.time()
        try:
            result = transcribe_unified(audio_path, provider=provider)
            elapsed = time.time() - start
            results.append({
                "provider": provider.value,
                "text": result["text"],
                "latency_s": round(elapsed, 2),
                "word_count": len(result["text"].split()),
                "char_count": len(result["text"]),
                "status": "success"
            })
        except Exception as e:
            results.append({
                "provider": provider.value,
                "status": "error",
                "error": str(e),
                "latency_s": round(time.time() - start, 2)
            })

    return results

Exercise 2: Transcription with formatted diarization

Use AssemblyAI or Deepgram to transcribe a conversation and format the output as a dialogue script.

def transcribe_as_dialogue(audio_path: str) -> str:
    config = aai.TranscriptionConfig(
        speaker_labels=True,
        language_code="es"
    )
    transcript = aai.Transcriber().transcribe(audio_path, config=config)

    if not transcript.utterances:
        return transcript.text

    dialogue_lines = []
    for utterance in transcript.utterances:
        start_min = utterance.start // 60000
        start_sec = (utterance.start % 60000) // 1000
        timestamp = f"[{start_min:02d}:{start_sec:02d}]"
        dialogue_lines.append(
            f"{timestamp} Speaker {utterance.speaker}: {utterance.text}"
        )

    return "\n\n".join(dialogue_lines)

Exercise 3: Automatic provider selector

Create a function that analyzes the user's requirements and recommends the best provider.

def recommend_provider(
    needs_diarization: bool = False,
    needs_pii_redaction: bool = False,
    needs_streaming: bool = False,
    needs_translation: bool = False,
    budget_priority: bool = False,
    language: str = "es"
) -> dict:
    scores = {
        "whisper": 0,
        "google": 0,
        "assemblyai": 0,
        "deepgram": 0
    }

    if needs_translation:
        scores["whisper"] += 10

    if needs_diarization:
        scores["assemblyai"] += 5
        scores["deepgram"] += 5
        scores["google"] += 3

    if needs_pii_redaction:
        scores["assemblyai"] += 10

    if needs_streaming:
        scores["deepgram"] += 5
        scores["google"] += 5

    if budget_priority:
        scores["deepgram"] += 5
        scores["google"] += 2

    if not any([needs_diarization, needs_pii_redaction, needs_streaming, needs_translation]):
        scores["whisper"] += 5

    recommended = max(scores, key=scores.get)

    reasons = {
        "whisper": "Best general option and the only one with built-in translation",
        "google": "Best for streaming and the widest language coverage",
        "assemblyai": "Best for diarization, PII, and advanced analysis",
        "deepgram": "Best cost-performance ratio and lowest latency"
    }

    return {
        "recommended": recommended,
        "reason": reasons[recommended],
        "scores": scores
    }

Summary

• 4 main STT providers: Whisper, Google STT, AssemblyAI, Deepgram.
• Whisper is the default option: simple, good quality, OpenAI ecosystem, the only one with built-in translation.
• Google STT stands out in languages (125+), streaming, and GCP integration.
• AssemblyAI is the leader in advanced features: diarization, PII redaction, sentiment analysis, automatic summaries.
• Deepgram has the lowest latency and the best price ($0.0043/min).
• Use the unified function to switch providers with a single parameter.
• Implement multi-provider fallback for resilience in production.

Additional Resources

1. Google Speech-to-Text — Google's official documentation
2. AssemblyAI Docs — Complete AssemblyAI documentation
3. Deepgram Docs — Deepgram documentation
4. AssemblyAI Python SDK — Official SDK
5. Deepgram Python SDK — Official SDK