2. Whisper (OpenAI)

Description

Whisper is OpenAI's transcription model. It supports 50+ languages, detects the language automatically, generates timestamps at the segment and word level, and produces output in multiple formats (plain text, JSON, SRT, VTT). In this capsule you'll master the complete API: transcription, translation, response formats, handling large files, batch processing, and cost optimizations.

Why it matters: Whisper is the industry standard for transcription via API. It's the STT component you'll use in the Audio Pipeline of the final project, and it's the provider against which all the alternatives in capsule 03 are compared. Mastering its parameters and limitations lets you implement robust transcription in any system.

Connection with the project: The Audio Pipeline (capsule 08) uses Whisper as its transcription engine. Everything you learn here — formats, chunking, error handling — applies directly to the project.

Key Concepts

Available model

OpenAI offers a single model through the API: whisper-1. Internally it's Whisper large-v2 optimized for production. You can't choose between tiny/small/medium/large models like in open-source Whisper.

Accepted audio formats

Limits

Basic Transcription

Transcription API

from openai import OpenAI

client = OpenAI()

def transcribe(audio_path: str, language: str = None) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language=language
        )
    return response.text

The language parameter is optional. If you omit it, Whisper detects the language automatically. Specifying it improves accuracy and reduces latency:

text_es = transcribe("meeting_es.mp3", language="es")

text_en = transcribe("meeting_en.mp3", language="en")

text_auto = transcribe("unknown_lang.mp3")

Language detection

Whisper detects the audio's language automatically. To get the detected language, use verbose_json:

def detect_language(audio_path: str) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json"
        )
    return {
        "language": response.language,
        "text": response.text
    }

Response Formats

Whisper supports 5 response formats. Each one has a specific use case:

Format comparison

text format — Text only

def transcribe_text(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="text"
        )
    return response

json format — Standard response

def transcribe_json(audio_path: str) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="json"
        )
    return response

srt format — Subtitles

def transcribe_srt(audio_path: str, output_path: str = "subtitles.srt") -> str:
    with open(audio_path, "rb") as f:
        srt_content = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="srt"
        )
    with open(output_path, "w", encoding="utf-8") as f:
        f.write(srt_content)
    return output_path

Example SRT output:

1
00:00:00,000 --> 00:00:04,500
Welcome to the team's weekly meeting.

2
00:00:04,500 --> 00:00:09,200
Today's main topic is the launch of version 2.0.

vtt format — WebVTT

def transcribe_vtt(audio_path: str, output_path: str = "subtitles.vtt") -> str:
    with open(audio_path, "rb") as f:
        vtt_content = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="vtt"
        )
    with open(output_path, "w", encoding="utf-8") as f:
        f.write(vtt_content)
    return output_path

verbose_json format — Detailed analysis

def transcribe_verbose(audio_path: str) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json"
        )
    return {
        "language": response.language,
        "duration": response.duration,
        "text": response.text,
        "segments": [
            {
                "start": seg.start,
                "end": seg.end,
                "text": seg.text
            }
            for seg in response.segments
        ]
    }

Word-Level Timestamps

Whisper can generate timestamps for each individual word. This is useful for precise synchronization, karaoke-style subtitles, or speech-rate analysis:

def transcribe_word_timestamps(audio_path: str) -> list[dict]:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",
            timestamp_granularities=["word"]
        )
    return [
        {
            "word": word.word,
            "start": word.start,
            "end": word.end
        }
        for word in response.words
    ]

You can combine both granularities:

def transcribe_full_timestamps(audio_path: str) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",
            timestamp_granularities=["word", "segment"]
        )
    return {
        "segments": [
            {"start": s.start, "end": s.end, "text": s.text}
            for s in response.segments
        ],
        "words": [
            {"word": w.word, "start": w.start, "end": w.end}
            for w in response.words
        ]
    }

Translation: Audio in Any Language → Text in English

Whisper has a translation endpoint that transcribes audio in any language and translates it to English in a single step:

def translate_to_english(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.translations.create(
            model="whisper-1",
            file=f
        )
    return response.text

Difference between transcription and translation:

def transcribe_and_translate(audio_path: str) -> dict:
    with open(audio_path, "rb") as f_transcribe:
        original = client.audio.transcriptions.create(
            model="whisper-1",
            file=f_transcribe
        ).text

    with open(audio_path, "rb") as f_translate:
        translated = client.audio.translations.create(
            model="whisper-1",
            file=f_translate
        ).text

    return {
        "original": original,
        "english": translated
    }

Advanced Parameters

Prompt to guide the transcription

The prompt parameter guides the transcription style. Useful for technical terms, proper names, or punctuation style:

def transcribe_with_prompt(audio_path: str, prompt: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            prompt=prompt
        )
    return response.text

result = transcribe_with_prompt(
    "tech_meeting.mp3",
    prompt="Kubernetes, Docker, CI/CD, FastAPI, PostgreSQL, Redis"
)

Common cases for using prompt:

Temperature

Controls the variability of the transcription. Low values produce more deterministic output:

def transcribe_deterministic(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            temperature=0.0
        )
    return response.text

Handling Long Audio (>25MB)

Strategy: split with pydub

When the file exceeds 25MB, split it into chunks, transcribe each one, and concatenate the results:

from pydub import AudioSegment
from pathlib import Path
import tempfile

def transcribe_long_audio(
    audio_path: str,
    chunk_duration_ms: int = 10 * 60 * 1000,
    language: str = None
) -> dict:
    audio = AudioSegment.from_file(audio_path)
    total_duration_s = len(audio) / 1000

    chunks = [
        audio[i:i + chunk_duration_ms]
        for i in range(0, len(audio), chunk_duration_ms)
    ]

    transcripts = []

    with tempfile.TemporaryDirectory() as tmp_dir:
        for i, chunk in enumerate(chunks):
            chunk_path = Path(tmp_dir) / f"chunk_{i:03d}.mp3"
            chunk.export(str(chunk_path), format="mp3", bitrate="128k")

            with open(chunk_path, "rb") as f:
                response = client.audio.transcriptions.create(
                    model="whisper-1",
                    file=f,
                    language=language
                )
            transcripts.append(response.text)

    return {
        "text": " ".join(transcripts),
        "chunks": len(chunks),
        "total_duration_s": total_duration_s
    }

Prior validation

from pathlib import Path

WHISPER_MAX_SIZE = 25 * 1024 * 1024
WHISPER_FORMATS = {".mp3", ".mp4", ".mpeg", ".mpga", ".m4a", ".wav", ".webm"}

def validate_audio_for_whisper(audio_path: str) -> dict:
    path = Path(audio_path)

    if not path.exists():
        return {"valid": False, "error": "File not found"}

    if path.suffix.lower() not in WHISPER_FORMATS:
        return {
            "valid": False,
            "error": f"Format {path.suffix} not supported. Use: {WHISPER_FORMATS}"
        }

    size = path.stat().st_size
    if size > WHISPER_MAX_SIZE:
        return {
            "valid": False,
            "error": f"File of {size / 1024 / 1024:.1f}MB exceeds the 25MB limit",
            "needs_splitting": True
        }

    return {"valid": True, "size_mb": size / 1024 / 1024}

Complete function: validate + transcribe

def smart_transcribe(audio_path: str, language: str = None) -> dict:
    validation = validate_audio_for_whisper(audio_path)

    if not validation["valid"]:
        if validation.get("needs_splitting"):
            return transcribe_long_audio(audio_path, language=language)
        return {"error": validation["error"]}

    return {
        "text": transcribe(audio_path, language=language),
        "chunks": 1,
        "total_duration_s": None
    }

Batch Transcription

To process multiple audio files in batch:

from pathlib import Path
from concurrent.futures import ThreadPoolExecutor

def batch_transcribe(
    audio_dir: str,
    language: str = None,
    max_workers: int = 3
) -> list[dict]:
    audio_files = [
        p for p in Path(audio_dir).iterdir()
        if p.suffix.lower() in WHISPER_FORMATS
    ]

    def process_file(path: Path) -> dict:
        try:
            result = smart_transcribe(str(path), language=language)
            return {
                "file": path.name,
                "status": "success",
                **result
            }
        except Exception as e:
            return {
                "file": path.name,
                "status": "error",
                "error": str(e)
            }

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        results = list(executor.map(process_file, audio_files))

    return results

Costs

Whisper pricing

Cost estimation

def estimate_whisper_cost(audio_path: str) -> dict:
    audio = AudioSegment.from_file(audio_path)
    duration_min = len(audio) / 1000 / 60
    cost = duration_min * 0.006

    return {
        "duration_minutes": round(duration_min, 2),
        "estimated_cost_usd": round(cost, 4)
    }

Quick reference table

Troubleshooting

Problem 1: Unsupported format

Symptom: Invalid file format or Could not process audio.

Solution:

def convert_to_mp3(input_path: str, output_path: str = None) -> str:
    if output_path is None:
        output_path = Path(input_path).with_suffix(".mp3")
    audio = AudioSegment.from_file(input_path)
    audio.export(str(output_path), format="mp3", bitrate="128k")
    return str(output_path)

Problem 2: Inaccurate transcription

Causes and solutions:

def preprocess_for_quality(input_path: str, output_path: str) -> str:
    audio = AudioSegment.from_file(input_path)
    audio = audio.set_frame_rate(16000).set_channels(1)
    audio = audio.normalize()
    audio.export(output_path, format="mp3", bitrate="64k")
    return output_path

Problem 3: File too large

Symptom: Maximum content size limit exceeded or error 413.

Solution: Use transcribe_long_audio() from the previous section. Split into 10-minute chunks.

Problem 4: Rate limits

Symptom: Rate limit reached (HTTP 429).

Solution:

import time

def transcribe_with_retry(audio_path: str, max_retries: int = 3) -> str:
    for attempt in range(max_retries):
        try:
            return transcribe(audio_path)
        except Exception as e:
            if "rate_limit" in str(e).lower() and attempt < max_retries - 1:
                wait = 2 ** attempt
                time.sleep(wait)
                continue
            raise

Exercises

Exercise 1: Multilingual transcription with language detection

Create a function that transcribes audio, detects the language, and returns both. If the language is neither Spanish nor English, also include the translation to English.

def transcribe_multilingual(audio_path: str) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json"
        )

    result = {
        "language": response.language,
        "text": response.text,
        "duration": response.duration
    }

    if response.language not in ("es", "en"):
        with open(audio_path, "rb") as f:
            translation = client.audio.translations.create(
                model="whisper-1",
                file=f
            )
        result["english_translation"] = translation.text

    return result

Exercise 2: Generate SRT subtitles with a character limit per line

Transcribe audio in SRT format, but post-process it so that no subtitle line exceeds 42 characters (the TV standard).

def generate_formatted_srt(audio_path: str, max_chars: int = 42) -> str:
    with open(audio_path, "rb") as f:
        srt = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="srt"
        )

    formatted_blocks = []
    for block in srt.strip().split("\n\n"):
        lines = block.split("\n")
        if len(lines) >= 3:
            index = lines[0]
            timestamp = lines[1]
            text = " ".join(lines[2:])

            wrapped = []
            words = text.split()
            current_line = ""
            for word in words:
                if len(current_line) + len(word) + 1 <= max_chars:
                    current_line = f"{current_line} {word}".strip()
                else:
                    wrapped.append(current_line)
                    current_line = word
            if current_line:
                wrapped.append(current_line)

            formatted_blocks.append(f"{index}\n{timestamp}\n" + "\n".join(wrapped))

    return "\n\n".join(formatted_blocks)

Exercise 3: Transcription with speech-rate analysis

Use word-level timestamps to calculate words per minute (WPM) per 30-second segment.

def analyze_speech_rate(audio_path: str, window_seconds: float = 30.0) -> list[dict]:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",
            timestamp_granularities=["word"]
        )

    if not response.words:
        return []

    total_duration = response.words[-1].end
    windows = []
    window_start = 0.0

    while window_start < total_duration:
        window_end = window_start + window_seconds
        words_in_window = [
            w for w in response.words
            if w.start >= window_start and w.start < window_end
        ]
        actual_duration = min(window_seconds, total_duration - window_start)
        wpm = (len(words_in_window) / actual_duration) * 60 if actual_duration > 0 else 0

        windows.append({
            "start": round(window_start, 1),
            "end": round(min(window_end, total_duration), 1),
            "word_count": len(words_in_window),
            "wpm": round(wpm, 1)
        })
        window_start += window_seconds

    return windows

Exercise 4: Robust pipeline with validation, preprocessing and transcription

Create a pipeline that: validates the file, converts to MP3 if necessary, splits it if it exceeds 25MB, and transcribes with retry.

from pathlib import Path
from pydub import AudioSegment
import tempfile
import time

def robust_transcription_pipeline(
    audio_path: str,
    language: str = None,
    prompt: str = None,
    max_retries: int = 3
) -> dict:
    path = Path(audio_path)
    if not path.exists():
        return {"error": "File not found"}

    with tempfile.TemporaryDirectory() as tmp_dir:
        working_path = audio_path
        if path.suffix.lower() not in WHISPER_FORMATS:
            working_path = str(Path(tmp_dir) / "converted.mp3")
            AudioSegment.from_file(audio_path).export(
                working_path, format="mp3", bitrate="128k"
            )

        working_file = Path(working_path)
        needs_split = working_file.stat().st_size > WHISPER_MAX_SIZE

        if needs_split:
            audio = AudioSegment.from_file(working_path)
            chunk_ms = 10 * 60 * 1000
            chunks = [audio[i:i + chunk_ms] for i in range(0, len(audio), chunk_ms)]
        else:
            chunks = None

        transcripts = []

        def transcribe_single(file_path: str) -> str:
            for attempt in range(max_retries):
                try:
                    with open(file_path, "rb") as f:
                        kwargs = {"model": "whisper-1", "file": f}
                        if language:
                            kwargs["language"] = language
                        if prompt:
                            kwargs["prompt"] = prompt
                        return client.audio.transcriptions.create(**kwargs).text
                except Exception as e:
                    if attempt < max_retries - 1:
                        time.sleep(2 ** attempt)
                        continue
                    raise

        if chunks:
            for i, chunk in enumerate(chunks):
                chunk_path = str(Path(tmp_dir) / f"chunk_{i:03d}.mp3")
                chunk.export(chunk_path, format="mp3", bitrate="128k")
                transcripts.append(transcribe_single(chunk_path))
        else:
            transcripts.append(transcribe_single(working_path))

    return {
        "text": " ".join(transcripts),
        "chunks_processed": len(transcripts),
        "converted": path.suffix.lower() not in WHISPER_FORMATS,
        "was_split": needs_split
    }

Summary

• Whisper (whisper-1) is OpenAI's transcription model: 50+ languages, $0.006/min, 25MB limit.
• It supports 5 response formats: text, json, srt, vtt, verbose_json.
• Timestamps available at the segment and word level with verbose_json.
• The translation endpoint converts audio in any language to text in English.
• The prompt parameter improves accuracy for technical terms and proper names.
• For audio >25MB: split with pydub into 10-minute chunks and transcribe in parts.
• Batch transcription with ThreadPoolExecutor to process multiple files.
• Always validate format and size before sending to the API.

Additional Resources

1. Whisper API Reference — Complete API reference
2. Speech-to-Text Guide — Official OpenAI guide
3. Whisper Open Source — Open-source model (for local deploy)
4. pydub — Audio manipulation in Python
5. Supported Languages — Complete list of languages