7. Audio Troubleshooting

Description

Working with audio in AI has specific problems: incompatible formats, files that exceed limits, inaccurate transcriptions, mis-detected languages, rate limits, poor audio quality, TTS problems, and out-of-control costs. In this capsule we document the 10 most common problems with concrete solutions, build an automated diagnostic function, a preprocessing pipeline, and a quick reference table.

Why it matters: In production, audio errors are the most frustrating because they're silent: an incorrect format doesn't produce a clear stack trace, an inaccurate transcription doesn't raise an exception. Without a troubleshooting strategy, you spend hours debugging problems that have known solutions.

Connection with the project: The Audio Pipeline (capsule 08) integrates the diagnostic and preprocessing functions from this capsule. Every error documented here has its handler in the final pipeline.

Problem 1: Unsupported Audio Format

Symptom

Error: Invalid file format. Supported formats: flac, m4a, mp3, mp4, mpeg, mpga, oga, ogg, wav, webm

Or: Could not process audio file.

Cause

The file has a format not supported by Whisper (e.g., .aac, .wma, standalone .opus, .amr).

Solution

from pydub import AudioSegment
from pathlib import Path

WHISPER_FORMATS = {".mp3", ".mp4", ".mpeg", ".mpga", ".m4a", ".wav", ".webm", ".flac", ".ogg", ".oga"}

def convert_to_supported_format(
    input_path: str,
    target_format: str = "mp3",
    bitrate: str = "128k"
) -> str:
    path = Path(input_path)

    if path.suffix.lower() in WHISPER_FORMATS:
        return input_path

    output_path = str(path.with_suffix(f".{target_format}"))
    audio = AudioSegment.from_file(input_path)
    audio.export(output_path, format=target_format, bitrate=bitrate)
    return output_path

Common formats and how to convert them

Problem 2: File Exceeds 25MB

Symptom

Error: Maximum content size limit (26214400) exceeded

Or HTTP 413: Request Entity Too Large.

Cause

The file exceeds Whisper's 25MB API limit. Common with WAV files (uncompressed) or long recordings.

Solution: Compress first, split if necessary

def reduce_file_size(
    input_path: str,
    target_size_mb: float = 24.0
) -> str:
    path = Path(input_path)
    current_size_mb = path.stat().st_size / (1024 * 1024)

    if current_size_mb <= target_size_mb:
        return input_path

    audio = AudioSegment.from_file(input_path)

    output_path = str(path.with_name(f"{path.stem}_compressed.mp3"))
    audio = audio.set_channels(1).set_frame_rate(16000)
    audio.export(output_path, format="mp3", bitrate="64k")

    new_size_mb = Path(output_path).stat().st_size / (1024 * 1024)

    if new_size_mb <= target_size_mb:
        return output_path

    return None

def split_large_audio(
    input_path: str,
    max_size_mb: float = 24.0,
    chunk_duration_ms: int = 10 * 60 * 1000
) -> list[str]:
    audio = AudioSegment.from_file(input_path)
    chunks = []
    output_dir = Path(input_path).parent / "chunks"
    output_dir.mkdir(exist_ok=True)

    for i in range(0, len(audio), chunk_duration_ms):
        chunk = audio[i:i + chunk_duration_ms]
        chunk_path = str(output_dir / f"chunk_{i // chunk_duration_ms:03d}.mp3")
        chunk.export(chunk_path, format="mp3", bitrate="128k")

        if Path(chunk_path).stat().st_size > max_size_mb * 1024 * 1024:
            sub_chunks = split_large_audio(chunk_path, max_size_mb, chunk_duration_ms // 2)
            chunks.extend(sub_chunks)
        else:
            chunks.append(chunk_path)

    return chunks

Problem 3: Poor-Quality Transcription

Symptom

Transcribed text with frequent errors: wrong words, nonsensical sentences, misspelled names.

Causes and solutions

Preprocessing to improve quality

def preprocess_audio_for_quality(
    input_path: str,
    output_path: str = None
) -> dict:
    if output_path is None:
        output_path = str(Path(input_path).with_name(
            f"{Path(input_path).stem}_preprocessed.mp3"
        ))

    audio = AudioSegment.from_file(input_path)

    original_dbfs = audio.dBFS
    audio = audio.set_frame_rate(16000)
    audio = audio.set_channels(1)
    audio = audio.normalize()
    audio = audio.set_sample_width(2)

    audio.export(output_path, format="mp3", bitrate="64k")

    return {
        "output_path": output_path,
        "original_dbfs": round(original_dbfs, 2),
        "normalized_dbfs": round(audio.dBFS, 2),
        "duration_s": round(len(audio) / 1000, 2),
        "output_size_mb": round(Path(output_path).stat().st_size / (1024 * 1024), 2)
    }

Use a prompt for technical vocabulary

def transcribe_with_vocabulary(
    audio_path: str,
    vocabulary: list[str],
    language: str = "es"
) -> str:
    prompt = ", ".join(vocabulary)
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language=language,
            prompt=prompt
        )
    return response.text

result = transcribe_with_vocabulary(
    "tech_meeting.mp3",
    vocabulary=["Kubernetes", "Docker", "CI/CD", "FastAPI", "PostgreSQL", "microservices"]
)

Problem 4: Wrong Detected Language

Symptom

Whisper transcribes in a language different from the audio's. Common when the audio has a strong accent, words in another language, or multilingual fragments.

Solution

def transcribe_with_language_verification(
    audio_path: str,
    expected_language: str = "es"
) -> dict:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json"
        )

    detected = response.language
    text = response.text

    if detected != expected_language:
        with open(audio_path, "rb") as f:
            forced = client.audio.transcriptions.create(
                model="whisper-1",
                file=f,
                language=expected_language
            )
        return {
            "text": forced.text,
            "detected_language": detected,
            "forced_language": expected_language,
            "warning": f"Detected language ({detected}) differs from the expected one ({expected_language}). Forced {expected_language}."
        }

    return {
        "text": text,
        "detected_language": detected,
        "forced_language": None,
        "warning": None
    }

Problem 5: Rate Limits

Symptom

Error 429: Rate limit reached for whisper-1 in organization ...

Cause

Too many requests per minute. The limits depend on your account tier.

Solution: Retry with exponential backoff

import time
from functools import wraps

def with_retry(max_retries: int = 5, base_delay: float = 1.0):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    error_str = str(e).lower()
                    is_rate_limit = "rate_limit" in error_str or "429" in error_str

                    if not is_rate_limit or attempt == max_retries - 1:
                        raise

                    delay = base_delay * (2 ** attempt)
                    time.sleep(delay)
            raise RuntimeError(f"Failed after {max_retries} attempts")
        return wrapper
    return decorator

@with_retry(max_retries=5, base_delay=2.0)
def transcribe_safe(audio_path: str, language: str = "es") -> str:
    with open(audio_path, "rb") as f:
        return client.audio.transcriptions.create(
            model="whisper-1", file=f, language=language
        ).text

Rate limits per tier

Problem 6: Poor Audio Quality

Symptom

Distorted audio, with echo, irregular volume, or clipping (saturation).

Diagnosis

def diagnose_audio_quality(audio_path: str) -> dict:
    audio = AudioSegment.from_file(audio_path)

    issues = []

    if audio.dBFS < -35:
        issues.append({
            "issue": "Volume too low",
            "value": f"{audio.dBFS:.1f} dBFS",
            "fix": "Normalize audio"
        })

    if audio.dBFS > -3:
        issues.append({
            "issue": "Possible clipping (saturation)",
            "value": f"{audio.dBFS:.1f} dBFS",
            "fix": "Reduce the audio gain"
        })

    if audio.frame_rate < 16000:
        issues.append({
            "issue": "Low sample rate",
            "value": f"{audio.frame_rate} Hz",
            "fix": "Resample to 16000 Hz minimum"
        })

    if audio.channels > 1:
        issues.append({
            "issue": "Stereo audio (unnecessary for STT)",
            "value": f"{audio.channels} channels",
            "fix": "Convert to mono to reduce size"
        })

    duration_s = len(audio) / 1000
    size_mb = Path(audio_path).stat().st_size / (1024 * 1024)
    bitrate_kbps = (size_mb * 1024 * 8) / duration_s if duration_s > 0 else 0

    if bitrate_kbps < 32:
        issues.append({
            "issue": "Low bitrate",
            "value": f"{bitrate_kbps:.0f} kbps",
            "fix": "Over-compressed audio, use a higher bitrate"
        })

    return {
        "path": audio_path,
        "duration_s": round(duration_s, 2),
        "size_mb": round(size_mb, 2),
        "sample_rate": audio.frame_rate,
        "channels": audio.channels,
        "dbfs": round(audio.dBFS, 2),
        "bitrate_kbps": round(bitrate_kbps, 0),
        "issues": issues,
        "quality": "good" if not issues else "needs_attention"
    }

Problem 7: TTS — Inappropriate Voice or Artifacts

Symptom

The generated voice sounds robotic, has cuts, or the pronunciation is incorrect.

Solutions by case

Function to improve text before TTS

def prepare_text_for_tts(text: str) -> str:
    text = text.strip()

    if not text.endswith((".", "!", "?", "...")):
        text += "."

    text = text.replace(" - ", ", ")
    text = text.replace("•", ".")
    text = text.replace("\n\n", ". ")
    text = text.replace("\n", " ")

    import re
    text = re.sub(r'\s+', ' ', text)

    text = text.replace("$", " dollars ")
    text = text.replace("%", " percent")
    text = text.replace("&", " and ")

    return text.strip()

Problem 8: Out-of-Control Costs

Symptom

The API bill is much higher than expected.

Diagnosis

def estimate_pipeline_cost(
    audio_duration_min: float,
    transcript_chars: int = None,
    include_tts: bool = False,
    tts_model: str = "tts-1"
) -> dict:
    if transcript_chars is None:
        transcript_chars = int(audio_duration_min * 150 * 5)

    costs = {
        "whisper": round(audio_duration_min * 0.006, 4),
        "gpt-4o-mini_input": round(transcript_chars * 0.00000015, 6),
        "gpt-4o-mini_output": round(500 * 0.0000006, 6),
    }

    if include_tts:
        summary_chars = min(transcript_chars // 5, 4000)
        cost_per_char = 0.000015 if tts_model == "tts-1" else 0.00003
        costs["tts"] = round(summary_chars * cost_per_char, 6)

    costs["total"] = round(sum(costs.values()), 4)

    return costs

Cost-reduction strategies

Problem 9: Timeout on Long Files

Symptom

The request to Whisper hangs or returns a timeout.

Solution

import httpx

def transcribe_with_timeout(
    audio_path: str,
    timeout_seconds: int = 300,
    language: str = "es"
) -> str:
    client_with_timeout = OpenAI(
        timeout=httpx.Timeout(timeout_seconds, connect=10.0)
    )

    with open(audio_path, "rb") as f:
        response = client_with_timeout.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language=language
        )
    return response.text

Problem 10: Audio with Multiple Speakers without Diarization

Symptom

Whisper transcribes everything as a single block of text without distinguishing who's speaking.

Solution

Whisper doesn't support native diarization. Options:

1. Use AssemblyAI with speaker_labels=True
2. Post-processing with an LLM to try to separate speakers

def infer_speakers_with_llm(transcript: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                "This is a transcript of a conversation between multiple people. "
                "Try to identify speaker changes based on the context "
                "(topic changes, answers to questions, etc). "
                "Format it as:\n"
                "Speaker A: ...\n"
                "Speaker B: ...\n\n"
                f"Transcript:\n{transcript[:8000]}"
            )
        }],
        max_tokens=2000
    )
    return response.choices[0].message.content

Complete Diagnostic Function

def full_audio_diagnostic(audio_path: str) -> dict:
    path = Path(audio_path)

    if not path.exists():
        return {"error": "File not found", "path": audio_path}

    diagnostic = {
        "file": path.name,
        "format": path.suffix.lower(),
        "size_mb": round(path.stat().st_size / (1024 * 1024), 2),
        "issues": [],
        "recommendations": [],
        "ready_for_whisper": True
    }

    if path.suffix.lower() not in WHISPER_FORMATS:
        diagnostic["issues"].append(f"Format {path.suffix} not supported")
        diagnostic["recommendations"].append("Convert to MP3 with convert_to_supported_format()")
        diagnostic["ready_for_whisper"] = False

    if diagnostic["size_mb"] > 25:
        diagnostic["issues"].append(f"File of {diagnostic['size_mb']}MB exceeds the 25MB limit")
        diagnostic["recommendations"].append("Compress with reduce_file_size() or split with split_large_audio()")
        diagnostic["ready_for_whisper"] = False

    try:
        audio = AudioSegment.from_file(audio_path)
        quality = diagnose_audio_quality(audio_path)
        diagnostic["audio_info"] = {
            "duration_s": quality["duration_s"],
            "sample_rate": quality["sample_rate"],
            "channels": quality["channels"],
            "dbfs": quality["dbfs"],
        }
        diagnostic["quality_issues"] = quality["issues"]
        if quality["issues"]:
            diagnostic["recommendations"].append("Preprocess with preprocess_audio_for_quality()")
    except Exception as e:
        diagnostic["issues"].append(f"Could not read the audio: {str(e)}")
        diagnostic["ready_for_whisper"] = False

    estimated_cost = None
    if "duration_s" in diagnostic.get("audio_info", {}):
        duration_min = diagnostic["audio_info"]["duration_s"] / 60
        estimated_cost = round(duration_min * 0.006, 4)
        diagnostic["estimated_whisper_cost"] = f"${estimated_cost}"

    return diagnostic

Complete Preprocessing Pipeline

def preprocess_pipeline(
    input_path: str,
    output_dir: str = None
) -> dict:
    path = Path(input_path)
    if output_dir is None:
        output_dir = str(path.parent / "preprocessed")
    Path(output_dir).mkdir(exist_ok=True)

    result = {
        "original": input_path,
        "steps": [],
        "final_path": input_path
    }

    current = input_path

    if path.suffix.lower() not in WHISPER_FORMATS:
        converted = str(Path(output_dir) / f"{path.stem}.mp3")
        audio = AudioSegment.from_file(current)
        audio.export(converted, format="mp3", bitrate="128k")
        current = converted
        result["steps"].append("format_conversion")

    audio = AudioSegment.from_file(current)
    needs_normalize = audio.dBFS < -30 or audio.channels > 1 or audio.frame_rate < 16000

    if needs_normalize:
        normalized = str(Path(output_dir) / f"{path.stem}_normalized.mp3")
        audio = audio.set_frame_rate(16000).set_channels(1).normalize()
        audio.export(normalized, format="mp3", bitrate="64k")
        current = normalized
        result["steps"].append("normalization")

    current_size = Path(current).stat().st_size / (1024 * 1024)
    if current_size > 25:
        chunks = split_large_audio(current)
        result["steps"].append("splitting")
        result["chunks"] = chunks
        result["final_path"] = chunks
    else:
        result["final_path"] = current

    return result

Quick Reference Table

Exercises

Exercise 1: Complete audio validator

Create a function that validates an audio file and returns a detailed report with all the problems found and the specific functions to solve them.

def validate_and_report(audio_path: str) -> dict:
    diagnostic = full_audio_diagnostic(audio_path)

    if diagnostic.get("error"):
        return diagnostic

    fixes = {}

    for issue in diagnostic.get("issues", []):
        if "format" in issue.lower():
            fixes["convert_format"] = {
                "function": "convert_to_supported_format(audio_path)",
                "description": "Converts to Whisper-compatible MP3"
            }
        if "25mb" in issue.lower() or "exceeds" in issue.lower():
            fixes["reduce_size"] = {
                "function": "reduce_file_size(audio_path)",
                "description": "Compresses to mono 16kHz 64kbps"
            }
            fixes["split_audio"] = {
                "function": "split_large_audio(audio_path)",
                "description": "Splits into 10-min chunks"
            }

    for qi in diagnostic.get("quality_issues", []):
        issue_type = qi["issue"].lower()
        if "volume" in issue_type:
            fixes["normalize"] = {
                "function": "preprocess_audio_for_quality(audio_path)",
                "description": "Normalizes volume and sample rate"
            }
        if "sample rate" in issue_type:
            fixes["resample"] = {
                "function": "audio.set_frame_rate(16000)",
                "description": "Resamples to 16kHz"
            }
        if "stereo" in issue_type:
            fixes["mono"] = {
                "function": "audio.set_channels(1)",
                "description": "Converts to mono"
            }

    diagnostic["fixes"] = fixes
    diagnostic["auto_fix_available"] = len(fixes) > 0
    diagnostic["auto_fix_command"] = "preprocess_pipeline(audio_path)" if fixes else None

    return diagnostic

Exercise 2: Accumulated cost monitor

Create a class that tracks the accumulated costs of all audio operations and alerts when it gets close to a defined budget.

from dataclasses import dataclass, field

@dataclass
class AudioCostMonitor:
    budget_usd: float = 5.0
    warning_threshold: float = 0.8
    costs: list[dict] = field(default_factory=list)

    @property
    def total_spent(self) -> float:
        return sum(c["cost"] for c in self.costs)

    @property
    def remaining(self) -> float:
        return max(0, self.budget_usd - self.total_spent)

    @property
    def utilization(self) -> float:
        return self.total_spent / self.budget_usd if self.budget_usd > 0 else 0

    def log_cost(self, service: str, cost: float, description: str = ""):
        self.costs.append({
            "service": service,
            "cost": round(cost, 6),
            "description": description,
            "timestamp": time.time()
        })

        if self.utilization >= self.warning_threshold:
            print(f"ALERT: {self.utilization:.0%} of the budget used. "
                  f"Remaining: ${self.remaining:.4f}")

    def check_budget(self, estimated_cost: float) -> dict:
        can_afford = estimated_cost <= self.remaining
        return {
            "can_afford": can_afford,
            "estimated_cost": round(estimated_cost, 6),
            "remaining_budget": round(self.remaining, 4),
            "total_spent": round(self.total_spent, 4)
        }

    def get_report(self) -> dict:
        by_service = {}
        for c in self.costs:
            svc = c["service"]
            by_service[svc] = by_service.get(svc, 0) + c["cost"]

        return {
            "total_spent": round(self.total_spent, 4),
            "budget": self.budget_usd,
            "remaining": round(self.remaining, 4),
            "utilization": f"{self.utilization:.1%}",
            "by_service": {k: round(v, 4) for k, v in by_service.items()},
            "operations": len(self.costs)
        }

Exercise 3: Auto-fix pipeline

Create a function that takes a problematic audio file, runs the complete diagnostic, and automatically applies all the necessary fixes.

def auto_fix_audio(
    audio_path: str,
    output_dir: str = "fixed_audio"
) -> dict:
    Path(output_dir).mkdir(exist_ok=True)

    diagnostic = full_audio_diagnostic(audio_path)
    if diagnostic.get("error"):
        return {"success": False, "error": diagnostic["error"]}

    if diagnostic["ready_for_whisper"] and not diagnostic.get("quality_issues"):
        return {
            "success": True,
            "output_path": audio_path,
            "fixes_applied": [],
            "message": "Audio is already ready for Whisper"
        }

    fixes_applied = []
    current_path = audio_path

    if Path(audio_path).suffix.lower() not in WHISPER_FORMATS:
        converted = str(Path(output_dir) / f"{Path(audio_path).stem}.mp3")
        AudioSegment.from_file(audio_path).export(converted, format="mp3", bitrate="128k")
        current_path = converted
        fixes_applied.append("format_conversion")

    quality = diagnose_audio_quality(current_path)
    if quality["issues"]:
        result = preprocess_audio_for_quality(
            current_path,
            str(Path(output_dir) / f"{Path(audio_path).stem}_normalized.mp3")
        )
        current_path = result["output_path"]
        fixes_applied.append("quality_normalization")

    size_mb = Path(current_path).stat().st_size / (1024 * 1024)
    if size_mb > 25:
        compressed = reduce_file_size(current_path)
        if compressed:
            current_path = compressed
            fixes_applied.append("compression")
        else:
            chunks = split_large_audio(current_path)
            return {
                "success": True,
                "output_paths": chunks,
                "fixes_applied": fixes_applied + ["splitting"],
                "message": f"Audio split into {len(chunks)} chunks"
            }

    return {
        "success": True,
        "output_path": current_path,
        "fixes_applied": fixes_applied,
        "message": f"Applied {len(fixes_applied)} fixes"
    }

Exercise 4: Pre/post-processing quality comparator

Create a function that transcribes the same audio before and after preprocessing, and compares the results.

def compare_preprocessing_quality(
    audio_path: str,
    language: str = "es"
) -> dict:
    t1 = time.time()
    with open(audio_path, "rb") as f:
        original_transcript = client.audio.transcriptions.create(
            model="whisper-1", file=f, language=language,
            response_format="verbose_json"
        )
    original_time = time.time() - t1

    prepped = preprocess_audio_for_quality(audio_path)

    t2 = time.time()
    with open(prepped["output_path"], "rb") as f:
        processed_transcript = client.audio.transcriptions.create(
            model="whisper-1", file=f, language=language,
            response_format="verbose_json"
        )
    processed_time = time.time() - t2

    original_words = set(original_transcript.text.lower().split())
    processed_words = set(processed_transcript.text.lower().split())
    common = original_words & processed_words
    similarity = len(common) / max(len(original_words | processed_words), 1)

    return {
        "original": {
            "text": original_transcript.text,
            "language": original_transcript.language,
            "duration": original_transcript.duration,
            "word_count": len(original_transcript.text.split()),
            "transcription_time_s": round(original_time, 2),
            "file_size_mb": round(Path(audio_path).stat().st_size / (1024 * 1024), 2)
        },
        "processed": {
            "text": processed_transcript.text,
            "language": processed_transcript.language,
            "duration": processed_transcript.duration,
            "word_count": len(processed_transcript.text.split()),
            "transcription_time_s": round(processed_time, 2),
            "file_size_mb": prepped["output_size_mb"]
        },
        "comparison": {
            "word_similarity": round(similarity, 3),
            "word_count_diff": len(processed_transcript.text.split()) - len(original_transcript.text.split()),
            "size_reduction_pct": round(
                (1 - prepped["output_size_mb"] / (Path(audio_path).stat().st_size / (1024 * 1024))) * 100, 1
            )
        }
    }

Summary

• 10 common problems documented with symptoms, causes and concrete solutions.
• Diagnostic function (full_audio_diagnostic()) analyzes format, size, quality and estimates costs.
• Preprocessing pipeline (preprocess_pipeline()) converts format, normalizes audio and splits large files automatically.
• Retry with backoff to handle rate limits resiliently.
• Quick reference table to solve problems in seconds.
• Always diagnose before transcribing — it's cheaper to debug audio than to retry failed transcriptions.

Additional Resources

1. pydub Documentation — Audio manipulation
2. FFmpeg — Base conversion tool
3. Whisper Supported Formats — Official formats
4. OpenAI Rate Limits — Limits per tier
5. Audio Quality for STT — Google's best practices (they apply to all providers)