3. The Audio Modality

Description

The audio modality spans two directions: converting audio to text (transcription/Speech-to-Text) and converting text to audio (speech synthesis/Text-to-Speech). In this capsule you'll learn how both directions work, which models exist, what their capabilities and limitations are, and you'll see runnable code to transcribe audio with Whisper and generate speech with OpenAI TTS.

Why it matters: Audio is the second most in-demand modality after vision. Transcribing meetings, creating voice assistants, generating podcasts, and building audio → text → LLM pipelines are real production use cases. Module 5 goes deep into audio; here you get the fundamentals.

Connection with the module: In capsule 02 you saw vision (image → text). Here you see audio: both the input direction (audio → text) and the output direction (text → audio). Together, vision and audio are the pillars of the multimodal ecosystem you'll master in this guide.

The Two Directions of Audio

Speech-to-Text (STT): Audio → Text

You take an audio file (MP3, WAV, M4A) and get transcribed text.

Input:  [meeting_30min.mp3]
Output: "Good morning everyone. Today we're going to review the Q4 results..."

Main models:

• Whisper (OpenAI): The industry standard. 50+ languages. API or local model.
• Google Speech-to-Text: A good alternative. 100+ languages. Real-time streaming.
• AssemblyAI: A modern API. Speaker diarization. Sentiment detection.

Text-to-Speech (TTS): Text → Audio

You take text and generate an audio file with a synthesized voice.

Input:  "The summary of the quarterly report is positive."
Output: [summary.mp3]  ← audio file with a natural voice

Main models:

• OpenAI TTS: Natural voices (alloy, echo, fable, onyx, nova, shimmer). Simple API.
• ElevenLabs: Ultra-realistic voices. Voice cloning. More expensive but superior in naturalness.
• Google Cloud TTS: Broad language support. WaveNet voices.

Transcription with Whisper (OpenAI)

Basic example

from openai import OpenAI
from pathlib import Path

client = OpenAI()

def transcribe_audio(audio_path: str) -> str:
    """Transcribe an audio file using Whisper."""
    with open(audio_path, "rb") as audio_file:
        transcription = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file
        )
    return transcription.text

# Usage
text = transcribe_audio("meeting.mp3")
print(text)
# Expected output:
# "Good morning everyone. Today we're going to review the fourth
#  quarter results. Sales grew 15% compared to the previous quarter..."

With advanced options

def transcribe_with_options(
    audio_path: str,
    language: str = None,
    prompt: str = None,
    response_format: str = "text",
    temperature: float = 0
) -> str:
    """Transcribe audio with advanced options.

    Args:
        audio_path: Path to the audio file
        language: ISO 639-1 code (e.g.: "es", "en", "fr")
        prompt: Context to improve transcription (proper nouns, jargon)
        response_format: "text", "json", "srt", "verbose_json", "vtt"
        temperature: 0-1. Higher = more creative (less literal)
    """
    with open(audio_path, "rb") as audio_file:
        kwargs = {
            "model": "whisper-1",
            "file": audio_file,
            "response_format": response_format,
            "temperature": temperature
        }
        if language:
            kwargs["language"] = language
        if prompt:
            kwargs["prompt"] = prompt

        transcription = client.audio.transcriptions.create(**kwargs)

    if response_format == "text":
        return transcription
    return transcription  # JSON, SRT, VTT

# Usage: transcription in English with context
text = transcribe_with_options(
    "tech_meeting.mp3",
    language="en",
    prompt="NIEVA, bootcamp, FastAPI, LangChain, RAG"
)

What is prompt for? Whisper can get confused with proper nouns, acronyms or technical jargon. Passing a prompt with the expected words improves accuracy:

# Without prompt: "The framework fast a pi lets you create APIs..."
# With prompt: "The framework FastAPI lets you create APIs..."
prompt = "FastAPI, LangChain, RAG, ChromaDB, Pydantic"

Supported audio formats

Limit: 25 MB per file. For larger files, you need to split into chunks.

Splitting long audio

from pydub import AudioSegment
import math

def split_audio(
    audio_path: str,
    chunk_duration_ms: int = 10 * 60 * 1000  # 10 minutes
) -> list[str]:
    """Split long audio into chunks for Whisper."""
    audio = AudioSegment.from_file(audio_path)
    total_duration = len(audio)
    num_chunks = math.ceil(total_duration / chunk_duration_ms)

    chunk_paths = []
    for i in range(num_chunks):
        start = i * chunk_duration_ms
        end = min((i + 1) * chunk_duration_ms, total_duration)
        chunk = audio[start:end]
        chunk_path = f"chunk_{i:03d}.mp3"
        chunk.export(chunk_path, format="mp3")
        chunk_paths.append(chunk_path)

    return chunk_paths


def transcribe_long_audio(audio_path: str) -> str:
    """Transcribe audio of any duration."""
    import os
    size_mb = os.path.getsize(audio_path) / (1024 * 1024)

    if size_mb <= 24:
        return transcribe_audio(audio_path)

    chunks = split_audio(audio_path)
    transcriptions = []
    for chunk_path in chunks:
        text = transcribe_audio(chunk_path)
        transcriptions.append(text)
        os.remove(chunk_path)  # cleanup

    return " ".join(transcriptions)

Speech Synthesis with OpenAI TTS

Basic example

from openai import OpenAI
from pathlib import Path

client = OpenAI()

def text_to_speech(
    text: str,
    output_path: str = "output.mp3",
    voice: str = "alloy",
    model: str = "tts-1"
) -> str:
    """Generate audio from text using OpenAI TTS.

    Available voices: alloy, echo, fable, onyx, nova, shimmer
    Models: tts-1 (fast), tts-1-hd (high quality)
    """
    response = client.audio.speech.create(
        model=model,
        voice=voice,
        input=text
    )
    response.stream_to_file(output_path)
    return output_path

# Usage
audio_file = text_to_speech(
    "The document analysis shows three main findings.",
    voice="nova"
)
print(f"Audio generated: {audio_file}")
# Expected output: Audio generated: output.mp3

Available voices

High-quality TTS

# Standard model: tts-1 (fast, good for streaming)
# HD model: tts-1-hd (slower, higher quality)

audio_standard = text_to_speech("Hello world", model="tts-1")
audio_hd = text_to_speech("Hello world", model="tts-1-hd")

When to use HD? For final content (podcast, video, presentation). For prototypes and development, tts-1 is enough and faster.

Output formats

# By default: MP3
# It also supports: opus, aac, flac

response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Example text",
    response_format="opus"  # Better for streaming
)

TTS costs

Example: A 500-word text (~3,000 characters) costs ~$0.045 with tts-1.

Comparison: Whisper vs Alternatives

When to use each one?

• Whisper: Simple API, good price, excellent quality, you already use OpenAI for other things
• Google STT: You need real-time streaming or many rare languages
• AssemblyAI: You need speaker diarization (who said what) or sentiment analysis

Comparison: OpenAI TTS vs ElevenLabs

When to use ElevenLabs? When voice quality is critical (podcast, audiobook, premium assistant) or you need to clone a specific voice.

Pipeline: Audio → Text → LLM

The most powerful pattern combines transcription with LLM processing:

def audio_to_summary(audio_path: str) -> dict:
    """Complete pipeline: audio → transcription → summary."""
    # Step 1: Transcribe
    transcript = transcribe_audio(audio_path)

    # Step 2: Summarize with an LLM
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "You are an assistant that summarizes meetings concisely."
            },
            {
                "role": "user",
                "content": (
                    f"Summarize this meeting transcript into:\n"
                    f"1. Executive summary (2-3 sentences)\n"
                    f"2. Key points (bullets)\n"
                    f"3. Action items (if any)\n\n"
                    f"Transcript:\n{transcript}"
                )
            }
        ],
        max_tokens=500
    )
    summary = response.choices[0].message.content

    return {
        "transcript": transcript,
        "summary": summary,
        "audio_file": audio_path
    }

# Usage
result = audio_to_summary("standup_meeting.mp3")
print(result["summary"])

This pipeline is the foundation of tools like Otter.ai, Fireflies, and Notion AI for meetings.

Variant: Audio → Text → LLM → Audio

You can close the loop: transcribe, process, and return audio.

Meeting (audio) → Whisper (transcription) → GPT-4o-mini (summary)
→ TTS (audio of the summary)

This lets you create assistants that "listen" to a meeting and "say" the summary. Exercise 3 implements exactly this.

Troubleshooting

Problem 1: "Invalid file format"

Cause: Unsupported audio format or corrupt file.

Solution:

from pydub import AudioSegment

def convert_to_mp3(audio_path: str) -> str:
    """Convert any audio format to MP3."""
    audio = AudioSegment.from_file(audio_path)
    output_path = audio_path.rsplit(".", 1)[0] + ".mp3"
    audio.export(output_path, format="mp3")
    return output_path

Problem 2: File larger than 25MB

Cause: A long uncompressed recording.

Solution: Use the split_audio function shown above, or compress first:

def compress_audio(audio_path: str, bitrate: str = "64k") -> str:
    """Compress audio by reducing the bitrate."""
    audio = AudioSegment.from_file(audio_path)
    output_path = audio_path.rsplit(".", 1)[0] + "_compressed.mp3"
    audio.export(output_path, format="mp3", bitrate=bitrate)
    return output_path

Problem 3: Inaccurate transcription

Cause: Low-quality audio, background noise, strong accents.

Solution:

• Specify language explicitly (don't let Whisper guess)
• Use prompt with the expected vocabulary
• For very noisy audio, preprocess with pydub (normalize volume, filter noise)

Problem 4: TTS sounds robotic

Cause: Text without punctuation or formatting to guide the prosody.

Solution:

• Use natural punctuation: commas, periods, question marks
• Split long texts into paragraphs
• Experiment with different voices (nova and fable tend to sound more natural)

Exercises

Exercise 1: Basic transcription (Easy)

Write a function that transcribes an English audio and returns the text in uppercase.

def transcribe_uppercase(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        transcription = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language="en"
        )
    return transcription.text.upper()

Exercise 2: Generate audio with a custom voice (Easy)

Create a function that generates audio from a text with the voice the user chooses, validating that it's a valid voice.

VALID_VOICES = {"alloy", "echo", "fable", "onyx", "nova", "shimmer"}

def generate_speech(text: str, voice: str = "alloy") -> str:
    if voice not in VALID_VOICES:
        raise ValueError(
            f"Voice '{voice}' is not valid. Options: {VALID_VOICES}"
        )

    output_path = f"speech_{voice}.mp3"
    response = client.audio.speech.create(
        model="tts-1",
        voice=voice,
        input=text
    )
    response.stream_to_file(output_path)
    return output_path

Exercise 3: Pipeline audio → summary → audio (Medium)

Create a pipeline that: (1) transcribes audio, (2) summarizes with an LLM, (3) generates audio of the summary.

def audio_summary_pipeline(
    audio_path: str,
    summary_voice: str = "nova"
) -> dict:
    # 1. Transcribe
    with open(audio_path, "rb") as f:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language="en"
        ).text

    # 2. Summarize
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                f"Summarize in 3 short sentences:\n\n{transcript}"
            )
        }],
        max_tokens=200
    )
    summary = response.choices[0].message.content

    # 3. Generate audio of the summary
    speech = client.audio.speech.create(
        model="tts-1",
        voice=summary_voice,
        input=summary
    )
    output_path = "summary_audio.mp3"
    speech.stream_to_file(output_path)

    return {
        "transcript": transcript,
        "summary": summary,
        "audio_summary": output_path
    }

Exercise 4: Transcription with timestamps (Medium)

Use response_format="verbose_json" to get a transcription with timestamps. Extract the first 5 sentences with their start time.

import json

def transcribe_with_timestamps(audio_path: str) -> list[dict]:
    with open(audio_path, "rb") as f:
        transcription = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json"
        )

    segments = transcription.segments[:5]
    result = []
    for seg in segments:
        result.append({
            "start": round(seg["start"], 1),
            "end": round(seg["end"], 1),
            "text": seg["text"].strip()
        })
    return result

# Expected output:
# [{"start": 0.0, "end": 3.5, "text": "Good morning everyone."},
#  {"start": 3.5, "end": 8.2, "text": "Today we're going to review the results."},
#  ...]

Summary

In this capsule you learned:

• Speech-to-Text (STT): Audio → text. Whisper is the standard; Google STT and AssemblyAI are alternatives with streaming
• Text-to-Speech (TTS): Text → audio. OpenAI TTS has 6 natural voices; ElevenLabs for premium quality
• Whisper accepts MP3, WAV, M4A and more. 25MB limit. Supports 50+ languages
• The prompt parameter in Whisper improves accuracy with specific vocabulary
• OpenAI TTS has two models: tts-1 (fast) and tts-1-hd (high quality)
• The audio → text → LLM pipeline is the most powerful pattern: transcribe, process, act
• For long audio, split into chunks before transcribing
• Costs: Whisper ~$0.006/min. TTS ~$15/1M characters

Next capsule: Multimodal combinations — how to connect vision, audio and text in pipelines.

Additional Resources

1. Whisper API Docs — Official Whisper documentation
2. OpenAI TTS Docs — Official TTS documentation
3. Whisper GitHub (open-source) — Local Whisper model
4. ElevenLabs API — Premium TTS with voice cloning
5. pydub Documentation — Audio manipulation in Python
6. AssemblyAI Docs — Alternative with speaker diarization