Module 5: Audio Processing

1. Introduction: Audio in AI

Description

This is the first capsule of Module 5 of the Multimodal AI Guide. Here you'll understand how AI processes audio: transcription (Speech-to-Text) with Whisper and alternatives, speech synthesis (Text-to-Speech) with OpenAI and ElevenLabs, and how to build complete pipelines that combine audio, text and LLMs.

Why it matters: Audio is the other big input/output modality. Meetings that get transcribed automatically. Podcasts that generate written summaries. Chatbots that respond with a voice. Accessibility systems that turn text into audio. Each of these cases depends on the same two capabilities: converting audio into text (STT) and converting text into audio (TTS). This module gives you both, plus the pipelines to integrate them with LLMs.

Without this module, you know how to process text and images but you can't do anything with the third most-used modality in production. With it, your multimodal toolkit is complete: text, image and audio. The Audio Pipeline you'll build at the end is a direct component of the Document Analyzer in Module 8.

Where Are We in the Guide?

Context

This guide has 8 modules organized into 3 phases:

Phase 1: Multimodal Foundations (Modules 1-3)
├── Module 1: Introduction to Multimodal AI
├── Module 2: Vision + LLMs
└── Module 3: Document Understanding

Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation
└── Module 5: Audio Processing    ← YOU ARE HERE

Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG
├── Module 7: Use Cases
└── Module 8: Final Project — Multimodal Document Analyzer

Estimated total duration of the guide: 6-7 hours (self-paced).

Where are we coming from?

In the previous modules you built a solid foundation:

Module	What you learned	How it connects with audio
M1: Multimodal Intro	Landscape of modalities, providers, formats	You understand where audio fits in the ecosystem
M2: Vision + LLMs	GPT-4V, Claude 3, image analysis	You know the pattern: input → model → output
M3: Documents	PDFs, OCR, data extraction	You processed complex files, now you'll do the same with audio
M4: Image Generation	DALL-E, Stable Diffusion, text → image	You mastered generation, now you move to text → audio

Where are we headed?

After this module:

Module 6 (Multimodal RAG) — Search over text, images and audio transcriptions
Module 7 (Use Cases) — Production patterns that combine everything
Module 8 (Document Analyzer) — Final project that integrates vision + audio + RAG

Your Audio Pipeline from this module feeds directly into the Document Analyzer: when the system receives an audio file, it will use exactly the pipeline you build here.

What Is Audio Processing in AI

The two directions of audio

Audio processing in AI splits into two fundamental directions:

Direction 1: Audio → Text (Speech-to-Text / STT)
  Input:  audio file (meeting, podcast, voice note)
  Output: transcribed text
  Use:    transcription, subtitles, audio search

Direction 2: Text → Audio (Text-to-Speech / TTS)
  Input:  written text
  Output: audio file with synthesized voice
  Use:    voice assistants, accessibility, narration

Speech-to-Text (STT): Main providers

Provider	Model	Languages	Accuracy	Cost	Standout feature
OpenAI	Whisper	50+	High	$0.006/min	Automatic language detection
Google	Speech-to-Text	125+	High	Variable	GCP integration, streaming
AssemblyAI	Universal	100+	Very high	$0.0065/min	Diarization, PII detection
Deepgram	Nova-2	30+	High	$0.0043/min	Low latency, streaming

Text-to-Speech (TTS): Main providers

Provider	Voices	Quality	Cost	Standout feature
OpenAI TTS	6	High	$15/1M chars	Integrated with the OpenAI ecosystem
ElevenLabs	100+	Very high	Variable	Voice cloning, emotions
Google TTS	200+	High	$4/1M chars	Widest variety of languages
Amazon Polly	60+	High	$4/1M chars	AWS integration

The complete flow: Audio Pipelines

The most powerful thing isn't STT or TTS on their own, but combining them with LLMs:

Complete pipeline:
  Meeting audio
    → Whisper (transcription)
    → GPT-4o-mini (summary + action items)
    → OpenAI TTS (spoken summary)
    → Audio file with the summary

Each step is an API call. The complete pipeline takes ~10 seconds.

Why Audio Matters in AI

The transition: from text-only to multimodal with audio

Before (manual):
  1. Listen to a 1-hour meeting
  2. Take notes manually
  3. Write a summary
  4. Send by email
  Time: 1.5 hours of human work

Now (with audio AI):
  1. Record the meeting
  2. Pipeline: transcribe → summarize → generate audio
  3. Receive the summary in text and audio
  Time: 30 seconds of processing + 0 minutes of human work

Real-world production use cases

Meeting transcription: Audio → transcription → summary with key points → action items. Tools like Otter.ai, Fireflies and Grain process millions of meetings daily. The version you'll build here replicates the core logic.
Podcasts and content: Transcribe episodes for SEO, generate automatic show notes, create clips with subtitles. Media companies use pipelines similar to the ones you'll implement.
Customer support: Transcribe support calls → analyze sentiment → extract recurring topics → generate reports. Contact centers process thousands of hours of audio daily.
Accessibility: Convert text to speech for visually impaired users. Generate audio descriptions for content. Provide real-time subtitles for hearing-impaired users.
Education: Transcribe classes → generate summaries → create flashcards. Students who prefer listening can convert written material into audio.
Voice assistants: Chatbots that listen to questions in audio, process them with an LLM, and respond with a synthesized voice. The complete loop audio → text → LLM → audio.
Legal and compliance: Transcribe depositions, hearings, regulated calls. Search for mentions of specific terms across hours of recording.
Medicine: Transcribe doctor-patient visits → generate structured clinical notes. Reduce the administrative burden on healthcare professionals.

Market data

Whisper processes audio in 50+ languages with human-comparable accuracy
The STT market grows ~20% per year and is projected to reach $30B by 2028
OpenAI TTS generates a voice nearly indistinguishable from a human's at $0.015/1K characters
ElevenLabs enables voice cloning with just a 30-second sample
73% of companies using AI plan to integrate audio capabilities in 2025-2026

Module Objective

By the end of this module you'll be able to:

✅ Transcribe audio with Whisper: formats, languages, timestamps, word-level, and handle files >25MB
✅ Compare STT providers (Whisper, Google, AssemblyAI, Deepgram) using cost, accuracy and feature criteria
✅ Generate speech with OpenAI TTS: voices, models, formats, speed, streaming
✅ Use ElevenLabs for premium TTS and compare it with OpenAI
✅ Build complete pipelines: audio → transcription → LLM → summary → TTS
✅ Diagnose and resolve common problems: formats, sizes, quality, costs
✅ Implement a functional Audio Pipeline as the module project

Professional objective

When someone tells you "we need to transcribe the support calls and generate automatic summaries," you'll know: which STT provider to choose based on volume and budget, how to handle long audio, which LLM model to use for the summary, how to generate the output in audio if required, and how much it will all cost. That's the level of judgment this module gives you.

Module Roadmap

Capsule map

#	Capsule	What you'll learn	Type
01	Introduction (this one)	Audio in AI, STT/TTS landscape, roadmap, setup	Intro
02	Whisper (OpenAI)	API, formats, languages, timestamps, long files, costs	Technical
03	Alternatives to Whisper	Google STT, AssemblyAI, Deepgram, comparison, decision	Technical
04	TTS: OpenAI	Voices, models, formats, speed, streaming, costs	Technical
05	TTS: ElevenLabs and others	ElevenLabs, Google TTS, Amazon Polly, comparison	Technical
06	Audio → text → LLM pipelines	4 complete pipelines, cost tracking, integration	Technical
07	Audio troubleshooting	8+ problems, diagnosis, preprocessing, reference	Technical
08	Project: Audio Pipeline	Complete pipeline: transcribe → process → generate audio	Project

Learning flow

First you master transcription with Whisper (capsule 02) and learn the alternatives (capsule 03). Then you learn speech synthesis with OpenAI TTS (capsule 04) and ElevenLabs (capsule 05). With both directions mastered, you build complete pipelines that combine STT + LLM + TTS (capsule 06). You know how to solve problems when something fails (capsule 07). And finally you integrate everything into a functional Audio Pipeline (capsule 08).

The progression is: STT → STT alternatives → TTS → TTS alternatives → pipelines → troubleshooting → project.

Estimated module duration: 50-60 minutes.

Connection with the Project

This module's project: Audio Pipeline

The Audio Pipeline receives an audio file and runs a complete pipeline:

Input: "team_meeting.mp3"
Output: {
  "transcript": "Full text of the meeting...",
  "summary": "1. Decided to launch v2.0 in March\n2. ...",
  "action_items": ["Prepare demo", "Review budget"],
  "audio_summary_path": "meeting_summary.mp3"
}

Connection with the final project: Multimodal Document Analyzer

The Document Analyzer from module 8 needs to process audio files. When it receives an .mp3 or .wav, it uses exactly this module's pipeline:

Module 5: Audio Pipeline → transcribes + summarizes + generates audio
    ↓
Module 8: Document Analyzer → detects the file type
    → If it's audio: uses the Audio Pipeline from M5
    → If it's an image: uses Vision from M2
    → If it's a PDF: uses Document Processing from M3

Prerequisites

Knowledge from previous modules

Previous module	What you need from it
M1: Intro	Understanding modalities, providers, the concept of a multimodal pipeline
M2: Vision	API usage pattern: client → request → response
M3: Documents	Handling binary files, processing in chunks
M4: Images	Experience with generation: input → model → output

Technical knowledge

Intermediate Python: Functions, classes, handling binary files, context managers (with open)
OpenAI APIs: You've used client.chat.completions.create() comfortably
File handling: You know how to read, write and manipulate files in Python
Terminal: You can install packages with pip and configure environment variables

If you don't have these prerequisites

What you're missing	Recommended resource
Python	Python Essentials Guide (NIEVA)
LLM APIs	AI Engineering Bootcamp Phase 1 (NIEVA)
Modules 1-4	Complete the previous modules of this guide

Technical Setup

Python and dependencies

python --version  # 3.11+ required

pip install openai>=1.0.0 python-dotenv

pip install pydub

pip install elevenlabs

FFmpeg (required for pydub)

pydub needs FFmpeg to convert audio formats. Without FFmpeg, you'll only be able to work with WAV.

# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt-get install ffmpeg

# Windows (with chocolatey)
choco install ffmpeg

Verify the installation:

ffmpeg -version

Environment variables

Create a .env file in your working directory:

# .env
OPENAI_API_KEY=sk-...

# Optional
ELEVENLABS_API_KEY=...
GOOGLE_APPLICATION_CREDENTIALS=path/to/service-account.json
ASSEMBLYAI_API_KEY=...

Load the variables in your code:

from dotenv import load_dotenv
import os

load_dotenv()

openai_key = os.getenv("OPENAI_API_KEY")
assert openai_key, "OPENAI_API_KEY missing in .env"

Quick verification

Run this script to verify that your setup works:

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply only 'OK' if you receive me."}],
    max_tokens=10
)
print(f"OpenAI API: {response.choices[0].message.content}")

Also verify pydub:

from pydub import AudioSegment

audio = AudioSegment.silent(duration=1000)
audio.export("/tmp/test_audio.mp3", format="mp3")
print(f"pydub + ffmpeg: OK (test file created)")

If both print OK, your setup is ready.

Estimated module costs

API	Operation	Cost	Estimate for the module
Whisper	Transcription	$0.006/min	~$0.10-0.30
GPT-4o-mini	Text processing	$0.15/1M input tokens	~$0.05-0.15
OpenAI TTS (tts-1)	Speech synthesis	$15/1M characters	~$0.05-0.15
OpenAI TTS (tts-1-hd)	HD synthesis	$30/1M characters	~$0.10-0.30
ElevenLabs	Premium synthesis	Variable (free tier available)	~$0.00 (free tier)

Total estimate for this module: $0.30-0.90 USD.

Tip: ElevenLabs has a free tier with ~10,000 characters/month. Enough for all the exercises in this module.

Recommended file structure

module_05/
├── transcribe.py          # Transcription functions
├── tts.py                 # Speech synthesis functions
├── pipeline.py            # Complete pipeline (project)
├── troubleshoot.py        # Diagnostic functions
├── audio_samples/
│   ├── test_short.mp3     # Short test audio (<1 min)
│   ├── test_long.mp3      # Long test audio (>10 min)
│   └── test_multilang.mp3 # Audio with multiple languages
└── outputs/
    ├── transcripts/       # Generated transcriptions
    └── tts/               # Synthesized audio

Limits: What This Module Does NOT Cover

❌ Music generation — Models like MusicGen, Suno, Udio. This module focuses on spoken voice (speech), not music.
❌ Real-time streaming — We don't implement real-time transcription (real-time STT). We work with pre-recorded audio files.
❌ Advanced voice cloning — We mention that ElevenLabs supports voice cloning, but we don't go deep into the cloning process.
❌ Local open-source models — We don't deploy Whisper locally or other local STT/TTS models. We use cloud APIs.
❌ Signal processing (DSP) — We don't cover spectrogram analysis, advanced noise filtering, or digital signal processing.
❌ Voice emotion recognition — We don't analyze tone, emotion or sentiment directly from the audio.
❌ Simultaneous multilingual — We don't cover transcription of audio with code-switching (language change within the same audio).

Evidence of Success

By the end of this module, you'll know you succeeded if:

✅ You can transcribe an audio file with Whisper and choose the right response format (text, json, srt, verbose_json)
✅ You know how to handle audio files larger than 25MB by splitting them into chunks
✅ You can compare Whisper, Google STT, AssemblyAI and Deepgram using cost, accuracy and feature criteria
✅ You generate speech with OpenAI TTS choosing the right voice and model
✅ You know ElevenLabs and when to choose it over OpenAI TTS
✅ Your Audio Pipeline works: it receives audio → transcribes → summarizes with an LLM → generates audio of the summary
✅ You can diagnose and resolve common problems (format, size, quality, costs)

Quick self-assessment test

If you can answer these questions, you're on the right track:

Which Whisper response format would you use to generate subtitles?
How would you handle a 50MB audio file that exceeds Whisper's limit?
What's the difference between tts-1 and tts-1-hd in OpenAI?
When would you choose ElevenLabs over OpenAI TTS?
Which pipeline would you use to turn a recorded meeting into an audio summary?
How much would it cost to transcribe 1 hour of audio with Whisper?

Summary

Audio processing in AI has two directions: STT (audio → text) and TTS (text → audio).
The main STT providers are: Whisper (OpenAI), Google Speech-to-Text, AssemblyAI, Deepgram.
The main TTS providers are: OpenAI TTS, ElevenLabs, Google Cloud TTS, Amazon Polly.
Pipelines combine STT + LLM + TTS to create complete flows: transcribe → process → generate audio.
Real-world use cases include: meeting transcription, podcast processing, customer support, accessibility, education.
The module project is an Audio Pipeline that transcribes, summarizes and generates audio.
This module is a prerequisite for the Document Analyzer in Module 8.

Additional Resources

Whisper API — OpenAI — Official transcription documentation
OpenAI TTS Guide — Official speech synthesis documentation
ElevenLabs — Premium TTS with voice cloning
AssemblyAI Docs — Transcription API with diarization
Deepgram — Low-latency STT
Google Speech-to-Text — Google Cloud STT
pydub Documentation — Audio manipulation in Python
FFmpeg — Audio/video conversion tool

← Previous8. Project: Image Generator with Fallback Next →2. Whisper (OpenAI)