Module 5: AWS Services for AI (S3, Lambda, SageMaker Basics)

1. Introduction: AWS Services for AI (S3, Lambda, SageMaker Basics)

Overview

This is the first capsule of Module 5 of the Deployment & Cloud Infrastructure Guide. Here you'll understand how S3, Lambda, and SageMaker fit together as pieces of an AI system on AWS — and why using them together is different from using them in isolation. In Module 4 you tried S3 and Lambda on LocalStack without spending a cent. Now you'll go deeper into how those same services operate in the real AWS context, with IAM, costs, and integration patterns that LocalStack doesn't teach you.

Why it matters: You already know how to create S3 buckets and Lambda functions locally. But there's a gap between "it works on LocalStack" and "it works in AWS production." That gap is called permissions (IAM), costs (S3 storage + Lambda invocations + SageMaker endpoints), and integration patterns (S3 triggers, Lambda layers, event-driven pipelines). This module closes that gap. By the end, you'll have a complete AI service where S3 stores your AI assets, Lambda runs inference, and you'll know when SageMaker is the better option — with numbers, not opinions.

Where Are We in the Guide?

Context

Phase 1: Deployment Strategies (Modules 1-3)
├── Module 1: Understanding Deployment Options    ✅ COMPLETED
├── Module 2: Local & Container Deployment        ✅ COMPLETED
└── Module 3: Serverless & Lambda for AI          ✅ COMPLETED

Phase 2: Cloud Infrastructure & Migration (Modules 4-6)
├── Module 4: LocalStack — AWS Local Development  ✅ COMPLETED
├── Module 5: AWS Services for AI                 ← YOU ARE HERE
└── Module 6: Cloud Migration Patterns

Phase 3: Alternatives & Production (Modules 7-8)
├── Module 7: Alternative Platforms (Render, Railway, Fly.io)
└── Module 8: Integrator Project — Deployed AI System

Total estimated guide duration: 10-12 hours (self-paced).

Transition from Module 4

In Module 4 you conquered LocalStack: you brought up S3 and Lambda locally, built an AI pipeline against simulated APIs, and validated that your boto3 code works without an AWS account. That gave you three things:

Confidence — Your code interacts with S3 and Lambda correctly; you tested it locally.
Speed — You iterate without waiting for cloud deploys or paying for mistakes.
Portability — You learned that boto3 works the same against LocalStack and AWS by changing only the endpoint.

Now comes the question LocalStack doesn't answer: how do you operate these services in real production?

Permissions: In LocalStack there's no real IAM. In AWS, your Lambda needs a role with specific permissions to access S3. Without the right role, your function fails with AccessDenied.
Costs: In LocalStack everything is free. In AWS, every S3 operation, every millisecond of Lambda, every hour of a SageMaker endpoint costs money. You need to estimate before you deploy.
Real integration: S3 can trigger Lambda automatically when a file is uploaded. Lambda can write results to S3. SageMaker can serve custom models. These integration patterns are the heart of an AI system on AWS.
SageMaker: LocalStack doesn't simulate SageMaker. Here you'll learn what it is, when to use it vs Lambda, and how a basic endpoint works.

This module takes you from "it works locally" to "I understand how it works on real AWS."

What AWS Services for AI Are

AWS as an AI platform: the pieces that matter

AWS has 200+ services. For an AI system, you need to understand three:

AWS Services for AI (this module):
├── S3 (Simple Storage Service)
│   ├── Stores models, embeddings, RAG documents
│   ├── Acts as the data lake of your AI system
│   └── Event trigger (new file → Lambda runs)
│
├── Lambda (Serverless Compute)
│   ├── Runs inference: reads assets from S3, invokes LLM
│   ├── Processes events: S3 trigger, API Gateway, scheduled
│   └── Scales automatically from 0 to thousands
│
└── SageMaker (ML Platform — basics)
    ├── Deploys custom models (not external APIs)
    ├── Dedicated inference endpoints
    └── When to use it vs Lambda → decision framework

Why these three and not others

You could use DynamoDB, SQS, Step Functions, ECS, and dozens more. But for a typical AI system, 80% of the value comes from S3 + Lambda:

S3 is where your data lives. Serialized models, embedding files, documents for RAG, versioned prompt templates, inference logs for auditing.
Lambda is where your logic runs. It reads from S3, builds the prompt, invokes the LLM (OpenAI, Anthropic), processes the response, and saves it to S3.
SageMaker is the option when Lambda isn't enough: custom models you need to serve (not external APIs), training, inference with models >1GB.

The AI architecture studio analogy

Think of AWS as an architecture studio for your AI system:

S3 is your blueprint archive. It stores everything: the designs (models), the specs (embeddings), the client documents (RAG data), the templates (prompt templates). All organized in folders (prefixes), accessible when you need it.
Lambda is your on-demand work crew. You don't pay to keep them sitting around waiting. When a project (request) arrives, they activate, consult the archive (S3), do the work (inference), deliver the result, and leave. You pay for the work done.
SageMaker is your specialized workshop. For when you need to fabricate custom parts (your own models), not just assemble third-party components (APIs). It has more capacity, but also more cost and complexity.

The question isn't "which one do I use?" but "which ones do I combine for my case?"

Module Objective

By the end of this module you'll be able to:

✅ Use S3 to store and retrieve AI assets: models, embeddings, RAG documents, prompt templates — with organization by prefixes
✅ Implement S3 operations with boto3: put_object, get_object, list_objects_v2, generate_presigned_url, delete_object
✅ Build a complete S3 → Lambda → S3 flow: Lambda reads a prompt template from S3, invokes an LLM, writes the response to S3
✅ Configure S3 event notifications that trigger Lambda automatically
✅ Explain what SageMaker is and when to use it vs Lambda, with technical judgment
✅ Create IAM roles with minimal permissions for Lambda → S3, following least privilege
✅ Estimate AWS costs for an AI system: S3 storage + requests, Lambda compute, SageMaker endpoint uptime
✅ Integrate your LocalStack knowledge (M4) with real AWS services

Professional objective

When your team says "let's put the RAG data in S3 and use Lambda for inference," you'll know exactly what to ask: "What bucket structure? What permissions does Lambda need? How much will it cost at our volume? Do we need SageMaker or is Lambda enough?" And you'll be able to implement it — not just have an opinion.

Module Roadmap

Capsule map

#	Capsule	What you'll learn	Type
01	Introduction (this one)	Context, objectives, setup, roadmap	Intro
02	S3 for AI Assets	Store models, embeddings, RAG docs with boto3. Bucket organization	Technical
03	Lambda for AI Inference	Lambda that invokes an LLM from AWS, error handling, retry, structured output	Technical
04	S3 + Lambda Integration	S3 trigger → Lambda → S3. Complete event-driven flow	Technical
05	SageMaker Basics	What it is, when to use it vs Lambda, basic endpoint deploy	Technical
06	IAM and Least Privilege	Roles, policies, least privilege for Lambda→S3	Technical
07	Cost Estimation AWS AI	Cost calculator for S3, Lambda, SageMaker with real numbers	Technical
08	Project: S3+Lambda AI Service	Complete AI service with S3, Lambda, IAM, cost estimation	Project

Learning flow

First you'll master S3 as the data layer of your AI system: organization, boto3 operations, storage patterns for models and RAG (capsule 02). Then you'll implement Lambda as the inference layer in a real AWS context: invoking LLMs, robust error handling, structured output (capsule 03). Next you'll integrate both into an event-driven flow: S3 trigger → Lambda → processes → writes to S3 (capsule 04). You'll explore SageMaker basics to know when Lambda isn't enough (capsule 05). You'll learn IAM so your system is secure in production (capsule 06). You'll estimate costs so you don't get surprises on the bill (capsule 07). And finally you'll integrate everything into a functional AI service (capsule 08).

The progression is: storage → compute → integration → alternatives → security → costs → project.

Estimated module duration: 1.25-1.5 hours.

Connection with the Project

This module's project: S3+Lambda AI Service

The S3+Lambda AI Service is a service that:

Stores AI assets in S3 (prompt templates, RAG documents, embeddings)
Uses Lambda to read assets, build contextualized prompts, invoke an LLM
Persists the responses in S3 for auditing and reuse
Has IAM permissions configured with least privilege
Includes cost estimation for different usage volumes

                    ┌─────────────────────────────────┐
                    │           S3 Bucket              │
                    │  ai-assets-{account-id}/         │
                    │  ├── prompts/                    │
                    │  │   └── summarizer-v1.txt       │
                    │  ├── documents/                  │
                    │  │   └── knowledge-base.json     │
                    │  └── responses/                  │
                    │      └── 2026-03-08/resp-001.json│
                    └───────┬───────────────┬──────────┘
                            │               ↑
                     S3 trigger        put_object
                     (new doc)         (response)
                            ↓               │
                    ┌───────┴───────────────┴──────────┐
                    │         Lambda Function           │
                    │  ├── Reads prompt template from S3│
                    │  ├── Reads document from S3       │
                    │  ├── Builds contextual prompt     │
                    │  ├── Invokes OpenAI API           │
                    │  └── Writes response to S3        │
                    └──────────────┬────────────────────┘
                                   │
                              ┌────┴────┐
                              │ OpenAI  │
                              │   API   │
                              └─────────┘

Connection with earlier and later modules

Module 3: Lambda AI Endpoint → functional Lambda function (base)
    ↓
Module 4: LocalStack → you tested S3 + Lambda locally, free
    ↓
Module 5: AWS Services → you deepen integration, IAM, costs ← YOU ARE HERE
    ↓
Module 6: Migration → same code on LocalStack and AWS
    ↓
Module 8: Integrator Project → S3+Lambda as a deployment option

Prerequisites

What you already know

✅ Lambda fundamentals — Handler, event, context, packaging (Module 3)
✅ Docker Compose — Multi-container, services (Module 2)
✅ LocalStack — Local S3, local Lambda, boto3 with endpoint_url (Module 4)
✅ Basic boto3 — Create buckets, upload files, invoke Lambda (Module 4)
✅ Intermediate Python — Classes, async, error handling
✅ AI apps — You've invoked LLMs from code (OpenAI SDK)

What you'll learn here (new)

S3 as a data layer for AI: organization, lifecycle, presigned URLs
Lambda for AI inference in a real AWS context: IAM roles, triggers, retry patterns
S3 ↔ Lambda integration: event-driven AI pipelines
SageMaker basics: what it is, when to use it, endpoint deploy
IAM least privilege: roles, policies, security boundaries
Cost estimation: pricing models for S3, Lambda, SageMaker with a calculator

If you're missing something

You're missing	Recommended resource
Lambda fundamentals	Module 3 of this guide
LocalStack	Module 4 of this guide
Docker Compose	Module 2 of this guide
Decision Framework	Module 1 of this guide
Python + FastAPI	Python REST APIs for AI Guide — NIEVA
AI apps	AI Engineering Bootcamp — NIEVA

A note on LocalStack vs real AWS

All the examples in this module work both with real AWS and with LocalStack. If you don't have an AWS account, keep using the LocalStack from Module 4 — the concepts are identical, the boto3 commands are the same. The main difference is IAM (LocalStack doesn't enforce it) and costs (LocalStack is free). When the module mentions "real AWS," read it as "the behavior you'll have when you migrate to AWS" — which is exactly what you'll do in Module 6.

Technical Setup

Required tools

# Python 3.10+ (same as previous modules)
python --version

# boto3 (AWS SDK for Python)
pip install boto3
python -c "import boto3; print(boto3.__version__)"

# AWS CLI v2 (configured in M3)
aws --version

# OpenAI SDK (for inference in Lambda)
pip install openai

Credentials configuration

You have two options for working through this module:

Option 1: Real AWS (if you have an account)

# Verify that AWS CLI is configured
aws sts get-caller-identity

# Expected output:
# {
#     "UserId": "AIDAEXAMPLE",
#     "Account": "123456789012",
#     "Arn": "arn:aws:iam::123456789012:user/your-user"
# }

# If it's not configured:
aws configure
# AWS Access Key ID: your-access-key
# AWS Secret Access Key: your-secret-key
# Default region: us-east-1
# Default output format: json

Option 2: LocalStack (no AWS account, no cost)

# Start LocalStack (if you don't have it from M4 yet)
docker run -d --name localstack \
  -p 4566:4566 \
  -e SERVICES=s3,lambda,iam,sagemaker \
  localstack/localstack

# Configure AWS CLI to use LocalStack
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Verify
aws --endpoint-url=http://localhost:4566 s3 ls

All the examples in this module work with both options. In boto3 code, the difference is one line:

import boto3

# Real AWS
s3 = boto3.client("s3")

# LocalStack
s3 = boto3.client("s3", endpoint_url="http://localhost:4566")

In Module 6 you'll learn to abstract this with environment variables so the same code works in both environments without changes.

Create the module structure

mkdir -p module-05/{s3-operations,lambda-inference,integration,tests}
cd module-05

# Structure
# module-05/
# ├── s3-operations/
# │   └── s3_ai_assets.py      # S3 operations for AI
# ├── lambda-inference/
# │   ├── handler.py            # Lambda for inference
# │   └── requirements.txt
# ├── integration/
# │   └── s3_lambda_flow.py     # S3 + Lambda integration
# ├── tests/
# │   └── test_s3_operations.py
# └── .env

Limits: What This Module Does NOT Cover

❌ Advanced SageMaker — Training jobs, pipelines, feature store, MLOps. SageMaker has enough depth for a full guide on its own. Here it's "basics": what it is, when to use it, endpoint deploy. If you need more, consult the SageMaker documentation directly.
❌ DynamoDB, SQS, SNS — Useful services but out of scope for this module. The focus is S3 + Lambda + SageMaker basics.
❌ Terraform/CDK — Advanced Infrastructure as Code. We use boto3 directly and SAM to keep the focus on the services, not on IaC tooling.
❌ Multi-account strategies — AWS Organizations, cross-account access. This is enterprise AWS, not deployment for AI systems.
❌ Complete data engineering — ETL pipelines, Glue, Athena. We use S3 as storage for AI assets, not as a data lake for analytics.
❌ Custom ML models — Training and serving your own model. If you need this, SageMaker (the full version, not basics) is your path.

What we do cover (and why)

Topic	Reason
S3 for AI assets	It's where your AI system's data lives
Lambda for inference	It's the serverless compute layer that connects S3 with LLMs
S3 ↔ Lambda integration	The event-driven pattern that makes the system run on its own
SageMaker basics	So you know when Lambda isn't enough
IAM least privilege	Without the right permissions, your system doesn't work (or is insecure)
Cost estimation	Without estimation, your system can cost you more than expected

Evidence of Success

By the end of this module, you'll know you succeeded if:

✅ Your boto3 code uploads and retrieves AI assets (models, embeddings, prompt templates) from S3
✅ Your Lambda reads from S3, invokes an LLM, and writes the response back to S3
✅ An S3 event trigger fires your Lambda automatically when a file is uploaded
✅ You can explain when to use SageMaker vs Lambda with technical arguments
✅ Your Lambda has an IAM role with minimal permissions (no * on resources)
✅ You can estimate the monthly cost of your AI service at 1K, 10K, and 100K invocations
✅ You can run the same code against LocalStack and AWS by changing only the endpoint

Quick self-assessment test

If you can answer these questions, you're on the right track:

How would you organize an S3 bucket for a RAG system with multiple document collections?
What minimal IAM permissions does Lambda need to read from a specific S3 bucket?
How much does it cost to store 10GB of embeddings in S3 for a month?
When would you choose SageMaker instead of Lambda for inference?

Summary

From M4 (LocalStack) to M5 (real AWS): you move from simulating services to understanding them in a production context — permissions, costs, and real integrations.
S3 + Lambda is 80% of your AI system on AWS. S3 stores assets, Lambda runs inference. Together they form a serverless event-driven pipeline.
SageMaker is covered at a basics level. Knowing what it is and when to use it is enough for this module — it's not a deep dive into MLOps.
IAM is not optional. In production, misconfigured permissions mean "AccessDenied" or, worse, security breaches.
Costs matter. Estimate before you deploy. S3 is cheap, Lambda is variable, SageMaker endpoints are expensive if you leave them on.
The code in this module is reused in M6 (Migration) and M8 (Integrator Project).

Additional Resources

AWS S3 Developer Guide — Official S3 documentation
boto3 S3 Client Reference — Complete S3 client reference
AWS Lambda Developer Guide — Python — Lambda with Python
Amazon SageMaker Documentation — Official SageMaker documentation
AWS IAM Best Practices — IAM best practices
AWS Pricing Calculator — AWS cost calculator
LocalStack Documentation — For local development at no cost

← Previous8. Project: LocalStack AI Pipeline Next →2. S3 for AI Assets