7. Debugging LocalStack

Overview

In this capsule you'll learn to diagnose and resolve the most common problems when working with LocalStack. It's not theory — it's the practical survival kit: read logs, verify service availability, resolve permission and format errors, and use diagnostic tools. When something fails (and something always fails), you'll know exactly where to look.

Context: In the previous capsules you built an S3 + Lambda pipeline with environment switching. Everything worked when you followed the steps. But in reality, errors appear: LocalStack won't start, Lambda returns a timeout, S3 says the bucket doesn't exist. This capsule prepares you for those moments. It's the difference between "it doesn't work and I don't know why" and "it doesn't work, but I know where to look."

The Diagnostic Flow

When something fails, follow this order

1. Is LocalStack running?
   └── docker compose ps / curl health endpoint
       ↓
2. Is the service you need available?
   └── curl /_localstack/health → check services
       ↓
3. Does your request reach LocalStack?
   └── docker compose logs localstack --tail 20
       ↓
4. Is the error in your code or in LocalStack?
   └── Try the same operation with the awslocal CLI
       ↓
5. Is it a known problem?
   └── Check the common-errors table below

This flow solves 90% of problems. Memorize it.

Level 1: Is LocalStack Running?

Check the container

# Is the container up?
docker compose ps localstack

# Possible outputs:
# localstack   Up (healthy)     ← All good
# localstack   Up (health: starting)  ← Still starting, wait
# localstack   Exited (1)       ← Crashed
# (nothing)                     ← Not defined in Compose

If the container isn't running

# See why it stopped
docker compose logs localstack --tail 30

# Common causes:
# 1. Port 4566 in use
lsof -i :4566
# If there's another process, kill it or change the port

# 2. Docker daemon isn't running
docker info
# If it fails, start Docker Desktop

# 3. Image not downloaded
docker pull localstack/localstack:latest

If the container is in "health: starting"

# LocalStack takes 10-20 seconds to start
# Wait and check:
for i in {1..10}; do
  STATUS=$(docker compose ps localstack --format json | python3 -c "import json,sys; data=json.load(sys.stdin); print(data.get('Health','unknown'))" 2>/dev/null || echo "checking")
  echo "Attempt $i: $STATUS"
  if [ "$STATUS" = "healthy" ]; then
    echo "LocalStack ready"
    break
  fi
  sleep 3
done

Level 2: Are the Services Available?

Health endpoint

# The health endpoint shows the status of each service
curl -s http://localhost:4566/_localstack/health | python3 -m json.tool

# Expected output:
# {
#     "services": {
#         "s3": "available",
#         "lambda": "available"
#     },
#     "version": "3.x.x"
# }

Interpret the statuses

"available"  → Service ready to use
"running"    → Service starting (wait a few seconds)
"disabled"   → Not included in SERVICES
"error"      → Problem initializing (see logs)
(not shown)  → Not configured in SERVICES

Complete health check script

# health_check.py
import requests
import json
import sys

ENDPOINT = "http://localhost:4566"

def check_localstack():
    """Complete diagnosis of LocalStack."""
    print("=== LocalStack Health Check ===\n")

    # 1. Connectivity
    try:
        resp = requests.get(f"{ENDPOINT}/_localstack/health", timeout=5)
        health = resp.json()
        print(f"Version: {health.get('version', 'unknown')}")
    except requests.ConnectionError:
        print("ERROR: Cannot connect to LocalStack")
        print(f"  Check: docker compose ps localstack")
        print(f"  Check: port 4566 accessible")
        sys.exit(1)
    except Exception as e:
        print(f"ERROR: {e}")
        sys.exit(1)

    # 2. Services
    services = health.get("services", {})
    print(f"\nServices ({len(services)}):")
    all_ok = True
    for name, status in services.items():
        icon = "✅" if status == "available" else "❌"
        print(f"  {icon} {name}: {status}")
        if status != "available":
            all_ok = False

    # 3. Verify S3
    print("\nS3 verification:")
    try:
        import boto3
        s3 = boto3.client(
            "s3", endpoint_url=ENDPOINT,
            aws_access_key_id="test", aws_secret_access_key="test",
            region_name="us-east-1",
        )
        buckets = s3.list_buckets()
        print(f"  ✅ S3 responds — {len(buckets['Buckets'])} buckets")
    except Exception as e:
        print(f"  ❌ S3 error: {e}")
        all_ok = False

    # 4. Verify Lambda
    print("\nLambda verification:")
    try:
        lam = boto3.client(
            "lambda", endpoint_url=ENDPOINT,
            aws_access_key_id="test", aws_secret_access_key="test",
            region_name="us-east-1",
        )
        functions = lam.list_functions()
        print(f"  ✅ Lambda responds — {len(functions['Functions'])} functions")
    except Exception as e:
        print(f"  ❌ Lambda error: {e}")
        all_ok = False

    # Result
    print(f"\n{'='*30}")
    if all_ok:
        print("Status: HEALTHY — everything working")
    else:
        print("Status: DEGRADED — check services with errors")

    return all_ok


check_localstack()

Level 3: Read the Container Logs

Basic logs

# Last 50 lines
docker compose logs localstack --tail 50

# Follow in real time (useful while debugging)
docker compose logs localstack -f

# Errors only
docker compose logs localstack 2>&1 | grep -i "error\|exception\|failed"

# Logs with timestamps
docker compose logs localstack --timestamps --tail 20

Enable DEBUG mode

If the normal logs don't give enough information, enable DEBUG:

# docker-compose.yml
localstack:
  environment:
    - DEBUG=1  # Change from 0 to 1

# Restart to apply
docker compose restart localstack

# Now the logs show details of each request
docker compose logs localstack -f
# You'll see every HTTP request that arrives at LocalStack

Go back to DEBUG=0 when you're done — DEBUG generates a lot of logs.

Lambda logs specifically

# When you invoke a Lambda, LocalStack logs the execution
# Look for the relevant lines:
docker compose logs localstack 2>&1 | grep -A5 "lambda.*invoke\|lambda.*create"

# If LAMBDA_EXECUTOR=docker, Lambdas run in separate containers
# List Lambda containers:
docker ps | grep "lambda"

# See the logs of a specific Lambda container:
docker logs <container-id>

Level 4: Common Errors and Solutions

Error 1: "Unable to connect to endpoint URL"

botocore.exceptions.EndpointConnectionError:
Could not connect to the endpoint URL: "http://localhost:4566/"

Cause: LocalStack isn't running or the port isn't accessible.

# Diagnosis:
docker compose ps localstack
curl http://localhost:4566/_localstack/health

# Solution:
docker compose up -d localstack
# Wait for the health check
sleep 15
curl http://localhost:4566/_localstack/health

Error 2: "NoSuchBucket"

botocore.exceptions.ClientError: An error occurred (NoSuchBucket)
when calling the PutObject operation

Cause: The bucket doesn't exist. In LocalStack Community, buckets are lost on restart.

# Diagnosis:
awslocal s3 ls
# Does your bucket appear?

# Solution:
awslocal s3 mb s3://ai-input
awslocal s3 mb s3://ai-output

# Prevention: use init scripts
# init-scripts/setup.sh creates the buckets on startup

Error 3: "ResourceNotFoundException" (Lambda)

botocore.exceptions.ClientError: An error occurred (ResourceNotFoundException)
when calling the Invoke operation: Function not found

Cause: The Lambda function doesn't exist or the name doesn't match.

# Diagnosis:
awslocal lambda list-functions --query 'Functions[].FunctionName'

# Does your function appear? Does the name match exactly?
# Lambda names are case-sensitive

# Solution:
# If it doesn't exist, deploy it:
awslocal lambda create-function --function-name ai-processor ...

# If the name doesn't match, use the correct name

Error 4: "Lambda timeout"

Task timed out after X seconds

Cause: Your handler takes longer than the configured timeout.

# Diagnosis:
awslocal lambda get-function-configuration \
  --function-name ai-processor \
  --query 'Timeout'
# Is it enough for your AI workload?

# Solution:
awslocal lambda update-function-configuration \
  --function-name ai-processor \
  --timeout 120

Error 5: "ModuleNotFoundError in Lambda"

[ERROR] Runtime.ImportModuleError:
Unable to import module 'handler': No module named 'openai'

Cause: The dependencies aren't in the deployment zip.

# Diagnosis:
unzip -l lambda/handler.zip | head -20
# Do the openai modules appear?

# Solution: repackage with dependencies
pip install -r lambda/requirements.txt -t lambda/package/
cp lambda/handler.py lambda/package/
cd lambda/package && zip -r ../handler.zip . && cd ../..

# Update:
awslocal lambda update-function-code \
  --function-name ai-processor \
  --zip-file fileb://lambda/handler.zip

Error 6: "Lambda can't connect to S3"

botocore.exceptions.EndpointConnectionError:
Could not connect to the endpoint URL: "http://localhost:4566/"

Cause: Lambda runs in a separate Docker container. localhost inside that container isn't your machine.

# Diagnosis:
awslocal lambda get-function-configuration \
  --function-name ai-processor \
  --query 'Environment.Variables.AWS_ENDPOINT_URL'

# If it says "http://localhost:4566" and LAMBDA_EXECUTOR=docker → that's the problem

# Solution:
awslocal lambda update-function-configuration \
  --function-name ai-processor \
  --environment "Variables={
    AWS_ENDPOINT_URL=http://host.docker.internal:4566,
    OPENAI_API_KEY=${OPENAI_API_KEY},
    MODEL_NAME=gpt-4o-mini
  }"
# host.docker.internal resolves to the host from inside containers

# If LAMBDA_EXECUTOR=local, localhost:4566 does work

Error 7: "Port 4566 already in use"

Error starting container: port is already allocated

Cause: Another process or container is using port 4566.

# Diagnosis:
lsof -i :4566

# Solution 1: kill the process using the port
kill -9 <PID>

# Solution 2: change the port in Compose
localstack:
  ports:
    - "4567:4566"
# And update AWS_ENDPOINT_URL to http://localhost:4567

# Solution 3: if it's another LocalStack container
docker stop localstack-test && docker rm localstack-test

Diagnostic Tools

Complete diagnostic script

#!/bin/bash
# scripts/diagnose.sh — Complete LocalStack diagnosis

echo "=== LOCALSTACK DIAGNOSIS ==="
echo ""

echo "1. Docker"
docker --version
docker compose version
echo ""

echo "2. Container Status"
docker compose ps
echo ""

echo "3. Health Check"
HEALTH=$(curl -s http://localhost:4566/_localstack/health 2>/dev/null)
if [ $? -ne 0 ]; then
  echo "   ERROR: Cannot connect to LocalStack"
  echo "   → Check: docker compose up -d localstack"
  exit 1
fi
echo "$HEALTH" | python3 -m json.tool
echo ""

echo "4. S3 Buckets"
awslocal s3 ls 2>/dev/null || echo "   S3 not available"
echo ""

echo "5. Lambda Functions"
awslocal lambda list-functions --query 'Functions[].{Name:FunctionName,Runtime:Runtime,Memory:MemorySize}' --output table 2>/dev/null || echo "   Lambda not available"
echo ""

echo "6. Latest errors in logs"
docker compose logs localstack --tail 50 2>&1 | grep -i "error\|exception\|failed" | tail -5
if [ $? -ne 0 ]; then
  echo "   No recent errors found"
fi
echo ""

echo "7. Docker Resources"
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" 2>/dev/null | grep -E "NAME|localstack"
echo ""

echo "8. Port 4566"
lsof -i :4566 2>/dev/null | head -5
echo ""

echo "=== END OF DIAGNOSIS ==="

Verify a specific operation

# If an operation fails, test it in isolation with awslocal:

# S3: create bucket
awslocal s3 mb s3://test-debug 2>&1
# If it works here but not in Python → problem in your code
# If it fails here too → problem in LocalStack

# Lambda: invoke
awslocal lambda invoke \
  --function-name ai-processor \
  --payload '{"document_key": "test.txt"}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/debug.json 2>&1
cat /tmp/debug.json | python3 -m json.tool

# This isolates whether the problem is your code or LocalStack

Monitor requests in real time

# With DEBUG=1, LocalStack logs every request
# In one terminal:
docker compose logs localstack -f

# In another terminal, run your operation:
awslocal s3 ls

# In the logs you'll see the exact HTTP request
# This helps understand what your code is sending

Performance Problems

LocalStack uses a lot of RAM

# Check memory usage
docker stats --no-stream localstack

# If it uses >2GB:
# 1. Reduce the enabled services
environment:
  - SERVICES=s3,lambda  # Only what's needed

# 2. If LAMBDA_EXECUTOR=docker, each Lambda creates a container
# Clean up old containers:
docker container prune -f

LocalStack takes too long to start

# Cause: too many enabled services
# Solution: only the ones you use
environment:
  - SERVICES=s3,lambda
  # Not: SERVICES= (all)

# Startup with s3,lambda takes ~10s
# Startup with all the services can take ~30-60s

Lambda takes too long to run

# With LAMBDA_EXECUTOR=docker, the first invocation is slow
# because it creates a Docker container for the function

# For fast iteration, use LAMBDA_EXECUTOR=local
environment:
  - LAMBDA_EXECUTOR=local
# Runs in the LocalStack process — faster, less isolated

Exercises

Exercise 1: Create the diagnostic script

Copy the diagnose.sh script from above, run it, and interpret each section of the output. If something fails, resolve it.

# Create the script
mkdir -p scripts
cat > scripts/diagnose.sh << 'SCRIPT'
#!/bin/bash
echo "=== LOCALSTACK DIAGNOSIS ==="

echo "1. Container Status"
docker compose ps

echo "2. Health Check"
curl -s http://localhost:4566/_localstack/health | python3 -m json.tool 2>/dev/null || echo "LocalStack not responding"

echo "3. S3"
awslocal s3 ls 2>/dev/null || echo "S3 not available"

echo "4. Lambda"
awslocal lambda list-functions --query 'Functions[].FunctionName' --output text 2>/dev/null || echo "Lambda not available"

echo "5. Recent errors"
docker compose logs localstack --tail 20 2>&1 | grep -i "error" | tail -3 || echo "No errors"

echo "=== END ==="
SCRIPT

chmod +x scripts/diagnose.sh
./scripts/diagnose.sh

Exercise 2: Trigger and resolve each error

Intentionally trigger 3 common errors and resolve them:

1. Invoke a Lambda that doesn't exist
2. Do put_object to a bucket that doesn't exist
3. Make a request with the wrong endpoint

import boto3
import json

endpoint = "http://localhost:4566"
kwargs = {
    "endpoint_url": endpoint,
    "aws_access_key_id": "test",
    "aws_secret_access_key": "test",
    "region_name": "us-east-1",
}

# Error 1: Lambda that doesn't exist
print("--- Error 1: Nonexistent Lambda ---")
lam = boto3.client("lambda", **kwargs)
try:
    lam.invoke(FunctionName="does-not-exist", Payload=b'{}')
except Exception as e:
    print(f"  Error: {type(e).__name__}: {e}")
    print("  Solution: verify the name with awslocal lambda list-functions")

# Error 2: Bucket that doesn't exist
print("\n--- Error 2: Nonexistent bucket ---")
s3 = boto3.client("s3", **kwargs)
try:
    s3.put_object(Bucket="does-not-exist", Key="test.txt", Body=b"data")
except Exception as e:
    print(f"  Error: {type(e).__name__}: {e}")
    print("  Solution: create the bucket with awslocal s3 mb s3://does-not-exist")

# Error 3: Wrong endpoint
print("\n--- Error 3: Wrong endpoint ---")
bad_s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:9999",
    aws_access_key_id="test",
    aws_secret_access_key="test",
    region_name="us-east-1",
)
try:
    bad_s3.list_buckets()
except Exception as e:
    print(f"  Error: {type(e).__name__}: {e}")
    print("  Solution: verify that AWS_ENDPOINT_URL=http://localhost:4566")

Exercise 3: Health check with alerts

Extend the health_check.py script so it verifies:

• That the ai-input and ai-output buckets exist
• That the ai-processor function is deployed
• That a test invocation returns status 200

import boto3
import json
import sys

ENDPOINT = "http://localhost:4566"
KWARGS = {
    "endpoint_url": ENDPOINT,
    "aws_access_key_id": "test",
    "aws_secret_access_key": "test",
    "region_name": "us-east-1",
}

checks = {}

# 1. Connectivity
try:
    import requests
    resp = requests.get(f"{ENDPOINT}/_localstack/health", timeout=5)
    checks["connectivity"] = "PASS"
except Exception:
    checks["connectivity"] = "FAIL — LocalStack not responding"
    for name, result in checks.items():
        print(f"  {'✅' if result == 'PASS' else '❌'} {name}: {result}")
    sys.exit(1)

# 2. Required buckets
s3 = boto3.client("s3", **KWARGS)
for bucket in ["ai-input", "ai-output"]:
    try:
        s3.head_bucket(Bucket=bucket)
        checks[f"bucket_{bucket}"] = "PASS"
    except Exception:
        checks[f"bucket_{bucket}"] = f"FAIL — bucket '{bucket}' doesn't exist"

# 3. Lambda function
lam = boto3.client("lambda", **KWARGS)
try:
    lam.get_function(FunctionName="ai-processor")
    checks["lambda_ai_processor"] = "PASS"
except Exception:
    checks["lambda_ai_processor"] = "FAIL — function 'ai-processor' doesn't exist"

# 4. Test invocation (only if the function exists)
if checks.get("lambda_ai_processor") == "PASS":
    try:
        resp = lam.invoke(
            FunctionName="ai-processor",
            InvocationType="RequestResponse",
            Payload=json.dumps({"document_key": "test-health.txt"}),
        )
        result = json.loads(resp["Payload"].read())
        status = result.get("statusCode", 0)
        checks["lambda_invoke"] = f"PASS (status={status})" if status in [200, 400, 404] else f"FAIL (status={status})"
    except Exception as e:
        checks["lambda_invoke"] = f"FAIL — {e}"

# Report
print("=== Pipeline Health Check ===\n")
all_pass = True
for name, result in checks.items():
    icon = "✅" if "PASS" in result else "❌"
    print(f"  {icon} {name}: {result}")
    if "FAIL" in result:
        all_pass = False

print(f"\nStatus: {'HEALTHY' if all_pass else 'NEEDS ATTENTION'}")

Exercise 4: Automate recovery

Create a script that detects problems and resolves them automatically: if the buckets don't exist, it creates them. If the Lambda isn't deployed, it deploys it. If LocalStack doesn't respond, it restarts it.

#!/bin/bash
# scripts/auto-recover.sh
set -e

echo "=== Auto-Recovery ==="

# 1. Does LocalStack respond?
echo "Checking LocalStack..."
if ! curl -s http://localhost:4566/_localstack/health > /dev/null 2>&1; then
  echo "  LocalStack not responding — restarting..."
  docker compose restart localstack
  echo "  Waiting 20 seconds..."
  sleep 20
  if ! curl -s http://localhost:4566/_localstack/health > /dev/null 2>&1; then
    echo "  FATAL: LocalStack won't start. Check docker compose logs localstack"
    exit 1
  fi
fi
echo "  OK"

# 2. Do the buckets exist?
echo "Checking buckets..."
for BUCKET in ai-input ai-output; do
  if ! awslocal s3 ls "s3://$BUCKET" > /dev/null 2>&1; then
    echo "  Creating bucket: $BUCKET"
    awslocal s3 mb "s3://$BUCKET"
  else
    echo "  Bucket $BUCKET: OK"
  fi
done

# 3. Does the Lambda exist?
echo "Checking Lambda..."
if ! awslocal lambda get-function --function-name ai-processor > /dev/null 2>&1; then
  echo "  Lambda ai-processor doesn't exist"
  if [ -f "lambda/processor.zip" ]; then
    echo "  Deploying from lambda/processor.zip..."
    awslocal lambda create-function \
      --function-name ai-processor \
      --runtime python3.11 \
      --handler processor.handler \
      --zip-file fileb://lambda/processor.zip \
      --role arn:aws:iam::000000000000:role/lambda-role \
      --timeout 90 --memory-size 768 \
      --environment "Variables={OPENAI_API_KEY=${OPENAI_API_KEY},MODEL_NAME=gpt-4o-mini,INPUT_BUCKET=ai-input,OUTPUT_BUCKET=ai-output,AWS_ENDPOINT_URL=http://host.docker.internal:4566}" > /dev/null
    echo "  Lambda deployed"
  else
    echo "  WARN: lambda/processor.zip not found — deploy manually"
  fi
else
  echo "  Lambda ai-processor: OK"
fi

echo ""
echo "=== Recovery complete ==="

Additional Troubleshooting

"LocalStack works but my app in Docker Compose won't connect"

# Inside Compose, use the service hostname, not localhost
# Your app should use: http://localstack:4566
# NOT: http://localhost:4566

# Check:
docker compose exec api env | grep AWS_ENDPOINT
# It should show: http://localstack:4566

"Data is lost on restart"

# Community Edition: limited persistence
# Solution 1: Docker volume
volumes:
  - localstack_data:/var/lib/localstack

# Solution 2: init scripts (recommended)
# Recreates everything on startup automatically
volumes:
  - ./init-scripts:/etc/localstack/init/ready.d

"awslocal returns 'command not found'"

# Install
pip install awscli-local

# If pip isn't in PATH:
python -m pip install awscli-local

# Alternative: use aws with --endpoint-url
aws --endpoint-url=http://localhost:4566 s3 ls

Summary

• The diagnostic flow is: container running → service available → request arrives → error in code vs LocalStack → known error.
• curl /_localstack/health is your first command when something fails.
• docker compose logs localstack shows what's happening internally.
• DEBUG=1 activates detailed logs of each request (useful but verbose).
• The most common errors: bucket doesn't exist, Lambda not deployed, wrong endpoint (localhost vs host.docker.internal), dependencies not packaged.
• Diagnostic scripts automate the verification and save time.
• Auto-recovery detects and resolves common problems without manual intervention.

Additional Resources

1. LocalStack Troubleshooting Guide — Official troubleshooting guide
2. LocalStack GitHub Issues — Known issues and solutions
3. Docker Compose Logs — docker compose logs reference
4. LocalStack Internal Endpoints — Diagnostic endpoints
5. Docker Debug Commands — Docker debugging commands
6. boto3 Error Handling — Error handling in boto3