6. Post-deploy monitoring

What this capsule covers

Your pipeline deploys and verifies with a health check at the end. But the health check is a moment, not a process. 5 minutes after the deploy, the app can degrade. 30 minutes later, it can have a high error rate without anyone noticing. 2 hours later, users report problems.

This capsule integrates the basic monitoring that closes the CI/CD loop: a robust health check (not just HTTP 200), error tracking with Sentry, basic metrics with Datadog/Prometheus, and alerts to Slack when something degrades. It isn't the complete observability guide (that's #14 in the path) — it's the minimum monitoring necessary for professional CI/CD.

By the end, you'll be able to:

• Implement a robust /health endpoint that validates real dependencies
• Integrate Sentry for automatic capture of unhandled errors
• Configure synthetic monitoring (uptime checks every N minutes)
• Distinguish between observability (continuous visibility) and monitoring (reactive alerts)
• Decide which metrics justify alerts and which to only log
• Build useful alerts that don't generate alert fatigue

The problem: a "successful" deploy but silent degradation

A typical case:

14:00 — Deploy to production
14:00 — Post-deploy health check: 200 OK ✅
14:01 — Slack: "🚀 Deploy successful"

15:00 — Error rate rises from 0.5% to 4%
15:00 — Nobody finds out
15:30 — p95 latency rises from 200ms to 1.5s
15:30 — Nobody finds out
16:00 — The container's memory reaches 95%
16:00 — Nobody finds out
16:30 — The container starts crashing (OOM)
17:00 — Users report on Twitter
17:30 — Someone investigates, identifies a memory leak introduced in the deploy
18:00 — Rollback executed

3 hours of silent degradation. The deploy health check was green — but it wasn't monitoring after that moment.

What's missing: continuous observability.

The mental model: three levels of monitoring

Level 1: Health checks (does the app respond?)
├── Who: your CI/CD workflow, uptime monitors
├── When: post-deploy + every 1-5 min
├── Output: HTTP 200 or not
└── Example: GET /health → 200

Level 2: Error tracking (does the app have bugs?)
├── Who: Sentry, Bugsnag, Rollbar
├── When: every unhandled exception at runtime
├── Output: stacktrace + context + frequency
└── Example: ZeroDivisionError in checkout_handler

Level 3: Metrics and traces (how fast is the app?)
├── Who: Datadog, New Relic, Prometheus
├── When: every request (sampled)
├── Output: latency p50/p95/p99, throughput, custom metrics
└── Example: p95 latency 350ms, 200 RPS

Professional CI/CD needs all 3 levels, not just the first.

This capsule configures levels 1 and 2 with free/cheap SaaS solutions. Level 3 is the complete observability guide (#14).

Step 1: robust health check

Your current endpoint is probably:

@app.get("/health")
def health():
    return {"status": "ok"}

It works but doesn't verify dependencies. The app responds even if the DB is down.

Production-ready health check:

from sqlalchemy import text
from sqlalchemy.orm import Session

@app.get("/health")
def health():
    """Liveness check — the app is alive."""
    return {"status": "ok"}

@app.get("/health/ready")
def readiness(db: Session = Depends(get_db)):
    """Readiness check — the app can serve requests."""
    checks = {}
    failed = False

    # Check DB
    try:
        db.execute(text("SELECT 1"))
        checks["database"] = {"status": "ok"}
    except Exception as e:
        checks["database"] = {"status": "error", "message": str(e)[:100]}
        failed = True

    # Check Redis (if you use it)
    try:
        from app.cache import redis_client
        redis_client.ping()
        checks["redis"] = {"status": "ok"}
    except Exception as e:
        checks["redis"] = {"status": "error", "message": str(e)[:100]}
        failed = True

    response = {"status": "ok" if not failed else "degraded", "checks": checks}

    if failed:
        raise HTTPException(status_code=503, detail=response)
    return response

Critical distinction:

• /health (liveness): is the process alive? It responds 200 if the process responds, regardless of the state of dependencies. Used by:

  • Container orchestrators (Railway, K8s) to decide whether to restart
  • Load balancers to decide whether to send traffic

• /health/ready (readiness): can the app serve requests? It responds 200 only if ALL the critical dependencies work. Used by:

  • Post-deploy health checks in CI/CD
  • Synthetic monitors

Why two endpoints:

If your DB has a transient blip (network issue, momentary lock):

• /health keeps returning 200 → the container does NOT restart
• /health/ready returns 503 → the load balancer can temporarily remove this pod

Without this distinction:

• Only /health: container restarted by a blip → you lose 30s of service
• Only /health/ready: dependencies down → container restarted unnecessarily

Recommended pattern: both endpoints, used by different systems.

Step 2: integrate Sentry

Sentry is the most common tool for error tracking in Python. Generous free tier (5k errors/month).

Setup in 3 steps

1. Create an account and project:

• sentry.io → New Project → Python → FastAPI
• Sentry gives you a DSN: https://abc123@o123.ingest.sentry.io/456

2. Install and configure:

pip install sentry-sdk[fastapi]

# app/main.py
import os
import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.sqlalchemy import SqlalchemyIntegration

sentry_sdk.init(
    dsn=os.environ.get("SENTRY_DSN"),
    integrations=[
        FastApiIntegration(),
        SqlalchemyIntegration(),
    ],
    environment=os.environ.get("DEPLOYMENT_ENV", "development"),
    release=os.environ.get("DEPLOYMENT_VERSION", "unknown"),
    traces_sample_rate=0.1,  # 10% of requests for performance monitoring
    profiles_sample_rate=0.1,
)

app = FastAPI(title="My App")

3. Configure env vars in Railway (per environment):

SENTRY_DSN = https://...@sentry.io/...
DEPLOYMENT_ENV = production  (or staging, dev)
DEPLOYMENT_VERSION = ${{ github.sha }}

What Sentry does automatically

Without writing more code, Sentry captures:

• Unhandled exceptions: any raise you don't catch
• HTTP 500 responses: errors returned by FastAPI
• Stack traces: complete with local variables
• Request context: URL, method, headers (sanitized)
• User context: if you configure set_user
• Performance: request latency if traces_sample_rate > 0

When an error occurs, Sentry notifies you:

• Configurable Slack notification
• Email
• PagerDuty (for incidents)
• Dashboard with frequency, affected users, environments

Custom captures

Sometimes you want to capture events that aren't exceptions:

from sentry_sdk import capture_message, capture_exception

@app.post("/payments")
def process_payment(payment: PaymentRequest):
    try:
        result = charge_card(payment)
        if result.status == "warning":
            # Capture as a warning, not an error
            capture_message(
                "Payment processed with warning",
                level="warning",
                extras={"payment_id": result.id, "warning": result.warning_msg}
            )
        return result
    except CardDeclined as e:
        # Expected error, log but don't capture in Sentry
        logger.info(f"Card declined: {e}")
        raise HTTPException(status_code=400, detail="Card declined")
    except Exception as e:
        # Unexpected error, capture in Sentry
        capture_exception(e)
        raise

Pattern: expected errors (validation, business logic) go to logs. Unexpected errors (bugs) go to Sentry.

Filter noise

By default, Sentry captures everything. This can generate noise:

sentry_sdk.init(
    dsn=...,
    ignore_errors=[
        "KeyboardInterrupt",   # Ctrl-C in dev
        "ConnectionRefusedError",  # client disconnected
    ],
    before_send=before_send_filter,
)

def before_send_filter(event, hint):
    """Custom logic to filter events."""
    # Don't report 404s
    if "exc_info" in hint:
        exc_type, exc_value, _ = hint["exc_info"]
        if isinstance(exc_value, HTTPException):
            if exc_value.status_code == 404:
                return None  # drop the event
    return event

Step 3: synthetic monitoring (uptime checks)

Your app can be responding OK locally but be blocked in the cloud by:

• DNS issues
• An expired SSL cert
• Firewall changes
• ISP routing issues

Synthetic monitors are automatic requests from multiple locations every N minutes. If they fail, alert.

Free options

1. UptimeRobot (50 monitors free):

• You configure URL + interval (5 min minimum on free)
• If it fails, alert by email/Slack/SMS
• The free tier is enough for hobby projects

2. Better Stack (Better Uptime) (10 monitors free, 30s checks):

• More professional, nice dashboards
• 30s interval on the free tier

3. Cronitor (5 monitors free):

• Specialized in monitoring cron jobs and APIs

Typical setup (UptimeRobot):

1. Sign up
2. Add monitor:
   • Type: HTTPS
   • URL: https://yourapp.com/health/ready
   • Interval: 5 min
   • Alert contacts: your email + Slack webhook
3. Test that it works

Why /health/ready and not /health

The synthetic monitor should detect:

• App not responding → alert
• App responding but DB down → alert
• App responding well → silence

/health/ready verifies dependencies. If the DB goes down, it returns 503 → UptimeRobot alerts.

/health returns 200 even with the DB down → UptimeRobot doesn't alert until the container dies.

Step 4: integrate monitoring into the pipeline

After the deploy, the workflow can validate that the monitoring is OK:

deploy-production:
  # ... existing deploy steps ...

  - name: Smoke test post-deploy
    run: |
      DEPLOY_URL="${{ secrets.DEPLOY_URL }}"

      # Test 1: liveness
      curl -f "$DEPLOY_URL/health" || exit 1

      # Test 2: readiness (DB + dependencies)
      curl -f "$DEPLOY_URL/health/ready" || exit 1

      # Test 3: smoke test of a critical endpoint
      RESPONSE=$(curl -s "$DEPLOY_URL/api/version")
      VERSION=$(echo $RESPONSE | jq -r .version)
      if [ "$VERSION" != "${{ github.sha }}" ]; then
        echo "❌ Wrong version deployed: $VERSION (expected ${{ github.sha }})"
        exit 1
      fi

      echo "✅ Smoke tests passed"

  - name: Notify Sentry of release
    run: |
      curl -X POST "https://sentry.io/api/0/organizations/YOUR_ORG/releases/" \
        -H "Authorization: Bearer ${{ secrets.SENTRY_AUTH_TOKEN }}" \
        -H "Content-Type: application/json" \
        -d '{
          "version": "${{ github.sha }}",
          "projects": ["my-app"],
          "ref": "${{ github.sha }}"
        }'

Post-deploy smoke tests:

• Verify critical endpoints, not just /health
• Verify the deployed version (detect deployment failing silently)
• If it fails, the red job → auto-rollback (from capsule 07 M4)

Notify Sentry:

Creates a "release" in Sentry linked to the SHA. Benefits:

• Errors are grouped by release
• Sentry knows which release introduced each bug
• Comparison between releases ("v1.2 has 2× more errors than v1.1")

Step 5: useful alerts vs alert fatigue

Alert fatigue is real: if Slack pings you 50 times/day, you eventually ignore everything.

Rules for useful alerts:

Rule 1: alert only what's actionable

❌ "Error rate is 0.6% (baseline 0.5%)" — not actionable, normal fluctuation

✅ "Error rate is 5% (10× baseline)" — actionable, clearly something is wrong

Rule 2: thresholds based on the baseline

Static: error_rate > 1%

Dynamic: error_rate > 3× last 7d average

The dynamic one adapts to the nature of your app. Apps with a naturally high error rate (1-2%) don't drown in alerts.

Rule 3: deduplication

If the same alert occurs 100 times in 1 hour, send a single notification with a count instead of 100 individual ones.

Rule 4: different routes by severity

Critical (prod down):     → PagerDuty (call/SMS)
High (error rate spike):  → Slack #alerts (notif + @here)
Medium (slow response):   → Slack #monitoring (silent notif)
Low (degraded):           → Dashboard, no Slack

Not every problem requires waking someone at 3 AM.

Example: configuring alerts in Sentry

Sentry has alert rules:

Rule: "New issue in production"
Conditions:
  - environment: production
  - level: error
  - is_unhandled: true
Action:
  - Send to Slack channel #engineering-alerts
  - Include: title, environment, link

Rule: "Error spike detected"
Conditions:
  - issue: any
  - frequency in 5 min > 10× frequency in last 24h
Action:
  - Send to Slack #engineering-alerts with @here
  - Create a JIRA ticket

Pattern: immediate alerts for new issues + spike alerts for known issues.

Step 6: minimal dashboard

Even if you don't implement full Datadog, having a basic dashboard helps:

Sentry dashboard (free with your account):

• Issue count by environment (last 24h)
• Top 10 issues by frequency
• Recent releases with error counts
• Affected users

UptimeRobot dashboard:

• Uptime % last 24h, 7d, 30d
• Response time graph
• Recent incidents

Railway dashboard:

• CPU/Memory per service
• Real-time logs
• Deployment history

Three dashboards, free, enough for monitoring an early-stage app.

For apps with real traffic, add Datadog/Grafana for custom metrics. That's guide #14 Monitoring & Observability.

Common traps

1. A health check that checks too much

@app.get("/health/ready")
def readiness():
    # ❌ check 10 external services
    check_stripe()
    check_sendgrid()
    check_aws_s3()
    check_external_api_1()
    # ...

If Stripe has a 30s blip, your health check fails → the load balancer removes your app → users without an app.

How to handle it: check only dependencies critical for serving requests: DB, cache. External services (Stripe, S3) should have fallbacks or circuit breakers, not affect readiness.

2. Sentry DSN in code

sentry_sdk.init(dsn="https://abc123@sentry.io/456")  # ❌ hardcoded

DSN exposed. Anyone with access to the code can send fake events to your Sentry.

How to handle it: an env var (SENTRY_DSN), never hardcoded.

3. traces_sample_rate: 1.0

sentry_sdk.init(
    traces_sample_rate=1.0,  # ❌ captures 100% of requests
)

In production with real traffic, this:

• Exceeds the Sentry quota quickly
• Adds latency to each request

How to handle it: a low sample (0.1 = 10%) in production. Enough data without killing the quota.

4. Not configuring environment in Sentry

sentry_sdk.init(dsn=...)
# Without environment configured

Dev, staging, and prod errors appear mixed together.

How to handle it:

sentry_sdk.init(
    dsn=...,
    environment=os.environ.get("DEPLOYMENT_ENV", "development"),
)

Sentry filters by environment. You can see only production errors.

5. An alert for every 500

Alert rule: "Any HTTP 500"

500s happen every day for various reasons. A noisy alert.

How to handle it: alert only if:

• The 500 rate > 5% (spike)
• Or: a new type of error never seen before
• Or: the 500s affect a critical endpoint (payments, login)

6. Synthetic monitor every 5 seconds

UptimeRobot interval: 5 seconds (paid feature)

If your app responds in 200ms and you test every 5s, that's 17280 requests/day just for monitoring. Cost in Railway: significant.

How to handle it: intervals of 1-5 min are enough. If you need more granularity, use APM (Datadog) instead of uptime checks.

Worked case: the incident monitoring caught

Without monitoring (scenario A):

14:00 — Deploy v2.5 to production
14:01 — Health check green
14:01 — Slack: "🚀 Deploy successful"

15:00 — v2.5 introduced a memory leak. Memory grows linearly
17:00 — Containers reach 80% memory
19:00 — OOM kills start, containers restart
       Users lose sessions
21:00 — Slack messages from users: "it crashes all the time"
21:30 — Someone investigates
22:00 — Identifies the memory leak in v2.5
22:30 — Rollback executed

8 hours of degradation. It started silent.

With monitoring (scenario B):

14:00 — Deploy v2.5
14:01 — Health check green
14:01 — Slack: "🚀 Deploy successful"

14:30 — Memory leak introduced by v2.5
15:00 — Datadog detects a memory growth anomaly
       Memory on deploy v2.5 is 30% above the v2.4 baseline
15:00 — Slack alert in #engineering-alerts:
       "⚠️ Memory growth anomaly detected
        Service: my-app-prod
        Current: 78% (last 24h avg: 45%)
        Trend: +5% per hour
        Started: ~14:30 (correlates with deploy v2.5)
        Link to dashboard: [...]"

15:05 — Engineer sees the alert, investigates
15:15 — Identifies the memory leak
15:20 — Decides to roll back (via the module 4 rollback workflow)
15:25 — Rollback complete, memory drops

Total degradation: 25 min
Total user impact: minimal (memory hadn't yet reached the limit)

Difference: 8 hours → 25 minutes. And zero downtime for users.

The secret: continuous monitoring + correlation with deploys (Sentry releases) + proactive alerts.

Exercise: basic monitoring

1. Health checks:

   • Implement /health (liveness) and /health/ready (readiness)
   • Configure Railway/Render to use /health/ready as the health check

2. Sentry:

   • Create an account at sentry.io
   • Configure the Sentry SDK in your app with env vars
   • Add SENTRY_DSN to the GitHub Secrets of each environment
   • Verify: trigger an intentional error in dev, see it appear in Sentry

3. UptimeRobot:

   • Sign up
   • Add monitor: https://yourapp.com/health/ready every 5 min
   • Configure an alert to Slack or email

4. Alert rules in Sentry:

   • Rule: "New issue in production" → Slack
   • Rule: "Error spike" → Slack with @here

5. Validate the setup:

   • Trigger an error in staging → see the Sentry alert
   • Turn off the DB temporarily → see the UptimeRobot alert
   • Validate that dev/staging don't alert the same channel as production

Self-check

Railway/K8s use: /health (liveness)
CI/CD post-deploy uses: /health/ready (readiness)
UptimeRobot uses: /health/ready

def handler():
    user_id = get_random_id()
    raise Exception(f"Error for user {user_id}")  # ❌ a unique message every time

raise Exception("Error in handler", extra={"user_id": user_id})  # ✅

@app.exception_handler(NotFoundError)
def handle_404(request, exc):
    return JSONResponse(...)  # ✅ this doesn't generate a Sentry event

# But if your code does:
raise NotFoundError("User 123 not found")  # ❌ the raise creates a Sentry event

def before_send(event, hint):
    if "NotFoundError" in str(hint.get("exc_info", "")):
        return None
    return event

GET /admin/.env → 404
GET /wp-admin/ → 404
GET /phpmyadmin/ → 404

Summary and next step

• Basic monitoring for CI/CD: robust health checks + Sentry + UptimeRobot
• /health (liveness) and /health/ready (readiness) serve different systems
• Sentry captures unhandled exceptions automatically; configure environment and release
• Synthetic monitors detect problems that post-deploy health checks don't catch
• Alert fatigue is real — alert only when actionable, deduplicate, separate by severity
• This is minimum monitoring for CI/CD. Complete observability is guide #14.

Bridge to the next step: Your pipeline is complete: modular, optimized, maintained, monitored. But it's still knowledge exclusive to you. In capsule 07 you're going to document everything — a professional README with badges, deployment and rollback runbooks, a troubleshooting guide. What turns your repo into something any developer on the team can understand and operate.

Resources

1. Sentry FastAPI integration — official reference.
2. Health checks pattern — the formalized pattern.
3. UptimeRobot — synthetic monitoring free tier.
4. Better Stack — a more modern alternative.
5. Observability Engineering (Honeycomb) — the modern observability book.

Capsule 06 of 08 — Module 5 — CI/CD for Python Backend Guide