7. Post-Deployment Validation

Overview

"The deploy completed" doesn't mean "the application works." The container can be running, the health endpoint can return 200, and your AI application can still be broken: the LLM doesn't respond, the responses are wrong, the cost per request tripled. Post-deployment validation is the difference between "we deployed" and "we confirmed it works."

In traditional software, an HTTP health check is enough. In AI systems, you need more: a prompt test that verifies the LLM generates coherent responses, a cost check that verifies the estimated cost didn't spike, and a latency check that verifies the response time is within acceptable limits. If any of these validations fails, the pipeline has to trigger an automatic rollback.

Connection with the final pipeline: In the capstone pipeline, post-deployment validation is the step that decides whether the deploy succeeded or whether an automatic rollback runs — it's the last line of defense before users see the change.

The levels of validation

Level 1: A Health Check (the minimum)

curl -sf https://your-app.com/health
# Expected: HTTP 200 with {"status": "ok"}

It verifies that the application is running and can respond to HTTP requests. It's the bare minimum, but it doesn't verify functionality.

Level 2: A Health Check + a Prompt Test (recommended)

# Health check
curl -sf https://your-app.com/health

# Prompt test
curl -sf -X POST https://your-app.com/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message":"What is 2+2?","max_tokens":10}'
# Expected: A response containing "4"

It verifies that the application can process a complete LLM request: receive a prompt, send it to the model, and return a response.

Level 3: Full Validation (production-grade)

# Health check
# Prompt test
# Latency check (response < 5s)
# Cost check (the estimate is within the threshold)
# Error rate check (no errors in the first 10 requests)

It verifies complete functionality including performance and costs.

The comparison of the levels

The recommendation: Level 2 for most deploys. Level 3 for critical deploys or after big changes.

The implementation: The Health Check

- name: Health check
  id: health
  continue-on-error: true
  run: |
    echo "Waiting for application startup..."
    sleep 15

    for i in $(seq 1 30); do
      STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
        https://your-app.com/health 2>/dev/null || echo "000")

      if [ "$STATUS" = "200" ]; then
        echo "Health check passed after $((15 + i*2))s"
        echo "result=healthy" >> $GITHUB_OUTPUT
        exit 0
      fi

      echo "Attempt $i/30 — HTTP $STATUS"
      sleep 2
    done

    echo "::error::Health check failed after 75s"
    echo "result=unhealthy" >> $GITHUB_OUTPUT
    exit 1

Why sleep 15 before the health check?

The container needs time to:

1. Download the image (if it isn't in the cache)
2. Start the Python/uvicorn process
3. Load the models or the configurations
4. Open the port and accept connections

15 seconds is a reasonable value for most applications. If your app takes longer to start (e.g. it loads big ML models), increase the sleep.

Why 30 attempts with 2 seconds between each one?

It's a balance between:

• Detecting fast whether the app is down (not waiting minutes)
• Tolerating a slow startup (the app can take a few seconds longer than expected)
• Not giving false positives (a single timeout doesn't mean it's down)

Total: 15s (the initial sleep) + 30×2s (the attempts) = a 75-second window.

The implementation: The Prompt Test

- name: Prompt smoke test
  id: prompt-test
  continue-on-error: true
  if: steps.health.outputs.result == 'healthy'
  run: |
    echo "Running prompt smoke test..."

    RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
      -H "Content-Type: application/json" \
      -d '{
        "message": "What is the capital of France? Answer in one word.",
        "max_tokens": 10
      }' \
      --max-time 30 2>/dev/null)

    if [ -z "$RESPONSE" ]; then
      echo "::error::AI endpoint returned empty response"
      echo "result=failed" >> $GITHUB_OUTPUT
      exit 1
    fi

    echo "Response: $RESPONSE"

    if echo "$RESPONSE" | grep -qi "paris"; then
      echo "Prompt test passed — correct response"
      echo "result=passed" >> $GITHUB_OUTPUT
    else
      echo "::warning::Unexpected response (may still be valid): $RESPONSE"
      echo "result=passed" >> $GITHUB_OUTPUT
    fi

Designing the validation prompt

The validation prompt has to be:

• ✅ Deterministic: A question with an obvious answer that any model answers the same way
• ✅ Fast: Few output tokens (max_tokens: 10)
• ✅ Cheap: A single request, modest in tokens
• ✅ Verifiable: The response can be validated programmatically

Good prompts for validation:

Bad prompts for validation:

The implementation: The Latency Check

- name: Latency check
  id: latency
  continue-on-error: true
  if: steps.health.outputs.result == 'healthy'
  run: |
    echo "Running latency check..."

    TOTAL_MS=0
    REQUESTS=5

    for i in $(seq 1 $REQUESTS); do
      START=$(date +%s%N)
      curl -sf -X POST https://your-app.com/api/chat \
        -H "Content-Type: application/json" \
        -d '{"message":"ping","max_tokens":5}' \
        --max-time 10 > /dev/null 2>&1
      END=$(date +%s%N)

      MS=$(( (END - START) / 1000000 ))
      TOTAL_MS=$((TOTAL_MS + MS))
      echo "  Request $i: ${MS}ms"
    done

    AVG_MS=$((TOTAL_MS / REQUESTS))
    echo "Average latency: ${AVG_MS}ms"
    echo "avg_ms=$AVG_MS" >> $GITHUB_OUTPUT

    if [ "$AVG_MS" -gt 5000 ]; then
      echo "::error::Average latency ${AVG_MS}ms exceeds 5000ms threshold"
      echo "result=slow" >> $GITHUB_OUTPUT
      exit 1
    else
      echo "Latency check passed: ${AVG_MS}ms < 5000ms"
      echo "result=ok" >> $GITHUB_OUTPUT
    fi

Latency thresholds for AI systems

AI endpoints are inherently slower than traditional endpoints because they involve a call to an external model. A response time of 2-3 seconds for a simple prompt is normal.

The complete integrated validation

  post-deploy-validation:
    name: "Post-Deploy Validation"
    needs: deploy-production
    runs-on: ubuntu-latest
    timeout-minutes: 5
    outputs:
      status: ${{ steps.result.outputs.status }}

    steps:
      - name: Health check
        id: health
        continue-on-error: true
        run: |
          sleep 15
          for i in $(seq 1 30); do
            STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
              https://your-app.com/health 2>/dev/null || echo "000")
            if [ "$STATUS" = "200" ]; then
              echo "result=healthy" >> $GITHUB_OUTPUT
              exit 0
            fi
            sleep 2
          done
          echo "result=unhealthy" >> $GITHUB_OUTPUT
          exit 1

      - name: Prompt test
        id: prompt
        continue-on-error: true
        if: steps.health.outputs.result == 'healthy'
        run: |
          RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
            -H "Content-Type: application/json" \
            -d '{"message":"What is 2+2?","max_tokens":10}' \
            --max-time 30 2>/dev/null)
          if [ -n "$RESPONSE" ]; then
            echo "result=passed" >> $GITHUB_OUTPUT
          else
            echo "result=failed" >> $GITHUB_OUTPUT
            exit 1
          fi

      - name: Determine result
        id: result
        if: always()
        run: |
          HEALTH="${{ steps.health.outputs.result }}"
          PROMPT="${{ steps.prompt.outputs.result }}"

          if [ "$HEALTH" = "healthy" ] && [ "$PROMPT" = "passed" ]; then
            echo "status=success" >> $GITHUB_OUTPUT
            echo "All validations passed"
          else
            echo "status=failed" >> $GITHUB_OUTPUT
            echo "::error::Validation failed — Health: $HEALTH, Prompt: $PROMPT"
          fi

      - name: Validation summary
        if: always()
        run: |
          echo "## Post-Deploy Validation" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Health | ${{ steps.health.outputs.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Prompt | ${{ steps.prompt.outputs.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "**Final status:** ${{ steps.result.outputs.status }}" >> $GITHUB_STEP_SUMMARY

  auto-rollback:
    name: "Auto-Rollback"
    needs: [post-deploy-validation, deploy-production]
    runs-on: ubuntu-latest
    if: needs.post-deploy-validation.outputs.status == 'failed'

    steps:
      - name: Execute rollback
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          PREV="${{ needs.deploy-production.outputs.previous-tag }}"
          echo "Auto-rollback to: $PREV"

          if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
            echo "::error::Cannot rollback — no previous version"
            exit 1
          fi

          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
          rm /tmp/key

      - name: Notify rollback
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "⚠️ Auto-rollback executed. Previous version restored."
            }

What to do if the validation fails

The decision tree

The validation fails
  │
  ├─ The health check fails
  │   ├─ The app doesn't start → Check the container's logs
  │   ├─ A port error → Check the docker-compose ports
  │   └─ A timeout → Increase the startup wait
  │
  ├─ The prompt test fails
  │   ├─ An empty response → An expired API key or rate limiting
  │   ├─ A wrong response → The model endpoint changed
  │   └─ A timeout → An LLM provider outage
  │
  └─ The latency check fails
      ├─ Consistently slow → Resource constraints
      └─ Intermittent → Network issues

Comparisons

Pre-deploy vs post-deploy validation

A health check vs a smoke test vs an integration test

Troubleshooting

"The health check passes but the prompt test fails"

Cause: The app is running but it can't connect to the LLM provider.

Diagnosis:

ssh deploy@your-server "cd /app && docker compose logs --tail 20"

Look for errors like: AuthenticationError, RateLimitError, ConnectionError.

Solution: Verify that the OPENAI_API_KEY secret in production is valid and has credits.

"The validation passes in staging but fails in production"

Cause: Differences between the environments: different API keys, different service URLs, different configurations.

Solution: Verify that the secrets and variables in staging and production are consistent.

"The prompt test gives different results every time"

Cause: LLM non-determinism.

Solution: Use prompts with extremely predictable responses ("What is 2+2?") and flexible validation (grep -qi "4" instead of an exact comparison).

"The latency is only high on the first request"

Cause: A cold start — the first request initializes connections, loads models, etc.

Solution: Add a "warmup" request before measuring latency, or discard the first request from the measurement.

Exercises

Exercise 1: Implement a health check with retries

Write a health check script that makes 3 retries before declaring a failure.

- name: Health check with retry
  run: |
    RETRIES=3
    for attempt in $(seq 1 $RETRIES); do
      echo "Attempt $attempt/$RETRIES"
      sleep $((attempt * 10))

      for i in $(seq 1 10); do
        if curl -sf https://your-app.com/health > /dev/null 2>&1; then
          echo "Healthy on attempt $attempt"
          exit 0
        fi
        sleep 2
      done
      echo "Attempt $attempt failed"
    done
    echo "::error::Health check failed after $RETRIES attempts"
    exit 1

Exercise 2: A prompt test with semantic validation

Write a prompt test that verifies the response is semantically correct, not just that it isn't empty.

- name: Semantic prompt test
  run: |
    RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
      -H "Content-Type: application/json" \
      -d '{"message":"List the first 3 prime numbers, separated by commas.","max_tokens":20}' \
      --max-time 30 2>/dev/null)

    echo "Response: $RESPONSE"

    VALID=true
    echo "$RESPONSE" | grep -q "2" || VALID=false
    echo "$RESPONSE" | grep -q "3" || VALID=false
    echo "$RESPONSE" | grep -q "5" || VALID=false

    if [ "$VALID" = "true" ]; then
      echo "Semantic validation passed"
    else
      echo "::error::Expected response containing 2, 3, 5 but got: $RESPONSE"
      exit 1
    fi

Exercise 3: Complete validation with auto-rollback

Combine a health check + a prompt test + auto-rollback into one complete flow.

- name: Full validation
  id: validate
  continue-on-error: true
  run: |
    sleep 15

    echo "Step 1: Health check"
    for i in $(seq 1 20); do
      if curl -sf https://your-app.com/health > /dev/null 2>&1; then
        echo "Health: OK"
        break
      fi
      [ "$i" = "20" ] && { echo "Health: FAILED"; exit 1; }
      sleep 2
    done

    echo "Step 2: Prompt test"
    RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
      -H "Content-Type: application/json" \
      -d '{"message":"ping","max_tokens":5}' --max-time 30 || echo "")
    [ -z "$RESPONSE" ] && { echo "Prompt: FAILED"; exit 1; }
    echo "Prompt: OK"

    echo "All validations passed"

- name: Auto-rollback
  if: steps.validate.outcome == 'failure'
  run: |
    PREV="${{ steps.current.outputs.tag }}"
    [ "$PREV" = "unknown" ] && { echo "Cannot rollback"; exit 1; }
    echo "Rolling back to $PREV"
    # ... the rollback commands ...
    exit 1

Validation best practices

1. Design idempotent validation prompts

The validation prompt will run on every deploy. It must have no side effects:

# ❌ Bad: it creates data in the database
{"message": "Create a test user", "max_tokens": 100}

# ✅ Good: it only generates text, with no side effects
{"message": "Reply with the word 'pong'", "max_tokens": 5}

2. Don't depend on the LLM's exact responses

LLMs are probabilistic. Don't validate against an exact response:

# ❌ Fragile: it expects the exact text
[ "$RESPONSE" = "pong" ] || exit 1

# ✅ Robust: it verifies that the response isn't empty and has the expected content
echo "$RESPONSE" | grep -qi "pong" || exit 1

Or better yet, validate the response's structure (it has valid JSON, it has the expected fields) instead of the literal content.

3. Include a "canary request" for cold starts

AI systems usually have cold starts from loading models. Make an initial "warmup" request before the real test:

# The warmup request (ignore the result)
curl -sf -X POST https://your-app.com/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message":"warmup","max_tokens":1}' \
  --max-time 60 || true

sleep 5

# The real test with a latency threshold
START=$(date +%s%N)
RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message":"ping","max_tokens":5}' \
  --max-time 30)
END=$(date +%s%N)
LATENCY=$(( (END - START) / 1000000 ))  # ms

echo "Latency: ${LATENCY}ms"
[ "$LATENCY" -lt 10000 ] || { echo "Too slow"; exit 1; }

4. Define realistic validation windows

Don't expect the service to respond in milliseconds immediately after the deploy:

Adjust the retries and the timeouts based on your type of service.

5. Monitor false positives

If the validation frequently fails from OpenAI API timeouts (not from problems in your app), you're generating unnecessary rollbacks. Add a retry specifically for API errors:

for attempt in 1 2 3; do
  RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
    -d '{"message":"ping","max_tokens":5}' \
    --max-time 30 2>/dev/null || echo "")

  if [ -n "$RESPONSE" ]; then
    echo "Validation passed on attempt $attempt"
    exit 0
  fi
  echo "Attempt $attempt failed, retrying..."
  sleep 10
done
echo "All attempts failed"
exit 1

Summary

• ✅ "The deploy completed" ≠ "the application works" — post-deploy validation closes that gap
• ✅ Three levels: A health check (the minimum), + a prompt test (recommended), + full validation (production-grade)
• ✅ The validation prompt has to be deterministic, fast, cheap, and verifiable
• ✅ A health check with retries tolerates slow startups without giving false positives
• ✅ If the validation fails → auto-rollback to the previous version
• ✅ Latency thresholds for AI systems are higher than for traditional software (~2-5s is normal)
• ✅ Pre-deploy (staging) and post-deploy (production) are complementary, not substitutes
• ✅ An immediate notification when the validation fails and the rollback runs

Additional resources

1. Health Check Patterns - Health check patterns for microservices
2. GitHub Actions — continue-on-error - Handling errors in steps
3. Smoke Testing - What smoke testing is and how to implement it
4. OpenAI API Status - OpenAI's status page for diagnosing outages
5. curl Timeout Options - curl's timeout options
6. Canary Deployments - The advanced pattern for gradual deployment