Webhooks Inbound

Capsule overview

Inbound webhooks are how your system receives automatic notifications from external systems when something happens there. Slack notifies you of a message. Stripe notifies you of a payment. GitHub notifies you of a commit. Instead of your system constantly asking "did something happen?" (polling), the external system proactively tells you when something happens (push). For AI systems, inbound webhooks are the canonical entry pattern: practically every Slack bot, Discord integration, or connector with external apps uses inbound webhooks.

The concept sounds trivial — "it's just receiving an HTTP request" — but the correct implementation has several subtleties that differentiate a fragile webhook handler from a production-ready one: signature verification (protect against malicious requests), validation challenges (Slack and others require an initial handshake), timeout responses (you must respond fast even if the processing is long), idempotency (the same event can arrive duplicated), and replay handling. Each of these is typically learned with a bug in production if not planned from design.

In this capsule you learn the complete flow of inbound webhooks applied to AI: registration with the provider, identity verification (signing secrets), the fast-response + async-processing pattern, error handling, and common pitfalls. The canonical case will be the Slack Events API, but the principles apply to any webhook (Stripe, GitHub, Twilio).

---

The complete flow

1. Registration with the provider

Before receiving webhooks, the provider must know your endpoint URL:

Slack admin panel → Create app → Event Subscriptions
   → Request URL: https://your-api.com/webhooks/slack
   → Subscribe to events: app_mention, message.im, etc.

The provider typically requires:

• A public and accessible URL (not localhost)
• HTTPS (not plain HTTP)
• The ability to respond to the verification challenge

2. Verification challenge (initial handshake)

When you register the URL, the provider sends a verification request to confirm that you control the URL:

# Slack sends:
POST /webhooks/slack
{
    "type": "url_verification",
    "challenge": "abc123def456",
    "token": "..."
}

# Your API must respond:
200 OK
{
    "challenge": "abc123def456"
}

Without a correct handshake, registration fails. Replaceable for Slack but similar for almost all providers.

3. Receive real events

After the handshake, the provider sends events:

POST /webhooks/slack
Headers:
  X-Slack-Signature: v0=abc123...
  X-Slack-Request-Timestamp: 1736000000

Body:
{
    "type": "event_callback",
    "event": {
        "type": "app_mention",
        "user": "U123",
        "text": "<@BOT> what is RAG?",
        "channel": "C456",
        "ts": "1736000000.000100"
    }
}

4. Verify signature

CRITICAL: verify that the request really comes from the provider, not from an attacker:

import hmac
import hashlib
import time

def verify_slack_signature(
    body: bytes, 
    timestamp: str, 
    signature: str,
    signing_secret: str
) -> bool:
    # Reject old requests (replay attack protection)
    if abs(time.time() - int(timestamp)) > 60 * 5:
        return False
    
    # Compute expected signature
    base_string = f"v0:{timestamp}:{body.decode('utf-8')}"
    expected = "v0=" + hmac.new(
        signing_secret.encode(),
        base_string.encode(),
        hashlib.sha256
    ).hexdigest()
    
    # Constant-time comparison (prevent timing attacks)
    return hmac.compare_digest(expected, signature)

Without this, anyone with your URL can trigger your system with fake events.

5. Respond fast

The provider typically requires a response within 3 seconds (Slack), 10 seconds (Stripe), etc. If you take longer, it considers the delivery failed, retries, and possibly disables your webhook after many failures.

That's why the typical AI pattern is: fast ACK + async processing (we go deeper in capsule 04).

@app.post("/webhooks/slack")
async def slack_webhook(request: Request):
    body = await request.body()
    
    # 1. Verify signature
    timestamp = request.headers.get("X-Slack-Request-Timestamp")
    signature = request.headers.get("X-Slack-Signature")
    if not verify_slack_signature(body, timestamp, signature, SLACK_SIGNING_SECRET):
        raise HTTPException(401, "Invalid signature")
    
    payload = await request.json()
    
    # 2. Handle verification challenge (initial setup)
    if payload.get("type") == "url_verification":
        return {"challenge": payload["challenge"]}
    
    # 3. Handle real event - enqueue for async processing
    if payload.get("type") == "event_callback":
        event = payload["event"]
        if event["type"] == "app_mention":
            await queue.enqueue({
                "user": event["user"],
                "text": event["text"],
                "channel": event["channel"],
                "ts": event["ts"]
            })
    
    # 4. Respond ASAP - actual processing happens in the worker
    return {"ok": True}

6. Process async

The worker pulls from the queue, processes with the LLM, and posts the response back to Slack via an API call (not via the bot's outbound webhook).

---

AI-specific cases

Case 1: Slack bot for Q&A

The inbound webhook receives messages mentioning the bot:

User: @ai_bot how do I configure Redis?
↓
Slack webhook → your API
↓ fast ACK
Queue
↓
Worker pulls
↓
RAG retrieval + LLM call (5s)
↓
Worker calls the Slack chat.postMessage API
↓
User sees the response in the thread

Case 2: Discord bot with slash commands

Discord uses the interactions API (a variation of webhooks):

User: /ask how do I configure Redis?
↓
Discord webhook → your API (3 sec timeout)
↓ ACK with "deferred response" (extended to 15 min)
↓
Async processing
↓
Worker calls the Discord followup endpoint
↓
User sees the response

Case 3: GitHub events for auto-PR review

PR opened in a repo
↓
GitHub webhook → your API
↓ ACK
↓
Worker: clone diff, RAG over history, LLM analyzes
↓
Comment on the PR via the GitHub API

Case 4: Sentry / error tracking integration

Error occurs in the app
↓
Sentry webhook → your API
↓ ACK
↓
Worker: classify error, RAG over similar issues, LLM suggests a fix
↓
Comment on Sentry or create a Linear ticket

---

Common pitfalls and mistakes

Pitfall 1: Processing synchronously when the provider has a timeout

A junior engineer receives the webhook, makes an LLM call directly, returns a response. The LLM takes 5s. The Slack timeout is 3s. Slack retries 3 times. Your system processes the same event 3 times, duplicated LLM calls, possible duplicated responses to the user.

How to detect it: does your webhook handler do heavy processing before responding? Have you seen events that seem to be processed multiple times?

How to fix it: separate receipt from processing. Receipt: verify signature, enqueue, return 200. Processing: async worker.

Pitfall 2: Not verifying signatures

An engineer omits verification "because only Slack sends to this URL". An attacker finds the URL (it's not secret), sends fake requests. Your system processes fake events as real.

How to detect it: does your webhook handler verify the signature with HMAC? If it only verifies the format of the payload, it's not protection.

How to fix it: signature verification as the first step. Constant-time comparison. Reject old timestamps.

Pitfall 3: Ignoring idempotency

The same event can arrive several times (provider retries, network issues). If your handler does non-idempotent operations, duplicated side effects.

How to detect it: do your webhooks execute actions with externally visible effects? What happens if they receive the same event twice?

How to fix it: each webhook has a unique event ID. Track processed IDs (Redis with 24h TTL). Skip if the ID is already processed.

async def process_event(event_id: str, payload: dict):
    if await redis.exists(f"processed:{event_id}"):
        return  # Already processed
    
    # Process
    await actually_process(payload)
    
    # Mark processed
    await redis.setex(f"processed:{event_id}", 86400, "1")

Pitfall 4: Webhook handler as a single point of failure

If your webhook handler crashes, events are lost. The provider will retry but limitedly. Then data lost.

How to detect it: what happens if your webhook handler is down for 10 minutes?

How to fix it: real HA (multiple instances), aggressive monitoring (alerts on errors), a DLQ for events that fail repeatedly.

Pitfall 5: Complete logging of the payload

Your webhook payloads have sensitive data (PII, tokens, etc.). Complete logging to Datadog means that data travels outside your security boundary.

How to detect it: do your logs have complete webhook payloads? Does any external logging service see that data?

How to fix it: log metadata (timestamps, IDs, types), not full payloads. If you need to debug, log to internal storage only.

---

Worked example: Slack bot Q&A end-to-end

Complete implementation:

import os
import hmac
import hashlib
import time
import json
from fastapi import FastAPI, Request, HTTPException
import redis.asyncio as redis

app = FastAPI()
r = redis.from_url("redis://localhost:6379")

SLACK_SIGNING_SECRET = os.environ["SLACK_SIGNING_SECRET"]

def verify_slack_signature(body: bytes, timestamp: str, signature: str) -> bool:
    if abs(time.time() - int(timestamp)) > 60 * 5:
        return False
    
    base_string = f"v0:{timestamp}:{body.decode('utf-8')}"
    expected = "v0=" + hmac.new(
        SLACK_SIGNING_SECRET.encode(),
        base_string.encode(),
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(expected, signature)

@app.post("/webhooks/slack")
async def slack_webhook(request: Request):
    # 1. Read body
    body = await request.body()
    
    # 2. Verify signature
    timestamp = request.headers.get("X-Slack-Request-Timestamp", "")
    signature = request.headers.get("X-Slack-Signature", "")
    
    if not timestamp or not signature:
        raise HTTPException(401, "Missing signature headers")
    
    if not verify_slack_signature(body, timestamp, signature):
        raise HTTPException(401, "Invalid signature")
    
    payload = json.loads(body)
    
    # 3. Handle verification challenge
    if payload.get("type") == "url_verification":
        return {"challenge": payload["challenge"]}
    
    # 4. Handle event
    if payload.get("type") == "event_callback":
        event = payload["event"]
        event_id = payload.get("event_id")
        
        # Idempotency check
        if event_id:
            already_processed = await r.exists(f"slack:processed:{event_id}")
            if already_processed:
                return {"ok": True}  # Skip duplicates
        
        # Filter events we care about
        if event["type"] == "app_mention":
            # Enqueue for async processing
            await r.xadd(
                "slack_queries",
                {
                    "event_id": event_id or "",
                    "user": event["user"],
                    "text": event["text"],
                    "channel": event["channel"],
                    "ts": event["ts"],
                }
            )
            
            # Mark processed
            if event_id:
                await r.setex(f"slack:processed:{event_id}", 86400, "1")
    
    # 5. Respond ASAP
    return {"ok": True}

Worker in a separate process:

# worker/process_slack.py
import asyncio
import redis.asyncio as redis
from src.services.query_service import QueryService
from src.services.slack_client import SlackClient

async def consume():
    r = redis.from_url("redis://localhost:6379")
    consumer_group = "slack_workers"
    
    try:
        await r.xgroup_create("slack_queries", consumer_group, mkstream=True)
    except redis.ResponseError:
        pass
    
    query_service = QueryService(...)
    slack = SlackClient(token=os.environ["SLACK_BOT_TOKEN"])
    
    while True:
        messages = await r.xreadgroup(
            consumer_group, "worker_1",
            {"slack_queries": ">"}, count=1, block=5000
        )
        
        if not messages:
            continue
        
        for stream, msgs in messages:
            for msg_id, data in msgs:
                try:
                    user_text = data[b"text"].decode()
                    channel = data[b"channel"].decode()
                    thread_ts = data[b"ts"].decode()
                    
                    # Strip bot mention
                    query = user_text.split(">", 1)[1].strip()
                    
                    # Show "thinking..." immediately
                    await slack.post_message(
                        channel=channel,
                        thread_ts=thread_ts,
                        text="🤔 Researching..."
                    )
                    
                    # Process (takes 5-15s)
                    response = await query_service.process(query)
                    
                    # Post response
                    await slack.post_message(
                        channel=channel,
                        thread_ts=thread_ts,
                        text=response.text
                    )
                    
                    # Ack
                    await r.xack("slack_queries", consumer_group, msg_id)
                    
                except Exception as e:
                    print(f"Error processing {msg_id}: {e}")
                    # Don't ack - it will be retried

Patterns applied:

• ✅ Signature verification
• ✅ Idempotency (Redis tracking)
• ✅ Verification challenge handling
• ✅ Quick ACK
• ✅ Async processing
• ✅ Error handling

---

Self-check

For a webhook integration you know or want to design, identify the critical aspects:

Aspect	Your approach	OK?
Signature verification	?	?
Verification challenge	?	?
Sync processing	?	?
Idempotency	?	?
Timeout handling	?	?
Error handling + retry	?	?

See example case (Discord bot)

Aspect	Approach
Signature verification	Discord uses Ed25519, verify with the nacl library
Verification challenge	Discord requires a PING response in the handshake
Sync processing	Use deferred response (15min window)
Idempotency	Track interaction.id in Redis
Timeout handling	Defer response immediately, async processing in the worker
Error handling	Edit the deferred response with an error message if it fails

---

Summary and next step

In this capsule you learned inbound webhooks:

• Complete flow: registration → verification challenge → events with signature
• Verification critical: signatures, timestamps, constant-time comparison
• ACK + async pattern: respond fast (3s), async processing via queue
• Idempotency: track event IDs to handle duplicates
• AI use cases: Slack bots, Discord bots, GitHub events, error tracking

Before moving on to capsule 03, you should be able to:

• Implement HMAC signature verification
• Design the ACK + async pattern for AI webhooks
• Identify 3+ common pitfalls in webhook handlers
• Articulate why idempotency is critical

In capsule 03 — Webhooks Outbound — you're going to see the other side: how YOUR system notifies externals via webhooks. AI cases: notify when a long-running job finishes, alert an admin of problematic queries, integrate with Slack/Discord to post an update. You're going to learn retry patterns, signing your webhooks, exponential backoff, dead letter queues — the operational aspects of outbound webhooks.

---

Resources

• Slack Events API — Documentation — Official
• Webhook Security Best Practices — Stripe — Canonical patterns
• Discord Interactions API — Discord-specific
• GitHub Webhooks Documentation — GitHub webhooks
• HMAC Signature Verification — Standards — RFC with cryptographic details
• Idempotency Tokens — Stripe Engineering — Canonical pattern