NIST AI RMF: Measure Function

Descripción

Map identificó risks. Measure los cuantifica. Sin métricas concretas, "tenemos bias" es opinión; "demographic parity ratio 0.65 entre groups A y B" es accionable.

Esta cápsula te enseña cómo medir cada risk category con métricas defendibles.

Al terminar vas a poder:

• Seleccionar métricas apropiadas para cada risk category
• Implementar measurement pipelines en producción
• Interpretar resultados con criteria objetivos
• Reportar measurements a stakeholders

---

Las 4 categorías de Measure

Measure 1: Identified risks measured

Para cada risk identificado en Map, asignás métricas:

Risk	Métrica	Frequency
Bias en responses	Demographic parity, equalized odds (M2)	Quarterly
Hallucination	Faithfulness score (Ragas), citation accuracy	Continuous
Tenant data leakage	Penetration test pass/fail	Annual
Privacy disclosure	Sensitive entity detection rate	Continuous
Performance degradation	P95 latency, error rate	Continuous
Cost explosion	$/query trend, budget usage %	Continuous

Measure 2: Effectiveness of risk mitigations

¿Tus mitigations actually work?

• Implementaste rate limiting → ¿Realmente protege contra abuse? Mide abuse incidents pre/post.
• Implementaste cache → ¿Reduce cost esperado? Mide cost savings.
• Implementaste bias mitigation → ¿Reduce disparidad? Mide before/after.

Measure 3: Tracking metrics

Continuous monitoring de métricas clave:

# Daily metrics export
def daily_metrics():
    return {
        "latency_p50_ms": measure_latency(p=50),
        "latency_p95_ms": measure_latency(p=95),
        "error_rate": count_errors() / count_total(),
        "bias_demographic_parity": calculate_demographic_parity(),
        "hallucination_rate": flag_hallucinations() / count_total(),
        "cost_per_query": total_cost() / count_total(),
        "user_satisfaction_score": average_feedback(),
    }

Measure 4: Performance and Trustworthy AI characteristics

NIST define 7 characteristics:

• Valid and reliable
• Safe
• Secure and resilient
• Accountable and transparent
• Explainable and interpretable
• Privacy-enhanced
• Fair (with harmful bias managed)

Para cada uno, métricas:

Valid/Reliable:
- Accuracy / pass rate on eval set
- Consistency (same input → similar output)

Safe:
- Harmful content detection rate
- Toxic output rate

Secure/Resilient:
- Adversarial attack detection rate
- Recovery time after incidents

Accountable/Transparent:
- % decisions with documented rationale
- Audit log completeness

Explainable:
- % responses with valid citations
- Citation accuracy

Privacy-enhanced:
- Sensitive entity disclosure rate (should be near 0)
- Data retention compliance

Fair:
- Demographic parity ratio
- Equalized odds gap
- Disparate impact

---

Implementación: measurement pipeline

# measurement_pipeline.py
class MeasurementPipeline:
    def __init__(self):
        self.collectors = [
            LatencyCollector(),
            BiasCollector(),
            HallucinationCollector(),
            CostCollector(),
            SecurityCollector(),
        ]

    async def run_continuous(self):
        """Real-time collection."""
        for collector in self.collectors:
            asyncio.create_task(collector.collect_continuously())

    async def run_periodic(self, frequency="daily"):
        """Aggregate metrics + alerts."""
        results = {}
        for collector in self.collectors:
            results[collector.name] = await collector.aggregate()
        
        # Alert if thresholds exceeded
        await self.check_thresholds(results)
        
        # Export to dashboard
        await self.export_to_dashboard(results)
        
        return results


class BiasCollector:
    name = "bias"
    
    async def collect_continuously(self):
        # Log each response with demographic features (if known)
        pass
    
    async def aggregate(self):
        # Calculate demographic parity for last 30 days
        return {
            "demographic_parity_gender": 0.92,
            "demographic_parity_age_bucket": 0.88,
            "equalized_odds_gap_gender": 0.05,
            # ... more metrics
        }

---

Criteria for "acceptable" measurements

Para cada métrica, define threshold:

Bias:
- demographic_parity_ratio > 0.8 → acceptable
- 0.7 - 0.8 → monitor closely
- < 0.7 → mitigation required

Hallucination:
- < 5% → acceptable
- 5-10% → mitigation in progress
- > 10% → blocked from production

Cost:
- within budget ±10% → acceptable
- 10-25% over → optimization required
- > 25% → immediate intervention

Threshold con justification documented: por qué 0.8 y no 0.7? Industry standard? Risk tolerance?

---

Reportting a stakeholders

Different stakeholders necesitan diferentes representations:

To engineering team

Detailed dashboard with all metrics + drill-down.

To CTO

Weekly summary with top 5 metrics + status (green/yellow/red).

To Tech Lead

Daily review with action items.

To regulators / auditors

Quarterly report with all measurements + interpretations + actions.

## Quarterly Measurement Report Q2 2026

### Executive Summary
- Bias: ACCEPTABLE (DPR 0.92 across all groups)
- Performance: ACCEPTABLE (P95 4.2s, SLA 8s)
- Hallucination: MONITORING (7% rate, mitigation in progress)
- Cost: ACCEPTABLE ($0.0024/query, within budget)

### Detailed Metrics
[Tables, charts]

### Trend Analysis
[Quarter-over-quarter comparison]

### Actions Taken
[Mitigations implemented this quarter]

### Outstanding Concerns
[Items requiring attention]

---

Trampas comunes

Trampa 1 — Métricas sin thresholds. "Medimos demographic parity" — pero no decís cuándo es problema. Sin threshold, no es actionable.

Trampa 2 — Demasiadas métricas, ninguna actionable. Mejor 10 métricas accionables que 100 que nadie revisa.

Trampa 3 — Measure pero never report. Métricas en database, nadie las ve. Necesitás dashboard + alerts + report cadence.

Trampa 4 — Thresholds sin justification. "Acceptable bias = 0.8" — pero ¿por qué 0.8? Documenta el reasoning.

Trampa 5 — No medir effectiveness de mitigations. Implementaste mitigation, asumís que funciona. Mide pre/post.

---

Ejercicio

Para tu Capstone:

1. Selecciona 5-7 métricas más importantes (de las identificadas en Map)
2. Define threshold para cada (acceptable / monitor / mitigate)
3. Diseñá pipeline de measurement (continuous vs periodic)
4. Diseñá quarterly report template

Ver solución (skeleton)

Top 5 métricas para Knowledge Assistant:

1. Hallucination rate: faithfulness score (Ragas) < 5%
2. Bias: demographic parity > 0.85 across detected groups
3. Performance: P95 latency < 8s
4. Cost: $/query within ±10% of $0.0024 target
5. User satisfaction: average feedback > 80% positive

Pipeline:

• Continuous (per-request): latency, errors, cost
• Daily aggregation: bias indicators, satisfaction
• Weekly: hallucination sampling (manual + automated)
• Quarterly: full bias audit, cost review, threshold review

Report: monthly summary to Tech Lead, quarterly comprehensive to CTO

---

Resumen

Aprendiste:

• ✅ 4 categorías de Measure (identified risks, mitigation effectiveness, tracking, trustworthy AI characteristics)
• ✅ NIST's 7 trustworthy AI characteristics with metrics
• ✅ Implementation pipeline pattern
• ✅ Threshold setting con justification
• ✅ Reporting a diferentes audiences
• ✅ Trampas: demasiadas métricas, sin thresholds, sin reporting

Checkpoint: si tenés métricas con thresholds + pipeline + reporting cadence, Measure está OK.

---

Siguiente cápsula

05 — NIST AI RMF: Manage function. Mediste los risks. Manage es priorizar y actuar — respond, mitigate, accept, transfer.

---

Recursos

1. NIST AI RMF Playbook — Measure.
2. Ragas — RAG metrics.
3. LangSmith — LLM observability.
4. Aequitas — Bias audit toolkit.