NIST AI RMF: The Measure Function

Description

Map identified the risks. Measure quantifies them. Without concrete metrics, "we have bias" is an opinion; "demographic parity ratio 0.65 between groups A and B" is actionable.

This capsule teaches you how to measure each risk category with defensible metrics.

By the end you'll be able to:

• Select appropriate metrics for each risk category
• Implement measurement pipelines in production
• Interpret the results against objective criteria
• Report the measurements to stakeholders

---

The 4 categories of Measure

Measure 1: Identified risks measured

For each risk identified in Map, you assign metrics:

Risk	Metric	Frequency
Bias in responses	Demographic parity, equalized odds (M2)	Quarterly
Hallucination	Faithfulness score (Ragas), citation accuracy	Continuous
Tenant data leakage	Penetration test pass/fail	Annual
Privacy disclosure	Sensitive entity detection rate	Continuous
Performance degradation	P95 latency, error rate	Continuous
Cost explosion	$/query trend, budget usage %	Continuous

Measure 2: Effectiveness of risk mitigations

Do your mitigations actually work?

• You implemented rate limiting → does it really protect against abuse? Measure abuse incidents pre/post.
• You implemented a cache → does it reduce cost as expected? Measure the cost savings.
• You implemented bias mitigation → does it reduce the disparity? Measure before/after.

Measure 3: Tracking metrics

Continuous monitoring of the key metrics:

# Daily metrics export
def daily_metrics():
    return {
        "latency_p50_ms": measure_latency(p=50),
        "latency_p95_ms": measure_latency(p=95),
        "error_rate": count_errors() / count_total(),
        "bias_demographic_parity": calculate_demographic_parity(),
        "hallucination_rate": flag_hallucinations() / count_total(),
        "cost_per_query": total_cost() / count_total(),
        "user_satisfaction_score": average_feedback(),
    }

Measure 4: Performance and Trustworthy AI characteristics

NIST defines 7 characteristics:

• Valid and reliable
• Safe
• Secure and resilient
• Accountable and transparent
• Explainable and interpretable
• Privacy-enhanced
• Fair (with harmful bias managed)

For each one, metrics:

Valid/Reliable:
- Accuracy / pass rate on the eval set
- Consistency (same input → similar output)

Safe:
- Harmful content detection rate
- Toxic output rate

Secure/Resilient:
- Adversarial attack detection rate
- Recovery time after incidents

Accountable/Transparent:
- % of decisions with a documented rationale
- Audit log completeness

Explainable:
- % of responses with valid citations
- Citation accuracy

Privacy-enhanced:
- Sensitive entity disclosure rate (should be near 0)
- Data retention compliance

Fair:
- Demographic parity ratio
- Equalized odds gap
- Disparate impact

---

Implementation: the measurement pipeline

# measurement_pipeline.py
class MeasurementPipeline:
    def __init__(self):
        self.collectors = [
            LatencyCollector(),
            BiasCollector(),
            HallucinationCollector(),
            CostCollector(),
            SecurityCollector(),
        ]

    async def run_continuous(self):
        """Real-time collection."""
        for collector in self.collectors:
            asyncio.create_task(collector.collect_continuously())

    async def run_periodic(self, frequency="daily"):
        """Aggregate metrics + alerts."""
        results = {}
        for collector in self.collectors:
            results[collector.name] = await collector.aggregate()
        
        # Alert if thresholds exceeded
        await self.check_thresholds(results)
        
        # Export to dashboard
        await self.export_to_dashboard(results)
        
        return results


class BiasCollector:
    name = "bias"
    
    async def collect_continuously(self):
        # Log each response with demographic features (if known)
        pass
    
    async def aggregate(self):
        # Calculate demographic parity for the last 30 days
        return {
            "demographic_parity_gender": 0.92,
            "demographic_parity_age_bucket": 0.88,
            "equalized_odds_gap_gender": 0.05,
            # ... more metrics
        }

---

Criteria for "acceptable" measurements

For each metric, define a threshold:

Bias:
- demographic_parity_ratio > 0.8 → acceptable
- 0.7 - 0.8 → monitor closely
- < 0.7 → mitigation required

Hallucination:
- < 5% → acceptable
- 5-10% → mitigation in progress
- > 10% → blocked from production

Cost:
- within budget ±10% → acceptable
- 10-25% over → optimization required
- > 25% → immediate intervention

A threshold with a documented justification: why 0.8 and not 0.7? Industry standard? Risk tolerance?

---

Reporting to stakeholders

Different stakeholders need different representations:

To the engineering team

A detailed dashboard with all the metrics + drill-down.

To the CTO

A weekly summary with the top 5 metrics + status (green/yellow/red).

To the Tech Lead

A daily review with action items.

To regulators / auditors

A quarterly report with all the measurements + interpretations + actions.

## Quarterly Measurement Report Q2 2026

### Executive Summary
- Bias: ACCEPTABLE (DPR 0.92 across all groups)
- Performance: ACCEPTABLE (P95 4.2s, SLA 8s)
- Hallucination: MONITORING (7% rate, mitigation in progress)
- Cost: ACCEPTABLE ($0.0024/query, within budget)

### Detailed Metrics
[Tables, charts]

### Trend Analysis
[Quarter-over-quarter comparison]

### Actions Taken
[Mitigations implemented this quarter]

### Outstanding Concerns
[Items requiring attention]

---

Common traps

Trap 1 — Metrics with no thresholds. "We measure demographic parity" — but you don't say when it's a problem. With no threshold, it isn't actionable.

Trap 2 — Too many metrics, none of them actionable. Better 10 actionable metrics than 100 nobody reviews.

Trap 3 — Measuring but never reporting. Metrics sit in a database and nobody sees them. You need a dashboard + alerts + a reporting cadence.

Trap 4 — Thresholds with no justification. "Acceptable bias = 0.8" — but why 0.8? Document the reasoning.

Trap 5 — Not measuring the effectiveness of the mitigations. You implemented a mitigation and assume it works. Measure pre/post.

---

Exercise

For your Capstone:

1. Select the 5-7 most important metrics (from the ones identified in Map)
2. Define a threshold for each (acceptable / monitor / mitigate)
3. Design the measurement pipeline (continuous vs. periodic)
4. Design the quarterly report template

See the solution (skeleton)

The top 5 metrics for the Knowledge Assistant:

1. Hallucination rate: faithfulness score (Ragas) < 5%
2. Bias: demographic parity > 0.85 across the detected groups
3. Performance: P95 latency < 8s
4. Cost: $/query within ±10% of the $0.0024 target
5. User satisfaction: average feedback > 80% positive

Pipeline:

• Continuous (per-request): latency, errors, cost
• Daily aggregation: bias indicators, satisfaction
• Weekly: hallucination sampling (manual + automated)
• Quarterly: full bias audit, cost review, threshold review

Reporting: a monthly summary to the Tech Lead, a comprehensive quarterly report to the CTO

---

Summary

You learned:

• ✅ The 4 categories of Measure (identified risks, mitigation effectiveness, tracking, trustworthy AI characteristics)
• ✅ NIST's 7 trustworthy AI characteristics with metrics
• ✅ The implementation pipeline pattern
• ✅ Threshold setting with justification
• ✅ Reporting to different audiences
• ✅ The traps: too many metrics, no thresholds, no reporting

Checkpoint: if you have metrics with thresholds + a pipeline + a reporting cadence, Measure is OK.

---

Next capsule

05 — NIST AI RMF: the Manage function. You measured the risks. Manage is about prioritizing and acting — respond, mitigate, accept, transfer.

---

Resources

1. NIST AI RMF Playbook — Measure.
2. Ragas — RAG metrics.
3. LangSmith — LLM observability.
4. Aequitas — Bias audit toolkit.