Module 1: Fundamentals of Prompt Engineering
8. Project: Prompt Analyzer
Project overview
In this project you'll build a Prompt Analyzer: a system that takes any prompt, classifies it by technique (zero-shot, few-shot, CoT, mixed), identifies its components (instruction, context, input, output format), scores the CRISPE dimensions present, and generates specific, actionable improvement suggestions. The output is structured JSON with a defined schema, validated with Pydantic.
This project integrates everything you learned in Module 1: prompt anatomy, roles, parameters, the CRISPE framework, the difference between casual and engineered prompts, and the differences between providers. It's not a theoretical exercise — it's a tool you'll use throughout the whole guide: when you design a CoT prompt in Module 4, you can run it through the Analyzer to check it has every component.
Goal: Build a working prompt analysis system with structured output that shows prompts have a structure you can analyze systematically.
Technical specifications
Project structure
prompt-analyzer/
├── analyzer.py # The main analysis logic
├── schemas.py # Pydantic models for input/output
├── main.py # CLI and test interface
├── test_cases.py # Test cases with example prompts
├── .env # API keys (don't commit)
└── requirements.txt # Dependencies
Dependencies
# requirements.txt
openai>=1.0.0
anthropic>=0.25.0
python-dotenv>=1.0.0
pydantic>=2.0.0
tiktoken>=0.5.0
Output schema
The Prompt Analyzer returns a JSON with four sections:
{
"classification": {
"technique": "zero-shot|few-shot|chain-of-thought|mixed",
"confidence": 0.0-1.0,
"reason": "An explanation of why it was classified this way"
},
"components": {
"instruction": "present|absent|implicit",
"context": "present|absent|implicit",
"input": "present|absent|implicit",
"output_format": "present|absent|implicit"
},
"crispe": {
"capacity": "present|absent|implicit",
"role": "present|absent|implicit",
"insight": "present|absent|implicit",
"statement": "present|absent|implicit",
"personality": "present|absent|implicit",
"experiment": "present|absent|implicit",
"score": 0-6
},
"suggestions": [
"Suggestion 1, specific and actionable",
"Suggestion 2, specific and actionable"
],
"quality": {
"level": "casual|basic|engineered|advanced",
"score": 0-100,
"summary": "An evaluation in 1 sentence"
},
"metadata": {
"character_length": 0,
"estimated_tokens": 0,
"has_system_prompt": true|false
}
}Step-by-step implementation
Step 1: schemas.py — Define the Pydantic models
# schemas.py
from pydantic import BaseModel, Field
from typing import Literal
ComponentState = Literal["present", "absent", "implicit"]
PromptTechnique = Literal["zero-shot", "few-shot", "chain-of-thought", "mixed"]
QualityLevel = Literal["casual", "basic", "engineered", "advanced"]
class Classification(BaseModel):
technique: PromptTechnique = Field(description="The prompting technique detected")
confidence: float = Field(ge=0.0, le=1.0, description="Confidence in the classification")
reason: str = Field(description="Why it was classified with this technique")
class Components(BaseModel):
instruction: ComponentState = Field(description="The instruction or main task")
context: ComponentState = Field(description="Context or background information")
input: ComponentState = Field(description="The input the model will process")
output_format: ComponentState = Field(description="The expected output format")
class CRISPE(BaseModel):
capacity: ComponentState
role: ComponentState
insight: ComponentState
statement: ComponentState
personality: ComponentState
experiment: ComponentState
score: int = Field(ge=0, le=6, description="How many CRISPE components are present")
class Quality(BaseModel):
level: QualityLevel
score: int = Field(ge=0, le=100)
summary: str = Field(description="An evaluation in 1 sentence")
class Metadata(BaseModel):
character_length: int
estimated_tokens: int
has_system_prompt: bool
class PromptAnalysis(BaseModel):
classification: Classification
components: Components
crispe: CRISPE
suggestions: list[str] = Field(min_length=1, max_length=6)
quality: Quality
metadata: MetadataStep 2: analyzer.py — The main logic
# analyzer.py
import json
import re
import tiktoken
from openai import OpenAI
from dotenv import load_dotenv
from schemas import PromptAnalysis, Metadata
load_dotenv()
client = OpenAI()
ANALYZER_SYSTEM = """
You are an expert prompt analyzer for AI systems. Analyze the prompt provided
and return a structured JSON with the complete analysis.
DEFINITIONS:
PROMPTING TECHNIQUES:
- zero-shot: Instruction only, no examples, no explicit reasoning
- few-shot: Includes input→output examples before the real task
- chain-of-thought: Includes or requests explicit step-by-step reasoning
- mixed: Combines several techniques
PROMPT COMPONENTS:
- instruction: The main task or action the model must carry out
- context: Background information, domain definitions, business rules
- input: The data or text the model must process (it can be a placeholder)
- output_format: The output format spec (JSON, a list, a specific format)
CRISPE DIMENSIONS:
- capacity: The main capability/action (the task's verb)
- role: The role, expertise or persona the model adopts
- insight: The context or background information needed
- statement: The concrete, specific instruction
- personality: Tone, style, behavior constraints
- experiment: The output format, examples
QUALITY:
- casual: No structure, no constraints, uncontrolled output
- basic: Has a clear instruction but no format and no constraints
- engineered: Has instruction + format + constraints + a system prompt or roles
- advanced: engineered + CRISPE ≥4 + examples or CoT + edge case handling
QUALITY SCORE (0-100):
- 0-25: Casual (a prompt with no structure)
- 26-50: Basic (a clear instruction, little more)
- 51-75: Engineered (clear structure, format, constraints)
- 76-100: Advanced (full CRISPE, examples, edge cases, production-ready)
SPECIFIC SUGGESTIONS (not generic ones):
- ❌ Bad: "Improve the prompt"
- ✅ Good: "output_format is missing. Add: 'Respond in JSON: {\"field\": \"value\"}'"
- ✅ Good: "Role is missing (R in CRISPE). Add: 'You are a [expert] specialized in [domain]'"
- ✅ Good: "No edge case handling: add 'If the input is empty, respond {\"error\": \"empty_input\"}'"
Respond ONLY with valid JSON that follows exactly the schema provided.
"""
SCHEMA_PROMPT = """
{
"classification": {
"technique": "zero-shot|few-shot|chain-of-thought|mixed",
"confidence": 0.0-1.0,
"reason": "string"
},
"components": {
"instruction": "present|absent|implicit",
"context": "present|absent|implicit",
"input": "present|absent|implicit",
"output_format": "present|absent|implicit"
},
"crispe": {
"capacity": "present|absent|implicit",
"role": "present|absent|implicit",
"insight": "present|absent|implicit",
"statement": "present|absent|implicit",
"personality": "present|absent|implicit",
"experiment": "present|absent|implicit",
"score": 0-6
},
"suggestions": ["string", "string"],
"quality": {
"level": "casual|basic|engineered|advanced",
"score": 0-100,
"summary": "string"
},
"metadata": {
"character_length": 0,
"estimated_tokens": 0,
"has_system_prompt": true|false
}
}
"""
def estimate_tokens(text: str, model: str = "gpt-4o-mini") -> int:
"""Counts tokens using tiktoken. Falls back to an approximation if it fails."""
try:
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
except Exception:
return len(text) // 4 # Approx: 4 chars ≈ 1 token
def detect_system_prompt(prompt: str) -> bool:
"""A heuristic: detects whether the prompt looks like a system prompt."""
indicators = [
"eres un", "you are a", "actúa como", "act as",
"tu rol es", "your role is", "## capacity", "## role"
]
lower = prompt.lower()
return any(ind in lower for ind in indicators)
def clean_json(raw: str) -> str:
"""Cleans up the model's response to extract valid JSON."""
# Remove markdown code blocks
if "```" in raw:
match = re.search(r'```(?:json)?\s*\n?(.*?)\n?```', raw, re.DOTALL)
if match:
return match.group(1).strip()
# Look for the first JSON object { ... }
start = raw.find('{')
if start != -1:
depth = 0
for i, char in enumerate(raw[start:], start):
if char == '{':
depth += 1
elif char == '}':
depth -= 1
if depth == 0:
return raw[start:i+1]
return raw.strip()
def analyze_prompt(prompt: str) -> PromptAnalysis:
"""
Analyzes a prompt and returns a structured analysis.
Args:
prompt: The prompt to analyze (it can be user only, or system+user)
Returns:
PromptAnalysis with classification, components, CRISPE, suggestions, quality
"""
tokens = estimate_tokens(prompt)
has_system = detect_system_prompt(prompt)
user_content = f"""Analyze this prompt:
---PROMPT START---
{prompt}
---PROMPT END---
Known data for metadata:
- character_length: {len(prompt)}
- estimated_tokens: {tokens}
- has_system_prompt: {has_system}
Respond with JSON that follows exactly this schema:
{SCHEMA_PROMPT}
"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": ANALYZER_SYSTEM},
{"role": "user", "content": user_content}
],
temperature=0,
max_tokens=800
)
raw = response.choices[0].message.content.strip()
json_str = clean_json(raw)
try:
data = json.loads(json_str)
except json.JSONDecodeError as e:
raise ValueError(f"The model did not return valid JSON: {e}\nRaw: {raw[:200]}")
# Make sure metadata has the right values
if "metadata" in data:
data["metadata"]["character_length"] = len(prompt)
data["metadata"]["estimated_tokens"] = tokens
data["metadata"]["has_system_prompt"] = has_system
return PromptAnalysis(**data)Step 3: main.py — Test interface and CLI
# main.py
import json
from analyzer import analyze_prompt
def print_report(analysis, prompt_preview: str):
"""Prints a readable report of the analysis."""
a = analysis
print("=" * 60)
print(f"PROMPT ANALYZED: {prompt_preview[:50]}...")
print("=" * 60)
print(f"\n📊 QUALITY: {a.quality.level.upper()} ({a.quality.score}/100)")
print(f" {a.quality.summary}")
print(f"\n🏷️ TECHNIQUE: {a.classification.technique.upper()} (confidence: {a.classification.confidence:.0%})")
print(f" {a.classification.reason}")
print("\n🧩 COMPONENTS:")
for comp, state in a.components.model_dump().items():
icon = "✅" if state == "present" else ("⚠️" if state == "implicit" else "❌")
print(f" {icon} {comp}: {state}")
print(f"\n🎯 CRISPE ({a.crispe.score}/6):")
for dim, state in a.crispe.model_dump().items():
if dim == "score":
continue
icon = "✅" if state == "present" else ("⚠️" if state == "implicit" else "❌")
print(f" {icon} {dim}: {state}")
print("\n💡 SUGGESTIONS:")
for i, suggestion in enumerate(a.suggestions, 1):
print(f" {i}. {suggestion}")
print(f"\n📏 METADATA:")
print(f" Characters: {a.metadata.character_length}")
print(f" Est. tokens: {a.metadata.estimated_tokens}")
print(f" System prompt: {'Yes' if a.metadata.has_system_prompt else 'No'}")
print()
# Test cases with prompts of different quality levels
PROMPTS_TEST = {
"casual": "Summarize this.",
"basic": "Summarize the following article in 3 main points.",
"engineered": """You are a technical editor specialized in AI articles.
Summarize the following article in exactly 3 key points.
FORMAT:
1. [Point in 20 words maximum]
2. [Point in 20 words maximum]
3. [Point in 20 words maximum]
No introduction, no conclusion.""",
"few_shot": """Classify the sentiment of the text.
Examples:
- "I loved the product" → POSITIVE
- "Horrible experience" → NEGATIVE
- "Fine, nothing special" → NEUTRAL
Text to classify: [INPUT]""",
"cot": """Solve this math problem step by step.
Think out loud, showing each step of the reasoning.
Verify your answer at the end.
Format:
Step 1: [reasoning]
Step 2: [reasoning]
...
Final answer: [result]
Verification: [check]
Problem: [INPUT]""",
"crispe_complete": """## Capacity
Classify support tickets into predefined categories.
## Role
You are an automatic classifier for the technical support system of a SaaS company.
## Insight
Categories:
- TECHNICAL: Software errors, feature failures, performance
- BILLING: Charges, invoices, plans, refunds
- GENERAL: Usage questions, features, onboarding
Priority: TECHNICAL > BILLING > GENERAL when there's ambiguity.
## Statement
Classify the following ticket into exactly ONE category.
## Personality
- The category name only, in uppercase
- No explanation, no punctuation
## Experiment
TECHNICAL → For: "Error 404 when accessing the dashboard"
BILLING → For: "My invoice has an incorrect charge"
GENERAL → For: "How do I export my data?"
"""
}
if __name__ == "__main__":
print("🔍 PROMPT ANALYZER — Case demo\n")
for name, prompt in PROMPTS_TEST.items():
print(f"\n{'='*60}")
print(f"CASE: {name.upper()}")
try:
analysis = analyze_prompt(prompt)
print_report(analysis, prompt)
except Exception as e:
print(f"Error analyzing '{name}': {e}")Step 4: test_cases.py — Classification tests
# test_cases.py
"""
Runs the Prompt Analyzer on known cases and verifies it classifies them correctly.
It doesn't use pytest — it's manual validation.
"""
from analyzer import analyze_prompt
KNOWN_CASES = [
{
"name": "zero_shot_simple",
"prompt": "Translate to Spanish: 'The dog eats'",
"expected_technique": "zero-shot",
"min_quality": 10,
"max_quality": 50
},
{
"name": "few_shot_with_examples",
"prompt": """Classify the email.
SPAM: "You won a prize, click here"
LEGITIMATE: "Your March invoice is available"
SPAM: "Exclusive offer today only!!!"
Email to classify: "Team meeting on Monday at 10am"
""",
"expected_technique": "few-shot",
"min_quality": 40,
"max_quality": 80
},
{
"name": "cot_explicit",
"prompt": """Solve step by step:
Think through each step before giving the final answer.
Show your full reasoning.
Question: If a store has 120 products and sells 30%, how many are left?
""",
"expected_technique": "chain-of-thought",
"min_quality": 30,
"max_quality": 70
}
]
def run_tests():
print("🧪 Running test cases...\n")
results = []
for case in KNOWN_CASES:
print(f"Case: {case['name']}")
analysis = analyze_prompt(case['prompt'])
# Check the technique
technique_ok = analysis.classification.technique == case['expected_technique']
# Check the quality is in range
quality_ok = (
case['min_quality'] <= analysis.quality.score <= case['max_quality']
)
status = "✅ PASS" if (technique_ok and quality_ok) else "❌ FAIL"
print(f" {status}")
print(f" Technique: {analysis.classification.technique} (expected: {case['expected_technique']}) {'✅' if technique_ok else '❌'}")
print(f" Quality: {analysis.quality.score}/100 (range: {case['min_quality']}-{case['max_quality']}) {'✅' if quality_ok else '❌'}")
print(f" CRISPE score: {analysis.crispe.score}/6\n")
results.append(technique_ok and quality_ok)
total = len(results)
passed = sum(results)
print(f"\n📊 FINAL RESULT: {passed}/{total} cases passed")
return passed == total
if __name__ == "__main__":
run_tests()Success criteria
When you finish the project, verify that:
-
analyze_prompt("Summarize this.")returns techniquezero-shot, qualitycasual(<50 points), and at least 2 specific suggestions about what to add -
analyze_promptwith a prompt that includes examples classifies asfew-shot -
analyze_promptwith a prompt that asks for step-by-step reasoning classifies aschain-of-thought - The CRISPE score of the
crispe_completetest prompt is ≥5/6 - Every output validates with Pydantic without errors (
PromptAnalysis(**data)) -
test_cases.pypasses the 3 known cases - The suggestions are specific: they mention exactly what to add, not "improve the prompt"
- The code runs without errors with
python main.py
A complete run
Input (a casual prompt):
Summarize this article in 3 points.
The Analyzer's output:
============================================================
PROMPT ANALYZED: Summarize this article in 3 points....
============================================================
📊 QUALITY: BASIC (35/100)
A clear instruction but no format and no output constraints
🏷️ TECHNIQUE: ZERO-SHOT (confidence: 95%)
A direct instruction only, no examples, no explicit reasoning
🧩 COMPONENTS:
✅ instruction: present
❌ context: absent
⚠️ input: implicit
❌ output_format: absent
🎯 CRISPE (1/6):
❌ capacity: implicit
❌ role: absent
❌ insight: absent
✅ statement: present
❌ personality: absent
❌ experiment: absent
💡 SUGGESTIONS:
1. output_format is missing: add 'Format: 1. [point] 2. [point] 3. [point]'
2. Role is missing (R): add 'You are a technical editor' to calibrate the vocabulary
3. Implicit input: add 'Article: [text]' to make clear what to process
4. No length constraints: add '20 words maximum per point'
📏 METADATA:
Characters: 35
Est. tokens: 10
System prompt: No
Optional extensions
Extension 1: Batch analysis
def analyze_batch(prompts: list[str]) -> list[dict]:
"""Analyzes several prompts and generates a comparative report."""
results = []
for prompt in prompts:
analysis = analyze_prompt(prompt)
results.append({
"preview": prompt[:50],
"technique": analysis.classification.technique,
"quality": analysis.quality.score,
"crispe_score": analysis.crispe.score,
"suggestions_count": len(analysis.suggestions)
})
return results
# Usage
prompts = [p for p in PROMPTS_TEST.values()]
report = analyze_batch(prompts)
for r in report:
print(f"{r['quality']:3d}/100 | CRISPE {r['crispe_score']}/6 | {r['technique']:18} | {r['preview']}")Extension 2: Multi-provider portability detection
def analyze_portability(prompt: str) -> dict:
"""
Analyzes whether the prompt uses features that aren't portable between providers.
"""
issues = []
# OpenAI-specific features
if "response_format" in prompt:
issues.append("Uses 'response_format' (OpenAI only). For Anthropic: a JSON instruction in text")
# Anthropic-specific features
if "<output>" in prompt or "</output>" in prompt:
issues.append("Custom XML tags — they work everywhere but they're idiomatic to Anthropic")
# Nicely portable
if '"format":' in prompt.lower() or "json" in prompt.lower():
pass # JSON in text is portable
return {
"is_portable": len(issues) == 0,
"issues": issues,
"recommendation": "Portable" if not issues else f"Adjust: {'; '.join(issues)}"
}Extension 3: Compare versions of a prompt
def compare_versions(prompt_v1: str, prompt_v2: str) -> dict:
"""Compares two versions of the same prompt and shows what improved."""
a1 = analyze_prompt(prompt_v1)
a2 = analyze_prompt(prompt_v2)
return {
"quality_delta": a2.quality.score - a1.quality.score,
"crispe_delta": a2.crispe.score - a1.crispe.score,
"technique_change": f"{a1.classification.technique} → {a2.classification.technique}",
"improvements": [
s for s in a1.suggestions
if not any(comp in str(a2.components) for comp in ["present"])
]
}Troubleshooting
Problem 1: The model doesn't return valid JSON
Cause: Sometimes it includes text before the JSON ("Here's the analysis:") or uses json ... .
Fix: The clean_json() function already handles this. If it still fails:
# Add to the system prompt
ANALYZER_SYSTEM += """
CRITICAL: Your response must start DIRECTLY with { and end with }.
No "```json", no introductory text, no explanations.
"""Problem 2: Generic suggestions ("improve the prompt")
Cause: The system prompt doesn't specify clearly enough what a specific suggestion is.
Fix: Add more negative examples to the system prompt:
# Add to ANALYZER_SYSTEM
"""
SUGGESTIONS: They must mention EXACTLY WHAT TO ADD with an example of the text to add.
- ❌ "Add more context"
- ✅ "Insight is missing: add 'Priorities: urgent > normal > low' to guide the classification"
- ❌ "Specify the format"
- ✅ "Experiment is missing: add 'Respond in JSON: {\"category\": \"TECHNICAL|BILLING|GENERAL\"}'"
"""Problem 3: Incorrect few-shot vs zero-shot classification
Cause: The model confuses examples in the context with real few-shot.
Fix: Add a more precise definition to the system prompt:
"""
CRITICAL for classification:
- zero-shot: An instruction ONLY. No input→output examples before the task.
- few-shot: The prompt INCLUDES examples in the format "Input: X → Output: Y" or similar, BEFORE asking for the classification of the current case.
- chain-of-thought: The prompt INCLUDES or explicitly REQUESTS step-by-step reasoning ("think step by step", "show your reasoning", etc.)
"""Problem 4: Pydantic validation error
Cause: The model's JSON doesn't match the schema's types (e.g. confidence outside 0-1, score outside 0-6).
Fix: Add sanitization before validating:
def sanitize_data(data: dict) -> dict:
"""Fixes out-of-range values before validating with Pydantic."""
if "classification" in data:
conf = data["classification"].get("confidence", 0.5)
data["classification"]["confidence"] = max(0.0, min(1.0, float(conf)))
if "crispe" in data:
score = data["crispe"].get("score", 0)
data["crispe"]["score"] = max(0, min(6, int(score)))
if "quality" in data:
pts = data["quality"].get("score", 0)
data["quality"]["score"] = max(0, min(100, int(pts)))
return data
# In analyze_prompt():
data = json.loads(json_str)
data = sanitize_data(data)
return PromptAnalysis(**data)Problem 5: tiktoken doesn't recognize the model
Cause: If you use a new model that tiktoken doesn't have in its registry yet.
Fix: The estimate_tokens function already has a fallback. If you want more precision:
def estimate_tokens(text: str, model: str = "gpt-4o-mini") -> int:
try:
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
except KeyError:
# Fallback: use the cl100k_base encoding (GPT-4 family)
encoding = tiktoken.get_encoding("cl100k_base")
return len(encoding.encode(text))Project summary
In this project you built:
- A complete Pydantic schema with type, range, and literal validation — the base for every structured output in the guide
- An LLM-based analysis system that uses a well-instructed model to analyze another prompt — the LLM-as-judge technique you'll see in detail in Module 7
- A robust JSON parser that handles the variable formats models can return
- A test case framework without pytest, but with systematic verification of expected results
- Heuristic detectors (system prompt, portability) that complement the LLM's analysis with deterministic logic
How you'll use this Analyzer in the following modules:
- Module 2: Check that your few-shot prompts have the right format in "experiment"
- Module 4: Check that your CoT has a "statement" with a step-by-step reasoning instruction
- Module 6: Check that every prompt in a chain has its components well defined
- Module 7: The Prompt Analyzer is a precursor of the evaluation framework with LLM-as-judge
Additional resources
- OpenAI Structured Outputs — JSON mode and JSON Schema in OpenAI, the base for the
response_formatyou use in the extractor - Pydantic v2 Validators — Advanced validators for the Analyzer's schema (useful if you add more validations)
- tiktoken GitHub — A library to count tokens exactly; crucial for optimizing costs in production
- OpenAI Evals — A framework for evaluating prompts at scale; what you built by hand here, but automated
- Pydantic BaseModel — BaseModel documentation with every field type and validator available