4. Temperature and Generation Parameters

Overview

The generation parameters (temperature, top_p, max_tokens, frequency_penalty, presence_penalty) control how the model produces text. Picking the wrong values can make a "perfect" prompt fail in production. In this capsule you'll learn what each parameter does, when to use temperature 0 vs 1, and the recommended settings per use case.

Why it matters: A classifier with temperature=0.9 will give inconsistent results even if the prompt is excellent. A creative generator with temperature=0 will sound robotic and repetitive. Parameters are part of the prompt's design: they are not an afterthought.

Temperature

What it does: Controls the randomness in the selection of the next token. It reshapes the probability distribution of the candidate tokens before sampling.

• temperature=0: The model always picks the most probable token. Deterministic.
• temperature=1: It samples according to the model's original probabilities.
• temperature>1: It flattens the distribution (less probable tokens get more of a chance). More random.

Typical range: 0.0 to 2.0 (OpenAI). 0.0 to 1.0 (Anthropic).

When to use each value

A practical example: comparing temperature

from openai import OpenAI

client = OpenAI()

classification_prompt = "Classify the sentiment: 'I loved the product, I recommend it'"
creative_prompt = "Write a 5-word slogan for an artisan coffee shop."

# Temperature 0: same result every time
print("=== Classification (temperature=0) ===")
for i in range(3):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Classify sentiment. Only: POSITIVE, NEGATIVE or NEUTRAL."},
            {"role": "user", "content": classification_prompt}
        ],
        temperature=0,
        max_tokens=5
    )
    print(f"  Attempt {i+1}: {response.choices[0].message.content.strip()}")
# Output: POSITIVE / POSITIVE / POSITIVE (always the same)

# Temperature 0.8: creativity, variation
print("\n=== Creative (temperature=0.8) ===")
for i in range(3):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": creative_prompt}],
        temperature=0.8,
        max_tokens=20
    )
    print(f"  Attempt {i+1}: {response.choices[0].message.content.strip()}")
# Output varies:
#   Attempt 1: "Your coffee, your perfect moment."
#   Attempt 2: "Flavors that warm the soul."
#   Attempt 3: "Every sip, a unique story."

top_p (Nucleus Sampling)

What it does: Limits the set of candidate tokens to those that accumulate top_p of the probability mass. E.g. top_p=0.1 only considers the most probable tokens up to 10% of probability.

Relationship with temperature: Both control randomness, but in different ways:

• temperature scales the whole distribution
• top_p cuts the distribution off after a certain threshold

OpenAI's recommendation: Change only temperature or top_p, not both at once.

# Low top_p = more determinism (a different mechanism than temperature)
response_low_p = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Give me one word that describes the ocean"}],
    temperature=1.0,  # No restriction from temperature
    top_p=0.1         # But only the top 10% probability tokens
)

# High top_p = more variety
response_high_p = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Give me one word that describes the ocean"}],
    temperature=1.0,
    top_p=0.95  # Allows less probable tokens
)

Typical values:

• 1.0: No filter (every token considered)
• 0.9: Variety with coherence
• 0.1-0.5: More determinism than with temperature but without reaching 0

max_tokens

What it does: The maximum limit of tokens in the model's response. It does not include the prompt's tokens.

When to adjust it:

# Low max_tokens for classification
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Classify the ticket into: TECHNICAL, BILLING, ACCOUNT. One word only."},
        {"role": "user", "content": "I can't log into my account"}
    ],
    temperature=0,
    max_tokens=5  # "ACCOUNT" fits in 1-2 tokens
)
print(response.choices[0].message.content)  # ACCOUNT

# Check whether the response was truncated
finish_reason = response.choices[0].finish_reason
print(f"Finish reason: {finish_reason}")
# "stop" = complete response
# "length" = truncated by max_tokens

Important: If finish_reason == "length", the response was cut off. Raise max_tokens or ask for a more concise response in the prompt.

frequency_penalty and presence_penalty

frequency_penalty

What it does: Penalizes tokens in proportion to how many times they already appeared. It reduces repetition of the same words.

• 0: No penalty (default)
• 0.5-1.0: Moderately reduces repetition
• 2.0: Maximum penalty (can degrade quality)

presence_penalty

What it does: Penalizes tokens that already appeared, regardless of frequency. It encourages talking about new topics.

• 0: No penalty (default)
• 0.3-0.6: Encourages a variety of topics
• 2.0: Maximum penalty

# Example: generating a list without repetition
list_prompt = "List 10 adjectives to describe good leadership:"

# Without penalty: it can repeat concepts
response_no_penalty = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": list_prompt}],
    temperature=0.7,
    frequency_penalty=0,
    presence_penalty=0,
    max_tokens=150
)

# With penalty: more diversity
response_with_penalty = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": list_prompt}],
    temperature=0.7,
    frequency_penalty=0.5,   # Reduces repetition of the same words
    presence_penalty=0.3,    # Encourages new concepts
    max_tokens=150
)

Cases where you should NOT use a penalty:

• Code: technical terms have to repeat (e.g. def, return, variable names)
• Extraction: if the same term appears several times in the text, you have to include it
• Technical answers where vocabulary precision matters

seed: reproducibility in OpenAI

OpenAI offers the seed parameter for approximate reproducibility (not guaranteed, but very consistent):

# With seed: more reproducibility
responses = []
for i in range(3):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Give me a name for an AI startup"}],
        temperature=0.9,  # High temperature
        seed=42           # But with a fixed seed
    )
    responses.append(response.choices[0].message.content)

print("Responses with seed=42:")
for r in responses:
    print(f"  - {r}")
# The 3 responses will be very similar or identical

Note: seed is useful for reproducibility in testing and debugging, but it isn't 100% guaranteed across model versions.

Configurations by use case

from openai import OpenAI
from dataclasses import dataclass

client = OpenAI()

@dataclass
class ModelConfig:
    temperature: float
    max_tokens: int
    frequency_penalty: float = 0.0
    presence_penalty: float = 0.0
    top_p: float = 1.0

# Predefined configurations
CONFIGS = {
    "classification": ModelConfig(
        temperature=0,
        max_tokens=20,
        frequency_penalty=0
    ),
    "extraction": ModelConfig(
        temperature=0,
        max_tokens=200,
        frequency_penalty=0
    ),
    "summary": ModelConfig(
        temperature=0.3,
        max_tokens=300,
        frequency_penalty=0.3
    ),
    "document_qa": ModelConfig(
        temperature=0.3,
        max_tokens=500
    ),
    "brainstorming": ModelConfig(
        temperature=0.8,
        max_tokens=400,
        frequency_penalty=0.5,
        presence_penalty=0.3
    ),
    "creative_copy": ModelConfig(
        temperature=0.7,
        max_tokens=200,
        frequency_penalty=0.4
    ),
    "code": ModelConfig(
        temperature=0,
        max_tokens=1500,
        frequency_penalty=0
    ),
}

def call_with_config(system: str, user: str, config_name: str) -> str:
    config = CONFIGS[config_name]
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user}
        ],
        temperature=config.temperature,
        max_tokens=config.max_tokens,
        frequency_penalty=config.frequency_penalty,
        presence_penalty=config.presence_penalty,
        top_p=config.top_p
    )
    return response.choices[0].message.content

# Usage
result = call_with_config(
    system="Classify the sentiment. Only: POSITIVE, NEGATIVE, NEUTRAL.",
    user="The product arrived fast but the quality is mediocre.",
    config_name="classification"
)
print(result)  # NEGATIVE or NEUTRAL

Summary table

Connection to the project

In the Prompt Analyzer (capsule 08) you could detect whether a prompt uses appropriate parameters:

• Is temperature 0 used for deterministic tasks (classification, extraction)?
• Is max_tokens right for the type of response expected?
• Is frequency_penalty on for long generations?

Troubleshooting

Problem 1: Truncated responses

Symptom: The response is cut off mid-sentence or before the JSON closes.

Diagnosis:

# Check the finish_reason
finish_reason = response.choices[0].finish_reason
print(f"Finish reason: {finish_reason}")  # "length" = truncated

Fix:

• Raise max_tokens gradually (use +200 until finish_reason == "stop")
• Or reduce the expected length in the prompt: "Answer in 50 words maximum"

Problem 2: Inconsistent responses in classification

Symptom: The same text is classified as POSITIVE sometimes and NEUTRAL other times.

Cause: temperature > 0 on a deterministic task.

Fix:

# ✅ Correct for classification
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    temperature=0  # Deterministic
)

# ❌ Problematic
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    temperature=0.7  # Introduces unnecessary variability
)

Problem 3: Very repetitive responses

Symptom: The model repeats the same phrases or ideas.

Fix:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Describe 5 benefits of exercise"}],
    temperature=0.7,
    frequency_penalty=0.8,  # Penalizes word repetition
    presence_penalty=0.3,   # Encourages new concepts
    max_tokens=300
)

Problem 4: Responses that are too conservative or generic

Symptom: For brainstorming, the model gives obvious and repetitive answers.

Cause: Temperature too low for a creative task.

Fix: Raise temperature to 0.7-0.9 for creative tasks. Check that you don't have a low top_p.

Exercises

Exercise 1: Pick the parameters

For each task, pick an appropriate temperature and max_tokens and justify them:

• (a) Extract emails from a text
• (b) Generate 10 ideas for product names
• (c) Answer questions about a technical document
• (d) Generate Python code for a sorting function

Exercise 2: Temperature experiment

Run the same prompt 3 times with temperature=0 and 3 times with temperature=0.9. Compare the variability.

from openai import OpenAI

client = OpenAI()

prompt = "Give me an analogy to explain what machine learning is."

for temp in [0, 0.9]:
    print(f"\n=== Temperature={temp} ===")
    for i in range(3):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=temp,
            max_tokens=80
        )
        print(f"  {i+1}: {r.choices[0].message.content.strip()[:100]}...")

# With temp=0: The 3 responses will be very similar or identical
# With temp=0.9: The 3 responses will be different (different analogies)

Exercise 3: Detect a truncated response

Write code that: (1) makes a call with max_tokens=10, (2) detects whether it was truncated, (3) retries with a higher max_tokens automatically.

from openai import OpenAI

client = OpenAI()

def call_with_retry(system: str, user: str, initial_max_tokens: int = 50) -> str:
    """
    Calls the model and retries with more tokens if the response was truncated.
    """
    max_tokens = initial_max_tokens
    max_retries = 3
    
    for attempt in range(max_retries):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system},
                {"role": "user", "content": user}
            ],
            temperature=0,
            max_tokens=max_tokens
        )
        
        finish_reason = response.choices[0].finish_reason
        content = response.choices[0].message.content
        
        if finish_reason == "stop":
            print(f"Completed on attempt {attempt+1} with max_tokens={max_tokens}")
            return content
        elif finish_reason == "length":
            print(f"Truncated with max_tokens={max_tokens}, retrying with {max_tokens * 2}...")
            max_tokens *= 2  # Double the tokens and retry
        else:
            break
    
    return content  # Return what we have

# Test
result = call_with_retry(
    system="Explain in detail what a transformer is in AI.",
    user="Give me a complete explanation",
    initial_max_tokens=10  # Very low, it will get truncated
)
print(f"\nFinal result:\n{result[:200]}...")

Exercise 4 (Advanced): Temperature benchmark

Implement a benchmark that runs a classification task with temperature 0, 0.3, 0.5, 0.7 and measures consistency (% of identical responses over 5 attempts).

from openai import OpenAI
from collections import Counter

client = OpenAI()

def benchmark_temperature(
    system: str,
    user: str,
    temperatures: list[float],
    n_tries: int = 5
) -> dict:
    """
    Tries different temperatures and measures consistency.
    """
    results = {}
    
    for temp in temperatures:
        responses = []
        for _ in range(n_tries):
            r = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": system},
                    {"role": "user", "content": user}
                ],
                temperature=temp,
                max_tokens=10
            )
            responses.append(r.choices[0].message.content.strip())
        
        counts = Counter(responses)
        most_common = counts.most_common(1)[0]
        consistency = most_common[1] / n_tries * 100
        
        results[temp] = {
            "responses": responses,
            "most_common": most_common[0],
            "consistency_pct": consistency
        }
    
    return results

# Run the benchmark
results = benchmark_temperature(
    system="Classify the sentiment. One word only: POSITIVE, NEGATIVE or NEUTRAL.",
    user="The product arrived late but it was in perfect condition.",
    temperatures=[0, 0.3, 0.5, 0.7, 1.0],
    n_tries=5
)

print("Temperature | Consistency | Most common response")
print("-" * 50)
for temp, data in results.items():
    print(f"    {temp:.1f}    |    {data['consistency_pct']:3.0f}%      | {data['most_common']}")

# Expected output (approximate):
# Temperature | Consistency | Most common response
# --------------------------------------------------
#     0.0    |    100%      | NEUTRAL
#     0.3    |     80%      | NEUTRAL
#     0.5    |     60%      | NEUTRAL
#     0.7    |     60%      | NEUTRAL
#     1.0    |     40%      | NEUTRAL

Summary

• temperature=0: Classification, extraction, code — any task with a correct answer
• temperature 0.2-0.4: Factual QA, summaries, analysis — a balance of precision and fluency
• temperature 0.7-0.9: Creativity, brainstorming, copy — intentional variety
• max_tokens: Tune it to the response type. Check finish_reason to detect truncation.
• top_p: An alternative to temperature. Don't combine both.
• frequency_penalty: Reduce word repetition in long generations.
• presence_penalty: Encourage a diversity of concepts.
• seed: For approximate reproducibility in testing (OpenAI).

Additional resources

1. OpenAI API Parameters — Complete documentation of every available parameter
2. OpenAI Prompt Engineering - Temperature — OpenAI's recommendations for each parameter
3. Anthropic Generation Parameters — Parameters available in the Claude API
4. Temperature in LLMs (Explained) — A technical explanation of how temperature works
5. tiktoken — To count tokens and size max_tokens correctly before calling