Module 5: LLMs (Large Language Models) - GPT, Claude and More

6. Model Parameters: Controlling Generation

Next →

Description

Model parameters (temperature, top_p, max_tokens, etc.) control how the LLM generates text. They're crucial for getting outputs appropriate to your use case.

What you'll learn:

  • Temperature: Creativity vs determinism.
  • top_p (nucleus sampling): Diversity control.
  • max_tokens: The response length limit.
  • frequency_penalty, presence_penalty: Reducing repetition.
  • How to choose values based on your use case.

Parameter 1: Temperature

What it controls: Randomness/creativity in generation.

Range: 0.0 - 2.0 (typically 0.0 - 1.0)

Temperature = 0 (deterministic)

Behavior: It always chooses the most probable token → the same output every time.

Use: Tasks that require consistent answers (classification, data extraction, factual answers).

Example:

Prompt: "Translate to Spanish: Hello"
Output (temp=0): "Hola" (always)

Temperature = 1 (creative)

Behavior: It considers multiple probable tokens → a different output every time.

Use: Creative writing, brainstorming, generating variations.

Example:

Prompt: "Write a poem about cats"
Output (temp=1, run 1): "Whiskers soft as morning dew..."
Output (temp=1, run 2): "Feline grace in moonlight gleams..."

Temperature > 1 (very random)

Behavior: It considers even improbable tokens → a very diverse output, but it can lose coherence.

Use: Experimentation, generating very divergent ideas.

Problem: It can generate nonsense.

Visual comparison

temp=0:  [▓▓▓▓▓▓▓▓▓▓] (always the most probable token)
temp=0.5:[▓▓▓▓▓░░░░░] (a mix of probable and somewhat random)
temp=1:  [▓▓░░░░░░░░] (high diversity)
temp=2:  [░░░░░░░░░░] (very random, it can lose coherence)

Parameter 2: top_p (Nucleus Sampling)

What it controls: The diversity of tokens considered (sampling based on cumulative probability).

Range: 0.0 - 1.0

How it works:

  • top_p = 0.1: It only considers tokens whose cumulative probability is ≤ 10% (very restrictive).
  • top_p = 0.9: It considers tokens up to a cumulative probability of 90% (more diverse).
  • top_p = 1.0: It considers all the tokens.

Common use: top_p = 0.9 - 1.0

Difference from temperature:

  • Temperature: It adjusts the probability distribution.
  • top_p: It filters out the less probable tokens.

Recommendation: Adjust temperature OR top_p, not both (they can interfere with each other).

Parameter 3: max_tokens

What it controls: The maximum length of the response (in tokens).

Example:

max_tokens = 50: The response is limited to 50 tokens (~37 words in English)
max_tokens = 1000: The response can be up to 1000 tokens (~750 words)

Use:

  • Short responses (a chatbot): max_tokens = 150-300.
  • Long articles: max_tokens = 2000-4000.

Important: If the response reaches max_tokens, it's truncated (it can end up incomplete).

Parameter 4: frequency_penalty

What it controls: It penalizes the repetition of tokens based on frequency in the output generated so far.

Range: -2.0 - 2.0

  • 0: No penalty (it can repeat).
  • Positive (e.g. 0.5): It reduces repetition.
  • Negative: It encourages repetition (rare).

Use: Preventing the model from repeating phrases/words.

Parameter 5: presence_penalty

What it controls: It penalizes the repetition of tokens based only on presence (not frequency).

Range: -2.0 - 2.0

Difference from frequency_penalty:

  • frequency_penalty: It penalizes more if a word appears many times.
  • presence_penalty: It penalizes the same whether a word appears 1 or 10 times.

Use: Encouraging topic diversity (the model explores different topics).

How to Choose Parameters Based on the Use Case

Case 1: Classification / Data Extraction

Goal: Consistent, factual answers.

Parameters:

temperature = 0
top_p = 1
max_tokens = 50-100
frequency_penalty = 0
presence_penalty = 0

Case 2: A Support Chatbot

Goal: Useful answers, somewhat varied but coherent.

Parameters:

temperature = 0.3-0.5
top_p = 0.9
max_tokens = 200-500
frequency_penalty = 0.3
presence_penalty = 0

Case 3: Creative Writing

Goal: Diverse, creative outputs.

Parameters:

temperature = 0.8-1.0
top_p = 0.95
max_tokens = 1000-2000
frequency_penalty = 0.5
presence_penalty = 0.5

Case 4: Code Generation

Goal: Correct, syntactically valid code.

Parameters:

temperature = 0-0.2
top_p = 1
max_tokens = 500-2000
frequency_penalty = 0
presence_penalty = 0

Experimentation: Same Prompt, Different Temperatures

Prompt: "Write a tagline for a coffee shop"

temp=0:

"The best coffee in town."
(always the same, generic)

temp=0.5:

"Where every cup tells a story."
(varied but reasonable)

temp=1.0:

"Awakening your senses, one sip at a time."
(more creative, different every time)

Why this matters for an AI Engineer

1. Quality optimization

Without tuning the parameters: Generic or inconsistent outputs.

With tuning: Outputs appropriate to your use case.

2. Debugging

Problem: The model generates different responses every time (it makes testing hard).

Solution: temp=0 during development/testing (consistent responses).

3. Reducing costs

A high max_tokens (e.g. 2000): The model generates long responses → high costs.

Solution: Set max_tokens to the minimum necessary (e.g. 300 for a chatbot).

Common Mistakes

1. Using temp=1 for factual tasks

Mistake: temp=1 for classification or data extraction.

Problem: Inconsistent outputs, they can be incorrect.

Solution: temp=0 for factual tasks.

2. Tuning temperature AND top_p

Mistake: temp=0.5 + top_p=0.5 (both restrict diversity).

Problem: The effects multiply, the output is too restrictive.

Solution: Tune one (usually temperature), leave the other at its default.

Summary

Temperature:

  • 0 = deterministic, 1 = creative.
  • Use 0 for factual tasks, 0.7-1.0 for creativity.

top_p:

  • It controls diversity (nucleus sampling).
  • Use 0.9-1.0 (the default).

max_tokens:

  • The response length limit.
  • Adjust it to your needs (avoid generating unnecessarily long responses).

frequency_penalty, presence_penalty:

  • They reduce repetition.
  • Use 0.3-0.5 if the model repeats a lot.

How to choose:

  • Factual: temp=0, top_p=1.
  • Chatbot: temp=0.3-0.5.
  • Creative: temp=0.8-1.0.
  • Code: temp=0-0.2.

Next step: Lesson 07: The Major LLMs — GPT-4, Claude 3, Llama 3, Gemini → a comparison.

← Previous5. Context Window: LLMs' Memory LimitNext →7. The Major LLMs: GPT-4, Claude 3, Llama 3, Gemini and More