Ollama Performance Tuning

Overview

You'll optimize Ollama's performance by adjusting GPU, RAM, quantization, and other parameters.

Time: 20 minutes
Difficulty: Medium

---

🎯 Objectives

• ✅ GPU acceleration
• ✅ Optimize RAM usage
• ✅ Quantization selection
• ✅ Measure performance

---

🚀 GPU Acceleration

Check for an available GPU:

NVIDIA:

nvidia-smi

Apple Silicon (M1/M2/M3):

system_profiler SPDisplaysDataType

---

Enable the GPU in Ollama:

Automatic in most cases. Verify with:

ollama run mistral
# In another terminal:
nvidia-smi  # Should show GPU usage

If it doesn't detect the GPU:

export CUDA_VISIBLE_DEVICES=0
ollama serve

---

💾 RAM Optimization

Custom Modelfile (reduce RAM):

# Create the Modelfile
cat > Modelfile <<EOF
FROM mistral:latest

# Reduce the context window (less RAM)
PARAMETER num_ctx 2048

# Reduce GPU layers
PARAMETER num_gpu 20
EOF

# Build the custom model
ollama create mistral-light -f Modelfile
ollama run mistral-light

---

Monitor RAM usage:

Linux:

watch -n 1 free -h

macOS:

while true; do
  vm_stat | grep "Pages active"
  sleep 1
done

---

📊 Quantization Trade-offs

Quantization	Size (7B)	Quality	Speed	RAM
Q8	7.5 GB	95%	Slow	10GB
Q5	5.0 GB	90%	Medium	7GB
Q4	4.0 GB	85%	Fast	6GB
Q2	2.5 GB	75%	Very fast	4GB

Recommendation:

• 16GB RAM: Q4
• 8GB RAM: Q2

ollama pull mistral:7b-q4
ollama pull mistral:7b-q2

---

⚙️ Parameters Tuning

Complete Modelfile:

cat > Modelfile <<EOF
FROM mistral:latest

# Context window (smaller = less RAM)
PARAMETER num_ctx 4096

# Temperature (0-2)
PARAMETER temperature 0.7

# GPU layers (more = faster)
PARAMETER num_gpu 35

# Threads (CPU cores)
PARAMETER num_thread 8

# Batch size
PARAMETER num_batch 512
EOF

ollama create mistral-tuned -f Modelfile

---

📊 Benchmark Performance

import time
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

def benchmark(model, num_runs=5):
    latencies = []
    
    for i in range(num_runs):
        start = time.time()
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "Hi"}]
        )
        latency = time.time() - start
        latencies.append(latency)
        print(f"Run {i+1}: {latency:.2f}s")
    
    avg = sum(latencies) / len(latencies)
    print(f"\nAverage: {avg:.2f}s")
    return avg

# Test
print("=== Mistral Q4 ===")
benchmark("mistral:7b-q4")

print("\n=== Mistral Q2 ===")
benchmark("mistral:7b-q2")

---

🔥 Hot Reload (Keep in Memory)

Avoid unloading between queries:

# Modelfile
cat > Modelfile <<EOF
FROM mistral:latest

# Keep loaded in memory
PARAMETER keep_alive 3600
EOF

ollama create mistral-persistent -f Modelfile

Default: Ollama unloads after 5 min of inactivity
Custom: Keep for 1 hour (3600s)

---

📊 CPU vs GPU Performance

Hardware	Latency (7B Q4)	Cost
CPU only (i7)	8-12s	$0
Apple M2	3-5s	$2000
NVIDIA RTX 3060	2-3s	$300
NVIDIA RTX 4090	1-2s	$1600

Conclusion: GPU improves speed by 3-5x

---

🐛 Troubleshooting

GPU not detected:

# NVIDIA
export CUDA_VISIBLE_DEVICES=0

# Check drivers
nvidia-smi

---

Out of memory:

# Reduce the context window
PARAMETER num_ctx 2048

# Or use a lower quantization
ollama pull mistral:7b-q2

---

✅ Summary

Key optimizations:

• GPU: 3-5x speed boost
• Quantization: Q4 balance, Q2 if RAM is limited
• Context window: Reduce if OOM
• Keep alive: Avoids cold starts

Workflow:

1. Start with Q4
2. If slow: Enable the GPU
3. If OOM: Drop to Q2
4. If production: Long keep alive

---

Next: 08-project-dockerized-chatbot.md

Final project: A chatbot in Docker with optimized Ollama.

Time: 60 min