Module 4: Ollama - Introduction

Mini-Project: Chatbot in Docker with Ollama

Next →

Project overview

Final project of Module 4: A production-ready chatbot in Docker with Ollama, health checks, and persistent storage.

Time: 60 minutes
Difficulty: Medium-High

🎯 Objective

Build a complete stack:

  • ✅ Ollama in a Docker container
  • ✅ Python chatbot API (FastAPI)
  • ✅ Persistent storage (models + conversations)
  • ✅ Health checks
  • ✅ Docker Compose orchestration

📁 Project Structure

chatbot-docker/
├── docker-compose.yml
├── ollama/
│   └── Modelfile
├── api/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── main.py
│   └── chatbot.py
└── README.md

🐳 Docker Compose Setup

docker-compose.yml:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3
    environment:
      - OLLAMA_HOST=0.0.0.0

  chatbot-api:
    build: ./api
    container_name: chatbot-api
    ports:
      - "8000:8000"
    depends_on:
      ollama:
        condition: service_healthy
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - conversations_data:/app/conversations
    restart: unless-stopped

volumes:
  ollama_data:
  conversations_data:

🔧 Ollama Modelfile

ollama/Modelfile:

FROM mistral:latest

# Optimized for production
PARAMETER num_ctx 4096
PARAMETER temperature 0.7
PARAMETER keep_alive 3600
PARAMETER num_gpu 35

🐍 API with FastAPI

api/requirements.txt:

fastapi==0.109.0
uvicorn==0.27.0
openai==1.12.0
pydantic==2.5.0

api/Dockerfile:

FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy app code
COPY . .

# Create conversations dir
RUN mkdir -p /app/conversations

# Expose port
EXPOSE 8000

# Run API
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

api/chatbot.py:

"""Chatbot logic"""

from openai import OpenAI
import os
import json
from datetime import datetime
from pathlib import Path

class OllamaChatbot:
    """Chatbot with an Ollama backend."""
    
    def __init__(self, model: str = "mistral"):
        self.client = OpenAI(
            base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434") + "/v1",
            api_key="ollama"
        )
        self.model = model
        self.conversations_dir = Path("/app/conversations")
        self.conversations_dir.mkdir(exist_ok=True)
    
    def chat(self, session_id: str, messages: list) -> dict:
        """Send chat and return response."""
        
        try:
            response = self.client.chat.completions.create(
                model=self.model,
                messages=messages,
                temperature=0.7,
                max_tokens=500
            )
            
            assistant_message = response.choices[0].message.content
            
            # Save conversation
            self._save_conversation(session_id, messages + [
                {"role": "assistant", "content": assistant_message}
            ])
            
            return {
                "response": assistant_message,
                "model": self.model,
                "session_id": session_id
            }
            
        except Exception as e:
            return {"error": str(e)}
    
    def _save_conversation(self, session_id: str, messages: list):
        """Save conversation to disk."""
        
        filepath = self.conversations_dir / f"{session_id}.json"
        
        data = {
            "session_id": session_id,
            "timestamp": datetime.now().isoformat(),
            "messages": messages
        }
        
        with open(filepath, "w") as f:
            json.dump(data, f, indent=2)

api/main.py:

"""FastAPI REST API"""

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List
from chatbot import OllamaChatbot
import uuid

app = FastAPI(title="Ollama Chatbot API")
chatbot = OllamaChatbot()

class Message(BaseModel):
    role: str
    content: str

class ChatRequest(BaseModel):
    session_id: str | None = None
    messages: List[Message]

class ChatResponse(BaseModel):
    response: str
    model: str
    session_id: str

@app.get("/")
def read_root():
    return {"status": "healthy", "service": "Ollama Chatbot API"}

@app.get("/health")
def health_check():
    return {"status": "ok"}

@app.post("/chat", response_model=ChatResponse)
def chat(request: ChatRequest):
    """
    Chat endpoint.
    
    Example:
    {
      "messages": [
        {"role": "user", "content": "Hi"}
      ]
    }
    """
    
    # Generate session ID if not provided
    session_id = request.session_id or str(uuid.uuid4())
    
    # Convert Pydantic models to dicts
    messages = [msg.dict() for msg in request.messages]
    
    # Get response from chatbot
    result = chatbot.chat(session_id, messages)
    
    if "error" in result:
        raise HTTPException(status_code=500, detail=result["error"])
    
    return result

@app.get("/models")
def list_models():
    """List available models."""
    return {"models": ["mistral"]}

🚀 Deployment Instructions

1. Build and start:

cd chatbot-docker

# Build images
docker-compose build

# Start stack
docker-compose up -d

# Check the logs
docker-compose logs -f

2. Pull the Ollama model:

docker exec ollama ollama pull mistral

3. Test the API:

# Health check
curl http://localhost:8000/health

# Chat request
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Hi, how are you?"}
    ]
  }'

Output:

{
  "response": "Hi! I'm doing well, thanks. How can I help you?",
  "model": "mistral",
  "session_id": "a1b2c3d4-..."
}

📊 Monitoring

Logs:

# All services
docker-compose logs -f

# Ollama only
docker-compose logs -f ollama

# API only
docker-compose logs -f chatbot-api

Stats:

docker stats ollama chatbot-api

Health checks:

# Ollama
curl http://localhost:11434/api/tags

# API
curl http://localhost:8000/health

🧪 Test Script

import requests

API_URL = "http://localhost:8000"

# Test 1: Health
response = requests.get(f"{API_URL}/health")
print("Health:", response.json())

# Test 2: Simple chat
response = requests.post(
    f"{API_URL}/chat",
    json={
        "messages": [
            {"role": "user", "content": "Hi"}
        ]
    }
)
print("\nChat:", response.json())

# Test 3: Conversation with context
session_id = response.json()["session_id"]
response = requests.post(
    f"{API_URL}/chat",
    json={
        "session_id": session_id,
        "messages": [
            {"role": "user", "content": "Hi"},
            {"role": "assistant", "content": "Hi, how are you?"},
            {"role": "user", "content": "What is the capital of France?"}
        ]
    }
)
print("\nContextual:", response.json())

🔧 Troubleshooting

Problem 1: The Ollama container won't start

Symptom:

Error: failed to start container: container init failed

Common causes:

  1. Port 11434 already in use
  2. Corrupted volume
  3. Insufficient permissions

Solution:

# 1. Check ports
lsof -i :11434
# If something is using the port, kill it or change the port in docker-compose.yml

# 2. Clean up volumes
docker-compose down -v
docker volume prune

# 3. Recreate everything
docker-compose up --build

Problem 2: GPU not detected in the container

Symptom:

WARNING: No GPU detected, using CPU

Cause: NVIDIA Container Toolkit not installed or misconfigured.

Solution:

# Check NVIDIA drivers
nvidia-smi

# Install NVIDIA Container Toolkit (Linux)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Update docker-compose.yml
services:
  ollama:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Verify:

docker exec -it ollama nvidia-smi

Problem 3: API returns "Connection refused"

Symptom:

ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Causes:

  1. The Ollama service isn't healthy
  2. Network issue between containers
  3. API using the wrong URL

Solution:

# 1. Check Ollama's health
docker-compose ps

# If it's not healthy, check the logs
docker-compose logs ollama

# 2. Check the network
docker network inspect chatbot-docker_default

# 3. Manual test from the API container
docker exec -it chatbot-api curl http://ollama:11434/api/tags

# If it fails, check that depends_on is configured with condition: service_healthy

Fix in docker-compose.yml:

chatbot-api:
  depends_on:
    ollama:
      condition: service_healthy  # Critical

Problem 4: Model not found

Symptom:

Error: model 'mistral:latest' not found

Cause: The model isn't pulled into the Ollama volume.

Solution:

# Enter the Ollama container
docker exec -it ollama bash

# List available models
ollama list

# If it's missing, pull it
ollama pull mistral:latest

# Verify
ollama list

To pre-pull on startup:

Modify the Ollama Dockerfile:

FROM ollama/ollama:latest

# Pre-pull models
RUN ollama pull mistral:latest && \
    ollama pull llama2:latest

Problem 5: Conversations don't persist

Symptom: After docker-compose down, conversations are lost.

Cause: Volume not mounted correctly.

Solution:

# Check that the volume exists
docker volume ls | grep conversations

# If it doesn't exist, recreate it
docker-compose down
docker-compose up -d

# Check the mount
docker exec -it chatbot-api ls -la /app/conversations

# If empty, check permissions
docker exec -it chatbot-api chown -R 1000:1000 /app/conversations

Problem 6: Slow performance (high latency)

Symptom: Responses take >30s.

Causes:

  1. Model too large for the CPU
  2. Not using the GPU
  3. num_ctx too high
  4. Cold start

Solution:

# 1. Check resources
docker stats

# 2. Optimize the Modelfile
PARAMETER num_ctx 2048  # Reduce from 4096
PARAMETER num_gpu 35    # Use the GPU
PARAMETER num_thread 8  # Increase threads

# 3. Use a smaller model
ollama pull mistral:7b-instruct-q4_0  # Quantized

Benchmark:

import time

start = time.time()
# ... request ...
latency = time.time() - start
print(f"Latency: {latency:.2f}s")

# Target: <5s on CPU, <2s on GPU

Problem 7: Docker Compose out of memory

Symptom:

Error: OOMKilled

Cause: Container exceeds its memory limit.

Solution:

# docker-compose.yml
services:
  ollama:
    deploy:
      resources:
        limits:
          memory: 8G  # Increase depending on the model
        reservations:
          memory: 4G

Verify:

docker stats ollama

Problem 8: Health check fails intermittently

Symptom:

Health check failed: connection timeout

Cause: Timeout too short or Ollama takes a while to start.

Solution:

ollama:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
    interval: 30s  # Increase from 10s
    timeout: 10s   # Increase from 5s
    retries: 5     # Increase from 3
    start_period: 60s  # Give it more initial time

Problem 9: Windows/Mac GPU issues

Symptom: GPU doesn't work in Docker Desktop (Windows/Mac).

Cause: Docker Desktop on Windows/Mac has limited GPU support.

Solution:

Windows (WSL2):

# Use WSL2 with NVIDIA support
wsl --install
# Install NVIDIA drivers in WSL2

Mac:

# There is no native GPU support on Mac
# Use CPU or deploy to the cloud

Alternative: Deploy to the cloud (Vast.ai, RunPod) with a GPU.

Problem 10: Logs too verbose

Symptom: Logs fill up the disk.

Solution:

services:
  ollama:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Check the size:

du -sh /var/lib/docker/containers/*/

✅ Pre-Deploy Checklist

Before considering it production-ready:

  • Health checks work consistently
  • GPU detected (if applicable)
  • Volumes persist data correctly
  • Logs configured with rotation
  • Appropriate memory limits (4-8GB)
  • Restart policy = unless-stopped
  • Latency test <5s average
  • Test of 100+ consecutive requests without a crash
  • Backup strategy for volumes
  • Monitoring configured (logs accessible)

✅ Self-Assessment Rubric

Setup (30 pts):

  • (10) Docker Compose works
  • (10) Ollama container healthy
  • (10) API container healthy

Functionality (40 pts):

  • (15) API responds to requests
  • (15) Conversations are saved
  • (10) Health checks work

Production-ready (30 pts):

  • (10) Persistent storage configured
  • (10) Restart policy configured
  • (10) Logs accessible

Total: ___/100

🚀 Optional Extensions

1. Nginx reverse proxy:

Add to docker-compose.yml:

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - chatbot-api

2. Redis cache:

  redis:
    image: redis:alpine
    ports:
      - "6379:6379"

Cache common responses.

3. Prometheus monitoring:

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

✅ Module 4 Summary

What you mastered:

  • ✅ Ollama CLI (pull, run, list)
  • ✅ Local REST API
  • ✅ Docker deployment
  • ✅ Performance tuning
  • ✅ Production-ready stack

Differences vs LM Studio:

  • CLI vs GUI
  • Docker support
  • Automation
  • Production focus

➡️ Next Module

Module 5: OpenRouter (Multi-Provider Aggregator)

You'll learn OpenRouter, an aggregator that gives you access to 100+ models (OpenAI, Anthropic, Google, etc.) with a single API.

Features:

  • 100+ models available
  • Cost optimization
  • Automatic fallback
  • Dynamic switching

Time: 2-3 hours

Congratulations! You completed Module 4. 🎉

← PreviousOllama Performance TuningNext →Module 5: OpenRouter - Introduction