4. Local models with Ollama: a $0 agent (Llama 3, Mistral)

Description

In the previous lesson you connected the agent's model node to Anthropic, OpenAI, and Google credentials: three cloud providers that charge per token and receive your prompts on their own servers. In this capsule you're going to do the opposite. You're going to spin up the Self-Hosted AI Starter Kit v2, connect that same node to Ollama, and run your agent with Llama 3.2 or Mistral running on your own machine. By the end you'll be able to spin up the local stack with the right profile for your hardware, connect the AI Agent to a local model with no per-token cost, and — most importantly — decide with judgment when a local model is enough for the job and when it's better to go back to the cloud.

This matters beyond the exercise. There are clients — a law firm, a clinic, a fintech with regulated data — where sending information out to an external provider isn't negotiable, no matter how strong its privacy policy sounds. Knowing how to build an agent that never touches the internet is as much a sales skill as a technical one: you can answer that objection with "this runs entirely on your server," instead of with a contractual promise.

Connection to the module: you already chose current models and connected cloud providers (lesson 3). Here you add the third option — local — before writing the system prompt (lesson 5) that gives the agent personality and boundaries, no matter what engine runs behind it.

Cloud vs. your own garage: the mental model

Think of the two ways to get around a city you don't know. Hiring a car with a driver — you tell them where you're going, they take you, you pay for the trip — is the cloud: Anthropic, OpenAI, and Google run the model on their infrastructure, you pay per token, and you depend on their server being up and your connection holding. Buying a used car and keeping it in your garage is the other option: the trip is free whenever you want to make it, you don't need anyone's permission to start it, but the car is yours — with your battery, your tank, your maintenance — and it doesn't have the same power as a professional driver's fleet.

A local model is that car in your garage. Ollama is the engine that starts it: it downloads a model's weights — Llama 3.2, Mistral, and dozens more — to your disk, and exposes an API on your own machine, with no key, no bill, no going out to the internet, so any program — n8n included — can send it prompts and get responses back.

The Self-Hosted AI Starter Kit v2 is the fastest way to have that garage already built. It's a Docker Compose stack that, with a single command, spins up four pieces: n8n (where your agent lives), Ollama (the local engine), Qdrant (a vector database, which you'll use later in the RAG modules), and Postgres (where n8n stores its workflows and credentials). It doesn't introduce any new concept: it bundles pieces you already knew under a single docker compose up.

Worked example

Step 1 — Clone the stack and spin it up.

git clone https://github.com/n8n-io/self-hosted-ai-starter-kit.git
cd self-hosted-ai-starter-kit
cp .env.example .env
docker compose --profile cpu up

The --profile flag tells Docker Compose which Ollama engine to spin up: cpu if your machine has no dedicated GPU (this includes Apple Silicon, because Docker on Mac can't expose the GPU to a container), gpu-nvidia if you have an NVIDIA card with CUDA drivers, or gpu-amd for AMD on Linux. If you have a Mac with a powerful GPU and want to take advantage of it, the alternative is installing Ollama natively on macOS (outside Docker) and pointing the n8n container at host.docker.internal:11434 instead of using the cpu profile.

What to expect: in the logs you'll see a container called ollama-pull-llama-cpu (the name changes depending on the profile: ollama-pull-llama-gpu or ollama-pull-llama-gpu-amd) running ollama pull llama3.2. That's the kit's default model downloading automatically — 2.0 GB — and it can take several minutes depending on your connection. When it finishes, n8n becomes available at http://localhost:5678.

Step 2 — Confirm Ollama responds before touching n8n.

docker exec -it $(docker ps -qf "name=ollama-cpu") ollama list

What to expect:

NAME               ID              SIZE      MODIFIED
llama3.2:latest    a80c4f17acd5    2.0 GB    3 minutes ago

If this list comes back empty, step 1's download hasn't finished yet — go back to docker compose's logs before continuing.

Step 3 — Create the Ollama credential in n8n.

Go to http://localhost:5678, create a credential of type Ollama, and under Base URL put:

http://ollama-cpu:11434

Here's the detail that trips up most people on the first try: it's not http://localhost:11434, even though that's the port Ollama exposes. n8n and Ollama run in different containers within the same Docker Compose network, and inside a container, localhost points to the container itself, not to its neighbors. The correct name is that of the service as defined in the kit's docker-compose.yml — ollama-cpu, ollama-gpu, or ollama-gpu-amd, depending on which profile you used to spin up the stack. Docker resolves that name as if it were a hostname within its own internal network.

Step 4 — Connect the model to the AI Agent.

Build a minimal workflow: Chat Trigger → AI Agent, and in the AI Agent connect the Ollama Chat Model node as the language model input. In its Model field you'll see a dropdown that fills itself in automatically with the models you have downloaded on that Ollama instance — the same result ollama list gave you in step 2. Choose llama3.2:latest.

Send a test message from the Chat Trigger:

Explain in one sentence what a vector database does.

What to expect (the exact wording varies with the model's sampling, this is a representative example):

A vector database stores information as numerical coordinates that
represent its meaning, so it can find similar content by closeness
instead of exact word matching.

Step 5 — Compare against Mistral.

docker exec -it $(docker ps -qf "name=ollama-cpu") ollama pull mistral

Once the download finishes (4.4 GB), go back to the Ollama Chat Model node, switch the dropdown to mistral:latest, and run the same test message. You'll notice differences in tone and length — and probably in latency, since Mistral has more than double the parameters of Llama 3.2 3B. Don't dig into measuring those differences precisely just yet — that's what the debugging engine in lesson 7 is for. Here the goal is only to confirm you can swap local models without touching the rest of the workflow.

When a local model is enough and when it isn't

The question isn't "is the local model worse?" It's "worse at what?"

Tools (tool calling). If in a later module you give your agent tools — searching an API, querying a database — the model needs to be able to emit a response in the tool_calls format n8n recognizes as "I want to run this tool." Ollama documents that support for specific models: Llama 3.1, Llama 3.2, Mistral Nemo, Firefunction v2, and Command-R+, among others. The base Mistral 7B you just downloaded — the one under the mistral:latest tag — isn't on that list. Before relying on tool calling with a local model, check its page at ollama.com/library: if it doesn't mention tool support, the agent might "say" it executed something without actually having done so.

Hardware and latency. A local model runs on whatever CPU or GPU you have, not on a data center's infrastructure. Llama 3.2's 3B version (2.0 GB on disk) responds in a couple of seconds on an ordinary laptop. Move up to a 13B model or bigger and, with no GPU, the wait becomes noticeable — that's the real cost of "free": you don't pay per token, you pay with your own machine's time and resources.

Data residency. This is where a local model wins even if its quality is lower: if the client's contract requires that information never leave their infrastructure, no cloud provider — however good — meets that condition. Ollama does, because the prompt never leaves the machine it runs on.

Depth of reasoning. For narrow, well-defined tasks — classifying, summarizing, drafting a short reply — a local model from 3B to 7B is usually enough. For multi-step reasoning, extensive context, or real ambiguity, the latest generation of cloud models still wins by a margin. The choice of model, at bottom, follows the same criterion you saw in lesson 2 — except now "current" also means "fits on your hardware."

Common mistakes

Using localhost as the Ollama credential's Base URL. What happens: the credential's connection test fails with something like "connection refused," even though curl http://localhost:11434 works perfectly from your terminal. Why: your terminal runs on the host operating system, where Ollama does publish port 11434 on localhost. But n8n runs inside its own container, with its own network namespace — there, localhost only refers to itself. How to spot it: the error only shows up when testing the credential from n8n's interface, never when testing from your machine. How to fix it: use the service name within the Docker Compose network (ollama-cpu, ollama-gpu, or ollama-gpu-amd, depending on which profile you spun up), or host.docker.internal if Ollama runs natively outside Docker.

Testing the workflow before the initial model download finishes. What happens: the Ollama Chat Model node returns a model-not-found error, or the execution hangs with no response. Why: the ollama-pull-llama-* service downloads llama3.2 in parallel with n8n's startup, and those 2.0 GB can take several minutes on a slow connection — n8n can become available on port 5678 before the download finishes. How to spot it: check docker compose's logs, look for the ollama-pull-llama-cpu container's line (or its GPU equivalent). How to fix it: wait for the log to confirm the download is complete, or run it manually and confirm with ollama list as in step 2 of the example.

Assuming "any local model works for everything the cloud one did." What happens: you connect mistral:latest to an agent thinking you'll give it tools later, and the agent fails to invoke them — or makes up that it did. Why: as you saw above, not every model that runs in Ollama emits the tool_calls format the AI Agent needs to trigger a real tool, and the base Mistral 7B isn't among the ones Ollama documents with that support. How to spot it: in the workflow's execution log, the tool node never actually fires even though the agent "narrates" in its response that it used it. How to fix it: check the model's tool-calling support on its ollama.com/library page before building an agent that depends on tools running on top of it.

Exercises

1. Spin up the stack with docker compose --profile cpu up and, once it's up, confirm with a Docker command that ollama list shows llama3.2:latest downloaded. Write the exact command you'd use.

docker exec -it $(docker ps -qf "name=ollama-cpu") ollama list

2. You spun up the stack with docker compose --profile gpu-nvidia up and you're setting up the Ollama credential in n8n. What value goes in Base URL, and why not localhost?

3. Run the same question against llama3.2:latest and mistral:latest on your AI Agent. Note one substantive difference — not just in wording — between the two responses.

4. A client asks you for an agent that summarizes internal contracts, and by regulation those documents can't leave the company's server under any circumstances. Cloud or local? Justify your answer with what you saw in this capsule.

Summary and next step

Before moving on you should be able to spin up the Self-Hosted AI Starter Kit v2 with the right profile for your hardware, connect the Ollama credential using the service name — not localhost —, build an AI Agent with the Ollama Chat Model node, and explain with a concrete case when a local model is enough and when it's better to go back to the cloud.

You now have the agent's engine sorted out — cloud or local, based on what you saw in lesson 3 and in this one. What's left is telling that engine who it is and what it shouldn't do, no matter what model runs behind it: that's the system prompt, and it's exactly what you build in the next capsule.

Resources

• Self-hosted AI Starter Kit v2 — the complete docker compose, the hardware profiles, and the reference .env file.
• Ollama credential in n8n — the credential's exact fields and notes on connectivity between containers.
• Ollama Chat Model node — how it connects to the AI Agent and what parameters it exposes.
• Llama 3.2 on Ollama — available sizes, context window, and tool-use support.
• Mistral on Ollama — available sizes and published versions.
• Tool support in Ollama — which models emit tool_calls and which don't.