Module 6: The AI API Ecosystem

5. Local vs Cloud: LM Studio, Ollama and When to Use Each One

Next →

Description

The key question: When should you run LLMs locally (on your laptop/server) vs use Cloud APIs (OpenAI, Anthropic)?

Local: Tools like LM Studio, Ollama, llama.cpp let you run open-source models (Llama 3, Mistral) on your own hardware.

Cloud: APIs (OpenAI, Anthropic) → models on external servers.

In this lesson you'll understand:

  • What LM Studio, Ollama, llama.cpp are.
  • The advantages/disadvantages of local vs cloud.
  • When to use each one.

What "Running Locally" Means

Definition: Downloading the model's weights (Llama 3) and running it on your hardware (a laptop, your own server).

Main tools:

  1. Ollama: A CLI for running models locally (simple, fast setup).
  2. LM Studio: A GUI (graphical interface) for running models (friendlier).
  3. llama.cpp: A C++ library for running LLMs (more control, but it requires compiling).

Ollama

What it is: A CLI for downloading and running open-source models locally.

Installation:

# macOS, Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from ollama.com

Use:

# Download Llama 3 (8B)
ollama pull llama3:8b

# Run the model (interactive chat)
ollama run llama3:8b

# Local API (OpenAI-compatible)
ollama serve  # Runs at http://localhost:11434

Advantages:

  • ✅ Setup in 5 minutes.
  • ✅ An OpenAI-compatible API → easy to integrate into code.
  • ✅ Free (open-source).

Disadvantages:

  • ❌ Only small models on a laptop (~7B-13B).
  • ❌ Slow on CPU (it requires a GPU for decent speed).

LM Studio

What it is: A GUI (desktop app) for downloading and running models locally.

Installation: Download from lmstudio.ai (macOS, Windows, Linux).

Advantages:

  • ✅ A graphical interface (no terminal required).
  • ✅ Download models with one click.
  • ✅ Built-in chat (for testing).
  • ✅ An OpenAI-compatible local API.

Disadvantages:

  • ❌ It requires a GUI (not ideal for servers).
  • ❌ Limited to local hardware.

llama.cpp

What it is: A C++ library for running LLMs efficiently (optimized for CPU, GPU, Metal on Mac).

Advantages:

  • ✅ Faster than Python (C++).
  • ✅ Optimized for Apple Silicon (Metal).
  • ✅ Lower memory usage (quantization).

Disadvantages:

  • ❌ It requires compiling.
  • ❌ Less beginner-friendly.

Quantization: How to Run Large Models on Limited Hardware

Problem: Llama 3 (70B) requires ~140 GB of RAM → impossible on a laptop.

Solution: Quantization (reducing the precision of the weights).

  • FP16 (16-bit): Full precision (2 bytes per parameter).
  • Q8 (8-bit): 50% less memory (1 byte per parameter).
  • Q4 (4-bit): 75% less memory (0.5 bytes per parameter).

Example:

  • Llama 3 (70B) in FP16: 140 GB.
  • Llama 3 (70B) in Q4: 35 GB → it fits on a consumer GPU (an RTX 4090: 24 GB VRAM + RAM).

Trade-off: Quantization reduces quality slightly (but it's usually imperceptible at Q8/Q4).

Comparison: Local vs Cloud

AspectLocal (Ollama, LM Studio)Cloud (the OpenAI API)
Setup5-30 minutes (downloading the model)5 minutes (an API key)
CostFree (only hardware)Per use ($0.0005-0.03/1K tokens)
PrivacyTotal (data doesn't leave your hardware)Low (data goes to the provider)
LatencyLow (local) or high (if the hardware is slow)Medium (an API call)
CapabilityLimited by hardware (8B-70B locally)Very high (GPT-4, Claude)
ScalabilityLimited (1 user)Unlimited (millions of users)
InternetNot required (offline)Required (an API call)

When to Use Local

1. Critical Privacy

Example: Analyzing confidential documents (contracts, medical data).

Decision: Local (Ollama + Llama 3) → the data doesn't leave your hardware.

2. Development/Testing

Example: Testing prompts without spending on an API.

Decision: Local → free, fast to iterate.

3. No Internet

Example: An application that must work offline (e.g. on a plane).

Decision: Local → it doesn't require internet.

4. Low Volume, Simple Tasks

Example: A personal assistant, categorizing local files.

Decision: Local (Llama 3 8B) → enough capability, free.

When to Use Cloud

1. Production with High Volume

Example: A chatbot with 100K users/day.

Decision: A cloud API → it scales automatically, your laptop can't handle that.

2. Complex Tasks

Example: Mathematical reasoning, complex code.

Decision: Cloud (GPT-4, Claude Opus) → a local Llama 3 (8B) is insufficient.

3. No Powerful Hardware

Example: You only have an office laptop (no GPU).

Decision: Cloud → local would be very slow.

A Hybrid Strategy: Local + Cloud

The best of both worlds:

  • Local (Ollama + Llama 3 8B): Development, testing, prototyping.
  • Cloud (GPT-4): Production, complex tasks.

Flow:

  1. Develop with Llama 3 locally (free, fast to iterate).
  2. When it works, deploy to production with GPT-4 (better quality).

Hardware Requirements

For small models (7B-13B)

CPU:

  • Llama 3 (8B) in Q4: ~6 GB RAM.
  • Latency: ~2-5 seconds per response (slow, but usable).

GPU (NVIDIA, AMD, Apple Silicon):

  • Llama 3 (8B) in Q4: ~6 GB VRAM.
  • Latency: ~0.5-1 second per response (fast).

For large models (70B-400B)

A powerful GPU:

  • Llama 3 (70B) in Q4: ~35 GB VRAM.
  • You need an RTX 4090 (24 GB) + RAM, or multiple GPUs.

Common Mistakes

1. Trying to run a very large model on limited hardware

Mistake: Downloading Llama 3 (70B) on a laptop with 8 GB of RAM.

Result: OOM (Out of Memory), a crash.

Solution: Use a small model (8B-13B) or quantization (Q4).

2. Expecting GPU speed on a CPU

Mistake: Running Llama 3 on a CPU, expecting low latency.

Result: 5-10 seconds per response (slow).

Solution: Use a GPU (NVIDIA, AMD, Apple Silicon) or a Cloud API.

Why this matters for an AI Engineer

1. Fast development

Local (Ollama):

  • Free prompt testing.
  • Fast iteration without worrying about API costs.

2. Privacy during development

Local: Sensitive data (contracts, proprietary code) doesn't leave your hardware during development.

3. Offline support

Local: Apps that work without internet (critical for certain use cases).

Summary

Local (Ollama, LM Studio):

  • Free, total privacy, offline.
  • Limited by hardware (8B-13B on a laptop).
  • Ideal for development, testing, privacy.

Cloud (OpenAI, Anthropic):

  • Unlimited scale, state-of-the-art (GPT-4).
  • Cost per use, requires internet.
  • Ideal for production, complex tasks.

When to use:

  • Local: Privacy, development, offline, low volume.
  • Cloud: Production, high volume, complex tasks.

A hybrid strategy: Develop locally (free), deploy to the cloud (scale).

Next step: Lesson 06: Aggregators and Routers — OpenRouter, Together.ai, Replicate → unified access to multiple models.

← Previous4. Pricing and Token Economics: How You're Charged and How to Optimize CostsNext →6. Aggregators and Routers: OpenRouter, Together.ai and Unified Access