Module 2: Machine Learning Fundamentals

6. Training vs Inference: The Most Important Distinction for AI Engineering

Next →

Description

In this lesson you'll understand the training vs inference distinction: the difference between training a model (the learning phase: expensive, long, requiring massive data and compute) and using an already-trained model (the application phase: fast, cheap, what you do as an AI Engineer). This is the most important distinction in the module and possibly in the whole guide, because it defines your role: as an AI Engineer, you don't train models from scratch; you integrate already-trained models (from OpenAI, Anthropic, Google, Meta, etc.) into applications.

This matters because when we talk in later modules about "calling an LLM via API" or "using GPT-4", you'll be in the inference phase; when we talk about "OpenAI trained GPT-4 on X terabytes of text", you'll be looking at the training phase. Without this distinction, terms like "pre-trained model", "fine-tuning" or "inference latency" would be black boxes. Here you build that foundation: what training is, what inference is, and why your role is almost always in inference.

What is Training?

Definition: The phase in which the model learns from data. The algorithm adjusts the model's internal parameters (weights, biases, etc.) to minimize errors on the training data. This phase is expensive (it requires a lot of data, massive compute, time) and is done by a specialized team or a provider (OpenAI, Google, Meta).

Flow:

Training data → Learning algorithm → Trained model

Concrete example (GPT-3):

  • Training data: 300 billion tokens of text (books, articles, the internet, code, etc.).
  • Learning algorithm: Training a Transformer (a neural network architecture) with backpropagation and optimization.
  • Compute: Thousands of GPUs over weeks.
  • Cost: Estimated in the millions of dollars.
  • Result: A model with 175 billion parameters that can generate coherent text.

Key point: This phase was done by OpenAI; as an AI Engineer you don't take part in GPT-3's training (unless you do fine-tuning, which is a smaller additional training run).

What is Inference?

Definition: The phase in which the already-trained model is used to make predictions or generate responses on new data. This phase is fast and cheap (compared with training); it only requires passing the new data through the model and getting the output. This is the phase you work in as an AI Engineer.

Flow:

New data → Trained model → Prediction / Response

Concrete example (using GPT-3 via API):

  • Model: GPT-3, already trained by OpenAI (you don't train it).
  • New data: A prompt you pass to the model (e.g. "Summarize this article").
  • Inference: The model generates a response (a summary of the article) in ~2-3 seconds.
  • Cost: Cents per call (depending on the number of tokens).

Key point: You didn't train GPT-3; you just use it. That's inference: using the already-trained model.

Comparison: Training vs Inference

AspectTrainingInference (Use)
What it doesLearns from data; adjusts parametersUses the trained model to predict
DataMillions/billions of examplesOne example at a time (or a small batch)
ComputeThousands of GPUs, days/weeks1 GPU (or CPU), seconds/milliseconds
CostMillions of $ (large models)Cents/dollars per call
Who does itML/Research teams, providersAI Engineers, developers, end users
FrequencyOnce (or a few times, if retraining)Thousands/millions of times a day
ResultA trained model (a file with parameters)A prediction or response

Key analogy: Training is like building a car in a factory (expensive, slow, requiring heavy machinery); inference is like driving the car (fast, cheap, what the end user does). As an AI Engineer you don't build the car (you don't train GPT-4); you drive it (you use GPT-4 via API).

Why this distinction defines your role

As an AI Engineer, you work in Inference

Your job isn't to train models from scratch (that's done by ML Engineering or Research teams at OpenAI, Anthropic, Google, Meta). Your job is to:

  1. Choose the right model: OpenAI, Anthropic, Google? GPT-4, Claude, Gemini? Open-source (Llama, Mistral) or proprietary?
  2. Design prompts: What input you give the model to get the response you need.
  3. Integrate the model into your application: Connecting the model's API with your backend, frontend, database, etc.
  4. Evaluate responses: Does the model respond well? Do you need RAG to improve accuracy?
  5. Handle fallbacks: What to do if the model fails or the API is down?

All of this is inference: using already-trained models. You don't touch the training phase (unless you do fine-tuning, which is a smaller additional training run).

Exception: Fine-tuning

Fine-tuning is additional training: you take a pre-trained model (e.g. GPT-3.5) and adjust it with your own data (e.g. your company's conversations). That's still training, but it's much cheaper and faster than training from scratch because the model already learned general patterns; you're only specializing it.

As an AI Engineer, you can do fine-tuning when:

  • You need to improve performance in a specific domain (e.g. your industry's jargon).
  • You have your own data that you want the model to learn.
  • The cost of fine-tuning justifies the improvement (vs using the base model with good prompts or RAG).

But it's still the exception; most of the work is inference (using base models via API).

Concrete examples: Training vs Inference

Example 1: Spam Filter (Gmail)

  • Training: Google trained a model with millions of labeled emails (spam/not-spam) over weeks. It required ML Engineering teams, large-scale data, and massive compute. Result: a model that classifies emails.
  • Inference: Every time you receive a new email, the model (already trained) classifies it in milliseconds. That happens millions of times a day for all Gmail users. It's fast and cheap (once the model is trained).

Example 2: GPT-4 (OpenAI)

  • Training: OpenAI trained GPT-4 with hundreds of billions of tokens of text (books, the internet, code, etc.) over months with thousands of GPUs. Estimated cost: tens of millions of dollars. Result: a model that generates coherent text.
  • Inference: Every time you call the OpenAI API with a prompt ("Summarize this article"), the model (already trained) generates a response in ~2-3 seconds. Cost: cents per call. That's what you do as an AI Engineer: use the model in inference.

Example 3: Image Recognition Model (Google Photos)

  • Training: Google trained a model with millions of labeled photos (cat, dog, person, car, etc.) using deep neural networks and GPUs over days. Result: a model that recognizes objects.
  • Inference: Every time you upload a photo to Google Photos, the model (already trained) analyzes it in milliseconds and predicts what's in it. That's inference.

Common pattern: Training is expensive and done by a provider; inference is fast and used by you (or the end user).

Why Training is expensive

Training requires:

  1. A lot of data: For large models (LLMs, vision), terabytes of data are used. Collecting, cleaning and labeling (where applicable) that data is expensive.

  2. A lot of compute: Training deep networks with billions of parameters requires thousands of GPUs (or TPUs) over days, weeks or months. The compute cost can run into millions of dollars for large models.

  3. Technical expertise: Training large models requires ML Engineering, Research and Data Engineering teams: choosing the architecture, tuning hyperparameters, evaluating results, iterating.

  4. Time: Training a large model can take weeks or months (from data collection to the final model).

That's why most AI Engineers don't train models from scratch; they use models pre-trained by providers (OpenAI, Anthropic, Google, Meta) via APIs.

Why Inference is cheap (relatively)

Once the model is trained, inference is much cheaper:

  1. No parameter adjustment: The model already learned; it just applies what it knows. There's no backpropagation or optimization.

  2. Less compute: Inference requires 1 GPU (or even a CPU) per call, not thousands of GPUs. For small models, it can run on CPU in milliseconds.

  3. Scalable: You can do millions of inferences a day with the same initial investment (the trained model). The cost per call is low.

  4. Fast: Inference takes seconds or milliseconds (depending on the model and hardware); training takes days or weeks.

Note: Inference isn't "free"; providers (OpenAI, Anthropic) charge you for the tokens processed. But it's orders of magnitude cheaper than training the model from scratch.

Implications for AI Engineering

1. Your role is inference, not training

When you design an AI system (chatbot, RAG, agent, classifier), you're choosing already-trained models and calling them via API. You don't train GPT-4; you use it. You don't train OpenAI's embedding model; you use it. That's the inference phase.

2. Fine-tuning is the exception, not the rule

Some AI Engineers do fine-tuning (training a pre-trained model a bit further with their own data). But that's an exception; most of the work is using base models via API (inference) with good prompts and RAG.

3. Optimizing inference costs

Since you work in inference, your concern is optimizing cost per call:

  • Choosing the right model (cheaper or more capable depending on the case).
  • Reducing the tokens sent (summarizing, bounding context).
  • Caching responses when possible.
  • Using small models for simple tasks (e.g. classification) and large models only for complex tasks (e.g. creative generation).

4. Latency and user experience

Inference must be fast (seconds, not minutes). If you use a very large model that takes 10 seconds to respond, the user experience suffers. That's why you choose models based on the latency/quality trade-off.

Common mistakes when confusing training and inference

1. Believing you can "train" GPT-4 with your data without fine-tuning

Mistake: Thinking that if you give it many examples in the prompt, the model "learns" from them.
Reality: Passing examples in the prompt is few-shot learning or in-context learning: the model uses those examples to do the task better in that specific call, but it doesn't adjust its parameters. On the next call, if you don't pass those examples again, it doesn't "remember" them. That isn't training; it's inference with context. Training requires adjusting parameters (e.g. fine-tuning).

2. Thinking inference is "free"

Mistake: Assuming that once the model is trained, using it costs nothing.
Reality: Inference has costs: tokens processed (OpenAI, Anthropic charge per token), compute (if you use local models, you need hardware), latency (large models take longer). Optimizing inference costs is part of your job.

3. Not distinguishing a "base model" from a "fine-tuned model"

Mistake: Confusing a base model (e.g. GPT-4) with a model adjusted (fine-tuned) with your own data.
Reality: Base model = trained by OpenAI with general data; you use it as-is via API (inference). Fine-tuned model = base model + additional training with your data; you have a specialized model. Fine-tuning is training (expensive); using the fine-tuned model is inference.

4. Believing you can "improve" the base model without retraining

Mistake: Thinking that with better prompts you can make GPT-4 "learn" new facts or capabilities permanently.
Reality: Prompts improve responses in that specific call (inference), but they don't adjust the model's parameters. For the model to "learn" permanently, you need fine-tuning (additional training) or to use RAG (connecting the model to your own documents, which is still inference with additional context, not training).

Analogies for understanding the distinction

Analogy 1: Building vs Driving a car

  • Training: Building a car in a factory (expensive, slow, requiring heavy machinery and specialized teams).
  • Inference: Driving the car (fast, cheap, what the end user does).

As an AI Engineer you don't build the car (you don't train GPT-4); you drive it (you use GPT-4 via API).

Analogy 2: Studying vs Taking an exam

  • Training: Studying for an exam (reading books, doing exercises, adjusting knowledge). It requires time and effort.
  • Inference: Taking the exam (applying what you learned). It's fast (compared with studying).

The model "studies" during training (it learns from data); the model "takes the exam" during inference (it answers new questions).

Analogy 3: Teaching vs Consulting an expert

  • Training: Teaching someone (giving them many examples, correcting them, adjusting their knowledge). It requires time and effort.
  • Inference: Consulting someone who's already an expert (you ask a question, they give you a quick answer).

You don't "teach" GPT-4 (that's training, OpenAI did it); you "consult" GPT-4 (you write a prompt, you get a response). That's inference.

Examples of Training vs Inference in products

1. Spam Filter (Gmail)

  • Training (Google): Google trained the model with millions of labeled emails, using GPUs, over days/weeks. Cost: millions of $ (infrastructure, personnel, data).
  • Inference (each user): Every time you receive an email, the model classifies it in milliseconds. It happens billions of times a day. Cost per email: imperceptible (for the user it's "free"; Google absorbs it).

2. GPT-4 (OpenAI)

  • Training (OpenAI): OpenAI trained GPT-4 with hundreds of billions of tokens, thousands of GPUs, over months. Estimated cost: tens of millions of $. Result: a model that generates coherent text.
  • Inference (you, the user, the AI Engineer): Every time you call the OpenAI API with a prompt, the model generates a response in ~2-3 seconds. Cost: $0.03 per 1K input tokens, $0.06 per 1K output tokens (GPT-4 pricing as of Feb 2026; may vary).

3. Recommendations (Netflix)

  • Training (Netflix): Netflix trains models with data from millions of users (what you watched, what you liked, how long you watched, etc.) using GPU clusters, at intervals (e.g. every week or month). Cost: internal ML infrastructure.
  • Inference (each user): Every time you open Netflix, the model recommends series/movies in milliseconds. It happens millions of times a day. Cost per user: low.

Fine-tuning: additional training on top of a pre-trained model

What is fine-tuning?

Taking a pre-trained model (e.g. GPT-3.5, Llama) and adjusting it with your own data (e.g. your company's conversations, technical documents from your domain). That's additional training, but much cheaper and faster than training from scratch because the model already learned general patterns; you're only specializing it.

Flow:

Pre-trained model (e.g. GPT-3.5) + Your own data → Additional training (fine-tuning) → Specialized model
New data → Specialized model → Prediction / Response (inference)

Concrete example (fine-tuning GPT-3.5 for technical support):

  1. Pre-trained model: GPT-3.5 (already trained by OpenAI with general data).
  2. Your own data: 10,000 pairs of (customer question, correct support answer) from your company.
  3. Fine-tuning: OpenAI (or you, if you use an open-source model) trains the model with your data. Cost: hundreds of $ (far less than training from scratch).
  4. Result: A specialized model that answers support questions in your company's style and with its knowledge.
  5. Use (inference): When a customer asks a question, the specialized model answers. That's inference.

Key point: Fine-tuning is training (you adjust parameters); using the fine-tuned model is inference. As an AI Engineer, you can do fine-tuning when the case justifies it; but most of the time you use base models (GPT-4, Claude) without fine-tuning.

Why it matters for AI Engineering: concrete cases

1. Choosing between fine-tuning and RAG

Question: Should I do fine-tuning or use RAG to connect the model with my own knowledge?

  • Fine-tuning: Additional training (expensive, requires labeled data, permanently improves the model in your domain). Useful when you need the model to "speak" in your style or use specific jargon.
  • RAG: Inference with additional context (cheaper, no retraining required, connects the model to documents in real time). Useful when you need the model to access up-to-date or specific information without retraining.

Trade-off: RAG is more common in AI Engineering because it's cheaper and more flexible; fine-tuning is useful when style or jargon are critical and you have enough data.

2. Estimating a project's costs

Training: If you were to train a model from scratch, the cost would be millions of $. That's why you don't do it; you use pre-trained models.
Inference: Your project's cost is: number of calls per day × cost per call (tokens). E.g. 10,000 calls/day × $0.10/call = $1,000/day. That's what you need to estimate and optimize.

3. Optimizing latency

Training: You don't worry about training latency (OpenAI did that over weeks; it's already done).
Inference: You worry about the latency of each call (2 seconds or 10 seconds?). You choose models based on the latency/quality trade-off: small models (fast) for simple tasks, large models (slow but better) for complex tasks.

4. Reading technical documentation

When you read documentation from OpenAI, Anthropic, etc., you'll see terms like:

  • "Pre-trained model" → Training already done; you use inference.
  • "Fine-tuning" → Additional training with your data.
  • "Inference latency" → The time the model takes to respond (inference).
  • "Input/output tokens" → What counts toward inference costs.

Without the training vs inference distinction, those terms would be confusing.

Exercises

Exercise 1: Classify as training or inference

Classify each activity as training or inference. Justify in one sentence.

  1. OpenAI trains GPT-4 with terabytes of text over months.
  2. You call the OpenAI API with a prompt and get a response.
  3. Netflix retrains its recommendation model with user data from the last week.
  4. A user opens Netflix and receives personalized recommendations.
  5. You fine-tune GPT-3.5 with 5,000 conversations from your company.
  6. You use the fine-tuned model to answer customer questions.
See solution
  1. Training. OpenAI adjusts the model's parameters with data; that's training.
  2. Inference. You use the already-trained model to get a response; you don't adjust parameters.
  3. Training (retraining). Netflix adjusts the model's parameters with new data.
  4. Inference. The already-trained model recommends; it isn't adjusted by that user's clicks at that moment (although those clicks can be used to retrain later).
  5. Training (fine-tuning is additional training). You adjust the model's parameters with your data.
  6. Inference. You use the fine-tuned model to answer; you don't adjust parameters at that moment.

Exercise 2: Costs

Why is inference much cheaper than training? Write 2-3 reasons.

See solution

Reasons:

  1. No parameter adjustment: Inference only applies the already-trained model; there's no backpropagation or optimization (far less compute).
  2. Less hardware: Inference requires 1 GPU (or a CPU) per call; training requires thousands of GPUs over days/weeks.
  3. Fast: Inference takes seconds/milliseconds; training takes days/weeks. The total cost of training is amortized across millions of inferences.

Exercise 3: Fine-tuning vs RAG

Explain in 2-3 sentences when you'd use fine-tuning and when you'd use RAG.

See answer guide

Possible guide:

  • Fine-tuning: When you need the model to "speak" in your style or use jargon specific to your domain, and you have enough labeled data (thousands of examples). It's additional training (expensive) but it permanently improves the model. Useful when style or jargon are critical.

  • RAG: When you need the model to access up-to-date or specific information (e.g. your company's documents, a knowledge base) without retraining. It's inference with additional context (cheaper and more flexible). Useful when the information changes frequently or when you don't have enough data for fine-tuning.

Trade-off: RAG is more common in AI Engineering because it's cheaper, requires no retraining, and you can update documents without touching the model. Fine-tuning is useful when style or deep specialization are critical.

Exercise 4: In-context learning

When you pass examples in the prompt to GPT-4 (e.g. "Classify these texts: Example 1: 'I loved it' → positive; Example 2: 'Horrible' → negative; Now classify: 'It was okay'"), is that training or inference? Justify in 2-3 sentences.

See solution

Inference (with context). You're not adjusting the model's parameters; you're only passing it examples in the prompt so the model uses them as context in that specific call. On the next call, if you don't pass those examples again, it doesn't "remember" them. That's called in-context learning or few-shot learning: the model uses the prompt's examples to do the task better, but it doesn't adjust its parameters. It's still inference, not training.

Exercise 5: Why this distinction defines your role

Why is the training vs inference distinction the most important one for AI Engineering? Write 2-3 reasons.

See answer guide

Possible guide:

  1. It defines your job: As an AI Engineer, your role is to integrate already-trained models (inference), not to train them from scratch (training). That determines which skills you need (APIs, prompts, RAG, evaluation) vs which you do NOT need (deep math, optimization algorithms, large-scale training infrastructure).
  2. Cost optimization: Your concern is optimizing inference costs (choosing the right model, reducing tokens, caching responses), not training costs (those were already absorbed by the provider).
  3. Reading documentation: When you read docs from OpenAI, Anthropic, etc., you'll understand terms like "pre-trained model" (training already done), "fine-tuning" (additional training), "inference latency" (response time), "input/output tokens" (what counts toward inference costs).

Summary

In one sentence: Training is when the model learns from data (expensive, long, done by providers); inference is when the already-trained model is used to predict (fast, cheap, what you do as an AI Engineer).

Key points:

  • Training: Adjusts the model's parameters with data. It requires a lot of data, a lot of compute (thousands of GPUs), time (days/weeks/months), technical expertise. It's done by ML/Research teams or providers (OpenAI, Google, Meta). Cost: millions of $ for large models.
  • Inference: Uses the trained model to predict on new data. It requires 1 GPU (or a CPU), seconds/milliseconds, no parameter adjustment. It's done by you (the AI Engineer), the end user, or your application. Cost: cents per call.
  • Fine-tuning: Additional training on top of a pre-trained model with your own data. Cheaper than training from scratch but more expensive than inference. Useful when style or deep specialization are critical.
  • RAG: Inference with additional context (your own documents). It isn't training; it's connecting the model to external information. More common than fine-tuning in AI Engineering.
  • In-context learning: Passing examples in the prompt to improve the response in that specific call. It's inference, not training (it doesn't adjust parameters permanently).
  • Why it matters: It defines your role (inference, not training), cost optimization (tokens, latency), and how you read technical documentation.

Quick reference table:

ActivityTraining or InferenceWho does itCost
Training GPT-4 from scratchTrainingOpenAIMillions of $
Calling the GPT-4 API with a promptInferenceAI Engineer / UserCents
Fine-tuning GPT-3.5 with your dataTraining (additional)AI Engineer / ProviderHundreds of $
Using the fine-tuned model via APIInferenceAI Engineer / UserCents
Passing examples in the prompt (few-shot)Inference (with context)AI Engineer / UserCents
RAG (connecting an LLM to your own docs)Inference (with context)AI EngineerCents + storage

Connection with the rest of the guide

In later modules:

  • Module 3 (Neural Networks): You'll see how a neural network's training works (backpropagation, weight adjustment). That's the training phase; when you use the trained network, it's inference.
  • Modules 4-5 (Transformers, LLMs): You'll see that LLMs (GPT, Claude) are trained with massive data (training) and then exposed via APIs for you to use (inference). The distinction you learned here is central.
  • Module 6 (APIs): Everything you do with APIs is inference: calling already-trained models. Providers charge you for the tokens processed in inference, not for the training (they already did that).
  • Module 7 (AI Engineering): Your role is inference: integrating already-trained models into applications. You don't train models from scratch (that's ML Engineering).

Additional resources

  1. Google ML Crash Course: Training and Loss — Explains what training is and how parameters are adjusted. In English.

  2. OpenAI: Fine-tuning Guide — Official documentation on fine-tuning (additional training). In English. Useful for understanding when and how to do fine-tuning.

  3. Stanford CS229: Training vs Testing — A course covering training, testing and generalization. In English. More technical.

  4. Hugging Face: Model Inference — Documentation on how to do inference with pre-trained models. In English.

  5. Fast.ai: Training Deep Learning Models — A practical course covering training and inference with code. In English.

  6. LangChain: Model I/O — Documentation on how to call models (inference) with LangChain. In English. Useful for connecting with practical work.

← Previous5. Reinforcement Learning: Learning by Trial and ErrorNext →7. Exercise: Classifying Machine Learning Problems