Module 3: Neural Networks & Deep Learning

6. Backpropagation: How a Neural Network Learns

Next →

Description

In the previous lesson you saw the forward pass: how data flows from input to output to generate a prediction. But that only explains how the network uses its current weights. It does NOT explain how it learns (how it adjusts those weights to improve).

That process is called backpropagation (or backward propagation): after the forward pass, the network computes how much error it made, and then propagates that error backward (from output to input) in order to adjust the weights and reduce the error in future predictions.

In this lesson you'll understand:

  • What backpropagation is (conceptually, without complex derivatives).
  • How it works (compute the error, propagate it backward, adjust the weights).
  • Why it's critical for Deep Learning.
  • Why it matters for AI Engineering (even though you will NOT implement it manually).

Approach: Conceptual, with analogies. You will NOT see derivative computations (frameworks like PyTorch do that automatically). You'll see the central idea and why it works.

The problem: the network makes mistakes

Example: the spam detector

Let's go back to the previous lesson's example (classifying emails as spam or not-spam).

Input: x = [5, 3] (5 exclamation marks, "free" 3 times)
The network's prediction: y_pred = 0.9993 (99.93% spam)
Real label: y_real = 1 (spam)

Error: Very small (the network predicted 0.9993, the real label is 1 → error ≈ 0.0007).

But let's imagine another case:

Input: x = [1, 0] (1 exclamation mark, "free" 0 times)
Prediction: y_pred = 0.75 (75% spam)
Real label: y_real = 0 (NOT spam)

Error: Large (the network predicted 0.75, but the real label is 0 → error = 0.75). The network is very wrong.

Question: How do we adjust the weights so the network predicts better in the future?

What backpropagation is

Definition: An algorithm that adjusts the network's weights by propagating the error backward (from output to input).

Flow:

1. Forward pass: Input → Output (generate a prediction)
2. Compute the error: Compare the prediction with the real label
3. Backward pass: Propagate the error backward (from output to input)
4. Adjust the weights: Reduce the error by modifying the weights according to their "responsibility" for the error

Analogy: A factory with departments.

  • Forward pass: The factory produces a product (a prediction).
  • Error: The customer (the supervisor) says "the product has defects".
  • Backward pass: The supervisor gives feedback to each department (layer) about what caused the defects.
  • Adjustment: Each department adjusts its process (weights) to reduce defects in future production.

How backpropagation works (conceptual)

Step 1: Forward pass (already seen in Lesson 05)

The network processes the input and generates a prediction.

Input → Hidden Layers → Output (prediction)

Step 2: Compute the error (Loss Function)

Loss Function: It measures how incorrect the prediction is.

Common examples:

  • Mean Squared Error (MSE): For regression. Loss = (y_real - y_pred)²
  • Binary Cross-Entropy: For binary classification. It penalizes incorrect predictions made with high confidence.
  • Categorical Cross-Entropy: For multiclass classification.

Example (binary classification with Cross-Entropy):

y_real = 0 (not-spam)
y_pred = 0.75 (75% spam)

Loss ≈ 1.39  (high → a very incorrect prediction)

The goal of training: Minimize the Loss (make the network commit fewer errors).

Step 3: Backward pass (propagate the error backward)

Central idea: Compute how much each weight contributed to the total error.

Questions backpropagation answers:

  • How much does the error change if I change the weight w₁ in the output layer?
  • How much does the error change if I change the weight w₅ in the first hidden layer?

Technical answer: Gradients are computed (partial derivatives of the Loss with respect to each weight). That tells you:

  • The gradient's sign: Whether increasing the weight increases or reduces the error.
  • The gradient's magnitude: How much the weight impacts the error.

Analogy: The factory supervisor asks each department: "How much did your work contribute to the product's defect?" Departments that contributed more receive stronger feedback.

Step 4: Adjust the weights (Gradient Descent)

After computing the gradients, we adjust the weights in the direction that reduces the error.

Update rule:

new_weight = current_weight - (learning_rate × gradient)

Learning rate: A hyperparameter that controls how much we adjust the weights at each step.

  • A high learning rate (e.g. 0.1): Large adjustments → fast learning BUT it can "overshoot" the optimum.
  • A low learning rate (e.g. 0.001): Small adjustments → slow learning BUT more precise.

Example:

Current weight: w = 0.5
Gradient: ∂Loss/∂w = 2.0  (increasing w increases the error)
Learning rate: α = 0.01

New weight:
w = 0.5 - (0.01 × 2.0)
w = 0.5 - 0.02
w = 0.48

Interpretation: We reduce the weight (because the gradient is positive → increasing w increases the error). Now the network will predict differently on the next iteration.

Step 5: Repeat with more examples (Epochs)

Backpropagation is repeated with many training examples (thousands, millions).

Epoch: One complete pass through all the training examples.

Typical training:

  1. Epoch 1: Process all the examples, adjust the weights.
  2. Epoch 2: Process all the examples again (now with adjusted weights), adjust further.
  3. ...
  4. Epoch N: After many iterations, the weights converge to values that minimize the error.

Result: The network learns to predict correctly.

Why backpropagation was revolutionary

History

Before backpropagation (1960s-1980s):

  • Neural networks existed, but there was no efficient way to train them.
  • Only 1-layer networks (simple perceptrons) could be trained.

1986: Backpropagation becomes popular (Rumelhart, Hinton, Williams):

  • The paper shows how to compute gradients efficiently in networks with many layers.
  • It makes it possible to train deep networks.

1990s-2000s: Backpropagation is used widely, but networks remain shallow (2-3 layers) due to vanishing gradient problems.

2012-today: The combination of backpropagation + ReLU + GPUs + large datasets → the Deep Learning boom.

Moral: Backpropagation is the algorithm that makes Deep Learning possible. Without it, we wouldn't have ChatGPT, facial recognition, machine translation, etc.

Common problems with backpropagation

1. Vanishing Gradients

Problem: In deep networks (50+ layers), the gradients become very small in the early layers → those layers learn very slowly or don't learn at all.

Cause: Activation functions like sigmoid/tanh compress gradients (derivative < 1). When gradients are multiplied through many layers, they become nearly 0.

Solution: Use ReLU (derivative = 1 for z > 0 → it doesn't compress gradients).

2. Exploding Gradients

Problem: The opposite of vanishing: the gradients become very large → the weights are updated wildly → training is unstable.

Solution: Gradient clipping (limiting the magnitude of gradients), batch normalization.

3. Local Minima

Problem: The network can get stuck in a local minimum (a point where the error is locally low, but it isn't the global minimum).

Solution (partial): Use advanced optimizers (Adam, RMSprop) that help escape local minima.

Reality: In deep networks with many parameters, local minima are usually "good enough" (you don't need the global optimum).

Why this matters for an AI Engineer

1. You do NOT need to implement backpropagation manually

Reality: Frameworks (PyTorch, TensorFlow) implement backpropagation automatically.

Your code:

# PyTorch
output = model(input)  # Forward pass
loss = loss_function(output, target)
loss.backward()  # Backpropagation (automatic)
optimizer.step()  # Update the weights

What .backward() does: It computes all the gradients automatically using backpropagation.

Moral: You do NOT need to implement derivatives manually. But you DO need to understand conceptually what backpropagation does (adjust weights to reduce error).

2. Understanding training problems

If your model isn't learning (accuracy stuck):

Possible cause: vanishing gradients

  • You used sigmoid/tanh in a deep network (50 layers).
  • Solution: Switch to ReLU.

Possible cause: an incorrect learning rate

  • A learning rate that's too high → unstable training (the loss goes up and down wildly).
  • A learning rate that's too low → very slow training (the loss goes down very slowly).
  • Solution: Adjust the learning rate (e.g. start with 0.001, experiment).

3. Understanding the trade-off between training and inference

Training (includes backpropagation):

  • Forward pass + backward pass + updating the weights.
  • Slow (hours, days).
  • It requires labeled data.
  • You do NOT do it as an AI Engineer (you use pre-trained models).

Inference (only the forward pass):

  • Only the forward pass (generating a prediction).
  • Fast (milliseconds).
  • YOU DO IT as an AI Engineer (integrating models into apps).

Exception: Fine-tuning (adjusting a pre-trained model with your data) requires backpropagation BUT with less data and fewer epochs than training from scratch.

Common mistakes

1. Thinking that backpropagation = AI

Mistake: Believing that backpropagation is the only thing that makes a neural network "intelligent".

Reality: Backpropagation is just the optimization algorithm (adjusting weights). The "intelligence" comes from the architecture (layers, activations) + data + training.

2. Implementing backpropagation manually in production

Mistake: Trying to compute gradients by hand in PyTorch/TensorFlow.

Reality: Frameworks do it automatically. You only need to call .backward().

Frequently asked questions

What is an optimizer?

Answer: An algorithm that uses gradients (from backpropagation) to update the weights.

Examples:

  • SGD (Stochastic Gradient Descent): Basic. w = w - α × gradient.
  • Adam: Advanced. It adjusts the learning rate automatically per parameter. Very widely used in modern Deep Learning.
  • RMSprop, AdaGrad, etc.: Variants with different strategies.

Use: When you train a model, you choose an optimizer (e.g. Adam with a learning rate of 0.001).

What is batch size?

Answer: The number of examples you process before updating the weights.

Mini-batch training:

  • Instead of processing 1 example → computing the gradient → updating (very slow).
  • You process N examples (e.g. 32) → compute the average of the gradients → update (more efficient).

Typical batch size: 32, 64, 128, 256.

How many epochs do I need?

Answer: It depends on the problem, the data, the architecture.

Rule of thumb:

  • Small datasets: 50-200 epochs.
  • Large datasets (e.g. ImageNet): 10-50 epochs.
  • Fine-tuning: 3-10 epochs (because you start from a pre-trained model).

How to know: Monitor the loss on a validation set. When it starts going up (overfitting), stop.

Practical exercise

Task: identify training problems

Case 1:

  • A 50-layer network with sigmoid in the hidden layers.
  • The loss drops very slowly (after 100 epochs, it's still high).

Question: What's the problem?

Answer: Vanishing gradients (sigmoid compresses gradients in deep networks). Solution: Switch to ReLU.

Case 2:

  • Learning rate = 1.0
  • The loss goes up and down wildly (it doesn't converge).

Question: What's the problem?

Answer: A learning rate that's too high (adjustments that are too large). Solution: Reduce the learning rate (e.g. to 0.001).

Summary

Backpropagation: An algorithm that adjusts weights by propagating the error backward (from output to input).

Steps:

  1. Forward pass (generate a prediction).
  2. Compute the error (loss function).
  3. Backward pass (compute gradients).
  4. Update the weights (gradient descent).
  5. Repeat with more examples.

Why it matters:

  • It makes it possible to train deep networks (Deep Learning).
  • Frameworks implement it automatically (you do NOT need to do it manually).
  • Understanding it lets you diagnose training problems (vanishing gradients, learning rate, etc.).

Next step: Lesson 07: Specialized Architectures (CNNs, RNNs, Feedforward) — What architectures exist and when to use each one.

Additional resources

  1. 3Blue1Brown: Backpropagation calculus — A visual video about backpropagation. In English with subtitles.

  2. CS231n: Backpropagation, Intuitions — Notes on backpropagation. In English. Technical (with derivatives).

  3. Deep Learning Book: Chapter 6.5 (Back-Propagation) — A technical chapter. In English. Very technical (calculus).

← Previous5. Forward Pass: How Data Flows in a Neural NetworkNext →7. Specialized Architectures: CNNs, RNNs and Feedforward