Module 3: Neural Networks & Deep Learning

5. Forward Pass: How Data Flows in a Neural Network

Next →

Description

In the previous lessons you saw what a neuron is, how neurons are organized into layers, and why they need activation functions. But you haven't yet seen how a neural network processes an input in order to generate a prediction (an output).

That process is called the forward pass (or forward propagation): the data "flows forward" from the input layer, passes through all the hidden layers (transforming at each layer), and finally reaches the output layer (the prediction).

In this lesson you'll understand:

  • What the forward pass is (the flow of data from input to output).
  • What operations happen in each layer (multiplying by weights, summing, activation).
  • A complete step-by-step example (binary classification).
  • Why it matters for AI Engineering (debugging, optimization, latency).

Approach: Conceptual, with a concrete numeric example. You'll see simple calculations (without advanced linear algebra) so you understand the flow.

What the forward pass is

Definition: The process of passing an input through all the network's layers in order to generate a prediction (an output).

Flow:

Input → Hidden Layer 1 → Hidden Layer 2 → ... → Output Layer → Prediction

At each layer:

  1. Multiply the inputs by the weights.
  2. Sum the results (+ bias).
  3. Apply the activation function.
  4. Pass the outputs to the next layer.

Analogy: A factory with departments. The raw material (the input) comes in through receiving, passes through departments (hidden layers) that transform it step by step, and comes out as the final product (the output).

Complete example: binary classification (a spam detector)

The problem

Task: Classify emails as spam (1) or not-spam (0).

Features (simplified):

  • x₁: The number of exclamation marks (e.g. 5).
  • x₂: The number of times "free" appears (e.g. 3).

The network's architecture:

Input Layer (2 neurons)
  ↓
Hidden Layer (3 neurons, ReLU)
  ↓
Output Layer (1 neuron, Sigmoid)

Step 1: Initialize the weights (already trained)

Let's assume the network is already trained and has these weights:

Weights between Input and Hidden:

W₁ = [[0.5, 0.3],    # Connections from x₁ to h₁, h₂
      [0.8, 0.6],    # Connections from x₂ to h₁, h₂
      [-0.2, 0.4]]   # Connections from x₃ (implicit bias)

(Simplified: each row represents the connections from one input to all the hidden neurons)

Hidden Layer biases:

b_hidden = [0.1, -0.3, 0.2]

Weights between Hidden and Output:

W₂ = [[1.2], [0.8], [-0.5]]

Output bias:

b_output = -1.0

Step 2: Forward pass (a specific input)

Input: x = [5, 3] (5 exclamation marks, "free" 3 times)

Hidden Layer (3 neurons, ReLU)

Neuron h₁:

z₁ = (x₁ × w₁₁) + (x₂ × w₂₁) + b₁
z₁ = (5 × 0.5) + (3 × 0.8) + 0.1
z₁ = 2.5 + 2.4 + 0.1
z₁ = 5.0

Activation (ReLU):
h₁ = max(0, z₁) = max(0, 5.0) = 5.0

Neuron h₂:

z₂ = (5 × 0.3) + (3 × 0.6) + (-0.3)
z₂ = 1.5 + 1.8 - 0.3
z₂ = 3.0

h₂ = max(0, 3.0) = 3.0

Neuron h₃:

z₃ = (5 × -0.2) + (3 × 0.4) + 0.2
z₃ = -1.0 + 1.2 + 0.2
z₃ = 0.4

h₃ = max(0, 0.4) = 0.4

Hidden Layer output: h = [5.0, 3.0, 0.4]

Output Layer (1 neuron, Sigmoid)

Neuron y:

z_out = (h₁ × w₁) + (h₂ × w₂) + (h₃ × w₃) + b_out
z_out = (5.0 × 1.2) + (3.0 × 0.8) + (0.4 × -0.5) + (-1.0)
z_out = 6.0 + 2.4 - 0.2 - 1.0
z_out = 7.2

Activation (Sigmoid):
y = 1 / (1 + e^(-7.2))
y ≈ 0.9993  (≈ 99.93%)

Final prediction: y ≈ 1.0 → Spam (with 99.93% confidence).

Interpretation

What the network learned:

  • The hidden layer captures features: h₁ = 5.0 (high → lots of activity from exclamation marks and "free"), h₂ = 3.0, h₃ = 0.4 (low).
  • The output layer combines those features: h₁ has a high weight (1.2) → a strong indicator of spam.
  • Result: A high probability of spam (99.93%).

If you change the input:

Input: x = [0, 0]  (no exclamation marks, no "free")
Hidden: h = [0.1, 0, 0.2]  (low activations)
Output: y ≈ 0.10  (10% spam → NOT spam)

Visualizing the flow

Input Layer       Hidden Layer (ReLU)      Output Layer (Sigmoid)

  x₁=5 ─────┬────→ h₁=5.0 ────┐
            │                  │
  x₂=3 ─────┼────→ h₂=3.0 ────┼────→ y=0.9993 (Spam)
            │                  │
            └────→ h₃=0.4 ────┘

Each arrow: Multiplication by a weight + sum + activation.

Why this matters for an AI Engineer

1. Understanding latency (prediction time)

The forward pass determines latency: Every time your app makes a prediction, the network does a forward pass.

Factors that affect latency:

  • The number of layers: More layers → more computation → more time.
  • The number of neurons per layer: More neurons → more multiplications → more time.
  • The number of parameters: More weights → more operations → more time.

Example:

  • Model A: 10 layers, 100 neurons per layer, 100K parameters → 50ms per prediction.
  • Model B: 50 layers, 1000 neurons per layer, 50M parameters → 500ms per prediction.

Trade-off: Model B is more accurate but 10× slower.

2. Debugging

If your model is slow, you can diagnose where the bottleneck is:

Option 1: Measure latency per layer

  • Tools: PyTorch Profiler, TensorBoard.
  • You identify which layers take the most time (e.g. the first convolutional layers in CNNs are usually the slowest).

Option 2: Reduce the model's size

  • Use fewer layers or fewer neurons per layer.
  • Use optimization techniques (quantization, pruning).

3. Model interpretation

The forward pass lets you see what each layer learns:

  • The first hidden layers: Basic features (edges, textures in images; letters, syllables in text).
  • The last hidden layers: Complex features specific to the problem (the shape of ears in dog vs cat classification).

Visualization tools:

  • Activation maps: Visualizing which neurons activate for a given input.
  • Feature visualization: Seeing which patterns most activate each neuron.

Common mistakes

1. Confusing the forward pass with training

Mistake: Thinking that the forward pass = training.

Reality:

  • Forward pass: Generating a prediction (inference). Fast (milliseconds).
  • Training: Forward pass + backpropagation (adjusting weights). Slow (hours, days).

2. Not considering the forward pass's impact on latency

Mistake: Choosing a model with 50 layers without considering latency.

Reality: A forward pass in large models (50+ layers, 100M+ parameters) can take seconds → unacceptable for apps that need fast responses.

Summary

Forward pass: The flow of data from input to output (Input → Hidden Layers → Output).

Operations at each layer: Multiply by weights, sum, apply activation, pass to the next layer.

Why it matters:

  • The forward pass determines latency (prediction time).
  • Understanding it lets you optimize models for production.
  • It lets you interpret what each layer learns.

Next step: Lesson 06: Backpropagation — How the network learns (adjusting weights to reduce error).

Additional resources

  1. 3Blue1Brown: Gradient descent, how neural networks learn — A visual video about the forward and backward pass. In English.

  2. CS231n: Backpropagation, Intuitions — Notes on the forward and backward pass. In English. Technical.

← Previous4. Activation Functions: Why Networks Need Non-LinearityNext →6. Backpropagation: How a Neural Network Learns