Module 3: Neural Networks & Deep Learning

4. Activation Functions: Why Networks Need Non-Linearity

Description

In the previous lessons you saw what a neuron is (inputs → weights → sum → activation → output) and how neurons are organized into layers. But there's a critical component we haven't explored in detail yet: the activation function.

Without activation functions, a neural network (no matter how many layers it has) can only learn linear relationships. That means it would be as limited as a simple linear model (linear regression). It could NOT recognize faces, translate languages, or do anything interesting.

Activation functions introduce non-linearity, allowing the network to learn complex relationships.

In this lesson you'll understand:

Why we need activation functions (what happens without them).
What functions exist (ReLU, Sigmoid, Tanh, Softmax).
When to use each one (depending on the problem and the layer).
What "non-linearity" means (without complex mathematical formulas).

Approach: Conceptual and visual. You'll see plots of the functions, analogies, and concrete examples.

The problem: networks without activation are linear

What happens if we don't use an activation function

Imagine a 3-layer neural network:

Input (x) → Hidden1 → Hidden2 → Output (y)

Without an activation function:

Each neuron does: output = (weighted sum of inputs)

Mathematically:

Hidden1 = W₁ × x
Hidden2 = W₂ × Hidden1 = W₂ × (W₁ × x) = (W₂ × W₁) × x
Output = W₃ × Hidden2 = W₃ × (W₂ × W₁ × x) = (W₃ × W₂ × W₁) × x

Simplifying:

Output = W_combined × x

Where W_combined = W₃ × W₂ × W₁ (a single combined weight).

Conclusion: Without activation, the whole network collapses into a simple linear transformation. It doesn't matter whether you have 1 layer or 100 layers; the result is equivalent to a linear regression.

Problem: You can only learn linear relationships (e.g. "y = 2x + 3"). You CANNOT learn complex relationships (e.g. recognizing faces, classifying images, translating languages).

Concrete example: XOR without activation

Remember the XOR problem (seen in Lesson 02):

x₁	x₂	y (XOR)
0	0	0
0	1	1
1	0	1
1	1	0

If you plot the points in 2D, you CANNOT draw a straight line that separates the cases where y = 1 from the cases where y = 0.

A network without activation: It can't solve XOR (it only learns linear relationships).

A network with activation (e.g. ReLU): It can solve XOR (it introduces non-linearity).

What non-linearity is

Simple definition

Linear: A relationship where the output is proportional to the input. E.g. y = 2x + 3 (if you double x, y roughly doubles too).

Non-linear: A relationship where the output is NOT proportional to the input. E.g. y = x² (if you double x, y quadruples).

Why it matters: The real world is non-linear. Recognizing a face isn't "if pixel 1 > 100 and pixel 2 > 50, then it's a face". It's a complex combination of patterns (edges, textures, shapes) that CANNOT be expressed linearly.

Analogy: complex vs simple decisions

Without activation (linear): Your decision to buy a house depends only on the price. "If price < $200K, buy; if not, don't buy." It's a simple linear rule.

With activation (non-linear): Your decision depends on price, location, size, neighborhood, schools, etc., combined in a complex way. "I'll buy if the price is reasonable AND the location is good OR the size compensates for other factors." That's non-linear.

Moral: Activation functions let the network make complex decisions (not just simple weighted sums).

The main activation functions

1. ReLU (Rectified Linear Unit)

Formula:

f(z) = max(0, z)

If z > 0, then f(z) = z. If z ≤ 0, then f(z) = 0.

Plot:

   f(z)
    |
    |      /
    |     /
    |    /
    |   /
    |__________
       0      z

Idea: It passes positive values through as-is; it zeroes out negative values.

Advantages:

✅ Very simple to compute (comparing with 0).
✅ It avoids vanishing gradients (a common problem with sigmoid/tanh).
✅ Faster than sigmoid or tanh.

Disadvantages:

❌ "Dying ReLU": If a neuron always receives negative inputs, its output is always 0 → the neuron "dies" (it stops learning).

Common use:

Hidden layers of deep networks (CNNs, feedforward).
It's the most used activation function in modern Deep Learning.

Example:

z = -2 → f(z) = 0
z = 0  → f(z) = 0
z = 3  → f(z) = 3

2. Sigmoid (Logistic)

Formula:

f(z) = 1 / (1 + e^(-z))

Plot:

   f(z)
  1 |     ___________
    |    /
  0.5|   /
    |  /
  0 |_____________
         0        z

Idea: It squashes any number into a range between 0 and 1.

Characteristics:

Output between 0 and 1 (it can be interpreted as a probability).
"S-shaped".

Advantages:

✅ Output between 0 and 1 → useful for binary classification (the probability of the positive class).

Disadvantages:

❌ Vanishing gradients: For very large or very small values of z, the derivative is nearly 0 → learning becomes very slow (backpropagation doesn't work well).
❌ Outputs not centered on 0: All the outputs are positive → it can make training less efficient.

Common use:

The output layer of binary classification (e.g. spam vs not-spam).
Less common in hidden layers (ReLU is preferred).

Example:

z = -10 → f(z) ≈ 0
z = 0   → f(z) = 0.5
z = 10  → f(z) ≈ 1

3. Tanh (Hyperbolic Tangent)

Formula:

f(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Plot:

   f(z)
  1 |     ___________
    |    /
  0 |___/___________
    |  /
 -1 |_____________
         0        z

Idea: It squashes any number into a range between -1 and 1.

Characteristics:

Output between -1 and 1 (centered on 0).
Similar to sigmoid but centered.

Advantages:

✅ Outputs centered on 0 → more efficient training than sigmoid.

Disadvantages:

❌ Vanishing gradients (the same problem as sigmoid).

Common use:

Hidden layers (before ReLU became popular).
RNNs (Recurrent Neural Networks) sometimes use tanh.

Example:

z = -10 → f(z) ≈ -1
z = 0   → f(z) = 0
z = 10  → f(z) ≈ 1

4. Softmax

Formula (for a vector of outputs z₁, z₂, ..., zₙ):

f(zᵢ) = e^(zᵢ) / (e^(z₁) + e^(z₂) + ... + e^(zₙ))

Idea: It converts a vector of numbers into a vector of probabilities (each number between 0 and 1, and they sum to 1).

Example:

Inputs: z = [2.0, 1.0, 0.5]
Softmax: f(z) = [0.66, 0.24, 0.10]  (they sum to 1)

Interpretation: The probability of each class (66% class 1, 24% class 2, 10% class 3).

Common use:

The output layer of multiclass classification (e.g. classifying images into 10 categories).

Advantages:

✅ Outputs interpretable as probabilities (they sum to 1).
✅ It penalizes incorrect predictions made with high confidence (e.g. if you predict 90% class A but the correct one is B, the error is large).

NOT commonly used for:

It is NOT used in hidden layers (only in the output layer of multiclass classification).

Comparison of activation functions

Function	Range	Common use	Advantages	Disadvantages
ReLU	[0, ∞)	Hidden layers	Simple, fast, avoids vanishing gradients	Dying ReLU
Sigmoid	(0, 1)	Binary output	Output as a probability	Vanishing gradients, not centered
Tanh	(-1, 1)	Hidden layers (formerly), RNNs	Centered on 0	Vanishing gradients
Softmax	(0, 1) and they sum to 1	Multiclass output	Interpretable probabilities	Only for multiclass output

How to choose an activation function

For hidden layers

Rule of thumb: Use ReLU (or variants like Leaky ReLU, ELU).

Why: Simple, fast, avoids vanishing gradients (a common problem in deep networks).

Exceptions:

RNNs: They sometimes use tanh (for historical and stability reasons).
Specific problems: If you run into dying ReLU, try Leaky ReLU or ELU (variants that don't completely zero out negative values).

For the output layer

It depends on the type of problem:

Type of problem	Activation function	Why
Regression (predicting a number)	None or linear	You want an output without restrictions (e.g. a house price can be any positive number)
Binary classification	Sigmoid	Output between 0 and 1 (the probability of the positive class)
Multiclass classification	Softmax	The outputs sum to 1 (the probabilities of each class)

Why this matters for an AI Engineer

1. Model selection

When you read model documentation, you'll see mentions like:

"ResNet-50 uses ReLU in all its hidden layers."
"BERT uses GELU (a ReLU variant) in its hidden layers and Softmax at the output."

If you understand activation functions:

You know that ReLU is standard for hidden layers (simple, effective).
You know that Softmax is for multiclass classification.

2. Debugging

If your model isn't learning (accuracy stuck at 50%):

Possible cause: the wrong activation function

You used no activation in the hidden layers → the network is linear (it can't learn complex relationships).
Solution: Add ReLU.

Possible cause: vanishing gradients

You used sigmoid/tanh in a deep network (50 layers) → gradients become very small → slow learning.
Solution: Switch to ReLU.

3. Fine-tuning

If you're fine-tuning a pre-trained model:

Don't change the activation functions (the layers are optimized for those functions).
Only adjust weights (parameters).

Common mistakes

1. Using sigmoid/tanh in the hidden layers of deep networks

Mistake: Using sigmoid or tanh in the hidden layers of a network with 50 layers.

Problem: Vanishing gradients → very slow or stalled learning.

Solution: Use ReLU (or its variants).

2. Not using an activation function in hidden layers

Mistake: Omitting activation functions, thinking it "simplifies the network".

Problem: The network becomes linear → it can't learn complex relationships.

Solution: Always use an activation function in hidden layers (ReLU is standard).

3. Using Softmax in hidden layers

Mistake: Using Softmax in hidden layers (thinking it "normalizes" the outputs).

Problem: Softmax is for multiclass output (it converts to probabilities). In hidden layers, we want simple non-linear transformations (ReLU).

Solution: Softmax only in the output layer of multiclass classification.

Frequently asked questions

What is Leaky ReLU?

Answer: A variant of ReLU that does NOT completely zero out negative values.

Formula:

f(z) = z if z > 0; otherwise, f(z) = 0.01 × z

Advantage: It avoids dying ReLU (neurons that "die" because they always receive negative inputs).

Use: An alternative to ReLU when you run into dying ReLU.

What is GELU?

Answer: Gaussian Error Linear Unit. A variant of ReLU used in Transformers (BERT, GPT).

Idea: Similar to ReLU but with a smoother transition.

Use: BERT, GPT, modern NLP models.

Why is ReLU so popular?

Answer:

Simple: It only requires comparing with 0 (very fast).
It avoids vanishing gradients: The derivative is 1 for z > 0 (it doesn't "vanish" like sigmoid/tanh).
It works well in practice: Networks with ReLU train faster and achieve better accuracy than with sigmoid/tanh.

Practical exercise

Task: choose an activation function

For each problem, choose an activation function for the output layer:

Predicting the price of a house (regression):
- Answer: None (linear). The price can be any positive number; we don't want restrictions.
Classifying emails as spam (1) or not-spam (0):
- Answer: Sigmoid. Output between 0 and 1 (the probability of spam).
Classifying images into 10 categories (dog, cat, car, etc.):
- Answer: Softmax. 10 outputs (one per category), each a probability between 0 and 1, summing to 1.

Summary

Why we need activation functions:

Without them, the network is linear (it can only learn simple relationships).
With them, the network is non-linear (it can learn complex relationships: recognizing faces, translating languages, etc.).

The main functions:

ReLU: Hidden layers (the standard in modern Deep Learning).
Sigmoid: The output of binary classification.
Tanh: Hidden layers (before ReLU), RNNs.
Softmax: The output of multiclass classification.

How to choose:

Hidden layers → ReLU (or Leaky ReLU, GELU).
Output → It depends on the problem (none for regression, sigmoid for binary, softmax for multiclass).

Why it matters:

Understanding activation functions lets you select correct architectures, diagnose problems (e.g. vanishing gradients), and reason about trade-offs.

Next step: Lesson 05: Forward Pass — How data flows through the network (from input to output).

Additional resources

CS231n: Neural Networks Part 1 - Activation Functions — Technical notes on activation functions. In English.
Deep Learning Book: Chapter 6.3 (Hidden Units) — A chapter on activation functions. In English. Technical.
Visualizing Activation Functions — Interactive plots of ReLU, sigmoid, tanh. In English.

← Previous3. Layers and Architecture of Neural Networks Next →5. Forward Pass: How Data Flows in a Neural Network