Module 3: Neural Networks & Deep Learning

2. What is a Neural Network?

Description

In this lesson you'll understand what a neural network is: a type of Machine Learning model inspired by the human brain that learns complex patterns by connecting artificial neurons in layers.

Neural networks are the technology behind practically all modern AI: ChatGPT, facial recognition, machine translation, Netflix recommendations, self-driving cars, etc. Understanding what they are and how they work is fundamental to working as an AI Engineer.

Approach: This lesson is conceptual and visual. You'll see analogies (a neuron = a worker who decides), diagrams (how neurons connect), and explanations without mathematical formulas. The goal is for you to understand what an artificial neuron does and how it decides, not for you to implement one from scratch.

The biological inspiration: real neurons

How a neuron in your brain works

Before understanding the artificial neuron, it's worth looking at the biological inspiration (even though they're very different):

Biological neuron (simplified):

It receives signals from other neurons (through dendrites).
It processes those signals in the cell body (soma).
It decides whether to send an electrical signal (a firing or spike).
It transmits the signal to other neurons (through the axon and synapses).

Simplified diagram:

Dendrites → Soma → Axon → Synapses → Other neurons
(input)   (processes) (output) (connection)

Key characteristics:

A neuron doesn't work alone; it's connected to thousands of other neurons.
The strength of the connections (synapses) determines how much influence one neuron has over another.
The brain learns by adjusting those connections (strengthening or weakening synapses based on experience).

Example: If you touch a hot stove, sensory neurons send pain signals; motor neurons send signals to pull your hand away; your brain strengthens those connections so that in the future you react faster.

Differences between biological and artificial neurons

Aspect	Biological Neuron	Artificial Neuron
Quantity	~86 billion in a human brain	Thousands to millions in a neural network
Connections	Thousands of synapses per neuron	Dozens to thousands of connections per neuron
Speed	Milliseconds (slow but parallel)	Nanoseconds (fast and sequential/parallel)
Learning	Chemical changes in synapses	Adjusting numeric weights (numbers)
Complexity	Extremely complex (chemical, electrical, genetic)	A simplified (mathematical) model

Moral: The artificial neuron is an extreme simplification of the biological neuron. It does NOT work the same way; it only takes inspiration from the idea: "receive inputs, process them, decide an output, adjust connections in order to learn".

The artificial neuron: the perceptron

What a perceptron is

The perceptron is the simplest model of an artificial neuron, invented in 1958 by Frank Rosenblatt. It's the basic unit of modern neural networks.

Central idea: An artificial neuron receives inputs (numbers), processes them (a weighted sum), and decides an output (a number).

Components:

Inputs (x₁, x₂, ..., xₙ): Numbers that represent features of the problem (e.g. the pixels of an image, the words of a text).
Weights (w₁, w₂, ..., wₙ): Numbers that represent the importance of each input. Adjustable during learning.
Bias (b): A number that adjusts the activation threshold. Also adjustable.
Weighted sum (z): z = (x₁ × w₁) + (x₂ × w₂) + ... + (xₙ × wₙ) + b
Activation function (f): Transforms z into a final output. E.g. "if z > 0, output = 1; otherwise, output = 0".
Output (y): The final result (a number).

Formula (optional, for reference only):

z = (x₁ × w₁) + (x₂ × w₂) + ... + (xₙ × wₙ) + b
y = f(z)

Where f is the activation function (e.g. step function, ReLU, sigmoid).

You do NOT need to memorize the formula. What matters is understanding the flow: inputs → weights → sum → activation → output.

Concrete example: classifying whether an email is spam

Problem: Decide whether an email is spam (1) or not-spam (0).

Inputs (simplified):

x₁: The number of exclamation marks (!!!) in the email. E.g. x₁ = 5.
x₂: The number of times the word "free" appears. E.g. x₂ = 3.
x₃: Whether the sender is known (1 = known, 0 = unknown). E.g. x₃ = 0.

Weights (adjusted during training):

w₁ = 0.8 (exclamation marks have a high weight → they indicate spam).
w₂ = 1.5 (the word "free" has a very high weight → a strong indicator of spam).
w₃ = -2.0 (a known sender has a negative weight → it indicates not-spam).

Bias:

b = -1.0 (the activation threshold).

Computation:

z = (x₁ × w₁) + (x₂ × w₂) + (x₃ × w₃) + b
z = (5 × 0.8) + (3 × 1.5) + (0 × -2.0) + (-1.0)
z = 4 + 4.5 + 0 - 1
z = 7.5

Activation (step function: if z > 0, y = 1; otherwise, y = 0):

y = 1 (because z = 7.5 > 0)

Result: The neuron decides that the email IS SPAM (y = 1).

Analogy: the neuron as a worker who decides

Imagine the neuron is a bank employee who decides whether to approve a loan (1) or reject it (0).

Inputs (the applicant's characteristics):

x₁: Monthly income (e.g. $3000).
x₂: Credit history (e.g. 750 points).
x₃: The loan amount requested (e.g. $20,000).

Weights (the importance of each factor, learned from experience):

w₁ = 0.5 (high income → more likely to approve).
w₂ = 1.2 (good credit history → very important).
w₃ = -0.8 (a high amount → less likely to approve).

Bias:

b = -500 (the base threshold).

The employee's process:

Receives information (inputs).
Weighs each factor by its importance (multiplies by the weights).
Sums everything (including the bias).
Decides: If the sum is positive, approve; if not, reject.

Computation:

z = (3000 × 0.5) + (750 × 1.2) + (20000 × -0.8) + (-500)
z = 1500 + 900 - 16000 - 500
z = -14100

Decision: z < 0 → Reject the loan.

Interpretation: Even though the applicant has good income and good history, the amount requested is too high (a strong negative weight) → the employee rejects.

Moral: The neuron "decides" by weighing inputs according to its weights. The weights represent what the neuron "learned" (e.g. "good credit history is important", "a high amount is risky").

How the neuron "learns"

The problem: incorrect weights

Initially, the weights (w₁, w₂, ...) are random numbers. That means the neuron makes incorrect predictions at first.

Example: Going back to the spam detector:

Real email: "Make money free!!!" → It should predict spam (1).
Initial prediction with random weights: y = 0 (not-spam) → INCORRECT.

Question: How do you correct the weights so the neuron predicts correctly?

The learning process (conceptual)

Step 1: Compute the error

Error = Real answer - The model's prediction
Error = 1 - 0 = 1 (the neuron failed)

Step 2: Adjust the weights

Increase the weights of the inputs that indicated spam (e.g. w₁ for exclamation marks, w₂ for "free") and reduce the ones that indicated not-spam.

Intuitive rule:

If the neuron predicted less than the correct answer (a positive error) → Increase the weights of the active inputs.
If the neuron predicted more than the correct answer (a negative error) → Reduce the weights of the active inputs.

Step 3: Repeat with more examples

You train the neuron with thousands of labeled emails (spam / not-spam). On each example, you compute the error and adjust the weights. Little by little, the weights converge to correct values.

After training:

w₁ (exclamation marks) → high positive (it indicates spam).
w₂ (the word "free") → very high positive (a strong indicator of spam).
w₃ (a known sender) → negative (it indicates not-spam).

Result: The neuron learns to distinguish spam from not-spam by adjusting its weights based on experience (the training data).

Analogy: learning as feedback

Going back to the bank employee analogy:

At the start: The employee has no experience → they approve/reject loans at random.

Feedback from the boss (supervisor):

"You approved this loan but the applicant didn't pay → next time, reduce the weight of income and increase the weight of credit history."
"You rejected this loan but the applicant was reliable → next time, reduce the negative weight of the amount."

After thousands of cases: The employee adjusts their criteria (weights) and learns to make good decisions.

Moral: The neuron "learns" the same way: it receives feedback (an error), adjusts weights, improves with experience.

Limitations of the simple perceptron

Problem: it can only learn linear relationships

A simple perceptron (1 neuron) can only learn problems where the classes are linearly separable (you can draw a straight line that separates the two classes).

Example of a linear problem (the spam detector):

If you plot emails in 2D (X axis = number of "free"s, Y axis = number of "!!!"s), you can draw a line that separates spam (upper right) from not-spam (lower left).

Example of a NON-linear problem (XOR):

The XOR problem: Given two inputs (x₁, x₂), predict 1 if only one of them is 1, otherwise predict 0.

x₁	x₂	y (XOR)
0	0	0
0	1	1
1	0	1
1	1	0

If you plot the points in 2D, you CANNOT draw a straight line that separates the cases where y = 1 from the cases where y = 0.

Conclusion: A simple perceptron CANNOT solve XOR. You need multiple neurons organized in layers (that's a neural network).

From neuron to neural network

The solution: connecting neurons in layers

If one neuron can learn simple linear relationships, what happens if we connect many neurons in layers?

Result: We can learn complex non-linear relationships (e.g. XOR, facial recognition, machine translation).

Basic structure:

Input layer: Neurons that receive the inputs (e.g. the pixels of an image).
Hidden layers: Intermediate layers of neurons that process information (they learn complex features).
Output layer: Neurons that generate the final prediction (e.g. "cat" or "dog").

Visual example (a network with 1 hidden layer):

Input Layer    Hidden Layer    Output Layer
(3 neurons)    (4 neurons)     (1 neuron)

  x₁  ───────┐
             ├───→ h₁ ───┐
  x₂  ───────┤           │
             ├───→ h₂ ───┼───→ y (output)
  x₃  ───────┤           │
             ├───→ h₃ ───┘
             └───→ h₄

Each arrow represents a connection with a weight (adjustable).

Key point: Each neuron in the hidden layer learns to detect features (e.g. edges in an image, words in a text). The output layer combines those features to make the final prediction.

Why multiple layers make it possible to learn complex relationships

Intuition:

1 layer (a simple perceptron): It can only learn linear relationships (e.g. "if x₁ > 5, then y = 1").
2 layers (1 hidden layer): It can learn simple non-linear relationships (e.g. XOR, simple curves).
Many layers (a deep network): It can learn extremely complex relationships (e.g. recognizing faces, translating languages, generating text).

Analogy: A factory with a single worker (1 neuron) can only do simple tasks. A factory with departments (layers of neurons) can do complex tasks (each department specialized in one task, then they're combined).

Concrete example: image recognition (dog vs cat)

Layer 1 (the first hidden layers): Learns basic features (edges, textures).
Layer 2 (the intermediate layers): Learns more complex features (ears, eyes, snout).
Layer 3 (the last hidden layers): Learns specific features (the shape of a dog's ears vs a cat's, whisker patterns).
Output layer: Combines everything and predicts "dog" or "cat".

Moral: Layers make a hierarchy of features possible: early layers learn the basics (edges), later layers learn the complex (whole objects).

Why this matters for an AI Engineer

1. Understanding model architectures

When you integrate a model from Hugging Face or from an API, you'll see descriptions like:

"ResNet-50: A neural network with 50 layers, a CNN architecture."
"BERT: A Transformer with 12 layers, 110 million parameters."

If you understand what a neuron and a layer are:

You know that "50 layers" means a deep network → more expressive but slower.
You know that "110M parameters" are the network's weights → more parameters → more compute, more memory.

Without understanding: You don't know what those numbers mean → you can't reason about trade-offs (speed vs accuracy).

2. Debugging

If your image classifier is failing (low accuracy), you can diagnose:

Possible cause 1: Inadequate architecture

You used a simple feedforward network (like the perceptron) instead of a CNN (specialized in images).
Solution: Switch to a CNN (you'll see this in Lesson 07).

Possible cause 2: A network that's too simple (underfitting)

You used 1 hidden layer with 10 neurons → it doesn't have enough capacity to learn complex patterns.
Solution: Add more layers or more neurons per layer.

Possible cause 3: A network that's too complex (overfitting)

You used 100 layers without regularization → it learns the training set's noise, it doesn't generalize.
Solution: Add dropout, reduce layers, or increase the data.

3. Optimization

If your app is slow, you can analyze:

Bottleneck: Each layer adds latency (the forward pass: passing data through all the layers).
Solution: Use a smaller model (fewer layers, fewer neurons) or optimization techniques (quantization, pruning).

Example:

Current model: 50 layers, 25M parameters, 2 seconds per prediction.
Optimized model: 18 layers, 11M parameters, 0.5 seconds per prediction.
Trade-off: You lose 2% accuracy but gain 4× speed.

Decision: It depends on product priorities (speed vs accuracy).

4. Communication

When product asks "Why does the model need so much GPU?", you can explain:

"The model has 100 million parameters (weights). Each prediction requires multiplying those parameters by the inputs and summing → millions of mathematical operations → we need a GPU to do it fast."

Without understanding: You can't explain it technically → product doesn't understand why the cost is high.

Common mistakes

1. Confusing "artificial neuron" with "biological neuron"

Mistake: Thinking that artificial neurons work the same way as biological ones (chemistry, electricity, synapses).

Reality: They're simplified mathematical models. They only take inspiration from the general idea (receive inputs, process them, adjust connections).

2. Thinking you need to implement a neuron from scratch

Mistake: Believing you don't understand neurons if you can't program them in Python from scratch.

Reality: Frameworks (PyTorch, TensorFlow) implement neurons for you. What matters is understanding what they do (inputs → weights → sum → activation → output), not coding them manually.

3. Assuming a single neuron solves complex problems

Mistake: Thinking a neuron (perceptron) can solve any problem.

Reality: A neuron only solves linear problems. For complex problems (images, text, etc.) you need multiple layers (deep neural networks).

Frequently asked questions

How are the initial weights chosen?

Answer: Randomly. During training (backpropagation, which you'll see in Lesson 06), the weights are adjusted to minimize the error.

Technical detail (optional): Techniques like "Xavier initialization" or "He initialization" are used to avoid vanishing/exploding gradient problems. But frameworks do it automatically.

What is the bias (b)?

Answer: An additional number that adjusts the neuron's activation threshold.

Analogy: In the bank example, the bias is like a "base threshold" (e.g. "the employee has a tendency to reject loans unless the inputs are very positive").

Technical detail: The bias lets the neuron activate even if all the inputs are 0. Without a bias, if x₁ = x₂ = ... = 0, the sum is 0 → the neuron can't learn in that case.

What is the activation function?

Short answer: A function that transforms the weighted sum (z) into the final output (y).

Example: Step function (if z > 0, y = 1; otherwise, y = 0), ReLU (if z > 0, y = z; otherwise, y = 0), sigmoid (y = 1 / (1 + e^(-z))).

Why it matters: You'll see it in detail in Lesson 04 (Activation Functions).

Practical exercise

Task: compute a neuron's output

Given:

Inputs: x₁ = 2, x₂ = 3, x₃ = 1
Weights: w₁ = 0.5, w₂ = -1.0, w₃ = 2.0
Bias: b = -1.0
Activation function: Step function (if z > 0, y = 1; otherwise, y = 0)

Question: What is the output y?

Solution:

z = (x₁ × w₁) + (x₂ × w₂) + (x₃ × w₃) + b
z = (2 × 0.5) + (3 × -1.0) + (1 × 2.0) + (-1.0)
z = 1 + (-3) + 2 + (-1)
z = -1

Activation: z = -1 < 0 → y = 0

Answer: y = 0

Task: interpret weights

Given:

Problem: Classify whether a student will pass an exam (1) or not (0).
Inputs: x₁ = hours studied, x₂ = class attendance (%), x₃ = previous exams passed.
Weights after training: w₁ = 0.8, w₂ = 0.3, w₃ = 1.2.

Question: Which factor is most important for passing the exam?

Answer: w₃ (previous exams passed) has the highest weight (1.2) → it's the most influential factor. Then w₁ (hours studied, 0.8), and finally w₂ (attendance, 0.3).

Interpretation: The neuron "learned" that prior history (exams passed) is the best predictor of success, followed by hours studied.

Summary

What is an artificial neuron?

A mathematical model inspired by biological neurons.
It receives inputs (numbers), weighs them with weights (adjustable), sums everything (+ bias), applies an activation function, generates an output (a number).

Flow:

Inputs → Weights → Sum → Activation → Output

How it learns:

Initially, the weights are random → incorrect predictions.
During training: compute the error, adjust the weights, repeat with more examples.
Result: The weights converge to correct values.

Limitation of the simple perceptron:

It can only learn linear relationships.
For complex problems, you need multiple neurons in layers (neural networks).

Why it matters for AI Engineering:

Understanding what neurons and layers are lets you interpret model architectures, diagnose problems, optimize, and communicate technically.

Next step: Lesson 03: Layers and Architecture — How neurons are organized into layers (input, hidden, output) and what "deep" network means.

Additional resources

3Blue1Brown: But what is a neural network? — A visual video about what an artificial neuron is. In English with subtitles. 19 minutes.
TensorFlow Playground — A visual interface for experimenting with neurons and layers (no code). You can see how the weights are adjusted during training.
Perceptron - Wikipedia — The history and technical details of the perceptron. In English.
Deep Learning Book: Chapter 6 (Deep Feedforward Networks) — A technical chapter about neurons and networks. In English. Very technical (with formulas).

← Previous1. Module Introduction: Neural Networks & Deep Learning Next →3. Layers and Architecture of Neural Networks