Module 3: Neural Networks & Deep Learning
3. Layers and Architecture of Neural Networks
Description
In the previous lesson you saw what an artificial neuron is and how it decides (inputs → weights → sum → activation → output). But a single neuron is very limited: it can only learn simple linear relationships.
The real power of neural networks comes from connecting many neurons in layers. That makes it possible to learn complex, non-linear relationships: recognizing faces, translating languages, generating text, etc.
In this lesson you'll understand:
- What layers are (input layer, hidden layers, output layer).
- How neurons connect between layers (fully connected, sparse connections).
- What parameters are (weights + biases) and why they matter.
- What "deep" network means vs "shallow" network.
Approach: Conceptual and visual. You'll see architecture diagrams, analogies (layers = the departments of a factory), and explanations without complex mathematical formulas.
The three types of layers
Every neural network has at least three types of layers:
1. Input Layer
What it is: The network's first layer. It receives the input data.
Characteristics:
- It doesn't do computation (it only receives data).
- The number of neurons = the number of features in the problem.
Examples:
- Image classification (28x28 pixels): The input layer has 784 neurons (28 × 28 = 784 pixels, each one is an input).
- Text classification (100 words represented as 50-dimensional vectors): The input layer has 50 neurons (the vector's dimension).
- House price prediction (5 features: size, location, rooms, year, garage): The input layer has 5 neurons.
Diagram:
Input Layer (3 neurons)
x₁ (e.g. house size)
x₂ (e.g. location)
x₃ (e.g. number of rooms)
Analogy: The input layer is like a factory's receiving dock: it receives raw material (data) without processing it.
2. Hidden Layers
What they are: Intermediate layers between input and output. This is where the processing and the learning of features happen.
Characteristics:
- They do computation (each neuron receives inputs from the previous layer, applies weights, sums, activates, and passes the output to the next layer).
- The number of neurons is adjustable (a hyperparameter you define).
- The number of hidden layers is also adjustable.
Examples:
- A simple network: 1 hidden layer with 10 neurons.
- A deep network: 50 hidden layers with 100-1000 neurons each (e.g. ResNet-50).
Diagram:
Hidden Layer 1 (4 neurons)
h₁
h₂
h₃
h₄
Analogy: Hidden layers are like a factory's departments: each department processes the raw material (the input layer), transforming it step by step until it reaches the final product (the output layer).
What they learn:
- The first hidden layers: Basic features (e.g. in images: edges, textures; in text: letters, syllables).
- The intermediate layers: More complex features (e.g. in images: shapes, parts of objects; in text: words, phrases).
- The last hidden layers: Features specific to the problem (e.g. in dog vs cat classification: the shape of the ears, whisker patterns).
Key point: Layers make a hierarchy of features possible (from simple to complex).
3. Output Layer
What it is: The network's last layer. It generates the final prediction.
Characteristics:
- The number of neurons = the number of outputs in the problem.
- The activation function depends on the type of problem (classification, regression).
Examples:
Binary classification (spam vs not-spam):
- Output layer: 1 neuron (output = the probability of spam, e.g. 0.8 → 80% spam).
- Activation function: Sigmoid (output between 0 and 1).
Multiclass classification (classifying images into 10 categories: dog, cat, car, etc.):
- Output layer: 10 neurons (one per category).
- Activation function: Softmax (each neuron gives the probability of its category, and they sum to 1).
Regression (predicting a house price):
- Output layer: 1 neuron (output = the price, e.g. $250,000).
- Activation function: None or linear (the output can be any number).
Diagram (binary classification):
Output Layer (1 neuron)
y (e.g. probability of spam: 0.8)
Analogy: The output layer is like the factory's shipping area: the final product (the prediction) is ready to be sent to the customer (the user).
How layers connect
Fully Connected (Densely Connected)
Definition: Each neuron in a layer is connected to all the neurons in the next layer.
Diagram (a 3-layer fully connected network):
Input Layer Hidden Layer Output Layer
(3 neurons) (4 neurons) (2 neurons)
x₁ ─────┬───→ h₁ ─────┬───→ y₁
│ │
x₂ ─────┼───→ h₂ ─────┼───→ y₂
│ │
x₃ ─────┼───→ h₃ ─────┘
│
└───→ h₄
Each line represents a connection with a weight (adjustable during training).
Number of connections:
- Between the input layer (3 neurons) and the hidden layer (4 neurons): 3 × 4 = 12 connections (12 weights).
- Between the hidden layer (4 neurons) and the output layer (2 neurons): 4 × 2 = 8 connections (8 weights).
- Total: 12 + 8 = 20 weights (+ each neuron's bias).
Advantages:
- Maximum expressiveness: Each neuron can learn from all the neurons in the previous layer.
Disadvantages:
- Many parameters: If you have large layers (e.g. 1000 neurons), the number of weights is enormous (1000 × 1000 = 1M weights between two layers) → more memory, more compute, more risk of overfitting.
Common use: Feedforward networks (you'll see them in Lesson 07), the last layers of CNNs.
Sparse Connections
Definition: Each neuron is only connected to some of the neurons in the previous layer (not all of them).
Example: CNNs (Convolutional Neural Networks) for images. Each neuron only looks at a small patch of the image (e.g. 3×3 pixels), not the whole image.
Advantages:
- Fewer parameters: Fewer connections → fewer weights → less memory, less compute, less overfitting.
- It captures local patterns: In images, it's more useful for a neuron to look at a small area (e.g. an edge) than at the whole image.
Common use: CNNs (you'll see them in Lesson 07).
Parameters: Weights and Biases
What parameters are
Parameters = Weights + Biases.
- Weights: Numbers that multiply the inputs of each connection.
- Biases: Numbers that adjust each neuron's activation threshold.
Number of parameters:
Total parameters = (number of weights) + (number of biases)
Example (a 3-layer fully connected network):
- Input layer: 3 neurons
- Hidden layer: 4 neurons
- Output layer: 2 neurons
Weights:
- Between input and hidden: 3 × 4 = 12
- Between hidden and output: 4 × 2 = 8
- Total weights: 20
Biases:
- Hidden layer: 4 biases (one per neuron)
- Output layer: 2 biases (one per neuron)
- Total biases: 6 (the input layer has no biases because it does no computation)
Total parameters: 20 + 6 = 26 parameters.
Why the number of parameters matters
More parameters:
- ✅ More expressiveness: The network can learn more complex relationships.
- ❌ More compute: More mathematical operations → more time, more GPU.
- ❌ More memory: More numbers to store → more RAM, more VRAM.
- ❌ More risk of overfitting: If you don't have enough data, the network can "memorize" the training set instead of learning general patterns.
Fewer parameters:
- ✅ Less compute: Faster, cheaper.
- ✅ Less memory: You can run on devices with fewer resources (e.g. a phone).
- ❌ Less expressiveness: The network may not have enough capacity to learn complex patterns (underfitting).
Trade-off: You need to find the right balance for your problem (available data, complexity, computational resources).
Example: large vs small models
| Model | Parameters | Common use | Trade-offs |
|---|---|---|---|
| GPT-3 | 175 billion | An LLM for text generation | Very expressive, very slow, very expensive ($$$) |
| GPT-3.5-turbo | ~10-20 billion | An LLM optimized for production | Expressive, faster, cheaper ($$) |
| DistilBERT | 66 million | Text classification | Less expressive, fast, cheap ($) |
| MobileNet | 4 million | Image classification on phones | Less expressive, very fast, very cheap |
Moral: As an AI Engineer, you'll choose models based on trade-offs (accuracy vs speed vs cost).
Depth: Shallow vs Deep Networks
Shallow Network
Definition: A network with few hidden layers (e.g. 1-2 hidden layers).
Example:
Input (3) → Hidden (10) → Output (1)
Characteristics:
- Few parameters → fast, simple.
- Limited capacity → it can only learn relatively simple relationships.
Common use: Simple problems (e.g. binary classification with few features, linear regression).
Deep Network
Definition: A network with many hidden layers (e.g. 10, 50, 100 or more).
Example:
Input (784) → Hidden1 (256) → Hidden2 (128) → Hidden3 (64) → Output (10)
Characteristics:
- Many parameters → expressive, it can learn very complex relationships.
- Slow, expensive → it requires more compute, more data to train.
Common use: Complex problems (images, text, audio, games).
Why "deep" in Deep Learning
Deep Learning is called that because it uses deep networks (many layers).
History:
- 1980s-2000s: Neural networks existed, but they were shallow (2-3 layers) because they were hard to train (vanishing gradient problems, little compute, little data).
- 2012: AlexNet (a deep network with 8 layers) won ImageNet (the computer vision competition) by a huge margin → it showed that deep networks work if you have enough data and GPUs.
- 2012-today: The Deep Learning boom: ever-deeper networks (ResNet-152: 152 layers; Transformers: 96 layers in GPT-3).
Why depth works:
- A hierarchy of features: Early layers learn the basics (edges), later layers learn the complex (objects).
- Greater expressiveness: More layers → more transformations → they can learn more complex relationships.
- Efficiency: A deep network with fewer neurons per layer can be more efficient than a shallow network with many neurons in a single layer.
Complete example: a network for classifying digits (MNIST)
The problem
Dataset: MNIST (70,000 images of handwritten digits, 28×28 pixels, 10 classes: 0-9).
Task: Given an image of a digit, predict which number it is (0, 1, 2, ..., 9).
The network's architecture
Input layer:
- 784 neurons (28 × 28 = 784 pixels, each pixel is an input with a value of 0-255).
Hidden layer 1:
- 128 neurons (fully connected with the input layer).
- Activation function: ReLU (you'll see it in Lesson 04).
Hidden layer 2:
- 64 neurons (fully connected with hidden layer 1).
- Activation function: ReLU.
Output layer:
- 10 neurons (one per digit: 0, 1, 2, ..., 9).
- Activation function: Softmax (each neuron gives the probability of its digit).
Diagram:
Input (784) → Hidden1 (128) → Hidden2 (64) → Output (10)
Number of parameters
Weights:
- Input → Hidden1: 784 × 128 = 100,352
- Hidden1 → Hidden2: 128 × 64 = 8,192
- Hidden2 → Output: 64 × 10 = 640
- Total weights: 109,184
Biases:
- Hidden1: 128
- Hidden2: 64
- Output: 10
- Total biases: 202
Total parameters: 109,184 + 202 = 109,386 parameters.
Interpretation
What each layer learns:
- Hidden1 (128 neurons): Learns basic features (edges, curves, lines).
- Hidden2 (64 neurons): Learns more complex features (combinations of edges that form parts of digits).
- Output (10 neurons): Combines Hidden2's features and predicts which digit it is.
Why it works:
- 109K parameters is enough to learn digit patterns (MNIST is relatively simple).
- 2 hidden layers make a hierarchy of features possible (basic → complex).
Typical performance: 98-99% accuracy on the test set (2-3 errors per 100 images).
Why this matters for an AI Engineer
1. Model selection
When you choose a model on Hugging Face, you'll see descriptions like:
- "BERT-base: 12 layers, 768 hidden size, 110M parameters."
- "DistilBERT: 6 layers, 768 hidden size, 66M parameters."
If you understand architectures:
- You know that DistilBERT has fewer layers (6 vs 12) → less expressive but faster.
- You know that "768 hidden size" means each layer has 768 neurons.
- You can reason: "If latency isn't critical, I use BERT (more accurate). If I need fast responses, I use DistilBERT."
2. Debugging
If your image classifier has low accuracy (60%), you can diagnose:
Cause 1: Underfitting (a network that's too simple)
- Architecture: 1 hidden layer with 10 neurons.
- Problem: It doesn't have enough capacity to learn complex image patterns.
- Solution: Add more layers or more neurons per layer.
Cause 2: Overfitting (a network that's too complex without regularization)
- Architecture: 10 hidden layers with 1000 neurons each (10M parameters).
- Problem: It learns the training set's noise, it doesn't generalize to the test set.
- Solution: Add dropout (you'll see it in Lesson 06), reduce layers/neurons, or increase the data.
3. Optimization
If your app is slow (2 seconds per prediction), you can analyze:
- Current architecture: 50 layers, 25M parameters.
- Bottleneck: The forward pass (passing data through 50 layers takes time).
- Solution: Use a smaller model (18 layers, 11M parameters) → 4× faster, you lose 2% accuracy.
4. Fine-tuning
If you need to adjust a pre-trained model:
- Which layers to adjust: The last layers (they learn features specific to your problem). The first layers (which learn general features) are left frozen.
- Why: The first layers already learned basic features (edges, textures) that are useful for any image problem. The last layers need to learn features specific to your domain (e.g. specific product types).
Common mistakes
1. Confusing the number of layers with the number of neurons
Mistake: Thinking that a "10-layer network" means "10 neurons".
Reality: Layers ≠ neurons. A network can have 10 layers with 1000 neurons each (10,000 neurons in total).
2. Assuming more layers = always better
Mistake: Thinking that adding more layers always improves accuracy.
Reality: More layers → more expressiveness BUT also → more risk of overfitting, harder to train (vanishing gradients), slower.
Rule of thumb: Start simple (2-3 hidden layers); add complexity only if necessary.
3. Not understanding the impact of parameters on latency
Mistake: Choosing a model with 175B parameters (GPT-3) for an app that needs responses in <100ms.
Reality: More parameters → more compute → more latency. For apps that need low latency, use small models (e.g. DistilBERT, MobileNet).
Frequently asked questions
How many hidden layers should I use?
Answer: It depends on the problem.
- Simple problems (e.g. binary classification with few features): 1-2 hidden layers.
- Complex problems (images, text, audio): 10-100 hidden layers (or use pre-trained architectures).
Practical rule: Start simple; add layers if you see underfitting (low accuracy on both training and test).
How many neurons per hidden layer?
Answer: It's a hyperparameter (you tune it by experimenting).
Practical rule:
- Hidden layers usually have between 50-1000 neurons.
- The first hidden layers usually have more neurons than the last ones (e.g. 256 → 128 → 64).
Which is better: a deep network (many layers, few neurons) or a wide network (few layers, many neurons)?
Answer: It depends, but deep networks are usually more efficient.
Why: A deep network can learn a hierarchy of features (basic → complex). A wide network learns everything in one layer (less efficient).
Example: A deep network with 5 layers of 100 neurons each (50K parameters) can outperform a wide network with 1 layer of 500 neurons (250K parameters).
Practical exercise
Task: design an architecture for binary classification
Problem: Classify emails as spam (1) or not-spam (0).
Data: 10,000 emails, each email represented as a vector of 50 features (e.g. word frequencies).
Task: Design a neural network architecture (number of layers, neurons per layer, activation function).
Suggested solution:
Input layer: 50 neurons (50 features)
Hidden layer 1: 32 neurons, ReLU activation
Hidden layer 2: 16 neurons, ReLU activation
Output layer: 1 neuron, Sigmoid activation (probability of spam)
Justification:
- 2 hidden layers → enough to learn non-linear relationships without overfitting.
- 32 → 16 neurons (decreasing) → a common pattern in feedforward networks.
- Sigmoid at the output → binary classification (output between 0 and 1).
Parameters:
- Input → Hidden1: 50 × 32 = 1,600 weights
- Hidden1 → Hidden2: 32 × 16 = 512 weights
- Hidden2 → Output: 16 × 1 = 16 weights
- Biases: 32 + 16 + 1 = 49
- Total: 2,177 parameters (relatively small → fast, little risk of overfitting with 10K examples).
Summary
Three types of layers:
- Input layer: Receives data (does no computation).
- Hidden layers: Process data, learn features (a hierarchy: basic → complex).
- Output layer: Generates the final prediction.
Connections:
- Fully connected: Each neuron connected to all the neurons in the next layer (maximum expressiveness, many parameters).
- Sparse: Each neuron connected to some of them (fewer parameters, captures local patterns).
Parameters:
- Weights + biases.
- More parameters → more expressiveness, more compute, more risk of overfitting.
Depth:
- Shallow: Few layers → simple, fast, less expressive.
- Deep: Many layers → complex, slow, very expressive.
Why it matters:
- Understanding architectures lets you select models, diagnose problems, optimize latency, and do effective fine-tuning.
Next step: Lesson 04: Activation Functions — Why networks need activation functions (ReLU, sigmoid, tanh) and what they do.
Additional resources
-
3Blue1Brown: Gradient descent, how neural networks learn — A visual video about how networks learn by adjusting weights. In English with subtitles.
-
TensorFlow Playground — Experiment with architectures (number of layers, neurons) and see how it affects learning. No code.
-
CS231n: Neural Networks Part 1 — Notes from the Stanford course on architectures. In English. Technical (with formulas).
-
Deep Learning Book: Chapter 6 (Deep Feedforward Networks) — A technical chapter about layers and architectures. In English. Very technical (linear algebra).