Module 3: Neural Networks & Deep Learning
7. Specialized Architectures: CNNs, RNNs and Feedforward
Description
In the previous lessons you saw the fundamentals of neural networks: what a neuron is, how neurons are organized into layers, activation functions, the forward pass, and backpropagation. But so far we've been talking about generic networks (fully connected / feedforward).
In practice, there are specialized architectures optimized for specific types of data:
- CNNs (Convolutional Neural Networks): For images (they capture spatial patterns: edges, textures, shapes).
- RNNs (Recurrent Neural Networks): For sequences (text, time series, audio → they capture temporal dependencies).
- Feedforward (Fully Connected): For tabular data (tables with rows and columns, without spatial or temporal structure).
In this lesson you'll understand:
- What problem each architecture solves (why the variants exist).
- How they work (conceptually, without implementing them from scratch).
- When to use each one (depending on your problem and type of data).
- Real examples (which models use each architecture).
Approach: Conceptual, with visualizations. You'll see diagrams, analogies, and examples of products that use each architecture.
Feedforward Networks (Fully Connected)
What they are
Definition: Networks where each neuron is connected to all the neurons in the next layer (fully connected).
Structure:
Input Layer → Hidden Layers (fully connected) → Output Layer
Example (already seen in previous lessons):
Input (5 features) → Hidden1 (32 neurons) → Hidden2 (16 neurons) → Output (1 neuron)
When to use them
Type of data: Tabular data (tables with rows and columns, without spatial or temporal structure).
Example problems:
- Predicting a house price (features: size, location, rooms, year).
- Classifying whether a customer will buy a product (features: age, purchase history, browsing).
- Predicting credit risk (features: income, credit history, loan amount).
Why it works: The features are independent (there's no spatial relationship as in images, or temporal one as in sequences). The network only needs to learn relationships between features.
Advantages and disadvantages
Advantages:
- ✅ Simple: Easy to understand and implement.
- ✅ It works well for tabular data.
Disadvantages:
- ❌ Many parameters: If you have many features (e.g. 1000), the number of weights is enormous (1000 × 500 = 500K weights between the input and the first hidden layer).
- ❌ It doesn't capture spatial patterns (images) or temporal ones (sequences).
CNNs (Convolutional Neural Networks)
The problem with feedforward for images
Imagine you want to classify 28×28 pixel images (e.g. MNIST digits).
Feedforward approach:
- Input layer: 784 neurons (28 × 28 = 784 pixels).
- Hidden layer 1: 500 neurons (fully connected).
- Number of weights: 784 × 500 = 392,000 weights (just between the input and the first hidden layer).
Problems:
- Too many parameters: 392K weights → a lot of memory, a lot of compute, a risk of overfitting.
- It doesn't capture spatial structure: Nearby pixels are related (they form edges, textures), but feedforward treats each pixel independently.
Solution: CNNs (they use convolutions to capture spatial patterns with fewer parameters).
How CNNs work
Central idea: Instead of connecting each neuron to all the pixels, each neuron looks at a small patch (e.g. 3×3 pixels) and detects local patterns (edges, textures).
Main components:
- Convolutional Layers: They detect local patterns (edges, textures, shapes) by applying filters (kernels) to patches of the image.
- Pooling Layers: They reduce resolution (e.g. from 28×28 to 14×14) while keeping important information. It reduces compute and parameters.
- Fully Connected Layers: At the end of the network, they combine the features detected by the convolutional layers and make the prediction.
Visual example (image classification):
Input Image (28×28)
↓
Conv Layer 1 (detects edges)
↓
Pooling (reduces to 14×14)
↓
Conv Layer 2 (detects textures)
↓
Pooling (reduces to 7×7)
↓
Fully Connected Layers (combines features)
↓
Output (10 classes: 0-9)
What each layer learns
Convolutional layer 1 (the first layers):
- It detects basic features: horizontal edges, vertical edges, curves.
Convolutional layer 2 (the intermediate layers):
- It combines edges and curves to detect more complex shapes: circles, corners, textures.
Convolutional layer 3+ (the later layers):
- It detects parts of objects: ears, eyes, wheels, windows.
Fully connected layers (the end):
- They combine the parts of objects to identify complete objects: dog, cat, car.
Advantages of CNNs
1. Fewer parameters: Each convolutional filter has few weights (e.g. 3×3 = 9 weights) but is applied across the whole image → far fewer connections than fully connected.
2. They capture spatial structure: Nearby pixels are processed together → the network learns local patterns (edges, textures).
3. Translation invariance: Whether a dog is in the upper-left corner or in the center, the CNN recognizes it just the same (because the filters are applied across the whole image).
When to use CNNs
Type of data: Images, videos, data with spatial structure.
Example problems:
- Classifying images (dog vs cat, digits, objects).
- Object detection (where the objects are in an image).
- Segmentation (identifying each pixel: sky, car, person).
- Facial recognition.
- Medical images (detecting tumors in X-rays).
Famous models (CNNs):
- AlexNet (2012): The first deep CNN to win ImageNet. 8 layers.
- VGG (2014): 16-19 layers. Widely used as a baseline.
- ResNet (2015): 50-152 layers. It uses "skip connections" to train very deep networks.
- MobileNet (2017): A CNN optimized for phones (few parameters, fast).
RNNs (Recurrent Neural Networks)
The problem with feedforward for sequences
Imagine you want to predict the next word in a sentence: "The cat is on the ___".
Feedforward approach:
- Input: Encode the 5 previous words as a vector.
- Problem: It doesn't capture temporal dependencies. The word "cat" (position 2) influences the prediction ("tree" is unlikely; "roof" or "floor" are likely), but feedforward treats each word independently.
Solution: RNNs (they process sequences step by step, keeping a "memory" of previous words).
How RNNs work
Central idea: The network has a hidden state that is updated at each step of the sequence. That state "remembers" information from previous steps.
Diagram (processing "The cat is"):
Input: "The" → "cat" → "is"
↓ ↓ ↓
State: h₁ → h₂ → h₃ → Output
At each step:
- The RNN receives the current input (e.g. "cat") + the previous state (h₁).
- It processes both and generates a new state (h₂).
- The new state "remembers" information from "The" and "cat".
Moral: The RNN maintains a "memory" of the sequence.
Advantages of RNNs
1. They capture temporal dependencies: Previous words influence future predictions.
2. Variable length: They can process sequences of any length (e.g. sentences of 5 words or 50 words).
Disadvantages of RNNs
1. The vanishing gradient problem: In long sequences (e.g. 100 words), information from early words is "forgotten" (the gradients become very small).
Solution: LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) → RNN variants with "long-term memory".
2. Sequential processing (slow): RNNs process one step at a time (they can't parallelize) → slow for long sequences.
Solution: Transformers (the modern architecture that replaced RNNs in NLP). You'll see them in Module 4.
When to use RNNs
Type of data: Sequences (text, time series, audio, video).
Example problems:
- Next-word prediction (autocomplete).
- Machine translation (a sequence of words in English → a sequence in Spanish).
- Sentiment analysis (classifying a review as positive/negative).
- Time series prediction (e.g. stock prices).
- Speech recognition (audio → text).
Famous models (RNNs/LSTMs):
- LSTMs: Used in Google Translate (before Transformers), speech recognition.
- GRUs: A simpler variant of LSTMs.
Note: In modern NLP (2020+), Transformers (you'll see them in Module 4) replaced RNNs because they're faster and capture long-range dependencies better. But RNNs are still used in time series and some audio problems.
Comparison of architectures
| Architecture | Type of data | Advantages | Disadvantages | Example uses |
|---|---|---|---|---|
| Feedforward | Tabular data | Simple, works well for tables | Many parameters, doesn't capture spatial/temporal patterns | Price prediction, binary classification |
| CNN | Images, spatial data | Captures spatial patterns, fewer parameters than feedforward | Specific to spatial data | Image classification, facial recognition, object detection |
| RNN/LSTM | Sequences (text, time series, audio) | Captures temporal dependencies, variable length | Slow (sequential), vanishing gradients | Translation, sentiment analysis, time series prediction |
How to choose an architecture
Step 1: Identify the type of data
- A table (rows, columns): Feedforward.
- An image: CNN.
- A sequence (text, time series, audio): RNN/LSTM (or Transformer, which you'll see in Module 4).
Step 2: Use pre-trained models whenever possible
- Images: ResNet, MobileNet, EfficientNet (pre-trained on ImageNet).
- Text: BERT, GPT, T5 (pre-trained on massive corpora). They're Transformers, NOT RNNs.
- Audio: Wav2Vec, Whisper.
Moral: You almost never design architectures from scratch. You use proven (pre-trained) architectures and adjust them (fine-tuning) if necessary.
Why this matters for an AI Engineer
1. Model selection
When you integrate a model from Hugging Face, you'll see descriptions like:
- "ResNet-50: A CNN with 50 layers, a residual architecture."
- "BERT: A Transformer (NOT an RNN) with 12 layers."
If you understand architectures:
- You know that ResNet is a CNN → for images.
- You know that BERT is a Transformer → for text (more modern than an RNN).
2. Debugging
If your image classifier has low accuracy:
Possible cause: You used feedforward instead of a CNN.
- Feedforward doesn't capture spatial patterns → low accuracy.
- Solution: Use a CNN (e.g. ResNet).
3. Optimization
If your app is slow:
- A large CNN (ResNet-152): Slow but accurate.
- A small CNN (MobileNet): Fast but less accurate.
Trade-off: Based on product priorities (speed vs accuracy).
Common mistakes
1. Using feedforward for images
Mistake: Using feedforward (fully connected) to classify images.
Problem: It doesn't capture spatial patterns → low accuracy, many parameters.
Solution: Use a CNN (e.g. ResNet, MobileNet).
2. Using an RNN for text in 2024
Mistake: Using an RNN/LSTM for text processing (NLP).
Problem: RNNs are slow and Transformers (BERT, GPT) are better.
Solution: Use Transformers (you'll see them in Module 4).
3. Designing architectures from scratch
Mistake: Trying to design a CNN or RNN from scratch for your problem.
Reality: Use proven architectures (ResNet, MobileNet, BERT) and adjust them (fine-tuning) if necessary.
Frequently asked questions
Can I combine architectures?
Yes. Many models combine CNNs, RNNs, and fully connected layers.
Example: Video classification (sequences of images).
- Use a CNN to extract features from each frame (image).
- Use an RNN to process the sequence of frames (capturing temporal dependencies).
- Use fully connected layers for the final classification.
What are Transformers?
Short answer: A modern architecture for sequences that replaced RNNs in NLP (text).
Why they're better than RNNs:
- Parallelization: They process the whole sequence at once (not sequentially like RNNs) → faster.
- Attention mechanism: They capture long-range dependencies better than RNNs.
You'll see them in detail in Module 4.
Summary
Three main architectures:
- Feedforward (fully connected): Tabular data. Simple, many parameters.
- CNN: Images, spatial data. Captures spatial patterns, fewer parameters.
- RNN/LSTM: Sequences (text, time series). Captures temporal dependencies, slow.
How to choose:
- Identify the type of data (table → feedforward, image → CNN, sequence → RNN or Transformer).
- Use pre-trained models (ResNet, BERT, etc.) instead of designing from scratch.
Next step: Lesson 08: Integrative Exercise — Designing architectures for given problems, identifying common mistakes.
Additional resources
-
CS231n: Convolutional Neural Networks — The Stanford course on CNNs. In English. Very technical.
-
Colah's Blog: Understanding LSTMs — A visual explanation of LSTMs. In English.
-
3Blue1Brown: But what is a convolution? — A visual video about convolutions. In English with subtitles.