8. Integrative Exercise: Designing Neural Network Architectures

Type of data: Images (256×256 pixels, grayscale)

Architecture: CNN (Convolutional Neural Network)

Design:

Input layer: 256×256×1 (1 channel: grayscale)
  ↓
Conv Layer 1: 32 filters 3×3, ReLU activation
  ↓
Pooling (2×2): Reduces to 128×128
  ↓
Conv Layer 2: 64 filters 3×3, ReLU activation
  ↓
Pooling (2×2): Reduces to 64×64
  ↓
Conv Layer 3: 128 filters 3×3, ReLU activation
  ↓
Pooling (2×2): Reduces to 32×32
  ↓
Flatten: Converts to a 1D vector
  ↓
Fully Connected Layer: 128 neurons, ReLU activation
  ↓
Fully Connected Layer: 64 neurons, ReLU activation
  ↓
Output layer: 1 neuron, Sigmoid activation (probability of pneumonia)

Justification:

• CNNs are ideal for images (they capture spatial patterns: edges, textures, shapes).
• The convolutional layers detect features (edges, spots in the lungs).
• Pooling reduces resolution (less compute) while keeping important information.
• The fully connected layers at the end combine the features and classify.
• Sigmoid at the output → binary classification (healthy vs pneumonia).

Alternative (better in practice):

• Use ResNet or EfficientNet pre-trained on ImageNet (transfer learning) and do fine-tuning with your 10,000 X-ray images. This is more effective than training from scratch.

Type of data: Tabular data (a table with 8 features per house)

Architecture: Feedforward (Fully Connected)

Design:

Input layer: 8 neurons (8 features)
  ↓
Hidden layer 1: 64 neurons, ReLU activation
  ↓
Hidden layer 2: 32 neurons, ReLU activation
  ↓
Hidden layer 3: 16 neurons, ReLU activation
  ↓
Output layer: 1 neuron, Linear activation (price: a number without restrictions)

Justification:

• Tabular data (without spatial or temporal structure) → Feedforward is appropriate.
• 3 hidden layers with decreasing sizes (64 → 32 → 16) is a common pattern for tables.
• ReLU in the hidden layers (the standard for Deep Learning).
• Linear output (regression: the price can be any positive number, without restrictions).

Parameters:

• Input → Hidden1: 8 × 64 = 512 weights
• Hidden1 → Hidden2: 64 × 32 = 2,048 weights
• Hidden2 → Hidden3: 32 × 16 = 512 weights
• Hidden3 → Output: 16 × 1 = 16 weights
• Biases: 64 + 32 + 16 + 1 = 113
• Total: ~3,200 parameters (small → fast, little risk of overfitting with 50K examples).

Type of data: Text (sequences of words)

Modern architecture (recommended): Transformer (e.g. BERT pre-trained in Spanish)

Traditional architecture (if you don't use Transformers): RNN/LSTM

Design (with LSTM, the traditional approach):

Input: A sequence of words (e.g. 100 words, each word represented as a 300-dimensional embedding)
  ↓
Embedding Layer: Converts words into vectors (300 dimensions)
  ↓
LSTM Layer 1: 128 units
  ↓
LSTM Layer 2: 64 units
  ↓
Fully Connected Layer: 32 neurons, ReLU activation
  ↓
Output layer: 3 neurons (positive, neutral, negative), Softmax activation

Justification (LSTM):

• Text is a sequence (temporal dependencies: previous words influence meaning).
• LSTM captures temporal dependencies (better than a simple RNN because it avoids vanishing gradients).
• Softmax at the output → multiclass classification (3 categories).

Design (with a Transformer, the modern approach):

Input: A sequence of words (tokenization)
  ↓
BERT pre-trained in Spanish (12 Transformer layers)
  ↓
Fully Connected Layer: 64 neurons, ReLU activation
  ↓
Output layer: 3 neurons, Softmax activation

Justification (Transformer/BERT):

• Transformers are better than RNNs for text (faster, they capture long-range dependencies better).
• A pre-trained BERT has already learned general characteristics of Spanish → you only need fine-tuning (adjusting the last layers with your 100K reviews).
• The recommended approach in 2024.

Type of data: Images (512×512 pixels, RGB)

Architecture: An Object Detection CNN (e.g. YOLO, Faster R-CNN, EfficientDet)

Design (conceptual, using YOLO as an example):

Input: A 512×512×3 image (RGB)
  ↓
CNN backbone (e.g. ResNet-50): Extracts features from the image
  ↓
Detection Head: Predicts bounding boxes (x, y, width, height) and classes (dog, cat, car)
  ↓
Output: A list of detected objects with coordinates and class

Justification:

• Images → CNN.
• Object detection requires specialized architectures (YOLO, Faster R-CNN) that predict multiple bounding boxes per image.
• It is NOT simple classification (which only predicts one class per image). It's detection (predicting multiple objects with locations).

Practical approach:

• Use a pre-trained model (e.g. YOLOv8, EfficientDet) on the COCO dataset (an object detection dataset).
• Fine-tune with your 50K images if your classes (dogs, cats, cars) aren't in COCO or you need to adapt to your domain.

Type of data: A time series (a sequence of months with features)

Architecture: RNN/LSTM or time-series-specific models (e.g. ARIMA, Prophet)

Design (with LSTM):

Input: A sequence of 12 months (each month: 4 features: sales, price, promotions, season)
  ↓
LSTM Layer 1: 64 units
  ↓
LSTM Layer 2: 32 units
  ↓
Fully Connected Layer: 16 neurons, ReLU activation
  ↓
Output layer: 1 neuron, Linear activation (sales prediction: a number)

Justification:

• A time series → a sequence with temporal dependencies (sales in previous months influence future sales).
• LSTM captures temporal dependencies (e.g. if there was a promotion 2 months ago, it can affect current sales).
• Linear output → regression (sales: a number without restrictions).

Alternative:

• Time-series-specific models like ARIMA or Prophet (from Facebook) sometimes work better than LSTMs for time series with clear trends and seasonality.
• Transformers for time series (e.g. the Temporal Fusion Transformer) are more modern than LSTMs.

Problems:

1. It uses Feedforward instead of a CNN: Images have spatial structure (nearby pixels are related). Feedforward treats each pixel independently → it does NOT capture spatial patterns (edges, textures).

2. Too many parameters: Input → Hidden1: 12,288 × 1,000 = 12.3 million weights (in the first layer alone) → a lot of memory, a lot of compute, a risk of overfitting.

Solution:

• Use a CNN (e.g. 3-4 convolutional layers + pooling + fully connected at the end).
• CNNs have fewer parameters (small filters: 3×3) and they capture spatial patterns.

Does it work? Yes, partially. CNNs can be used for text (1D convolutions capture local patterns of words).

Is it optimal? No.

Problems:

1. CNNs are less common for text: In modern NLP, Transformers (BERT, GPT) are better than CNNs because they capture long-range dependencies (e.g. the word at position 1 influences the word at position 50).

2. CNNs capture local patterns: Useful if the order of nearby words matters a lot (e.g. "not good" vs "very good"). But for complex sentiment analysis, Transformers are better.

Solution:

• Use a pre-trained BERT (a Transformer) and do fine-tuning → better accuracy than a CNN or an RNN.

Problem: Vanishing gradients.

• Sigmoid compresses gradients (derivative < 1 at almost every point).
• In a deep network (50 layers), the gradients are multiplied through many layers → they become nearly 0 → the early layers don't learn.

Solution:

• Change Sigmoid to ReLU in the hidden layers (ReLU doesn't compress gradients: derivative = 1 for z > 0).
• Sigmoid only in the output layer (binary classification).

1. What type of data?

• Images (a CNN).

2. Why so many parameters?

• 50 layers → many connections between layers → many weights (25.6M).
• Trade-off: More parameters → more expressiveness (better accuracy) BUT more compute, more memory.

3. Can you use it for X-rays?

• Yes, through transfer learning:

  1. ResNet-50 is pre-trained on ImageNet (natural photos: dogs, cats, cars).
  2. You use ResNet-50 as a base (keeping the convolutional layers that detect general features: edges, textures).
  3. You replace the last layer (output layer: 1,000 classes → 2 classes: healthy vs pneumonia).
  4. You fine-tune (adjust) with your X-ray data (you only adjust the last layers; the first ones stay frozen).

• Advantage: You need less data (e.g. 1,000-10,000 X-rays) than training ResNet from scratch (which would require millions of images).