Unit II - Artificial Neural Network (ANN)

📌 Syllabus Topics Covered

14 Hrs Weightage

2.1 Artificial Neural Network (ANN)
2.1.1 The architecture of an artificial neural network
2.2.2 Single layer perceptron
2.1.3 Multilayer perceptron
2.2 Back Propagation Learning
2.2.1 Forward pass
2.2.2 Backward pass
2.3 Models improvement
2.4 Training Neural Networks
2.5 Activation functions (Step, Sigmoid, ReLU, Leaky ReLU)
2.6 Gradient Descent
2.7 Common Challenges in Training - Overfitting and underfitting

📖 Comprehensive Theoretical Notes

Exam-Oriented Theory

2.1 Single Layer Perceptron (SLP) vs Multilayer Perceptron (MLP)

Single Layer Perceptron (Rosenblatt, 1958): The simplest feedforward neural network consisting of input nodes directly connected to output threshold units without hidden layers. It can only classify linearly separable patterns (e.g., AND, OR logic gates) and fundamentally fails on non-linear problems like the XOR problem (proven by Minsky & Papert, 1969).

Multilayer Perceptron (MLP): Stacks one or more hidden layers between inputs and outputs with non-linear activations. By learning composite hyperplanes, MLPs can approximate arbitrary continuous non-linear decision boundaries.

2.2 Backpropagation Algorithm: Forward Pass & Backward Pass

Backpropagation (Rumelhart et al., 1986) is the foundational learning algorithm for training multi-layer neural networks via the calculus Chain Rule:

Forward Pass: Input data vectors propagate through network layers. Each layer calculates $\mathbf{z}^{[l]} = \mathbf{W}^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}$ and $\mathbf{a}^{[l]} = g^{[l]}(\mathbf{z}^{[l]})$. The final layer produces $\hat{\mathbf{y}}$, and the loss function $\mathcal{L}(\hat{\mathbf{y}}, \mathbf{y})$ evaluates the prediction error.
Backward Pass: The loss gradient is calculated with respect to output activations, then propagated backward through layers using the chain rule to obtain partial derivatives: $$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{[l]}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{[l]}} (\mathbf{a}^{[l-1]})^T \quad \text{and} \quad \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{[l]}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{[l]}}$$
Parameter Update: Weights and biases are updated in the opposite direction of the gradient: $$\mathbf{W}^{[l]} := \mathbf{W}^{[l]} - \alpha \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{[l]}}$$

2.5 Activation Functions: Mathematical Formulations & Comparison

Activation functions introduce non-linearity, allowing neural networks to model non-linear relationships:

Step Function: $f(z) = 1 \text{ if } z \ge 0 \text{ else } 0$. Derivative is zero everywhere, preventing gradient descent training.
Sigmoid Function: $\sigma(z) = \frac{1}{1 + e^{-z}}$, range $(0, 1)$. Output derivative is $\sigma(z)(1 - \sigma(z))$. Suffer from vanishing gradient problem when $|z|$ is large.
Rectified Linear Unit (ReLU): $f(z) = \max(0, z)$. Derivative is $1$ for $z > 0$ and $0$ for $z < 0$. Eliminates vanishing gradients in positive domain and provides fast computation.
Leaky ReLU: $f(z) = \max(\alpha z, z)$ with small $\alpha \approx 0.01$. Prevents the "Dying ReLU" neuron death problem by maintaining a non-zero gradient for negative inputs.

2.6 Gradient Descent & 2.7 Training Challenges (Overfitting / Underfitting)

Gradient Descent Variants:

Batch Gradient Descent: Computes gradients over the entire training dataset. Stable convergence but extremely slow for massive datasets.
Stochastic Gradient Descent (SGD): Computes gradients on one single sample at a time. Fast and memory-efficient but noisy, oscillating convergence.
Mini-Batch Gradient Descent: Computes gradients over a small batch (e.g., 32, 64, 128 samples). Balances computational vectorization and convergence stability.

Mitigating Overfitting & Underfitting:

Overfitting: High variance (high training accuracy, poor validation accuracy). Solutions: L1/L2 Regularization (Weight Decay), Dropout, Data Augmentation, Early Stopping.
Underfitting: High bias (poor training and validation accuracy). Solutions: Increase model capacity (add layers/neurons), train longer, reduce regularization.

🔑 Key Concepts & Examination Keywords

Quick Terminology

Vanishing Gradient

Problem where backpropagated error gradients shrink exponentially towards zero in early layers due to saturating activations (Sigmoid/Tanh).

Dying ReLU

Occurs when neurons get stuck in negative territory where ReLU gradient is 0, permanently deactivating the neuron.

Dropout

Regularization technique where random neurons are deactivated during training with probability $p$ to prevent co-adaptation.

Learning Rate

Hyperparameter $\alpha$ governing the step size taken in parameter space along the negative loss gradient vector.

🎯 High-Yield Important Examination Questions

8–10 Descriptive Points Each

Q1. Explain the step-by-step mathematical derivation of the Backpropagation algorithm including Forward Pass, Backward Pass, and Weight Update rules.

10 MarksMathematical DerivationCore Algorithm

▼

📝 Detailed Examination Answer (10-Point Model):

Core Objective of Backpropagation: Backpropagation calculates the exact analytical gradient of the global scalar loss function with respect to all internal weight matrices and bias vectors.
Feedforward Signal Flow: In the forward pass, layer $l$ calculates intermediate pre-activation $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and post-activation $a^{[l]} = g^{[l]}(z^{[l]})$. Target loss $L(\hat{y}, y)$ is evaluated at layer $L$.
Loss Gradient at Output Layer: The error delta at the final output layer is defined as $\delta^{[L]} = \frac{\partial L}{\partial z^{[L]}} = \nabla_{a^{[L]}} L \odot {g^{[L]}}'(z^{[L]})$, representing the initial sensitivity vector.
Application of Multivariate Chain Rule: Using the chain rule, the error sensitivity of intermediate layer $l$ is propagated backward via: $\delta^{[l]} = ((W^{[l+1]})^T \delta^{[l+1]}) \odot {g^{[l]}}'(z^{[l]})$.
Weight Gradient Matrix Formulation: The partial derivative of the loss with respect to weight matrix $W^{[l]}$ is given by the outer product: $\frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T$.
Bias Gradient Vector Formulation: The partial derivative with respect to bias vector $b^{[l]}$ equals the accumulated error delta: $\frac{\partial L}{\partial b^{[l]}} = \delta^{[l]}$.
Gradient Descent Parameter Update: Parameters are updated iteratively: $W^{[l]} \leftarrow W^{[l]} - \alpha \frac{\partial L}{\partial W^{[l]}}$ and $b^{[l]} \leftarrow b^{[l]} - \alpha \frac{\partial L}{\partial b^{[l]}}$, where $\alpha$ is the learning rate.
Vectorized Batch Backpropagation: For mini-batches of size $m$, gradients are averaged over matrix operations: $\frac{\partial L}{\partial W^{[l]}} = \frac{1}{m} \Delta^{[l]} (A^{[l-1]})^T$ ensuring parallel GPU efficiency.
Computational Complexity: Backpropagation computes all gradients across the entire network in time proportional to a single forward pass ($O(W)$ where $W$ is total weights).
Impact of Activation Choice: The term ${g^{[l]}}'(z^{[l]})$ dictates gradient propagation; activations with vanishing derivatives stifle backward flow in early layers.

Q2. Compare Step, Sigmoid, ReLU, and Leaky ReLU activation functions. Explain their mathematical definitions, advantages, limitations, and use cases.

10 MarksActivation Functions

▼

📝 Detailed Examination Answer (10-Point Model):

Step Function Formulation & Limitations: Defined as $f(z) = 1$ if $z \ge 0$ else $0$. It produces a derivative of 0 almost everywhere, making gradient-based optimization impossible in multi-layer networks.
Sigmoid Activation Formulation: Defined as $\sigma(z) = \frac{1}{1 + e^{-z}}$, mapping real numbers smoothly to $(0, 1)$, historically popular for probabilistic output interpretation.
Vanishing Gradient in Sigmoid: Sigmoid's maximum derivative is $0.25$ at $z=0$. For $|z| > 4$, derivative $\approx 0$, causing gradients to vanish when backpropagating through deep networks.
Sigmoid Non-Zero Centered Issue: Sigmoid outputs are strictly positive, causing zig-zagging gradient updates during backpropagation because all weight updates share the same sign.
ReLU (Rectified Linear Unit) Formula: Defined as $f(z) = \max(0, z)$. It is computationally efficient as it only requires thresholding at zero without exponential operations.
ReLU Vanishing Gradient Resolution: For positive inputs ($z > 0$), ReLU maintains a constant gradient of $1.0$, allowing gradients to flow unimpeded through hundreds of hidden layers.
Dying ReLU Problem: If a large gradient updates weights such that a neuron outputs negative values for all dataset inputs, its gradient becomes permanently zero, killing the neuron.
Leaky ReLU Formulation: Defined as $f(z) = \max(\alpha z, z)$ where $\alpha \approx 0.01$. Provides a small positive slope for negative inputs to keep dead neurons active.
Parametric & Exponential Variants (PReLU/ELU): PReLU treats $\alpha$ as a learnable parameter, whereas ELU uses smooth exponential curves for negative values to push mean activations closer to zero.
Practical University Exam Selection Rule: Hidden layers: Use ReLU or Leaky ReLU; Binary Output Layer: Sigmoid; Multi-Class Output Layer: Softmax.

Q3. Describe the common challenges in training neural networks (Overfitting, Underfitting, Vanishing Gradients) and explain 6 proven techniques to overcome them.

10 MarksOptimization & Regularization

▼

📝 Detailed Examination Answer (10-Point Model):

Understanding Overfitting (High Variance): The model memorizes training noise and outliers, resulting in near-zero training error but high validation/test generalization error.
Understanding Underfitting (High Bias): The model is too simplistic to capture underlying patterns, resulting in poor accuracy on both training and validation sets.
L1 & L2 Weight Regularization: Adds penalty terms to the loss function: $\Omega(\theta) = \lambda \sum |w_i|$ (L1 Lasso, induces sparsity) or $\Omega(\theta) = \frac{1}{2}\lambda \sum w_i^2$ (L2 Ridge / Weight Decay, prevents large weights).
Dropout Regularization: Randomly sets a fraction $p$ of hidden activations to zero during each forward training step, breaking co-adaptation among feature detectors.
Batch Normalization (BatchNorm): Normalizes layer inputs to zero mean and unit variance per mini-batch, stabilizing internal covariate shift and allowing higher learning rates.
Data Augmentation: Synthetically expands training data using domain-preserving transformations (rotations, cropping, flips, color jitter, noise injection) to improve invariance.
Early Stopping: Monitors validation loss during training and halts optimization when validation performance ceases to improve, saving model checkpoints at minimum loss.
Weight Initialization Strategies: Using Xavier/Glorot initialization for Sigmoid/Tanh or He/Kaiming initialization for ReLU prevents activations and gradients from exploding or vanishing at initialization.
Adaptive Optimization Algorithms: Optimizers like RMSprop and Adam adjust per-parameter learning rates dynamically using exponentially decaying moving averages of past gradients and squared gradients.
Addressing Underfitting: Resolved by increasing model complexity (adding layers/neurons), engineering better features, or decreasing regularization constraints.

⚖️ Comprehensive Comparison & Difference Tables

8+ Comparison Criteria

📊 Single Layer Perceptron vs Multilayer Perceptron (MLP)

Comparison Parameter	Single Layer Perceptron (SLP)	Multilayer Perceptron (MLP)
Layer Architecture	Contains only Input Layer and Output Layer (0 hidden layers).	Contains Input Layer, one or more Hidden Layers, and Output Layer.
Decision Boundary	Can only generate linear decision boundaries (hyperplanes).	Can generate highly complex non-linear arbitrary decision surfaces.
XOR Problem	Fails to solve the non-linear XOR logic function.	Easily solves XOR and non-linear parity problems.
Activation Functions	Typically uses Step / Threshold activation functions.	Uses continuous non-linear activations (ReLU, Sigmoid, Tanh).
Training Algorithm	Trained using simple Perceptron Learning Rule / Delta Rule.	Trained using Backpropagation with Gradient Descent.
Computational Complexity	Very low computational and memory complexity ($O(N)$).	High computational complexity requiring matrix operations on GPUs.
Differentiability	Step function is non-differentiable at $z=0$ and has zero gradient elsewhere.	Activations are differentiable almost everywhere for calculus chain rule.
Universal Approximation	Cannot approximate general non-linear functions.	Satisfies Universal Approximation Theorem given sufficient hidden units.
Application Scope	Simple binary linear classification (e.g., AND, OR gates).	Complex pattern recognition, speech, vision, and tabular prediction.

📊 Overfitting vs Underfitting in Neural Networks

Comparison Parameter	Overfitting (High Variance)	Underfitting (High Bias)
Core Problem	Model learns training noise and specific sample peculiarities.	Model is too simple to capture underlying underlying relationships.
Training Loss	Extremely low training loss / high training accuracy.	High training loss / low training accuracy.
Validation / Test Loss	Significantly higher than training loss (generalization gap).	High validation loss, comparable to poor training loss.
Model Capacity	Model capacity is excessively high relative to training data size.	Model capacity is insufficient (too few layers or parameters).
Decision Boundary	Overly complex, highly oscillating, and sensitive to noise.	Overly rigid, simple, or linear, failing to separate data.
Root Causes	Lack of regularization, small dataset, excessive training epochs.	Oversimplified model, excessive regularization, premature stopping.
Remediation Techniques	Apply Dropout, L2 weight decay, Data Augmentation, Early Stopping.	Increase network depth/width, reduce regularization, train longer.
Visual Characteristic	Fits every single training outlier perfectly.	Fails to fit even standard training trendlines.

⚡ Quick Pre-Exam Revision Summary

5-Minute Recap

💡 Core Takeaways & High-Yield Summary

Single Layer Perceptron can only classify linearly separable data; MLPs solve non-linear classification (XOR).
Backpropagation calculates loss gradients across layers using the multivariate Chain Rule.
Forward pass computes layer activations; backward pass propagates error deltas $\delta$ to compute $\frac{\partial L}{\partial W}$.
ReLU ($f(z)=\max(0,z)$) prevents vanishing gradients in positive domains; Leaky ReLU prevents the Dying ReLU problem.
Overfitting is mitigated by Dropout, L2 regularization, Data Augmentation, and Early Stopping.
Batch Normalization stabilizes internal covariate shift and accelerates training convergence.