Unit IV - Recurrent Neural Networks (RNNs)

📌 Syllabus Topics Covered

14 Hrs Weightage

4.1 Introduction to Recurrent Neural Networks
4.2 Types of Recurrent Neural Networks
4.3 Recurrent Neural Networks Architecture
4.4 Examples of sequential data: Text, time-series, and speech
4.5 Working of RNNs
4.6 Training Process in Recurrent Neural Networks
4.7 Vanishing Gradients
4.8 Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)
4.9 Encoder-Decoder Model
4.10 Applications of RNNs (e.g., text generation, language translation)

📖 Comprehensive Theoretical Notes

Exam-Oriented Theory

4.1 - 4.4 Sequential Data & RNN Architecture

Standard feedforward neural networks assume all inputs and outputs are independent of each other. However, sequential data (e.g., sentences, audio streams, financial time-series, DNA sequences) possesses strong temporal / sequential dependencies where previous elements dictate the context of current elements.

RNN Hidden State & Mathematical Formulation:

At each time step $t$, an RNN maintains an internal memory called the hidden state ($\mathbf{h}_t$) computed from current input $\mathbf{x}_t$ and previous hidden state $\mathbf{h}_{t-1}$:

$$\mathbf{h}_t = \tanh(\mathbf{W}_{hh} \mathbf{h}_{t-1} + \mathbf{W}_{xh} \mathbf{x}_t + \mathbf{b}_h)$$ $$\hat{\mathbf{y}}_t = \text{Softmax}(\mathbf{W}_{hy} \mathbf{h}_t + \mathbf{b}_y)$$

Types of RNN Sequences:

One-to-One: Standard feedforward classification (non-sequential).
One-to-Many: Image Captioning (single image input $\rightarrow$ sequence of words).
Many-to-One: Sentiment Analysis / Text Classification (sequence of words $\rightarrow$ single sentiment class).
Many-to-Many (Synced): Video Frame Classification / Named Entity Recognition (NER).
Many-to-Many (Async / Seq2Seq): Machine Translation / Speech-to-Text (Encoder-Decoder model).

4.5 - 4.7 Training via BPTT and the Vanishing / Exploding Gradient Problem

Backpropagation Through Time (BPTT): To train an RNN, the recurrent loop is "unfolded" across all time steps $T$. The total loss is the sum of losses at each time step: $\mathcal{L} = \sum_{t=1}^T \mathcal{L}_t$. Gradients are accumulated across all unfolded time steps:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{W}_{hh}} = \sum_{t=1}^T \sum_{k=1}^t \frac{\partial \mathcal{L}_t}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k} \frac{\partial \mathbf{h}_k}{\partial \mathbf{W}_{hh}}$$

Vanishing / Exploding Gradients:

The Jacobian term $\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k} = \prod_{j=k+1}^t \frac{\partial \mathbf{h}_j}{\partial \mathbf{h}_{j-1}}$ involves multiplying weight matrix $\mathbf{W}_{hh}$ repeated $t-k$ times. If the largest eigenvalue of $\mathbf{W}_{hh} < 1$, gradients decay exponentially to zero (vanishing gradients), rendering standard RNNs incapable of learning long-term dependencies (>10 time steps). If eigenvalue $> 1$, gradients explode to infinity (mitigated by Gradient Clipping).

4.8 & 4.9 Long Short-Term Memory (LSTM), GRU, and Encoder-Decoder Architecture

LSTM (Hochreiter & Schmidhuber, 1997): Solves vanishing gradients using a linear Cell State ($\mathbf{C}_t$) regulated by 3 multiplicative gates:

Forget Gate ($\mathbf{f}_t$): Decides what information to discard from cell state: $\mathbf{f}_t = \sigma(\mathbf{W}_f [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f)$.
Input Gate ($\mathbf{i}_t, \tilde{\mathbf{C}}_t$): Decides what new information to store: $\mathbf{i}_t = \sigma(\mathbf{W}_i [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i)$, $\tilde{\mathbf{C}}_t = \tanh(\mathbf{W}_c [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c)$.
Cell State Update ($\mathbf{C}_t$): $\mathbf{C}_t = \mathbf{f}_t \odot \mathbf{C}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{C}}_t$ (Constant error carousel allows gradients to flow linearly without decay).
Output Gate ($\mathbf{o}_t, \mathbf{h}_t$): $\mathbf{o}_t = \sigma(\mathbf{W}_o [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o)$ and $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{C}_t)$.

Gated Recurrent Unit (GRU): Merges cell state and hidden state, using only 2 gates: Reset Gate and Update Gate, offering faster training with fewer parameters.

Encoder-Decoder (Seq2Seq): An Encoder RNN compresses an input sequence into a fixed-length Context Vector, which a Decoder RNN unrolls into the target language sequence.

🔑 Key Concepts & Examination Keywords

Quick Terminology

Hidden State

A recurrent vector capturing memory of past sequence inputs up to the current time step.

Backpropagation Through Time (BPTT)

The optimization algorithm that unrolls an RNN across temporal time steps to compute cumulative gradients.

Constant Error Carousel (CEC)

The additive linear cell state update path in LSTM that prevents vanishing gradients over long sequences.

Seq2Seq Model

An architecture comprising an Encoder network that creates a latent thought vector and a Decoder network generating target sequences.

🎯 High-Yield Important Examination Questions

8–10 Descriptive Points Each

Q1. Explain the architectural design, mathematical working, and gate equations of Long Short-Term Memory (LSTM) networks in detail.

10 MarksLSTMCore Architecture

▼

📝 Detailed Examination Answer (10-Point Model):

Core Problem Addressed by LSTM: Standard RNNs fail to preserve dependencies over long sequences due to exponential gradient decay during backpropagation through time.
Dual State Architecture: LSTMs maintain two distinct recurrent states: the Cell State ($C_t$, long-term highway) and the Hidden State ($h_t$, short-term working memory).
Forget Gate Equation & Role: Calculates $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$ using Sigmoid; outputs near 0 discard past information, while outputs near 1 retain it.
Input Gate Equation & Candidate Memory: Calculates $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$ to scale candidate vector $\tilde{C}_t = \tanh(W_c [h_{t-1}, x_t] + b_c)$ of prospective new information.
Linear Cell State Update: Cell state updates via additive operation: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$, providing a linear gradient superhighway.
Output Gate Equation & Hidden State Calculation: Calculates $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$ and filters squashed cell state: $h_t = o_t \odot \tanh(C_t)$.
Resolution of Vanishing Gradient: Because $\frac{\partial C_t}{\partial C_{t-1}} = f_t$, when forget gate is active ($f_t \approx 1$), gradient flows over hundreds of time steps without decaying.
Multiplicative Gating Mechanism: Element-wise multiplication ($\odot$) controls information passage dynamically based on current context.
Parameter Complexity: Each LSTM cell contains 4 distinct weight matrix sets ($W_f, W_i, W_c, W_o$), requiring $4 \times (d_h(d_h + d_x) + d_h)$ parameters.
Applications & Industry Relevance: Powers industrial speech recognition, multi-lingual machine translation, financial forecasting, and musical generation.

Q2. Explain the training process of Recurrent Neural Networks using Backpropagation Through Time (BPTT) and why the Vanishing Gradient problem occurs.

10 MarksBPTT & Vanishing Gradients

▼

📝 Detailed Examination Answer (10-Point Model):

Temporal Unfolding Concept: An RNN is conceptually unrolled into a feedforward graph of $T$ sequential stages, sharing identical weight matrices ($W_{xh}, W_{hh}, W_{hy}$) across all steps.
Forward Sequence Propagation: At each time step $t$, $h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$ and $\hat{y}_t = \text{Softmax}(W_{hy} h_t + b_y)$ are evaluated sequentially.
Total Sequence Loss Aggregation: Global loss is the cumulative sum of step-wise cross-entropy errors: $L_{total} = \sum_{t=1}^T L_t(\hat{y}_t, y_t)$.
Chain Rule Across Time Steps: Computing $\frac{\partial L}{\partial W_{hh}}$ requires summing partial derivatives across all historical time steps $k \le t$: $\frac{\partial L_t}{\partial W_{hh}} = \sum_{k=1}^t \frac{\partial L_t}{\partial h_t} \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial W_{hh}}$.
Temporal Jacobian Product Chain: The term $\frac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^t \frac{\partial h_j}{\partial h_{j-1}} = \prod_{j=k+1}^t W_{hh}^T \text{diag}(1 - \tanh^2(z_j))$ involves repeated matrix multiplications.
Mathematical Origin of Vanishing Gradients: Since $\tanh'(z) \le 1$ and if the spectral radius of $W_{hh} < 1$, the continuous product approaches zero exponentially fast as $t - k > 10$.
Consequence on Long-Term Dependencies: Early historical inputs receive virtually zero error gradient, making the network blind to contextual relationships in distant past tokens.
Exploding Gradient Phenomenon: Conversely, if spectral radius of $W_{hh} > 1$, gradient magnitudes grow exponentially, causing NaN weights and catastrophic training divergence.
Gradient Clipping Solution: Exploding gradients are effectively neutralized by thresholding gradient norm: $g \leftarrow g \cdot \frac{\text{threshold}}{\|g\|}$ if $\|g\| > \text{threshold}$.
Architectural Solutions (LSTM / GRU): Vanishing gradient is definitively resolved by switching from standard RNN units to gated cells (LSTM/GRU) with linear cell states.

Q3. Describe the Sequence-to-Sequence (Seq2Seq) Encoder-Decoder architecture, its working mechanism, and its application in Neural Machine Translation.

10 MarksSeq2Seq & Applications

▼

📝 Detailed Examination Answer (10-Point Model):

Seq2Seq Architectural Motivation: Standard RNNs require input and output sequences to have identical lengths; Seq2Seq maps variable-length input sequences to variable-length output sequences.
Encoder Sub-Network Function: The Encoder RNN processes input tokens $x_1, x_2, \dots, x_{T_x}$ sequentially, updating its hidden state at each time step until reaching the end-of-sentence () token.
Context Vector (Thought Vector): The final hidden state of the Encoder $h_{T_x}$ summarizes the semantic meaning of the entire source sentence into a single fixed-size embedding vector.
Decoder Initialization: The Context Vector is passed as the initial hidden state $s_0$ to the Decoder RNN network.
Autoregressive Decoding Process: The Decoder generates target tokens sequentially: at step $t$, it predicts output word $y_t$ given previous output $y_{t-1}$ and current hidden state $s_t$.
Teacher Forcing Technique: During training, ground truth target tokens are fed as inputs to the Decoder at the next step rather than model predictions, stabilizing and accelerating convergence.
Inference Beam Search: During testing/inference, Beam Search explores the top $K$ most probable sequence hypotheses simultaneously rather than greedy token selection.
Information Bottleneck Limitation: Compressing long sentences (e.g., >30 words) into a single fixed-length context vector causes information loss and poor translation quality for long passages.
Attention Mechanism Enhancement: Attention allows the Decoder to dynamically look back and take a weighted average of all intermediate Encoder hidden states at each decoding step.
Industrial NLP Use Cases: Powers Google Translate, automated code generation, text summarization, and speech-to-text transcription systems.

⚖️ Comprehensive Comparison & Difference Tables

8+ Comparison Criteria

📊 Standard RNN vs LSTM vs Gated Recurrent Unit (GRU)

Comparison Parameter	Standard RNN	LSTM (Long Short-Term Memory)	GRU (Gated Recurrent Unit)
Internal States	Single Hidden State ($h_t$).	Dual States: Cell State ($C_t$) & Hidden State ($h_t$).	Single Hidden State ($h_t$).
Gating Mechanism	No gates; only simple tanh non-linearity.	3 Gates: Forget Gate, Input Gate, Output Gate.	2 Gates: Reset Gate and Update Gate.
Cell State Update	Overwrites hidden state via non-linear tanh matrix product.	Linear additive cell state update ($f_t C_{t-1} + i_t \tilde{C}_t$).	Linearly interpolates previous state using update gate $z_t$.
Vanishing Gradient	Severely suffers from vanishing/exploding gradients.	Effectively eliminates vanishing gradients via linear cell state.	Eliminates vanishing gradients with simpler architecture.
Parameter Count	Lowest parameter count ($1\times$ baseline).	High parameter count ($4\times$ baseline parameters).	Moderate parameter count ($3\times$ baseline, ~25% fewer than LSTM).
Training Speed	Fastest per epoch but fails to learn long sequences.	Slower per epoch due to 4 gate matrix computations.	Faster to train than LSTM due to fewer gate operations.
Memory Capacity	Fails on sequences longer than ~10 steps.	Maintains dependencies across hundreds of time steps.	Maintains long-term dependencies comparable to LSTM.
Overfitting Risk	Low parameter capacity but high training instability.	Higher risk of overfitting on smaller datasets.	Slightly lower overfitting risk on smaller datasets compared to LSTM.

📊 Feedforward Neural Network vs Recurrent Neural Network

Comparison Parameter	Feedforward Neural Network (FNN)	Recurrent Neural Network (RNN)
Data Assumption	Assumes all input samples are independent and identically distributed (i.i.d.).	Assumes sequential/temporal dependency between consecutive time steps.
Memory / Feedback Loops	No internal memory; signals move strictly forward in one direction.	Contains cyclic recurrent feedback loops that store past memory.
Input/Output Dimensions	Requires fixed-dimension input vectors and produces fixed outputs.	Handles variable-length input sequences and variable-length output sequences.
Weight Sharing Over Time	Weights are layer-specific and not shared across temporal sequences.	The same weight matrices ($W_{hh}, W_{xh}, W_{hy}$) are reused at every time step.
Training Algorithm	Standard Backpropagation with Gradient Descent.	Backpropagation Through Time (BPTT) with temporal unfolding.
Temporal Order Sensitivity	Indifferent to order; shuffling input features does not affect MLP learning.	Critically sensitive to sequence order; shuffling tokens alters semantic meaning.
Computation Execution	Can process entire batch in a single parallel feedforward matrix pass.	Inherently sequential; time step $t$ strictly depends on result of $t-1$.
Primary Domains	Tabular classification, static pattern recognition, regression.	Natural language processing, time series forecasting, speech recognition, music synthesis.

⚡ Quick Pre-Exam Revision Summary

5-Minute Recap

💡 Core Takeaways & High-Yield Summary

RNNs maintain internal hidden state $h_t = anh(W_{hh} h_{t-1} + W_{xh} x_t + b)$ to process sequential dependencies.
Backpropagation Through Time (BPTT) unrolls the network across time steps $T$ to compute gradients.
Vanishing gradients in standard RNNs occur due to repeated multiplication of weight matrices across time steps.
LSTM uses Forget ($f_t$), Input ($i_t$), and Output ($o_t$) gates with a linear Cell State ($C_t$) to preserve long-term memory.
GRU simplifies LSTM by combining cell and hidden states using Update and Reset gates.
Seq2Seq Encoder-Decoder models transform variable-length input sequences into variable-length output sequences.