Elements of Deep Learning
Teaching
- 28 lectures
- 2 paper reading sessions (student presentations)
- [groups of upto 4]
- Tentative lecture plan:
Why Deep Learning?
- Data and compute availability in large quantities
- Hard to define features. Need to be automatically learnt.
- DL allows a hierarchical feature composition to handle complexity
- Real life data often has the same property. Think images, text, sound, etc.
Hierarchical Feature Composition
Source: link
The process
- The data that we can learn from.
- A model of how to transform the data.
- An objective function that quantifies how well (or badly) the model is doing.
- An algorithm to adjust the model’s parameters to optimize the objective function.
Kinds of (Supervised) Machine Learning Problems
- Classification
- Binary, Multiclass, Hierarchical
- Tagging
- Multilabel classification, e.g., topics in a forum or categories of clothes
- Regression
- Search
- How to rank pages as per the relevance for the query (original PageRank from Google didn’t do this!)
- Sequence Learning
- Recommender systems
- Will this user like this item? (Classification)
- Rating prediction for the user (Regression)
d2l.ai
Generality of log likelihood
Why CNN?
- Increasing receptive field
![]()
Source: UDLBook
Much faster convergence due to reduced increased model bias
Convolution Effectiveness
(Down and Up) Sampling
Down and up-sampling (UDLbook)
Transposed convolutions (upsampling)
The DL revival?
- What started the DL revival?
DL Meme
The Imagenet dataset
Imagenet dataset considerations
Alexnet
AlexNet
What progress looks like
ImageNet performance
Perception of images
- Classification
- Object Detection
- Semantic Segmentation
- Pose
- Dense Pose
Perception of images
- Monocular depth prediction
- Stereo depth prediction
- Anomaly detection
Depth Prediction
Creation of images
- Conditioned on text prompt/other images
- High resolution from low resolution - forensics, satellite, movies, etc.
Midjourney creations
Modifying images
Style Transfer
Modifying images - contd (LEDITS++)
Potter
Shattered Gradients
- Increasingly complex way of calculating gradients in deeper networks
![]()
![Shattered Gradients]()
Saviors: Residual connections + batch normalization
- Intuitively, allows for different complexities in the learned concepts
Residual connections
Expressive functions
Expressive ability
Loss surface with skip connections
On CIFAR-10
Numeric control
- Control activations (forward pass) & gradients (backward pass)
- (keep same expected variance)
- He initialization: \(\beta\) initialized to 0, \(\omega\) initialized to normal distribution - (0, \(\frac{2}{D_h}\)) where \(D_h\) is the number of hidden neurons in the previous layer
- Residual : might still explode with increasing variance throughout the layers. Helps establish a numerical range after every layer.
- Batch Normalization
![]()
Step1: Normalization ![]()
Step2: Scale & Shift - \(m_h\) and \(s_h\) at test time?: calculated from the entire training dataset
- Batch Normalization
Bottleneck residuals
- Limiting the number of parameters
Residual blocks
Image Classification with residual blocks
DenseNet
- either concatenate or do a 1x1 convolution to add a weighted combination
- comparable performance as compared to resnet
![Densenet architecture]()
U-net
- Bring dexterity with semantics
![Architecture]()
Semantic Segmentation examples
Semantic Segmentation
Pose Prediction Example (Hourglass)
Different types of normalization
- Note: Input is 1D
Normalization Types
[End of UDL References]
Other influential advances: NiN (Network in Network)
- purpose: reduce computation
- Same architecture as alexnet except with a max pool of 3x3 window
- output layers = number of labels followed by 1x1 convolution + average pooling instead of FC layer
- slower training
ref: D2L.ai
GoogLeNet (2014 ImageNet Winner)
- First clear distinction between stem (data ingest), body (processing), head (prediction)
ResNext
ResNext
- Grouped Convolutions is \(g\times\) faster than a dense convolution
- The 1x1 convolutions allows for information sharing between groups
Image Augmentations
- Flips
- Crops/Resize/Rescale
- Color Augmentations
Augmentations
Single Shot Multibox Detector (2016)
- Multibox detector -> Assigns ground truth boxes to anchor boxes
- Loss is a weighted sum of confidence and localization loss. N = number of matched boxes.
SSD - Loss functions
- How to handle class imbalance (more negative boxes than positives)?
- Sample from negative (take only top 3 most confident boxes)
SSD Vs YOLO
YOLO Comparison
Retina Net
RetinaNet
CornerNet
Proposal based methods
Deformable convolutions
Deformable Convolutions
CentripetalNet
CentripetalNet
Semantic Segmentation: Capturing multi-scale context (from deeplabv3)
Capturing multi-scale context
Side: Spatial pooling pyramids
Blending with laplacian pyramids
Learning similar embeddings
- Face Recognition
- Object Reidentification also uses a triplet loss framework after making object level representations
- Note how reidentification and detection are conflicting objectives?
ReID Task
FairMOT
FairMOT
Cosine Similarity
- Take a q-dimensional output -> L2 Normalization -> Use cosine similarity
Fine grained/Hierarchical classification
- How to highlight distinctions?
- Part based embeddings
Adaptive Style Transfer
- Adaptive instance normalization allows the activation distribution to match a style activation distribution
Example outputs (Adaptive Style Transfer)
output
GANs (Generative Adversarial Networks)
Common issues in GANs
- Mode Collapse
![Mode Collapse]()
- Vanishing gradients
- Discriminator performs better than generator; lack of challenging examples for discriminator -> vanishing gradient
- Convergence
![Convergence issues in GANs]()
DCGAN
- Deep Convolutional Generative Adversarial Networks
- both generator and discriminator are convolutions/transposed-convolutions
- wasserstein loss functions (critique instead of discriminator) for handling the common issues
Conditional GANs
Conditioning on label/other modalities of data
Style GAN
- Style Mixing (Mixing regularization)
- Stochastic variation by introducing noise
- Separation of global effects from stochasticity
- focuses on generating effects indistinguishable by discriminator
Example outputs
Autoencoders
![]()
Simple autoencoder
AnoGAN
GANomaly
Masked autoencoders
Recurrent Neural Networks
- Ordered list of features \(x_1, .. x_T\) where \(x_t\) is the \(t^{th}\) time step in the sequence
- Sequence independence, not feature independence
- Aligned (e.g., POS tagging) and unaligned predictions (translation, sentiment)
Auto-regressive models
- Quantity of interest: \(P(x_t | x_1, x_2 .. x_{t-1})\)
- Notice varying sequence length after every time step –> New model after every timestep?
Stock Value over years
Example - Markov model: input
- Markov Assumption
- \(P(x_1, \ldots, x_T) = P(x_1) \prod_{t=2}^T P(x_t \mid x_{t-1})\)
Input data
Model: k-step ahead prediction
1-step ahead
k-step ahead
Token Frequency
Zipf’s Law: \(n_i \propto \frac{1}{i^\alpha}\)
What do the top ten words tell you?
- Word frequencies
[('the', 2261),
('i', 1267),
('and', 1245),
('of', 1155),
('a', 816),
('to', 695),
('was', 552),
('in', 541),
('that', 443),
('my', 440)]- Bigrams
[('of--the', 309),
('in--the', 169),
('i--had', 130),
('i--was', 112),
('and--the', 109),
('the--time', 102),
('it--was', 99),
('to--the', 85),
('as--i', 78),
('of--a', 73)]- Trigrams
[('the--time--traveller', 59),
('the--time--machine', 30),
('the--medical--man', 24),
('it--seemed--to', 16),
('it--was--a', 15),
('here--and--there', 15),
('seemed--to--me', 14),
('i--did--not', 14),
('i--saw--the', 13),
('i--began--to', 13)]
- Indicates structure
Training the network
- Fixed length neural network based language model
I/O
- skip \(d\) random tokens at the start to get random subsequences
- m partitioned subsequences: \(\mathbf x_d, \mathbf x_{d+n}, \ldots, \mathbf x_{d+n(m-1)}\) where \(m = \frac{T-d}{n}\)
Recurrent Neural Networks
- Refer notes
Cell Architecture
Character level RNN-based language model
BPTT
- Refer notes
Truncation
- Comparing strategies for computing gradients in RNNs. From top to bottom:
- randomized truncation
- regular truncation
- full computation
LSTM
LSTM
GRU
GRU
Deep RNNs
Deep Recurrent Neural Network
Other tasks - fill in the blanks
- I am ___.
- I am ___ hungry.
- I am ___ hungry, and I can eat half a pig.
Class feedback form
- Anonymous
- Will be done after every class. Also, open after class but please refer to a specific class while giving feedback.
- Link
![QR Code for class feedback]()
Use cases - Encoder Decoder
- Teacher forcing
- Loss?
Test time - encoder decoder
- Beam Search
Building applications in NLP
Structure
Skip-gram
- \(P(\textrm{"the"},\textrm{"man"},\textrm{"his"},\textrm{"son"}\mid\textrm{"loves"})\)
- \(P(\textrm{"the"}\mid\textrm{"loves"})\cdot P(\textrm{"man"}\mid\textrm{"loves"})\cdot P(\textrm{"his"}\mid\textrm{"loves"})\cdot P(\textrm{"son"}\mid\textrm{"loves"})\)
probability
Model probability with softmax, \(P(w_o \mid w_c) = \frac{\exp(\mathbf{u}_o^\top \mathbf{v}_c)}{ \sum_{i \in \mathbb{V}} \exp(\mathbf{u}_i^\top \mathbf{v}_c)}\)
For multiple context words, \(\prod_{t=1}^{T} \prod_{-m \leq j \leq m,\ j \neq 0} P(w^{(t+j)} \mid w^{(t)})\)
POS Tagging
Summarization
- Extractive
- Abstractive
Extractive summarization through ranking (TextRank)
![]()
Abstractive Summarization
Sentiment Classification
Sentiment Classification
Image Captioning
Image Captioning
Visual QnA
Visual QnA
What is a graph?
The graphs of our lives
- Nodes and vertices
- Typically sparse
- More examples of graphs - social network, computer programs, protein interactions, scientific literature, geometric point clouds, images
Examples of graphs
Examples
- Which of these can you process with DL?
Graphical musketeers
Source: UDLBook
Properties of adjacency matrix
Source: UDLbook
Approach - Graph Task
- For individual nodes, understand the “meaning & context” of the nodes
- Take \(X\) and \(A\) as input –> Pass through \(K\) layers –> \(H_k \forall k \in [1, K]\) hidden representations
- For the graph task,
- Average all node embeddings (also called mean pooling)
- Follow it up with a linear layer/MLP
- Use BCE for classification/MSE for regression
![]()
Classification
Approach - Node Task
- For example, predict the part of the plane using the point cloud node
- Node embeddings are used as input to classifier/regression stage directly
- Same \(\beta_k\) and \(\omega_k\) are used across nodes
Per node optimization
Approach - Edge Prediction Task
- For example, “friend” suggestions
- A similarity metric for two nearby nodes to predict if an edge should ‘indeed’ exist
- Dot product for similarity
Graph tasks
Requirements of equivariance and invariance
- Equivariance/Covariance, e.g., image segmentation
- f[t[x]] = t[f[x]]
- Invariance, e.g., classification of a flip
- f[t[x]] = f[x]
- For graph tasks, mean pooling is ________ to permutation matrix \(P\)
![]()
Graph Convolutions
- The \(K\) layers induce a relational inductive bias
- e.g., predict plane part using nodes of point cloud
GCN
The convolution in GCN
- Convolution: Aggregates information from neighbors
- Message passing from neighbors
- Fixed weights according to the location of neighbor in a CNN. What about graphs?
- Uniformly weighted combination of all neighbors
![]()
![]()
- Writing combined for all nodes
![]()
Check design considerations
- equivariant to permutations of node indices
![]()
- can cope with any number of neighbors
![]()
- exploits graph structure to provide relational inductive bias
![]()
- shares weights across nodes
![]()
Example: Drug classification
- 118 elements in the periodic table, N = number of nodes in graph
- \(X \in R^{118\times N}\), \(A \in R^{N \times N}\), \(\Omega_0 \in R^{D \times 118}\)
Inductive vs transductive
- Supervised vs semi-supervised (imagine labeling more nodes in the academic literature graph)
- Graph tasks –> inductive
- Node, Edge tasks –> inductive or transductive.
Node task example
- Transductive task on a large graph - label each node
- Impossible to fit the entire graph, i.e., subselect
- How to sample nodes?
![k-hop sampling]()
Smarter subsampling
- Neighborhood sampling (randomly subsample from hops)
- Think dropout
- Graph partitioning
- Think clustering graph nodes to simplify graphs or
- Removing some edges which might be ‘weaker’
Edge graphs
- Transform into a node graph and use
Properties of attention-based querying
- Typically generates a linear sum over “relevant” values. Some special cases:
![]()
Attention Pooling Architecture
Scoring Function
- Dot Product Attention
![]()
- All keys are assumed to be zero mean, unit variance (normalization)
- Ideally, we’d like the dot products to have unit variance as well.
- Therefore, we normalize by \(1/\sqrt{d}\) where \(k \in R^d\)
\(\alpha(\mathbf{q}, \mathbf{k}_i) = \mathrm{softmax}(a(\mathbf{q}, \mathbf{k}_i)) = \frac{\exp(\mathbf{q}^\top \mathbf{k}_i / \sqrt{d})}{\sum_{j=1} \exp(\mathbf{q}^\top \mathbf{k}_j / \sqrt{d})}\)
Limitations of the RNN
- Too much is expected of the state
- In Bahdanau Attention, When predicting a token, if not all the input tokens are relevant, the model aligns (or attends) only to parts of the input sequence that are deemed relevant to the current prediction. This is then used to update the current state before generating the next token.
Introducing the relevance in decoding
Attention in sequences
Interpretability ++
Attention in Images
CNN + RNN + Attention
Attention in graphs
Multiple heads of attention
Mathematically, MHA
Self attention
- With multiple heads, the attention mechanism learns queries, keys and values as linear transformations of the input X itself. This is why it’s called Self Attention.
num_hiddens, num_heads = 100, 5
attention = d2l.MultiHeadAttention(num_hiddens, num_heads, 0.5)
batch_size, num_queries, valid_lens = 2, 4, torch.tensor([3, 2])
X = torch.ones((batch_size, num_queries, num_hiddens))
d2l.check_shape(attention(X, X, X, valid_lens),
(batch_size, num_queries, num_hiddens))Understanding positional encoding
- Encouraging unique encodings through decreasing frequencies and offsets
- Learning different frequency information for long sequences
Transformer
- An architecture without recurrent connections that can capture sequential as well as long range information
- Four elements
- Multi-head self attention (MHA)
- Feedforward Network (FFN)
- Residual connections
- Encoder-Decoder Attention
Vision Transformer
Swin Transformer
- Hierarchical feature maps for linear scaling with image size
![]()
![]()
Hierarchical Features & Shifted Windows
Hierarchical Features & Shifted WindowsGraph Transformer
Pros
- Long range connections
- Navigating the maze with the map
- Node update still unrelated to graph structure
Cons
- Identifiability of nodes (need better encodings)
- Loss of relational inductive bias
- Increased computational complexity
Graph Transformer detailed
Multimodal (Unified) transformers
- VQA: Visual Question Answering
- SNLI-VE: Stanford Natural Language Inference - Visual Entailment
- MNLI: Multi-Genre Natural Language Inference
- QNLI: Question Natural Language Inference
- QQP: Quora Question Pairs
- SST-2: Stanford Sentiment Treebank - 2
Architecture
The unification
Encoder-only
- BERT (Bidirectional Encoding Representations from Transformers)
- Masking words randomly
- output embedding is projected for classification/regression tasks
Fine-tune BERT
Encoder-decoder: Pre-training T5
- T5 = Text to text transfer transformer
- Includes the “task” in the encoder, e.g., summarize..
![]()
- Includes the “task” in the encoder, e.g., summarize..
Fine-tuning T5
Decoder only: GPT
Using GPT2 for different tasks without fine-tuning
Scalability of transformers
Sometimes, these things go beyond imagination
- OpenAI CLIP (Contrastive Language Image Pretraining)
- SSL on 400 million text-image pairs
- Beats robustness gap by 75%
- Beats Imagenet classification without using Imagenet labels (zero-shot)
Architecture
- Train time?
- 256 GPUs for 2 weeks
On satellite images
Related Work:
SatCLIP