INFO 557: Neural Networks

Dr. Greg Chism

This page contains an outline of the topics, content, and assignments for the semester. Note that this schedule will be updated as the semester progresses and the timeline of topics and assignments might be updated throughout the semester.

Week Date Topic Readings Due
1 Mon, Aug 25 🎥 0.0 Neurons
🎥 0.0 Neural Networks		 🎥 Instructor Introduction
📃 Codespaces
📃 Course Tech
📚 Understanding DL - Chps 1-3		 ✅ 0.0 Quiz
🎙️ 0.0 Discussion

Fri, Aug 29 🎥 6.1 Activation Functions
🎥 6.1 Broadcasting
🎥 6.1 XOR Problem
🎥 6.1 Drawing NNs		 📚 DL Book - Chp 6 Intro
📚 DL Book - Chp 6.1		 ⌨️ 6.1 Activity
✅ 6.1 Quiz
🎙️ 6.1 Discussion
2 Mon, Sep 1 Labor Day (No Class)


Wed, Sep 3 🎥 6.2 Gradient Descent
🎥 6.2 Cost Functions		 📚 Understanding DL - Chp 5
📚 DL Book - Chp 6.2		 ⌨️ 6.2 Activity 1
✅ 6.2 Quiz 1
🎙️ 6.2 Discussion 1

Fri, Sep 5 🎥 6.2 Other Costs
🎥 6.2 Linear Regression
🎥 6.2 Sigmoid Outputs
🎥 6.2 Softmax Outputs
🎥 6.2 Softmax Units for Multiclass Classification
🎥 6.2 Other Output Units
⌨️ 6.2 Activity 2
✅ 6.2 Quiz 2
🎙️ 6.2 Discussion 2
⌨️ 6.2 Activity 3
3 Mon, Sep 8 🎥 6.3 Hidden Units
🎥 6.3 ReLU
🎥 6.3 Non Differentiable Activation Functions
🎥 6.3 Absolute Value Rectification		 📚 Understanding DL - Chp 6
📚 DL Book - Chp 6.3		 ✅ 6.3 Quiz
🎙️ 6.3 Discussion

Fri, Sep 12 🎥 6.3 Leaky ReLU
🎥 6.3 Smooth ReLUs
🎥 6.3 Maxout Units
⌨️ 6.3 Activity
4 Mon, Sep 15 🎥 6.3 Sigmoid and Tanh
🎥 6.3 Linear Units
🎥 6.3 Less Common Units
🎥 Activation vs. Cost


Wed, Sep 17

💻 Program 1

Fri, Sep 19 🎥 6.4 Universal Approximation
🎥 6.4 Other Considerations		 📚 DL Book - Chp 6.4 ✅ 6.4 Quiz
🎙️ 6.4 Discussion
⌨️ 6.4 Activity
5 Mon, Sep 22 🎥 6.5 Computational Graphs
🎥 6.5 Chain Rule
🎥 6.5 Forward Propagation
🎥 6.5 Back propogation
🎥 6.5 Software Implementation		 📚 Understanding DL
📚 DL Book - Chp 6.5		 ✅ 6.5 Quiz 1
🎙️ 6.5 Discussion 1
⌨️ 6.5 Activity 1
✅ 6.5 Quiz 2
🎙️ 6.5 Discussion 2
⌨️ 6.5 Activity 2

Fri, Sep 26 🎥 7.1 Evaluating Neural Networks
🎥 7.1 Learning Curves
🎥 7.1 L1 + L2 Regularization		 📚 DL Book - Chp 7 Intro
📚 DL Book - Chp 7.1		 ✅ 7.1 Quiz
🎙️ 7.1 Discussion
⌨️ 7.1 Activity
6 Mon, Sep 29 🎥 7.4 Generating Training Data
🎥 7.8 Early Stopping
🎥 7.8 Early Stopping as Regularization		 📚 DL Book - Chp 7.4
📚 DL Book - Chp 7.8		 ✅ 7.8 Quiz
🎙️ 7.8 Discussion
⌨️ 7.8 Activity

Fri, Oct 3 🎥 7.12 Dropout Training
🎥 7.12 Dropout Prediction
🎥 7.12 Dropout Considerations
🎥 7.KD Why Distillation
🎥 7.KD Knowledge Distillation
🎥 7.KD Distillation as Regularization		 📚 DL Book - Chp 7.12
📄 Knowledge Distillation		 ✅ 7.12 Quiz
🎙️ 7.12 Discussion
⌨️ 7.12 Activity
✅ 7.KD Quiz
🎙️ 7.KD Discussion
⌨️ 7.KD Activity
7 Mon, Oct 6 🎥 8.1 Learning vs. Pure Optimization
🎥 8.1 Estimating Gradient Steps
🎥 8.1 SGD and Generalization Error		 📚 DL Book - Chp 8 Intro
📚 DL Book - Chp 8.1		 ✅ 8.1 Quiz
🎙️ 8.1 Discussion
⌨️ 8.1 Activity

Wed, Oct 8

💻 Program 2

Fri, Oct 10 🎥 8.2 Ill Conditioning
🎥 8.2 Local Minima
🎥 8.2 Saddles
🎥 8.2 Vanishing and Exploding Gradients		 📚 DL Book - Chp 8.2 ✅ 8.2 Quiz
🎙️ 8.2 Discussion
⌨️ 8.2 Activity
8 Mon, Oct 13 🎥 8.3 Stochastic Gradient Descent
🎥 8.3 Momentum		 📚 DL Book - Chp 8.3 ✅ 8.3 Quiz
🎙️ 8.3 Discussion
⌨️ 8.3 Activity


✏️ Midterm is Live


Fri, Oct 17 🎥 8.4 Parameter Initialization
🎥 8.4 Random Initialization
🎥 8.4 Debugging Initialization Problems
🎥 8.4 Bias Initialization
🎥 8.4 Pre-Training Models		 📚 DL Book - Chp 8.4 ✅ 8.4 Quiz
🎙️ 8.4 Discussion
⌨️ 8.4 Activity
9 Mon, Oct 20 🎥 8.5 AdaGrad
🎥 8.5 RMSProp
🎥 8.5 Adam
🎥 8.5 Selecting an Algorithm		 📚 DL Book - Chp 8.5 ✅ 8.5 Quiz
🎙️ 8.5 Discussion
⌨️ 8.5 Activity

Fri, Oct 24 🎥 8.7.1 Batch Normalization
🎥 8.7.1 Smoother Loss
🎥 8.7.1 Batch Normalization in Practice		 📚 DL Book - Chp 8.7.1 ✅ 8.7.1 Quiz
🎙️ 8.7.1 Discussion
⌨️ 8.7.1 Activity
10 Mon, Oct 27 🎥 9.1 Definition of Convolution
🎥 9.1 Cross Correlation
🎥 9.1 Convolution Properties		 📚 DL Book - Chp 9 Intro
📚 DL Book - Chp 9.1
📚 DL Book - Chp 9.2		 ✅ 9.1 + 9.2 Quiz
🎙️ 9.1 + 9.2 Discussion
⌨️ 9.1 Activity

Fri, Oct 31 🎥 9.3 Pooling Basics
🎥 9.3 Types of Pooling
🎥 9.3 Translation Invariance
🎥 9.3 Convolution Properties		 📚 DL Book - Chp 9.3 ✅ 9.3 Quiz
🎙️ 9.3 Discussion
11 Mon, Nov 3 🎥 9.4 Convolution and Pooling as Priors
🎥 9.4 Convolution for Language
🎥 9.4 Convolution for Images
🎥 9.GC Graph Convolution		 📚 DL Book - Chp 9.4
📄 How GNNs Work		 ⌨️ 9.4 Activity
✅ 9.GC Quiz
🎙️ 9.GC Discussion
⌨️ 9.GC Activity

Wed, Nov 5

💻 Program 3

Fri, Nov 7 🎥 10.1 Feedforward vs. Variable Other
🎥 10.1 Convolution vs. Long Distance
🎥 10.1 Recurrent Network Computational Graph		 📚 DL Book - Chp 10
📚 DL Book - Chp 10.1		 ⌨️ 10.1 Activity 1
✅ 10.1 Quiz
🎙️ 10.1 Discussion
⌨️ 10.1 Activity 2
12 Mon, Nov 10 🎥 10.2 One Output per Input
🎥 10.2 Output to Hidden Connections
🎥 10.2 One Output at End
🎥 10.2 Back-propagation Through Time		 📚 DL Book - Chp 10.2 ⌨️ 10.2 Activity 1
✅ 10.2 Quiz 1
🎙️ 10.2 Discussion 1
⌨️ 10.2 Activity 2
✅ 10.2 Quiz 2
🎙️ 10.2 Discussion 2

Wed, Nov 12

✏️ Midterm - Diagnostic Stage
✏️ Midterm - Output Prediction
✏️ Midterm - Training + Regularization
✏️ Midterm - Computational Graph

Fri, Nov 14 🎥 10.3 Bidirectional RNNs
🎥 10.3 Bidirectional RNN Merging
🎥 10.3 Bidirectional RNN Properties
🎥 10.4 Sequence to Sequence Networks
🎥 10.4 Machine Translation
🎥 10.4 Sequence to Sequence Considerations		 📚 DL Book - Chp 10.3
📚 DL Book - Chp 10.4		 ✅ 10.3 Quiz
🎙️ 10.3 Discussion
⌨️ 10.3 Activity
✅ 10.4 Quiz
🎙️ 10.4 Discussion
⌨️ 10.4 Activity
13 Mon, Nov 17 🎥 10.6 Recursive Neural Networks
🎥 10.6 Constructing Syntactic Trees
🎥 10.6 Classifying Sentiment
🎥 10.7 Composing Functions
🎥 10.7 RNN Challenges		 📚 DL Book - Chp 10.6
📚 DL Book - Chp 10.7		 ⌨️ 10.6 Activity
✅ 10.7 Quiz
🎙️ 10.7 Discussion
⌨️ 10.7 Activity

Fri, Nov 21 🎥 10.9 Skip Connections
🎥 10.9 Leaky Units
🎥 10.10 LSTM
🎥 10.10 GRU		 📚 DL Book - Chp 10.9
📚 DL Book - Chp 10.10		 ⌨️ 10.10 Activity
✅ 10.10 Quiz
🎙️ 10.10 Discussion
14 Mon, Nov 24 🎥 10.TF Query Keys Values
🎥 10.TF Self-Attention
🎥 10.TF Multi-Head Attention
🎥 10.TF Layer Normalization
🎥 10.TF Transformer Properties		 📄 Illustrated Transformer ⌨️ 10.TF Activity
✅ 10.TF Quiz
🎙️ 10.TF Discussion

Fri, Nov 28 Happy Thanksgiving 🦃

15 Mon, Dec 1 🎥 11.1 Design Process
🎥 11.1 Performance Metrics
🎥 11.2 First Model
🎥 11.3 Analyzing Low Performance
🎥 11.4 Hyperparameter Search
🎥 11.4 Hyperparameters and Model Capacity
🎥 11.4 Automatic Hyperparameter Optimization		 📚 DL Book - Chp 11 Intro
📚 DL Book - Chp 11.1
📚 DL Book - Chp 11.2
📚 DL Book - Chp 11.3
📚 DL Book - Chp 11.4
⌨️ 11.1 Activity
✅ 11.2 Quiz
🎙️ 11.2 Discussion
⌨️ 11.3 Activity
⌨️ 11.4 Activity
✅ 11.4 Quiz
🎙️ 11.4 Discussion

Wed, Dec 3 ✏️ Final Exam is Live
💻 Program 4

Fri, Dec 5 🎥 11.5 Debugging Machine Learning Code
🎥 11.5 Visualize Predictions
🎥 11.5 Reason about Error
🎥 11.5 Fit a Tiny Dataset
🎥 11.5 Monitor Activations and Gradients
🎥 11.6 Street View Transcription		 📚 DL Book - Chp 11.5
📚 DL Book - Chp 11.6 Ex		 ⌨️ 11.4 Activity
✅ 11.4 Quiz
🎙️ 11.4 Discussion
16 Mon, Dec 8 🎥 11.LS Local Surrogate Models
🎥 11.LS Neighborhoods in Tabular Data
🎥 11.LS Neighborhoodsin Text
🎥 11.LS Neighborhoodsin Images
🎥 11.LS LIME Properties		 📄 Interpretable ML 9.2 ⌨️ 11.LS Activity
✅ 11.LS Quiz
🎙️ 11.LS Discussion

Wed, Dec 10 🎥 11.GA Gradient Attribution
🎥 11.GA Gradient Attribution Variations
🎥 11.GA Gradient Attribution Visualization
🎥 11.GA Gradient Attribution Demo		 📄 Interpretable ML 10.2 ⌨️ 11.GA Activity
✅ 11.GA Quiz
🎙️ 11.GA Discussion

Fri, Dec 12 No Classes

17 Mon, Dec 15



Wed, Dec 17

✏️ Final - The Trial of Truth
✏️ Final - The Chamber of Meaning
✏️ Final - The Hall of Decisions
✏️ Final - The Puzzle Forge
✏️ Final - The Final Trials of Insight