Skip to main content

Ctrl+K

Introduction

Getting Started

Holy Trinity for Machine Learning
- Data
- Model
- Loss Function
Approximation models
Gradients
- Do loss functions have to be differentiable?
- How Gradients Are Calculated?

Common Tasks

Types of tasks
Regression
- Auto Regression
Classification

Common Building Blocks

Layers
Linear Layer
- Calculate gradients for Linear Layers
Convolution Layer
Recurrent Layer
- Long Short Term Memory
- Gated Linear Unit
Embedding Layer
Dropout Layer
Normalization Layer
Padding Layer
Pooling Layer
Transformer Block
Activation Functions

Some Other Important Things To Notice

Other Things To Notice
Batch size
Gradient Vanishing / Gradient Explosion
Saddle point
Learning Rate
Optimizer
Overfit
Underfit

Generative Models

Generative Models
AutoEncoder Model
Generative Adversarial Models
Gaussian Mixture Model

Improving Models

Improvements to a model
Explainable AI
- Saliency Maps
Meta Learning
Life Long Learning
Model Compression

Reuse Existing Models

Reusing Existing Models
Transfer Learning and Domain Adaptation
- Transfer Learning vs Domain Adaptation
Knowledge Distillation

Beyond Supervised Training

Unsupervised Learning
Clustering
Decision Tree
Self Supervised Learning
Semi Supervised Training

Reinforcement Learning

Reinforcement Learning
Value
- Q Learning
Policy
- Policy Gradient
Actor Critic

Repository
Suggest edit
Open issue

.ipynb

Training Transformers

Training Transformers#

previous

Transformer vs RNN

next

Teacher Forcing vs Scheduled Sampling vs Normal Mode

By RenChu Wang

© Copyright RenChu Wang, 2021.