Advanced Applied Deep Learning
Lecture Course
Sheng Yun Wu
Lecture Course
Sheng Yun Wu
Objective:
To introduce students to the concept of transfer learning and how pre-trained models can be fine-tuned for new tasks. Students will learn why transfer learning is useful for deep learning, especially when working with smaller datasets, and will get hands-on experience with pre-trained models such as VGG, ResNet, and Inception.
Lecture 1: Introduction to Transfer Learning
3.1 What is Transfer Learning?
Definition:
Transfer learning is the process of taking a pre-trained model that has been trained on one task and reusing it for a new task.
Why Use Transfer Learning?
Saves time and computational resources.
Prevents the need for massive datasets from scratch training.
Effective for tasks with limited data (e.g., medical imaging, small-scale image classification).
3.2 How Transfer Learning Works:
Feature Extraction vs. Fine-tuning:
Feature extraction: Use the pre-trained model as a feature extractor and only train the final classification layer.
Fine-tuning: Unfreeze some layers of the pre-trained model and retrain them along with the final layers for the new task.
Pre-trained Models:
Models like VGG, ResNet, Inception, DenseNet are pre-trained on large datasets like ImageNet.
These models have learned useful features (like edges, textures) that can be applied to new tasks.
3.3 Benefits and Challenges:
Advantages:
Accelerates the training process.
Often achieves better accuracy than training from scratch, especially on small datasets.
Challenges:
Overfitting on the new dataset.
Ensuring that the pre-trained model’s learned features are relevant to the new task.
Lecture 2: Popular Pre-trained Models (VGG, ResNet, Inception)
3.4 Overview of VGG Networks:
VGG16 and VGG19:
Known for their simple, uniform architecture with 3x3 convolutional filters.
Deep networks (16 or 19 layers) with small filter sizes and max pooling.
Advantages of VGG:
Easy to implement and modify.
Suitable for fine-tuning because of its regular and simple architecture.
Limitations:
VGG is computationally expensive due to the large number of parameters.
3.5 ResNet (Residual Networks):
Key Idea:
Introduces skip connections (residual connections) to solve the problem of vanishing gradients in deep networks.
Why ResNet is Powerful:
Able to train very deep networks (e.g., 50, 101, 152 layers) without degradation in performance.
Common ResNet Variants:
ResNet50, ResNet101, ResNet152.
3.6 Inception Networks:
Inception Modules:
Uses multi-scale convolutions (1x1, 3x3, 5x5) in parallel to capture features at different scales.
Key Features:
Efficient in terms of computation, while still being deep and accurate.
Inception v3 and v4 are widely used variants.
Practical Session: Fine-tuning a Pre-trained Model (Transfer Learning)
Objective: Use transfer learning to fine-tune a pre-trained model (VGG16 or ResNet50) on a new dataset.
Dataset: Choose a custom dataset or a smaller dataset like Cats vs. Dogs or Flowers.
Key Steps:
Step 1: Load a Pre-trained Model
Use TensorFlow/Keras to load the pre-trained VGG16 or ResNet50 model, with pre-trained weights from ImageNet.
Step 2: Freezing Layers
Freeze the earlier layers (i.e., prevent them from being updated during training) and only fine-tune the last few layers for the new task.
Step 3: Replace the Output Layer
Replace the top fully connected layer with a new classification head specific to the new task (e.g., 2 classes for Cats vs. Dogs).
Use softmax activation for multi-class classification, sigmoid for binary classification.
Step 4: Compile and Train the Model
Compile the model using an appropriate optimizer (e.g., Adam) and a loss function (e.g., binary crossentropy for binary classification).
Train the model for a few epochs and monitor the performance on a validation set.
Step 5: Fine-tuning
Unfreeze some of the deeper layers in the pre-trained model and re-train with a lower learning rate for fine-tuning.
Continue training for a few more epochs.
Step 6: Evaluation
Evaluate the model on the test set to compare the results before and after fine-tuning.
Observe the improvement in performance due to fine-tuning.
Assignment for Week 3:
Coding Assignment:
Implement transfer learning with ResNet50 or VGG16 on a different dataset.
Perform two experiments:
Use the pre-trained model as a feature extractor (freeze all layers except the final one).
Fine-tune the deeper layers of the pre-trained model.
Analysis:
Compare the performance of both approaches.
Discuss when to use feature extraction versus fine-tuning.
Reading Assignment:
Read Chapter 4 of "Advanced Applied Deep Learning" by Umberto Michelucci.
Focus on understanding the use cases for transfer learning and pre-trained models.
Summary of Key Concepts:
Transfer learning: Using pre-trained models for new tasks to save time and improve performance.
Feature extraction vs. fine-tuning: Two main approaches to transfer learning.
VGG, ResNet, and Inception: Popular pre-trained models and their key characteristics.
Practical experience in fine-tuning a pre-trained model using TensorFlow/Keras.
This week introduces students to the highly practical and impactful technique of transfer learning, allowing them to apply pre-trained models for new tasks. They will gain hands-on experience working with some of the most widely used architectures in deep learning, setting the stage for more advanced model tuning in the upcoming weeks.