Advanced Applied Deep Learning
Practice Course
Sheng Yun Wu
Practice Course
Sheng Yun Wu
In Week 11, students will focus on evaluating and improving object detection performance. This includes understanding key performance metrics such as mean Average Precision (mAP), Intersection over Union (IoU), Precision, Recall, and optimizing these metrics through techniques such as hard negative mining, data balancing, and model ensembling. The week also covers error analysis and using advanced tools for model diagnostics.
Description:
This example explains the key performance metrics for object detection tasks, including Precision, Recall, IoU (Intersection over Union), and mean Average Precision (mAP). These metrics provide a quantitative measure of the model's performance.
No Code for this example – Theoretical Explanation
IoU: Measures the overlap between predicted and ground truth bounding boxes.
Precision and Recall: Evaluate the accuracy and coverage of detected objects.
mAP: Aggregates precision-recall curve information to assess model performance.
Description:
This example demonstrates how to calculate IoU between predicted and ground truth bounding boxes, a key metric used to determine the quality of detections.
def compute_iou(box1, box2):
# box1 and box2 are [x1, y1, x2, y2]
x1_inter = max(box1[0], box2[0])
y1_inter = max(box1[1], box2[1])
x2_inter = min(box1[2], box2[2])
y2_inter = min(box1[3], box2[3])
intersection_area = max(0, x2_inter - x1_inter) * max(0, y2_inter - y1_inter)
box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
iou = intersection_area / float(box1_area + box2_area - intersection_area)
return iou
# Example usage
box1 = [100, 100, 200, 200]
box2 = [150, 150, 250, 250]
iou = compute_iou(box1, box2)
print(f"IoU: {iou}")
Description:
This example shows how to compute Precision and Recall for an object detection model to evaluate the model's ability to correctly detect objects.
from sklearn.metrics import precision_score, recall_score
# True labels and predicted labels
true_labels = [1, 0, 0, 1, 1]
pred_labels = [1, 0, 1, 1, 0]
# Calculate Precision and Recall
precision = precision_score(true_labels, pred_labels)
recall = recall_score(true_labels, pred_labels)
print(f"Precision: {precision}")
print(f"Recall: {recall}")
Description:
This example demonstrates how to calculate the mean Average Precision (mAP), a key metric for object detection performance evaluation.
from sklearn.metrics import average_precision_score
# Ground truth labels and predicted scores for multiple objects
true_labels = [1, 0, 1, 1, 0, 1]
pred_scores = [0.9, 0.6, 0.8, 0.75, 0.5, 0.95]
# Calculate mAP
mAP = average_precision_score(true_labels, pred_scores)
print(f"mean Average Precision (mAP): {mAP}")
Description:
This example introduces hard negative mining, a technique used to improve object detection performance by focusing on hard-to-classify examples, such as background regions falsely classified as objects.
import numpy as np
# Example dataset: true labels and predicted scores
predictions = np.array([0.1, 0.4, 0.9, 0.8, 0.3])
true_labels = np.array([0, 0, 1, 1, 0])
# Select hard negatives (background predicted as object)
threshold = 0.5
hard_negatives = predictions[(predictions >= threshold) & (true_labels == 0)]
print(f"Hard Negatives: {hard_negatives}")
Description:
This example explains how to perform data balancing in object detection to ensure that all object classes are equally represented in the training dataset.
from sklearn.utils import resample
# Example dataset with imbalance
data = [['dog', 1], ['cat', 0], ['dog', 1], ['dog', 1], ['cat', 0]]
# Separate classes
dogs = [x for x in data if x[0] == 'dog']
cats = [x for x in data if x[0] == 'cat']
# Balance the dataset by resampling
cats_upsampled = resample(cats, replace=True, n_samples=len(dogs), random_state=42)
# Combine the upsampled minority class with the majority class
balanced_data = dogs + cats_upsampled
print(balanced_data)
Description:
This example teaches students how to perform error analysis by identifying and visualizing false positives and false negatives in object detection results.
import cv2
# Example ground truth and predicted bounding boxes
true_boxes = [[100, 150, 200, 250]] # Ground truth boxes
pred_boxes = [[110, 160, 210, 260]] # Predicted boxes
# Compute IoU and classify false positives and false negatives
iou = compute_iou(true_boxes[0], pred_boxes[0])
threshold = 0.5
if iou < threshold:
print("False Positive")
if len(true_boxes) == 0:
print("False Negative")
# Visualize false positives on the image
image = cv2.imread('image.jpg')
cv2.rectangle(image, (110, 160), (210, 260), (0, 0, 255), 2) # False positive in red
cv2.imshow("False Positive Example", image)
cv2.waitKey(0)
Description:
This example demonstrates how to use model ensembling to combine the predictions of multiple object detection models, improving overall performance by reducing false positives and false negatives.
import numpy as np
# Example predictions from three different models
model1_boxes = np.array([[100, 150, 200, 250], [300, 350, 400, 450]])
model2_boxes = np.array([[105, 155, 205, 255], [295, 345, 395, 445]])
model3_boxes = np.array([[110, 160, 210, 260], [290, 340, 390, 440]])
# Ensemble predictions (simple average of bounding boxes)
ensemble_boxes = (model1_boxes + model2_boxes + model3_boxes) / 3
print(f"Ensemble bounding boxes: {ensemble_boxes}")
Description:
This example introduces hyperparameter tuning for object detection models. Students will experiment with different learning rates, batch sizes, and anchor box sizes to find the optimal configuration.
# Hyperparameter tuning for learning rate and batch size
learning_rates = [0.001, 0.0001]
batch_sizes = [8, 16]
for lr in learning_rates:
for batch_size in batch_sizes:
print(f"Training with learning rate: {lr}, batch size: {batch_size}")
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lr), loss='categorical_crossentropy')
model.fit(train_dataset, batch_size=batch_size, epochs=10)
Description:
This example demonstrates how to use TensorBoard to visualize training metrics (loss, accuracy, mAP) and perform diagnostics on object detection models to understand performance bottlenecks.
import tensorflow as tf
# Set up TensorBoard callback
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir='./logs')
# Train model with TensorBoard
model.fit(train_dataset, epochs=10, validation_data=val_dataset, callbacks=[tensorboard_callback])
# Launch TensorBoard
%tensorboard --logdir ./logs
Objective: Evaluate and optimize object detection models using performance metrics, advanced techniques, and diagnostics.
Skills Developed:
Understand and compute key object detection metrics (Precision, Recall, IoU, mAP).
Apply techniques such as hard negative mining, data balancing, and model ensembling.
Perform error analysis and fine-tune hyperparameters for improved performance.
Use TensorBoard for model diagnostics and visualization.
Tools: PyTorch, TensorFlow, OpenCV, TensorBoard, Scikit-learn.
By the end of Week 11, students will have a strong understanding of how to evaluate object detection models and apply advanced techniques to improve performance, making them capable of analyzing real-world detection systems for accuracy and efficiency.